As an avid user of esoteric programming languages, I'm currently designing one, interpreted in Python.
The language will be stack-based and scripts will be a mix of symbols, letters, and digits, like so:
This will be read from the file and passed to a new
Script object, to tokenize. It needs to be tokenized into a list of strings, where adjacent digits are grouped together and converted into ints, but letters are individual, resulting in this array:
['a', 'b', 123, 'c', '-', 43, '=', 'f', 54, 'g', 6, '!', 6]
Currently, I am doing this using regex, with the following code (
get_tokens is the main method):
import re class Script: def __init__(self, text): self._tokens = Script.get_tokens(text) def get_tokens(text): """ Static tokenizer function. Uses regex to turn the string into a list of tokens. Groups adjacent digits together and converts to integers. """ tokens = re.findall('\d+|[^\d]', text) for i in range(len(tokens)): if tokens[i].isdigit(): tokens[i] = int(tokens[i]) return tokens def run(self): raise NotImplementedError()
The resulting tokens can be seen with
I have a few questions about my method:
- Is there any more Pythonic or elegant way to do this? (e.g, an alternative instead of regexing)
- Should I really be using a static function like that, or should I move it outside the class?
- If the loop is to be kept, should I be using
for i,t in enumerate(tokens): if t.isdigit(): tokens[i] = int(t)