# Tokenize a mathematical statement string

The function tokenize generates tokens from a mathematical string.

It can chain characters together to form longer tokens.

This can be used to build a parse tree, in order to evaluate the math expression.

def get_char_type(char):
"""
Returns a string denoting the type of char.

>>> get_char_type('x')
'letter'
"""
operators = {'*', '/', '+', '-', '=', \
'%', '~', '/', '&', '|', '>', '<'}
if char.isdigit():
char_type = 'number'
elif char.isalpha():
char_type = 'letter'
elif char in operators:
char_type = 'operator'
else:
char_type = 'other'
return char_type

def tokenize(string):
"""
Generates tokens from a mathematical statement string.

>>> list(tokenize('3 + (4 ∗ 5)'))
['3', '+', '(', '4', '∗', '5', ')']

>>> list(tokenize('x+=10'))
['x', '+=', '10']

>>> list(tokenize('(729 + 4 * variable) ** 22 // 3'))
['(', '729', '+', '4', '*', 'variable', ')', '**', '22', '//', '3']
"""
token_type = get_char_type(string[0])
token = ''
for char in string:
if char == ' ':
continue  # Spaces are not included
new_type = get_char_type(char)
if new_type != token_type:  # A new type of token has been found
yield token
token_type = new_type
token = ''
token += char
if len(token) > 0:
yield token


# Possible errors/suspicious statements

In def get_char_type(char), you have repeated the statement

if char.isdigit():
char_type = 'number'


twice. This is probably unintentional.

Note: The OP has removed this error in an edit before this answer was posted.

# Suggestions

Instead of explicitly checking for whitespace in your generator expression and using a continue to skip whitespace, you can filter the input string by a regex (regular expression) to remove all whitespace before you run the loop over the characters of the string. Kind of like this:

whitespace_pattern = re.compile(r"\s")
for char in whitespace_pattern.sub("",string):
new_type = get_char_type(char)
...


for char in string:
if char == ' ':
continue  # Spaces are not included
...


This looks a bit better, although it might not be as performant as your original code. You will also need an import re somewhere at the top of your code.

Also, instead of that if-elif-else in get_token_type(), what about this?

def get_char_type(char):
"""
Returns a string denoting the type of char.

>>> get_char_type('x')
'letter'
"""
operators = {'*', '/', '+', '-', '=', '%', '~', '/', '&', '|', '>', '<'}
characteristics_of_token_types = {
(lambda char: char.isdigit()): 'number',
(lambda char: char.isalpha()): 'letter',
(lambda char: char in operators): 'operator',
(lambda char: len(char) == 1): 'other' #default case, if extending functionality, put any new case before this
}
for characteristic in characteristics_of_token_types.keys():
if(characteristic(char)):
return characteristics_of_token_types[characteristic]


Why is this better? If you want to add more types, it just becomes a matter of specifying a characteristic of a token type (via a lambda dictionary key), and a name for the type (as the lambda key's value). It makes a bit more sense to me logically, however, opinions vary.

Also, note that what I did above is somewhat along the canonical path of emulating a switch-case in Python, which is, in fact, a better fit for this problem than if-else. I was looking for something like guards in Haskell, as I've recently become a functional programming guy, but I've got to make do with Python, so this is what I could think up.

Finally, I did not bother making your token-parsing code more "functional" as I think the result would be more complicated than what you already have.

# Style

You have docstrings. That's a definite plus. You also follow the proper naming conventions for methods and variables, and there is adequate commenting, so no issues there.

However, a minor peeve - do you really need that line continuation when declaring the operators list? It doesn't look like you do, the whole thing fits in a line in below 40 characters all right.