Not lexing correctly
Your lexer does not handle strings properly:
from json import dumps, loads
double_quotes = '"A quoted random UTF-8 string €.'
json = dumps(double_quotes)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()
Results in:
python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
l.parse()
File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
tok = self.getNextToken()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '
Which, as you can see from the JSON library should be a totally fine JSON string.
You should consider writing unit test for your lexer and testing it against valid JSON input such as above.
Outdated Python version
Judging from the use of the print
statement you seem to be using Python 2.x which is unmaintained for two years now.
Switch to Python 3 for all future projects.