Skip to main content
1 of 3

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

double_quotes = '"A quoted random UTF-8 string €.'
json = dumps(double_quotes)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the JSON library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.