Skip to main content
Code Review

Return to Answer

added 181 characters in body
Source Link
Richard Neumann
Richard Neumann
  • 6.2k
  • 21
  • 41

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

utf8_string = '"A quoted random UTF-8 string €.'
json = dumps(utf8_string)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the comparison to the json library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Use custom exceptions

Consider using a custom exception, such as

class JSONLexingError(Exception):
    """Indicate an error when lexing a JSON string."""

to indicate errors during lexing instead of using a bare RuntimeError.

PEP-8

It is unlawful for a Python code review to be missing the PEP-8 hammer. ;) Use PascalCase for class names (which you already do) and snake_case for function names.

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

utf8_string = '"A quoted random UTF-8 string €.'
json = dumps(utf8_string)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the comparison to the json library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Use custom exceptions

Consider using a custom exception, such as

class JSONLexingError(Exception):
    """Indicate an error when lexing a JSON string."""

to indicate errors during lexing instead of using a bare RuntimeError.

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

utf8_string = '"A quoted random UTF-8 string €.'
json = dumps(utf8_string)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the comparison to the json library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Use custom exceptions

Consider using a custom exception, such as

class JSONLexingError(Exception):
    """Indicate an error when lexing a JSON string."""

to indicate errors during lexing instead of using a bare RuntimeError.

PEP-8

It is unlawful for a Python code review to be missing the PEP-8 hammer. ;) Use PascalCase for class names (which you already do) and snake_case for function names.

added 254 characters in body
Source Link
Richard Neumann
Richard Neumann
  • 6.2k
  • 21
  • 41

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

double_quotesutf8_string = '"A quoted random UTF-8 string €.'
json = dumps(double_quotesutf8_string)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the JSONcomparison to the json library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Use custom exceptions

Consider using a custom exception, such as

class JSONLexingError(Exception):
    """Indicate an error when lexing a JSON string."""

to indicate errors during lexing instead of using a bare RuntimeError.

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

double_quotes = '"A quoted random UTF-8 string €.'
json = dumps(double_quotes)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the JSON library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

utf8_string = '"A quoted random UTF-8 string €.'
json = dumps(utf8_string)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the comparison to the json library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.

Use custom exceptions

Consider using a custom exception, such as

class JSONLexingError(Exception):
    """Indicate an error when lexing a JSON string."""

to indicate errors during lexing instead of using a bare RuntimeError.

Source Link
Richard Neumann
Richard Neumann
  • 6.2k
  • 21
  • 41

Not lexing correctly

Your lexer does not handle strings properly:

from json import dumps, loads

double_quotes = '"A quoted random UTF-8 string €.'
json = dumps(double_quotes)
print(loads(json))
l = Lexer(json)
l.parse()
l.pprint()

Results in:

python .\jsonlex.py
"A quoted random UTF-8 string €.
Traceback (most recent call last):
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 59, in <module>
    l.parse()
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 49, in parse
    tok = self.getNextToken()
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Richard Neumann\Desktop\jsonlex.py", line 45, in getNextToken
    raise RuntimeError("Unrecognized token at position %d -- '%s'" % (self.idx, substring[:5]))
RuntimeError: Unrecognized token at position 0 -- '"\"A '

Which, as you can see from the JSON library should be a totally fine JSON string.

You should consider writing unit test for your lexer and testing it against valid JSON input such as above.

Outdated Python version

Judging from the use of the print statement you seem to be using Python 2.x which is unmaintained for two years now. Switch to Python 3 for all future projects.