JSON string parsing

Question

I need to parse a simple JSON string (flat JSON, no hierarchy) for keys and values, and there is a system constraint that I cannot use any built-in JSON library and can only read a string once due to latency requirements. I need to use Python 2.7.x series and cannot use a higher version. It looks a bit ugly and I'm looking for advice in this problem, for both efficiency and reliability.

def findKeyValue(str1, lastndex):
    keyIndexBegin=str1.rfind(":",0,lastndex)
    keyIndexBegin=str1.rfind("\"",0,keyIndexBegin)
    keyIndexBegin=str1.rfind("\"",0,keyIndexBegin)
    keyIndexBegin+=1
    keyIndexEnd=str1.find("\"",keyIndexBegin)
    print "Key is: " + str1[keyIndexBegin:keyIndexEnd]

    valueBeginIndex=str1.find(":",keyIndexEnd+1)
    valueEndIndex=index
    print "Value is: "+ str1[valueBeginIndex:valueEndIndex]

if __name__ == "__main__":

    #JSONString = '{ "id": 1, "name": "A green door", "price": 12.50, "tags": ["home", "green"]}'
    JSONString = '{ "id": 1, "name": "A green door", "price": 12.50, "tags": "home green"}'
    index = JSONString.find(",")
    while index != -1:
        findKeyValue(JSONString, index)
        index = JSONString.find(",",index+1)

    # handle the last one
    index=len(JSONString)-1
    findKeyValue(JSONString,index)

Answer 1

Some style comments

First a few code style comments:

• Follow style guidelines for naming variables and functions – According to PEP8 the guidelines indicate to use snake_case for variable and function names.
• Add spaces after commas and around operators – Change JSONString.find(",",index+1) by adding spaces to open up the statement into JSONString.find(",", index + 1). This makes your code easier to read and understand.

Algorithmic considerations

Instead of doing a backwards search from a random comma, I would rather change the algorithm to doing forward searches and picking id's, values or list of values. Changing into forwarding searching algorithm would also allow for proper handling of commas and escaping of quotes and similar.

Some things you should look for when doing a proper search:

• Start- and end-quotes, i.e. "simple"
• Escaping of quotes using backslashes, i.e. "a \" quote" and possibly "a backslash, \\"
• Colon's separating id's and values
• Start- and end-brackets, like in "id" : [ "one", "two" ]

Secondly I would make the functions actually return values and not printing them directly. In your case I would probably make the function return a triplet of id, value, index where the index is used for offsetting the next run.

Answer 2

High-level considerations

• Given a set of key-value pairs encoded as JSON, I’d really like to get them into a Python dictionary. How do I do that with your code? It’s not obvious.

  Ideally there’d be a function json_to_dict() into which I can pass a JSON string, and get a Python dictionary back.

  Most of the work is already there

• Your findKeyValue() function isn’t very re-usable – it does some work, and prints the results to stdout, but it’s difficult for the caller to make good use of those values.

  It would be better if you returned a (key, value) tuple, and didn’t print anything – that can be used by the caller to construct a full dictionary.

• There aren’t any comments. Parsing JSON is non-trivial, and it’s not immediately obvious why you’ve written findKeyValue() in the way that you have. More comments would make it easier to follow the code, spot bugs, and maintain in future.

• Read PEP 8, the Python style guide, in particular the sections on variable names and spaces around operators.

Bugs/strange output

Here are a few dicts I found that product unusual or misleading output. I changed the value of JSONString, ran your code, and looked at the output. I found several edge cases that aren’t correctly covered.

(I’ve changed the printing in findKeyValue to use repr(), so we see the bounds of the strings.)

• The empty dict, i.e. {}. This is just nonsense:

  Key is: '{'
  Value is: ''
  

  That seems plain odd.

• A dict with empty values, e.g. {"0": ""}. Plenty of those produce very strange looking output:

  Key is: '0'
  Value is: ': ""'`
  

• A dict with empty keys, e.g. {"": "0"}. Once again, some odd output:

  Key is: ''
  Value is: ': "0"'
  

• Backslash escaping, e.g. {"\\": ""}. Two problems here:

  Key is: '\\\\'
  Value is: ': ""'
  

  The original key was a single backslash, doubled-up for escaping in the JSON – but that’s not the number I got back. And the value is bizarre.

To find some of these examples, I used hypothesis to generate random examples of key/value dictionaries, fired them into your code (with some small changes to return a dict), and tested:

assert parseFromFindKeyValue(json.dumps(d)) == d

It’s a powerful and fast way to find inconsistencies.

You may decide that these are all edge cases, and you’re going to leave them broken. That might be fine, but you should at least be aware of their existence.

(And if this was assessed work, I would be more impressed by somebody who handled these cases correctly.)