0
\$\begingroup\$

I need to parse a simple JSON string (flat JSON, no hierarchy) for keys and values, and there is a system constraint that I cannot use any built-in JSON library and can only read a string once due to latency requirements. I need to use Python 2.7.x series and cannot use a higher version. It looks a bit ugly and I'm looking for advice in this problem, for both efficiency and reliability.

def findKeyValue(str1, lastndex):
    keyIndexBegin=str1.rfind(":",0,lastndex)
    keyIndexBegin=str1.rfind("\"",0,keyIndexBegin)
    keyIndexBegin=str1.rfind("\"",0,keyIndexBegin)
    keyIndexBegin+=1
    keyIndexEnd=str1.find("\"",keyIndexBegin)
    print "Key is: " + str1[keyIndexBegin:keyIndexEnd]

    valueBeginIndex=str1.find(":",keyIndexEnd+1)
    valueEndIndex=index
    print "Value is: "+ str1[valueBeginIndex:valueEndIndex]

if __name__ == "__main__":

    #JSONString = '{ "id": 1, "name": "A green door", "price": 12.50, "tags": ["home", "green"]}'
    JSONString = '{ "id": 1, "name": "A green door", "price": 12.50, "tags": "home green"}'
    index = JSONString.find(",")
    while index != -1:
        findKeyValue(JSONString, index)
        index = JSONString.find(",",index+1)

    # handle the last one
    index=len(JSONString)-1
    findKeyValue(JSONString,index)
\$\endgroup\$
  • \$\begingroup\$ In current problem, the value cannot contain ",", but if it contains ",", how to write more reliable code? Your advice is appreciated. I commented one string with "," as part of values. \$\endgroup\$ – Lin Ma Feb 27 '16 at 2:40
  • 2
    \$\begingroup\$ Please read What you may and may not do after receiving answers \$\endgroup\$ – holroy Feb 27 '16 at 23:05
2
\$\begingroup\$

Some style comments

First a few code style comments:

  • Follow style guidelines for naming variables and functions – According to PEP8 the guidelines indicate to use snake_case for variable and function names.
  • Add spaces after commas and around operators – Change JSONString.find(",",index+1) by adding spaces to open up the statement into JSONString.find(",", index + 1). This makes your code easier to read and understand.

Algorithmic considerations

Instead of doing a backwards search from a random comma, I would rather change the algorithm to doing forward searches and picking id's, values or list of values. Changing into forwarding searching algorithm would also allow for proper handling of commas and escaping of quotes and similar.

Some things you should look for when doing a proper search:

  • Start- and end-quotes, i.e. "simple"
  • Escaping of quotes using backslashes, i.e. "a \" quote" and possibly "a backslash, \\"
  • Colon's separating id's and values
  • Start- and end-brackets, like in "id" : [ "one", "two" ]

Secondly I would make the functions actually return values and not printing them directly. In your case I would probably make the function return a triplet of id, value, index where the index is used for offsetting the next run.

\$\endgroup\$
  • \$\begingroup\$ Thanks holroy, awesome reply and vote up. For your comments, "Escaping of quotes using backslashes", do you mean escape processing in original JSON string? \$\endgroup\$ – Lin Ma Feb 27 '16 at 7:20
  • 1
    \$\begingroup\$ @LinMa Yes, escaping to allow quotes and backslashes \$\endgroup\$ – holroy Feb 27 '16 at 10:24
  • 1
    \$\begingroup\$ @LinMa, You are not supposed to add/edit the code of your question. You should rather ask a new question with the updated code. I've rolled back your edit. \$\endgroup\$ – holroy Feb 27 '16 at 22:59
  • 1
    \$\begingroup\$ I've only used the library versions, but have knowledge related to how to do it, so that I could review your version. \$\endgroup\$ – holroy Feb 27 '16 at 23:00
  • 1
    \$\begingroup\$ @LinMa, It's actually recommended to post a question, watch the response, incorporate the response and come back for a second round! :-) \$\endgroup\$ – holroy Feb 27 '16 at 23:24
2
\$\begingroup\$

High-level considerations

  • Given a set of key-value pairs encoded as JSON, I’d really like to get them into a Python dictionary. How do I do that with your code? It’s not obvious.

    Ideally there’d be a function json_to_dict() into which I can pass a JSON string, and get a Python dictionary back.

    Most of the work is already there

  • Your findKeyValue() function isn’t very re-usable – it does some work, and prints the results to stdout, but it’s difficult for the caller to make good use of those values.

    It would be better if you returned a (key, value) tuple, and didn’t print anything – that can be used by the caller to construct a full dictionary.

  • There aren’t any comments. Parsing JSON is non-trivial, and it’s not immediately obvious why you’ve written findKeyValue() in the way that you have. More comments would make it easier to follow the code, spot bugs, and maintain in future.

  • Read PEP 8, the Python style guide, in particular the sections on variable names and spaces around operators.

Bugs/strange output

Here are a few dicts I found that product unusual or misleading output. I changed the value of JSONString, ran your code, and looked at the output. I found several edge cases that aren’t correctly covered.

(I’ve changed the printing in findKeyValue to use repr(), so we see the bounds of the strings.)

  • The empty dict, i.e. {}. This is just nonsense:

    Key is: '{'
    Value is: ''
    

    That seems plain odd.

  • A dict with empty values, e.g. {"0": ""}. Plenty of those produce very strange looking output:

    Key is: '0'
    Value is: ': ""'`
    

  • A dict with empty keys, e.g. {"": "0"}. Once again, some odd output:

    Key is: ''
    Value is: ': "0"'
    

  • Backslash escaping, e.g. {"\\": ""}. Two problems here:

    Key is: '\\\\'
    Value is: ': ""'
    

    The original key was a single backslash, doubled-up for escaping in the JSON – but that’s not the number I got back. And the value is bizarre.

To find some of these examples, I used hypothesis to generate random examples of key/value dictionaries, fired them into your code (with some small changes to return a dict), and tested:

assert parseFromFindKeyValue(json.dumps(d)) == d

It’s a powerful and fast way to find inconsistencies.

You may decide that these are all edge cases, and you’re going to leave them broken. That might be fine, but you should at least be aware of their existence.

(And if this was assessed work, I would be more impressed by somebody who handled these cases correctly.)

\$\endgroup\$
  • \$\begingroup\$ Thanks alexwlchan, I have updated code and fix some problems you mentioned. I post my new code in my original post and please refer to method parseElegant, now I met with issues about how to write elegant and reliable code to deal with values which are not quote with "" (you can see my test case with JSONString2)? Your advice is appreciated. \$\endgroup\$ – Lin Ma Feb 27 '16 at 22:58
  • \$\begingroup\$ BTW, if you have simple JSON parser in Python or Java (which parse strings only once, easy to read), appreciate for your recommendation in advance. \$\endgroup\$ – Lin Ma Feb 27 '16 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.