2
\$\begingroup\$

I have JSON files of size data_large(150.1mb). The content inside the file is of type [{"score": 68},{"score": 78}]. I need to find the list of unique scores from each file.

This is what I'm doing:

import ijson  # since JSON file is large, hence making use of ijson

f = open ('data_large')
content = ijson.items(f, 'item') # json loads quickly here as compared to when json.load(f) is used.
print set(i['score'] for i in content) #this line is actually taking a long time to get processed.

Can I make the last line more efficient? It's currently taking 201 secs to execute.

\$\endgroup\$
  • \$\begingroup\$ ijson is an iterative parser, so ijson.items will not load the whole file. The set creation likely isn’t the bottleneck; reading the huge file is. \$\endgroup\$ – poke Jan 4 '14 at 20:35
  • \$\begingroup\$ @poke - When I remove the print statement, it executes within no time. But when applying the logic for unique value it takes time there. Can I make it any more efficient? \$\endgroup\$ – PythonEnthusiast Jan 5 '14 at 4:57
  • \$\begingroup\$ You mean when you just remove the print or the whole line (including the set(…))? If you remove the whole line, then of course nothing is processed because ultimately, the file is only opened but never read from. And no, I doubt you can make it more efficient than this; you do have to read a 150 MB file after all, and that’s always slow. \$\endgroup\$ – poke Jan 5 '14 at 15:28
4
\$\begingroup\$

So… I have spent a bit time on this and generated myself a random JSON file matching your format. Mine is 167 MB in size, so not too far off from yours.

First of all, ijson’s performance is terrible. I’ve tried it multiple times and it’s very slow for me (much slower than your results actually). While the idea sounds great, it took very long to read and parse the file.

Using the normal json module gave me much better results. In fact, reading and parsing the file took only ~20 seconds, of which reading the file was just a very small fraction. Most of the time was actually spent on extracting the scores; generating the set from them again is very quick.

So to speed this up, we obviously want to avoid creating all those intermediary objects in between, and just extract the data we are interested in. In your case where the file has a rather simple structure, we don’t really need to parse it as a JSON object but we can just operate on the string directly and search for the scores.

As we discussed this on the SO Python chat, Jon Clements came up with this solution:

with open('file.json') as f:
    content = f.read()
    print len(set(m.group(1) for m in re.finditer('"score": (\d+)', content)))

This in fact runs in a few seconds, 8 on my machine, and is a multitude faster than anything that utilizes JSON. Of course this highly depends on the format of the file, so if you have additional data inside (which you left out in the question) you might need to adjust this.

\$\endgroup\$
  • \$\begingroup\$ Ijson is slow by default because it uses the native Python backend. But it can use the C library "yajl" if available. The yajl backend can be imported directly: import ijson.backends.yajl2 as ijson. \$\endgroup\$ – isagalaev May 21 '15 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.