# How to find the unique values from the JSON file?

I have JSON files of size data_large(150.1mb). The content inside the file is of type [{"score": 68},{"score": 78}]. I need to find the list of unique scores from each file.

This is what I'm doing:

import ijson  # since JSON file is large, hence making use of ijson

f = open ('data_large')
content = ijson.items(f, 'item') # json loads quickly here as compared to when json.load(f) is used.
print set(i['score'] for i in content) #this line is actually taking a long time to get processed.


Can I make the last line more efficient? It's currently taking 201 secs to execute.

• ijson is an iterative parser, so ijson.items will not load the whole file. The set creation likely isn’t the bottleneck; reading the huge file is. – poke Jan 4 '14 at 20:35
• @poke - When I remove the print statement, it executes within no time. But when applying the logic for unique value it takes time there. Can I make it any more efficient? – PythonEnthusiast Jan 5 '14 at 4:57
• You mean when you just remove the print or the whole line (including the set(…))? If you remove the whole line, then of course nothing is processed because ultimately, the file is only opened but never read from. And no, I doubt you can make it more efficient than this; you do have to read a 150 MB file after all, and that’s always slow. – poke Jan 5 '14 at 15:28

So… I have spent a bit time on this and generated myself a random JSON file matching your format. Mine is 167 MB in size, so not too far off from yours.

First of all, ijson’s performance is terrible. I’ve tried it multiple times and it’s very slow for me (much slower than your results actually). While the idea sounds great, it took very long to read and parse the file.

Using the normal json module gave me much better results. In fact, reading and parsing the file took only ~20 seconds, of which reading the file was just a very small fraction. Most of the time was actually spent on extracting the scores; generating the set from them again is very quick.

So to speed this up, we obviously want to avoid creating all those intermediary objects in between, and just extract the data we are interested in. In your case where the file has a rather simple structure, we don’t really need to parse it as a JSON object but we can just operate on the string directly and search for the scores.

As we discussed this on the SO Python chat, Jon Clements came up with this solution:

with open('file.json') as f:

• Ijson is slow by default because it uses the native Python backend. But it can use the C library "yajl" if available. The yajl backend can be imported directly: import ijson.backends.yajl2 as ijson. – isagalaev May 21 '15 at 16:35