How to find the unique values from the JSON file?

Question

I have JSON files of size data_large(150.1mb). The content inside the file is of type [{"score": 68},{"score": 78}]. I need to find the list of unique scores from each file.

This is what I'm doing:

import ijson  # since JSON file is large, hence making use of ijson

f = open ('data_large')
content = ijson.items(f, 'item') # json loads quickly here as compared to when json.load(f) is used.
print set(i['score'] for i in content) #this line is actually taking a long time to get processed.

Can I make the last line more efficient? It's currently taking 201 secs to execute.

Answer 1

So… I have spent a bit time on this and generated myself a random JSON file matching your format. Mine is 167 MB in size, so not too far off from yours.

First of all, ijson’s performance is terrible. I’ve tried it multiple times and it’s very slow for me (much slower than your results actually). While the idea sounds great, it took very long to read and parse the file.

Using the normal json module gave me much better results. In fact, reading and parsing the file took only ~20 seconds, of which reading the file was just a very small fraction. Most of the time was actually spent on extracting the scores; generating the set from them again is very quick.

So to speed this up, we obviously want to avoid creating all those intermediary objects in between, and just extract the data we are interested in. In your case where the file has a rather simple structure, we don’t really need to parse it as a JSON object but we can just operate on the string directly and search for the scores.

As we discussed this on the SO Python chat, Jon Clements came up with this solution:

with open('file.json') as f:
    content = f.read()
    print len(set(m.group(1) for m in re.finditer('"score": (\d+)', content)))

This in fact runs in a few seconds, 8 on my machine, and is a multitude faster than anything that utilizes JSON. Of course this highly depends on the format of the file, so if you have additional data inside (which you left out in the question) you might need to adjust this.