So… I have spent a bit time on this and generated myself a random JSON file matching your format. Mine is 167 MB in size, so not too far off from yours.
First of all,
ijson’s performance is terrible. I’ve tried it multiple times and it’s very slow for me (much slower than your results actually). While the idea sounds great, it took very long to read and parse the file.
Using the normal
json module gave me much better results. In fact, reading and parsing the file took only ~20 seconds, of which reading the file was just a very small fraction. Most of the time was actually spent on extracting the scores; generating the set from them again is very quick.
So to speed this up, we obviously want to avoid creating all those intermediary objects in between, and just extract the data we are interested in. In your case where the file has a rather simple structure, we don’t really need to parse it as a JSON object but we can just operate on the string directly and search for the scores.
As we discussed this on the SO Python chat, Jon Clements came up with this solution:
with open('file.json') as f:
content = f.read()
print len(set(m.group(1) for m in re.finditer('"score": (\d+)', content)))
This in fact runs in a few seconds, 8 on my machine, and is a multitude faster than anything that utilizes JSON. Of course this highly depends on the format of the file, so if you have additional data inside (which you left out in the question) you might need to adjust this.