Due to another module coupled to my function, I can only receive the input to my part in the form of a JSON object structured roughly like this:
[{'id':0, 'y':4, 'value':25},{'id':0, 'y':2, 'value':254}]
Note that I do know that the data will arrive in exactly that format. Now, I need to cast this to lists such that I can pass it to the constructor of scipy.sparse.coo_matrix()
.
Since I have frequent incoming calls, I want to perform this cast as quickly as possible, which is why I am concerned with the optimization of this operation. Below are three different approaches that come up with this type of question. Note that the casting itself has been frequently addressed on Stackoverflow, even with respect to performance in some cases (mostly in terms of list comprehension), but I could not find anything that would give me an optimal solution.
To quickly address the three different methods I use:
- evaluate each line as a tuple of the dictionary values. Comparable in speed with something like
[[el['id'], el['y'], el['values']] for el in x]
. - Let
pandas
do the casting for you. Extremly slow. - Cast three separate lists. Since the allocation of lists in list comprehensions is way slower (compare
[[el['id']] for el in x]
to[el['id'] for el in x]
), this seems to be the currently best-performing solution.
According to the articles I found, list comprehensions outperform any python-native method using .append()
, but I might add an example and timing for that later.
The benchmarks are as follows:
import timeit as ti
# 2.107 seconds
print(ti.timeit("z = [tuple(el.values()) for el in x]",
setup="import random; x = [{'id':random.randint(0,5),'y':random.randint(0,5), 'value':random.randint(0,500)}]*60000",
number=100))
# 8.93 seconds
print(ti.timeit("z = pd.DataFrame(x)",
setup="import pandas as pd; import random; x = [{'id':random.randint(0,5),'y':random.randint(0,5), 'value':random.randint(0,500)}]*60000",
number=100))
# 0.717 seconds
print(ti.timeit("z1 = [el['id'] for el in x]; z2 = [el['y'] for el in x]; z3 = [el['value'] for el in x]",
setup="import random; x = [{'id':random.randint(0,5),'y':random.randint(0,5), 'value':random.randint(0,500)}]*60000",
number=100))
I also include the "raw code" for the three snippets:
import random
import pandas as pd
if __name__ == "__main__":
# ignore the fact that this actually isn't random for individual values
x = [{'id':random.randint(0,5),'y':random.randint(0,5), 'value':random.randint(0,500)}]*60000
# first method
z1 = [tuple(el.values()) for el in x]
# second method
z2 = pd.DataFrame(x)
# third method
z_3a = [el['id'] for el in x]
z_3b = [el['y'] for el in x]
z_3c = [el['value'] for el in x]
The question is whether there is any significant improvement on this (maybe by using a specialized library I don't know of, or any trick with numpy, etc.) to easily improve the speed on this. I'm currently assuming that, following the 80/20 principle, it is unlikely I'll get more performance out of this without spending a lot more effort on it...