3
\$\begingroup\$

I have a list of dictionaries, with keys 'a', 'n', 'o', 'u'. Is there a way to speed up this calculation, for instance with NumPy? There are tens of thousands of items in the list.

The data is drawn from a database, so I must live with that it's in the form of a list of dictionaries originally.

x = n = o = u = 0
for entry in indata:
    x += (entry['a']) * entry['n']  # n - number of data points
    n += entry['n']
    o += entry['o']
    u += entry['u']

    loops += 1

average = int(round(x / n)), n, o, u
\$\endgroup\$
  • \$\begingroup\$ what are you trying to calculate? \$\endgroup\$ – Oved D Oct 22 '12 at 15:40
  • \$\begingroup\$ @OvedD, an average of all 'a' values and a bunch of sums for the others. \$\endgroup\$ – Prof. Falken Oct 22 '12 at 15:44
2
\$\begingroup\$

I'm not sure if there really is a better way to do this. The best I could come up with is:

import itertools
from collections import Counter

def convDict(inDict):
    inDict['a'] = inDict['a'] * inDict['n']
    return Counter(inDict)

average = sum(itertools.imap(convDict, inData), Counter())
average['a'] = average['a'] / average['n']

But I'm still not sure if that is better than what you originally had.

Counter is a subclass of dict. You can get items from them the same way you get items from a normal dict. One of the most important differences is that the Counter will not raise an Exception if you try to select a non-existant item, it will instead return 0.

\$\endgroup\$
  • \$\begingroup\$ Interesting but I don't see how I can replace my code with that. +1 though \$\endgroup\$ – Prof. Falken Oct 22 '12 at 16:02
  • \$\begingroup\$ I realized after posting this that you wanted to do some multiplication before summing. Ill look into that \$\endgroup\$ – Matt Oct 22 '12 at 16:06
  • \$\begingroup\$ Wouldn't that require to access the full list twice instead of once? In that case, it must be slower. \$\endgroup\$ – Prof. Falken Oct 22 '12 at 16:16
  • \$\begingroup\$ @AmigableClarkKant Yeah, I just got done editing the question to state that. It is probably slower than what you already have. I'm not sure how much better Counter is (if any) than manually summing. \$\endgroup\$ – Matt Oct 22 '12 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.