-1
\$\begingroup\$

I'm writing some Python code that will handle huge datasets.

On a small scale, this snippet works fine, but as the datasets become bigger it grinds my script to a halt (I know it is this part as I have surrounded all the possible offenders with timers, and this one takes the longest):

#example variables
neighbours = [1, 5, 2, 6, 3, 8]
coverage = {1:0, 2:3, 3:1, 4:4, 5:1, 6:2, 7:4, 8:1}
is_repeat = True
for n in neighbours:
    if coverage.get(n,0)<2:
        is_repeat = False
        break

The loop prevents the same list of values being handled more than twice - if at least one element of the list has not been handled twice then the list is still handled further on making the is_repeat flag crucial to the next steps.

How can I optimise this section?

\$\endgroup\$
  • \$\begingroup\$ What is the purpose of your loop? Is is_repeat flag the crucial variable that coordinates further processing? \$\endgroup\$ – RomanPerekhrest Jan 2 at 10:33
  • \$\begingroup\$ @RomanPerekhrest the loop prevents the same list of values being handled more than twice - if at least one element of the list has not been handled twice then the list is still handled further on, so the is_repeat flag is indeed crucial to the next steps. I will edit my question to reflect that \$\endgroup\$ – Jessica Chambers Jan 2 at 10:36
  • \$\begingroup\$ I don't think there's a way to optimize this piece of code in isolation in such a way as to have it do the same thing, but maybe there's some other way to accomplish the wider aim? Could you maybe keep track of the aggregate "is_repeat" state for this list of values as you're handling it the previous time so you don't need to go back through it later to re-compute that state? \$\endgroup\$ – Samwise Jan 2 at 15:52
2
\$\begingroup\$

The more "Pythonic" way (aka using Pythonic syntactic sugar to make the code more concise) of writing this:

is_repeat = True
for n in neighbours:
    if coverage.get(n,0)<2:
        is_repeat = False
        break

would be:

is_repeat = all([coverage.get(n, 0) >= 2 for n in neighbors])

but I'm not sure this will necessarily be any more efficient.

Is is_repeat true most of the time (forcing this iteration to go through all of the neighbors)? If so, maybe the thing to focus on is optimizing earlier detection of that case in some other part of the script where it'll be easier to figure out.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ It's True about half of the time. Thank you for your answer though! I will see if there is anything I can do earlier in the code but this has given me some helpful pointers :) \$\endgroup\$ – Jessica Chambers Jan 3 at 7:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.