I am working on a P2P streaming sim (which I think is correct), however I've discovered a huge bottleneck in my code. During the run of my sim, the function I've included below gets called about 25000 times and takes about 560 seconds from a total of about 580 seconds (I used pycallgraph).
Can anyone identify any way I could speed the execution up, by optimizing the code (I think I won't be able to reduce the times it gets executed)? The basics needed to understand what the function does, is that the incoming messages consist of
nr_of_known_nodes_to_sender (or -1 if the sender departs the network),
time_to_live], while the
mCache entries these messages update, consist of the same data (except for the third field, which cannot be -1, instead if such a message is received the whole entry will be deleted), with the addition of a fifth field containing the update time.
def msg_io(self): # Node messages (not datastream) control # filter and sort by main key valid_sorted = sorted((el for el in self.incoming if el != 0), key=ig(1)) self.incoming =  # ensure identical keys have highest first element first valid_sorted.sort(key=ig(0), reverse=True) # group by second element grouped = groupby(valid_sorted, ig(1)) # take first element for each key internal = [next(item) for group, item in grouped] for j in internal: found = False if j > 0 : # We are only interested in non-expired messages for i in self.mCache[:]: if j == i: # If the current mCache entry corresponds to the current incoming message if j == -1: # If the message refers to a node departure self.mCache.remove(i) # We remove the node from the mCache self.partnerCache = [partner for partner in self.partnerCache if not partner.node_id == j] else: if j > i: # If the message isn't a leftover (i.e. lower sequence number) i[:4] = j[:] # We update the relevant mCache entry i = turns # We add the current time for the update found = True else: k = j[:] #k -= 1 id_to_node[i].incoming.append(k) # and we forward the message if not j == -1 and not found and (len(self.mCache) < maxNodeCache): # If we didn't have the node in the mCache temp = j[:] temp.append(turns) # We insert a timestamp... self.mCache.append(temp) # ...and add it to the mCache self.internal.remove(j)