The problem is that a list made from the rows of a file with ~500M rows will have millions of entries, and that’s going to really hamper Python. As long as you’re using a list, I don’t think there’s any magic you can do here to magically fix that problem.
So you should ask: do I need a list? You may be able to make to do with a generator, which is much faster and more memory efficient (because it only computes one element at a time, not precomputing everything).
For a generator, I’m assuming that the customer IDs are monotonically increasing. If not, you really do have to go through the entire file to be sure you’ve got everything for each group. But your existing code seems to assume that, so I’ll assume I can as well.
Here’s a slight reworking of your code to use generators:
Reads customer IDs and events from a CSV file. Yields events for
with open(myfile, 'rb') as csvfile:
reader = csv.reader(csvfile, delimiter='|', quotechar='"')
next(reader) ## skip header
current_cust_id = 1
seq = 
for row in reader:
cust_id, event = row
if ('word' in event) or ('say' in event):
while int(cust_id) != current_cust_id:
seq = 
current_cust_id += 1
I’ve done away with the massive list (
sequence), and we just yield the list of per-customer events whenever the customer ID changes.
A few other comments on your code:
- Put your imports at the top of your file; don’t hide them inside a function.
- There isn’t much in the way of documentation. Your function should at least have a docstring, and some more comments would be nice. For example, tell me why we’re skipping lines with “Say” or “Word”. I can read that the code is doing it, but I can’t read your mind to find out why.
You were assuming that customer IDs increase by 1 every time, and that there’s never going to be a gap. What happens if you run your code against:
Maybe you know that will be the case; I wasn’t sure, so my code is defensive against this possibility.
Rather than doing
seq.extend([row]), it’s much cleaner to do
seq.append(row). It saves you creating an additional list.