Deep iterations with side effects [closed]

Question

Suppose I need to process a list of list of list of many objects, where the data usually comes from third party APIs.

For the sake of an example, lets assume we have many users, and for each user we get many days of data, and for each day you get many events. We want to process all of these objects and store in our database

I thought about some differents design patterns I could use, but I'm not sure what is considered to be the best approach, if any.

These are my thoughts:

1) Nested iterations and do the side-effect (save to database) at the lowest-level loop.

for user in users:
    for day in user.days:
        for event in day.event:
            processed_object = process_event(event)
            save_to_db(processed_object)

2) Similar to one, except that each nested iteration is 'hidden' in another method.

for user in users:
    process_user(user)

def process_user(user):
   for day in user.days:
        process_day(day)

def process_day(day):
    for event in day.events:
        process_event(event)

def process_event(event):
    save_to_db(event)

3) Return everything from deep in the nest and then do the side-effects afterwards.

def process_users(users):
    processed = []
    for user in users:
        processed.append(process_user(user))
    return processed

def process_user(user):
    processed = []
    for day in user.days:
        processed.append(process_day(day))
    return processed

def process_day(day):
    processed = []
    for event in day.events:
        processed.append(process_event)
    return processed

processed_events = process_users(users) 

for event in processed_events:
    save_to_db(event)

Is there a general, agreed on approach for these type of programs, where side-effects are involved?

Answer 1

Not sure what is "generally agreed upon", if anything, but in my opinion the first variant is by far the easiest to read and understand, let alone most compact.

As @Alex points out, the second and third variant give more flexibility, e.g., you can process the events of just a single user, or a single day, but judging from your question this seems not to be necessary.

Another variant would be: First collect the events in a list, then process them in a separate loop, i.e., a mixture of variant 1 and 3:

all_events = []
for user in users:
    for day in user.days:
        all_events.extend(day.event)

for event in all_events:
    processed_object = process_event(event)
    save_to_db(processed_object)

You could also use a list comprehension for the first part...

all_events = [event for user in users for day in user.days for event in day.event]

...or even for the whole thing (although I don't think this is very readable).

[save_to_db(process_event(event)) for user in ... ]

Answer 2

Your third approach is the most pythonic, in the sense that you create a function for each specific purpose, avoiding repetition and giving the user more options (process the users, only the days, etc...) and the option to save or not to save the processed data to the database.

It is also the fastest. If you benchmark it, that kind of approach (separating things into functions) will almost always be the faster one with Python.

Using your first approach it took roughly 0.00027 seconds to process a list of 100 events for each day, having two days for each user and two users.

Using the second approach it takes roughly 0.00018 seconds.

Using the third approach it takes roughly 0.00012 seconds.

Here are all the approaches and their output: http://pastebin.com/vU12HaQW

Notice your third approach yields a list of list of lists, whereas the first and the second return a list.