11
\$\begingroup\$

I have a function that processes a large amount of data. I call this function as part of a wider process via a command line script in which many similar but shorter jobs are conducted in sequence.

I have something like this:

import time

def initialise_foo():
    """do something quickly"""
    time.sleep(0.1)

def initialise_bar():
    """do something else quickly"""
    time.sleep(0.1)

def load_data(n_items):
    """do another thing quickly"""
    return range(n_items)    

def process_datum(datum, sleep):
    """"do something with datum that takes a while"""
    time.sleep(sleep)

def cleanup():
    """do a final thing quickly"""
    time.sleep(0.1)

My problem is that I want my script to provide feedback on progress and I was considering converting my data processor into an iterator and using yield to emit progress information.

I can do this but there is a long wait with no feedback...

def process_data1(data, sleep):
    """loop over a lot of data"""
    for item in data:
        process_datum(item, sleep)


def main1(n_items, sleep):
    print("initialising foo")
    initialise_foo()
    print("initialising bar")      
    initialise_bar()
    print("loading data")
    mydata = load_data(n_items)
    print("processing data")
    process_data1(mydata, sleep)
    print("cleaning up")
    cleanup()
    print("done")

So what about this?

def process_data2(data, sleep, report_frequency):
    """loop over a lot of data"""
    for i, item in enumerate(data):
        process_datum(item, sleep)
        if not (i % report_frequency):
            yield i

def main2(n_items, sleep, report_frequency):
    print("initialising foo")
    initialise_foo()
    print("initialising bar")      
    initialise_bar()
    print("loading data")
    mydata = load_data(n_items)
    print("processing data")
    for i in process_data2(mydata, sleep, report_frequency):
        print("%i items processed" % i)
    print("cleaning up")
    cleanup()
    print("done")

run it like this.

if __name__ == "__main__":
    print ("\nwithout progress")
    main1(100, 0.01)
    print ("\nwith progress")
    main2(100, 0.01, 10)

Does this look like a reasonable approach? Any comments on how I have coded this?

\$\endgroup\$
  • 1
    \$\begingroup\$ Alternatively, you could pass in an object like a file or something and write progress to that, perhaps. \$\endgroup\$ – rlms Sep 25 '13 at 15:41
8
\$\begingroup\$

I see three basic options you have, to structure progress feedback:

  1. yield
  2. a callback function
  3. inline, hard-coded feedback

Using yield, as you've shown, is one approach. Ensure you are familiar with Python's implementation of yield and iterables and generators, since several languages share the yield keyword superficially but have subtle implementation differences that might surprise you if, for example, you're used to C#'s yield. Here's a reference on that -- it's subtle but worth reading. Essentially, in Python when your function yields, it returns a generator which can be usefully assigned to a variable, capturing iteration to that point, but you may or may not want to do that.

So there is fundamentally nothing wrong with using yield for this job. It carries the big advantage of separating pure computing from progress IO.

A decent alternative to yield which is easy to understand is a callback function. If you passed a callable as a final parameter to your process_data function then a block of code can be executed, instead of a yield or hardcoded inline feedback, also separating processing from feedback.

One advantage a callback approach carries is it allows communication back to the process_data function which could be used to pause or halt processing. For example, if your callback function is aware of the console or UI, then while updating progress it can also monitor buttons or keys and return a value to the process_data function. This could indicate, for example, that the user wants to abort processing and this doesn't pollute the process_data function with awareness of specific UI or IO. So a callback would also work well and allow you to pass a lambda block or full function in Python. (Languages without a yield facility may have no choice but to use callbacks.)

Example code:

def process_data(data, sleep, report_frequency, report_callback=None):
    for ...
        ...
        if not (i % report_frequency):  # or whatever criteria
            if report_callback:
                if report_callback(i) == USER_ABORT:  # or other codes, by convention
                  return

Please let me also address what I see as a big, very practical, issue with the code defining report_frequency in terms of data elements or loop iterations.

I have see two problems with this pattern of progress tracking:

  1. It's difficult to know in advance what a good value of report_frequency is, when it's expressed in loop iterations. Trial and error can quickly get you into the ballpark, but that leaves problem #2.

  2. Depending on the nature of your computation, the process_datum function may not run in constant time. For example, it may take longer as you get deeper into the dataset, if numbers are getting larger and depending on what you're calculating. What I'm getting at is there may not be a constant number of loop iterations per second.

So I'll propose a better way for you to think of report_frequency. Use seconds, not iterations or data elements. Time is likely what you had in mind, anyways. It's natural to want to see an update every X seconds, regardless of how many loop iterations that is. This eliminates both problems and is especially useful when your process_datum function does not run in constant time.

Your code:

def process_data2(data, sleep, report_frequency):
    """loop over a lot of data"""
    for i, item in enumerate(data):
        process_datum(item, sleep)
        if not (i % report_frequency):
            yield i 

Can become this: (I'm putting the yield vs inline vs callback issue aside for the moment, to illustrate the concept)

def process_data3(data, sleep, report_frequency):
    """loop over a lot of data"""
    t_updlast = datetime.now()
    t_update = t_updlast + timedelta(seconds=report_frequency)    
    for i, item in enumerate(data):
        if datetime.now() >= t_update:
            print("%6.2f%%\n" % (i*100.0/len(data)))
            t_update = datetime.now() + timedelta(seconds=report_frequency)
        process_datum(item, sleep)

Now you'll see a report every report_frequency seconds that looks like:

56.12%
56.83%
...

And this simplifies how you call process_data3 from main. By dropping the yield, you simply call the following from main, without wrapping it in a for loop:

process_data3(mydata, sleep, report_frequency)

And now that you're in the domain of measuring time, if your process_datum function runs in constant time, you can add a fairly simple calculation for "estimated time to completion". I'll leave that as an exercise for you. :-)

Remember to add this at the top of your module:

from datetime import datetime, timedelta

Sorry for the long answer and timedelta diversion. I hope this is helpful. Good luck!

\$\endgroup\$
  • 1
    \$\begingroup\$ Nice answer, I agree that using time to determine the feedback is a great approach. However, my question was about the use of yield for this kind of feedback. The caller of my function may want to process the feedback in another way other than printing. I would like to know if yield is a sensible approach and what other approaches I might take. \$\endgroup\$ – Graeme Stuart Nov 5 '13 at 13:41
  • \$\begingroup\$ @GraemeStuart, thanks, that's fair feedback! I did fixate on your example code's use of iteration-based progress measurement because I've seen that pattern fail many times in practical terms. I will edit my answer shortly to address your (real) question about the suitability of yield. :) \$\endgroup\$ – Darren Stone Nov 5 '13 at 19:15
  • \$\begingroup\$ Great thanks. I hadn't considered a callback! I will mull it over. Looks like there are advantages. If my process_data function knows what to expect from the callback then it can indeed be used to control the data processing as well as to provide feedback. Thanks again. If I don't get any more answers then this will be marked as accepted. \$\endgroup\$ – Graeme Stuart Nov 6 '13 at 11:36
5
\$\begingroup\$

Use A generator as a callback function. A generator is absolutely perfect for this. Using your example I'll demonstrate

def process_datum(item):
    import random
    import time
    # Sleep for some randomly small time to mimic processing the item
    time.sleep(random.random() / 100.0)


def process_data(data, callback=None):
    if callback is not None:  # Prime the generator
        next(callback)

    for i, item in enumerate(data):
        process_datum(item)
        if callback is not None:
            callback.send(float(i) / len(data))

    if callback is not None:  # Close the generator
        callback.close()


def callback(report_frequency):
    counter = 0
    while True:
        progress = yield
        counter += 1
        if counter % report_frequency:
            print("Progress: %s%%" % (progress * 100.0,))


def main():
    data = range(1000)
    process_data(data, callback(10))


if __name__ == '__main__':
    main()

The great thing about this is that you can make your call back function as complicated or as simple as you like, it also has access to the local scope as well so, if you want to pass arguments (like I did i.e. report_frequency), you can pass it to the generator initialiser, instead of having to do something like

def process_data(data, callback_function, callback_function_args, callback_function_kwargs):
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.