5
\$\begingroup\$

I've written a function to call with the deferred library to generate a large task queue, and at first without the recursion it timed out (deadlineexceeded) so now I'm trying with a recursion but is it safe that the function will terminate? I actually ran it with GAE and it did the correct task - deleted all the empty blobs that were written previously because of a bug. (It seems that empty blobs with 0 bytes get written several times a day and the code is to delete those blobs.)

def do_blobs_delete(msg=None, b=None, c=None):
    logging.info(msg)
    edge = datetime.now() - timedelta(days=750)
    statmnt = "WHERE size < 1" # size in bytes
    logging.debug(statmnt)
    try:
        gqlQuery = blobstore.BlobInfo.gql(statmnt)
        blobs = gqlQuery.fetch(999999)
        for blob in blobs:
            if blob.size < 1 or blob.creation < edge:
                blob.delete()
                continue
    except DeadlineExceededError:
        # Queue a new task to pick up where we left off.
        deferred.defer(do_blobs_delete, "Deleting empty blobs", 42, c=True)
        return
    except Exception, e:
        logging.error('There was an exception:%s' % str(e))
\$\endgroup\$

1 Answer 1

5
\$\begingroup\$

Yes, that function terminate (unless the semantics of blobstore are stranger than expected and blobs is an infinite generator). One thing to be concerned about might be that, should your query take a very long time, you would not delete a single blob before a DeadlineExceededError is thrown, and so schedule another task without doing any work. This is a bad thing, as you may end up with many jobs that simply get half way through a query, give up and then schedule themselves to run again. The worst part is that your only indication would be that your log would be full of info level messages (i.e. messages that will be ignored), giving you no idea that this travesty was unfolding in your task queue.

I would recommend you add some kind of limit to make sure you are decreasing the number of blobs towards zero each time. You could think of it as an inductive proof almost; see Burstall's 69 paper on structural induction if you feel like formulating a mathematical proof. However, that is probably not necessary in this case. My suggested rewrite would be something like:

# The number of times one call to do_blobs_delete may `recurse`, should probably be
# made a function of the count of all blobs
MAX_ATTEMPTS = 10

def do_blobs_delete(msg=None, b=None, c=None, attempt=0):
    logging.info(msg)
    edge = datetime.now() - timedelta(days=750)
    statmnt = "WHERE size < 1" # size in bytes
    logging.debug(statmnt)
    try:
        gqlQuery = blobstore.BlobInfo.gql(statmnt)
        blobs = gqlQuery.fetch(999999)
        for blob in blobs:
            if blob.size < 1 or blob.creation < edge:
                blob.delete()
    except DeadlineExceededError:
        if MAX_ATTEMPTS <= attempt:
            logging.warning("Maximum blob delete attempt reached")
            return
        # Queue a new task to pick up where we left off.
        attempt++
        deferred.defer(do_blobs_delete, "Deleting empty blobs",
                       42, c=True,
                       attempts=attempt)
        return
    except Exception, e:
        logging.error('There was an exception:%s' % str(e))

Note that this does not explicitly address the lack of an inductive variable, though it does limit any damage done.

Another thing to note is that you have a race condition on the .delete. Should the method be called concurrently, say by process A and B, then the call to .fetch could return the same set of bobs to each. They would then attempt to delete the elements twice, which would raise an error on the .delete call, leading to fewer blob deletions than expected. This problem gets a lot worse when we consider more processes and things like unspecified ordering. The correct way to handle this is to treat any exceptions thrown by the .delete in a more nuanced way than you are currently, and accommodate multiple attempted deletes without prejudice.

Your current code will work fine at the moment, the problems will manifest themselves when things get more complicated. While I am sure Google's infrastructure can handle these upsets, your bottom line may be less flexible.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.