I'm using some slow-ish emit() methods in Python (2.7) logging (email, http POST, etc.) and having them done synchronously in the calling thread is delaying web requests. I put together this function to take an existing handler and make it asynchronous, but I'm new to python and want to make sure my assumptions about object scope are correct. Here's the code:

def patchAsyncEmit(handler):                                                     
    base_emit = handler.emit                                                     
    queue = Queue.Queue()                                                        
    def loop():                                                                  
        while True:                                                              
            record = queue.get(True) # blocks                                    
            try :                                                                
            except: # not much you can do when your logger is broken                                                           
                print sys.exc_info()                                      
    thread = threading.Thread(target=loop)                                       
    thread.daemon = True                                                         
    def asyncEmit(record):                                                       
    handler.emit = asyncEmit                                                     
    return handler

That let's me create an asynchronous LogHandler using code like:

handler = patchAsyncEmit(logging.handlers.HttpHandler(...))

I've confirmed the messages are sent, and not in the calling thread, but I want to make sure I'm getting only one queue and only one thread per Handler, and that I'm not at risk of those somehow going out of scope.

Am I off base or did I get that okay?


1 Answer 1


It would seem to me that writing a class would be better:

class AsyncHandler(threading.Thread):
    def __init__(self, handler):
         self._handler = handler
         self._queue = Queue.Queue()

         self.daemon = True

    def run(self):
        while True:
           record = self._queue.get(True)

    def emit(self, record):

Patching objects can be useful, but its usually clearer to create another object like this.

  • \$\begingroup\$ A decorator like that (the pattern not the @pythonthing) is the route I wanted to go, but the LogHandler creation, at least when using the dictionary schema docs.python.org/library/… ), doesn't allow for passing a handler into a handler's constructor -- only literals. So I couldn't think of a way to make that work using dict config (which I recognize I didn't mention in the question), but if there is a way I'd welcome it. \$\endgroup\$ Sep 23, 2011 at 16:09
  • \$\begingroup\$ @Ry4an, ah. Actually, I believe you can pass objects using the cfg:// syntax. \$\endgroup\$ Sep 23, 2011 at 16:36
  • \$\begingroup\$ oh, wow, I hadn't previously seen that at all. \$\endgroup\$ Sep 23, 2011 at 16:49
  • \$\begingroup\$ A LogHandler has a few other important methods (configuration, etc.) would all of those have to be written into the AsyncHander to delegate correctly to the _handler? Is there an easy way to pull in all of _handler's method except the one we want to override? \$\endgroup\$ Sep 27, 2011 at 4:53
  • \$\begingroup\$ @Ry4an, in that case I'd probably want to subclass the handler of interest. \$\endgroup\$ Sep 27, 2011 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.