Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

I'm using some slow-ish emit() methods in Python (2.7) logging (email, http POST, etc.) and having them done synchronously in the calling thread is delaying web requests. I put together this function to take an existing handler and make it asynchronous, but I'm new to python and want to make sure my assumptions about object scope are correct. Here's the code:

def patchAsyncEmit(handler):                                                     
    base_emit = handler.emit                                                     
    queue = Queue.Queue()                                                        
    def loop():                                                                  
        while True:                                                              
            record = queue.get(True) # blocks                                    
            try :                                                                
                base_emit(record)                         
            except: # not much you can do when your logger is broken                                                           
                print sys.exc_info()                                      
    thread = threading.Thread(target=loop)                                       
    thread.daemon = True                                                         
    thread.start()                                                               
    def asyncEmit(record):                                                       
        queue.put(record)                                                        
    handler.emit = asyncEmit                                                     
    return handler

That let's me create an asynchronous LogHandler using code like:

handler = patchAsyncEmit(logging.handlers.HttpHandler(...))

I've confirmed the messages are sent, and not in the calling thread, but I want to make sure I'm getting only one queue and only one thread per Handler, and that I'm not at risk of those somehow going out of scope.

Am I off base or did I get that okay?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

It would seem to me that writing a class would be better:

class AsyncHandler(threading.Thread):
    def __init__(self, handler):
         self._handler = handler
         self._queue = Queue.Queue()

         self.daemon = True
         self.start()

    def run(self):
        while True:
           record = self._queue.get(True)
           self._handler.emit(record)

    def emit(self, record):
        self._queue.put(record)

Patching objects can be useful, but its usually clearer to create another object like this.

share|improve this answer
    
A decorator like that (the pattern not the @pythonthing) is the route I wanted to go, but the LogHandler creation, at least when using the dictionary schema docs.python.org/library/… ), doesn't allow for passing a handler into a handler's constructor -- only literals. So I couldn't think of a way to make that work using dict config (which I recognize I didn't mention in the question), but if there is a way I'd welcome it. –  Ry4an Sep 23 '11 at 16:09
    
@Ry4an, ah. Actually, I believe you can pass objects using the cfg:// syntax. –  Winston Ewert Sep 23 '11 at 16:36
    
oh, wow, I hadn't previously seen that at all. –  Ry4an Sep 23 '11 at 16:49
    
A LogHandler has a few other important methods (configuration, etc.) would all of those have to be written into the AsyncHander to delegate correctly to the _handler? Is there an easy way to pull in all of _handler's method except the one we want to override? –  Ry4an Sep 27 '11 at 4:53
    
@Ry4an, in that case I'd probably want to subclass the handler of interest. –  Winston Ewert Sep 27 '11 at 12:47
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.