Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Any suggestions/improvements for the following custom thread-pool code?

import threading
from Queue import Queue

class Worker(threading.Thread):
    def __init__(self, function, in_queue, out_queue):
        self.function = function
        self.in_queue, self.out_queue = in_queue, out_queue
        super(Worker, self).__init__()

    def run(self):
        while True:
            if self.in_queue.empty(): break
            data = in_queue.get()
            result = self.function(*data)
            self.out_queue.put(result)
            self.in_queue.task_done()

def process(data, function, num_workers=1):
    in_queue = Queue()
    for item in data: 
        in_queue.put(item)

    out_queue = Queue(maxsize=in_queue.qsize())
    workers = [Worker(function, in_queue, out_queue) for i in xrange(num_workers)]

    for worker in workers: 
        worker.start()

    in_queue.join()

    while not out_queue.empty():
        yield out_queue.get() 
share|improve this question
    
I recommend looking into the multiprocessing.pool.ThreadPool object, also explained here. – Theuni Dec 7 '12 at 21:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.