Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

Any suggestions/improvements for the following custom thread-pool code?

import threading
from Queue import Queue

class Worker(threading.Thread):
    def __init__(self, function, in_queue, out_queue):
        self.function = function
        self.in_queue, self.out_queue = in_queue, out_queue
        super(Worker, self).__init__()

    def run(self):
        while True:
            if self.in_queue.empty(): break
            data = in_queue.get()
            result = self.function(*data)
            self.out_queue.put(result)
            self.in_queue.task_done()

def process(data, function, num_workers=1):
    in_queue = Queue()
    for item in data: 
        in_queue.put(item)

    out_queue = Queue(maxsize=in_queue.qsize())
    workers = [Worker(function, in_queue, out_queue) for i in xrange(num_workers)]

    for worker in workers: 
        worker.start()

    in_queue.join()

    while not out_queue.empty():
        yield out_queue.get() 
share|improve this question
add comment

1 Answer

I recommend looking into the 'multiprocessing.ThreadPool' object. Also explained here: http://stackoverflow.com/questions/3033952/python-thread-pool-similar-to-the-multiprocessing-pool

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.