3
\$\begingroup\$

I want to run all the coroutines from the list in a pool of constant size and would like to ask if this is the right way how to achieve it. Is there any built in solution for this problem? I have not found any.

def subprocess_pool(coroutines, pool_size=3):
    loop = asyncio.get_event_loop()
    event_finished = asyncio.Event()

    def done(future):
        event_finished.set()

    @asyncio.coroutine 
    def scheduler(coroutines):
        num = len(coroutines)
        finished = 0
        started = 0
        while finished<num:
            if coroutines and (started - finished) < pool_size:
                cor = coroutines.pop(0)
                task = loop.create_task(cor) 
                task.add_done_callback(done)
                started += 1
            else:
                yield from event_finished.wait()
                event_finished.clear()
                finished += 1
        return True

    loop.run_until_complete(scheduler(coroutines))
    loop.close()
\$\endgroup\$
2
\$\begingroup\$

Making a pool of coroutines seems not to be the way why we invented coroutines. Coroutines means to be lightweight, so that they can be created in very large numbers. Rather than limiting the number of coroutines, I think the right way to do this is to limit the running coroutines, and:

use an asyncio.Semaphore

async def worker(semaphore, coro):
    async with semaphore:
        await coro()


async def scheduler(corotines):
    tasks = []
    semaphore = asyncio.BoundedSemaphore(pool_size)
    for coro in corotines:
        tasks.append(loop.create_task(worker(coro))
    await asyncio.gather(*tasks)


loop.run_until_complete(scheduler(coroutines))
loop.close()

check out this article for a full example.

\$\endgroup\$
  • \$\begingroup\$ Should pass the semaphore in create_task: worker(semaphore, coro) \$\endgroup\$ – Joseph Sheedy Apr 25 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.