I wrote this ~100 line script recently, where I tried out Python's asyncio library. When it comes to asynchronous programming in Python, I just have 2 days of experience. I spent the last two days reading about it for the first time.

Based on all my readings and understanding, I think, there is no mess-up in the script, but there are some specific parts which I am a little skeptical about because of my inexperience in this area.

To be specific I am doing a asyncio.gather(<unpacking a value here>, return_exceptions=True). This is the relevant function:

async def main():
districts = json.loads(os.environ['DISTRICTS'])
while True:
    start = perf_counter()
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(*(process_district(d, session) for d in districts), return_exceptions=True)
        for district, result in zip(districts, results):
            if isinstance(result, Exception):
                error = f"exception occurred when district {district['district_id']} was being processed: {result}"
                await send_telegram(os.environ['BOT_CHAT_ID'], error, session)
    print(f'time taken: {perf_counter() - start} seconds')

Now somewhere in the function call hierarchy of process_district(), there comes another asyncio.gather(), without return_exceptions flag. This is the relevant function:

async def send_notifications(chat_ids, cova_18_message, cova_45_message, covi_18_message, covi_45_message, session):
await asyncio.gather(
    send_telegram(chat_ids.get('cova_18'), cova_18_message, session),
    send_telegram(chat_ids.get('cova_45'), cova_45_message, session),
    send_telegram(chat_ids.get('covi_18'), covi_18_message, session),
    send_telegram(chat_ids.get('covi_45'), covi_45_message, session)

My script is working all well as expected. However, I am a little bit uncertain if I am just doing anything wrong/bad here. To be specific, some of the worries that I have:

  1. Is it fine where asyncio.gather() is "nested" in the hierarchy call. Can it have any bad implications?
  2. My "inner" asyncio.gather() doesn't have return_exceptions=True. To be honest, this looks completely fine to me, but still, is there any catch / any code smell here? (I have also tested this, where the exceptions raised by the coroutines inside the inner gather() are also being awaited and iterated at the outer gather())
  3. Any other issue that is there with the way my script is written? Other than my inexperience, the only reason I am sceptical about this is, I was not able to find a single example on the wide Internet where people are calling gather() "inside" gather().
  • \$\begingroup\$ I don't think there is anything inherently wrong or dangerous with nested asyncio.gather calls. However I think this question may be more suited to SO \$\endgroup\$
    – amin_nejad
    Commented Oct 18, 2021 at 22:37


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.