I wrote this ~100 line script recently, where I tried out Python's asyncio
library. When it comes to asynchronous programming in Python, I just have 2 days of experience. I spent the last two days reading about it for the first time.
Based on all my readings and understanding, I think, there is no mess-up in the script, but there are some specific parts which I am a little skeptical about because of my inexperience in this area.
To be specific I am doing a asyncio.gather(<unpacking a value here>, return_exceptions=True)
. This is the relevant function:
async def main():
districts = json.loads(os.environ['DISTRICTS'])
while True:
start = perf_counter()
async with aiohttp.ClientSession() as session:
results = await asyncio.gather(*(process_district(d, session) for d in districts), return_exceptions=True)
for district, result in zip(districts, results):
if isinstance(result, Exception):
error = f"exception occurred when district {district['district_id']} was being processed: {result}"
print(error)
await send_telegram(os.environ['BOT_CHAT_ID'], error, session)
print(f'time taken: {perf_counter() - start} seconds')
sleep(int(os.environ['SLEEP_SECONDS']))
Now somewhere in the function call hierarchy of process_district()
, there comes another asyncio.gather()
, without return_exceptions
flag. This is the relevant function:
async def send_notifications(chat_ids, cova_18_message, cova_45_message, covi_18_message, covi_45_message, session):
await asyncio.gather(
send_telegram(chat_ids.get('cova_18'), cova_18_message, session),
send_telegram(chat_ids.get('cova_45'), cova_45_message, session),
send_telegram(chat_ids.get('covi_18'), covi_18_message, session),
send_telegram(chat_ids.get('covi_45'), covi_45_message, session)
)
My script is working all well as expected. However, I am a little bit uncertain if I am just doing anything wrong/bad here. To be specific, some of the worries that I have:
- Is it fine where
asyncio.gather()
is "nested" in the hierarchy call. Can it have any bad implications? - My "inner" asyncio.gather() doesn't have
return_exceptions=True
. To be honest, this looks completely fine to me, but still, is there any catch / any code smell here? (I have also tested this, where the exceptions raised by the coroutines inside the innergather()
are also being awaited and iterated at the outergather()
) - Any other issue that is there with the way my script is written? Other than my inexperience, the only reason I am sceptical about this is, I was not able to find a single example on the wide Internet where people are calling
gather()
"inside"gather()
.
asyncio.gather
calls. However I think this question may be more suited to SO \$\endgroup\$