I need a program that needs to gather data from large list of API end points. Below is a mock-up program that attempts to make 10000 requests as fast as possible. Any suggestions on how to improve on this (especially for speed) is highly welcome. Experimentation showed that the Semaphore cap of around 100 gave the best speed.

import asyncio
from aiohttp import ClientSession
import datetime
import time
import sys

def processData(data):
    return data

async def fetch(url, session):
    async with session.get(url) as response:
        data = await response.read()
        data = processData(data)
        return data

async def bound_fetch(sem, url, session):
    async with sem:
        return await fetch(url, session)

async def run(loop,N):
    url = "https://www.example.com"
    tasks = []
    sem = asyncio.Semaphore(100)
    async with ClientSession() as session:
        for i in range(N):
            task = loop.create_task(bound_fetch(sem, url, session))

        print("Done starting {} tasks".format(N))
        starttime = time.time()
        responses = await  asyncio.gather(*tasks)
        print("Done completing {} tasks in: {}".format(N,time.time()-starttime))

        return responses

args = sys.argv
loop = asyncio.get_event_loop()

if __name__ == "__main__":
    if len(sys.argv) == 2:
        N = int(sys.argv[1])
        N = 10000
    maintask = loop.create_task(run(loop, N))
    result = loop.run_until_complete(maintask)
  • 1
    \$\begingroup\$ Why do you have time.sleep(0.001)? \$\endgroup\$
    – hjpotter92
    Apr 4 '18 at 0:39
  • \$\begingroup\$ @hjpotter92 That is for simulating some data-processing time. \$\endgroup\$
    – rajendra
    Apr 4 '18 at 19:28
  • 1
    \$\begingroup\$ I've flagged this question as off-topic, because you have no concrete implementation of processData, and provided no 'real' URL. \$\endgroup\$
    – Daniel
    Apr 5 '18 at 5:56
  • \$\begingroup\$ If process_data is synchronous, CPU-bound, offload it -- opt1. fetch, save, then process; opt2. fetch, push to queue, have threaded or forked workers process; opt3. use many machines :) \$\endgroup\$
    – user134596
    Dec 12 '18 at 1:34
  • \$\begingroup\$ You could conceivably make 10K requests in parallel, but check system limits first, max number of open file descriptors, max tcp connections, network stack memory, as well as network path to your endpoints -- e.g. if your machine is behind NAT or firewall, that could have a limit. \$\endgroup\$
    – user134596
    Dec 12 '18 at 1:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.