I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 days or so and 1 day of scanning is too long. From what I read so far my options are threading or something like an asynchronous requests library like grequests. Here is my code:

def sortiment(Supermarkt_link, Supermarkt_name):
with open("progress.txt", "r") as t: 
    lines = t.readlines()
    paused_loop_start = int(lines[0])   
    for e in range(paused_loop_start + 1, get_max_product_number(Supermarkt_link) ):
        json_list = []
        page = requests.get(f"https://www.supermarktcheck.de{Supermarkt_link}sortiment/?page={e}")
        soup = BeautifulSoup(page.content, "html.parser")
        products_list = soup.find_all("a", class_="h3", href=True)
        for single_product in products_list: 
            product_string = single_product["href"]
            product_page = requests.get(f"https://www.supermarktcheck.de/{product_string}")

--This code scrapes a landing page and than scans each individual page on that landing page for information that are than saved into a database.--

  • \$\begingroup\$ What does get_max_product_number do? Can you post the entire script? \$\endgroup\$
    – C.Nivs
    Jul 14 at 16:56
  • 1
    \$\begingroup\$ Please add all the imports required to run this script. \$\endgroup\$ Jul 14 at 17:59
  • \$\begingroup\$ You have an indentation error. \$\endgroup\$
    – Booboo
    Jul 21 at 11:59
  • \$\begingroup\$ If your trying to speed up your code, try using numba. Search numba up on youtube. \$\endgroup\$
    – Kimi M
    Aug 15 at 19:39

2 Answers 2


Using asynchronous requests is a go to method.

But you can also check out multithreading. Why not take advantage of multi-core CPU's to maximize performance.


Even without changing this code, using sessions could boost performance.

And indeed you need to add some kind of parallel processing. Python has different options for that, and there are many different approaches - example.

If you can spread concurrent requests among different websites this is even better. You can normally crawl more than one page at a time on the same webserver but you don't want to overwhelm the server. And quite likely, there is rate limiting in place. Consider yourself lucky if you don't have to deal with Cloudflare captchas.

It's not clear what the purpose of json_list is, or paused_loop_start, or how get_max_product_number works. Maybe this function is slowing you down further.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.