I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and iterates through every pages. What would you recommend to improve on?

import requests
from bs4 import BeautifulSoup, SoupStrainer
import pandas as pd

list_topic = []
list_time = []

SESSION = requests.Session()

def get_response(url):  # Gets the <html> structure from the website #
    response = SESSION.get(url)
    soup = BeautifulSoup(response.text, 'lxml',
                         parse_only=SoupStrainer('ul', {'class': 'posts posts-archive'}))
    return soup

def iteration(url, max_page=52):
    starting_page = 1
    while starting_page <= max_page:
        ## formats the new URL etc (https://paultan.org/topics/test-drive-reviews/page/1) ##
        new_url = url + f"page/{starting_page}"
        data = get_response(new_url)
        ## iteration starts ##
        starting_page += 1

def get_reviews(response):
    for container in response('article'):
        title = container.h2.a.text
        time = container.time.text

def create_pdReview():
    return pd.DataFrame({'Title': list_topic, 'Time': list_time})

if __name__ == '__main__':
    URL = 'https://paultan.org/topics/test-drive-reviews/'

I've been wondering; would using yield improve the efficiency and simplicity of the code? How would it be done? Because I've been trying to learn from my previous inquiries that has been answered on earlier. Here is a similar question and I'm trying to put to practice what has been recommended so far.


1 Answer 1


When we are discussing performance of a particular piece of code, it's important to recognize bottlenecks and major contributors to the runtime of the program.

In your particular case, even though you've applied some optimizations like SoupStrainer speed-up for HTML parsing, the synchronous nature of the script is the biggest problem by far. The script is processing pages one by one, not getting to the next page until the processing for the current page is finished.

Switching to an asynchronous approach would be the natural next step in your optimizations. Look into using third-party frameworks like Scrapy or, if you are adventurous, things like asyncio or grequests.

You could apply one more optimization to your current script which should help you optimize the "crawling/scraping" part - instead of using requests.get(), initialize session = requests.Session() and use session.get() to make requests (documentation). This would allow the underlying TCP connection to be re-used for subsequent requests resulting in a performance increase.

  • \$\begingroup\$ I've changed the way of iteration; but it looks like an honest mess at the moment. I've used Session as you've recommended ~ but at the moment I'm trying to think how to simplify and get it to work with my previous version =l. \$\endgroup\$
    – Minial
    Commented Jan 15, 2019 at 2:39
  • \$\begingroup\$ @Minial I usually use a class and store session as an instance attribute which is initialized in a class constructor. \$\endgroup\$
    – alecxe
    Commented Jan 15, 2019 at 4:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.