# Webcrawler for a car forum

I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and iterates through every pages. What would you recommend to improve on?

import requests
from bs4 import BeautifulSoup, SoupStrainer
import pandas as pd

list_topic = []
list_time = []

SESSION = requests.Session()

def get_response(url):  # Gets the <html> structure from the website #
response = SESSION.get(url)
soup = BeautifulSoup(response.text, 'lxml',
parse_only=SoupStrainer('ul', {'class': 'posts posts-archive'}))
return soup

def iteration(url, max_page=52):
starting_page = 1
while starting_page <= max_page:
## formats the new URL etc (https://paultan.org/topics/test-drive-reviews/page/1) ##
new_url = url + f"page/{starting_page}"
data = get_response(new_url)
get_reviews(data)
## iteration starts ##
starting_page += 1

def get_reviews(response):
for container in response('article'):
title = container.h2.a.text
time = container.time.text
list_topic.append(title)
list_time.append(time)
else:
None

def create_pdReview():
return pd.DataFrame({'Title': list_topic, 'Time': list_time})

if __name__ == '__main__':
URL = 'https://paultan.org/topics/test-drive-reviews/'
print(iteration(URL))
print(create_pdReview())


I've been wondering; would using yield improve the efficiency and simplicity of the code? How would it be done? Because I've been trying to learn from my previous inquiries that has been answered on earlier. Here is a similar question and I'm trying to put to practice what has been recommended so far.

In your particular case, even though you've applied some optimizations like SoupStrainer speed-up for HTML parsing, the synchronous nature of the script is the biggest problem by far. The script is processing pages one by one, not getting to the next page until the processing for the current page is finished.
Switching to an asynchronous approach would be the natural next step in your optimizations. Look into using third-party frameworks like Scrapy or, if you are adventurous, things like asyncio or grequests.
You could apply one more optimization to your current script which should help you optimize the "crawling/scraping" part - instead of using requests.get(), initialize session = requests.Session() and use session.get() to make requests (documentation). This would allow the underlying TCP connection to be re-used for subsequent requests resulting in a performance increase.
• @Minial I usually use a class and store session as an instance attribute which is initialized in a class constructor. – alecxe Jan 15 '19 at 4:09