Scraping the names of a whole category with a three-liner code

Question

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for importing libraries, I used three lines of code to do this. I think this time I've done this errorlessly.

Here is what I've tried with:

import requests
from bs4 import BeautifulSoup 

for i in range(1, 52):
    for title in BeautifulSoup(requests.get("https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}".format(i)).text, "lxml").findAll("h2",{"class":"n"},"a"):
        print(title.text)

Answer 1

I would add more lines, but improve on performance:

• parse only the relevant parts of the page with SoupStrainer
• re-use the same session when making requests

Improved version:

import requests
from bs4 import BeautifulSoup, SoupStrainer


URL_TEMPLATE = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}"

parse_only = SoupStrainer("h2", class_="n")

with requests.Session() as session:
    for page_number in range(1, 52):
        response = session.get(URL_TEMPLATE.format(page_number))

        soup = BeautifulSoup(response.content, "lxml", parse_only=parse_only)
        for title in soup.select("h2.n"):
            print(title.get_text())

Also note a variable name change - page_number would be more descriptive than i.