# Scraping the names of a whole category with a three-liner code

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for importing libraries, I used three lines of code to do this. I think this time I've done this errorlessly.

Here is what I've tried with:

import requests
from bs4 import BeautifulSoup

for i in range(1, 52):
for title in BeautifulSoup(requests.get("https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}".format(i)).text, "lxml").findAll("h2",{"class":"n"},"a"):
print(title.text)


I would add more lines, but improve on performance:

Improved version:

import requests
from bs4 import BeautifulSoup, SoupStrainer

URL_TEMPLATE = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}"

parse_only = SoupStrainer("h2", class_="n")

with requests.Session() as session:
for page_number in range(1, 52):
response = session.get(URL_TEMPLATE.format(page_number))

soup = BeautifulSoup(response.content, "lxml", parse_only=parse_only)
for title in soup.select("h2.n"):
print(title.get_text())


Also note a variable name change - page_number would be more descriptive than i.

• Thanks sir alecxe for your descriptive review. This SoupStrainer stuff is foreign to me. Btw, it seems to me that you used the tag and class twice in your script. Firstly using soup style and then selector. Does it bring efficiency? Forgive my ignorance and thanks once again. – MITHU Aug 2 '17 at 20:33
• @Mithu sure, you are welcome. From what I recall, SoupStrainer requires the "find/find_all" definition style, but to actually find elements I usually prefer CSS selectors - it is not really a big deal here at all since we've already cut the page down to only what we need. You can probably even go with soup("h2") version. Thanks. – alecxe Aug 2 '17 at 20:35