6
\$\begingroup\$

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for importing libraries, I used three lines of code to do this. I think this time I've done this errorlessly.

Here is what I've tried with:

import requests
from bs4 import BeautifulSoup 

for i in range(1, 52):
    for title in BeautifulSoup(requests.get("https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}".format(i)).text, "lxml").findAll("h2",{"class":"n"},"a"):
        print(title.text)
\$\endgroup\$
3
\$\begingroup\$

I would add more lines, but improve on performance:

  • parse only the relevant parts of the page with SoupStrainer
  • re-use the same session when making requests

Improved version:

import requests
from bs4 import BeautifulSoup, SoupStrainer


URL_TEMPLATE = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los+Angeles%2C+CA&page={0}"

parse_only = SoupStrainer("h2", class_="n")

with requests.Session() as session:
    for page_number in range(1, 52):
        response = session.get(URL_TEMPLATE.format(page_number))

        soup = BeautifulSoup(response.content, "lxml", parse_only=parse_only)
        for title in soup.select("h2.n"):
            print(title.get_text())

Also note a variable name change - page_number would be more descriptive than i.

\$\endgroup\$
  • \$\begingroup\$ Thanks sir alecxe for your descriptive review. This SoupStrainer stuff is foreign to me. Btw, it seems to me that you used the tag and class twice in your script. Firstly using soup style and then selector. Does it bring efficiency? Forgive my ignorance and thanks once again. \$\endgroup\$ – MITHU Aug 2 '17 at 20:33
  • \$\begingroup\$ @Mithu sure, you are welcome. From what I recall, SoupStrainer requires the "find/find_all" definition style, but to actually find elements I usually prefer CSS selectors - it is not really a big deal here at all since we've already cut the page down to only what we need. You can probably even go with soup("h2") version. Thanks. \$\endgroup\$ – alecxe Aug 2 '17 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.