E-mail crawler for yellowpages

Question

I've written a script in python to scrape e-mail addresses from different pizza shops located in los-angeles available in yellowpage traversing multiple pages. It is able to go one-layer deep and dig out email addresses. I believe, this crawler has got the ability to parse all the emails from any link no matter how many pages it has spread across. Just needed to adjust the last page number in the crawler.

Here is what I've written:

import requests
from lxml import html

link = "https://www.yellowpages.com"
for page_num in range(1,10):
  for item in html.fromstring(requests.get("https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Los%20Angeles%2C%20CA&page={0}".format(page_num)).text).cssselect('div.info h2.n a:not([itemprop="name"]).business-name'):
    for data in html.fromstring(requests.get(link + item.attrib['href']).text).cssselect('a.email-business'):
      print(data.attrib['href'].replace("mailto:",""))

Answer 1

Knowing that you had a ton of scrappers questions, this doesn't look nice at all. It looks like you were in a hurry and didn't much care about how the code looks.

---

Styling

• Use 4 spaces per indentation level
• Constants should be LOWERCASED
• You should have a space after ,
• Your lines are way too long. Stick to 72 characters or 120 (at most)

---

Code

First, you don't have to hardcode your search items. Instead, you might as well define a function which returns the data you need (the entire URL):

def build_url(item, location, page):
    params = urllib.parse.urlencode({
        'search_terms': item,
        'geo_location_terms': location,
        'page': page
    })

    return '{}/search?{}'.format(URL, params)

With that in mind, your code might be restructured nicer, like this:

import requests
import urllib

from lxml import html

URL = "https://www.yellowpages.com"
LOWER_LIMIT = 1
UPPER_LIMIT = 10


def build_url(item, location, page):
    params = urllib.parse.urlencode({
        'search_terms': item,
        'geo_location_terms': location,
        'page': page
    })

    return '{}/search?{}'.format(URL, params)


def scrape():
    for page in range(LOWER_LIMIT, UPPER_LIMIT):
        url = build_url('pizza', 'Los Angeles CA', page)
        html_ = requests.get(url).text
        for item in html.fromstring(html_).cssselect('div.info h2.n a:not([itemprop="name"]).business-name'):
            for data in html.fromstring(requests.get(URL + item.attrib['href']).text).cssselect('a.email-business'):
                print(data.attrib['href'].replace("mailto:", ""))


if __name__ == '__main__':
    scrape()

Other changes that I did above:

• Move the logic into separate functions
• Make the range arguments constants so that they can be easily modified
• Add the guard check if __name__ == '__main__'

---

As is, your scrapper is pretty slow because it has to search in each page the entire html for that specific a info. You might consider using Scrapy.

Answer 2

• Your current code has lots of hardcoded stuff and is not re-useable in anyway.

• For performance improvement use a session instead of opening a new HTTP connection every time.

  The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3's connection pooling. So if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).

• Make sure your code is PEP8 compliant to make it more-readable. Don't have 239 character long lines. 80 or 120 etc is the standard. Some issues pointed out here.

---

This is how I would break your program to make it re-useable and testable.

from urllib.parse import quote_plus

import requests
from lxml import html


base_url = 'https://www.yellowpages.com'
search_url = base_url + '/search?search_terms={search_term}&geo_location_terms={geo_location}&page={page_num}'


def get_business_urls_from_page(page):
    businesses = html.fromstring(page).cssselect('div.info h2.n a:not([itemprop="name"]).business-name')
    for business in businesses:
        yield base_url + business.attrib['href']


def get_business_emails(page):
    for data in html.fromstring(page).cssselect('a.email-business'):
        yield data.attrib['href'].replace("mailto:", "")


def get_email_addresses(search_term, geo_location, page_count=10):
    session = requests.Session()

    for page_num in range(1, page_count + 1):
        search_page_content = session.get(search_url.format(
            search_term=quote_plus(search_term),
            geo_location=quote_plus(geo_location),
            page_num=page_num
        )).text

        for business_url in get_business_urls_from_page(search_page_content):
            business_page_content = session.get(business_url).text
            for email in get_business_emails(business_page_content):
                print(email)

Usage:

>>> for email in get_email_addresses(search_term='pizza', geo_location='Los Angeles CA'):
...     print(email)
...
alien@et.com
bark@dog.com
meow@cat.com