Customized class crawler to harvest names and phone numbers from a webpage

Question

I've written some code using python to scrape names and phone numbers from a webpage which is working nice at this moment.

However, the way I've written my class crawler is unconventional. As I've hardly seen any crawler which have been written the way I did here, I'm little dubious as to whether this style I should keep up. It serves the purpose, though! Expecting reviews to do the betterment of this crawler. Thanks in advance.

Here is the script:

import requests

from lxml import html

class YelPage:

    base_url = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Los+Angeles%2C+CA"
    url = "https://www.yellowpages.com"

    def __init__(self):

        self.vault = []

    def parse(self):           

        self.get_urls(self.base_url)
        self.process_links(self.base_url)

    # Going to the main page for collecting links leading to the target page
    def get_urls(self, link):

        response = requests.get(link)
        tree = html.fromstring(response.text)

        for items in tree.xpath("//div[@class='info']"):
            item_link = items.xpath(".//a[@class='business-name'][not(@itemprop='name')]/@href")
            for page in item_link:
                self.target_page(self.url + page)

    # Creating links leading to the next page
    def process_links(self, link): 

        response = requests.get(link)
        tree = html.fromstring(response.text)
        next_page = tree.xpath("//div[@class='pagination']//li/a/@href")

        for n_page in next_page: 
            self.get_urls(self.url + n_page)

    # Scraping required data reaching this target page
    def target_page(self, urls):

        print(urls)
        response = requests.get(urls)
        tree = html.fromstring(response.text)

        for posts in tree.xpath("//*[@id='main-header']"):
            name = posts.xpath(".//div[@class='sales-info']/h1/text()")[0] if posts.xpath(".//div[@class='sales-info']/h1/text()") else ""
            phone = posts.xpath('.//p[@class="phone"]/text()')[0] if posts.xpath('.//p[@class="phone"]/text()') else ""
            records = name, phone
            if records not in self.vault:
                self.vault.append(records)

    def __str__(self):

        return self.vault

if __name__ == '__main__':

    crawler = YelPage()
    crawler.parse()
    for item in crawler.vault:
        print(item)

Answer 1

In some sense, you are reinventing what Scrapy web-scraping framework provides from out-of-the-box. Take a look how Scrapy spiders are written and you may find something similar, but better organized with less boilerplate code. Since, you've been doing quite a lot of web-scraping, I would go ahead and give Scrapy a try here.

Some notes about the code:

• naming. Remember: code is much often read than written. YelPage can be replaced with a more explicit YellowPageScraper; next_page actually means next_pages since you locate multiple of them - etc. (please refer to the code below for the other improved variable names)
• as I've mentioned before, you can reuse a session by having a session instance as your instance variable
• comments before each of the methods should be transformed to proper method documentation strings

Improved code:

import requests
from lxml import html


class YellowPageScraper:
    base_url = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Los+Angeles%2C+CA"
    url = "https://www.yellowpages.com"

    def __init__(self):
        self.vault = []
        self.session = requests.Session()

    def crawl(self):
        self.get_urls(self.base_url)
        self.process_links(self.base_url)

    def get_urls(self, link):
        """Going to the main page for collecting links leading to the target page."""
        response = self.session.get(link)
        tree = html.fromstring(response.text)

        for item in tree.xpath("//div[@class='info']"):
            item_links = item.xpath(".//a[@class='business-name'][not(@itemprop='name')]/@href")
            for page in item_links:
                self.target_page(self.url + page)

    def process_links(self, link):
        """Creating links leading to the next page."""
        response = self.session.get(link)
        tree = html.fromstring(response.text)

        next_urls = tree.xpath("//div[@class='pagination']//li/a/@href")
        for next_url in next_urls:
            self.get_urls(self.url + next_url)

    def target_page(self, urls):
        """Scraping required data reaching this target page."""
        response = self.session.get(urls)
        tree = html.fromstring(response.text)

        for post in tree.xpath("//*[@id='main-header']"):
            name = post.findtext(".//div[@class='sales-info']/h1")
            phone = post.findtext('.//p[@class="phone"]')

            records = name, phone
            if records not in self.vault:
                self.vault.append(records)

    def __str__(self):
        return self.vault


if __name__ == '__main__':
    crawler = YellowPageScraper()
    crawler.crawl()
    for item in crawler.vault:
        print(item)