In some sense, you are reinventing what Scrapy web-scraping framework provides from out-of-the-box. Take a look how Scrapy spiders are written and you may find something similar, but better organized with less boilerplate code. Since, you've been doing quite a lot of web-scraping, I would go ahead and give Scrapy a try here.
Some notes about the code:
- naming. Remember: code is much often read than written.
YelPage can be replaced with a more explicit YellowPageScraper; next_page actually means next_pages since you locate multiple of them - etc. (please refer to the code below for the other improved variable names)
- as I've mentioned before, you can reuse a session by having a session instance as your instance variable
- comments before each of the methods should be transformed to proper method documentation strings
Improved code:
import requests
from lxml import html
class YellowPageScraper:
base_url = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Los+Angeles%2C+CA"
url = "https://www.yellowpages.com"
def __init__(self):
self.vault = []
self.session = requests.Session()
def crawl(self):
self.get_urls(self.base_url)
self.process_links(self.base_url)
def get_urls(self, link):
"""Going to the main page for collecting links leading to the target page."""
response = self.session.get(link)
tree = html.fromstring(response.text)
for item in tree.xpath("//div[@class='info']"):
item_links = item.xpath(".//a[@class='business-name'][not(@itemprop='name')]/@href")
for page in item_links:
self.target_page(self.url + page)
def process_links(self, link):
"""Creating links leading to the next page."""
response = self.session.get(link)
tree = html.fromstring(response.text)
next_urls = tree.xpath("//div[@class='pagination']//li/a/@href")
for next_url in next_urls:
self.get_urls(self.url + next_url)
def target_page(self, urls):
"""Scraping required data reaching this target page."""
response = self.session.get(urls)
tree = html.fromstring(response.text)
for post in tree.xpath("//*[@id='main-header']"):
name = post.findtext(".//div[@class='sales-info']/h1/text()")
phone = post.findtext('.//p[@class="phone"]/text()')
records = name, phone
if records not in self.vault:
self.vault.append(records)
def __str__(self):
return self.vault
if __name__ == '__main__':
crawler = YellowPageScraper()
crawler.crawl()
for item in crawler.vault:
print(item)
