I've written a script using python to parse the names and email addresses of different pizza shops in USA. I am very new in writing classes using python so I'm not very sure I didn't do anything wrong with it's design.
However, it serves the purpose scraping the required fields. Firstly, it scrapes all the links of different pizza shops then parses the links to the next page and finally tracking down the links of each pizza shops to it's main page to harvest the documents from there. I would be very happy if I could learn the way I could improve my class crawler.
Here is what I have tried with:
import requests from lxml import html class YellowPage: main_url = "https://www.yellowpages.com/search?search_terms=pizza&geo_location_terms=Los+Angeles%2C+CA" def __init__(self): self.links = [self.main_url] self.storage =  def crawl(self): for link in self.links : self.get_link(link) def get_link(self, link): print('Scraping Now: ' + link) url = "https://www.yellowpages.com" response = requests.get(link) tree = html.fromstring(response.text) #scraping links of each pizza shop for items in tree.xpath("//div[@class='info']"): link_page = items.xpath(".//a[@class='business-name'][not(@itemprop='name')]/@href") for page in link_page: if page and url + page not in self.links: self.links += [url + page] #parsing the links to the next page next_page = tree.xpath("//div[@class='pagination']//li/a/@href") for nepage in next_page: if nepage and url + nepage not in self.links: self.links += [url + nepage] #going to the main page of each pizza shop and harvest the record for posts in tree.xpath("//*[@id='main-header']"): name = posts.xpath(".//div[@class='sales-info']/h1/text()") if posts.xpath(".//div[@class='sales-info']/h1/text()") else "" email = posts.xpath(".//a[@class='email-business']/@href") if posts.xpath(".//a[@class='email-business']/@href") else "" records = name, email self.storage.append(records) def __str__(self): return self.storage crawler = YellowPage() crawler.crawl() for item in crawler.storage: print(item)
Btw, when I write the above code using function then i find that very speedy.