I've written a script in python to scrape names and phone numbers of a certain search from yellowpage traversing multiple pages. The scraper has the ability to collect all the links connected to all the names. Once the links are collected, the scraper goes to the target page to parse the name and website address. Finally, it will produce a csv file with data filled-in. Although the scraper is doing it's job errorlessly, I'm little dubious as to whether I used the best way to write items in a csv file. Any betterment to the pattern will be highly appreciated.
Here is what I've written:
import requests import csv from urllib.parse import urljoin from bs4 import BeautifulSoup url = "https://www.yellowpages.com/search?search_terms=+Injury+Law+Attorneys&geo_location_terms=California" def get_links(session,link): session.headers['User-Agent'] = 'Mozilla/5.0' res = session.get(link) soup = BeautifulSoup(res.text,"lxml") for items in soup.select(".info h2 a[data-analytics]"): get_info(session,urljoin(url,items.get("href"))) nextpage = soup.select_one("a.next") if nextpage: #If there is no more next page link, it should break nexturl = urljoin(link,nextpage.get("href")) print(nexturl) #check to see which page I'm on get_links(session,nexturl) #supply the newly produced next page link to grab the next one def get_info(session,tlink): session.headers['User-Agent'] = 'Mozilla/5.0' res = session.get(tlink) soup = BeautifulSoup(res.text,"lxml") name = soup.select_one("#main-header .sales-info h1").get_text(strip=True) website = soup.select_one("#main-header a.website-link").get("href") if soup.select_one("#main-header a.website-link") else "" print(name,website) writer.writerow([name,website]) #writing data in a csv file if __name__ == '__main__': with open("yellowpageinfo.csv","w",newline="") as infile: #creating a csv file to write populated results writer = csv.writer(infile) writer.writerow(['Name','Website']) with requests.Session() as session: get_links(session,url)