I've written a script in python to scrape names and phone numbers of a certain search from yellowpage traversing multiple pages. The scraper has the ability to collect all the links connected to all the names. Once the links are collected, the scraper goes to the target page to parse the name and website address. Finally, it will produce a csv file with data filled-in. Although the scraper is doing it's job errorlessly, I'm little dubious as to whether I used the best way to write items in a csv file. Any betterment to the pattern will be highly appreciated.
Here is what I've written:
import requests
import csv
from urllib.parse import urljoin
from bs4 import BeautifulSoup
url = "https://www.yellowpages.com/search?search_terms=+Injury+Law+Attorneys&geo_location_terms=California"
def get_links(session,link):
session.headers['User-Agent'] = 'Mozilla/5.0'
res = session.get(link)
soup = BeautifulSoup(res.text,"lxml")
for items in soup.select(".info h2 a[data-analytics]"):
get_info(session,urljoin(url,items.get("href")))
nextpage = soup.select_one("a.next")
if nextpage: #If there is no more next page link, it should break
nexturl = urljoin(link,nextpage.get("href"))
print(nexturl) #check to see which page I'm on
get_links(session,nexturl) #supply the newly produced next page link to grab the next one
def get_info(session,tlink):
session.headers['User-Agent'] = 'Mozilla/5.0'
res = session.get(tlink)
soup = BeautifulSoup(res.text,"lxml")
name = soup.select_one("#main-header .sales-info h1").get_text(strip=True)
website = soup.select_one("#main-header a.website-link").get("href") if soup.select_one("#main-header a.website-link") else ""
print(name,website)
writer.writerow([name,website]) #writing data in a csv file
if __name__ == '__main__':
with open("yellowpageinfo.csv","w",newline="") as infile: #creating a csv file to write populated results
writer = csv.writer(infile)
writer.writerow(['Name','Website'])
with requests.Session() as session:
get_links(session,url)