# Crawler for performing reverse search and writing results into a CSV file

I've written a script which is able to perform reverse search in the website using the Name and Lid from a predefined CSV file. However, when the search is done then it can put the results containing Address and Phone Number adjacent to those Name and Lid creating a new CSV file. It is working errorlessly now. I tried to make the total process clean. Any suggestion to do betterment of this script will be highly appreciated. Here is the code I have tried with:

import csv
import requests
from lxml import html

with open("predefined.csv", "r") as f, open('newly_created.csv', 'w', newline='') as g:

writer = csv.writer = csv.DictWriter(g, fieldnames = newfieldnames)

Page = "https://www.yellowpages.com/los-angeles-ca/mip/{}-{}".format(entry["Name"].replace(" ","-"), entry["Lid"])
response = requests.get(Page)
tree = html.fromstring(response.text)

Contact = title.xpath('.//p[@class="phone"]/text()')[0]

new_row = entry
new_row['Phone'] = Contact
writer.writerow(new_row)


Here is the link to the results.

There are multiple things we can do to improve the code:

• variable naming - try to be consistent with PEP8 naming suggestions - for instance:
• Page should probably be page - or even better url
• Address would be address
• Contact would be contact
• f can be input_file
• g can be output_file
• titles variable is never used
• move the url format string into a constant
• you don't need writer = csv.writer = csv.DictWriter(...) - just assign the writer to the DictWriter instance directly
• since you are crawling the same domain, re-using requests.Session() instance should have a positive impact on performance
• use .findtext() method instead of xpath() and then getting the first item
• I would also create a separate crawl function to keep the web-scraping logic separate

Here is the modified code with the above and other improvements combined:

import csv

import requests
from lxml import html

URL_TEMPLATE = "https://www.yellowpages.com/los-angeles-ca/mip/{}-{}"

def crawl(entries):
with requests.Session() as session:
for entry in entries:
url = URL_TEMPLATE.format(entry["Name"].replace(" ", "-"), entry["Lid"])
response = session.get(url)
tree = html.fromstring(response.text)

for title in titles:
contact = title.findtext('.//p[@class="phone"]')

entry['Phone'] = contact
yield entry

if __name__ == '__main__':
with open("predefined.csv", "r") as input_file, open('newly_created.csv', 'w', newline='') as output_file: