# Scraping restaurant names and addresses from a website using Scrapy and Selenium

I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is in javascript.

The actual list of restaurants per area is two links deep (three in London) - hence the initial three:

• def(parse)
• def(parse_dir_contents)
• def(parse_dir_contents1)

to get me to that list. And then I need to use Selenium to hit the "Show more" button. At that point I can then go one more link deeper to get the data I need.

The one piece of code I am still thinking about is where to put self.driver.close() - if I leave it where it is then the program does have a habit of ending before it should.

Also, though I suspect this is an issue of my PC - when I launch the program I get the message "Please insert a disk into drive E", which I dismiss and then the program works as it should.

If anyone has any ideas on how to improve the Code or on the self.driver.close() position then I am always listening.

import scrapy
import urlparse
import time

from hungryhouse.items import HungryhouseItem
from selenium import webdriver
from scrapy.http import TextResponse

class HungryhouseSpider(scrapy.Spider):
name = "hungryhouse"
allowed_domains = ["hungryhouse.co.uk"]
start_urls =["https://hungryhouse.co.uk/takeaway"]

# parse / parse_dir_contents / parse_dir_contents1 follow the links the page with the restaurant lists
# parse starts at the city list page
def parse(self,response):
for href in response.xpath('//*[@class="CmsRestcatCityLandingLocations"]/ul[@class="cities"]/li/a/@href'):
url = urlparse.urljoin('https://hungryhouse.co.uk/',href.extract())
yield scrapy.Request(url, callback=self.parse_dir_contents)

# parse_dir_contents will get to the web page with the lists except for London
def parse_dir_contents(self, response):
for href1 in response.xpath('//*[contains(text(),"Choose your location")]/../ul/li/a/@href'):
url1 = urlparse.urljoin('https://hungryhouse.co.uk/',href1.extract())
if "london-takeaway" in url1:
yield scrapy.Request(url1, callback=self.parse_dir_contents1)
yield scrapy.Request(url1, callback=self.parse_dir_contents2)

# parse_dir_contents1 is needed for London which is one link deeper
def parse_dir_contents1(self, response):
for href2 in response.xpath('//*[contains(text(),"Choose your location")]/../ul/li/a/@href'):
url2 = urlparse.urljoin('https://hungryhouse.co.uk/',href2.extract())
yield scrapy.Request(url2, callback=self.parse_dir_contents2)

def __init__(self):

# and now we are on the web page where the restaurants are listed
# now we need to use Selenium to activate a javescript button to reveal all the page
def parse_dir_contents2(self,response):
self.driver.get(response.url)

# Pressing the "Show More" button until there are no more on the page to reveal all the page
while True:
next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a')
try:
next.click()
time.sleep(3) # waiting 3 seconds for the page to load fully
except:
break

# Now that the webpage is all revealed Scrapy can bring down all the restaurant URLs
# I.e. we need to follow the link for every restuarant to get onto its page to get our data
response1 = TextResponse(url=response.url, body=self.driver.page_source, encoding='utf-8')
for href in response1.xpath('//*[@class="restsRestInfo"]/a/@href'):
url = response1.urljoin(href.extract())
yield scrapy.Request(url, callback=self.parse_dir_contents3)

#        self.driver.close()

# and now Scrapy can take the names, addresses and postcodes of all the restaurants from their URL page
def parse_dir_contents3(self, response):
item = HungryhouseItem()
for sel in response.xpath('//*[@class="restBoxInfo"]'):
item['name']=sel.xpath('//div/div/div/h1/span/text()').extract()[0].strip()

A good place to close the driver would be the the closed method, which is called when the spider is closed:
def closed(self, reason):