I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is in javascript.

The actual list of restaurants per area is two links deep (three in London) - hence the initial three:

  • def(parse)
  • def(parse_dir_contents)
  • def(parse_dir_contents1)

to get me to that list. And then I need to use Selenium to hit the "Show more" button. At that point I can then go one more link deeper to get the data I need.

The one piece of code I am still thinking about is where to put self.driver.close() - if I leave it where it is then the program does have a habit of ending before it should.

Also, though I suspect this is an issue of my PC - when I launch the program I get the message "Please insert a disk into drive E", which I dismiss and then the program works as it should.

If anyone has any ideas on how to improve the Code or on the self.driver.close() position then I am always listening.

import scrapy
import urlparse
import time

from hungryhouse.items import HungryhouseItem
from selenium import webdriver
from scrapy.http import TextResponse

class HungryhouseSpider(scrapy.Spider):
    name = "hungryhouse"
    allowed_domains = ["hungryhouse.co.uk"]
    start_urls =["https://hungryhouse.co.uk/takeaway"]

    # parse / parse_dir_contents / parse_dir_contents1 follow the links the page with the restaurant lists
    # parse starts at the city list page
    def parse(self,response):
        for href in response.xpath('//*[@class="CmsRestcatCityLandingLocations"]/ul[@class="cities"]/li/a/@href'):
           url = urlparse.urljoin('https://hungryhouse.co.uk/',href.extract())
           yield scrapy.Request(url, callback=self.parse_dir_contents)

    # parse_dir_contents will get to the web page with the lists except for London
    def parse_dir_contents(self, response):
        for href1 in response.xpath('//*[contains(text(),"Choose your location")]/../ul/li/a/@href'):
           url1 = urlparse.urljoin('https://hungryhouse.co.uk/',href1.extract())
           if "london-takeaway" in url1:
               yield scrapy.Request(url1, callback=self.parse_dir_contents1)
           yield scrapy.Request(url1, callback=self.parse_dir_contents2)

    # parse_dir_contents1 is needed for London which is one link deeper
    def parse_dir_contents1(self, response):
        for href2 in response.xpath('//*[contains(text(),"Choose your location")]/../ul/li/a/@href'):
           url2 = urlparse.urljoin('https://hungryhouse.co.uk/',href2.extract())
           yield scrapy.Request(url2, callback=self.parse_dir_contents2)       

    def __init__(self):
        self.driver = webdriver.Chrome("C:/Users/andrew/Downloads/chromedriver_win32/chromedriver.exe")

    # and now we are on the web page where the restaurants are listed
    # now we need to use Selenium to activate a javescript button to reveal all the page
    def parse_dir_contents2(self,response):

        # Pressing the "Show More" button until there are no more on the page to reveal all the page
        while True:
            next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a')
                time.sleep(3) # waiting 3 seconds for the page to load fully

        # Now that the webpage is all revealed Scrapy can bring down all the restaurant URLs
        # I.e. we need to follow the link for every restuarant to get onto its page to get our data
        response1 = TextResponse(url=response.url, body=self.driver.page_source, encoding='utf-8')
        for href in response1.xpath('//*[@class="restsRestInfo"]/a/@href'):
            url = response1.urljoin(href.extract())
            yield scrapy.Request(url, callback=self.parse_dir_contents3)

#        self.driver.close()

    # and now Scrapy can take the names, addresses and postcodes of all the restaurants from their URL page
    def parse_dir_contents3(self, response):
        item = HungryhouseItem()
        for sel in response.xpath('//*[@class="restBoxInfo"]'):
            yield item

1 Answer 1


A good place to close the driver would be the the closed method, which is called when the spider is closed:

def closed(self, reason):

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.