Parsing a slow-loading webpage with scrapy in combination with selenium

Question

I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible to make the site load all the items unless the scraper is able to scroll that page downmost. However, my scraper can reach the lowest portion of this webpage and parse the aforesaid category flawlessly. I've set explicit wait in my scraper instead of any hardcoded delay so that it doesn't take longer than necessary.

As this is my first time to work with selenium along with scrapy, there might be scopes to do betterment of this script to make it more robust.

import scrapy
from selenium import webdriver
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


class ProductSpider(scrapy.Spider):
    name = "productsp"
    start_urls = ['http://fortune.com/fortune500/list/']

    def __init__(self):
        self.driver = webdriver.Chrome()
        self.wait = WebDriverWait(self.driver, 10)

    def parse(self, response):
        self.driver.get(response.url)
        check_height = self.driver.execute_script("return document.body.scrollHeight;")
        while True:
            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            try:
                self.wait.until(lambda driver: self.driver.execute_script("return document.body.scrollHeight;")  > check_height)
                check_height = self.driver.execute_script("return document.body.scrollHeight;") 
            except:
                 break

        for item in self.driver.find_elements_by_css_selector(".row"):
            name = item.find_element_by_css_selector(".company-title").text
            revenue = item.find_element_by_css_selector(".company-revenue").text
            yield {"Title":name,"Revenue":revenue}

Answer 1

The spider is readable and understandable. I would only extract some of the things into separate methods for readability. For example, the "infinite scroll" should probably be just defined in a separate method.

And, the bare except can be replaced with handling a more specific TimeoutException:

def scroll_until_loaded(self):
    check_height = self.driver.execute_script("return document.body.scrollHeight;")
    while True:
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        try:
            self.wait.until(lambda driver: self.driver.execute_script("return document.body.scrollHeight;")  > check_height)
            check_height = self.driver.execute_script("return document.body.scrollHeight;") 
        except TimeoutException:
             break

def parse(self, response):
    self.driver.get(response.url)
    self.scroll_until_loaded()

    for item in self.driver.find_elements_by_css_selector(".row"):
        name = item.find_element_by_css_selector(".company-title").text
        revenue = item.find_element_by_css_selector(".company-revenue").text
        yield {"Title":name,"Revenue":revenue}