Scraper to deal with some complicated site with ads

I've written a script in Python in combination with Selenium to scrape data from a slightly complicated web page. When the web page is made to open, it comes up with an ad sitting on the data making the browser break unless it is kicked out or the scraper waits a certain time for the ad to disappear.

I have made the scraper remove it successfully and reach the data as i wanted to. Another problem was to click on the next page button which I've been able to do effectively as well. Now, it is working perfectly along with writing the scraped data in a CSV file. There are around 100 pages the website has traversed. Although it's a very small script to get reviewed, I suppose any clue to make it better might be useful for future reference.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
import csv

driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)

driver.get("https://www.inc.com/inc5000/list/2017")

try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[.='SKIP']"))).click()
except TimeoutException: pass

outfile = open("inc.csv","w",newline='')
writer = csv.writer(outfile)
writer.writerow(["Rank","Company","Growth","Revenue","Industry"])

while True:
for item in driver.find_elements_by_css_selector("#data-container .row"):
rank = item.find_elements_by_css_selector(".rank")[0].text
company = item.find_elements_by_css_selector(".company a")[0].text
growth =item.find_elements_by_css_selector(".growth")[0].text
revenue = item.find_elements_by_css_selector(".revenue")[0].text
industry = item.find_elements_by_css_selector(".industry")[0].text
writer.writerow([rank,company,growth,revenue,industry])
print(rank,company,growth,revenue,industry)
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#arrow-r .fa-caret-right"))).click()
except:break

driver.quit()


There is a room for improvement - like there always is:

• avoid bare exception handling. Instead of having an except clause without specifying exception classes to handle, you can handle TimeoutException (which you even already have imported)
• use with context manager when dealing with files
• instead of using the find_elements_by_css_selector() method and then getting the first result - just use find_element_by_css_selector() directly
• use try/finally to make sure the browser is closed in case of an unexpected exception during the web-scraping
• respect PEP8 rules - specifically: spaces between arguments, order of imports
• I think you can switch to "by id" locator for the "next" button

Improved version:

import csv

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

driver.get("https://www.inc.com/inc5000/list/2017")

# skip ad popup if present
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[.='SKIP']"))).click()
except TimeoutException:
pass

try:
with open("inc.csv", "w", newline='') as outfile:
writer = csv.writer(outfile)
writer.writerow(["Rank", "Company", "Growth", "Revenue", "Industry"])

while True:
for row in driver.find_elements_by_css_selector("#data-container .row"):
rank = row.find_element_by_css_selector(".rank").text
company = row.find_element_by_css_selector(".company a").text
growth = row.find_element_by_css_selector(".growth").text
revenue = row.find_element_by_css_selector(".revenue").text
industry = row.find_element_by_css_selector(".industry").text

writer.writerow([rank, company, growth, revenue, industry])
print(rank, company, growth, revenue, industry)

# next page
try:
wait.until(EC.element_to_be_clickable((By.ID, "arrow-r"))).click()
except TimeoutException:
break  # reached the end of the page
finally:
driver.quit()

• Thanks sir alecxe, for this descriptive review. It's always a great pleasure to have you in the loop. I tried with 'ID' for clicking the next page for the first time but it was breaking sometimes then. Thanks a lot again. – SIM Oct 16 '17 at 4:55

I somewhat doubt that you need to actually web-scrape, here. It's entirely likely that you should instead pay attention to how the site is fetching the data, and cut to the source:

https://www.inc.com/inc5000list/json/inc5000_2017.json

If this is true, you should get rid of Selenium, and use Requests with a single GET instead.