I've written a script in Python in combination with Selenium to scrape data from a slightly complicated web page. When the web page is made to open, it comes up with an ad sitting on the data making the browser break unless it is kicked out or the scraper waits a certain time for the ad to disappear.
I have made the scraper remove it successfully and reach the data as i wanted to. Another problem was to click on the next page button which I've been able to do effectively as well. Now, it is working perfectly along with writing the scraped data in a CSV file. There are around 100 pages the website has traversed. Although it's a very small script to get reviewed, I suppose any clue to make it better might be useful for future reference.
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.common.exceptions import TimeoutException from selenium.webdriver.support import expected_conditions as EC import csv driver = webdriver.Chrome() wait = WebDriverWait(driver,10) driver.get("https://www.inc.com/inc5000/list/2017") try: wait.until(EC.element_to_be_clickable((By.XPATH, "//span[.='SKIP']"))).click() except TimeoutException: pass outfile = open("inc.csv","w",newline='') writer = csv.writer(outfile) writer.writerow(["Rank","Company","Growth","Revenue","Industry"]) while True: for item in driver.find_elements_by_css_selector("#data-container .row"): rank = item.find_elements_by_css_selector(".rank").text company = item.find_elements_by_css_selector(".company a").text growth =item.find_elements_by_css_selector(".growth").text revenue = item.find_elements_by_css_selector(".revenue").text industry = item.find_elements_by_css_selector(".industry").text writer.writerow([rank,company,growth,revenue,industry]) print(rank,company,growth,revenue,industry) try: wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#arrow-r .fa-caret-right"))).click() except:break driver.quit()