I've written some code in python in combination with selenium to scrape populated result from a website after performing a reverse search.
My scraper opens that site clicks on the "search by address" button and then takes the street number and address from the "original.csv" file and then put it in the searchbox and hit the search button.
Once the result is populated my scraper grabs it and write the result in a new csv file creating new columns in it along with the previous columns in the "Original Csv" file.
It is necessary to switch two iframes to get to the result. To get result for all searches it is necessary to write complex xpaths which can grab data by searching two different locations because sometimes the data are not in a particular location.
I've used try except block in my script so that it can take care of the result with no value. I tried to write all the data in "Number" and "City" column but as I'am very weak in handling try except functionality that is why I created extra column named "Number1" and "City1" so that no data are missing. "Number1" and "City1" both fall under different xpaths, though!
However, my script is running errorlessly and fetching desired results. Any input on this will be highly appreciated.
Here is what I've written to get the job done:
import csv from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC def get_info(driver, wait): with open("Original.csv", "r") as f, open('Updated.csv', 'w', newline='') as g: reader = csv.DictReader(f) newfieldnames = reader.fieldnames + ['Number','City','Number1','City1'] writer = csv.writer = csv.DictWriter(g, fieldnames = newfieldnames) writer.writeheader() for item in reader: driver.get('http://hcad.org/quick-search/') driver.switch_to_frame(driver.find_element_by_tag_name("iframe")) driver.find_element_by_id("s_addr").click() wait.until(EC.presence_of_element_located((By.NAME, 'stnum'))) driver.find_element_by_name('stnum').send_keys(item["Street"]) driver.find_element_by_name('stname').send_keys(item["Address"]) driver.find_element_by_xpath("//input[@value='Search']").click() try: driver.switch_to_frame(driver.find_element_by_id("quickframe")) try: element = driver.find_element_by_xpath("//td[@class='data']/table//th") name = driver.execute_script("return arguments.childNodes.textContent", element).strip() or driver.execute_script("return arguments.childNodes.textContent", element).strip() except: name = "" try: element = driver.find_element_by_xpath("//td[@class='data']/table//th") pet = driver.execute_script("return arguments.childNodes.textContent", element).strip() or driver.execute_script("return arguments.childNodes.textContent", element).strip() except: pet = "" try: name1 = driver.find_element_by_xpath("//table[@class='bgcolor_1']//tr/td").text except Exception: name1 = "" try: pet1 = driver.find_element_by_xpath("//table[@class='bgcolor_1']//tr/td").text except Exception: pet1 = "" item["Number"] = name item["City"] = pet item["Number1"] = name1 item["City1"] = pet1 print(item) writer.writerow(item) except Exception as e: print(e) if __name__ == '__main__': driver = webdriver.Chrome() wait = WebDriverWait(driver, 10) try: get_info(driver, wait) finally: driver.quit()
Here is the link to the csv file which I used to search the result. "https://www.dropbox.com/s/etgj0bbsav4ex4y/Original.csv?dl=0"