Scraping data from tables connected to each dot on a map

Question

I've written a script in python with selenium to scrape different table data lie within different dots on a map in a certain website. Each table is connected to each dot. The table is activated once either of the dots is clicked. However, my script is able to open that webpage, traverse the map, click each dot to activate each table and finally parse the data of each table available on that map. Any input on this to make it more robust will be highly appreciated.

Here is what I've written to do the whole thing:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome()
driver.get("https://acwi.gov/monitoring/vm/programs/vm_map.html")
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))

#Using iframe link to get to the map

driver.get(driver.find_element_by_tag_name("iframe").get_attribute("src")) 
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#mapDiv_zoom_slider")))

#Zooming in for the browser to locate elements properly

driver.find_element_by_css_selector("div#mapDiv_zoom_slider").click()

#Hardcoded time to wait until certain moment to avoid stale element error

time.sleep(5)

#Finding each tag to click to get the table

for item in driver.find_elements_by_tag_name("circle"):
    ActionChains(driver).move_to_element(item).click().perform()

    #Go for the next item to click

    elem = driver.find_element_by_css_selector(".titleButton.next")
    if elem.is_displayed():
        elem.click()
        time.sleep(5)

        #Finding table element

        items = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.attrTable")))
        wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "tr")))
        list_of_data = [[item.text for item in data.find_elements_by_css_selector('td')]
                    for data in items.find_elements_by_css_selector('tr')]
        for tab_data in list_of_data:
            print(tab_data)

driver.quit()

Here is a link to an image to give the clarity of my above description: https://www.dropbox.com/s/axd66fvozexbefp/Untitled.jpg?dl=0

Answer 1

Some key things to take into account when using selenium:

• if you have an id of an element - use it - it's the fastest way to locate an element
• wait.until() returns a WebElement with most of the Expected Conditions - there is no need to find the element again if you want to use the element you've been waiting for
• WebDriverWait() can be instantiated once per driver and reused
• time.sleep() waits have to be avoided - they are making the code slower than needed most of the time, they are unreliable and not tied to any condition on a page, and sometimes the delay you set is not enough

You don't have to click through every circle on the map

We can seriously improve on web-scraping speed by extracting the "token" (let's call it this way) from the iframe URL and then making a REST API request to a specific endpoint with the data using requests:

from urllib.parse import parse_qs, urlparse

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC    


driver = webdriver.Chrome()
driver.get("https://acwi.gov/monitoring/vm/programs/vm_map.html")

wait = WebDriverWait(driver, 10)

# get the token from the frame src
frame = wait.until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
token = parse_qs(urlparse(frame.get_attribute("src")).query)['webmap'][0]

driver.close()

# parse data
response = requests.get("http://www.arcgis.com/sharing/rest/content/items/{token}/data?f=json".format(token=token))
data = response.json()

for item in data["operationalLayers"][0]["featureCollection"]["layers"][0]["featureSet"]["features"]:
    coordinates = item["geometry"]["x"], item["geometry"]["y"]
    attributes = item["attributes"]

    print(coordinates, attributes["MPN"])