2
\$\begingroup\$

I've written a script in python with selenium to scrape different table data lie within different dots on a map in a certain website. Each table is connected to each dot. The table is activated once either of the dots is clicked. However, my script is able to open that webpage, traverse the map, click each dot to activate each table and finally parse the data of each table available on that map. Any input on this to make it more robust will be highly appreciated.

Here is what I've written to do the whole thing:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome()
driver.get("https://acwi.gov/monitoring/vm/programs/vm_map.html")
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))

#Using iframe link to get to the map

driver.get(driver.find_element_by_tag_name("iframe").get_attribute("src")) 
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#mapDiv_zoom_slider")))

#Zooming in for the browser to locate elements properly

driver.find_element_by_css_selector("div#mapDiv_zoom_slider").click()

#Hardcoded time to wait until certain moment to avoid stale element error

time.sleep(5)

#Finding each tag to click to get the table

for item in driver.find_elements_by_tag_name("circle"):
    ActionChains(driver).move_to_element(item).click().perform()

    #Go for the next item to click

    elem = driver.find_element_by_css_selector(".titleButton.next")
    if elem.is_displayed():
        elem.click()
        time.sleep(5)

        #Finding table element

        items = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.attrTable")))
        wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "tr")))
        list_of_data = [[item.text for item in data.find_elements_by_css_selector('td')]
                    for data in items.find_elements_by_css_selector('tr')]
        for tab_data in list_of_data:
            print(tab_data)

driver.quit()

Here is a link to an image to give the clarity of my above description: https://www.dropbox.com/s/axd66fvozexbefp/Untitled.jpg?dl=0

\$\endgroup\$
1
\$\begingroup\$

Some key things to take into account when using selenium:

  • if you have an id of an element - use it - it's the fastest way to locate an element
  • wait.until() returns a WebElement with most of the Expected Conditions - there is no need to find the element again if you want to use the element you've been waiting for
  • WebDriverWait() can be instantiated once per driver and reused
  • time.sleep() waits have to be avoided - they are making the code slower than needed most of the time, they are unreliable and not tied to any condition on a page, and sometimes the delay you set is not enough

You don't have to click through every circle on the map

We can seriously improve on web-scraping speed by extracting the "token" (let's call it this way) from the iframe URL and then making a REST API request to a specific endpoint with the data using requests:

from urllib.parse import parse_qs, urlparse

import requests
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC    


driver = webdriver.Chrome()
driver.get("https://acwi.gov/monitoring/vm/programs/vm_map.html")

wait = WebDriverWait(driver, 10)

# get the token from the frame src
frame = wait.until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
token = parse_qs(urlparse(frame.get_attribute("src")).query)['webmap'][0]

driver.close()

# parse data
response = requests.get("http://www.arcgis.com/sharing/rest/content/items/{token}/data?f=json".format(token=token))
data = response.json()

for item in data["operationalLayers"][0]["featureCollection"]["layers"][0]["featureSet"]["features"]:
    coordinates = item["geometry"]["x"], item["geometry"]["y"]
    attributes = item["attributes"]

    print(coordinates, attributes["MPN"])
\$\endgroup\$
  • \$\begingroup\$ Thanks sir alecxe, for your updated script. My head still spins to see the performance of your code. It took no more than 3 seconds. When it comes to code for scraping the web, you are second to none. Thanks a trillion. \$\endgroup\$ – SIM Aug 15 '17 at 15:55
  • \$\begingroup\$ One little asking on this sir. Using chrome dev tools i could find that there are several json links. Is there any hard and fast rules to be sure which one to track? \$\endgroup\$ – SIM Aug 15 '17 at 16:28
  • 1
    \$\begingroup\$ @Shahin good question - I had to manually explore which of them contained the desired data. There is probably a better way to do that, but I just copied one of the titles and then looked for it in the json responses until I found the desired one. Sometimes, it's easy to see just by looking at the names of the endpoints though. Thanks, interesting question! \$\endgroup\$ – alecxe Aug 15 '17 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.