2
\$\begingroup\$

The script I've written is able to scrape name, address, phone and web address from a webpage using python and selenium. The main barrier I had to face was to exhaust the load more button to get the more content until all are displayed. I know the way I have written xpaths in the script is fragile but they serve the purpose for now. I ran my script and found the results as I expected. I hope there will be any better way to improve the design and performance of my script. Thanks in advance for taking care of it. Here is the full code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

driver.get("https://www.zebra.com/us/en/partners/partner-application-locator.html")
driver.find_element_by_xpath('//div[@class="ft-dropdown"]/input[@placeholder="City, State/Province, Country"]').clear()
driver.find_element_by_xpath('//div[@class="ft-dropdown"]/input[@placeholder="City, State/Province, Country"]').send_keys("Colorado, USA")

wait.until(EC.visibility_of_element_located((By.XPATH, '//a[contains(@class,"ng-binding")]')))
driver.find_element_by_xpath('//a[contains(@class,"ng-binding")]').click()

while True:
    try:
        wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'showmore-bg')))
        driver.find_element_by_class_name('showmore-bg').click()
    except Exception:
        break

wait.until(EC.visibility_of_element_located((By.XPATH, '//div[contains(@class,"padding0")]')))
for items in driver.find_elements_by_xpath('//div[contains(@class,"padding0")]'):
    try:
        name = items.find_element_by_xpath('.//a[@class="ng-binding"]').text
    except Exception:
        name = ""
    try:
        address = items.find_element_by_xpath('.//div[contains(@class,"fullDetail-cmpAdres")]//p[@class="ng-binding"]').text
    except Exception:
        address = ""    
    try:
        phone = items.find_element_by_xpath('.//div[contains(@class,"fullDetail-cmpAdres")]//p[contains(@class,"ng-scope")]').text
    except Exception:
        phone = ""      
    try:
        website = items.find_element_by_xpath('.//a[contains(@class,"ng-scope")]').get_attribute("href")
    except Exception:
        website = ""

    print(name ,address, phone, website)

driver.quit()
\$\endgroup\$
  • \$\begingroup\$ You've been posting a lot of scrappers lately. May I ask what purpose do they serve? \$\endgroup\$ – Grajdeanu Alex. Jul 14 '17 at 9:07
  • \$\begingroup\$ Yeah sure, MrGrj. if necessary, I can deal with different webpages with complicated structures. Few months back I knew nothing about it and now at least I can understand a little. Btw, did I hurt anybody through my posting? If it is, I will not practise posting then. Thanks. \$\endgroup\$ – SIM Jul 14 '17 at 9:24
2
\$\begingroup\$

There are a few major things that immediately caught my eye:

  • bare except clauses should be avoided in favor of catching more specific exceptions
  • element locators quality - you should NOT be relying on layout-oriented classes like padding0 and pure-technical Angular classes like ng-binding and ng-scope
  • wait.until() calls in your case return a WebElement instance - you don't have to re-find an element after the wait
  • I think the code may benefit from comments explaining the logic behind the browser element interactions

Improved code:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException, NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def get_field_text_if_exists(item, selector):
    """Extracts a field by a CSS selector if exists."""
    try:
        return item.find_element_by_css_selector(selector).text
    except NoSuchElementException:
        return ""


def get_link_if_exists(item, selector):
    """Extracts an href attribute value by a CSS selector if exists."""
    try:
        return item.find_element_by_css_selector(selector).get_attribute("href")
    except NoSuchElementException:
        return ""


driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

driver.get("https://www.zebra.com/us/en/partners/partner-application-locator.html")

location = driver.find_element_by_css_selector('.partnerLocation input')
location.clear()
location.send_keys("Colorado, USA")

# select the first suggestion from a suggestion dropdown
dropdown_suggestion = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'ul[id^=typeahead] li a')))
dropdown_suggestion.click()

# click more until no more results to load
while True:
    try:
        more_button = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'showmore-bg'))).click()
    except TimeoutException:
        break

# wait for results to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.seclection-result .partners-detail')))

# parse results
for result in driver.find_elements_by_css_selector('.seclection-result .partners-detail'):
    name = get_field_text_if_exists(result, 'a')
    address = get_field_text_if_exists(result, '.fullDetail-cmpAdres')
    phone = get_field_text_if_exists(result, '.fullDetail-cmpAdres p[ng-if*=phone]')
    website = get_link_if_exists(result, 'a[ng-if*=website]')

    print(name, address, phone, website)

driver.quit()
\$\endgroup\$
  • \$\begingroup\$ Thanks sir for the review along with an epic script. This is by far the best code on python + selenium I've ever come across . What I did in [while true block ,as in firstly waited then clicked] was making my browser run slowly. You rectified that and now it is working very fast. I'm unable to wait to see the full results as I'm getting busy otherwise. However, I observed for a few minutes and found it running very fast. 'll give you the feedback when I'm free. Thanks again sir. \$\endgroup\$ – SIM Jul 14 '17 at 14:38
  • \$\begingroup\$ @SMth80 you are welcome! I actually enjoy reviewing your code - I can see you are improving the code quality and web-scraping skills quickly. If you want to further optimize this code, please post separately with "performance" tag - I think there may be ways to make the "load more" and other parts faster. Thanks. \$\endgroup\$ – alecxe Jul 14 '17 at 14:40
  • \$\begingroup\$ I ran your code sir alecxe. It does get to the bottom of the page exhausting load more button, but prints nothing in the console. I suppose the selectors might need a little twitch. Thanks. \$\endgroup\$ – SIM Jul 14 '17 at 16:49
  • \$\begingroup\$ @SMth80 oops, I am an idiot - was a typo, please try now. \$\endgroup\$ – alecxe Jul 14 '17 at 17:00
  • \$\begingroup\$ Tried and got the results as I get always with your code. Thanks a zillion sir. \$\endgroup\$ – SIM Jul 14 '17 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.