Parser for gleaning data from twitter

Question

I've written a script in python to parse the name, tweets, following and follower of those people available in view all section in my profile page of twitter. My scraper is able to parse those aforesaid fields flawlessly. Any input on the improvement of my parser will be highly appreciated. Here is what I've written:

from selenium import webdriver
import time

def browsing_pages():

    driver = webdriver.Chrome()
    driver.get('https://twitter.com/?lang=en')

    driver.find_element_by_xpath('//input[@id="signin-email"]').send_keys('username')
    driver.find_element_by_xpath('//input[@id="signin-password"]').send_keys('password')
    driver.find_element_by_xpath('//button[@type="submit"]').click()
    time.sleep(5)

    #Clicking the viewall link
    driver.find_element_by_xpath("//small[@class='view-all']//a[contains(@class,'js-view-all-link')]").click()
    time.sleep(5)

    for links in driver.find_elements_by_xpath("//div[@class='stream-item-header']//a[contains(@class,'js-user-profile-link')]"):
        scraping_docs(links.get_attribute("href"))
        #tracking down each profile links under viewall section

def scraping_docs(item_link):

    driver = webdriver.Chrome()
    driver.get(item_link)

    # gleaning information of each profile holder
    for prof in driver.find_elements_by_xpath("//div[@class='route-profile']"):
        name = prof.find_elements_by_xpath(".//h1[@class='ProfileHeaderCard-name']//a[contains(@class,'ProfileHeaderCard-nameLink')]")[0]
        tweet = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[0]
        following = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[1]
        follower = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[2]
        print(name.text, tweet.text, following.text, follower.text)

        driver.quit()

browsing_pages()

Answer 1

I'd focus on 3 main things:

• don't use time.sleep() to wait for elements on a page. With hardcoded time delays there is a tendency to wait more than actually needed most of the time and less than needed sometimes - not reliable at all. Instead, use Explicit Waits with WebDriverWait class and a set of Expected Conditions
• remove the overhead of firing up a separate Chrome instance - collect the links into a list a reuse the same WebDriver instance - you should expect improvements in page load times as well
• improve your locators - XPath locators are generally the slowest - use "by id" locators whenever possible; handling class attributes with CSS selectors is more reliable (raw contains() in XPath may generate false positives - it can be a bit better with concat())

In the end, you should have something like this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def browsing_pages(driver):
    driver.get('https://twitter.com/login')
    wait = WebDriverWait(driver, 10)

    driver.find_element_by_css_selector('form.signin input.email-input').send_keys('username')
    driver.find_element_by_css_selector('form.signin input.js-password-field').send_keys('password')
    driver.find_element_by_css_selector('form.signin button[type=submit]').click()

    view_all = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".view-all a.js-view-all-link")))
    view_all.click()

    # wait for a profile link to become visible
    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".stream-item-header a.js-user-profile-link")))

    links = [link.get_attribute("href") for link in driver.find_elements_by_css_selector(".stream-item-header a.js-user-profile-link")]
    for link in links:
        scraping_docs(driver, link)


def scraping_docs(driver, item_link):
    driver.get(item_link)

    # gleaning information of each profile holder
    for prof in driver.find_elements_by_css_selector(".route-profile"):
        name = prof.find_element_by_css_selector("h1.ProfileHeaderCard-name a.ProfileHeaderCard-nameLink")
        tweet = prof.find_element_by_css_selector(".ProfileNav-value")

        _, following, follower = prof.find_elements_by_css_selector(".ProfileNav-value")[:3]
        print(name.text, tweet.text, following.text, follower.text)


if __name__ == '__main__':
    driver = webdriver.Chrome()

    try:
        browsing_pages(driver)
    finally:
        driver.quit()

You may also go for a class with the driver instance kept at self.driver, but I think it's pretty much okay to do it this "functional" way considering that you have only two functions. Good read on the subject:

• Stop Writing Classes
• Start Writing More Classes