I've written a script in python to parse the name, tweets, following and follower of those people available in view all section in my profile page of twitter. My scraper is able to parse those aforesaid fields flawlessly. Any input on the improvement of my parser will be highly appreciated. Here is what I've written:

from selenium import webdriver
import time

def browsing_pages():

    driver = webdriver.Chrome()


    #Clicking the viewall link

    for links in driver.find_elements_by_xpath("//div[@class='stream-item-header']//a[contains(@class,'js-user-profile-link')]"):
        #tracking down each profile links under viewall section

def scraping_docs(item_link):

    driver = webdriver.Chrome()

    # gleaning information of each profile holder
    for prof in driver.find_elements_by_xpath("//div[@class='route-profile']"):
        name = prof.find_elements_by_xpath(".//h1[@class='ProfileHeaderCard-name']//a[contains(@class,'ProfileHeaderCard-nameLink')]")[0]
        tweet = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[0]
        following = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[1]
        follower = prof.find_elements_by_xpath(".//span[@class='ProfileNav-value']")[2]
        print(name.text, tweet.text, following.text, follower.text)



1 Answer 1


I'd focus on 3 main things:

  • don't use time.sleep() to wait for elements on a page. With hardcoded time delays there is a tendency to wait more than actually needed most of the time and less than needed sometimes - not reliable at all. Instead, use Explicit Waits with WebDriverWait class and a set of Expected Conditions
  • remove the overhead of firing up a separate Chrome instance - collect the links into a list a reuse the same WebDriver instance - you should expect improvements in page load times as well
  • improve your locators - XPath locators are generally the slowest - use "by id" locators whenever possible; handling class attributes with CSS selectors is more reliable (raw contains() in XPath may generate false positives - it can be a bit better with concat())

In the end, you should have something like this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def browsing_pages(driver):
    wait = WebDriverWait(driver, 10)

    driver.find_element_by_css_selector('form.signin input.email-input').send_keys('username')
    driver.find_element_by_css_selector('form.signin input.js-password-field').send_keys('password')
    driver.find_element_by_css_selector('form.signin button[type=submit]').click()

    view_all = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".view-all a.js-view-all-link")))

    # wait for a profile link to become visible
    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".stream-item-header a.js-user-profile-link")))

    links = [link.get_attribute("href") for link in driver.find_elements_by_css_selector(".stream-item-header a.js-user-profile-link")]
    for link in links:
        scraping_docs(driver, link)

def scraping_docs(driver, item_link):

    # gleaning information of each profile holder
    for prof in driver.find_elements_by_css_selector(".route-profile"):
        name = prof.find_element_by_css_selector("h1.ProfileHeaderCard-name a.ProfileHeaderCard-nameLink")
        tweet = prof.find_element_by_css_selector(".ProfileNav-value")

        _, following, follower = prof.find_elements_by_css_selector(".ProfileNav-value")[:3]
        print(name.text, tweet.text, following.text, follower.text)

if __name__ == '__main__':
    driver = webdriver.Chrome()


You may also go for a class with the driver instance kept at self.driver, but I think it's pretty much okay to do it this "functional" way considering that you have only two functions. Good read on the subject:

  • \$\begingroup\$ Omg!! Who did I get response from!! It's always a great pleasure to have you in the loop sir, alecxe. I just executed your code. However, it breaks when it hits the line containing "links = [link.get_attribute("href") for link----". I suppose you wanna take a look. Thanks again sir. \$\endgroup\$
    – SIM
    Commented Jun 10, 2017 at 20:57
  • \$\begingroup\$ @SMth80 thanks, yeah, we keep stumbling upon each other on multiple SE forums :) Thanks for the good question! How does it fail? \$\endgroup\$
    – alecxe
    Commented Jun 10, 2017 at 21:04
  • \$\begingroup\$ @SMth80 I've added a wait before grabbing the links - might help - if you meant by "breaks" that it could not find the links to follow (with no errors). Thanks. \$\endgroup\$
    – alecxe
    Commented Jun 10, 2017 at 21:11
  • \$\begingroup\$ This is the very message I'm getting. " raise exception_class(message, screen, stacktrace) selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document". Forgive my ignorance. \$\endgroup\$
    – SIM
    Commented Jun 10, 2017 at 21:16
  • 1
    \$\begingroup\$ That Jack Diederich video is one of my all time favourites, I show it to Java people all the time ;D \$\endgroup\$
    – cat
    Commented Jun 11, 2017 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.