I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from the last 3 years. When I run my code it takes hours for only a period of a few days. Does anyone have any tips for me on how I can optimize my code to run faster? Sorry in advance for the probably trivial question, but I'm a beginner and I am trying to get started gradually.

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

options = Options()
options.headless = True

web = "https://stocktwits.com/search"
driver = webdriver.Chrome(r"C:\Users\veron\Downloads\chromedriver\chromedriver.exe", options=options)

username = driver.find_element_by_xpath('//input[@placeholder = "Symbol or @Username"]')




def get_tweet(element):
        user = element.find_element_by_xpath('.//span[@class = "st_2JY3sEE"]/a[contains(@href, "/")]/span[text()]').text
        text = element.find_element_by_xpath('.//div[@class="st_3SL2gug"]').text
        date = element.find_element_by_xpath('.//a[@class ="st_28bQfzV st_1E79qOs st_3TuKxmZ st_1VMMH6S"]').text
        # date = date_old.replace("\n", "")
        tweet_data = [user, text, date]
        tweet_data = ['user', 'text', 'date']
    return tweet_data

user_data = []
text_data = []
date_data = []
scrolling = True

while scrolling:

    tweets = WebDriverWait(driver, 5).until(
        EC.presence_of_all_elements_located((By.XPATH, '//div[@class = "st_2o0zabc st_jGV698i st_PLa30pM"]')))
    # print(len(tweets))
    for tweet in tweets:
        tweet_list = get_tweet(tweet)
        text_data.append(" ".join(tweet_list[1].split()))

    # Get the initial scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")
    # Specified date
    str1 = "4/30/22"

    while True:
        # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        # Wait to load page
        # Calculate new scroll height and compare it with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        # check if the date substring from above is in the date list, condition 1
        res = any(str1 in string for string in date_data)
        if res is True:
            scrolling = False
        # condition 2
        if new_height == last_height:
            scrolling = False

            last_height = new_height

df_tweets = pd.DataFrame({'user': user_data, 'text': text_data, 'date': date_data})  # , 'date': date_data
df_tweets.to_csv('stocktwits_tweets.csv', index=False)
  • 2
    \$\begingroup\$ There are many good, free Twitter API libraries. Why not use one? \$\endgroup\$
    – Reinderien
    Commented May 1, 2022 at 17:08
  • \$\begingroup\$ Unfortunately I have to use StockTwits, and StockTwits currently doesn't accept new registrations for the API. \$\endgroup\$
    – VaKt
    Commented May 1, 2022 at 17:41
  • 1
    \$\begingroup\$ Why do you have to use StockTwits? \$\endgroup\$
    – Reinderien
    Commented May 1, 2022 at 17:48
  • \$\begingroup\$ It's for a replication of a research paper \$\endgroup\$
    – VaKt
    Commented May 1, 2022 at 19:01
  • \$\begingroup\$ The answer is probably you are rate limited by their API by design. \$\endgroup\$
    – qwr
    Commented May 1, 2022 at 20:54

1 Answer 1


Does anyone have any tips for me on how I can optimize my code to run faster?

Measure what's slow. A typical approach is:

  • Select some block of code you think might be slow. Try to make it about half the code if possible
  • Measure the time before the start of the code and after the end
  • Subtract the two to check how long it took. If you're getting a rate of 60 requests per second, each one is taking 0.016 seconds. If the code takes 0.015 seconds, it's the "slow half". If the code takes 0.0001 seconds, it's the "fast half".
  • Slowly narrow in on what the slowest part of the code is

Once you know what's slowest, you may already have a solution. Or if not, feel free to ask someone at that point. By doing this investigation yourself, you're respecting peoples' time more, and much more likely to get an answer because you showed you did the initial work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.