I wrote a script that will web scrape data for a list of stocks. The scraper has to get the data from 2 separate pages so each stock symbol must scrape 2 different pages. If I run the process on a list that is 1000 items long it will take around 30 minutes to complete. It's not horrible, I can set it and forget it, but I'm wondering if there is a way to speed up the process. Maybe store the data and wait to write it all at the end instead of on each loop? Any other ideas appreciated.

import requests
from BeautifulSoup import BeautifulSoup
from progressbar import ProgressBar
import csv

pbar = ProgressBar()

with open('industrials.csv', "ab") as csv_file:
    writer = csv.writer(csv_file, delimiter=',')
    writer.writerow(['Symbol','5 Yr EPS','EPS TTM'])
    for s in pbar(symbols):
            url1 = 'https://research.tdameritrade.com/grid/public/research/stocks/fundamentals?symbol='
            full1 = url1 + s
            response1 = requests.get(full1)
            html1 = response1.content
            soup1 = BeautifulSoup(html1)

            for hist_div in soup1.find("div", {"data-module-name": "HistoricGrowthAndShareDetailModule"}):
                EPS5yr = hist_div.find('label').text

        except Exception as e:
            EPS5yr = 'Bad Data'

            url2 = 'https://research.tdameritrade.com/grid/public/research/stocks/summary?symbol='
            full2 = url2 + s
            response2 = requests.get(full2)
            html2 = response2.content
            soup2 = BeautifulSoup(html2)

            for div in soup2.find("div", {"data-module-name": "StockSummaryModule"}):
                EPSttm = div.findAll("dd")[11].text

        except Exception as e:
            EPSttm = "Bad data"


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.