Downloads historical data from a yahoo finance csv and returns a pandas DataFrame

Question

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to grab the correct cookie and crumb pair and then implemented that into it's own function.

I am new to Pandas and don't know the best practices for accessing data and creating DataFrames. Are there any ways that I can make this project more efficient?

This project can be downloaded here https://github.com/AndrewRPorter/yahoo-historical

import datetime as dt
import pandas as pd
import requests
import re
import csv
import time

class Fetcher:
    def __init__(self, ticker, start, *args):
        self.ticker = ticker.upper()
        self.cookie, self.crumb = self.init()

        self.start = int(time.mktime(dt.datetime(start[0],start[1],start[2]).timetuple()))

        if args:
            end = args[0]
            self.end = int(time.mktime(dt.datetime(end[0],end[1],end[2]).timetuple()))
        else:
            self.end = int(time.time())

    def init(self):
        """Returns a tuple pair of cookie and crumb used in the request"""
        url = 'https://finance.yahoo.com/quote/%s/history' % (self.ticker)
        r = requests.get(url)
        txt = r.content
        cookie = r.cookies['B']
        pattern = re.compile('.*"CrumbStore":\{"crumb":"(?P<crumb>[^"]+)"\}')

        for line in txt.splitlines():
            m = pattern.match(line.decode("utf-8"))
            if m is not None:
                crumb = m.groupdict()['crumb']
        return cookie, crumb  # return a tuple of crumb and cookie

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s" % (self.ticker, self.start, self.end, self.crumb)
        data = requests.get(url, cookies={'B':self.cookie})
        content = data.content.decode("utf-8")
        csv_content = csv.reader(content.splitlines(), delimiter=',')
        return pd.DataFrame(list(csv_content))

    def getDatePrice(self):
        """Returns a DataFrame for Date and Price from getHistorical()"""
        return self.getHistorical().ix[0:,5]

    def getDateVolume(self):
        """Returns a DataFrame for Date and Volume from getHistorical()"""
        return self.getHistorical().ix[0:,6]

Answer 1

You could be using pandas DataFrame.read_csv method, instead of manually going through csv.reader first (you need to go through StringIO, though...):

try:
    from io import StringIO
except ImportError:
    # python 2.x
    from StringIO import StringIO
...

class Fetcher:
    ...

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        data = requests.get(self.url, cookies={'B':self.cookie})
        content = StringIO(data.content.decode("utf-8"))
        return pd.DataFrame.read_csv(content, sep=',')

If you didn't need the cookie, you could just do the following, alas you can't get around that part...

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        return pd.DataFrame.read_csv(self.url, sep=',')

I would also expect a class called Fetcher to accept an url parameter to determine where it is fetching from (hence I already used self.url above). Since it is some arcane API url here, I would make it at least a class variable that can be modified if needed in a sub-class or on-the-fly:

class Fetcher:
    api_url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s"

    def __init__(self, ticker, start, *args):
        self.ticker = ticker.upper()
        self.cookie, self.crumb = self.init()

        self.start = int(time.mktime(dt.datetime(start[0],start[1],start[2]).timetuple()))

        if args:
            end = args[0]
            self.end = int(time.mktime(dt.datetime(end[0],end[1],end[2]).timetuple()))
        else:
            self.end = int(time.time())
        self.url = self.api_url % (self.ticker, self.start, self.end, self.crumb)

    ...

if __name__ == "__main__":
    fetcher = Fetcher(...)

I also added a section for you to actually call the code, guarded by a if __name__ == "__main__": guard.