3
\$\begingroup\$

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to grab the correct cookie and crumb pair and then implemented that into it's own function.

I am new to Pandas and don't know the best practices for accessing data and creating DataFrames. Are there any ways that I can make this project more efficient?

This project can be downloaded here https://github.com/AndrewRPorter/yahoo-historical

import datetime as dt
import pandas as pd
import requests
import re
import csv
import time

class Fetcher:
    def __init__(self, ticker, start, *args):
        self.ticker = ticker.upper()
        self.cookie, self.crumb = self.init()

        self.start = int(time.mktime(dt.datetime(start[0],start[1],start[2]).timetuple()))

        if args:
            end = args[0]
            self.end = int(time.mktime(dt.datetime(end[0],end[1],end[2]).timetuple()))
        else:
            self.end = int(time.time())

    def init(self):
        """Returns a tuple pair of cookie and crumb used in the request"""
        url = 'https://finance.yahoo.com/quote/%s/history' % (self.ticker)
        r = requests.get(url)
        txt = r.content
        cookie = r.cookies['B']
        pattern = re.compile('.*"CrumbStore":\{"crumb":"(?P<crumb>[^"]+)"\}')

        for line in txt.splitlines():
            m = pattern.match(line.decode("utf-8"))
            if m is not None:
                crumb = m.groupdict()['crumb']
        return cookie, crumb  # return a tuple of crumb and cookie

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s" % (self.ticker, self.start, self.end, self.crumb)
        data = requests.get(url, cookies={'B':self.cookie})
        content = data.content.decode("utf-8")
        csv_content = csv.reader(content.splitlines(), delimiter=',')
        return pd.DataFrame(list(csv_content))

    def getDatePrice(self):
        """Returns a DataFrame for Date and Price from getHistorical()"""
        return self.getHistorical().ix[0:,5]

    def getDateVolume(self):
        """Returns a DataFrame for Date and Volume from getHistorical()"""
        return self.getHistorical().ix[0:,6]
\$\endgroup\$
1
\$\begingroup\$

You could be using pandas DataFrame.read_csv method, instead of manually going through csv.reader first (you need to go through StringIO, though...):

try:
    from io import StringIO
except ImportError:
    # python 2.x
    from StringIO import StringIO
...

class Fetcher:
    ...

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        data = requests.get(self.url, cookies={'B':self.cookie})
        content = StringIO(data.content.decode("utf-8"))
        return pd.DataFrame.read_csv(content, sep=',')

If you didn't need the cookie, you could just do the following, alas you can't get around that part...

    def getHistorical(self):
        """Returns a list of historical data from Yahoo Finance"""
        return pd.DataFrame.read_csv(self.url, sep=',')

I would also expect a class called Fetcher to accept an url parameter to determine where it is fetching from (hence I already used self.url above). Since it is some arcane API url here, I would make it at least a class variable that can be modified if needed in a sub-class or on-the-fly:

class Fetcher:
    api_url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s"

    def __init__(self, ticker, start, *args):
        self.ticker = ticker.upper()
        self.cookie, self.crumb = self.init()

        self.start = int(time.mktime(dt.datetime(start[0],start[1],start[2]).timetuple()))

        if args:
            end = args[0]
            self.end = int(time.mktime(dt.datetime(end[0],end[1],end[2]).timetuple()))
        else:
            self.end = int(time.time())
        self.url = self.api_url % (self.ticker, self.start, self.end, self.crumb)

    ...

if __name__ == "__main__":
    fetcher = Fetcher(...)

I also added a section for you to actually call the code, guarded by a if __name__ == "__main__": guard.

\$\endgroup\$
5
  • 1
    \$\begingroup\$ Thanks! I noticed that there are different imports depending on which version of python I am running. What would be the best way to import based on the running version? \$\endgroup\$ Jul 14 '17 at 17:34
  • \$\begingroup\$ By wrapping it in a try: ... except ImportError: stackoverflow.com/questions/342437/… \$\endgroup\$
    – Graipher
    Jul 14 '17 at 17:35
  • \$\begingroup\$ @APorter1031 Added it to the answer. \$\endgroup\$
    – Graipher
    Jul 14 '17 at 17:38
  • \$\begingroup\$ Why would I need an if __name__ ... if I am distributing this code as an API? \$\endgroup\$ Jul 17 '17 at 0:28
  • 1
    \$\begingroup\$ @APorter1031 you could add someexamole or test code there. Then you could run the tests by directly running the script but not when importing it from another file \$\endgroup\$
    – Graipher
    Jul 17 '17 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.