GET multiple data frames using external module's function that uses requests

Question

Purpose

The point of this mini project is to quickly gather data from a website's API and combine the collected DataFrames into a "master" DataFrame with all of the stocks I am interested in looking at. I do this using the iexfinance module. If needed, here is the code behind iexfinance.

Improvements?

While this module does a great job of dealing with the API, I'm not sure my code is as fast as it could be. Because the iexfinance module uses requests to make GET requests, I believe there should be a faster way to asynchronously send GET requests instead of having to use multiprocessing. Below is my code:

import pandas as pd
import numpy as np
from iexfinance import Stock
from multiprocessing import Pool 
import os


def iex_get_stat(batch):
    """ 
    :param batch: batch is a list of stock tickers (ex: ["AAPL", "MSFT", "TSLA"]
    Gets and returns DataFrames of stats on a list of stocks 
    """
    frame = Stock(batch, output_format="pandas").get_key_stats().T
    return frame


def get_stats(symbols):
    """
    :param symbols: List of symbols (can only send 100 at a time, 
    so these may need to be broken up)
    :return: Pandas DataFrame with stats
    """
    symbols = [symbols[i : i + 99] for i in range(0, len(symbols), 99)]
    frames = []
    pool = Pool(processes=os.cpu_count())
    frames.append(pool.starmap(iex_get_stat, [[batch] for batch in symbols]))
    pool.close()
    pool.join()
    stats = pd.concat(frames[0])
    return stats

if __name__ == "__main__":
    symbols = ["AAPL", "TSLA", "MSFT"]
    stats = get_stats(symbols)
    print(stats)

Answer 1

• Your docstrings look a bit weird to me. The first line is usually a summary of what the function does. In your first function that line comes after the parameters and in the second one it does not exist at all. In get_stats when describing the parameter symbols there is no need to describe what the function internally does with it (i.e. split it up). This is an implementation detail (if it mutated the original list that might be worth mentioning but you don't do that).

• I would put the chunking line into its own function, so you could change it if you want to at some time:

  def chunks(it, n):
      """Splits `it` into chunks of length `n` (last chunk might be smaller)."""
     return [it[i : i + n] for i in range(0, len(it), n)]
  

• The processing part is unnecessarily long. You could just write:

  def get_stats(symbols):
      """
      :param symbols: List of symbols (can only send 100 at a time, 
      so these may need to be broken up)
      :return: Pandas DataFrame with stats
      """
      with Pool() as pool:
          return pd.concat(pool.starmap(iex_get_stat, chunks(symbols, 99))))
  

  Note that when exiting, this calls pool.terminate() instead of pool.close(); pool.join(). If you want to switch to the async methods this means that it will not wait for your workers to finish, but since you are not (I think), it should be fine.

  Pool also automatically chooses os.cpu_count() when no number of processes is passed.

• I would also move encapsulating each batch within another list into iex_get_stat, so you can just pass it a list instead (which is then actually what it says in the docstring):

  def iex_get_stat(batch):
      """ 
      :param batch: batch is a list of stock tickers (ex: ["AAPL", "MSFT", "TSLA"]
      Gets and returns DataFrames of stats on a list of stocks 
      """
      return Stock([batch], output_format="pandas").get_key_stats().T