1
\$\begingroup\$

The function findMinimal() receives a tuple from a list which contains 171 tuples, each of them go into the function one by one and, for each tuple, it searches the best pair via minimalTscore out of those that match by industry (approx. 20 names, for each one). How can I make the function to run faster? At the moment, it's quite slow.

def findMinimal(tup): #Finds cointegrated pair with minimal t-score
    stockPair = []
    stock = tup[0]
    industry = tup[1]
    minimalTscore = 0
    stockListLocal = [i for i in stockList if i[0] != stock]
    clf = linear_model.LinearRegression()
    for i in stockListLocal:
        if i[1] == industry and stock != i[0]:
            Y = data[stock][datesIterable[startTrain:endTrain]]
            X = data[i[0]][datesIterable[startTrain:endTrain]]
            clf.fit(np.array(X).reshape(len(X),1), np.array(Y).reshape(len(Y),1))
            beta_hr = clf.coef_[0]
            alpha = clf.intercept_[0]
            residuals = Y - beta_hr*X - alpha
            cadf = ts.adfuller(residuals)

            if cadf[4]['1%'] > cadf[0]:
                if hurst(residuals) < 0.4:
                    if minimalTscore == 0:
                        minimalTscore = cadf[0]
                        stockPair = i[0]
                    else:
                        if cadf[0] < minimalTscore:
                            minimalTscore = cadf[0]
                            stockPair = i[0]
    if (stockPair,stock) not in pairsOnlyStocks and len(stockPair) > 0:
        pairs.append((stock, stockPair, minimalTscore)) #Y, X, tscore
        pairsOnlyStocks.append((stock, stockPair)) #this is only for filtering
\$\endgroup\$
  • 1
    \$\begingroup\$ There are a bunch of symbols that are not defined in this function. Can you provide a bit more context so we understand more what is going on. The code calling this function would also be of great value. \$\endgroup\$ – Mathias Ettinger May 5 '16 at 16:28
1
\$\begingroup\$

1. Review

  1. As Mathias Ettinger points out in comments, it is hard to review this code because we have no idea what the global variables stockList, linear_model, data, datesIterable, startTrain, endTrain, ts, hurst, pairsOnlyStocks and pairs mean.

  2. There's no docstring. What does this function do? What parameters does it take? What does it return? What side effects does it cause?

  3. Code that depends on global variables is hard to understand, hard to test, and hard to parallelize. It would be better to turns some of these variables into function parameters, or into attributes on objects belonging to a class for which findMinimal is a method.

  4. The code filters stockList on i[0] != stock to get stockListLocal. But then it iterates over stockListLocal and filters again on the same condition. This is wasted work.

  5. Each time findMinimal is called, it looks at all the stocks and filters on i[1] == industry. If you maintained a list of stocks by industry then you would could save a lot of work.

  6. pairsOnlyStocks seems to be a list (it has an append method). Lists do not have an efficient membership test: Python has to compare the item against each list member in turn. So (stockPair,stock) not in pairsOnlyStocks will be slow. It would be better to use a set.

  7. minimalTscore starts at zero, which requires the code to have a special case if minimalTscore == 0:. Better to use Python's built-in min function and avoid the special case.

  8. Instead of indexing:

    stock = tup[0]
    industry = tup[1]
    

    use tuple assignment:

    stock, industry, *_ = tup
    

    But note that this only works in Python 3. In Python 2 you can write:

    stock, industry = tup[:2]
    

    and I think this is still clearer than indexing.

  9. Similarly, instead of:

    cadf = ts.adfuller(residuals)
    

    use tuple assignment with wildcards:

    score, _, _, _, critical_values, *_ = ts.adfuller(residuals)
    
  10. There's duplicated code of the form:

    data[stock][datesIterable[startTrain:endTrain]]
    

    Putting this in a function would reduce duplication and make the code easier to follow, and it would be another place to add a docstring.

  11. Instead of:

    .reshape(len(X),1)
    

    write:

    .reshape(-1, 1)
    
  12. The computation of residuals:

    Y - beta_hr*X - alpha
    

    suggests that X and Y are already NumPy arrays and so you don't need to call np.array on them.

  13. The array Y gets computed each again for each stock in the list. It could be computed just once by moving it out of the loop.

  14. The arbitrary values '1%' and 0.4 should be given names and documented, for example by turning then into optional function parameters.

2. Revised code

This code implements some of the suggestions above:

def stock_data(stock):
    """Return stock data for [explanation here]."""
    return data[stock][datesIterable[startTrain:endTrain]]

def add_best_pair(stock_tuple, significance='1%', max_hurst_exponent=0.4):
    """Given a tuple (stock0, industry, ...), find the tuple (stock1,
    industry, ...) in the global stocks such that the pair
    (stock0, stock1) has the smallest Augmented Dickey-Fuller score
    among all such pairs, and add it to the global pairs if not
    already present in either order.

    Optional parameter significance gives the required significance
    level for the Augmented Dickey-Fuller test, and max_hurst_exponent
    gives the maximum Hurst exponent for the autocorrelations of the
    residuals.

    """
    stock0, industry0, *_ = stock_tuple
    Y = stock_data(stock0)
    clf = linear_model.LinearRegression()
    def scores():
        for stock1, industry1, *_ in stocks:
            if industry0 == industry1 and stock0 != stock1:
                X = stock_data(stock1)
                clf.fit(X.reshape(-1, 1), Y.reshape(-1, 1))
                beta_hr = clf.coef_[0]
                alpha = clf.intercept_[0]
                residuals = Y - beta_hr*X - alpha
                score, _, _, _, critical_values, *_ = ts.adfuller(residuals)
                if (score <= critical_values[significance]
                    and hurst(residuals) <= max_hurst_exponent):
                    yield score, stock1
    try:
        _, stock1 = min(scores())
    except ValueError:
        # None of the candidate pairs passed the statistical tests.
        return
    if stock1, stock0 not in pairs_set:
        stock_pair = stock0, stock1
        pairs.append(stock_pair)
        pairs_set.add(stock_pair)

3. Performance

It's hard to see how to implement this without carrying out the linear regression and Augmented Dickey-Fuller test on all pairs of stocks in each industry. But here are some ideas on speeding it up:

  1. Are the statistical tests symmetric? That is, do we expect to get the same score when comparing stock A with stock B as we do when we compare stock B with stock A? If so, it would be possible to cache the score computed for (A, B) and look it up when computing (B, A). That would save half the work.

  2. It might be possible to use a rough test to quickly reject pairs of stocks that won't pass the statistical tests. For example, you could start by testing daily/weekly/monthly averages, and only if those test pass do you go on to test the high-resolution data.

  3. It might be straightforward to parallelize the computation of statistical tests, using concurrent.futures.ThreadPoolExecutor in Python 3, or the concurrent.futures backport in Python 2.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.