Finding cointegrated pair with minimal t-score

The function findMinimal() receives a tuple from a list which contains 171 tuples, each of them go into the function one by one and, for each tuple, it searches the best pair via minimalTscore out of those that match by industry (approx. 20 names, for each one). How can I make the function to run faster? At the moment, it's quite slow.

def findMinimal(tup): #Finds cointegrated pair with minimal t-score
stockPair = []
stock = tup[0]
industry = tup[1]
minimalTscore = 0
stockListLocal = [i for i in stockList if i[0] != stock]
clf = linear_model.LinearRegression()
for i in stockListLocal:
if i[1] == industry and stock != i[0]:
Y = data[stock][datesIterable[startTrain:endTrain]]
X = data[i[0]][datesIterable[startTrain:endTrain]]
clf.fit(np.array(X).reshape(len(X),1), np.array(Y).reshape(len(Y),1))
beta_hr = clf.coef_[0]
alpha = clf.intercept_[0]
residuals = Y - beta_hr*X - alpha

if hurst(residuals) < 0.4:
if minimalTscore == 0:
stockPair = i[0]
else:
stockPair = i[0]
if (stockPair,stock) not in pairsOnlyStocks and len(stockPair) > 0:
pairs.append((stock, stockPair, minimalTscore)) #Y, X, tscore
pairsOnlyStocks.append((stock, stockPair)) #this is only for filtering

• There are a bunch of symbols that are not defined in this function. Can you provide a bit more context so we understand more what is going on. The code calling this function would also be of great value. – 301_Moved_Permanently May 5 '16 at 16:28

1. Review

1. As Mathias Ettinger points out in comments, it is hard to review this code because we have no idea what the global variables stockList, linear_model, data, datesIterable, startTrain, endTrain, ts, hurst, pairsOnlyStocks and pairs mean.

2. There's no docstring. What does this function do? What parameters does it take? What does it return? What side effects does it cause?

3. Code that depends on global variables is hard to understand, hard to test, and hard to parallelize. It would be better to turns some of these variables into function parameters, or into attributes on objects belonging to a class for which findMinimal is a method.

4. The code filters stockList on i[0] != stock to get stockListLocal. But then it iterates over stockListLocal and filters again on the same condition. This is wasted work.

5. Each time findMinimal is called, it looks at all the stocks and filters on i[1] == industry. If you maintained a list of stocks by industry then you would could save a lot of work.

6. pairsOnlyStocks seems to be a list (it has an append method). Lists do not have an efficient membership test: Python has to compare the item against each list member in turn. So (stockPair,stock) not in pairsOnlyStocks will be slow. It would be better to use a set.

7. minimalTscore starts at zero, which requires the code to have a special case if minimalTscore == 0:. Better to use Python's built-in min function and avoid the special case.

stock = tup[0]
industry = tup[1]


use tuple assignment:

stock, industry, *_ = tup


But note that this only works in Python 3. In Python 2 you can write:

stock, industry = tup[:2]


and I think this is still clearer than indexing.

cadf = ts.adfuller(residuals)


use tuple assignment with wildcards:

score, _, _, _, critical_values, *_ = ts.adfuller(residuals)

10. There's duplicated code of the form:

data[stock][datesIterable[startTrain:endTrain]]


Putting this in a function would reduce duplication and make the code easier to follow, and it would be another place to add a docstring.

.reshape(len(X),1)


write:

.reshape(-1, 1)

12. The computation of residuals:

Y - beta_hr*X - alpha


suggests that X and Y are already NumPy arrays and so you don't need to call np.array on them.

13. The array Y gets computed each again for each stock in the list. It could be computed just once by moving it out of the loop.

14. The arbitrary values '1%' and 0.4 should be given names and documented, for example by turning then into optional function parameters.

2. Revised code

This code implements some of the suggestions above:

def stock_data(stock):
"""Return stock data for [explanation here]."""
return data[stock][datesIterable[startTrain:endTrain]]

"""Given a tuple (stock0, industry, ...), find the tuple (stock1,
industry, ...) in the global stocks such that the pair
(stock0, stock1) has the smallest Augmented Dickey-Fuller score
among all such pairs, and add it to the global pairs if not

Optional parameter significance gives the required significance
level for the Augmented Dickey-Fuller test, and max_hurst_exponent
gives the maximum Hurst exponent for the autocorrelations of the
residuals.

"""
stock0, industry0, *_ = stock_tuple
Y = stock_data(stock0)
clf = linear_model.LinearRegression()
def scores():
for stock1, industry1, *_ in stocks:
if industry0 == industry1 and stock0 != stock1:
X = stock_data(stock1)
clf.fit(X.reshape(-1, 1), Y.reshape(-1, 1))
beta_hr = clf.coef_[0]
alpha = clf.intercept_[0]
residuals = Y - beta_hr*X - alpha
score, _, _, _, critical_values, *_ = ts.adfuller(residuals)
if (score <= critical_values[significance]
and hurst(residuals) <= max_hurst_exponent):
yield score, stock1
try:
_, stock1 = min(scores())
except ValueError:
# None of the candidate pairs passed the statistical tests.
return
if stock1, stock0 not in pairs_set:
stock_pair = stock0, stock1
pairs.append(stock_pair)

3. It might be straightforward to parallelize the computation of statistical tests, using concurrent.futures.ThreadPoolExecutor in Python 3, or the concurrent.futures backport in Python 2.