# Faster way to 3 nested for loop python

I am calculating parameters for a time series and the data size is very big. How can I make my function faster? The following function, param, calculates parameters for a time series model.

Input:

import pandas as pd
import statsmodels.api as sm

data = [['01-01-2018', 150,661,396,286,786],['01-02-2018',231,341,57,768,941], ['01-03-2018',486,526,442,628,621],
['01-04-2018',279,336,140,705,184],['01-05-2018',304,137,800,94,369],['01-06-2018',919,340,372,494,117],
['01-07-2018',947,920,848,716,719],['01-08-2018',423,20,313,368,909],['01-09-2018',422,678,656,604,674],
['01-10-2018',422,678,656,604,674],['01-11-2018',337,501,743,606,991],['01-12-2018',408,536,669,903,463]]
df = pd.DataFrame(data, columns = ['date', 'A','B','C','D','E'])
df.index = df.date

def param(data_param):
w = []
x = []
y = []
z = []
p = d = q = range(0, 2) # Define the p, d and q parameters to take any value between 0 and 2
pdq = list(itertools.product(p, d, q)) # Generate all different combinations of p, q and q triplets
seasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(itertools.product(p, d, q))] # Generate all different combinations of seasonal p, q and q triplets

for i in data_param:
for param in pdq:
for param_seasonal in seasonal_pdq:
try:
results = sm.tsa.statespace.SARIMAX(data_param[i],
order=param,
seasonal_order=param_seasonal,
enforce_stationarity=False,
enforce_invertibility=False).fit()
#print(i,param, param_seasonal,results.aic)
w.append(i)
x.append(param)
y.append(param_seasonal)
z.append(results.aic)
mod_score_param = pd.DataFrame({
'id': w,
'param': x,
'param_seasonal': y,
'results_aic': z,
})
except:
continue
mod_score_param = mod_score_param.sort_values(by='results_aic')
mod_score_param = mod_score_param.dropna()
mod_score_param = mod_score_param[mod_score_param['results_aic']>3.0]
mod_score_param = mod_score_param.sort_values(['id','results_aic'])
mod_score_param = mod_score_param.drop_duplicates(['id'],keep='first')
return(mod_score_param)

Output= param(df)


Output:

+-----+----+-----------+----------------+-------------+
|     | id |   param   | param_seasonal | results_aic |
+-----+----+-----------+----------------+-------------+
| 199 | A  | (1, 0, 1) | (1, 0, 0, 12)  |       4.565 |
| 197 | B  | (1, 0, 0) | (1, 0, 0, 12)  |      21.752 |
|  30 | C  | (0, 1, 1) | (1, 0, 0, 12)  |      87.847 |
|  22 | D  | (1, 1, 1) | (0, 1, 0, 12)  |      91.183 |
|  50 | E  | (0, 1, 1) | (0, 1, 0, 12)  |       92.87 |
+-----+----+-----------+----------------+-------------+

• If you add a sample of data, will we be able to run the code and see it working? – C. Harley Apr 12 at 14:52
• Code is working perfectly fine, but it is very slow. Check, its time series data – A14 Apr 12 at 14:57
• What is sm and what output is produced by that call? Also please add the example input not (only) as a pretty but useless table, but such that it is easy for reviewers to run your code without having to type out your data. – Graipher Apr 12 at 15:08
• ok, did necessary changes – A14 Apr 12 at 15:21
• If I copy/paste your code, I receive the following message Traceback (most recent call last): File "20190413a.py", line 6, in <module> ['01-08-2018',423,20,313,368,909],['01-09-2018',422,678,656,604,674],['01-10-2018',422,678,656,604,674]['01-11-2018',337,501,743,606,991]['01-12-2018',408,536,669,903,463]] TypeError: list indices must be integers or slices, not tuple . Regardless, I fixed the code but your program doesn't output anything like you post. Please confirm working code. – C. Harley Apr 13 at 1:20

Without really understanding what you're attempting to do (and sorry, not enough time to dig into the subject matter), I've rewritten you code a little and profiled it.
The biggest bottleneck I see have to do with the panda/numpy definitions, there is a lot of code querying what your data structure is to determine how to handle it. Perhaps review all your cells in the data frame and ensure they're defined correctly?

Full image (but I'm pretty sure it's not helpful): https://i.imgur.com/nSiOrqu.png

A lot of those hints come from the exceptions which are raised - something you could have found easily if you didn't disable them with continue. More on that later.

Onto the code. Firstly was the imports, make sure you have them defined - when I loaded up your code I had a missing import, and that could have been either your paste (missing) or settings on your computer.

import pandas as pd
import statsmodels.api as sm
import itertools


Next is you're missing the entry point for your python code, always have this as it performs several things. First, it lets people reading your code understand where it begins and ends, secondly if you use tools like an auto-documentor, they load your code to perform reflection on all the objects.
Currently your code would just execute immediately on loading (that's loading, not executing), and that's not good - think of it like your car taking off in gear as soon as you turn the engine on.

if __name__ == "__main__":


Is your standard entry point for python code. Next, we have data initialisations at the top of the code - again, they should be after the entry point, or better yet, external (like an .ini file) and imported at runtime. The code comes out to:

if __name__ == "__main__":
data = [['01-01-2018', 150, 661, 396, 286, 786], ['01-02-2018', 231, 341, 57, 768, 941],
['01-03-2018', 486, 526, 442, 628, 621],
['01-04-2018', 279, 336, 140, 705, 184], ['01-05-2018', 304, 137, 800, 94, 369],
['01-06-2018', 919, 340, 372, 494, 117],
['01-07-2018', 947, 920, 848, 716, 719], ['01-08-2018', 423, 20, 313, 368, 909],
['01-09-2018', 422, 678, 656, 604, 674],
['01-10-2018', 422, 678, 656, 604, 674], ['01-11-2018', 337, 501, 743, 606, 991],
['01-12-2018', 408, 536, 669, 903, 463]]
df = pd.DataFrame(data, columns=['date', 'A', 'B', 'C', 'D', 'E'])
result = param(df)
print(result)


When looking at the function def param(data_param):, the first thing that my IDE highlighted was mod_score_param was used inside a loop on a conditional path - but never initialised outside the loop.
When you write code, you need to be aware of what scope your variables have. Usually variables used inside loops are discarded at the end of the loop. This changes the start of def param(data_param): slightly to:

    seasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(
itertools.product(p, d, q))]  # Generate all different combinations of seasonal p, q and q triplets

mod_score_param = pd.DataFrame({
'id': w,
'param': x,
'param_seasonal': y,
'results_aic': z,
})

for i in data_param:


The next modification I did was to extract the call to the stats package into it's own function, this is because we try to adhere to the Single Responsibility Principle (from S.O.L.I.D) where each function should only perform a single action - it makes it much easier to track down bugs, and we usually only change a single line of code when fixing problems (great when our single change doesn't work - we don't need to debug 10 new lines of code to find out what went wrong on top of the original bug).
Much easier in the long-run (and the code looks very clean). Here is the function:

def model_params(data_param, i, param, param_seasonal):
results = sm.tsa.statespace.SARIMAX(data_param[i],
order=param,
seasonal_order=param_seasonal,
enforce_stationarity=False,
enforce_invertibility=False).fit()
return results.aic


So the caller becomes:

            try:
results = model_params(data_param, i, param, param_seasonal)
w.append(i)
x.append(param)
y.append(param_seasonal)
z.append(results)


The exception was modified slightly too:

            except Exception as e:
print("Exception reached: ", e)
continue


So we can see immediately what is broken and needs fixing. It's important with any coding - you should never let your exceptions be silenced, because it will lead to bigger problems in the future.
If they are known, catch them with the appropriate exception handler, such as ZeroDivisionError or TypeError, and handle them at the time of the exception.
If they don't match any existing error type, create a custom error handler and handle that particular error before continuing on.

That wraps up my code review and attempt to help you find the slow down in your code. Please apply these changes and see if the modifications to your data frame definitions (such as defining the frequency of the date strings - that's one of the warnings I saw) make the intended improvements.
Good Luck!

• Thank you @C. Harley. I changed the code & it seems little speed increased. If I want to use multiprocessing how will I use? For same. Any guidance? – A14 Apr 22 at 13:21
• I'd start with threading to work out the engineering challenges of having multiple processes writing back into the dataframe in an out-of-order method and a thread-safe work queue. Multiprocessing is a little heavier as each spawned mp object is a full copy of Python, and you need to work on heavier data sharing techniques (doable, but faster to thread then mp). Firstly, I'd spawn the threads in daemon mode (pointing at the model_params function monitoring a queue), then each loop place a copy of the data onto the queue. Results to a second queue, then reintegrate and compare results match. – C. Harley Apr 23 at 15:15