2
\$\begingroup\$

I am calculating parameters for a time series and the data size is very big. How can I make my function faster? The following function, param, calculates parameters for a time series model.

Input:

import pandas as pd
import statsmodels.api as sm



  data = [['01-01-2018', 150,661,396,286,786],['01-02-2018',231,341,57,768,941], ['01-03-2018',486,526,442,628,621],
    ['01-04-2018',279,336,140,705,184],['01-05-2018',304,137,800,94,369],['01-06-2018',919,340,372,494,117],
    ['01-07-2018',947,920,848,716,719],['01-08-2018',423,20,313,368,909],['01-09-2018',422,678,656,604,674],
    ['01-10-2018',422,678,656,604,674],['01-11-2018',337,501,743,606,991],['01-12-2018',408,536,669,903,463]]
df = pd.DataFrame(data, columns = ['date', 'A','B','C','D','E'])
df.index = df.date 

def param(data_param):
    w = []
    x = []
    y = []
    z = []
    p = d = q = range(0, 2) # Define the p, d and q parameters to take any value between 0 and 2
    pdq = list(itertools.product(p, d, q)) # Generate all different combinations of p, q and q triplets
    seasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(itertools.product(p, d, q))] # Generate all different combinations of seasonal p, q and q triplets    


    for i in data_param:
        for param in pdq:
            for param_seasonal in seasonal_pdq:
                try:
                    results = sm.tsa.statespace.SARIMAX(data_param[i],
                                                        order=param, 
                                                        seasonal_order=param_seasonal, 
                                                        enforce_stationarity=False, 
                                                        enforce_invertibility=False).fit()
                    #print(i,param, param_seasonal,results.aic)
                    w.append(i)
                    x.append(param)
                    y.append(param_seasonal)
                    z.append(results.aic)
                    mod_score_param = pd.DataFrame({
                        'id': w,
                        'param': x,
                        'param_seasonal': y,
                        'results_aic': z,
                        })
                except:
                    continue
    mod_score_param = mod_score_param.sort_values(by='results_aic')
    mod_score_param = mod_score_param.dropna()
    mod_score_param = mod_score_param[mod_score_param['results_aic']>3.0]
    mod_score_param = mod_score_param.sort_values(['id','results_aic'])
    mod_score_param = mod_score_param.drop_duplicates(['id'],keep='first')
    return(mod_score_param)

Output= param(df)

Output:

+-----+----+-----------+----------------+-------------+
|     | id |   param   | param_seasonal | results_aic |
+-----+----+-----------+----------------+-------------+
| 199 | A  | (1, 0, 1) | (1, 0, 0, 12)  |       4.565 |
| 197 | B  | (1, 0, 0) | (1, 0, 0, 12)  |      21.752 |
|  30 | C  | (0, 1, 1) | (1, 0, 0, 12)  |      87.847 |
|  22 | D  | (1, 1, 1) | (0, 1, 0, 12)  |      91.183 |
|  50 | E  | (0, 1, 1) | (0, 1, 0, 12)  |       92.87 |
+-----+----+-----------+----------------+-------------+
\$\endgroup\$
  • \$\begingroup\$ If you add a sample of data, will we be able to run the code and see it working? \$\endgroup\$ – C. Harley Apr 12 at 14:52
  • \$\begingroup\$ Code is working perfectly fine, but it is very slow. Check, its time series data \$\endgroup\$ – A14 Apr 12 at 14:57
  • 2
    \$\begingroup\$ What is sm and what output is produced by that call? Also please add the example input not (only) as a pretty but useless table, but such that it is easy for reviewers to run your code without having to type out your data. \$\endgroup\$ – Graipher Apr 12 at 15:08
  • \$\begingroup\$ ok, did necessary changes \$\endgroup\$ – A14 Apr 12 at 15:21
  • 2
    \$\begingroup\$ If I copy/paste your code, I receive the following message Traceback (most recent call last): File "20190413a.py", line 6, in <module> ['01-08-2018',423,20,313,368,909],['01-09-2018',422,678,656,604,674],['01-10-2018',422,678,656,604,674]['01-11-2018',337,501,743,606,991]['01-12-2018',408,536,669,903,463]] TypeError: list indices must be integers or slices, not tuple . Regardless, I fixed the code but your program doesn't output anything like you post. Please confirm working code. \$\endgroup\$ – C. Harley Apr 13 at 1:20
1
\$\begingroup\$

Without really understanding what you're attempting to do (and sorry, not enough time to dig into the subject matter), I've rewritten you code a little and profiled it.
The biggest bottleneck I see have to do with the panda/numpy definitions, there is a lot of code querying what your data structure is to determine how to handle it. Perhaps review all your cells in the data frame and ensure they're defined correctly?

Profiling Output

Full image (but I'm pretty sure it's not helpful): https://i.imgur.com/nSiOrqu.png

A lot of those hints come from the exceptions which are raised - something you could have found easily if you didn't disable them with continue. More on that later.

Exceptions

Onto the code. Firstly was the imports, make sure you have them defined - when I loaded up your code I had a missing import, and that could have been either your paste (missing) or settings on your computer.

import pandas as pd
import statsmodels.api as sm
import itertools

Next is you're missing the entry point for your python code, always have this as it performs several things. First, it lets people reading your code understand where it begins and ends, secondly if you use tools like an auto-documentor, they load your code to perform reflection on all the objects.
Currently your code would just execute immediately on loading (that's loading, not executing), and that's not good - think of it like your car taking off in gear as soon as you turn the engine on.

if __name__ == "__main__":

Is your standard entry point for python code. Next, we have data initialisations at the top of the code - again, they should be after the entry point, or better yet, external (like an .ini file) and imported at runtime. The code comes out to:

if __name__ == "__main__":
    data = [['01-01-2018', 150, 661, 396, 286, 786], ['01-02-2018', 231, 341, 57, 768, 941],
            ['01-03-2018', 486, 526, 442, 628, 621],
            ['01-04-2018', 279, 336, 140, 705, 184], ['01-05-2018', 304, 137, 800, 94, 369],
            ['01-06-2018', 919, 340, 372, 494, 117],
            ['01-07-2018', 947, 920, 848, 716, 719], ['01-08-2018', 423, 20, 313, 368, 909],
            ['01-09-2018', 422, 678, 656, 604, 674],
            ['01-10-2018', 422, 678, 656, 604, 674], ['01-11-2018', 337, 501, 743, 606, 991],
            ['01-12-2018', 408, 536, 669, 903, 463]]
    df = pd.DataFrame(data, columns=['date', 'A', 'B', 'C', 'D', 'E'])
    result = param(df)
    print(result)

When looking at the function def param(data_param):, the first thing that my IDE highlighted was mod_score_param was used inside a loop on a conditional path - but never initialised outside the loop.
When you write code, you need to be aware of what scope your variables have. Usually variables used inside loops are discarded at the end of the loop. This changes the start of def param(data_param): slightly to:

    seasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(
        itertools.product(p, d, q))]  # Generate all different combinations of seasonal p, q and q triplets

    mod_score_param = pd.DataFrame({
        'id': w,
        'param': x,
        'param_seasonal': y,
        'results_aic': z,
    })

    for i in data_param:

The next modification I did was to extract the call to the stats package into it's own function, this is because we try to adhere to the Single Responsibility Principle (from S.O.L.I.D) where each function should only perform a single action - it makes it much easier to track down bugs, and we usually only change a single line of code when fixing problems (great when our single change doesn't work - we don't need to debug 10 new lines of code to find out what went wrong on top of the original bug).
Much easier in the long-run (and the code looks very clean). Here is the function:

def model_params(data_param, i, param, param_seasonal):
    results = sm.tsa.statespace.SARIMAX(data_param[i],
                                        order=param,
                                        seasonal_order=param_seasonal,
                                        enforce_stationarity=False,
                                        enforce_invertibility=False).fit()
    return results.aic

So the caller becomes:

            try:
                results = model_params(data_param, i, param, param_seasonal)
                w.append(i)
                x.append(param)
                y.append(param_seasonal)
                z.append(results)

The exception was modified slightly too:

            except Exception as e:
                print("Exception reached: ", e)
                continue

So we can see immediately what is broken and needs fixing. It's important with any coding - you should never let your exceptions be silenced, because it will lead to bigger problems in the future.
If they are known, catch them with the appropriate exception handler, such as ZeroDivisionError or TypeError, and handle them at the time of the exception.
If they don't match any existing error type, create a custom error handler and handle that particular error before continuing on.

That wraps up my code review and attempt to help you find the slow down in your code. Please apply these changes and see if the modifications to your data frame definitions (such as defining the frequency of the date strings - that's one of the warnings I saw) make the intended improvements.
Good Luck!

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.