Optimize this Python Code

Question

See the following equation:

I have made the following code for it:

def i_t(t, t_s, t_max, i_min, i_max):
    if t >= t_max:
        return i_max
    elif t_s <= t < t_max:
        s = (i_max - i_min) / (t_max - t_s)
        return s*(t-t_s)
    else:
        return 0
x_real = pd.DataFrame([[0.322,0.062,0.045,0.05,0.094,0.067,0.045,0.042,0.009,0.047,0.109,0.077,0.051,0.046,0.019,
                      0.033,0.122,0.085,0.028,0.344,0.208,0.166,0.247,0.231,0.149,0.157,0.137,0.061,0.118,0.135,
                      0.152,0.155,0.243,0.27,0.321,0.386,0.444,0.464,0.382,0.397,0.41,0.397,0.42,0.368,0.356,0.322,
                      0.295,0.309,3],
                     [1.01,0.661,0.627,0.662,0.626,0.647,0.638,0.633,0.653,1.07,0.664,0.628,0.661,1.138,1.403,
                      0.998,0.634,0.639,0.618,0.654,0.619,0.655,0.621,0.652,0.624,0.634,0.644,0.623,0.654,0.616,
                      0.653,0.62,0.643,0.635,0.624,0.65,0.621,1.044,1.335,1.371,1.341,1.376,1.386,1.378,1.404,1.366,
                      1.413,1.388,2],
                     [0.484,0.429,0.3,0.146,0.146,0.264,0.242,0.147,0.146,0.16,0.265,0.252,0.144,0.13,0.088,0.194,
                      0.199,0.115,0.088,0.088,0.136,0.572,0.365,0.222,0.369,0.242,0.732,0.457,0.339,0.285,0.645,
                      0.461,0.813,2.317,1.745,0.89,0.193,1.419,1.586,1.068,0.844,0.35,0.355,0.338,0.323,0.331,0.336,
                      0.322,1],
                     [0.467,0.561,0.272,0.265,0.235,0.246,0.258,0.249,0.247,0.265,0.259,0.259,0.405,0.24,0.222,
                      0.134,0.114,0.219,0.573,0.268,0.522,0.233,0.852,1.78,0.397,0.27,2.229,2.738,0.404,0.415,0.527,
                      1.174,2.506,2.333,1.367,1.871,3.831,3.043,1.347,1.026,1.17,1.021,0.837,0.97,0.84,0.861,0.853,
                      1.783,1]], columns=['00:29:59','00:59:59','01:29:59','01:59:59','02:29:59','02:59:59',
                                          '03:29:59','03:59:59','04:29:59','04:59:59','05:29:59','05:59:59',
                                          '06:29:59','06:59:59','07:29:59','07:59:59','08:29:59','08:59:59',
                                          '09:29:59','09:59:59','10:29:59','10:59:59','11:29:59','11:59:59',
                                          '12:29:59','12:59:59','13:29:59','13:59:59','14:29:59','14:59:59',
                                          '15:29:59','15:59:59','16:29:59','16:59:59','17:29:59','17:59:59',
                                          '18:29:59','18:59:59','19:29:59','19:59:59','20:29:59','20:59:59',
                                          '21:29:59','21:59:59','22:29:59','22:59:59','23:29:59','23:59:59','Code'])
x_labels = x_real.columns[0:48]
imin = 0
imax = 0.9
time_arr = np.arange(48)
x_8 = pd.DataFrame(columns=x_labels)
for i, row in x_real.iterrows():
    x_aux = pd.DataFrame(columns=x_labels)
    if row['Code'] == 1:
        it = []
        ts = random.choice(x_labels[36:39])
        tmax = random.choice([t for t in x_labels if t > ts])
        ts = x_real.columns.get_loc(ts)
        tmax = x_real.columns.get_loc(tmax)
        for t in time_arr:
            it.append(i_t(t, ts, tmax, imin, imax))
        x_aux.loc[i] = it
        x_aux.loc[i] = np.asarray([1 - x for x in it])*x_real.iloc[i, 0:48]
        x_aux = x_aux.squeeze(axis=0)
        x_aux = pd.concat([x_aux, x_real.iloc[i, 48:]])
        x_aux = x_aux.to_frame()
        x_8 = pd.concat([x_8, x_aux.T])
    elif row['Code'] == 2:
        it = []
        ts = random.choice(x_labels[15:33])
        tmax = random.choice([t for t in x_labels if t > ts])
        ts = x_real.columns.get_loc(ts)
        tmax = x_real.columns.get_loc(tmax)
        for t in time_arr:
            it.append(i_t(t, ts, tmax, imin, imax))
        x_aux.loc[i] = it
        x_aux.loc[i] = np.asarray([1 - x for x in it])*x_real.iloc[i, 0:48]
        x_aux = x_aux.squeeze(axis=0)
        x_aux = pd.concat([x_aux, x_real.iloc[i, 48:]])
        x_aux = x_aux.to_frame()
        x_8 = pd.concat([x_8, x_aux.T])
    else:
        it = []
        ts = random.choice(x_labels[12:35])
        tmax = random.choice([t for t in x_labels if t > ts])
        ts = x_real.columns.get_loc(ts)
        tmax = x_real.columns.get_loc(tmax)
        for t in time_arr:
            it.append(i_t(t, ts, tmax, imin, imax))
        x_aux.loc[i] = it
        x_aux.loc[i] = np.asarray([1 - x for x in it])*x_real.iloc[i, 0:48]
        x_aux = x_aux.squeeze(axis=0)
        x_aux = pd.concat([x_aux, x_real.iloc[i, 48:]])
        x_aux = x_aux.to_frame()
        x_8 = pd.concat([x_8, x_aux.T])
        

I am getting the results i expect, but since the data i am working on has around 350 thousands rows, it takes forever to run.

My pandas dataframe has 49columns. Each row is the data of a consumer. The first 48 columns contain half-hour measurements of a variable (hence we have 1 day of data at each row) for a single consumer. The last column x_real['Code'] contains a code that represents the type of consumers (it can be 1, 2 or 3). Below some pictures to get the idea of the data i am working on:

So, i need to apply that equation for each row (consumer) of my dataset. And it depends of the value of 'Code', because there are different range of start, ts, and max time, tmax. Based on those random ts and tmax, i need to calculate the slope, s, to determine the values of it so i can finally determine my x_8 dataframe.

How could i optmize this code, so it can run faster? In resume, i need to determine the s value for each row, to be able to compute it and finally create my x_8 dataframe with the values of it.

Answer 1

A few improvements I could see, which you could implement to handle such large datasets and speed up the program, are vectorized operations so it should perform on the entire dataframe at once.

You could also implement parallel processing which should be able to speed up the code significantly, however your hardware would determine how effective this would be for you.

A couple other options would be to implement memory-mapped arrays or streaming data processing techniques to load and process the data in smaller chunks and, as this code acts as a scraper for your dataset, I would always recommend including a caching function in any scraper based code.