# Calculating difference statistics over a moving window

I'd like to calculate some statistics of given data (frequencies of difference among elements on various distances in percent multiplied by 10) using moving window within that data. Is it possible to speed up the code below? I noticed that some calculations are repeating. But I was not able to exclude them without additional slowness.

def get_dist_stat(pdata, pwin_length):
''' pdata - given data array
pwin_length - the lenght of window
the function returns stat table where
row represents the distance between elements
col represents the difference for that distance in percent multiplied by 10 (assume that maximum difference can be 20 percent)

'''

l_data = len(pdata)
l_win = pwin_length
print("l_data=", l_data)
print("l_win=", l_win)

# stat table
stat_table = np.zeros((l_win-1, 20*10), dtype = int)

# loop over all data
for k in range(l_data - l_win + 1):

win = pdata[k : k + l_win]
print('-' * 10)
print("k=", k, " kend=", k + l_win )

print("win=", win)

# loop over window
for i  in range(1 , l_win):
b=win[i:]
a=win[:-i]
diff=(abs((b-a)/a*100 ) * 10).astype(int)
print("i=",i)
print("b=", b)
print("a=", a)
print("diff=",diff)

# storing found differences into stat table
apercents, acount = np.unique(diff, return_counts = True)
l_apercents = len(apercents)
for j in range(l_apercents):
stat_table[i-1, apercents[j]] += acount[j]
return stat_table

print(astat_table)


And that is its output

adata= [1.1 1.2 1.3 1.4 1.5]
l_data= 5
l_win= 3
----------
k= 0  kend= 3
win= [1.1 1.2 1.3]
i= 1
b= [1.2 1.3]
a= [1.1 1.2]
diff= [90 83]
i= 2
b= [1.3]
a= [1.1]
diff= 
----------
k= 1  kend= 4
win= [1.2 1.3 1.4]
i= 1
b= [1.3 1.4]
a= [1.2 1.3]
diff= [83 76]
i= 2
b= [1.4]
a= [1.2]
diff= 
----------
k= 2  kend= 5
win= [1.3 1.4 1.5]
i= 1
b= [1.4 1.5]
a= [1.3 1.4]
diff= [76 71]
i= 2
b= [1.5]
a= [1.3]
diff= 
[[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
0 0 0 0 2 0 0 0 0 0 0 2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]


You've already made the key observation here, which is that most of the work is redone. Each time you pick a window, most of the calculations are the same as the previous window.

In fact it's much faster to do all the calculations ahead of time into one big ndarray, and then for each window, pick out the calculations that are relevant. So we don't need the temporary a and b lists.

How many dimensions do we need? Just starting point and length. It's going to be a triangular array, so we'll waste some space.

precomputed_results = np.zeros(l_win+1, l_data), dtype = int)
# First pass
for interval in range(1, l_win):
for first_point_index in range(l_data-interval):
# compute diff relative to elements [first_point_index] and [first_point_index+interval]
# line will be similar to precomputed_results[...] = ...

# Second pass
for interval in range(1, l_win):
for first_point_index in range(l_data-interval):
# use slicing on precomputed_results

• It looks like the window is not moving in your code. For example, if we have l_data = 10 and l_win = 7, we won't get difference between elements 8 and 9. – Prokhozhii Jul 26 '18 at 10:13
• @Prokhozhii Um, yes we will: when interval is 1 (which is certainly in range(1, 7), and first_point_index is 8 (which is certainly in range(1, 10-1) then first_point_index + interval is 9. I didn't write all the code, but the comments indicate which elements are getting compared. – Snowbody Jul 27 '18 at 12:57