4
\$\begingroup\$

Skippable Intro:

I have a dataset that corresponds to an observed time-series. The dataset is organized into a dictionary - which contains an array of years, an array of months, an array of days, an array of hours, an array of observed speeds, etc., each obtainable via the appropriate key. My goal is to find all indices at which all input values match the dictionary values of all input keys. I did this by using numpy where on each sub-array from the dictionary (via input keys) to construct sub-array masks of zeros or ones (which alternate by index in range(len(keys))); values are filled in the sub-array masks at indices obtained from the numpy where routine. These filled sub-array masks are then used to construct a multi-dimensional array.

The part I'd like to improve:

The multi-dimensional array, consisting of sub-array masks of indices, is datain the code below. Consider this as a short sample problem.

import numpy as np

# assume a,b,c,d are array masks of indices obtained via numpy where
a = np.array([0, 9, 0]) # mask -- index of years
b = np.array([1, 9, 1]) # mask -- index of months
c = np.array([0, 9, 0]) # mask -- index of days
d = np.array([1, 9, 1]) # mask -- index of hours
data = np.array([a, b, c, d]) # multi-dimensional array of index masks

# routine to get indices where all indices are equal
condition = np.diff(data, axis=0) # get consecutive difference across rows at each column
rows, cols = np.where(condition == np.zeros(len(data)-1)) # get row and column where difference is zero
res = [datum[cols[0]] for datum in data] # check that values agree

# print(data)
[[0 9 0]
 [1 9 1]
 [0 9 0]
 [1 9 1]]

# print(condition)
[[ 1  0  1]
 [-1  0 -1]
 [ 1  0  1]]

# print(rows)
[0 1 2]

# print(cols)
[1 1 1]

# print(res)
[9, 9, 9, 9]

I was initially worried that this algorithm failed in edge-cases where the indices of a column would sum to zero without having all indices of the respective column being equal, but I see this is not the case via print(condition). Are there any bugs I missed? Is there a better implementation of this algorithm?

EDIT:

The algorithm actually fails when adding another sub-array to data. I'm not sure if it's ok to ask in this stack exchange, so I can delete it here and repost it in stackoverflow if necessary.

...
e = np.array([0, 9, 0]) # mask -- index of speeds
data = np.array([a, b, c, d, e]) # multi-dimensional array of index masks
...

# rows, cols = np.where(condition == np.zeros(len(data)-1)) # get row and column where difference is zero
## ^^ fails

Traceback (most recent call last):
   rows, cols = np.where(condition == np.zeros(len(data)-1)) # get row and column where difference is zero
ValueError: not enough values to unpack (expected 2, got 1)


rows, cols = np.where(condition == np.zeros(len(data)-2)) # get row and column where difference is zero
## ^^ works

...

# print(res)
[9, 9, 9, 9, 9]

Why does the original algorithm fail when modifying data?

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.