1
\$\begingroup\$

I have written a script that I believe works and covers all edge-cases. I am curious about ways to improve upon speed. While the given example below covers a multi-dimensional array of 3 rows x 10 columns, my actual use case will be n rows x ~70,000 columns (where n depends upon the number of data parameters being searched).

Given individual arrays of data points, the goal is to combine them into a multi-dimensional array and find the columns in which all conditions are satisfied. If the same column of each row satisfies a given condition, the index that corresponds to that column is output; otherwise, an error is raised.

I have included a small class named MaskOps() because it has many other functions relevant in my main code, though I've only included the parts relevant to the goal in this question.

import numpy as np

class MaskOps():

    @staticmethod
    def get_base(shape, value, dtype=int):
        """ This function produces a base-mask, the values of which may be overwritten. """
        if isinstance(value, (float, int)):
            res = np.ones(shape, dtype=dtype) * value
        elif isinstance(value, str):
            res = np.array([value for idx in range(np.prod(shape))]).reshape(shape)
        return res

    @staticmethod
    def alternate_base(shape, key):
        """ This function creates base-masks that consist of one of two value; the value depends on the index input as the parameter key. """
        if key % 2 == 0:
            value = 0.25
        else:
            value = 0.5
        return MaskOps().get_base(shape, value, dtype=float)

MO = MaskOps()

Sample Data

row_a = np.linspace(1, 10, 10)
row_b = row_a * 10
row_c = row_a + 20
data = np.array([row_a, row_b, row_c])

Main Search Function

def core_algorithm(ndata, search_value):
    """ 
    This function prints values and indices that match the search condition. 

    An index mask of non-zero values is created per row of the input data, 
    and the values of the index mask are overwritten to be a zero at each
    column at which the condition is satisfied - per condition and row of data.
    Then, the columns of the index masks that sum to zero are the column-indices 
    that satisfy all input conditions.
    """

    print("\nSEARCH VALUES:\n{}\n".format(search_value))
    print("NDATA:\n{}\n".format(ndata))

    bases = np.array([MO.alternate_base(len(ndata.T), idx) for idx in range(len(ndata))])
    print("ORIGINAL BASES:\n{}\n".format(bases))

    locs = np.array([np.where(ndata[idx] == search_value[idx])[0] for idx in range(len(search_value))])
    print("LOCS:\n{}\n".format(locs))

    for idx in range(len(bases)):
        bases[idx][locs[idx]] = 0
    print("UPDATED BASES:\n{}\n".format(bases))

    res_idx = np.where(np.sum(bases, axis=0) == 0)[0]
    print("RES COLUMN:\n{}\n".format(res_idx))

    if len(res_idx) == 0:
        raise ValueError("match could not be found")

    res_val = np.array([ndata[idx][res_idx] for idx in range(len(ndata))])
    print("VALUES FROM COL-INDICES\n{}\n".format(res_val))

core_algorithm(data, search_value=(3, 30, 23)) # works successfully
# core_algorithm(data, search_value=(3, 30, 24)) # throws an error

One alternative method I have yet to explore is using set intersection/unions to find the same indices, though I'm not sure if that would necessarily improve performance. I posted a similar example some time ago, though I later realized the code had bugs and could have been improved upon as an example.

\$\endgroup\$
1
\$\begingroup\$

Python is not Java

Not everything needs to be in a class

class MaskOps():

    @staticmethod
    def get_base(shape, value, dtype=int):
        """ This function produces a base-mask, the values of which may be overwritten. """
        if isinstance(value, (float, int)):
            res = np.ones(shape, dtype=dtype) * value
        elif isinstance(value, str):
            res = np.array([value for idx in range(np.prod(shape))]).reshape(shape)
        return res

    @staticmethod
    def alternate_base(shape, key):
        """ This function creates base-masks that consist of one of two value; the value depends on the index input as the parameter key. """
        if key % 2 == 0:
            value = 0.25
        else:
            value = 0.5
        return MaskOps().get_base(shape, value, dtype=float)

MO = MaskOps()

can just be

def get_base(..):
    ...
def alternate_base(...):
    ...

looping

Python has a lot of elegant looping constructs

for idx in range(len(bases)):
    bases[idx][locs[idx]] = 0

for example is not one of them. This can be done with zip

for base, loc in zip(bases, locs):
     base[loc] = 0

Check out this talk (slides) for tips on enumerate, zip, generators etc.

return values, don't print them

When you print the value, it makes it harder to reuse this part of the code in another place. Better would be to split the core_algorithm in different functions all doing their part of the calculation, so you can test each of these parts individually, and decide how to print the result

use numpy (#1)

You have a lot of numpy arrays that you make from list comprehensions. It would be a lot easier and clearer to vectorize this, and use numpy's large arsenal of native methods

def get_base(shape, value, dtype=int):
    """ This function produces a base-mask, the values of which may be overwritten. """
    if isinstance(value, (float, int)):
        res = np.ones(shape, dtype=dtype) * value
    elif isinstance(value, str):
        res = np.array([value for idx in range(np.prod(shape))]).reshape(shape)
    return res

can be more easily written as

def get_base(shape, value, dtype=int):
    dtypes = {int: int, float: float}
    if not dtype:
        dtype = dtypes.get(type(value), object)
    return np.ones(shape, dtype=dtype) * value

use numpy (#2)

Instead of making this boolean mask yourself, why not just do:

mask = data == [[i] for i in search_value]
collapsed_mask = mask.all(axis=0)
result = data[:, collapsed_mask]
\$\endgroup\$
  • \$\begingroup\$ With regard to your first point, my intention was to make different classes to organize functions into logically consistent groups because my main code is very long. Is there a better way to organize such a code without using lots of globally defined functions? With regard to your second point, I thought there might be a numpy way to do this instead of looping; your example was very helpful. Thanks for also providing a link for extra information. I was not aware boolean values could be used for indexing! \$\endgroup\$ – MPath Mar 13 '18 at 0:55
  • \$\begingroup\$ Also, my main code is actually slightly different. Where it says locs = np.array([np.where(ndata[idx] == search_value[idx])[0] for idx in range(len(search_value))]), specifically the part that reads ndata[idx] == search_value[idx], my main code actually implements a custom condition inputter, so that one can specify <, <=, >, >=, !=, nearest-neighbor, etc. Is the mask.all(axis=0) approach adaptable in this way? \$\endgroup\$ – MPath Mar 13 '18 at 1:07
  • \$\begingroup\$ I've posted the updated question in a separate post on stackoverflow. \$\endgroup\$ – MPath Mar 13 '18 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.