2
\$\begingroup\$
def check_rule(token, training_data, mode):
    if mode == "frag":
        children = training_data["words"]
        remove = list.remove
        def in_child(x):
            for i in token:
                if i not in x:
                    return False
                remove(x, i)
            return True

        for i in children:
            if in_child(i):
                return False
        return True

        # is_present = children.map(in_child)
    elif mode == "wild":

        # is_present = training_data[1].map(methodcaller("__contains__", token))

        for i in training_data[1]:
            if token in i:
                return False
        return True

    elif mode == "prefix":
        # is_present = training_data[1].str.startswith(token)

        token_len = len(token)
        for i in training_data[1]:
            if i[:token_len] == token:
                return False
        return True

        # token_len = len(token)
        # is_present = training_data[1].str[:token_len] == token

        # is_present = training_data[1].map(methodcaller("startswith", token))

    # if is_present.any():
    #     return False
    # return True

This function takes in:

token: string
training_data: A pandas dataframe with 2 columns having ID and vendor name columns
mode: Which can be any one of the prefix, wild, frag.

I have 5 data structures:

  1. Vendor data: Having vendor names in one column (there are other columns as well but only this name is relevant).

  2. Sub vendor data: A dict, for each <vendor name> there is a list of <sub-vendor>.

  3. Rule data: For each mode there is a dict with <vendor name><sub vendor name> as key and value is a string rule.

So, for each vendor and sub vendor, whole data minus the rows having the respective vendor, the corresponding rule is checked if it is present in any of the rows. This rule checking is done by the code segment I have provided.

The commented parts are the optimizations I tried after profiling. But, it still takes a lot of time. Can this be optimized further?

\$\endgroup\$
  • 3
    \$\begingroup\$ Your title is too generic to be useful for this site. It should state the purpose of the code, as per the How to Ask guidelines. \$\endgroup\$ – 200_success Dec 14 '16 at 14:51
  • 1
    \$\begingroup\$ This question is incomplete. To help reviewers give you better answers, please add sufficient context to your question. The more you tell us about what your code does and what the purpose of doing that is, the easier it will be for reviewers to help you. Questions should include a description of what the code does \$\endgroup\$ – Mathias Ettinger Dec 14 '16 at 15:24
  • \$\begingroup\$ The time it takes to run on different datasets can be seen here - stackoverflow.com/q/41139223/2650427 \$\endgroup\$ – TrigonaMinima Dec 15 '16 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.