# Optimization of startswith, list intersection with duplicates and substring search

def check_rule(token, training_data, mode):
if mode == "frag":
children = training_data["words"]
remove = list.remove
def in_child(x):
for i in token:
if i not in x:
return False
remove(x, i)
return True

for i in children:
if in_child(i):
return False
return True

# is_present = children.map(in_child)
elif mode == "wild":

# is_present = training_data[1].map(methodcaller("__contains__", token))

for i in training_data[1]:
if token in i:
return False
return True

elif mode == "prefix":
# is_present = training_data[1].str.startswith(token)

token_len = len(token)
for i in training_data[1]:
if i[:token_len] == token:
return False
return True

# token_len = len(token)
# is_present = training_data[1].str[:token_len] == token

# is_present = training_data[1].map(methodcaller("startswith", token))

# if is_present.any():
#     return False
# return True


This function takes in:

token: string
training_data: A pandas dataframe with 2 columns having ID and vendor name columns
mode: Which can be any one of the prefix, wild, frag.

I have 5 data structures:

1. Vendor data: Having vendor names in one column (there are other columns as well but only this name is relevant).

2. Sub vendor data: A dict, for each <vendor name> there is a list of <sub-vendor>.

3. Rule data: For each mode there is a dict with <vendor name><sub vendor name> as key and value is a string rule.

So, for each vendor and sub vendor, whole data minus the rows having the respective vendor, the corresponding rule is checked if it is present in any of the rows. This rule checking is done by the code segment I have provided.

The commented parts are the optimizations I tried after profiling. But, it still takes a lot of time. Can this be optimized further?

• Your title is too generic to be useful for this site. It should state the purpose of the code, as per the How to Ask guidelines. – 200_success Dec 14 '16 at 14:51
• This question is incomplete. To help reviewers give you better answers, please add sufficient context to your question. The more you tell us about what your code does and what the purpose of doing that is, the easier it will be for reviewers to help you. Questions should include a description of what the code does – 301_Moved_Permanently Dec 14 '16 at 15:24
• The time it takes to run on different datasets can be seen here - stackoverflow.com/q/41139223/2650427 – TrigonaMinima Dec 15 '16 at 7:39