# Find matches between two tables after downsampling

I am trying to find matches between two tables by downsampling so that they fit into memory and we are able to find a good number of matches. For one of the tables, I have created an inverted index on one table and I am trying to probe the other table - so that down sampled tables have matches for data science purposes in the workflow. I am trying to improve the performance of the code so that it performs better. I have tried to improve but it's still taking 90s for 2 million transactions and I would like to get some suggestions because I am very new to Python. Main Code calls function with 4 parameters (table1, table2, down_sampled_table2_size, y)

s_inv_index = create_inv_index(small_table)
big_sample_size = min(math.floor(size/y), len(big_table))
big_tbl_indices = list(np.random.choice(len(big_table), big_sample_size, replace=False))

small_tbl_indices = probing(big_table.ix[big_tbl_indices], y,
len(small_table), s_inv_index)

def create_inv_index(table):
stop_words = set(_get_stop_words())
str_cols_ix = _get_str_cols_list(table)
n = len(table)
key_pos = dict(zip(range(n), range(n)))
inv_index = dict()
pos = 0
for row in table.itertuples():
s = ''
for ix in str_cols_ix:
s += str(row[ix+1]).lower() + ' '
s = s.rstrip()
# tokenize them
s = set(s.split())
s = s.difference(stop_words)
for token in s:
lst = inv_index.get(token, None)
if lst is None:
inv_index[token] = [pos]
else:
lst.append(pos)
inv_index[token] = lst
pos += 1
return inv_index

def probing(big_table, y, small_tbl_sz, s_inv_index):
y_pos = math.floor(y/2)
h_table = set()
str_cols_ix = _get_str_cols_list(big_table) //this will return column  numbers that are string - for ex. [2, 3, 4]

for row in big_table.itertuples():
s = ''
for ix in str_cols_ix:
s += str(row[ix+1]).lower() + ' '
s = s.rstrip()
s = set(s.split())
s = s.difference(stop_words)

m = set()
for token in s:
ids = s_inv_index.get(token, None)
if ids is not None:
m.update(ids)

# pick y/2 elements from m
k = min(y_pos, len(m))
m = list(m)
smpl_pos = np.random.choice(m, k, replace=False)
s_pos_set = set()
s_pos_set.update(smpl_pos)
s_tbl_ids = set(range(s_tbl_sz))
rem_locs = list(s_tbl_ids.difference(s_pos_set))
if y - k > 0:
s_neg_set = np.random.choice(rem_locs, y - k, replace=False)
h_table.update(s_pos_set, s_neg_set)
return h_table

• Welcome to Code Review! Please edit your question title to describe what your code does. Everybody posting on this site wants better code, and we don't want all questions having a title like "How can I improve this code?" - rule of thumb, if your title reads like a question, it's probably not telling readers what it's doing. See How to Ask for more details. – Mathieu Guindon May 16 '16 at 13:42
• This code does not seem correct to me — the function probing takes a parameter big_table but this is ignored and a global variable b_table is used instead. This looks like a mistake. – Gareth Rees May 18 '16 at 11:48
• Also, I don't see how to review this unless you tell us what create_inv_index does (or show its code), and give us some example data to test it on. – Gareth Rees May 18 '16 at 11:52
• @GarethRees I have updated the code with create index and have corrected too. Please review. – yguw May 18 '16 at 13:22