I am trying to find matches between two tables by downsampling so that they fit into memory and we are able to find a good number of matches. For one of the tables, I have created an inverted index on one table and I am trying to probe the other table - so that down sampled tables have matches for data science purposes in the workflow. I am trying to improve the performance of the code so that it performs better. I have tried to improve but it's still taking 90s for 2 million transactions and I would like to get some suggestions because I am very new to Python. Main Code calls function with 4 parameters (table1, table2, down_sampled_table2_size, y)

s_inv_index = create_inv_index(small_table)
big_sample_size = min(math.floor(size/y), len(big_table))
big_tbl_indices = list(np.random.choice(len(big_table), big_sample_size, replace=False))

small_tbl_indices = probing(big_table.ix[big_tbl_indices], y,
                         len(small_table), s_inv_index)

def create_inv_index(table):
    stop_words = set(_get_stop_words())
    str_cols_ix = _get_str_cols_list(table)                                                                                                                                                                
    n = len(table)
    key_pos = dict(zip(range(n), range(n)))
    inv_index = dict()
    pos = 0
    for row in table.itertuples():                                                                                                                                                                         
        s = ''                                                                                                                                                                                             
        for ix in str_cols_ix:                                                                                                                                                                             
            s += str(row[ix+1]).lower() + ' '                                                                                                                                                          
        s = s.rstrip()                                                                                                                                                                                     
        # tokenize them                                                                                                                                                                                  
        s = set(s.split())
        s = s.difference(stop_words)
        for token in s:
            lst = inv_index.get(token, None)
            if lst is None:
                inv_index[token] = [pos]
                inv_index[token] = lst
        pos += 1
    return inv_index

def probing(big_table, y, small_tbl_sz, s_inv_index):
    y_pos = math.floor(y/2)
    h_table = set()
    str_cols_ix = _get_str_cols_list(big_table) //this will return column  numbers that are string - for ex. [2, 3, 4]

    for row in big_table.itertuples():
        s = ''
        for ix in str_cols_ix:
            s += str(row[ix+1]).lower() + ' '
        s = s.rstrip()
        s = set(s.split())
        s = s.difference(stop_words)

        m = set()
        for token in s:
            ids = s_inv_index.get(token, None)
            if ids is not None:

        # pick y/2 elements from m                                                                                                                                                                          
        k = min(y_pos, len(m))
        m = list(m)
        smpl_pos = np.random.choice(m, k, replace=False)
        s_pos_set = set()
        s_tbl_ids = set(range(s_tbl_sz))
        rem_locs = list(s_tbl_ids.difference(s_pos_set))
        if y - k > 0:
            s_neg_set = np.random.choice(rem_locs, y - k, replace=False)
            h_table.update(s_pos_set, s_neg_set)
    return h_table
  • 2
    \$\begingroup\$ Welcome to Code Review! Please edit your question title to describe what your code does. Everybody posting on this site wants better code, and we don't want all questions having a title like "How can I improve this code?" - rule of thumb, if your title reads like a question, it's probably not telling readers what it's doing. See How to Ask for more details. \$\endgroup\$ – Mathieu Guindon May 16 '16 at 13:42
  • \$\begingroup\$ This code does not seem correct to me — the function probing takes a parameter big_table but this is ignored and a global variable b_table is used instead. This looks like a mistake. \$\endgroup\$ – Gareth Rees May 18 '16 at 11:48
  • \$\begingroup\$ Also, I don't see how to review this unless you tell us what create_inv_index does (or show its code), and give us some example data to test it on. \$\endgroup\$ – Gareth Rees May 18 '16 at 11:52
  • \$\begingroup\$ @GarethRees I have updated the code with create index and have corrected too. Please review. \$\endgroup\$ – yguw May 18 '16 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.