I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other part is then fed into the model to find the indexed text that is most similar. This involves a search function to enumerate over each item in the indexed part.

The goal is to compare each row in df_yes with the sims index (which consists of all rows from df_no. So the function is applied to each row in df_yes and, for each row, find the maximum similarity value and index(ices) of df_no.

With the toy data, it is fast, but with my real data, it is much too slow using apply. Here is the code example:

import pandas as pd
import gensim
import nltk
from nltk.tokenize import word_tokenize

d = {'number': [1,2,3,4,5], 'text': ['do you like python', 'do you hate python','do you like apples','who is nelson mandela','i am not interested'], 'answer':['no','yes','no','no','yes']}
df = pd.DataFrame(data=d)

df_yes = df[df['answer']=='yes']

df_no = df[df['answer']=='no']
df_no = df_no.reset_index()

docs = df_no['text'].tolist()
genDocs = [[w.lower() for w in word_tokenize(text)] for text in docs]
dictionary = gensim.corpora.Dictionary(genDocs)
corpus = [dictionary.doc2bow(genDoc) for genDoc in genDocs]
tfidf = gensim.models.TfidfModel(corpus)
sims = gensim.similarities.MatrixSimilarity(tfidf[corpus], num_features=len(dictionary))

def get_search_results(row):
    tokenized_row = word_tokenize(row)
    query_bag_of_words = dictionary.doc2bow(tokenized_row)
    query_tfidf = tfidf[query_bag_of_words]
    search_result = sims[query_tfidf]
    max_similarity = max(search_result)
    index = [i for i, j in enumerate(search_result) if j == max_similarity]
    return max_similarity, index

df_yes = df_yes.copy()

df_yes['max_similarity'], df_yes['index'] = zip(*df_yes['text'].apply(get_search_results))

I have tried converting the operations to dask dataframes to no avail, as well as python multiprocessing. How would I make these functions more efficient? Is it possible to vectorize some/all of the functions?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.