I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really complicated it. Do you know if I can improve it ? It is important as far as I apply it to a big file then.
import numpy as np, pandas as pd # Break the paragraph/context into multiple sentences. import spacy en_nlp = spacy.load('en') # synonyms from itertools import chain from nltk.corpus import wordnet # stemmer/ root maker from nltk.stem.lancaster import LancasterStemmer st = LancasterStemmer() # to evaluate literal expressions (sorry, I know it might not mean anything ...) import ast def match_roots(x): # we flatten the question question = x["question"].lower() # We get the sentences of the question in seperated words sentences = en_nlp(x["context"].lower()).sents # taking anything that look like a root in the question, not just the dominating one question_roots = [chunk.root.head.text.lower() for chunk in en_nlp(question).noun_chunks] # Attempt of using synonyms synonyms =  for word in set(question_roots): words =  words = wordnet.synsets(word) for synset in words: name = synset.lemmas().name() synonyms.append(st.stem(name)) question_roots.extend(synonym for synonym in set(synonyms)) # end of attempt li =  # Here we prepare the ranking list ranking_list =  # for each sentence of a bunch of sentences for i,sent in enumerate(sentences): # we store the roots of a sentence roots = [st.stem(chunk.root.head.text.lower()) for chunk in sent.noun_chunks] common_roots =  if sum(1 for root in roots if root in question_roots)>0: # Here we score common_roots = [root for root in roots if root in question_roots] for k,item in enumerate(ast.literal_eval(x["sentences"])): if str(sent) in item.lower(): li.append(k) ranking_list.append((len(common_roots),i)) return [max(ranking_list,key=lambda item:item)]
The input can be :
>>>predicted["question"] What role did Beyoncé have in Destiny's Child? >>>predicted["context"] 'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\'s Child. Managed by her father, Mathew Knowles, the group became one of the world\'s best-selling girl groups of all time. Their hiatus saw the release of Beyoncé\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".'
The output would be 1. The sentence at rank 1 (the second sentence would be elected as the one containing the answer).