# Function for root matching between two paragraphs

I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really complicated it. Do you know if I can improve it ? It is important as far as I apply it to a big file then.

import numpy as np, pandas as pd
# Break the paragraph/context into multiple sentences.
import spacy

# synonyms
from itertools import chain
from nltk.corpus import wordnet

# stemmer/ root maker
from nltk.stem.lancaster import LancasterStemmer
st = LancasterStemmer()

# to evaluate literal expressions (sorry, I know it might not mean anything ...)
import ast

def match_roots(x):
# we flatten the question
question = x["question"].lower()

# We get the sentences of the question in seperated words
sentences = en_nlp(x["context"].lower()).sents

# taking anything that look like a root in the question, not just the dominating one
question_roots = [chunk.root.head.text.lower() for chunk in en_nlp(question).noun_chunks]
# Attempt of using synonyms
synonyms = []
for word in set(question_roots):
words = []
words = wordnet.synsets(word)
for synset in words:
name = synset.lemmas()[0].name()
synonyms.append(st.stem(name))
question_roots.extend(synonym for synonym in set(synonyms))
# end of attempt
li = []
# Here we prepare the ranking list
ranking_list = []
# for each sentence of a bunch of sentences
for i,sent in enumerate(sentences):
# we store the roots of a sentence
roots = [st.stem(chunk.root.head.text.lower()) for chunk in sent.noun_chunks]
common_roots = []
if sum(1 for root in roots if root in question_roots)>0:
# Here we score
common_roots = [root for root in roots if root in question_roots]
for k,item in enumerate(ast.literal_eval(x["sentences"])):
if str(sent) in item.lower():
li.append(k)
ranking_list.append((len(common_roots),i))
return [max(ranking_list,key=lambda item:item[0])[1]]


The input can be :

>>>predicted["question"][0]
What role did Beyoncé have in Destiny's Child?
>>>predicted["context"][0]
'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\'s Child. Managed by her father, Mathew Knowles, the group became one of the world\'s best-selling girl groups of all time. Their hiatus saw the release of Beyoncé\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".'


The output would be 1. The sentence at rank 1 (the second sentence would be elected as the one containing the answer).

• Please provide some context for your code. Are there some missing function definitions or import statements? – 200_success Aug 26 '18 at 22:18
• @200_success Yes! I think I addded all of them. Let me know if this is not the case – Revolucion for Monica Aug 26 '18 at 23:39