1
\$\begingroup\$

I created an algorithm that match roots of two texts, a question and a paragraph made of sentences. I aim at predicting in which sentence it exists the answer of a question. Yet It seems that I really complicated it. Do you know if I can improve it ? It is important as far as I apply it to a big file then.

import numpy as np, pandas as pd
# Break the paragraph/context into multiple sentences. 
import spacy
en_nlp = spacy.load('en')

# synonyms
from itertools import chain
from nltk.corpus import wordnet

# stemmer/ root maker 
from nltk.stem.lancaster import LancasterStemmer
st = LancasterStemmer()

# to evaluate literal expressions (sorry, I know it might not mean anything ...)
import ast 

def match_roots(x):
    # we flatten the question
    question = x["question"].lower()

    # We get the sentences of the question in seperated words
    sentences = en_nlp(x["context"].lower()).sents

    # taking anything that look like a root in the question, not just the dominating one
    question_roots = [chunk.root.head.text.lower() for chunk in en_nlp(question).noun_chunks]
    # Attempt of using synonyms
    synonyms = []
    for word in set(question_roots):
        words = []
        words = wordnet.synsets(word)
        for synset in words:
            name = synset.lemmas()[0].name()
            synonyms.append(st.stem(name))
    question_roots.extend(synonym for synonym in set(synonyms))
    # end of attempt
    li = []
    # Here we prepare the ranking list
    ranking_list = []
    # for each sentence of a bunch of sentences
    for i,sent in enumerate(sentences):
        # we store the roots of a sentence
        roots = [st.stem(chunk.root.head.text.lower()) for chunk in sent.noun_chunks]
        common_roots = []
        if sum(1 for root in roots if root in question_roots)>0:
            # Here we score
            common_roots = [root for root in roots if root in question_roots]
            for k,item in enumerate(ast.literal_eval(x["sentences"])):
                if str(sent) in item.lower(): 
                    li.append(k)
        ranking_list.append((len(common_roots),i))
    return [max(ranking_list,key=lambda item:item[0])[1]]

The input can be :

>>>predicted["question"][0]
What role did Beyoncé have in Destiny's Child?
>>>predicted["context"][0]
'Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny\'s Child. Managed by her father, Mathew Knowles, the group became one of the world\'s best-selling girl groups of all time. Their hiatus saw the release of Beyoncé\'s debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".'

The output would be 1. The sentence at rank 1 (the second sentence would be elected as the one containing the answer).

\$\endgroup\$
  • \$\begingroup\$ Please provide some context for your code. Are there some missing function definitions or import statements? \$\endgroup\$ – 200_success Aug 26 '18 at 22:18
  • \$\begingroup\$ @200_success Yes! I think I addded all of them. Let me know if this is not the case \$\endgroup\$ – ThePassenger Aug 26 '18 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.