1
\$\begingroup\$

I am trying to train a question-answer system, where I am trying to group similar questions, and identify the most apt response. The program should identify the intent/focus.

To do it, I have tagged the questions as NER so that I may know which intent it has, like PERSON or LOCATION etc. I am trying to group similar questions in a classifier, where NE tagged questions are X, and answers are Y.

I tried to write the following code. I am trying to paste the .pos file data in Pastebin. The code is running on toy-sized data thus I could not keep proper training/testing data partitions. But it seems giving me satisfactory results.

Please suggest how I can write the code beautifully.

import nltk
import collections
import nltk.metrics
from nltk.corpus.reader import TaggedCorpusReader
from nltk.tokenize import word_tokenize
from itertools import chain
from nltk.classify import NaiveBayesClassifier as nbc
from itertools import chain

#HMM 
reader = TaggedCorpusReader('.', r'.*\.pos')
f1=reader.fileids()
print f1
sents=reader.tagged_sents(fileids='Monishnew1.pos')
ls=len(sents)
print ls
train_sents=sents[:7]
test_sents=sents[7:]
class_sents=reader.sents(fileids='Monishnew1.pos')
appli_sent1=class_sents[5]
print "Sentence Is:",appli_sent1 
hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_sents)
test=hmm_tagger.test(test_sents)
print test
classf1=hmm_tagger.tag(appli_sent1)
print classf1
joiner1 = "/".join
classf2=map(joiner1, classf1)
print classf2
classf3=" ".join(classf2)
print classf3

#NBC
#CLASSIFYING THE ANSWER
training_data=[('Hello/GREET',
' How can I assist you'),
               ('Hi/GREET','How can I assist you'),
               ('want/NA to/NA speak/NA to/NA customer/GREET care/GREETC','How can I assist you'),
               ('issue/NA with/NA the/NA report/REPORT', 
' Please tell me your exact problem'),
('want/NA to/NA see/NA my/NA test/REPORTP report/REPORT','Sure Please share the lab number'),
('know/NA my/NA report/REPORT test/REPORTC done/NA today/NAgot/NA the/NA message/NA for/NA the/NA report/REPORT generation/REPORTC but/NA unable/NA to/NA see/NA','Apologies for the inconvenience caused Please share the lab number'),
('the/NA report/REPORT 224622415/REPVW  231223456/REPVW how/REPVW to/NA check/REPVW it/NA','please share you lab number to check this'),
('thanks/CLOSEok/CLOSEthank/CLOSE you/CLOSEC When/CLOSE will/NA I/NA get/NA the/NA call/CLOSE','is there anything else I can help you with'),
('I/NA will/NA wait/NA for/NA the/NA call/CLOSE','is there anything else I can help you with'),
('when/CLOSE call/CLOSE me/NA back/NA','you will get a call in the next 12 hours Is there anything else I can help you with')]

print training_data
vocabulary = set(chain(*[word_tokenize(i[0].lower()) for i in training_data]))
feature_set = [({i:(i in word_tokenize(sentence.lower())) for i in vocabulary},tag) for sentence, tag in training_data]
train_set, test_set = feature_set[:5], feature_set[5:]
classifier = nbc.train(train_set)
refsets = collections.defaultdict(set)
testsets = collections.defaultdict(set)
for i, (feats, label) in enumerate(test_set):
    refsets[label].add(i)
    observed = classifier.classify(feats)
    testsets[observed].add(i)

test_sentence=classf3
featurized_test_sentence = {i:(i in word_tokenize(test_sentence.lower())) for i in vocabulary}
cat1=classifier.classify(featurized_test_sentence)
print "The Answer Is:",cat1
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.