I am trying to write a script of Python code, for entity extraction and resolution.
The excerpts of the algorithm:
- It is trying to extract the entity as PoS Tag with Hidden Markov Model(HMM).
- After training and testing, application data is given to tagger.
- It is extracting the user defined entities.
- A different Naive Bayes Classifier (NBC) is trained to identify the sentences with relations like, 'APPO'->for apposition, 'Alias'->for alias, 'DT'-> for Determiner, etc.
- The data of HMM and NBC are different.
- The application data of HMM is the application data of NBC, post tagging or identifying of entities.
- I have used mainly Relationship extraction procedure suggested by Daniel Zurafsky to resolve entities.
Based on this idea, I wrote the following script:
import nltk
from nltk.corpus.reader import TaggedCorpusReader
from sklearn import cross_validation
from nltk.tokenize import word_tokenize
from itertools import chain
def ERR():
#HMM PORTION
reader = TaggedCorpusReader('.', r'.*\.pos')
files=reader.fileids()
tagcorpus=reader.tagged_sents('TotalData.pos')
testcorpus=reader.tagged_sents('TestHMM.pos')
app_corp=reader.sents('HMMApplication.pos')
len_tagcorp=len(tagcorpus)
print "The Length of the Annotated Training Corpus",len_tagcorp
#train_corp=tagcorpus[:1380]
train_corp=tagcorpus
test_corp=testcorpus
hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_corp)
hmm_score=hmm_tagger.evaluate(test_corp)
print "HMM Score Is:",hmm_score
#NAIVE BAYES TRAINING
nbf1=open("/python27/classfdata/Classif1data/AppoNew3.txt","r").read().lower() #New Set1
nbf2=open("/python27/classfdata/Classif1data/AliasNew3.txt","r").read().lower() #New Set1
nbf3=open("/python27/classfdata/Classif1data/DTNew3.txt","r").read().lower() #New Set1
nbf4=open("/python27/classfdata/Classif1data/PRNNew3.txt","r").read().lower() #New Set1
nbf5=open("/python27/classfdata/Classif1data/SMNew3.txt","r").read().lower() #New Set1
nbf6=open("/python27/classfdata/Classif1data/AmbNew1.txt","r").read().lower() #AMBIGUITY
nbf7=open("/python27/classfdata/Classif1data/SMAMB1.txt","r").read().lower() #AMBIGUITY
list2=[(nbf1,'APPO'),(nbf2,'Alias'),(nbf3,'DT'),(nbf4,'PRNAMB'),(nbf5,'SM'),(nbf6,'AMB'),(nbf7,'SMAMB')]
len_list2=len(list2)
train_sents=list2[:5]
test_sents=list2[3:]
all_words = set(word.lower() for passage in list2 for word in word_tokenize(passage[0]))
t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]
t1=[({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]
classifier = nltk.NaiveBayesClassifier.train(t)
print "NLTK NAIVE BAYES ACCURACY:",(nltk.classify.accuracy(classifier, t1))
for i in app_corp:
classif=hmm_tagger.tag(i)
#FLATTENING THE LIST OF TUPLES
list3=list(chain(*classif))
try:
a3=[w.replace('NA', '/NA').replace('COMP','/COMP').replace('COMPSu','/COMPSu').replace('EXET','/EXET').replace('AMT','/AMT').replace('ArtDef','/ArtDef').replace('AdjDef','/AdjDef').replace('AdjLoc','/AdjLoc').replace('PERSB','/PERSB').replace('DPERS','/DPERS').replace('DT','/DT').replace('PERSPL','/PERSPL').replace('EXET','/EXET').replace('LOC','/LOC').replace('LOCC','/LOCC').replace('TRM','/TRM').replace('ORGRel','/ORGRel').replace('ORGP','/ORGP').replace('PERSRel','/PERSRel').replace('PERSREM','/PERSREM').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('LOCDi','/LOCDi').replace('LOCPOS','/LOCPOS').replace('ORGREL','/ORGREL').replace('ARTDEF','/ARTDEF').replace('STR','/STR').replace('GM','/GM').replace('PERS','/PERS') for w in list3]
a4=" ".join(a3).replace(' /','/').replace('//','/')
#CLASSIFYING
x1=a4
print "The Sentence Is:",x1
test_sent_features1 = {word.lower(): (word in word_tokenize(x1.lower())) for word in all_words}
result=classifier.classify(test_sent_features1)
print "CLASSIFIER RESULT:",result
except AttributeError:
print "Error"
Am I being able to tackle the problem? Could I combine HMM and NBC fine?
NB: Please see link for sample HMM training data. NBC training data is very close in nature though labels are quite different.