Entity resolution with NLTK

Question

I am trying to write a script of Python code, for entity extraction and resolution.

The excerpts of the algorithm:

1. It is trying to extract the entity as PoS Tag with Hidden Markov Model(HMM).
2. After training and testing, application data is given to tagger.
3. It is extracting the user defined entities.
4. A different Naive Bayes Classifier (NBC) is trained to identify the sentences with relations like, 'APPO'->for apposition, 'Alias'->for alias, 'DT'-> for Determiner, etc.
5. The data of HMM and NBC are different.
6. The application data of HMM is the application data of NBC, post tagging or identifying of entities.
7. I have used mainly Relationship extraction procedure suggested by Daniel Zurafsky to resolve entities.

Based on this idea, I wrote the following script:

import nltk
from nltk.corpus.reader import TaggedCorpusReader
from sklearn import cross_validation
from nltk.tokenize import word_tokenize
from itertools import chain
def ERR():
    #HMM PORTION
    reader = TaggedCorpusReader('.', r'.*\.pos')
    files=reader.fileids()
    tagcorpus=reader.tagged_sents('TotalData.pos')
    testcorpus=reader.tagged_sents('TestHMM.pos')
    app_corp=reader.sents('HMMApplication.pos')
    len_tagcorp=len(tagcorpus)
    print "The Length of the Annotated Training Corpus",len_tagcorp
    #train_corp=tagcorpus[:1380]
    train_corp=tagcorpus
    test_corp=testcorpus
    hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_corp)
    hmm_score=hmm_tagger.evaluate(test_corp)
    print "HMM Score Is:",hmm_score
    #NAIVE BAYES TRAINING
    nbf1=open("/python27/classfdata/Classif1data/AppoNew3.txt","r").read().lower() #New Set1
    nbf2=open("/python27/classfdata/Classif1data/AliasNew3.txt","r").read().lower() #New Set1
    nbf3=open("/python27/classfdata/Classif1data/DTNew3.txt","r").read().lower() #New Set1
    nbf4=open("/python27/classfdata/Classif1data/PRNNew3.txt","r").read().lower() #New Set1
    nbf5=open("/python27/classfdata/Classif1data/SMNew3.txt","r").read().lower() #New Set1
    nbf6=open("/python27/classfdata/Classif1data/AmbNew1.txt","r").read().lower() #AMBIGUITY
    nbf7=open("/python27/classfdata/Classif1data/SMAMB1.txt","r").read().lower() #AMBIGUITY 
    list2=[(nbf1,'APPO'),(nbf2,'Alias'),(nbf3,'DT'),(nbf4,'PRNAMB'),(nbf5,'SM'),(nbf6,'AMB'),(nbf7,'SMAMB')]
    len_list2=len(list2)
    train_sents=list2[:5]
    test_sents=list2[3:]
    all_words = set(word.lower() for passage in list2 for word in word_tokenize(passage[0]))
    t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]
    t1=[({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]
    classifier = nltk.NaiveBayesClassifier.train(t)
    print "NLTK NAIVE BAYES ACCURACY:",(nltk.classify.accuracy(classifier, t1))
    for i in app_corp:
        classif=hmm_tagger.tag(i)
        #FLATTENING THE LIST OF TUPLES
        list3=list(chain(*classif))
        try:
            a3=[w.replace('NA', '/NA').replace('COMP','/COMP').replace('COMPSu','/COMPSu').replace('EXET','/EXET').replace('AMT','/AMT').replace('ArtDef','/ArtDef').replace('AdjDef','/AdjDef').replace('AdjLoc','/AdjLoc').replace('PERSB','/PERSB').replace('DPERS','/DPERS').replace('DT','/DT').replace('PERSPL','/PERSPL').replace('EXET','/EXET').replace('LOC','/LOC').replace('LOCC','/LOCC').replace('TRM','/TRM').replace('ORGRel','/ORGRel').replace('ORGP','/ORGP').replace('PERSRel','/PERSRel').replace('PERSREM','/PERSREM').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('LOCDi','/LOCDi').replace('LOCPOS','/LOCPOS').replace('ORGREL','/ORGREL').replace('ARTDEF','/ARTDEF').replace('STR','/STR').replace('GM','/GM').replace('PERS','/PERS') for w in list3]
            a4=" ".join(a3).replace(' /','/').replace('//','/')
            #CLASSIFYING
            x1=a4
            print "The Sentence Is:",x1
            test_sent_features1 = {word.lower(): (word in word_tokenize(x1.lower())) for word in all_words}
            result=classifier.classify(test_sent_features1)
            print "CLASSIFIER RESULT:",result
        except AttributeError:
            print "Error"

Am I being able to tackle the problem? Could I combine HMM and NBC fine?

NB: Please see link for sample HMM training data. NBC training data is very close in nature though labels are quite different.

Answer 1

Style

There is an official python style guide, PEP8. One of the things it recommends is using lower_case for variable and function names. Another recommendation is to leave two empty lines before functions. Also, you should surround operators (such as + and =) with one space on each side to improve readability. Same goes for a space after a comma, in an argument list.

Reduce duplication

The variables tagcorpus and testcorpus are never really used. Just use train_corp and test_corp right away.

The length of the tagcorpus is used only once.

Use functions

You should split up your one function into smaller parts, at least into an HMM and an NBC part.

The file reading can be put in its own function as well.

That replace monstrosity needs to go as well.

Use better names

Names such as list2, list3, t, t1, a3 and x1 do not help in understanding the code. Choose better names (In can't even recommend a better name, because I have no clue what e.g. those lists contain).

Use better scope

As of now, your try..except block contains a lot of operations, any of which might fail with an AttributeError. it is better to protect only the one operation which might fail this way (I don't see which one that is here, though). Also capture the error message itself by using except AttributeError as e: print "Error:", e

Use with..as

To ensure that files are closed again:

def read_file(file_name):
    with open(DIR + file_name) as f:
        return f.read().lower()

Use __name__ hook

In order to ease importing you code from other scripts, you might want to wrap calling your function like this:

if __name__ == "__main__":
    entity_resolution()

General remarks

Your train and test sentences seem to overlap, since l[:5] will give all elements up to and including the fourth, l[3:] will give all elements starting with the third. Therefore the sentences in l[3] and l[4] are used twice.

For the train_words and test_words you use lists instead of sets. These could also be sets.

Result

Here is the result of my preliminary remarks (I'll be back after lunch for more). I did not yet factor out the NBC part into its own function and did not look too hard at the algorithm building the word sets:

from nltk import HiddenMarkovModelTagger, NaiveBayesClassifier, classify
from nltk.corpus.reader import TaggedCorpusReader
from nltk.tokenize import word_tokenize
from sklearn import cross_validation
from itertools import chain

DIR = "/python27/classfdata/Classif1data/"


def hmm(train_corp, test_corp):
    # HMM PORTION
    hmm_tagger = HiddenMarkovModelTagger.train(train_corp)
    hmm_score = hmm_tagger.evaluate(test_corp)
    return hmm_tagger, hmm_score


def read_file(file_name):
    with open(DIR + file_name) as f:
        return f.read().lower()


def add_slashes(word):
    replace_words = ('NA', 'COMP', 'COMPSu', 'EXET', 'AMT', 'ArtDef', 'AdjDef',
                     'AdjLoc', 'PERSB', 'DPERS', 'DT', 'PERSPL', 'EXET', 'LOC',
                     'LOCC', 'TRM', 'ORGRel', 'ORGP', 'PERSRel', 'PERSREM',
                     'PRNM3PAS', 'PRNneu3PAS', 'PRNM3PAS', 'PRNneu3PAS',
                     'LOCDi', 'LOCPOS', 'ORGREL', 'ARTDEF', 'STR', 'GM', 'PERS')
    for repl_word in replace_words:
        word = word.replace(repl_word, "/" + repl_word)
    return word


def entity_resolution():
    reader = TaggedCorpusReader('.', r'.*\.pos')
    files = reader.fileids()
    train_corp = reader.tagged_sents('TotalData.pos')
    test_corp = reader.tagged_sents('TestHMM.pos')
    app_corp = reader.sents('HMMApplication.pos')
    print "The Length of the Annotated Training Corpus is", len(train_corp)

    hmm_tagger, hmm_score = hmm(train_corp, test_corp)
    print "HMM score is:", hmm_score

    # NAIVE BAYES TRAINING
    file_names = ["AppoNew3.txt", "AliasNew3.txt", "DTNew3.txt", "PRNNew3.txt",
                  "SMNew3.txt", "AmbNew1.txt", "SMAMB1.txt"]
    names = ["APPO", "Alias", "DT", "PRNAMB", "SM", "AMB", "SMAMB"]
    sentences = zip([read_file(file_name) for file_name in file_names], names)
    train_sents = sentences[:5]
    test_sents = sentences[3:]
    all_words = set(word.lower() for sentence in sentences for word in word_tokenize(sentence[0]))
    train_words = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]
    test_words = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]
    classifier = NaiveBayesClassifier.train(train_words)

    print "NLTK NAIVE BAYES ACCURACY:", (classify.accuracy(classifier, test_words))

    for corp in app_corp:
        sentence = list(chain(*hmm_tagger.tag(corp)))
        try:
            sentence = " ".join([add_slashes(word) for word in sentence])
            sentence = sentence.replace(' /', '/').replace('//', '/').lower()

            # CLASSIFYING
            print "The Sentence Is:", sentence
            test_sent_features1 = {word: (word in word_tokenize(sentence)) for word in all_words}
            result = classifier.classify(test_sent_features1)
            print "CLASSIFIER RESULT:", result
        except AttributeError as e:
            print "Error", e

if __name__ == "__main__":
    entity_resolution()