1
\$\begingroup\$

I am trying to write a script of Python code, for entity extraction and resolution.

The excerpts of the algorithm:

  1. It is trying to extract the entity as PoS Tag with Hidden Markov Model(HMM).
  2. After training and testing, application data is given to tagger.
  3. It is extracting the user defined entities.
  4. A different Naive Bayes Classifier (NBC) is trained to identify the sentences with relations like, 'APPO'->for apposition, 'Alias'->for alias, 'DT'-> for Determiner, etc.
  5. The data of HMM and NBC are different.
  6. The application data of HMM is the application data of NBC, post tagging or identifying of entities.
  7. I have used mainly Relationship extraction procedure suggested by Daniel Zurafsky to resolve entities.

Based on this idea, I wrote the following script:

import nltk
from nltk.corpus.reader import TaggedCorpusReader
from sklearn import cross_validation
from nltk.tokenize import word_tokenize
from itertools import chain
def ERR():
    #HMM PORTION
    reader = TaggedCorpusReader('.', r'.*\.pos')
    files=reader.fileids()
    tagcorpus=reader.tagged_sents('TotalData.pos')
    testcorpus=reader.tagged_sents('TestHMM.pos')
    app_corp=reader.sents('HMMApplication.pos')
    len_tagcorp=len(tagcorpus)
    print "The Length of the Annotated Training Corpus",len_tagcorp
    #train_corp=tagcorpus[:1380]
    train_corp=tagcorpus
    test_corp=testcorpus
    hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_corp)
    hmm_score=hmm_tagger.evaluate(test_corp)
    print "HMM Score Is:",hmm_score
    #NAIVE BAYES TRAINING
    nbf1=open("/python27/classfdata/Classif1data/AppoNew3.txt","r").read().lower() #New Set1
    nbf2=open("/python27/classfdata/Classif1data/AliasNew3.txt","r").read().lower() #New Set1
    nbf3=open("/python27/classfdata/Classif1data/DTNew3.txt","r").read().lower() #New Set1
    nbf4=open("/python27/classfdata/Classif1data/PRNNew3.txt","r").read().lower() #New Set1
    nbf5=open("/python27/classfdata/Classif1data/SMNew3.txt","r").read().lower() #New Set1
    nbf6=open("/python27/classfdata/Classif1data/AmbNew1.txt","r").read().lower() #AMBIGUITY
    nbf7=open("/python27/classfdata/Classif1data/SMAMB1.txt","r").read().lower() #AMBIGUITY 
    list2=[(nbf1,'APPO'),(nbf2,'Alias'),(nbf3,'DT'),(nbf4,'PRNAMB'),(nbf5,'SM'),(nbf6,'AMB'),(nbf7,'SMAMB')]
    len_list2=len(list2)
    train_sents=list2[:5]
    test_sents=list2[3:]
    all_words = set(word.lower() for passage in list2 for word in word_tokenize(passage[0]))
    t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]
    t1=[({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]
    classifier = nltk.NaiveBayesClassifier.train(t)
    print "NLTK NAIVE BAYES ACCURACY:",(nltk.classify.accuracy(classifier, t1))
    for i in app_corp:
        classif=hmm_tagger.tag(i)
        #FLATTENING THE LIST OF TUPLES
        list3=list(chain(*classif))
        try:
            a3=[w.replace('NA', '/NA').replace('COMP','/COMP').replace('COMPSu','/COMPSu').replace('EXET','/EXET').replace('AMT','/AMT').replace('ArtDef','/ArtDef').replace('AdjDef','/AdjDef').replace('AdjLoc','/AdjLoc').replace('PERSB','/PERSB').replace('DPERS','/DPERS').replace('DT','/DT').replace('PERSPL','/PERSPL').replace('EXET','/EXET').replace('LOC','/LOC').replace('LOCC','/LOCC').replace('TRM','/TRM').replace('ORGRel','/ORGRel').replace('ORGP','/ORGP').replace('PERSRel','/PERSRel').replace('PERSREM','/PERSREM').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('LOCDi','/LOCDi').replace('LOCPOS','/LOCPOS').replace('ORGREL','/ORGREL').replace('ARTDEF','/ARTDEF').replace('STR','/STR').replace('GM','/GM').replace('PERS','/PERS') for w in list3]
            a4=" ".join(a3).replace(' /','/').replace('//','/')
            #CLASSIFYING
            x1=a4
            print "The Sentence Is:",x1
            test_sent_features1 = {word.lower(): (word in word_tokenize(x1.lower())) for word in all_words}
            result=classifier.classify(test_sent_features1)
            print "CLASSIFIER RESULT:",result
        except AttributeError:
            print "Error"

Am I being able to tackle the problem? Could I combine HMM and NBC fine?

NB: Please see link for sample HMM training data. NBC training data is very close in nature though labels are quite different.

\$\endgroup\$
4
  • \$\begingroup\$ what's the size of your training and testing data ? Could you create a link to Daniel Zurafsky article? could you post a sample of the corpus you used for training the HMM or a link to where you have all this data e.g github or pastebin? \$\endgroup\$
    – Tolani
    Jul 27, 2016 at 7:56
  • \$\begingroup\$ Daniel Zurafsky's Lecture: Please refer to Coursera Course in NLP, Week 4, talking about Relationship Extraction, please refer that. Other than this I found a nice paper A Machine Learning Approach to Coreference Resolution of Noun Phrases, by Soon, Lim, Ng. They used Decision Tree, I am using NBC. Training corpus is 1380, Test is 410. I am changing the data size to see the optimum size for accuracy. I am trying to edit my question or add another comment with link for the data. Thank you for your kind time. \$\endgroup\$
    – HIGGINS
    Jul 27, 2016 at 16:53
  • \$\begingroup\$ For the benefit of others, those links should be provided in the post rather than in the comment. Note : you can actually have a link in your question. \$\endgroup\$
    – Tolani
    Jul 27, 2016 at 17:08
  • \$\begingroup\$ Did as you kindly suggested. \$\endgroup\$
    – HIGGINS
    Jul 27, 2016 at 17:15

1 Answer 1

1
\$\begingroup\$

Style

There is an official python style guide, PEP8. One of the things it recommends is using lower_case for variable and function names. Another recommendation is to leave two empty lines before functions. Also, you should surround operators (such as + and =) with one space on each side to improve readability. Same goes for a space after a comma, in an argument list.

Reduce duplication

The variables tagcorpus and testcorpus are never really used. Just use train_corp and test_corp right away.

The length of the tagcorpus is used only once.

Use functions

You should split up your one function into smaller parts, at least into an HMM and an NBC part.

The file reading can be put in its own function as well.

That replace monstrosity needs to go as well.

Use better names

Names such as list2, list3, t, t1, a3 and x1 do not help in understanding the code. Choose better names (In can't even recommend a better name, because I have no clue what e.g. those lists contain).

Use better scope

As of now, your try..except block contains a lot of operations, any of which might fail with an AttributeError. it is better to protect only the one operation which might fail this way (I don't see which one that is here, though). Also capture the error message itself by using except AttributeError as e: print "Error:", e

Use with..as

To ensure that files are closed again:

def read_file(file_name):
    with open(DIR + file_name) as f:
        return f.read().lower()

Use __name__ hook

In order to ease importing you code from other scripts, you might want to wrap calling your function like this:

if __name__ == "__main__":
    entity_resolution()

General remarks

Your train and test sentences seem to overlap, since l[:5] will give all elements up to and including the fourth, l[3:] will give all elements starting with the third. Therefore the sentences in l[3] and l[4] are used twice.

For the train_words and test_words you use lists instead of sets. These could also be sets.

Result

Here is the result of my preliminary remarks (I'll be back after lunch for more). I did not yet factor out the NBC part into its own function and did not look too hard at the algorithm building the word sets:

from nltk import HiddenMarkovModelTagger, NaiveBayesClassifier, classify
from nltk.corpus.reader import TaggedCorpusReader
from nltk.tokenize import word_tokenize
from sklearn import cross_validation
from itertools import chain

DIR = "/python27/classfdata/Classif1data/"


def hmm(train_corp, test_corp):
    # HMM PORTION
    hmm_tagger = HiddenMarkovModelTagger.train(train_corp)
    hmm_score = hmm_tagger.evaluate(test_corp)
    return hmm_tagger, hmm_score


def read_file(file_name):
    with open(DIR + file_name) as f:
        return f.read().lower()


def add_slashes(word):
    replace_words = ('NA', 'COMP', 'COMPSu', 'EXET', 'AMT', 'ArtDef', 'AdjDef',
                     'AdjLoc', 'PERSB', 'DPERS', 'DT', 'PERSPL', 'EXET', 'LOC',
                     'LOCC', 'TRM', 'ORGRel', 'ORGP', 'PERSRel', 'PERSREM',
                     'PRNM3PAS', 'PRNneu3PAS', 'PRNM3PAS', 'PRNneu3PAS',
                     'LOCDi', 'LOCPOS', 'ORGREL', 'ARTDEF', 'STR', 'GM', 'PERS')
    for repl_word in replace_words:
        word = word.replace(repl_word, "/" + repl_word)
    return word


def entity_resolution():
    reader = TaggedCorpusReader('.', r'.*\.pos')
    files = reader.fileids()
    train_corp = reader.tagged_sents('TotalData.pos')
    test_corp = reader.tagged_sents('TestHMM.pos')
    app_corp = reader.sents('HMMApplication.pos')
    print "The Length of the Annotated Training Corpus is", len(train_corp)

    hmm_tagger, hmm_score = hmm(train_corp, test_corp)
    print "HMM score is:", hmm_score

    # NAIVE BAYES TRAINING
    file_names = ["AppoNew3.txt", "AliasNew3.txt", "DTNew3.txt", "PRNNew3.txt",
                  "SMNew3.txt", "AmbNew1.txt", "SMAMB1.txt"]
    names = ["APPO", "Alias", "DT", "PRNAMB", "SM", "AMB", "SMAMB"]
    sentences = zip([read_file(file_name) for file_name in file_names], names)
    train_sents = sentences[:5]
    test_sents = sentences[3:]
    all_words = set(word.lower() for sentence in sentences for word in word_tokenize(sentence[0]))
    train_words = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]
    test_words = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]
    classifier = NaiveBayesClassifier.train(train_words)

    print "NLTK NAIVE BAYES ACCURACY:", (classify.accuracy(classifier, test_words))

    for corp in app_corp:
        sentence = list(chain(*hmm_tagger.tag(corp)))
        try:
            sentence = " ".join([add_slashes(word) for word in sentence])
            sentence = sentence.replace(' /', '/').replace('//', '/').lower()

            # CLASSIFYING
            print "The Sentence Is:", sentence
            test_sent_features1 = {word: (word in word_tokenize(sentence)) for word in all_words}
            result = classifier.classify(test_sent_features1)
            print "CLASSIFIER RESULT:", result
        except AttributeError as e:
            print "Error", e

if __name__ == "__main__":
    entity_resolution()
\$\endgroup\$
6
  • \$\begingroup\$ Thank you for taking your kind time and showing me how to write code. I know PEPs, I try to do a rough coding first, and then make into classes and all. But your code is very nice. Thank you. I am slightly busy for a day, I am running and letting you know. \$\endgroup\$
    – HIGGINS
    Jul 27, 2016 at 16:34
  • \$\begingroup\$ Sir, sorry for the delay. I tried to run the code. Approach-wise it is far superior but there are few errors coming. I repaired one , and would repair others soon. I trying to get back to you maximum by this weekend. \$\endgroup\$
    – HIGGINS
    Jul 30, 2016 at 11:08
  • \$\begingroup\$ Dear Sir, I created a Pastebin file in link, as writing everything in comment box may be bit tricky. If you may please see. \$\endgroup\$
    – HIGGINS
    Jul 30, 2016 at 15:50
  • \$\begingroup\$ Fixed the missing train_corp in hmm(). Regarding the unhashable dict: Just keep it as lists, then. Sorry, I couldn't test my code, because getting nltk suetup the same way you did would have taken way too much time... \$\endgroup\$
    – Graipher
    Jul 31, 2016 at 10:53
  • \$\begingroup\$ Dear Sir, Thank you for your kind time. I would read your notes and check the code. I should let you know by weekend. As you are a super coder, may I ask one more suggestion? What objective should I have to write excellent code? How would I achieve it? \$\endgroup\$
    – HIGGINS
    Aug 2, 2016 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.