0
\$\begingroup\$

I have sentences expressed as Parts of speech (POS) tagged words. I want to have all the short phrases in them to be joined up with underscores. I want them to have the Part of speech tag of the final word in the phrase -- not because this is necessarily linguistically correct, but because it will perform correctly in my system later when it has to "Unstem"/"Unlemmaize" derived words/phrases.

For example, if I had the following sentence, (based off of the first sentence in the Microsoft research paraphrase corpus):

PCCW's chief operating officer, Mike Butcher, and the Arena brothers, the chief financial officers, will report directly to the police officer.

Then the POS tagged text is:

[('PCCW', 'NNP'), ("'s", 'POS'), ('chief', 'NN'),('operating', 'VBG'), ('officer', 'NN'),(',', ','),('Mike', 'NNP'),('Butcher', 'NNP'), (',', ','),('and', 'CC'),('the', 'DT'),('Arena', 'NNP'),('brothers', 'NNS'),(',', ','),('the', 'DT'),('chief', 'JJ'), ('financial', 'JJ'),('officers', 'NNS'),(',', ','),('will', 'MD'),('report', 'VB'),('directly','RB'),('to', 'TO'),('the', 'DT'), ('police', 'NN'),('officer', 'NN'), ('.', '.')]

And the POS tagged phrases (ie the output of my function) is:

[('PCCW', 'NNP'), ("'s", 'POS'), ('chief_operating_officer', 'NN'), (',', ','), ('Mike', 'NNP'), ('Butcher', 'NNP'), (',', ','), ('and', 'CC'), ('the', 'DT'), ('Arena', 'NNP'), ('brothers', 'NNS'), (',', ','), ('the', 'DT'), ('chief_financial_officers', 'NNS'), (',', ','), ('will', 'MD'), ('report', 'VB'), ('directly', 'RB'), ('to', 'TO'), ('the', 'DT'), ('police_officer', 'NN'), ('.', '.')]

I am prepared to accept WordNet as the ground truth as to the existence or not of a phrase.

from nltk.corpus import wordnet as wn    
def get_tagged_phrases(tagged_sent, max_phrase_length):

    tagged_phrase_sent = list(tagged_sent)
    for phrase_len in range(max_phrase_length,1,-1): #Go from largest to smallest to keep information
        for indexes in n_wise(phrase_len, range(len(tagged_sent))):
            tagged_words = [tagged_phrase_sent[index] for index in indexes]
            if not(any([tagged_word is None for tagged_word in tagged_words])):
                words, tags = zip(*tagged_words)
                possible_phrase = "_".join(words)
                if wn.synsets(possible_phrase): #If there are any, then it is a phrase
                    for index in indexes:
                        tagged_phrase_sent[index] = None #Blank them out with Nones which we will remove later
                    pos = tags[-1] #Use final tag, it will be the one we need for handling plurals
                    tagged_phrase_sent[indexes[0]] = (possible_phrase, pos)
    return [tagged_phrase for tagged_phrase in tagged_phrase_sent if not tagged_phrase is None]

The obvious code smell is that it is nested about 5 deep. That is too much state to remember, perhaps.


This is in Python 2

\$\endgroup\$
  • \$\begingroup\$ If you're specifically targeting Python 2.6 or 2.7 you can also add a tag for that. \$\endgroup\$ – ferada Jul 13 '15 at 15:23
  • \$\begingroup\$ I guess I should use it, one of my dependancies (pattern) doesn't come in python 3 flavour \$\endgroup\$ – Lyndon White Jul 13 '15 at 23:55
1
\$\begingroup\$

Apart from choosing more optimal functions (xrange, in), this looks okay I think; a bit more nesting can be removed by inverting the conditions and use continue instead; it also looks like the last return statement could be faster if the intermediate results from the inner loops were collected in a separate result, but I may read that wrong.

from nltk.corpus import wordnet as wn


def get_tagged_phrases(tagged_sent, max_phrase_length):
    tagged_sent = list(tagged_sent)

    for phrase_len in xrange(max_phrase_length, 1, -1): #Go from largest to smallest to keep information
        for indexes in n_wise(phrase_len, xrange(len(tagged_sent))):
            tagged_words = [tagged_sent[index] for index in indexes]

            if None in tagged_words:
                continue

            words, tags = zip(*tagged_words)
            possible_phrase = "_".join(words)

            if not wn.synsets(possible_phrase): #If there are any, then it is a phrase
                continue

            for index in indexes:
                tagged_sent[index] = None #Blank them out with Nones which we will remove later

            pos = tags[-1] #Use final tag, it will be the one we need for handling plurals
            tagged_sent[indexes[0]] = (possible_phrase, pos)

    return [tagged_phrase for tagged_phrase in tagged_sent if not tagged_phrase is None]
\$\endgroup\$
  • \$\begingroup\$ good point about xrange (I forgot I was using python 2), \$\endgroup\$ – Lyndon White Jul 13 '15 at 12:30
  • \$\begingroup\$ None in tagged_words is a lot clearer. I'm not sure about the general use of continue over nested iffs inside loops. (Actually unsure, I see pros and cons) \$\endgroup\$ – Lyndon White Jul 13 '15 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.