Converting from POS tagged word-tokens to POS tagged Phrases

Question

I have sentences expressed as Parts of speech (POS) tagged words. I want to have all the short phrases in them to be joined up with underscores. I want them to have the Part of speech tag of the final word in the phrase -- not because this is necessarily linguistically correct, but because it will perform correctly in my system later when it has to "Unstem"/"Unlemmaize" derived words/phrases.

For example, if I had the following sentence, (based off of the first sentence in the Microsoft research paraphrase corpus):

PCCW's chief operating officer, Mike Butcher, and the Arena brothers, the chief financial officers, will report directly to the police officer.

Then the POS tagged text is:

[('PCCW', 'NNP'), ("'s", 'POS'), ('chief', 'NN'),('operating', 'VBG'), ('officer', 'NN'),(',', ','),('Mike', 'NNP'),('Butcher', 'NNP'), (',', ','),('and', 'CC'),('the', 'DT'),('Arena', 'NNP'),('brothers', 'NNS'),(',', ','),('the', 'DT'),('chief', 'JJ'), ('financial', 'JJ'),('officers', 'NNS'),(',', ','),('will', 'MD'),('report', 'VB'),('directly','RB'),('to', 'TO'),('the', 'DT'), ('police', 'NN'),('officer', 'NN'), ('.', '.')]

And the POS tagged phrases (ie the output of my function) is:

[('PCCW', 'NNP'), ("'s", 'POS'), ('chief_operating_officer', 'NN'), (',', ','), ('Mike', 'NNP'), ('Butcher', 'NNP'), (',', ','), ('and', 'CC'), ('the', 'DT'), ('Arena', 'NNP'), ('brothers', 'NNS'), (',', ','), ('the', 'DT'), ('chief_financial_officers', 'NNS'), (',', ','), ('will', 'MD'), ('report', 'VB'), ('directly', 'RB'), ('to', 'TO'), ('the', 'DT'), ('police_officer', 'NN'), ('.', '.')]

I am prepared to accept WordNet as the ground truth as to the existence or not of a phrase.

from nltk.corpus import wordnet as wn    
def get_tagged_phrases(tagged_sent, max_phrase_length):

    tagged_phrase_sent = list(tagged_sent)
    for phrase_len in range(max_phrase_length,1,-1): #Go from largest to smallest to keep information
        for indexes in n_wise(phrase_len, range(len(tagged_sent))):
            tagged_words = [tagged_phrase_sent[index] for index in indexes]
            if not(any([tagged_word is None for tagged_word in tagged_words])):
                words, tags = zip(*tagged_words)
                possible_phrase = "_".join(words)
                if wn.synsets(possible_phrase): #If there are any, then it is a phrase
                    for index in indexes:
                        tagged_phrase_sent[index] = None #Blank them out with Nones which we will remove later
                    pos = tags[-1] #Use final tag, it will be the one we need for handling plurals
                    tagged_phrase_sent[indexes[0]] = (possible_phrase, pos)
    return [tagged_phrase for tagged_phrase in tagged_phrase_sent if not tagged_phrase is None]

The obvious code smell is that it is nested about 5 deep. That is too much state to remember, perhaps.

---

This is in Python 2

Answer 1

Apart from choosing more optimal functions (xrange, in), this looks okay I think; a bit more nesting can be removed by inverting the conditions and use continue instead; it also looks like the last return statement could be faster if the intermediate results from the inner loops were collected in a separate result, but I may read that wrong.

from nltk.corpus import wordnet as wn


def get_tagged_phrases(tagged_sent, max_phrase_length):
    tagged_sent = list(tagged_sent)

    for phrase_len in xrange(max_phrase_length, 1, -1): #Go from largest to smallest to keep information
        for indexes in n_wise(phrase_len, xrange(len(tagged_sent))):
            tagged_words = [tagged_sent[index] for index in indexes]

            if None in tagged_words:
                continue

            words, tags = zip(*tagged_words)
            possible_phrase = "_".join(words)

            if not wn.synsets(possible_phrase): #If there are any, then it is a phrase
                continue

            for index in indexes:
                tagged_sent[index] = None #Blank them out with Nones which we will remove later

            pos = tags[-1] #Use final tag, it will be the one we need for handling plurals
            tagged_sent[indexes[0]] = (possible_phrase, pos)

    return [tagged_phrase for tagged_phrase in tagged_sent if not tagged_phrase is None]