Multiword Expression Tagging in Python

Question

I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs).

I am trying to tag each one, as follows.

I have a dictionary of entries, containing the MWE and tagged MWE. I am tagging them as replace, and whichever I could not tag as MWE -the NMWE I am marking them so.

I wrote the following code

def rule_disc():
    corp=open("/python27/MWETagtext1.txt","r").read().lower()
    print "The Text file given Is:",corp
    mwedict={'prime minister':'prime/MWE minister/MWE','new delhi':'new/MWE    delhi/MWE','reserve bank':'reserve/MWE bank/MWE'}
    dict=mwedict
    mwetag=reduce(lambda x, y: x.replace(y, dict[y]), dict, corp)
    print "MWE Tagged String Is:",mwetag
    mwetagw=mwetag.split()
    list1=[]
    for word in mwetagw:
        if "/" in word:
            list1.append(word)
        else:
            word1=word+"[NMWE]"
            list1.append(word1)
    nlist=list1
    nstring=" ".join(nlist)
    print "The Tagged Text File Is:",nstring 

on the given sample data, which is producing me the result.

My question is, is there a smarter way to do it? Actual data size may run on millions of files.

I am using Python2.7.12 on MS-Windows 7

Answer 1

Several things we can improve:

• don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
• use with context manager to read the file contents
• use a list comprehension for tagging all other words
• generalize the function a bit and provide it with data as an argument

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def tag_words(text):
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in text:
            for word in multi_word.split():
                text = text.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in text.split()])

with open("/python27/MWETagtext1.txt") as f:
    corp = f.read().lower()

print(tag_words(corp))

Since you are doing natural language processing, I think you should also be using word_tokenize() instead of the naive str.split().