# Multiword Expression Tagging in Python

I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs).

I am trying to tag each one, as follows.

I have a dictionary of entries, containing the MWE and tagged MWE. I am tagging them as replace, and whichever I could not tag as MWE -the NMWE I am marking them so.

I wrote the following code

def rule_disc():
print "The Text file given Is:",corp
mwedict={'prime minister':'prime/MWE minister/MWE','new delhi':'new/MWE    delhi/MWE','reserve bank':'reserve/MWE bank/MWE'}
dict=mwedict
mwetag=reduce(lambda x, y: x.replace(y, dict[y]), dict, corp)
print "MWE Tagged String Is:",mwetag
mwetagw=mwetag.split()
list1=[]
for word in mwetagw:
if "/" in word:
list1.append(word)
else:
word1=word+"[NMWE]"
list1.append(word1)
nlist=list1
nstring=" ".join(nlist)
print "The Tagged Text File Is:",nstring


on the given sample data, which is producing me the result.

My question is, is there a smarter way to do it? Actual data size may run on millions of files.

I am using Python2.7.12 on MS-Windows 7

Several things we can improve:

• don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
• use with context manager to read the file contents
• use a list comprehension for tagging all other words
• generalize the function a bit and provide it with data as an argument

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def tag_words(text):
# tag all MWEs
for multi_word in MWE:
if multi_word in text:
for word in multi_word.split():
text = text.replace(word, word + "/MWE")

# tag all NMWEs and return
return " ".join([word if "/" in word else word + "[NMWE]"
for word in text.split()])

with open("/python27/MWETagtext1.txt") as f:

Since you are doing natural language processing, I think you should also be using word_tokenize() instead of the naive str.split().
• @HIGGINS ah, if tagging may differ depending on the multi-word, then yes, you better have a dictionary, but, try not to repeat the multi words in keys and values and have e.g. {'new delphi': 'GPE', 'reserve bank': 'ORG'} etc. Hope I understood the problem correctly. Feb 1 '17 at 21:30