Several things we can improve:
- don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the
corp
, replace each word with a tagged word - use
with
context manager to read the file contents - use a list comprehension for tagging all other words
- generalize the function a bit and provide it with data as an argument
Improved version:
MWE = ['prime minister', 'new delhi', 'reserve bank']
def rule_disc():
with opentag_words("/python27/MWETagtext1.txt"text) as f:
corp = f.read().lower()
# tag all MWEs
for multi_word in MWE:
if multi_word in corptext:
for word in multi_word.split():
corptext = corptext.replace(word, word + "/MWE")
# tag all NMWEs and return
return " ".join([word if "/" in word else word + "[NMWE]"
for word in corptext.split()])
with open("/python27/MWETagtext1.txt") as f:
corp = f.read().lower()
print(tag_words(corp))
Since you are doing natural language processing, I think you should also be using word_tokenize()
instead of the naive str.split()
.