I am trying to write a small python code,where I am reading a text file-which contains both Multiwords (MWEs) and singular words (NMWEs).

I am trying to tag each one, as follows.

I have a dictionary of entries, containing the MWE and tagged MWE. I am tagging them as replace, and whichever I could not tag as MWE -the NMWE I am marking them so.

I wrote the following code

def rule_disc():
    print "The Text file given Is:",corp
    mwedict={'prime minister':'prime/MWE minister/MWE','new delhi':'new/MWE    delhi/MWE','reserve bank':'reserve/MWE bank/MWE'}
    mwetag=reduce(lambda x, y: x.replace(y, dict[y]), dict, corp)
    print "MWE Tagged String Is:",mwetag
    for word in mwetagw:
        if "/" in word:
    nstring=" ".join(nlist)
    print "The Tagged Text File Is:",nstring 

on the given sample data, which is producing me the result.

My question is, is there a smarter way to do it? Actual data size may run on millions of files.

I am using Python2.7.12 on MS-Windows 7


Several things we can improve:

  • don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
  • use with context manager to read the file contents
  • use a list comprehension for tagging all other words
  • generalize the function a bit and provide it with data as an argument

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def tag_words(text):
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in text:
            for word in multi_word.split():
                text = text.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in text.split()])

with open("/python27/MWETagtext1.txt") as f:
    corp = f.read().lower()


Since you are doing natural language processing, I think you should also be using word_tokenize() instead of the naive str.split().

  • \$\begingroup\$ Sir, thanks for your time and prompt answer. Now if I change the MWE value for each MWE item like new delhi may be new/GPE delhi/GPE, reserve bank may be reserve/ORG bank/ORG etc. \$\endgroup\$
    Feb 1 '17 at 21:28
  • \$\begingroup\$ @HIGGINS ah, if tagging may differ depending on the multi-word, then yes, you better have a dictionary, but, try not to repeat the multi words in keys and values and have e.g. {'new delphi': 'GPE', 'reserve bank': 'ORG'} etc. Hope I understood the problem correctly. \$\endgroup\$
    – alecxe
    Feb 1 '17 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.