Skip to main content
added 91 characters in body
Source Link
alecxe
  • 17.3k
  • 8
  • 51
  • 93

Several things we can improve:

  • don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
  • use with context manager to read the file contents
  • use a list comprehension for tagging all other words
  • generalize the function a bit and provide it with data as an argument

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def rule_disc():
    with opentag_words("/python27/MWETagtext1.txt"text) as f:
        corp = f.read().lower()
    
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in corptext:
            for word in multi_word.split():
                corptext = corptext.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in corptext.split()]) 

with open("/python27/MWETagtext1.txt") as f:
    corp = f.read().lower()

print(tag_words(corp))

Since you are doing natural language processing, I think you should also be using word_tokenize() instead of the naive str.split().

Several things we can improve:

  • don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
  • use with context manager to read the file contents
  • use a list comprehension for tagging all other words

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def rule_disc():
    with open("/python27/MWETagtext1.txt") as f:
        corp = f.read().lower()
    
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in corp:
            for word in multi_word.split():
                corp = corp.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in corp.split()])

Several things we can improve:

  • don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
  • use with context manager to read the file contents
  • use a list comprehension for tagging all other words
  • generalize the function a bit and provide it with data as an argument

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def tag_words(text):
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in text:
            for word in multi_word.split():
                text = text.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in text.split()]) 

with open("/python27/MWETagtext1.txt") as f:
    corp = f.read().lower()

print(tag_words(corp))

Since you are doing natural language processing, I think you should also be using word_tokenize() instead of the naive str.split().

Source Link
alecxe
  • 17.3k
  • 8
  • 51
  • 93

Several things we can improve:

  • don't repeat the multi words and the same multi words with tags - just have a list of multi-words and, if it is in the corp, replace each word with a tagged word
  • use with context manager to read the file contents
  • use a list comprehension for tagging all other words

Improved version:

MWE = ['prime minister', 'new delhi', 'reserve bank']

def rule_disc():
    with open("/python27/MWETagtext1.txt") as f:
        corp = f.read().lower()
    
    # tag all MWEs
    for multi_word in MWE:
        if multi_word in corp:
            for word in multi_word.split():
                corp = corp.replace(word, word + "/MWE")

    # tag all NMWEs and return
    return " ".join([word if "/" in word else word + "[NMWE]"
                     for word in corp.split()])