I am trying to extract names from a body of text to use as stopwords. I tried a few different approaches to identifying names (or proper nouns in general) below. The second approach is much faster than the first, but is admittedly a much more naïve approach (misses out on names like 'PETER' or misspellings like 'jOHN'). Specifically, I'm looking for ways to improve the extract of names (places, people, etc.), but not sacrifice speed. Most of
nltk has been pretty bulky, so if there's a better 'naïve' approach to the one I use below, I would appreciate any input.
I've generated some random strings below for demonstration.
import random, string from time import time from nltk import pos_tag def randomword(length): return ''.join(random.choice(string.ascii_letters) for i in range(length)) def remainder_lower(string): if not string.isupper(): return False for letter in string[1:]: if letter.isupper(): return False return True def has_excluded_elements(string, to_exclude): for letter in string: if letter in to_exclude: return True return False # generate 50000 strings with four words in each string words = [" ".join(randomword(10) for y in range(4)) for x in range(50000)] # 2 minutes on my machine to run pos_tag_words = [word for row in words for word in pos_tag(set(row.split())) if word == 'NNP'] # 0.2 seconds on my machine possible_names1 = [word for row in words for word in set(row.split()) \ if remainder_lower(word) and not has_excluded_elements(word, string.punctuation)]