1
\$\begingroup\$

I am trying to extract names from a body of text to use as stopwords. I tried a few different approaches to identifying names (or proper nouns in general) below. The second approach is much faster than the first, but is admittedly a much more naïve approach (misses out on names like 'PETER' or misspellings like 'jOHN'). Specifically, I'm looking for ways to improve the extract of names (places, people, etc.), but not sacrifice speed. Most of nltk has been pretty bulky, so if there's a better 'naïve' approach to the one I use below, I would appreciate any input.

I've generated some random strings below for demonstration.

import random, string
from time import time
from nltk import pos_tag

def randomword(length):
   return ''.join(random.choice(string.ascii_letters) for i in range(length))

def remainder_lower(string):
    if not string[0].isupper():
        return False
    for letter in string[1:]:
        if letter.isupper():
            return False
    return True

def has_excluded_elements(string, to_exclude):
    for letter in string:
        if letter in to_exclude:
            return True
    return False

# generate 50000 strings with four words in each string
words = [" ".join(randomword(10) for y in range(4)) for x in range(50000)]

# 2 minutes on my machine to run
pos_tag_words = [word for row in words for word in pos_tag(set(row.split())) if word[1] == 'NNP']

# 0.2 seconds on my machine
possible_names1 = [word for row in words for word in set(row.split()) \
                   if remainder_lower(word) and not has_excluded_elements(word, string.punctuation)]
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.