6
\$\begingroup\$

What is the fastest Pythonic way to remove all stopwords from a list of words in a document? Right now I am using a list comprehension that contains a for loop.

from nltk.corpus import stopwords

''' Push stopwords to a list '''
stop = stopwords.words('english')
Document = ' Some huge text .......................... '
''' Tokenize the doc '''
words = nltk.word_tokenize(Document)
''' Comparing two lists '''
stopwordsfree_words = [word for word in words if word not in stop]

Is there a faster way to do this?

\$\endgroup\$
8
\$\begingroup\$

If stop is a list containing \$s\$ stopwords, and words is a list containing \$w\$ words, then the loop in the list comprehension will be \$O(w s)\$, since it basically has to iterate over both lists in a nested loop.

However, if you make the stopwords into a set

stop = set(stopwords.words('english'))

… then each lookup can be done in \$O(1)\$ time. You would get \$O(w)\$ running time just by changing the data structure like that.

Another minor issue is that by convention, Document should be lowercase, because it is a variable rather than a class.

\$\endgroup\$
  • 1
    \$\begingroup\$ Oh, so sets in python are like dictionaries with no values. Thanks that helps. \$\endgroup\$ – Siddarth May 13 '15 at 23:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.