Remove words from an exclude list in each row of a pandas.Series

Question

I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach.

import operator,re

exclude=[]
for i in range(len(freq_select)):
  if freq_select[i][1]<=25:
    exclude.append(freq_select[i][0])



def remove_nonfrequent(note):
  print(remove_nonfrequent.i)
  sentences=[]
  for sent in note.split('.'):
    words=sent.split(' ')
    #print("Before:",len(words))
    for word in words:
      for j in exclude:
        if j==word:
          #print(j,word)
          words.remove(word)
    #print("After:",len(words))
    sent=' '.join(words)
    sentences.append(sent)
  note='. '.join(sentences)
  remove_nonfrequent.i+=1
  return '. '.join([i for i in note.split('.') if i!=''])



remove_nonfrequent.i=0
jdf.NOTES=jdf.NOTES.apply(remove_nonfrequent)

I was initially looping over all the notes in pandas series. But now I am using apply() and I can say performance increased little bit. But it still takes a very long time.

Each note in the pandas.Series is of variable length. However, an average note can contain somewhere between 3000-6000 words. One note even has 13000 words.

I would like to get some help in this.

Answer 1

You currently have to loop over all exclude words to check if it is equal to the current word. If your exclude was a set, you could just do if j in exclude. This change alone should speed up your code a lot (how much depends on how large your list of exclude words actually is).

In addition, you could simplify getting the exclude words:

from collections import Counter
form itertools import takewhile

THRESHOLD = 25

words = Counter(whatever_generates_your_words())
exclude = set(t[0] for t in takewhile(lambda t: t[1] <= THRESHOLD,
                                      reversed(words.most_common())))

This uses the fact that collections.Counter.most_common is sorted by frequency, so reversing it starts with the least common words. itertools.takewhile stops taking when the condition is no longer true, so you don't need to go through all words in the Counter.

For the filtering you should probably use some list comprehensions. All three of the following functions are doing the same thing:

from itertools import filterfalse

def remove_nonfrequent(note):
    sentences = []
    for sentence in note.split('. '):
        sentence_ = []
        for word in sentence:
            if word not in exclude:
                sentence_.append(word)
        if sentence:
            sentences.append(" ".join(sentence))
    return ". ".join(sentences)

def remove_nonfrequent(note):
    return ". ".join(" ".join(filterfalse(exclude.__contains__,
                              sentence.split(" ")))
                     for sentence in note.split('. '))
                     if sentence)

def remove_nonfrequent(note):        
    return ". ".join(" ".join(word for word in sentence.split(" ")
                              if word not in exclude)
                     for sentence in note.split('. ') if sentence)

I personally prefer the last one, as it is the most readable one to me.

Note that Python has an official style-guide, PEP8, which recommends using 4 spaces as indentation and surrounding operators with spaces.

---

Since you are using this function with pandas later, a different approach might actually be faster. There is a pd.Series.replace method that can take a dictionary of replacements. You can use it like this:

from itertools import repeat

replacements = dict(zip((fr'\b{word}\b' for word in exclude), repeat("")))
df.NOTES.replace(replacements, regex=True, inplace=True)
df.NOTES.replace({r' +': ' ', r' +\.': '.'}, regex=True, inplace=True)

The first replace does all the word replacements (using \b to ensure only complete words are replaced), while the latter one fixes multiple spaces and spaces before a period.