I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach.

import operator,re

for i in range(len(freq_select)):
  if freq_select[i][1]<=25:

def remove_nonfrequent(note):
  for sent in note.split('.'):
    words=sent.split(' ')
    for word in words:
      for j in exclude:
        if j==word:
    sent=' '.join(words)
  note='. '.join(sentences)
  return '. '.join([i for i in note.split('.') if i!=''])


I was initially looping over all the notes in pandas series. But now I am using apply() and I can say performance increased little bit. But it still takes a very long time.

Each note in the pandas.Series is of variable length. However, an average note can contain somewhere between 3000-6000 words. One note even has 13000 words.

I would like to get some help in this.


You currently have to loop over all exclude words to check if it is equal to the current word. If your exclude was a set, you could just do if j in exclude. This change alone should speed up your code a lot (how much depends on how large your list of exclude words actually is).

In addition, you could simplify getting the exclude words:

from collections import Counter
form itertools import takewhile


words = Counter(whatever_generates_your_words())
exclude = set(t[0] for t in takewhile(lambda t: t[1] <= THRESHOLD,

This uses the fact that collections.Counter.most_common is sorted by frequency, so reversing it starts with the least common words. itertools.takewhile stops taking when the condition is no longer true, so you don't need to go through all words in the Counter.

For the filtering you should probably use some list comprehensions. All three of the following functions are doing the same thing:

from itertools import filterfalse

def remove_nonfrequent(note):
    sentences = []
    for sentence in note.split('. '):
        sentence_ = []
        for word in sentence:
            if word not in exclude:
        if sentence:
            sentences.append(" ".join(sentence))
    return ". ".join(sentences)

def remove_nonfrequent(note):
    return ". ".join(" ".join(filterfalse(exclude.__contains__,
                              sentence.split(" ")))
                     for sentence in note.split('. '))
                     if sentence)

def remove_nonfrequent(note):        
    return ". ".join(" ".join(word for word in sentence.split(" ")
                              if word not in exclude)
                     for sentence in note.split('. ') if sentence)

I personally prefer the last one, as it is the most readable one to me.

Note that Python has an official style-guide, PEP8, which recommends using 4 spaces as indentation and surrounding operators with spaces.

Since you are using this function with pandas later, a different approach might actually be faster. There is a pd.Series.replace method that can take a dictionary of replacements. You can use it like this:

from itertools import repeat

replacements = dict(zip((fr'\b{word}\b' for word in exclude), repeat("")))
df.NOTES.replace(replacements, regex=True, inplace=True)
df.NOTES.replace({r' +': ' ', r' +\.': '.'}, regex=True, inplace=True)

The first replace does all the word replacements (using \b to ensure only complete words are replaced), while the latter one fixes multiple spaces and spaces before a period.

| improve this answer | |
  • \$\begingroup\$ Is there any way i can track how much of the execution is done instead of a blank output? i cannot insert print statements in one liners right? \$\endgroup\$ – Aditya Vartak Nov 17 '19 at 17:36
  • 1
    \$\begingroup\$ This works like charm. Thanks @Graipher \$\endgroup\$ – Aditya Vartak Nov 17 '19 at 17:46
  • 1
    \$\begingroup\$ @AdityaVartak: I added a more vectorized version that might be even faster. \$\endgroup\$ – Graipher Nov 18 '19 at 9:30
  • 1
    \$\begingroup\$ Thanks for the effort, but for now the original answer would suffice. Would use this if more optimization is needed. \$\endgroup\$ – Aditya Vartak Nov 18 '19 at 10:53
  • 1
    \$\begingroup\$ @AdityaVartak: Fair enough. I just couldn't leave it be without also including a pandas version :). Which is also a lot shorter and could be encapsulated into its own function. \$\endgroup\$ – Graipher Nov 18 '19 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.