Manipulating list values in python

Question

I wrote a function that I use to preprocess pandas dataframes before running them through a machine learning model. The function works perfectly, however I don't think it's the most pythonic way to write it.

This function accepts a list of words:

['here', 'is', 'a','sample', 'of','what','the','function','accepts']

def clean_text(x):

    stopwords_english = stopwords.words('english')

    for i,word in enumerate(x):
        if word.lower() in stopwords_english:
            x[i] = ''
        else:
            for punct in "/-'":
                x[i] = word.replace(punct, ' ')
            for punct in '&':
                x[i] = word.replace(punct, f' {punct} ')
            for punct in '?!.,"#$%\'()*+-/:;<=>@[\\]^_`{|}~' + '“”’':
                x[i] = word.replace(punct, '')
    return x

Here I am using enumerate to change the value inside of the list. I would have assumed a more pythonic way of doing it would be writing it as follows:

def clean_text(x):

    stopwords_english = stopwords.words('english')

    for word in x:
        if word.lower() in stopwords_english:
            word = ''
        else:
            for punct in "/-'":
                word = word.replace(punct, ' ')
            for punct in '&':
                word = word.replace(punct, f' {punct} ')
            for punct in '?!.,"#$%\'()*+-/:;<=>@[\\]^_`{|}~' + '“”’':
                word = word.replace(punct, '')
    return x

The function is being called as follows:

train['question_text'].progress_apply(lambda x: clean_text(x))

Where train is a pandas dataframe and 'question_text' is a column in the dataframe.

Is my current implementation the most pythonic way, or is there a better way?

Answer 1

This may be a good case for generator functions. And splitting it into two parts might make things more flexible: first, remove the stopwords; and second, handle the punctuation.

Also, str.maketrans and str.translate can do the punctuation mapping.

def remove_stopwords(text_iterable, stopwords):
    for word in text_iterable:
        if word.lower() not in stopwords:
            yield word


def handle_punctuation(text_iterable, table):
    for word in text_iterable:
        yield word.translate(table)


# format is ("chars", "replacement")          
mappings = (("/-'", ' '),
            ('&', ' & '),
            ('?!.,"#$%\'()*+-/:;<=>@[\\]^_`{|}~' + '“”’', None))

table = str.maketrans({c:v for k,v in mappings for c in k})

stopword_free = remove_stopwords(text, stopwords)
cleaned_text = handle_punctuation(stopword_free, table)

cleaned_text is a generator, use list(handle_punctuation(stopword_free, table)) if you need an actual list of words.

Answer 2

Honestly, I can not consider neither of proposed approaches (except for applying generators) as efficient enough.

Here are my arguments:

• stopwords.words('english') sequence.
  As clean_text(x) will be applied for each column cell it's better to move a common sequence of stopwords to the top level at once. But that's easy. stopwords.words('english') is actually a list of stopwords and a much more efficient would be to convert it into set object for fast containment check (in if word.lower() in stopwords_english):

  stopwords_english = set(stopwords.words('english'))
  

• instead of yielding a words that aren't contained in stopwords_english set for further replacements - in opposite, words that are stopwords can be just skipped at once:

  if word.lower() in stopwords_english:
      continue
  

• subtle nuance: the pattern "/-'" at 1st replacement attempt (for punct in "/-'") is actually contained in a longer pattern of punctuation chars ?!.,"#$%\'()*+-/:;<=>@[\\]^_`{|}~' + '“”’'.
  Thus, it's unified into a single pattern and considering that there could be multiple consequent occurrences of punctuation/special char within a word - I suggest to apply a compiled regex pattern with + quantifier (to replace multiple occurrences at once) defined at top level.

---

Finally, the optimized approach would look as follows:

import re

...
stopwords_english = set(stopwords.words('english'))
punct_pat = re.compile(r'[?!.,"#$%\'()*+-/:;<=>@\[\\\]^_`{|}~“”’]+')


def clean_text(x):
    for word in x:
        if word.lower() in stopwords_english:
            continue
        if '&' in word:
            word = word.replace('&', ' & ')
        yield punct_pat.sub('', word)

Applied as:

train['question_text'].progress_apply(lambda x: list(clean_text(x)))