I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset through following cleaning step:

import re
from nltk.corpus import stopwords
import pandas as pd

def preprocess(raw_text):

    # keep only words
    letters_only_text = re.sub("[^a-zA-Z]", " ", raw_text)

    # convert to lower case and split 
    words = letters_only_text.lower().split()

    # remove stopwords
    stopword_set = set(stopwords.words("english"))
    meaningful_words = [w for w in words if w not in stopword_set]

    # join the cleaned words in a list
    cleaned_word_list = " ".join(meaningful_words)

    return cleaned_word_list

def process_data(dataset):
    tweets_df = pd.read_csv(dataset,delimiter='|',header=None)

    num_tweets = tweets_df.shape[0]
    print("Total tweets: " + str(num_tweets))

    cleaned_tweets = []
    print("Beginning processing of tweets at: " + str(datetime.now()))

    for i in range(num_tweets):
        cleaned_tweet = preprocess(tweets_df.iloc[i][1])
        if(i % 10000 == 0):
            print(str(i) + " tweets processed")

    print("Finished processing of tweets at: " + str(datetime.now()))
    return cleaned_tweets

cleaned_data = process_data("tweets.csv)

And here is the relevant output:

Total tweets: 216041
Beginning processing of tweets at: 2017-05-16 13:45:47.183113
Finished processing of tweets at: 2017-05-16 13:47:01.436338

It's taking approx. 2 minutes to process the tweets. Although it looks relatively a small timeframe for current dataset I would like to improve it further especially when I use a dataset of much bigger size.

Can the steps/code in the preprocess(raw_text) method be improved in order to achieve faster execution?

  • 1
    \$\begingroup\$ Have you tried profiling your code to see what the bottleneck is? \$\endgroup\$
    – Mast
    May 16, 2017 at 9:12
  • \$\begingroup\$ No. Never tried that. I am fairly new to python. My bad. \$\endgroup\$
    – Ravindra S
    May 16, 2017 at 9:14

4 Answers 4


Copying my answer from SO:

You can use pandas vectorized string methods to do your processing and it also removes the for loop for more efficient pandas operations, this should give you some speed.

# column you are working on
df_ = tweets_df[1]

stopword_set = set(stopwords.words("english"))

# convert to lower case and split 
df_ = df_.str.lower().split()

# remove stopwords
df_ = df_.apply(lambda x: [item for item in x if item not in stopword_set])

# keep only words
regex_pat = re.compile(r'[^a-zA-Z\s]', flags=re.IGNORECASE)
df_ = df_.str.replace(regex_pat, '')

# join the cleaned words in a list

Also I've changed your regex to [^a-zA-Z\s] so that it does not match the space character.


You can replace all those steps with a single list comprehension which will give you a bit more speed. This runs around 30-40% quicker:

def preprocess2(raw_text):
    stopword_set = set(stopwords.words("english"))
    return " ".join([i for i in re.sub(r'[^a-zA-Z\s]', "", raw_text).lower().split() if i not in stopword_set])

I can't comment because low rep but I just wanted to add a quick note. It's the same comment i made on your post on StackOverflow before you moved your post.

If you use a Deque instead of a list for cleaned_tweets, and you insert in front of the list instead of appending, do you gain performance?

I believe it would speed this part cleaned_tweets.append(cleaned_tweet).

Edit: For the preprocess, in fact I thought about something. Did you try passing the stopwords set as a parameter instead of creating it each time for every tweet? If the stopwords set is big, could help a bit. It also make your preprocess function a bit more generic.

  • \$\begingroup\$ Sure thing. But I am more interested in increasing the performance of what is happening inside the preprocess(raw_text) function. \$\endgroup\$
    – Ravindra S
    May 16, 2017 at 9:09
  • \$\begingroup\$ Ok sorry then, I have nothing to add on the preprocess sadly, good luck! \$\endgroup\$ May 16, 2017 at 9:11

Pretty sure I can speed it up.

First: Detailed measurements. You have a time-measurement already. You are doing 4 fairly heavy processing tasks (split, join, etc). Measure each of these seperately.

Speedup: Your stopwords is an unordered set. Python must search the entire set to know if a word matches. Change this to a map - searching will be much faster.

Let me know if this helps.

  • 2
    \$\begingroup\$ set uses hash-based lookup, same as dict. \$\endgroup\$ May 16, 2017 at 10:37
  • \$\begingroup\$ Argh. Facepalm. You are right! \$\endgroup\$
    – user24119
    May 16, 2017 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.