# Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset through following cleaning step:

import re
from nltk.corpus import stopwords
import pandas as pd

def preprocess(raw_text):

# keep only words
letters_only_text = re.sub("[^a-zA-Z]", " ", raw_text)

# convert to lower case and split
words = letters_only_text.lower().split()

# remove stopwords
stopword_set = set(stopwords.words("english"))
meaningful_words = [w for w in words if w not in stopword_set]

# join the cleaned words in a list
cleaned_word_list = " ".join(meaningful_words)

return cleaned_word_list

def process_data(dataset):

num_tweets = tweets_df.shape[0]
print("Total tweets: " + str(num_tweets))

cleaned_tweets = []
print("Beginning processing of tweets at: " + str(datetime.now()))

for i in range(num_tweets):
cleaned_tweet = preprocess(tweets_df.iloc[i][1])
cleaned_tweets.append(cleaned_tweet)
if(i % 10000 == 0):
print(str(i) + " tweets processed")

print("Finished processing of tweets at: " + str(datetime.now()))
return cleaned_tweets

cleaned_data = process_data("tweets.csv)


And here is the relevant output:

Total tweets: 216041
Beginning processing of tweets at: 2017-05-16 13:45:47.183113
Finished processing of tweets at: 2017-05-16 13:47:01.436338


It's taking approx. 2 minutes to process the tweets. Although it looks relatively a small timeframe for current dataset I would like to improve it further especially when I use a dataset of much bigger size.

Can the steps/code in the preprocess(raw_text) method be improved in order to achieve faster execution?

• Have you tried profiling your code to see what the bottleneck is? – Mast May 16 '17 at 9:12
• No. Never tried that. I am fairly new to python. My bad. – Pale Blue Dot May 16 '17 at 9:14

You can use pandas vectorized string methods to do your processing and it also removes the for loop for more efficient pandas operations, this should give you some speed.

# column you are working on
df_ = tweets_df[1]

stopword_set = set(stopwords.words("english"))

# convert to lower case and split
df_ = df_.str.lower().split()

# remove stopwords
df_ = df_.apply(lambda x: [item for item in x if item not in stopword_set])

# keep only words
regex_pat = re.compile(r'[^a-zA-Z\s]', flags=re.IGNORECASE)
df_ = df_.str.replace(regex_pat, '')

# join the cleaned words in a list
df_.str.join("")


Also I've changed your regex to [^a-zA-Z\s] so that it does not match the space character.

You can replace all those steps with a single list comprehension which will give you a bit more speed. This runs around 30-40% quicker:

def preprocess2(raw_text):
stopword_set = set(stopwords.words("english"))
return " ".join([i for i in re.sub(r'[^a-zA-Z\s]', "", raw_text).lower().split() if i not in stopword_set])


I can't comment because low rep but I just wanted to add a quick note. It's the same comment i made on your post on StackOverflow before you moved your post.

If you use a Deque instead of a list for cleaned_tweets, and you insert in front of the list instead of appending, do you gain performance?

I believe it would speed this part cleaned_tweets.append(cleaned_tweet).

Edit: For the preprocess, in fact I thought about something. Did you try passing the stopwords set as a parameter instead of creating it each time for every tweet? If the stopwords set is big, could help a bit. It also make your preprocess function a bit more generic.

• Sure thing. But I am more interested in increasing the performance of what is happening inside the preprocess(raw_text) function. – Pale Blue Dot May 16 '17 at 9:09
• Ok sorry then, I have nothing to add on the preprocess sadly, good luck! – Julien Rousé May 16 '17 at 9:11

Pretty sure I can speed it up.

First: Detailed measurements. You have a time-measurement already. You are doing 4 fairly heavy processing tasks (split, join, etc). Measure each of these seperately.

Speedup: Your stopwords is an unordered set. Python must search the entire set to know if a word matches. Change this to a map - searching will be much faster.

Let me know if this helps.

• set uses hash-based lookup, same as dict. – Janne Karila May 16 '17 at 10:37
• Argh. Facepalm. You are right! – user24119 May 16 '17 at 11:36