I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier).

My data is being read in via a csv file, with 200,000 rows, that looks like this:

AUSTINLOVESBEER,NHS fails to treat one in six cancer patients on time https://t.co/8M6o3wn7WG,41,34,1,1
AUSTINLOVESBEER,Real reason Alexis Sanchez walked out of Arsenal training revealed https://t.co/zNkBLkCuz9 https://t.co/S0fIhs70sv,41,34,1,1
AUSTINLOVESBEER,George Michael cause of death revealed: What is a dilated cardiomyopathy and fatty liver? https://t.co/luiDpJOc4h https://t.co/g3S2LGJ8hq,41,34,1,1
AUSTINLOVESBEER,Russian TV crew 'offer Swedish teenagers money' to cause trouble in front of cameras https://t.co/kzm7e7U5S9,41,34,1,1
AUSTINLOVESBEER,Donald Trump met Russian ambassador during election campaign https://t.co/oArd5BSGBK,41,34,1,1
AUSTINLOVESBEER,MPs raise concerns over tougher knife crime guidelines' impact on prison populations https://t.co/KJ0tkclknd,41,34,1,1
AUSTINLOVESBEER,Ex-Arsenal coach Laura Harvey blazes a trail in America https://t.co/nYllUTzEws,41,34,1,1
AUSTINLOVESBEER,Russia deploys deadly 'SIZZLER' nuclear submarines as US conflict looms https://t.co/YIqseUIyHq https://t.co/gBabIzOmRh,41,34,1,1
AUSTINLOVESBEER,#Towie: 'Nothing happened with Dan' https://t.co/YNdnqlfeaA https://t.co/DOkZ7h7Gtc,41,34,1,1
AUSTINLOVESBEER,Player ratings: Can any Arsenal players hold their heads high after last night? https://t.co/HM1aFTcIxK https://t.co/z3FmbJkLUG,41,34,1,1
AUSTINLOVESBEER,"Cannabis and prescription painkillers flooding Gaza Strip, Hamas warns https://t.co/muEYFZvx8O",41,34,1,1

I read the data into a pandas data frame, and from there, attempt to vectorize the tweets with a TFxIDF weighting scheme using sklearn's TFxIDF vectorizer. It will create an insanely large feature vector, but that is OK.

When running the code, I continuously get errors related to MemoryError. I am requesting a review of my code to see where I can be more memory efficient and, therefore, not have a memory error occur.

NOTE - I initially tried to take in over ~1M tweets, and so I read those into two separate data frames and performed a np.vstack operation on them. I got the memory error there, so that is when I reduced this to only 200K samples, but am still getting the same error:

Traceback (most recent call last):
  File "d:\Grad School\Fall 2019\605.744.81.FA19 - Information Retrieval\Project\Data\Trainer.py", line 90, in <module>
    df = pd.DataFrame(data=test.todense(), columns=vectorizer.get_feature_names())
  File "C:\Users\Kelly\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\base.py", line 849, in todense
    return np.asmatrix(self.toarray(order=order, out=out))
  File "C:\Users\Kelly\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\compressed.py", line 962, in toarray
    out = self._process_toarray_args(order, out)
  File "C:\Users\Kelly\AppData\Local\Programs\Python\Python37\lib\site-packages\scipy\sparse\base.py", line 1187, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)

Codes below:

import pandas as pd, numpy as np, re, string
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import confusion_matrix
from nltk.corpus import stopwords

stop_words = ['ourselves', 'hers', 'between', 'yourself', 'but', 
'again', 'there', 'about', 'once', 'during', 'out', 'very', 
'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 
'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 
'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 
'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 
'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 
'were', 'her', 'more', 'himself', 'this', 'down', 'should', 
'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 
'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 
'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 
'does', 'yourselves', 'then', 'that', 'because', 'what', 
'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 
'he', 'you', 'herself', 'has', 'just', 'where', 'too', 
'only', 'myself', 'which', 'those', 'i', 'after', 'few', 
'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 
'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']

# This function removes numbers from an array
def remove_nums(arr): 
    # Declare a regular expression
    pattern = '[0-9]'  
    # Remove the pattern, which is a number
    arr = [re.sub(pattern, '', i) for i in arr]    
    # Return the array with numbers removed
    return arr

# This function cleans the passed in paragraph and parses it
def get_words(para):   
    # Split it into lower case    
    lower = para.lower().split()
    # Remove punctuation
    no_punctuation = (nopunc.translate(str.maketrans('', '', string.punctuation)) for nopunc in lower)
    # Remove integers
    no_integers = remove_nums(no_punctuation)
    # Remove stop words
    dirty_tokens = (data for data in no_integers if data not in stop_words)
    # Ensure it is not empty
    tokens = (data for data in dirty_tokens if data.strip())
    # Ensure there is more than 1 character to make up the word
    tokens = (data for data in tokens if len(data) > 1)

    # Return the tokens
    return tokens 

# Function to collect required F1, Precision, and Recall Metrics
def collect_metrics(actuals, preds):
    # Create a confusion matrix
    matr = confusion_matrix(actuals, preds, labels=[2, 4])
    # Retrieve TN, FP, FN, and TP from the matrix
    true_negative, false_positive, false_negative, true_positive = confusion_matrix(actuals, preds).ravel()

    # Compute precision
    precision = true_positive / (true_positive + false_positive)
    # Compute recall
    recall = true_positive / (true_positive + false_negative)
    # Compute F1
    f1 = 2*((precision*recall)/(precision + recall))

    # Return results
    return precision, recall, f1

# not_bot = pd.read_csv("filepath", skiprows=1)
# bot = pd.read_csv("filepath", skiprows=1)
# csv_table = pd.DataFrame(np.vstack((not_bot.values, bot.values)))
# csv_table.columns = ['username', 'tweet', 'following', 'followers', 'is_retweet', 'is_bot']

csv_table = pd.read_csv("filepath", dtype={'username': str, 'tweet': str, 'following': int, 'followers': int, 'is_retweet': int, 'is_bot': int})

# Create the overall corpus
s = pd.Series(csv_table['tweet'])
corpus = s.apply(lambda s: ' '.join(get_words(s)))

# Create a vectorizer
vectorizer = TfidfVectorizer()
# Compute tfidf values
# This also updates the vectorizer
test = vectorizer.fit_transform(corpus)

# Create a dataframe from the vectorization procedure
df = pd.DataFrame(data=test.todense(), columns=vectorizer.get_feature_names())

# Merge results into final dataframe
result = pd.concat([csv_table, df], axis=1, sort=False)

labels = result['is_bot']
result = result.drop(['is_bot'])

X_train, y_train, X_test, y_test = train_test_split(result, labels, test_size = 0.2)

knn = KNeighborsClassifier(n_neighbors = 7)
rf = RandomForestClassifier()
mlp = MLPClassifier()

knn.fit(X_train, y_train)
rf.fit(X_train, y_train)
mlp.fit(X_train, y_train)

knn_preds = knn.predict(X_test)
rf_preds = rf.predict(X_test)
mlp_preds = mlp.predict(X_test)

knn_precision, knn_recall, knn_f1 = collect_metrics(knn_preds, y_test)
rf_precision, rf_recall, rf_f1 = collect_metrics(rf_preds, y_test)
mlp_precision, mlp_recall, mlp_f1 = collect_metrics(mlp_preds, y_test)

# Pretty print the results
print("KNN | Recall: {} | Precision: {} | F1: {}".format(knn_recall, knn_precision, knn_f1))
print("MLP     | Recall: {} | Precision: {} | F1: {}".format(mlp_recall, mlp_precision, mlp_f1))
print("RF      | Recall: {} | Precision: {} | F1: {}".format(rf_recall, rf_precision, rf_f1))

NOTE: I can read in the data just fine...I can print it, and do some pandas calculations with it. It is once I use sklearn's vectorizer that I run into problems.


As this is Code review I'll provide general (but important) optimizations:

  • stop_words = [...]. To obtain a fast membership testing it should be defined as set object (not as list).
    Here's time performance comparison:

    In [262]: %timeit 'here' in stop_words                                                                                       
    1.83 µs ± 77.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
    In [263]: stop_words_set = set(stop_words)                                                                                   
    In [264]: %timeit 'here' in stop_words_set                                                                                   
    33.5 ns ± 1.97 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

    33.5 nanoseconds against 1.83 microseconds is about 54 times faster.

  • remove_nums function.
    To prevent multiple regexp pattern generation/compilation - prepare precompiled regex pattern at once with re.complie function. Also, use + quantifier to perform substitution for 2 and more digit occurrences at once:

    def remove_nums(arr): 
        pattern = re.compile(r'\d+')   
        # Return the array with numbers removed
        return [pattern.sub('', i) for i in arr]
  • get_words function.
    The whole sequence of subsequent traversals

    dirty_tokens = (data for data in no_integers if data not in stop_words)
    # Ensure it is not empty
    tokens = (data for data in dirty_tokens if data.strip())
    # Ensure there is more than 1 character to make up the word
    tokens = (data for data in tokens if len(data) > 1)

    can be effectively reduced to a single one generator expression:

    tokens = (data for data in no_integers 
              if data not in stop_words and len(data.strip()) > 1)
  • \$\begingroup\$ Thanks for the suggestions. Unfortunately, this doesn't help the issue with the todense() matrix being too large, but I appreciate your input. I will implement these suggestions. \$\endgroup\$
    – wundermahn
    Nov 18 '19 at 19:33
  • \$\begingroup\$ @JerryM., Code review is not actually about fixing/trobleshooting \$\endgroup\$ Nov 18 '19 at 19:37
  • \$\begingroup\$ I've used codereview before :). But could codereview is, as far as I understand it, a way to make your code more efficient. If there is a more efficient method for creating the vector space I need other than doing it the way I currently am, that meets the criteria, no? \$\endgroup\$
    – wundermahn
    Nov 18 '19 at 22:27
  • \$\begingroup\$ @JerryM. I would say that your case is specific and balancing between "fixing" and "optmizing" for now. But if, let's say the assumed workload has grown and the load is established as * +1M tweets per action*, then you'd have a constant problem/error that needs to be fixed before review. \$\endgroup\$ Nov 19 '19 at 6:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.