Alternative to Python's Naive Bayes Classifier for Twitter Sentiment Mining

Question

I am doing sentiment analysis on tweets. I have code that I developed from following an online tutorial (found here) and adding in some parts myself, which looks like this:

#!/usr/bin/env python

import csv, string, HTMLParser, nltk, pickle
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

test_file = 'Dataset/SmallSample.csv'
#test_file = 'Dataset/Dataset.csv'
csv_file = csv.DictReader(open(test_file, 'rb'), delimiter=',', quotechar='"')

pos_tweets = {}
neg_tweets = {}

for line in csv_file:
    if line['Sentiment']=='1':
        pos_tweets.update({(line['SentimentText'],"positive")})
    else:
        neg_tweets.update({(line['SentimentText'],"negative")})

tweets = []
labeltweets = []
for (text, sentiment) in pos_tweets.items() + neg_tweets.items():
    text = HTMLParser.HTMLParser().unescape(text.decode('utf-8'))
    remove_punctuation_map = dict((ord(char), None) for char in string.punctuation)
    cleanedText = [e.translate(remove_punctuation_map).lower() for e in text.split() if not e.startswith(('http', '@')) ]
    shortenedText = [e for e in cleanedText if len(e) >= 3]
    tweets.append((shortenedText, sentiment))

# Produces list of all words in text of tweets (including duplicates).
def get_words_in_tweets(tweets):
    all_words = []
    for (text, sentiment) in tweets:
        all_words.extend(text)
    return all_words

def get_word_features(wordlist):
    # This line calculates the frequency distrubtion of all words in tweets
    # e.g. word "love" appears 5 times, word "dog" appears 3 times etc.
    wordlist = nltk.FreqDist(wordlist)
    word_features = wordlist.keys()
    # This prints out the list of all distinct words in the text in order
    # of their number of occurrences.
    return word_features

word_features = get_word_features(get_words_in_tweets(tweets))

def extract_features(document):
    setOfDocument = set(document)
    features = {}
    for word in word_features:
        features['contains(%s)' % word] = (word in setOfDocument)
    return features

training_set = nltk.classify.apply_features(extract_features, tweets)
classifer = nltk.NaiveBayesClassifier.train(training_set)

The code imports a large csv file and creates two dictionaries out of it, depending on whether the tweet is positive or negative. It maps these dictionaries like so:

("United won a !game today", "positive")

The for loop that follows this performs basic HTML stripping for unescaped syntax and removes all punctuation and words less than length of three, combining these into one large list, with this format:

(["United", "won", "game", "today"], "positive")

It then creates a list of just the text words (ignoring the positive/negative sentiment) and uses the nltk FreqDist method to create a frequency distribution of all words used like this:

<FreqDist:
    'this': 6,
    'car': 2,
    'concert': 2,
    'feel': 2,
    'morning': 2,
    'not': 2,
    'the': 2,
    'view': 2,
    'about': 1,
    'amazing': 1,
    ...
>

and then removing the accompanying numbers, just leaving the "word features".

The method extract_features then compares the tweet input to a list of words, like this:

Input: ['love', 'this', 'car'] 
Output:
{'contains(not)': False,
 'contains(view)': False,
 'contains(best)': False,
 'contains(excited)': False,
 'contains(morning)': False,
 'contains(about)': False,
 'contains(horrible)': False,
 'contains(like)': False,
 'contains(this)': True,
 'contains(friend)': False,
 'contains(concert)': False,
 'contains(feel)': False,
 'contains(love)': True,
 'contains(looking)': False,
 'contains(tired)': False,
 'contains(forward)': False,
 'contains(car)': True,
 'contains(the)': False,
 'contains(amazing)': False,
 'contains(enemy)': False,
 'contains(great)': False}

Finally, the line:

training_set = nltk.classify.apply_features(extract_features, tweets)

is used to build this training set of data from all provided tweets, breaking them down like this:

[({'contains(not)': False,
   ...
   'contains(this)': True,
   ...
   'contains(love)': True,
   ...
   'contains(car)': True,
   ...
   'contains(great)': False},
  'positive'),
 ({'contains(not)': False,
   'contains(view)': True,
   ...
   'contains(this)': True,
   ...
   'contains(amazing)': True,
   ...
   'contains(enemy)': False,
   'contains(great)': False},
  'positive'),
  ...]

This information is finally fed in to the trainer, like so:

classifier = nltk.NaiveBayesClassifier.train(training_set)

When I ran this on my sample dataset, it all worked perfectly, although a little inaccurately (training set only had 50 tweets). My REAL training set however has 1.5 million tweets. I'm finding that using the default trainer provided by Python is just far too slow.

Is this too large a dataset to be used with the default Python classifier? Does anybody have any suggestions or alternatives that could be used to do this operation? In all responses please bear in mind I could only accomplish this with a tutorial and am totally new to Python (am usually a Java coder).

Original SO post

Answer 1

take a look at the answers in

Is there a rule-of-thumb for how to divide a dataset into training and validation sets?

"If you have 100,000 instances, ... indeed you may choose to use less training data if your method is particularly computationally intensive)."

Good luck!