MemoryError for a small-ish dataset with RandomForestClassifier()

Question

I have a not-so-big dataset having 100,000 rows and 6k columns and I'm using the following code to fit a Random Forest to it:

# Read csv and create dummy variables
Sessions = pd.read_csv('filename.csv')
cols_to_transform = ['a','b','c','d','e','f']
Sessions = pd.get_dummies( Sessions, columns = cols_to_transform )

# Create train and test set
Sessions['is_train'] = np.random.uniform(0, 1, len(Sessions)) <= .85
train, test = Sessions[Sessions['is_train']==True], Sessions[Sessions['is_train']==False]
del Sessions

y = pd.factorize(train['targetname'])[0]
features = train.columns[:4].append(train.columns[6:])

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
clf.fit(train[features], y)

This gives me MemoryError. My RAM size is 32GB which should be plenty for this size of data. Why am I running into this error then? (There's nothing else running on the server, Python is the only application)

Here are the outputs of memory_usage() method for the variables test,train and Sessions:

Sessions.memory_usage(index=True, deep=True).sum()/1024/1024
603L
train.memory_usage(index=True, deep=True).sum()/1024/1024
513L
test.memory_usage(index=True, deep=True).sum()/1024/1024
90L

Here's the traceback:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-8-bc5dc9fc8fd3> in <module>()
      1 from sklearn.ensemble import RandomForestClassifier
      2 clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
----> 3 clf.fit(train[features], y)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in fit(self, X, y, sample_weight)
    324                     t, self, X, y, sample_weight, i, len(trees),
    325                     verbose=self.verbose, class_weight=self.class_weight)
--> 326                 for i, t in enumerate(trees))
    327 
    328             # Collect newly grown trees

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)
    756             # was dispatched. In particular this covers the edge
    757             # case of Parallel used with an exhausted iterator.
--> 758             while self.dispatch_one_batch(iterator):
    759                 self._iterating = True
    760             else:

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
    606                 return False
    607             else:
--> 608                 self._dispatch(tasks)
    609                 return True
    610 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in _dispatch(self, batch)
    569         dispatch_timestamp = time.time()
    570         cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 571         job = self._backend.apply_async(batch, callback=cb)
    572         self._jobs.append(job)
    573 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in apply_async(self, func, callback)
    107     def apply_async(self, func, callback=None):
    108         """Schedule a func to be run"""
--> 109         result = ImmediateResult(func)
    110         if callback:
    111             callback(result)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in __init__(self, batch)
    324         # Don't delay the application, to avoid keeping the input
    325         # arguments in memory
--> 326         self.results = batch()
    327 
    328     def get(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees, verbose, class_weight)
    118             curr_sample_weight *= compute_sample_weight('balanced', y, indices)
    119 
--> 120         tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False)
    121     else:
    122         tree.fit(X, y, sample_weight=sample_weight, check_input=False)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    737             sample_weight=sample_weight,
    738             check_input=check_input,
--> 739             X_idx_sorted=X_idx_sorted)
    740         return self
    741 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    348                                            self.min_impurity_split)
    349 
--> 350         builder.build(self.tree_, X, y, sample_weight, X_idx_sorted)
    351 
    352         if self.n_outputs_ == 1:

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:5002)()

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:4829)()

MemoryError: 

How do I fix it?

EDIT: This code works for a small dataset. I subset my dataset to contain 1000 rows and the get_dummies() then gave me 670 columns, and the fit() method works within a second.

Answer 1

The traceback shows that the MemoryError is raised by the DepthFirstTreeBuilder.build method, which is building a decision tree for the random forest classifier.

A look at the documentation for the sklearn.ensemble.RandomForestClassifier class reveals that it constructs n_estimators (default 10) decision trees, and with the default values for the max_depth (infinite) and min_samples_split (2) options, those trees end up including one leaf node for every row in the data set.

Consider using the max_depth and/or min_samples_split options to reduce the size of these decision trees.

(Originally posted as a comment, but copied and expanded so that the question has an answer.)