I have a not-so-big dataset having 100,000 rows and 6k columns and I'm using the following code to fit a Random Forest to it:

# Read csv and create dummy variables
Sessions = pd.read_csv('filename.csv')
cols_to_transform = ['a','b','c','d','e','f']
Sessions = pd.get_dummies( Sessions, columns = cols_to_transform )

# Create train and test set
Sessions['is_train'] = np.random.uniform(0, 1, len(Sessions)) <= .85
train, test = Sessions[Sessions['is_train']==True], Sessions[Sessions['is_train']==False]
del Sessions

y = pd.factorize(train['targetname'])[0]
features = train.columns[:4].append(train.columns[6:])

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
clf.fit(train[features], y)

This gives me MemoryError. My RAM size is 32GB which should be plenty for this size of data. Why am I running into this error then? (There's nothing else running on the server, Python is the only application)

Here are the outputs of memory_usage() method for the variables test,train and Sessions:

Sessions.memory_usage(index=True, deep=True).sum()/1024/1024
train.memory_usage(index=True, deep=True).sum()/1024/1024
test.memory_usage(index=True, deep=True).sum()/1024/1024

Here's the traceback:

MemoryError                               Traceback (most recent call last)
<ipython-input-8-bc5dc9fc8fd3> in <module>()
      1 from sklearn.ensemble import RandomForestClassifier
      2 clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
----> 3 clf.fit(train[features], y)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in fit(self, X, y, sample_weight)
    324                     t, self, X, y, sample_weight, i, len(trees),
    325                     verbose=self.verbose, class_weight=self.class_weight)
--> 326                 for i, t in enumerate(trees))
    328             # Collect newly grown trees

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)
    756             # was dispatched. In particular this covers the edge
    757             # case of Parallel used with an exhausted iterator.
--> 758             while self.dispatch_one_batch(iterator):
    759                 self._iterating = True
    760             else:

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
    606                 return False
    607             else:
--> 608                 self._dispatch(tasks)
    609                 return True

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in _dispatch(self, batch)
    569         dispatch_timestamp = time.time()
    570         cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 571         job = self._backend.apply_async(batch, callback=cb)
    572         self._jobs.append(job)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in apply_async(self, func, callback)
    107     def apply_async(self, func, callback=None):
    108         """Schedule a func to be run"""
--> 109         result = ImmediateResult(func)
    110         if callback:
    111             callback(result)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in __init__(self, batch)
    324         # Don't delay the application, to avoid keeping the input
    325         # arguments in memory
--> 326         self.results = batch()
    328     def get(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self)
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    133     def __len__(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees, verbose, class_weight)
    118             curr_sample_weight *= compute_sample_weight('balanced', y, indices)
--> 120         tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False)
    121     else:
    122         tree.fit(X, y, sample_weight=sample_weight, check_input=False)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    737             sample_weight=sample_weight,
    738             check_input=check_input,
--> 739             X_idx_sorted=X_idx_sorted)
    740         return self

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    348                                            self.min_impurity_split)
--> 350         builder.build(self.tree_, X, y, sample_weight, X_idx_sorted)
    352         if self.n_outputs_ == 1:

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:5002)()

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:4829)()


How do I fix it?

EDIT: This code works for a small dataset. I subset my dataset to contain 1000 rows and the get_dummies() then gave me 670 columns, and the fit() method works within a second.

  • 1
    \$\begingroup\$ RandomForestClassifier needs to make n_estimators (default 10) decision trees, and since you don't set max_depth or min_samples_split, those trees will end up including all the rows as leaf nodes. Consider using these parameters to reduce the size of the decision trees. \$\endgroup\$ Commented Jul 11, 2017 at 19:35

1 Answer 1


The traceback shows that the MemoryError is raised by the DepthFirstTreeBuilder.build method, which is building a decision tree for the random forest classifier.

A look at the documentation for the sklearn.ensemble.RandomForestClassifier class reveals that it constructs n_estimators (default 10) decision trees, and with the default values for the max_depth (infinite) and min_samples_split (2) options, those trees end up including one leaf node for every row in the data set.

Consider using the max_depth and/or min_samples_split options to reduce the size of these decision trees.

(Originally posted as a comment, but copied and expanded so that the question has an answer.)

  • \$\begingroup\$ I'm having similar issue with multinomialNB and GaussianNB. I am working on a spam classification problem. SVM, logistic regression and decision trees are not running into memory error but NB is running into memory error. Moreover, the size of dataset is just 5500(approx). I'm running all algorithms in one code. Without NB, the code runs fine. Is there an explanation? \$\endgroup\$
    – Eswar
    Commented Aug 10, 2019 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.