1
\$\begingroup\$

I have a not-so-big dataset having 100,000 rows and 6k columns and I'm using the following code to fit a Random Forest to it:

# Read csv and create dummy variables
Sessions = pd.read_csv('filename.csv')
cols_to_transform = ['a','b','c','d','e','f']
Sessions = pd.get_dummies( Sessions, columns = cols_to_transform )

# Create train and test set
Sessions['is_train'] = np.random.uniform(0, 1, len(Sessions)) <= .85
train, test = Sessions[Sessions['is_train']==True], Sessions[Sessions['is_train']==False]
del Sessions

y = pd.factorize(train['targetname'])[0]
features = train.columns[:4].append(train.columns[6:])

from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
clf.fit(train[features], y)

This gives me MemoryError. My RAM size is 32GB which should be plenty for this size of data. Why am I running into this error then? (There's nothing else running on the server, Python is the only application)

Here are the outputs of memory_usage() method for the variables test,train and Sessions:

Sessions.memory_usage(index=True, deep=True).sum()/1024/1024
603L
train.memory_usage(index=True, deep=True).sum()/1024/1024
513L
test.memory_usage(index=True, deep=True).sum()/1024/1024
90L

Here's the traceback:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-8-bc5dc9fc8fd3> in <module>()
      1 from sklearn.ensemble import RandomForestClassifier
      2 clf = RandomForestClassifier(n_jobs=1,n_estimators=100)
----> 3 clf.fit(train[features], y)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in fit(self, X, y, sample_weight)
    324                     t, self, X, y, sample_weight, i, len(trees),
    325                     verbose=self.verbose, class_weight=self.class_weight)
--> 326                 for i, t in enumerate(trees))
    327 
    328             # Collect newly grown trees

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self, iterable)
    756             # was dispatched. In particular this covers the edge
    757             # case of Parallel used with an exhausted iterator.
--> 758             while self.dispatch_one_batch(iterator):
    759                 self._iterating = True
    760             else:

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in dispatch_one_batch(self, iterator)
    606                 return False
    607             else:
--> 608                 self._dispatch(tasks)
    609                 return True
    610 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in _dispatch(self, batch)
    569         dispatch_timestamp = time.time()
    570         cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 571         job = self._backend.apply_async(batch, callback=cb)
    572         self._jobs.append(job)
    573 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in apply_async(self, func, callback)
    107     def apply_async(self, func, callback=None):
    108         """Schedule a func to be run"""
--> 109         result = ImmediateResult(func)
    110         if callback:
    111             callback(result)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.pyc in __init__(self, batch)
    324         # Don't delay the application, to avoid keeping the input
    325         # arguments in memory
--> 326         self.results = batch()
    327 
    328     def get(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.pyc in __call__(self)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/ensemble/forest.pyc in _parallel_build_trees(tree, forest, X, y, sample_weight, tree_idx, n_trees, verbose, class_weight)
    118             curr_sample_weight *= compute_sample_weight('balanced', y, indices)
    119 
--> 120         tree.fit(X, y, sample_weight=curr_sample_weight, check_input=False)
    121     else:
    122         tree.fit(X, y, sample_weight=sample_weight, check_input=False)

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    737             sample_weight=sample_weight,
    738             check_input=check_input,
--> 739             X_idx_sorted=X_idx_sorted)
    740         return self
    741 

/home/prateek/anaconda2/lib/python2.7/site-packages/sklearn/tree/tree.pyc in fit(self, X, y, sample_weight, check_input, X_idx_sorted)
    348                                            self.min_impurity_split)
    349 
--> 350         builder.build(self.tree_, X, y, sample_weight, X_idx_sorted)
    351 
    352         if self.n_outputs_ == 1:

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:5002)()

sklearn/tree/_tree.pyx in sklearn.tree._tree.DepthFirstTreeBuilder.build (sklearn/tree/_tree.c:4829)()

MemoryError: 

How do I fix it?

EDIT: This code works for a small dataset. I subset my dataset to contain 1000 rows and the get_dummies() then gave me 670 columns, and the fit() method works within a second.

\$\endgroup\$
  • 1
    \$\begingroup\$ RandomForestClassifier needs to make n_estimators (default 10) decision trees, and since you don't set max_depth or min_samples_split, those trees will end up including all the rows as leaf nodes. Consider using these parameters to reduce the size of the decision trees. \$\endgroup\$ – Gareth Rees Jul 11 '17 at 19:35
3
\$\begingroup\$

The traceback shows that the MemoryError is raised by the DepthFirstTreeBuilder.build method, which is building a decision tree for the random forest classifier.

A look at the documentation for the sklearn.ensemble.RandomForestClassifier class reveals that it constructs n_estimators (default 10) decision trees, and with the default values for the max_depth (infinite) and min_samples_split (2) options, those trees end up including one leaf node for every row in the data set.

Consider using the max_depth and/or min_samples_split options to reduce the size of these decision trees.

(Originally posted as a comment, but copied and expanded so that the question has an answer.)

\$\endgroup\$
  • \$\begingroup\$ I'm having similar issue with multinomialNB and GaussianNB. I am working on a spam classification problem. SVM, logistic regression and decision trees are not running into memory error but NB is running into memory error. Moreover, the size of dataset is just 5500(approx). I'm running all algorithms in one code. Without NB, the code runs fine. Is there an explanation? \$\endgroup\$ – Eswar Aug 10 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.