SKlearn automate data pre treatment

Question

I want to make a simple wrapper for sklearn models. The idea is that the wrapper automatically takes care of factors (columns of type "object") replacing them with the average value of the target, while preserving the syntax of sklearn models.

If the factor is too scarce, it is replaced by the overall average value of the target. It seems simple but issues arise when a factor is in the test set and was not in the train set. I came up with the following solution, that looks really awkward to me.

class ModelEmbedder :

    def __init__(self, model, rare_threshold) :
        self.model = model
        self.means = {}
        self.rare_threshold = rare_threshold
        self.train = None
        self.origin_train = None
        self.average = 0


    def fit(self,train,target):
        self.origin_train = train.copy().fillna(-1)
        self.train = train.copy()
        self.train = self.train.fillna(-1)
        self.train['target'] = target
        self.average = target.mean()

        for feat in train.columns:
            if feat != 'target' :
                if self.train[feat].dtype=='object' :
                    self.train.loc[self.train[feat].value_counts()[self.train[feat]].values <  self.rare_threshold, feat] = "RARE"
                    self.origin_train.loc[self.origin_train[feat].value_counts()[self.origin_train[feat]].values <  self.rare_threshold, feat] = "RARE"
                    self.means[feat] = self.train.groupby([feat])['target'].mean()
                    self.means[feat]["RARE"] = self.average

                    self.train[feat] = self.train[feat].replace(self.means[feat], inplace=False)

        del self.train['target']

        self.model.fit(self.train,target)

    def _pre_treat_test(self,test) :
        test = test.copy()
        test = test.fillna(-1)

        for feat in self.origin_train.columns:
            if self.origin_train[feat].dtype=='object' :
                test.loc[self.origin_train[feat].value_counts()[self.origin_train[feat]].values  <  self.rare_threshold, feat] = "RARE"

                criterion = ~test[feat].isin(set(self.origin_train[feat]))

                test.loc[criterion,feat] = self.average


                test[feat]  = test[feat].replace(self.means[feat], inplace=False)

        return test        

    def predict_proba(self,test) :
        test = self._pre_treat_test(test)
        return self.model.predict_proba(test)

    def get_params(self, deep = True):
        return self.model.get_params(deep)

Then, every model can be wrapped:

rf = ensemble.ExtraTreesClassifier(n_jobs=7, 
    n_estimators = n_estimators, 
    random_state = 11)

rf_embedded = model_embedder.ModelEmbedder(rf,10)

and sent to a cross-validation loop or any pipeline.

Answer 1

A few observations:

• You are adding spaces before most of your : throughout, which reads weird. I'd remove this as I've never seen any other Python code that does this
• I would check out Pep8 as a way to clean up a variety of formatting things, as it is the more or less accepted Python formatting standard
  • This will show things like lacking spaces around operators, too many line breaks, etc

Also, I am not sure how you are using underscores here. Generally, underscores signify private members/methods in Python and it looks like you have quite a few places you could make your member variables private.

Also this could be a bit more clear:

self.train.loc[self.train[feat].value_counts()[self.train[feat]].values <  self.rare_threshold, feat] = "RARE"

You might consider some intermediate steps here to help with readability. Even something as simple as:

val = self.train[feat].value_counts()[self.train[feat]].values <  self.rare_threshold
self.train.loc[val, feat] = "RARE"

is more clear. Trying to read that many nested dictionary lookups is not simple.