I have a pandas.Dataframe with a single (new) record and the following function:

self.encoders # is a dict and stores a sklearn LabelEncoder() for each column
self.fillNewLabels # is a pd.Series and has an entry for each column with the most frequent element

def transform(self, data):
    for column, encoder in self.encoders.items():
        fittedLabels = encoder.classes_
        replacementForUnseen = self.fillNewLabels[column]
        data[column] = data[column].astype(str) # deliberately call str as found no better way to handle categoricals directly
        data.loc[~data[column].isin(fittedLabels), column] = str(replacementForUnseen)
        data.loc[:, column] = encoder.transform(data[column])
    return data

For around 100 columns this step takes around 3 seconds for a single row of new data. What can I do to speed it up?


A nice picture from profiling results can be found here: http://imgur.com/a/K08GN


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.