I have a
pandas.Dataframe with a single (new) record and the following function:
self.encoders # is a dict and stores a sklearn LabelEncoder() for each column self.fillNewLabels # is a pd.Series and has an entry for each column with the most frequent element def transform(self, data): for column, encoder in self.encoders.items(): fittedLabels = encoder.classes_ replacementForUnseen = self.fillNewLabels[column] data[column] = data[column].astype(str) # deliberately call str as found no better way to handle categoricals directly data.loc[~data[column].isin(fittedLabels), column] = str(replacementForUnseen) data.loc[:, column] = encoder.transform(data[column]) return data
For around 100 columns this step takes around 3 seconds for a single row of new data. What can I do to speed it up?
A nice picture from profiling results can be found here: http://imgur.com/a/K08GN