Box and Cox process

Question

The code obtains Box and Cox transformed data. The original data is a Pandas DataFrame and the process uses this DataFrame.

I am trying to speed up the following code. If you can elaborate in faster way of doing this it would be very helpful.

import numpy as np
import pandas as pd
from numpy.random import randn
from scipy import stats

np.random.seed(1)
df = pd.DataFrame(np.random.randint(1, 100,size=(100, 4)))

def st_bc(data):
    data_bc = pd.DataFrame()
    for column in list(data):
            data_bc[column], lam = stats.boxcox(data[column])
    return data_bc

st_bc(data = df)

Answer 1

The code and what you're doing are very simple, there's probably not a whole lot you can do here. Here are some thoughts:

• Styling: consistent indentation and following PEP8, please.

• If the original data isn't important to keep around, consider overwriting the data in-place. So, data[column], _ = stats.boxcox(data[column]). This allows Pandas to forget the old data after the transformation has been applied to it, saving a little memory. Probably no noticeable time savings from this, though.

• I note from the boxcox documentation that, if lambda isn't given, it's computed to maximize the log-likelihood of the result. I don't know anything about this transform, but depending on how that parameter is being computed and what it means, there may be a way to pre-compute it for your given data more efficiently and thus save that computation time.

Unfortunately, when your code is basically one line long (the boxcox call is the only thing that's substantial here), there's only so much you can do to make it faster, especially if that one line is already calling a high-performance library like Scipy. If that's not fast enough, then you'll need to break the underlying algorithm apart and look for ways that it can be optimized when you apply stronger assumptions, e.g. if you assume your data is all non-negative rather than checking it at runtime, or can find a more efficient way to compute an optimal lambda.