Cross validation of gradient boosting machines

Question

I am fairly new to Python. I implemented a short cross-validation tool for gradient boosting methods.

import numpy as np

from sklearn.metrics import roc_auc_score as auc
from sklearn import cross_validation
from time import time

def heldout_auc(model, X_test, y_test):
    score = np.zeros((model.get_params()["n_estimators"],), dtype=np.float64)
    for i, y_pred in enumerate(model.staged_decision_function(X_test)):
        score[i] = auc(y_test, y_pred)
    return score

def cv_boost_estimate(X,y,model,n_folds=3):
    cv = cross_validation.StratifiedKFold(y, n_folds=n_folds, shuffle=True, random_state=11)
    val_scores = np.zeros((model.get_params()["n_estimators"],), dtype=np.float64)
    t = time()
    i = 0
    for train, test in cv:
        i = i + 1
        print('FOLD : ' + str(i) + '-' + str(n_folds))
        model.fit(X.iloc[train,], y.iloc[train])
        val_scores += heldout_auc(model, X.iloc[test,], y.iloc[test])
    val_scores /= n_folds
    return val_scores,(time()-t)

Then I can look for the optimal number of trees with the following:

print('AUC : ' + str(max(auc)) + ' - index : ' + str(auc.tolist().index(max(auc))))

Everything is working, but the syntax feels uneasy and "not Pythonic". Does someone have improvements to suggest?

Answer 1

You didn't use enumerate for your cv loop, I assume you tried this and found it didn't work:

for i, train, test in enumerate(cv):

To understand this, you need to know what enumerate is actually doing. It's wrapping each element of cv in a 2 item tuple, with index as the first item and the element from the iterable in the other. Basically it's using this structure: (i, (train, test)).

Fortunately this means you can get the result you need with just one modification:

for i, (train, test) in enumerate(cv):

Now it can extract all 3 values correctly without error. Even though you want i to start at 1, I find this to be more clear and readable than a manually incrementing value. You can just make the allowance to push the i value up by 1.

Also you're manually concatenating strings when you should use str.format. It allows you to insert values into strings, this is how you'd use it:

    print('FOLD : {}-{}'.format(i, n_folds))

format will replace any {} with the parameters passed to it. It will also automatically try to call str on any objects that can be turned into strings so you no longer need those manual calls.

Lastly, you don't need to wrap your time expression in brackets, it returns fine without it and it's more Pythonic to leave out the brackets.

return val_scores, time() - t

You should read the Python style guide, it has a lot of info about Pythonic style coding. You have it mostly correct, but some lines are a bit long and sometimes you should put more whitespace in (eg. time()-t -> time() - t).