Cross validation of gradient boosting machines

I am fairly new to Python. I implemented a short cross-validation tool for gradient boosting methods.

import numpy as np

from sklearn.metrics import roc_auc_score as auc
from sklearn import cross_validation
from time import time

def heldout_auc(model, X_test, y_test):
score = np.zeros((model.get_params()["n_estimators"],), dtype=np.float64)
for i, y_pred in enumerate(model.staged_decision_function(X_test)):
score[i] = auc(y_test, y_pred)
return score

def cv_boost_estimate(X,y,model,n_folds=3):
cv = cross_validation.StratifiedKFold(y, n_folds=n_folds, shuffle=True, random_state=11)
val_scores = np.zeros((model.get_params()["n_estimators"],), dtype=np.float64)
t = time()
i = 0
for train, test in cv:
i = i + 1
print('FOLD : ' + str(i) + '-' + str(n_folds))
model.fit(X.iloc[train,], y.iloc[train])
val_scores += heldout_auc(model, X.iloc[test,], y.iloc[test])
val_scores /= n_folds
return val_scores,(time()-t)


Then I can look for the optimal number of trees with the following:

print('AUC : ' + str(max(auc)) + ' - index : ' + str(auc.tolist().index(max(auc))))


Everything is working, but the syntax feels uneasy and "not Pythonic". Does someone have improvements to suggest?

• I've commented on similar SO questions - loops like this are right at home in Python. It's numpy that encourages non-looping methods. Whether the 1st loop can be eleminated depends on what auc can handle. The 2nd looks iterative by nature. Nov 13 '15 at 17:28

You didn't use enumerate for your cv loop, I assume you tried this and found it didn't work:

for i, train, test in enumerate(cv):


To understand this, you need to know what enumerate is actually doing. It's wrapping each element of cv in a 2 item tuple, with index as the first item and the element from the iterable in the other. Basically it's using this structure: (i, (train, test)).

Fortunately this means you can get the result you need with just one modification:

for i, (train, test) in enumerate(cv):


Now it can extract all 3 values correctly without error. Even though you want i to start at 1, I find this to be more clear and readable than a manually incrementing value. You can just make the allowance to push the i value up by 1.

Also you're manually concatenating strings when you should use str.format. It allows you to insert values into strings, this is how you'd use it:

    print('FOLD : {}-{}'.format(i, n_folds))


format will replace any {} with the parameters passed to it. It will also automatically try to call str on any objects that can be turned into strings so you no longer need those manual calls.

Lastly, you don't need to wrap your time expression in brackets, it returns fine without it and it's more Pythonic to leave out the brackets.

return val_scores, time() - t


You should read the Python style guide, it has a lot of info about Pythonic style coding. You have it mostly correct, but some lines are a bit long and sometimes you should put more whitespace in (eg. time()-t -> time() - t).

• Thank you ! Why is using enumerate considered a better practice ? Nov 16 '15 at 12:15
• @RUser4512 It cuts out two lines of code (i = 0 and i += 1) but also people immediately know what enumerate means, while a manually incremented value might be only counting specific cases. Or worse, a manually incremented value should always go up but a mistake means that it's only incrementing in certain cases. Since you still need to manually add 1 anyway, it's more debateable whether it's worth it but I still prefer it. Nov 16 '15 at 12:20