5
\$\begingroup\$

Using the wine quality dataset, I'm attempting to perform a simple KNN classification (w/ a scaler, and the classifier in a pipeline). It works, but I've never used cross_val_scores this way and I wanted to be sure that there isn't a better way. For me, it became a little less straightforward using the cross_val_scores inside the pipeline, including the n range

I've included my entire code, as it's fairly short.

import pandas as pd

import pylab as pl

from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline, Pipeline, FeatureUnion
from sklearn.cross_validation import cross_val_score

X = pd.read_csv("https://s3.amazonaws.com/demo-datasets/wine.csv")
y = X.pop('high_quality')

le = LabelEncoder()
X.color = le.fit_transform(X.color)

results = []

for n in range(1, 50, 2):
   pipe = make_pipeline(StandardScaler(),
   KNeighborsClassifier(n_neighbors=n))
   c_val = cross_val_score(pipe, X, y, cv=5, scoring='accuracy').mean()
   results.append([n, c_val])


print results

results = pd.DataFrame(results, columns=["n", "accuracy"])

pl.plot(results.n, results.accuracy)
pl.title("Accuracy with Increasing K")
pl.show()
\$\endgroup\$
  • \$\begingroup\$ You're asking if you're implementing it correctly, which to me sounds like you don't know if it works, or it doesn't seem to work. Is your real question "it works, but is there a better way" or is it more like "it seems to not work, how should I do it"? The first is on topic, the second is not. \$\endgroup\$ – Dannnno Jul 23 '16 at 0:43
  • \$\begingroup\$ I see, sorry if it was ambiguous. My question is, it works, and I get the score for each 'n', yet i'm wondering if this is improper use of cross_val_score? I believe it is correct, but i haven't used it in this fashion before. \$\endgroup\$ – chrymxbrwn Jul 23 '16 at 0:58
1
\$\begingroup\$

Your usage (and your code) is actually fine, which means there isn't much to review.

The only tip I would give is that having only the mean of the cross validation scores is not enough to determine if your model did well. Imagine, for instance, that you have 4 cv that gave the following accuracy scores : [0.01, 0.90, 0.90, 0.90] / 4 = 0.677 vs another set of hyper-parameters that gave [0.77, 0.79, 0.76, 0.80] / 4 = 0.78. That's a fairly crazy example, because such a big gap in the first cv scores is unrealistic, but it's only an example. The mean isn't very robust to outliers, that's what I mean. In order to have more information, it's great to also keep tracking of the standard deviation. You can also plot it using errorbar in matplotlib.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.