1
\$\begingroup\$

I'm trying to improve my Alternating Least Squares (ALS) model performance by testing using different model parameters, but the model performance is seemingly very low.

Since the Alternating Least Squares model is relatively different from other models under the evaluation perspective, I needed to set up my custom evaluation and cross-validation method in order for the GridSearchCV function to correctly work. To improve the performance, I tried to set up different param_grid and model to the testing function with apparently no results at all. My guess is that the custom parameters are not properly working. For reference, I looked at Jbochi's piece of code (https://gist.github.com/jbochi/2e8ddcc5939e70e5368326aa034a144e) to set up Discounted cumulative gain (DCG) for measuring my model score and the function Predefinedsplit from sklearn.model_selection (as you can see in the LeavePOutByGroup class) to set up the cv parameter.

class LeavePOutByGroup():
    def __init__(self, X, p=5, n_splits = 3):
        self.X = X
        self.p = p
        self.n_splits = n_splits
        test_fold = self.X.groupby("fullvisitorid").cumcount().apply(lambda x: int(x / p) if x < (n_splits * p) else -1)
        self.s = PredefinedSplit(test_fold)

grid_search = GridSearchCV(rec_pipeline, param_grid,
                           cv=LeavePOutByGroup(train_set, p=5, n_splits=3),
                           scoring=ndcg_scorer, verbose=1)
grid_search.fit(train_set)

I expected the mean_test_score to significantly change by changing the model parameters, instead I got almost no improvements at all. This is what I got (the float shows the mean_test_score for each set of parameters tested with the ALS model):

cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(mean_score, params)

Output:

0.012434668656604229 {'als__factors': 20, 'als__regularization': 0.001, 'matrix__confidence': 10}

0.011882640120018911 {'als__factors': 20, 'als__regularization': 0.001, 'matrix__confidence': 40}

0.01271910407436217 {'als__factors': 20, 'als__regularization': 0.0001, 'matrix__confidence': 10}
...

0.011947024379421394 {'als__factors': 100, 'als__regularization': 0.0001, 'matrix__confidence': 10}

0.010698034031201297 {'als__factors': 100, 'als__regularization': 0.0001, 'matrix__confidence': 40}

0.010091750744302954 {'als__factors': 100, 'als__regularization': 0.0001, 'matrix__confidence': 100}

My question is: is there any other approach I can try for solving the bad performance issue? What is your experience on that? Or should I just try another evaluation metric for my ALS model? If so, which one?

\$\endgroup\$
  • 2
    \$\begingroup\$ I'm not immediately convinced that this should be closed. You would definitely need to add more code for this to be reviewable. Also, when you say performance... do you mean execution speed, or numerical performance? We regularly help with the former, but the latter wouldn't be as clearly on-topic. \$\endgroup\$ – Reinderien Sep 28 '19 at 2:23