# R Function to Generate Predictions from Ratings and Save Results

I am trying to generate predictions using the function below in combination with sapply. My actual datasets are very large, I am attempting to make 1.5 million predictions with this function and it is currently taking about 10 seconds per prediction which is... prohibitive to say the least.

mean_rating <- function(df){
#mean_rating <- function(query_user,query_movie){
user<-df$user movie<-df$movie
u_row<-which(U_lookup == user)[1]
m_row<-which(M_lookup==movie)

knn_match<- knn_txt[u_row,]

knn_match<-as.numeric(unlist(knn_match))

dfm_mov<- dfm[,m_row]

dfm_test<- dfm_mov[knn_match]

c<-mean(dfm_test[dfm_test!=0])
c1<-mean(dfm_mov[dfm_mov!=0])
ifelse(c!="NaN",c,c1)
}
test<-sapply(1:nrow(probe),
function(x) mean_rating(probe[x,]))


The inputs to this function include the following. Sparse matrix dfm

library(Matrix)
dput(dfm)
new("dgTMatrix"
, i = c(0L, 1L, 2L, 4L, 5L, 6L, 8L, 0L, 1L, 2L, 3L, 4L, 6L, 7L, 8L,
0L, 2L, 3L, 6L, 7L, 8L, 1L, 2L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L,
3L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L,
9L, 0L, 1L, 3L, 4L, 6L, 7L, 8L, 9L, 0L, 2L, 3L, 5L, 6L, 7L, 9L,
0L, 1L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L,
7L, 9L)
, j = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L,
5L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L,
8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L,
9L, 9L)
, Dim = c(10L, 10L)
, Dimnames = list(NULL, NULL)
, x = c(4, 3, 1, 2, 3, 1, 2, 1, 3, 3, 2, 3, 3, 3, 4, 2, 1, 2, 3, 2,
1, 4, 1, 2, 2, 3, 2, 3, 4, 1, 4, 1, 3, 4, 3, 2, 2, 2, 4, 1, 2,
2, 1, 2, 3, 1, 1, 1, 4, 1, 1, 2, 1, 1, 1, 4, 3, 3, 2, 1, 2, 2,
1, 1, 3, 3, 4, 1, 2, 4, 2, 4, 1, 2, 2, 3, 4, 2, 1, 2, 4)
, factors = list()
)


My probe dataset, these are the user movie combinations I am trying to predict for.

dput(probe)
structure(list(X = c(1145185L, 951920L, 1137277L, 180365L, 353195L
), movie = c(1L, 100L, 10000L, 10002L, 10004L), user = c(10L,
1000004L, 1000033L, 1000035L, 1000053L), Rating = c(4L, 4L, 3L,
5L, 4L)), .Names = c("X", "movie", "user", "Rating"), row.names = c("1145185",
"951920", "1137277", "180365", "353195"), class = "data.frame")


My U_lookup table, this is where I convert from the actual id of a user to the row number of the matrix they are in.

dput(U_lookup[1:10,])
c(10L, 100000L, 1000004L, 1000027L, 1000033L, 1000035L, 1000038L,
1000051L, 1000053L, 1000057L)


My M_lookup table, this is where I convert from the actual id of a movie to the row number of the matrix they are in.

dput(M_lookup[1:10,])
c(1L, 10L, 100L, 1000L, 10000L, 10001L, 10002L, 10003L, 10004L,
10005L)


knn_txt is where I'm storing all the users nearest neighbors. Dfm and knn_text would have the same user in every row. I.e., row 1 in dfm would equate to the same user that is in row 1 of knn_text.

dput(knn_txt)
structure(list(V1 = c(1, 2, 1, 7, 5, 3, 2, 9, 1, 3), V2 = c(7,
9, 5, 5, 3, 4, 2, 7, 4, 9), V3 = c(2, 9, 6, 3, 8, 9, 4, 7, 1,
6)), .Names = c("V1", "V2", "V3"), row.names = c(NA, -10L), class = "data.frame")


Edit: As I have done some more testing myself it does seem like a lot of the time savings could come from more efficiently subsetting my large sparse matrix, dfm. I am planning on trying some different formats of the matrix, in addition to maybe looking at looping rather than using sapply (just because then I could say for user i, if user i=i-1 then I don't need to re-subset the matrix).

• the code can not bee run + create larger data using sample or something similar. – minem Jul 20 '18 at 11:13
• I apologize, there was a syntax problem in this example unrelated to my actual problem which I have corrected. I am happy to make a larger dataset, but my understanding was that it should be a MWE. Is there something specific you need from a larger dataset? – user2355903 Jul 20 '18 at 13:45
• This being Code Review (not Stack Overflow), we prefer to see as much code and background information as possible to fully understand your situation so that we can give you the best advice possible. – 200_success Jul 20 '18 at 13:59
• Oh ok, my mistake. The problem I run into is all of my data has to be analogous to each other, i.e. a user movie combo in the probe set has to appear in all the other data sets so that the function can use all of them. Using sample therefore is out of the question because it has a high likelihood of sampling data from each set that is for different movie user ids. I can try to expand my example if that is what people need, but would you be able to tell me if that's an acceptable edit to make to a question or should it be posted as a new question? – user2355903 Jul 20 '18 at 14:03
• No answer has been posted yet, so you are free to improve the code in the question. Also, you are encouraged to add example data at any time. – 200_success Jul 20 '18 at 14:53

dfm<-as(dfm,"dgCMatrix")

dfm<-as(dfm,"RsparseMatrix")