I am trying to generate predictions using the function below in combination with sapply. My actual datasets are very large, I am attempting to make 1.5 million predictions with this function and it is currently taking about 10 seconds per prediction which is... prohibitive to say the least.
mean_rating <- function(df){
#mean_rating <- function(query_user,query_movie){
user<-df$user
movie<-df$movie
u_row<-which(U_lookup == user)[1]
m_row<-which(M_lookup==movie)
knn_match<- knn_txt[u_row,]
knn_match<-as.numeric(unlist(knn_match))
dfm_mov<- dfm[,m_row]
dfm_test<- dfm_mov[knn_match]
c<-mean(dfm_test[dfm_test!=0])
c1<-mean(dfm_mov[dfm_mov!=0])
ifelse(c!="NaN",c,c1)
}
test<-sapply(1:nrow(probe),
function(x) mean_rating(probe[x,]))
The inputs to this function include the following. Sparse matrix dfm
library(Matrix)
dput(dfm)
new("dgTMatrix"
, i = c(0L, 1L, 2L, 4L, 5L, 6L, 8L, 0L, 1L, 2L, 3L, 4L, 6L, 7L, 8L,
0L, 2L, 3L, 6L, 7L, 8L, 1L, 2L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L,
3L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L,
9L, 0L, 1L, 3L, 4L, 6L, 7L, 8L, 9L, 0L, 2L, 3L, 5L, 6L, 7L, 9L,
0L, 1L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L,
7L, 9L)
, j = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L,
2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 4L, 4L,
4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L,
5L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L,
8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L,
9L, 9L)
, Dim = c(10L, 10L)
, Dimnames = list(NULL, NULL)
, x = c(4, 3, 1, 2, 3, 1, 2, 1, 3, 3, 2, 3, 3, 3, 4, 2, 1, 2, 3, 2,
1, 4, 1, 2, 2, 3, 2, 3, 4, 1, 4, 1, 3, 4, 3, 2, 2, 2, 4, 1, 2,
2, 1, 2, 3, 1, 1, 1, 4, 1, 1, 2, 1, 1, 1, 4, 3, 3, 2, 1, 2, 2,
1, 1, 3, 3, 4, 1, 2, 4, 2, 4, 1, 2, 2, 3, 4, 2, 1, 2, 4)
, factors = list()
)
My probe dataset, these are the user movie combinations I am trying to predict for.
dput(probe)
structure(list(X = c(1145185L, 951920L, 1137277L, 180365L, 353195L
), movie = c(1L, 100L, 10000L, 10002L, 10004L), user = c(10L,
1000004L, 1000033L, 1000035L, 1000053L), Rating = c(4L, 4L, 3L,
5L, 4L)), .Names = c("X", "movie", "user", "Rating"), row.names = c("1145185",
"951920", "1137277", "180365", "353195"), class = "data.frame")
My U_lookup table, this is where I convert from the actual id of a user to the row number of the matrix they are in.
dput(U_lookup[1:10,])
c(10L, 100000L, 1000004L, 1000027L, 1000033L, 1000035L, 1000038L,
1000051L, 1000053L, 1000057L)
My M_lookup table, this is where I convert from the actual id of a movie to the row number of the matrix they are in.
dput(M_lookup[1:10,])
c(1L, 10L, 100L, 1000L, 10000L, 10001L, 10002L, 10003L, 10004L,
10005L)
knn_txt is where I'm storing all the users nearest neighbors. Dfm and knn_text would have the same user in every row. I.e., row 1 in dfm would equate to the same user that is in row 1 of knn_text.
dput(knn_txt)
structure(list(V1 = c(1, 2, 1, 7, 5, 3, 2, 9, 1, 3), V2 = c(7,
9, 5, 5, 3, 4, 2, 7, 4, 9), V3 = c(2, 9, 6, 3, 8, 9, 4, 7, 1,
6)), .Names = c("V1", "V2", "V3"), row.names = c(NA, -10L), class = "data.frame")
Edit: As I have done some more testing myself it does seem like a lot of the time savings could come from more efficiently subsetting my large sparse matrix, dfm. I am planning on trying some different formats of the matrix, in addition to maybe looking at looping rather than using sapply (just because then I could say for user i, if user i=i-1 then I don't need to re-subset the matrix).
sampleor something similar. \$\endgroup\$