0
\$\begingroup\$

I am trying to generate predictions using the function below in combination with sapply. My actual datasets are very large, I am attempting to make 1.5 million predictions with this function and it is currently taking about 10 seconds per prediction which is... prohibitive to say the least.

mean_rating <- function(df){
  #mean_rating <- function(query_user,query_movie){
  user<-df$user
  movie<-df$movie
  u_row<-which(U_lookup == user)[1]
  m_row<-which(M_lookup==movie)

  knn_match<- knn_txt[u_row,]

  knn_match<-as.numeric(unlist(knn_match))

  dfm_mov<- dfm[,m_row]

  dfm_test<- dfm_mov[knn_match]

  c<-mean(dfm_test[dfm_test!=0])
  c1<-mean(dfm_mov[dfm_mov!=0])
  ifelse(c!="NaN",c,c1)
}
test<-sapply(1:nrow(probe),
             function(x) mean_rating(probe[x,]))

The inputs to this function include the following. Sparse matrix dfm

library(Matrix)
    dput(dfm)
new("dgTMatrix"
    , i = c(0L, 1L, 2L, 4L, 5L, 6L, 8L, 0L, 1L, 2L, 3L, 4L, 6L, 7L, 8L, 
0L, 2L, 3L, 6L, 7L, 8L, 1L, 2L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 
3L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 
9L, 0L, 1L, 3L, 4L, 6L, 7L, 8L, 9L, 0L, 2L, 3L, 5L, 6L, 7L, 9L, 
0L, 1L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 
7L, 9L)
    , j = c(0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
2L, 2L, 2L, 2L, 2L, 2L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 6L, 7L, 7L, 7L, 7L, 7L, 7L, 7L, 
8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 8L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 
9L, 9L)
    , Dim = c(10L, 10L)
    , Dimnames = list(NULL, NULL)
    , x = c(4, 3, 1, 2, 3, 1, 2, 1, 3, 3, 2, 3, 3, 3, 4, 2, 1, 2, 3, 2, 
1, 4, 1, 2, 2, 3, 2, 3, 4, 1, 4, 1, 3, 4, 3, 2, 2, 2, 4, 1, 2, 
2, 1, 2, 3, 1, 1, 1, 4, 1, 1, 2, 1, 1, 1, 4, 3, 3, 2, 1, 2, 2, 
1, 1, 3, 3, 4, 1, 2, 4, 2, 4, 1, 2, 2, 3, 4, 2, 1, 2, 4)
    , factors = list()
)

My probe dataset, these are the user movie combinations I am trying to predict for.

dput(probe)
structure(list(X = c(1145185L, 951920L, 1137277L, 180365L, 353195L
), movie = c(1L, 100L, 10000L, 10002L, 10004L), user = c(10L, 
1000004L, 1000033L, 1000035L, 1000053L), Rating = c(4L, 4L, 3L, 
5L, 4L)), .Names = c("X", "movie", "user", "Rating"), row.names = c("1145185", 
"951920", "1137277", "180365", "353195"), class = "data.frame")

My U_lookup table, this is where I convert from the actual id of a user to the row number of the matrix they are in.

dput(U_lookup[1:10,])
c(10L, 100000L, 1000004L, 1000027L, 1000033L, 1000035L, 1000038L, 
1000051L, 1000053L, 1000057L)

My M_lookup table, this is where I convert from the actual id of a movie to the row number of the matrix they are in.

dput(M_lookup[1:10,])
c(1L, 10L, 100L, 1000L, 10000L, 10001L, 10002L, 10003L, 10004L, 
10005L)

knn_txt is where I'm storing all the users nearest neighbors. Dfm and knn_text would have the same user in every row. I.e., row 1 in dfm would equate to the same user that is in row 1 of knn_text.

dput(knn_txt)
structure(list(V1 = c(1, 2, 1, 7, 5, 3, 2, 9, 1, 3), V2 = c(7, 
9, 5, 5, 3, 4, 2, 7, 4, 9), V3 = c(2, 9, 6, 3, 8, 9, 4, 7, 1, 
6)), .Names = c("V1", "V2", "V3"), row.names = c(NA, -10L), class = "data.frame")

Edit: As I have done some more testing myself it does seem like a lot of the time savings could come from more efficiently subsetting my large sparse matrix, dfm. I am planning on trying some different formats of the matrix, in addition to maybe looking at looping rather than using sapply (just because then I could say for user i, if user i=i-1 then I don't need to re-subset the matrix).

\$\endgroup\$
  • \$\begingroup\$ the code can not bee run + create larger data using sample or something similar. \$\endgroup\$ – minem Jul 20 '18 at 11:13
  • \$\begingroup\$ I apologize, there was a syntax problem in this example unrelated to my actual problem which I have corrected. I am happy to make a larger dataset, but my understanding was that it should be a MWE. Is there something specific you need from a larger dataset? \$\endgroup\$ – user2355903 Jul 20 '18 at 13:45
  • \$\begingroup\$ This being Code Review (not Stack Overflow), we prefer to see as much code and background information as possible to fully understand your situation so that we can give you the best advice possible. \$\endgroup\$ – 200_success Jul 20 '18 at 13:59
  • \$\begingroup\$ Oh ok, my mistake. The problem I run into is all of my data has to be analogous to each other, i.e. a user movie combo in the probe set has to appear in all the other data sets so that the function can use all of them. Using sample therefore is out of the question because it has a high likelihood of sampling data from each set that is for different movie user ids. I can try to expand my example if that is what people need, but would you be able to tell me if that's an acceptable edit to make to a question or should it be posted as a new question? \$\endgroup\$ – user2355903 Jul 20 '18 at 14:03
  • \$\begingroup\$ No answer has been posted yet, so you are free to improve the code in the question. Also, you are encouraged to add example data at any time. \$\endgroup\$ – 200_success Jul 20 '18 at 14:53
0
\$\begingroup\$

So after a lot of testing I found something that significantly reduces calculation time. Converting the format of my dfm to a different sparse format saves significant time when subsetting my main matrix (dfm).

dfm<-as(dfm,"dgCMatrix")

This took my calculation time from approximately 11-12 seconds per line, to .2 seconds per line. As I understand it, this is because dgCMatrix format is optimized for subsetting by columns/column operations which is the first subsetting operation in my function. If you were wanting to subset by rows the optimal format would be RsparseMatrix, optimized for rows.

dfm<-as(dfm,"RsparseMatrix")
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.