2
\$\begingroup\$

I have a function that finds the most frequent level of string and factor variables. This is used in a data.table aggregation on Big Data, for non-numeric columns.

Right now the overall aggregation runs extremely slow. I'm trying to make improvements in other aspects as well (reducing observations and columns, etc) and I would like to see if there's a want to improve this function:

MaxTable <- function(InVec, mult = FALSE) {
  if (!is.factor(InVec)) InVec <- factor(InVec)
  A <- tabulate(InVec)
  if (isTRUE(mult)) {
    levels(InVec)[A == max(A)]
  }
  else levels(InVec)[which.max(A)]
} 
\$\endgroup\$
1
\$\begingroup\$

Your function isn't very complicated so I don't know how much more efficient you can get. However, maybe removing the if and else states may help. How about something like this?

Code

# Sample Data
dat <- c("a", "a", "b", "c", "d","a", "a", "b", "c", "d","a", "a", "b", "c", "d","a", "a", "b", "c", "d","a", "a", "b", "c", "d")

# Mode function
MaxTable <- function(x){
     dd <- unique(x)
     dd[which.max(tabulate(match(x,dd)))]
}

Output

> MaxTable(dat)

[1] "a"

How many variables are you trying to process? You may want to think about writing a function that splits up the number of obs. for parallel processing, tabulate the mode for each section, merge each section, and then do which.max() . Although, I'm not sure if you are reaching the limits of R. Also, maybe a database such as mysql would be another option.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.