0
\$\begingroup\$

In order to have a "unified" (I realize how ambitious this actually is) syntax when working on dataframes, I wrote the following functions that is a general purpose dataframe set of tools in R.

is.constant <- function(x) {return(length(unique(x))==1)}

is.binary <- function(x) {return(length(unique(x))==2)}

level.count <- function(x) {length(unique(x))}

any.na <- function(x) {any(is.na(x))}

dataframe.drop <- function(df,drops) { df[,!(names(df) %in% drops)] }

dataframe.keep <- function(df,keeps) { df[,(names(df) %in% keeps)] }

dataframe.constant_columns <- function(df) { names(which(apply(df, 2, is.constant))) }

daraframe.low_variance <- function(df, threshold = 0.001) { names(which(apply(df, 2, var)<threshold)) }

dataframe.binary_columns <- function(df) { names(which(apply(df, 2, is.binary))) }

dataframe.rename_column <- function(df,old_name,new_name) 
{
  names(df)[names(df)==old_name]<- new_name
  return(df)
}

dataframe.summarize_in_file <- function(df, filename)
{
  require(readr)

  coltypes <- sapply(df, class)
  colnames <- names(df)
  collevels <- sapply(df, level_count)
  colnas <- sapply(df, any_na)
  colmin <- sapply(df, min)
  colmax <- sapply(df, max)

  result <- data.frame(Name = colnames, Type = coltypes, Level = collevels, NAs = colnas, Min = colmin, Max = colmax)

  result <- result[order(-result$Level),]
  result <- result[order(result$Type),]

  write_csv(x = result,path = filename)
}

dataframe.duplicated_columns <- function(df) {
  features_pair <- combn(names(df), 2, simplify = F)
  toRemove <- c()
  for(pair in features_pair) {
    f1 <- pair[1]
    f2 <- pair[2]
    if (!(f1 %in% toRemove) & !(f2 %in% toRemove)) {
      if (all(df[[f1]] == df[[f2]])) {
        toRemove <- c(toRemove, f2)
      }
    }
  }
  return(toRemove)
}

dataframe.highly_correlated_columns <- function(df,threshold=0.995) {
  features_pair <- combn(names(df), 2, simplify = F)
  toRemove <- c()
  for(pair in features_pair) {
    f1 <- pair[1]
    f2 <- pair[2]
    if (!(f1 %in% toRemove) & !(f2 %in% toRemove)) {
      if (cor(df[[f1]],df[[f2]])>threshold) {
        toRemove <- c(toRemove, f2)
      }
    }
  }
  return(toRemove)
}

dataframe.order_by <- function(df,column_name)
{
  df[with(df, order(df[,column_name])), ]
}

Is there anything I could have done better in terms of performance/readability ?

\$\endgroup\$
  • \$\begingroup\$ This is all pretty basic, so I don't think anything can be improved performance wise; however, in terms of readability, someone who is reading your code would find it hard because of the unknown functions. If I were reading your code I would have to find out what each of those functions do at each point in your code you call them. Honestly, I would probably stop if I saw this. It's important to be able to read your code without much effort and this adds a lot of effort to the reader. . \$\endgroup\$ – Amstell Jun 21 '16 at 19:17
  • 2
    \$\begingroup\$ I know you're writing your own, but there is some overlap with dplyr (e.g., select() handles dataframe.drop, dataframe.keep, and dataframe.rename_column; dplyr::arrange for dataframe.order_by). If you import its namespace, you can use those functions directly with no loss of your other functions. \$\endgroup\$ – r2evans Jun 21 '16 at 20:49
  • 1
    \$\begingroup\$ Also, you might consider using lapply or sapply in place of apply on data.frames (three functions, such as sapply(df, is.binary)); it handles things per-column (as you are using), and using apply on data.frames is generally discouraged. (sapply is also typically much faster, lapply even more so.) \$\endgroup\$ – r2evans Jun 21 '16 at 20:53
  • \$\begingroup\$ I would mention your use of require, and would suggest the use of library instead. In short, require is try(library) with a logical indication for success or failure. It will not however stop execution if success == FALSE. That is a non-welcomed behavior. See this post by Yihui Xie on this matter. \$\endgroup\$ – dof1985 Jul 12 '16 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.