1
\$\begingroup\$

I'm writing an Exploratory Data Analysis using R Markdown. First of all, I need to check the "sanity" of the input data set. Among the other check I perform, I need to check if at least one between the variables var_in and var_out are present in the dataset. If at least one is present, the other one can be computed from the first one. Thus, I want to check which of them is missing, and store its name in a character vector. If both are missing, the analysis is impossible and I need to exit, preferably throwing a meaningful error.

I cannot include the actual dataset on which the check is performed, because it's \$10^6\$ rows with confidential data, so I need to use fake data. The code, however, is as similar to the real code as possible.

# fake data
n <- 10^6
input_df <- data.frame(wind_speed = 10*abs(rnorm(n)), wind_direction = runif(n, 0, 2*pi),
                       var_in = NA, var_out = 3)

# nearly real code
missing_variables <- character(0)

var_in_is_missing <- all(is.na(input_df$var_in))
var_out_is_missing <- all(is.na(input_df$var_out))

if (var_in_is_missing & var_out_is_missing) {
  stop("both var_in and var_out are completely missing, so I cannot continue the EDA")
}
if (var_in_is_missing) {
  missing_variables <- c("var_in", missing_variables)
}
if (var_out_is_missing) {
  missing_variables <- c("var_out", missing_variables)
}

Note that according to the principle of early return, I put the stopping test before the other two. The code runs, but it doesn't seem that readable to me:

  1. I test both var_in_is_missing and var_out_is_missing twice: this is definitely not going to impact notably the performance of my code, but it still feels useless
  2. is it really necessary to use three separate if statements? Isn't there a way to do it in R in a more...compact way, without sacrificing readability?
\$\endgroup\$
1
  • 2
    \$\begingroup\$ An additional minor comment, in this situation you should use && rather than &, cleaner as you're not comparing vector, and this way RHS won't be tested if LHS is FALSE. \$\endgroup\$ Oct 3, 2018 at 2:20

2 Answers 2

2
\$\begingroup\$

I would do something like this:

tt <- function() {
  naI <- sapply(input_df[, c('var_in', 'var_out')], function(x) all(is.na(x)))
  if (sum(naI) == 2) {
    stop("both var_in and var_out are completely missing, so I cannot continue the EDA")
  } else {
    missing_variables <- names(naI[naI == TRUE])
  }
  missing_variables
}

I put the code into function for testing. Tests:

input_df <- data.frame(var_in = NA, var_out = 3)
tt()
#> [1] "var_in"
input_df <- data.frame(var_in = NA, var_out = NA)
tt()
#> Error in tt(): both var_in and var_out are completely missing, so I cannot continue the EDA
input_df <- data.frame(var_in = 3, var_out = NA)
tt()
#> [1] "var_out"
input_df <- data.frame(var_in = 3, var_out = 3)
tt()
#> character(0)

P.S. maybe you also want to add some kind of treatment if the one/both columns are missing in data.frame...

\$\endgroup\$
5
  • \$\begingroup\$ Very nice! Also, the way you wrote it, it's easy to generalize it if in the future I need to check for more variables. I'll probably replace sapply with purrr::map_lgl or something, but other than that, it's great. Only, I don't get your last point: "maybe you also want to add some kind of treatment if the one/both columns are missing in data.frame". I already added some action: I update missing_variables accordingly. What diid you have in mind? \$\endgroup\$
    – DeltaIV
    Sep 28, 2018 at 7:40
  • 1
    \$\begingroup\$ @DeltaIV if var_in or var_out were not in the data.frame then the code wouldn't work. why purrr::map_lgl? \$\endgroup\$
    – minem
    Sep 28, 2018 at 7:43
  • \$\begingroup\$ I mean, functions downstream get missing_variables as an input and process the data.frame accordingly. I'd rather have each function do one thing: this one only checks for missingness of these two "complementary" variables. \$\endgroup\$
    – DeltaIV
    Sep 28, 2018 at 7:43
  • 1
    \$\begingroup\$ Ah, I got your point now, thanks. The variables have to be in the data frame. It's the result from an automated system. The system may not have measured anything because of sensor failures (i.e., variables are missing), but the file header (i.e., the column names) is fixed. \$\endgroup\$
    – DeltaIV
    Sep 28, 2018 at 7:46
  • \$\begingroup\$ purrr::map_lgl because the output is a logical vector. I like the type consistency of the purrr functions: just a pet peeve of mine. \$\endgroup\$
    – DeltaIV
    Sep 28, 2018 at 7:52
1
\$\begingroup\$

Another possibility :

cols <- c('var_in', 'var_out')
missing_variables <- cols[!colSums(!is.na(input_df[cols]))]
if (length(missing_variables) == 2) 
  stop("both var_in and var_out are completely missing, so I cannot continue the EDA")

Let's unwrap the definition of missing_variables :

  • !is.na(input_df[cols]) creates a matrix of logicals indicating if elements are NA (FALSE) or not (TRUE),
  • colSums(!is.na(input_df[cols])) sum the columns of this matrix, FALSE is coerced to 0 and TRUE to 1 so a column will be full of NA if and only if this sum is O
  • In !colSums(!is.na(input_df[cols])), ! changes 0 to TRUE and any other number to FALSE, so we get a vector of logical indicating if the variable is missing (TRUE) or not
  • cols[!colSums(!is.na(input_df[cols]))] subsets cols to keep only "missing variables"
\$\endgroup\$
4
  • \$\begingroup\$ Can you explain me what missing_variables <- cols[!colSums(is.na(input_df[cols]))] does? \$\endgroup\$
    – DeltaIV
    Oct 3, 2018 at 10:09
  • 1
    \$\begingroup\$ Sure, I edited my answer, let me know if it makes sense to you. \$\endgroup\$ Oct 3, 2018 at 10:25
  • \$\begingroup\$ I had a mistake and a typo, now corrected! (needed a 2nd !) \$\endgroup\$ Oct 3, 2018 at 15:51
  • \$\begingroup\$ The 2nd ! is still missing in your answer. \$\endgroup\$
    – DeltaIV
    Oct 3, 2018 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.