How do I speed up the following R code?

Question

I have files containing data set which contain 11,000 rows. I have to run a loop through each row, this is taking me 25minutes for each file. I am using following code:

     z <- read.zoo("Title.txt", tz = "")
  for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
   ctr <- i
    while(ctr < length(z[,1])) {
     if(       (   abs  ( coredata(z[i,1]) -coredata(z[ctr+1,1])  )      )  > std_d) {
       z[ctr+1,1] <- NA
       ctr <- ctr + 1

      } else {
  break
    }
 }
}

Where Title. txt is file containing 11,000rows. It looks like (first five rows):

"timestamp" "mesured_distance" "IFC_Code" "from_sensor_to_river_bottom"
 "1" "2012-06-03 12:30:07-05" 3188 1005 3500
 "2" "2012-06-03 12:15:16-05" 3189 1005 3500
 "3" "2012-06-03 12:00:08-05" 3185 1005 3500
 "4" "2012-06-03 11:45:11-05" 3191 1005 3500
 "5" "2012-06-03 11:30:15-05" 3188 1005 3500

I wish to receive help on how should I increase the speed for this code? Thank you for your consideration.

Answer 1

In your code, std_d is not defined. Presumably it is some threshold value, since it is used in a comparison of absolute differences.

First, reformat your code so that I can better determine what it is supposed to do:

for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
    ctr <- i
    while(ctr < length(z[,1])) {
        if((abs(coredata(z[i,1])-coredata(z[ctr+1,1]))) > std_d) {
            z[ctr+1,1] <- NA
            ctr <- ctr + 1
        } else {
            break
        }
    }
}

Then determine what the algorithm is that you are trying to do:

Looping over each element of the first column of z, skipping any missing values, compare that value to every other value to the end of column. If the current value and some subsequent value differ by more than some threshold (in absolute difference), set that subsequent value to NA.

R is a vectorized language, so explicit for loops are often not the most efficient way to perform an operation.

Let's vectorize that inner while loop.

for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
    ctr <- (i+1):length(z[,1])
    z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1] <- NA
}

I make ctr a vector of all the indexes higher than i, and use that most functions are vectorized. For example, with i equal to 1, (and picking std_d equal to 2) working the statements from inside out:

> ctr <- (i+1):length(z[,1])
> ctr
[1] 6 5
> i
[1] 5
> i <- 1
> ctr <- (i+1):length(z[,1])
> ctr
[1] 2 3 4 5
> coredata(z[i,1]) - coredata(z[ctr,1])
 4  3  2  1 
-3  3 -1  0 
> abs(coredata(z[i,1]) - coredata(z[ctr,1]))
4 3 2 1 
3 3 1 0 
> abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d
    4     3     2     1 
 TRUE  TRUE FALSE FALSE 
> which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d)
4 3 
1 2 
> i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d)
4 3 
2 3 
> z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1]
2012-06-03 11:45:11 2012-06-03 12:00:08 
               3191                3185 
> z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1] <- NA
> z
                    mesured_distance IFC_Code from_sensor_to_river_bottom
2012-06-03 11:30:15             3188     1005                        3500
2012-06-03 11:45:11               NA     1005                        3500
2012-06-03 12:00:08               NA     1005                        3500
2012-06-03 12:15:16             3189     1005                        3500
2012-06-03 12:30:07             3188     1005                        3500

This change alone would probably speed up your code a lot. You could aslo pull the computation of length(z[,1]) outside the loop, but that will be a much smaller improvement.

The sample of data you gave is too small to do any reasonable benchmarking on.