Looping through rows of data sets in files

Question

I have files containing data sets which contain 11,000 rows. I have to run a loop through each row, which is taking me 25 minutes for each file.

     z <- read.zoo("Title.txt", tz = "")
  for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
   ctr <- i
    while(ctr < length(z[,1])) {
     if(       (   abs  ( coredata(z[i,1]) -coredata(z[ctr+1,1])  )      )  > std_d) {
       z[ctr+1,1] <- NA
       ctr <- ctr + 1

      } else {
  break
    }
 }
}

Where Title.txt is a file containing 11,000 rows. It looks like this (first five rows):

"timestamp" "mesured_distance" "IFC_Code" "from_sensor_to_river_bottom"
 "1" "2012-06-03 12:30:07-05" 3188 1005 3500
 "2" "2012-06-03 12:15:16-05" 3189 1005 3500
 "3" "2012-06-03 12:00:08-05" 3185 1005 3500
 "4" "2012-06-03 11:45:11-05" 3191 1005 3500
 "5" "2012-06-03 11:30:15-05" 3188 1005 3500

How can I increase the speed of this code?

Answer 1

In your code, std_d is not defined. Presumably it is some threshold value, since it is used in a comparison of absolute differences.

First, reformat your code so that I can better determine what it is supposed to do:

for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
    ctr <- i
    while(ctr < length(z[,1])) {
        if((abs(coredata(z[i,1])-coredata(z[ctr+1,1]))) > std_d) {
            z[ctr+1,1] <- NA
            ctr <- ctr + 1
        } else {
            break
        }
    }
}

Then determine what the algorithm is that you are trying to do:

Looping over each element of the first column of z, skipping any missing values, compare that value to every other value to the end of column. If the current value and some subsequent value differ by more than some threshold (in absolute difference), set that subsequent value to NA.

R is a vectorized language, so explicit for loops are often not the most efficient way to perform an operation.

Let's vectorize that inner while loop.

for(i in seq_along(z[,1])) {
    if(is.na(coredata(z[i,1]))) next
    ctr <- (i+1):length(z[,1])
    z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1] <- NA
}

I make ctr a vector of all the indexes higher than i, and use that most functions are vectorized. For example, with i equal to 1, (and picking std_d equal to 2) working the statements from inside out:

> ctr <- (i+1):length(z[,1])
> ctr
[1] 6 5
> i
[1] 5
> i <- 1
> ctr <- (i+1):length(z[,1])
> ctr
[1] 2 3 4 5
> coredata(z[i,1]) - coredata(z[ctr,1])
 4  3  2  1 
-3  3 -1  0 
> abs(coredata(z[i,1]) - coredata(z[ctr,1]))
4 3 2 1 
3 3 1 0 
> abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d
    4     3     2     1 
 TRUE  TRUE FALSE FALSE 
> which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d)
4 3 
1 2 
> i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d)
4 3 
2 3 
> z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1]
2012-06-03 11:45:11 2012-06-03 12:00:08 
               3191                3185 
> z[i+which(abs(coredata(z[i,1]) - coredata(z[ctr,1])) > std_d),1] <- NA
> z
                    mesured_distance IFC_Code from_sensor_to_river_bottom
2012-06-03 11:30:15             3188     1005                        3500
2012-06-03 11:45:11               NA     1005                        3500
2012-06-03 12:00:08               NA     1005                        3500
2012-06-03 12:15:16             3189     1005                        3500
2012-06-03 12:30:07             3188     1005                        3500

This change alone would probably speed up your code a lot. You could aslo pull the computation of length(z[,1]) outside the loop, but that will be a much smaller improvement.

The sample of data you gave is too small to do any reasonable benchmarking on.