# More elegant filter script in R

I'm very new both to programming and to R so please bear with me :-) I've written the following bit of code, which works perfectly well and runs through a data file with 17446 rows in about 35 seconds. I don't really have a problem with this but am sure that this could be a lot more elegant with probably the use of the tapply function. I would love to see how you experts would rewrite this more efficiently and am sure the rewrite would teach me a lot. I've included the first few rows of the output file and hopefully the code should be fairly obvious (it must be if I managed to write it!!); it's a simple filter based on the standard deviation of a subset in one column dictating the output to another column which is set up at the start of the code. If a value in the NEE column is more than 2 stdevs of the preceeding three values then the value is taken from the mean of a subset in the NEE2 column else the value is copied from the first (if that makes sense). Please also note the "count" variable as I need to retrieve the amount of values replaced. Hope this piques someone's interest and thanks in advance for all of your time. JonP

Dspke <- read.csv(file.choose())
nr = nrow(Dspke)
count = 0

Dspke$NEE2 <- (1:nr) #creates a new column ready for input of values for (i in 4:nr) { #standard deviation of the previous three values in NEE stdev <- sd(Dspke$NEE[(i-2):i])

#if stdev>2 then NEE2 value is mean of the previous 3 values in NEE2 else copy value from NEE
if(stdev>=2) {
Dspke$NEE2[i] <- mean(Dspke$NEE2[(i-2):i-1])
count = count+1

}else { Dspke$NEE2[i] <- Dspke$NEE[i]
}

}
write.csv(Dspke,"Dspke.csv")

Date_Time       NEE     NEE2
1 03/01/2012 13:00 -2.300000  1.00000
2 03/01/2012 13:30 -2.385610  2.00000
3 03/01/2012 14:00 -2.081935  3.00000
4 03/01/2012 14:30 -1.778260 -1.77826
5 03/01/2012 15:00 -2.409490 -2.40949
6 03/01/2012 15:30 -0.741030 -0.74103

-

## migrated from stackoverflow.comJan 14 '13 at 22:03

This question came from our site for professional and enthusiast programmers.

Can you confirm that NEE2 in mean(Dspke$NEE2[(i-2):i-1]) is really meant to be NEE2 and not NEE? NEE2 seems like an odd choice, and this has a bearing on the complexity of the solution. – Matthew Plourde Jan 11 '13 at 18:23 Sorry, struggling to keep up with the replies, yes the NEE2 is intentional, the new value is the mean of the corrected not the original values, otherwise I would just carry the outliers with me. As for "count", I just retrieve the value with command line afterwards. – JonP Jan 11 '13 at 18:36 ## 1 Answer The embed function is well-suited for constructing trailing vectors by row:  Dspke[ , paste0("E3", 1:3)] <- NA # create columns to hold the trailing values Dspke[4:nrow(Dspke), 5:7] <- embed(Dspke$NEE, 3)
Dspke\$NEE3 <- ifelse( apply(Dspke[,5:7], 1, sd) >2,   #test
rowMeans(Dspke[,5:7]),   # result if TRUE
Dspke[,"NEE"])           # result if FALSE


I see the count vector being created but cannot tell from your description what it is supposed to do. Perhaps you should store the value of:

> apply(Dspke[,5:7], 1, sd) > 2)
1     2     3     4     5     6
NA    NA    NA FALSE FALSE FALSE


(Not a very good test of the code presented by the data since all of the tests are FALSE.)

-
In OP's example, the current mean's calculated from the last three corrected values, not the last three original values. –  Matthew Plourde Jan 11 '13 at 18:20
Groan. My comment about the lack of a good test set is bearing bitter fruit. Well, my solution IS more elegant. –  DWin Jan 11 '13 at 18:24
Hi, thanks for the reply. Well I tried that but get the error message...Error in *tmp*[[j]] : recursive indexing failed at level 2 In addition: Warning message: In matrix(value, n, p) : data length [52332] is not a sub-multiple or multiple of the number of rows [17443] –  JonP Jan 11 '13 at 18:32
Are you now seeing why it is strongly recommended that you post the output of dput() on your sample data rather than relying on us to reconstruct it from console output? –  DWin Jan 11 '13 at 18:40
Yes tried that, it outputs a file with 11639 rows which appears to bear very little relationship to my actual output file. Bearing in mind my input file is 17446 rows long. No worries, I'll see if I can get a good understanding of the script you posted, I'm sure once I understand it how it works I'll be able to apply it to my file. Thanks again. –  JonP Jan 11 '13 at 18:50