# Unefficent data wrangling in R

so I've been working with R quite a while now, but due to very stressful deadlines in my last university module I had, I wrote code that I'm not very proud of. I would like to be more efficient in data wrangling, e.g. reading files, rearranging and cropping dfs. Here's a short excerpt of my code

 # Reading files

# Format Dates
org_bm$datetime <- as_datetime(org_bm$datetime) + years(98)
t_bm$datetime <- as_datetime(t_bm$datetime) + years(98)
co2_bm$datetime <- as_datetime(co2_bm$datetime) + years(98)

# Select cols
org_ghg <- select(org_ghg, c(datetime,
aN_n2o_emis.kgNha.1.,
aC_ch4_emis.kgCha.1.,
aC_co2_emis_hetero.kgCha.1.,
aC_co2_emis_auto.kgCha.1.))
t_ghg <- select(t_ghg, c(datetime,
aN_n2o_emis.kgNha.1.,
aC_ch4_emis.kgCha.1.,
aC_co2_emis_hetero.kgCha.1.,
aC_co2_emis_auto.kgCha.1.))
co2_ghg <- select(co2_ghg, c(datetime,
aN_n2o_emis.kgNha.1.,
aC_ch4_emis.kgCha.1.,
aC_co2_emis_hetero.kgCha.1.,
aC_co2_emis_auto.kgCha.1.))


As you can see I mastered c/p and "search and replace" XD But seriously, I'm wondering how I could "automate" these tasks. Maybe loading everything into a list and than write helper functions to apply certain actions (i.e. date formatting, selecting cols) to every list entry ?

I've always struggled to find good practice tasks, b/c during class I usually don't have the time to read up on nicer ways to make things work. So if anybody has some suggestions on where to find exercises on data analysis tasks, I would very much appreciate that.

You can do more conversion as you read the data. The magic is done by the colClasses parameter to read.delim(). That parameter allows you to (1) convert the DateTime whilst reading and (2) drop unwanted columns.

Now I don't know what your data looks like, but here's a rough outline of how to read a datetime (untested). Note that you can do the + years(98) within the data-parsing class as well.

library(methods)
setClass("myDateTime")
setAs("character",
"myDateTime",
function(from) as.DateTime(from, format="%Y-%m-%d %H:%M:%S", tz=Sys.timezone())) + years(98)

d = read.delim(..., colClasses = c(..., "myDateTime", ...))


For the columns you want to drop, code them as "NULL" (note the quotes).

d = read.delim(..., colClasses = c("numeric", "myDateTime", "NULL", ...))


This not only shortens your code, but by explicitly telling read.delim() the datatype of each column the read is 2x faster. And of course there aren't two copies of the data in memory after the parsing.

You might also want to look at the skip= and col.names=c() parameters as they allow you to decide the names of the variables, rather than having those decided by the input file. If you decide to use colCLasses=c() you'll almost certainly want to skip the column headings rather than program a class which reads strings for row 1 and a different datatype for the remaining rows.

BTW, if you have to do complex manipulation of input data, then the tidyverse's readr and dplyr libraries have useful improvement over the shipped R libraries. Regardless of your use or not of Tidyverse, its book is worth reading if you are an R programmer: https://r4ds.had.co.nz/