2
\$\begingroup\$

I'm writing a lot of R in my spare time and am trying to get into good habits by using TidyVerse as much as possible. A large part of this has required reading entire directories of dated log files into R for processing. The volumes of data are well into the GBs so I won't post one here and it shouldn't matter so long as you have filenames matching "./accounts_20180217.csv" or "./symbols_20180217.csv" in the working or data directory.

Example file contents is:

./accounts_20180217.csv

 16 109=18A
175 109=1FN    
 10 109=ABP
  6 109=ABW
787 109=ACI
226 109=ACL
163 109=AD
644 109=AGM
 79 109=AGN
 40 109=AIG
  2 109=AJT
991 109=ALG
  4 109=ALGR
 14 109=AM1
 17 109=AN1
 34 109=AP1
136 109=APA
267 109=APJ    
160 109=ARE
 64 109=ARX
329 109=AS1

I've been using the following to read in and categorise the file data:

library(tidyverse)
library(stringr)

all_data <- tibble(
              filename = list.files(
                   "./", pattern = "^(accounts|symbols)_201[7-9][0-9]{2}[0-9]{2}\\.csv$"
               , full.names = TRUE
              )
            ) %>%
            mutate(
              date = str_extract(filename, "201[7-9][0-9]{4}")
              , type = str_extract(filename, "(accounts|symbols)")
              , data = map2(
                      filename
                      , date
                      , ~read.csv(
                           .x
                           , header = FALSE
                           , sep = "="
                           , as.is = TRUE
                           , col.names = c("count", "names")
                           , colClasses = c("integer", "character") ))
  )

My question is whether this is a good and Tidy way to achieve this or is there a "better" or "Tidier" way?

\$\endgroup\$
2
  • \$\begingroup\$ What are you planing to do with these tables after reading them? \$\endgroup\$
    – Hugh
    Sep 15, 2018 at 15:57
  • \$\begingroup\$ @Hugh they get pivoted and sliced and diced then graphed in various ways to demonstrate message flow statistics to a customer. \$\endgroup\$
    – MD-Tech
    Sep 17, 2018 at 7:40

1 Answer 1

4
\$\begingroup\$

A consistent tidyverse solution would use readr::read_csv I guess.

You don't need map2 if you don't use .y, map is enough.

You can use read.csv as your .f argument, and leave the rest to the ... in your map call, might be a matter of taste but I prefer it that way as with the formula notation you have to read through to make sure read.csv is the only function called.

as.is doesn't seem necessary if you give column classes already.

Your mutate call is not necessary, tibble will evaluate lazily the definition of filename given at the previous step.

It seems you really want a "one liner" but in my opinion it would be much clearer in a few steps, I propose the following:

files <- list.files(
  "./",
  full.names = TRUE,
  pattern = "^(accounts|symbols)_201[7-9][0-9]{4}\\.csv$")

data_lst <- map(
  files, 
  read_csv,
  header = FALSE,
  sep = "=",
  coltypes = "ic",
  col_names = c("count", "names"))

tibble(
  file_name = files,
  date = str_extract(filename, "201[7-9][0-9]{4}"),
  type = str_extract(filename, "(accounts|symbols)"),
  data = data_lst)

In fact as you're working with a lot of data files I'd tend to write a for loop instead of a map call, so I could more easily debug and proceed without restarting all if an issue arises.

\$\endgroup\$
3
  • \$\begingroup\$ Just so you know I did know that I need to move to read_csv and have done in other projects. Also I hadn't spotted that map2 error so thanks! \$\endgroup\$
    – MD-Tech
    Sep 13, 2018 at 7:36
  • \$\begingroup\$ +1 but I won't accept the answer for a few days to allow others to answer since R on Code Review SO isn't exactly epically followed \$\endgroup\$
    – MD-Tech
    Sep 14, 2018 at 9:23
  • \$\begingroup\$ Indeed, and it's a shame considering how people fight for answers on SO... Take your time ;) \$\endgroup\$ Sep 14, 2018 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.