Using Reddit API in R

I'm scraping some comments from Reddit using Reddit JSON API and R. Since the data does not have a flat structure, extracting it is a little tricky, but I've found a way.

To give you a flavour of what I'm having to do, here is a brief example:

x = "http://www.reddit.com/r/funny/comments/2eerfs/fifa_glitch_cosplay/.json" # example url
rawdat   = fromJSON(rawdat) # formatting
dat_list = repl = rawdat[[2]][[2]][[2]] # this will be used later
sq       = seq(dat_list)[-1]-1 # number of comments
txt      = unlist(lapply(sq,function(x)dat_list[[x]][[2]][[14]])) # comments (not replies)

# loop time:

for(a in sq){
repl  = tryCatch(repl[[a]][[2]][[5]][[2]][[2]],error=function(e) NULL) # getting replies all replies to comment a

if(length(repl)>0){ # in case there are no replies
sq  = seq(repl)[-1]-1 # number of replies
txt    = c(txt,unlist(lapply(sq,function(x)repl[[x]][[2]][[14]]))) # this is what I want

# next level down
for(b in sq){
repl  = tryCatch(repl[[b]][[2]][[5]][[2]][[2]],error=function(e) NULL) # getting all replies to reply b of comment a

if(length(repl)>0){
sq  = seq(repl)[-1]-1
txt    = c(txt,unlist(lapply(sq,function(x)repl[[x]][[2]][[14]])))
}
}
}
}


The above example gets all comments, the first level of replies to each of these comments and the second level of replies (i.e. replies to each of the replies), but this could go down much deeper, so I'm trying to figure out an efficient way of handling this. To achieve this manually, what I'm having to do is this:

1. Copy the following code from the last loop:

for(b in sq){
repl  = tryCatch(repl[[b]][[2]][[5]][[2]][[2]],error=function(e) NULL)

if(length(repl)>0){
sq  = seq(repl)[-1]-1
txt = c(txt,unlist(lapply(sq,function(x)repl[[x]][[2]][[14]])))
}
}

2. Paste that code right after the line that starts with txt = ... and change b in the loop to c.

3. Repeat this procedure approximately 20 times or so, to make sure everything is captured, which as you can imagine creates a huge loop. I was hoping that there must be a way to fold this loop somehow and make it more elegant.

Here are my main recommendations:

1. use recursion
2. use names instead of list indices, for example node$data$reply$data$children reads much better than node[[2]][[5]][[2]][[2]] and it is also more robust to data changes.
3. use well-named variables so you code reads easily

Now for the code:

url       <- "http://www.reddit.com/r/funny/comments/2eerfs/fifa_glitch_cosplay/.json"
rawdat    <- fromJSON(readLines(url, warn = FALSE))
main.node <- rawdat[[2]]$data$children

comment     <- node$data$body
replies     <- node$data$replies
reply.nodes <- if (is.list(replies)) replies$data$children else NULL

• is there a difference between unlist(lapply(items, fun)) and sapply(items, fun) ? – janos Aug 31 '14 at 18:34
• Yes. If fun were to return a vector of the same length for each item, then sapply would put all the output vectors into a matrix, otherwise in a list. lapply on the other hand always returns a list. unlist(lapply(...)) will unwrap the list into a vector. – flodel Aug 31 '14 at 19:07
• If I browse to the url and search for "body:", chrome tells me there are 198 hits (not sure where the one-off comes from) so I'm leaning towards an API limitation. – flodel Sep 3 '14 at 11:23