Find Reddit comments with a particular keyword

Question

I am writing a loop (in R) to webscrape Reddit posts - using Reddit's API ("Pushshift").

Essentially, I would like to get every comment that contains the word "Trump" between now and until 20,000 hours ago at an hourly basis. The API stores the comments in a JSON frame - I wrote the following code in R to obtain these comments (note - I made it so that the results are saved after every 200 iterations in case of a crash):

library(jsonlite)

part1 = "https://api.pushshift.io/reddit/search/comment/?q=trump&after="    
part2 = "h&before="
part3 = "h&size=500"

results = list()
for (i in 1:20000)
{tryCatch({
    {
        url_i<-  paste0(part1, i+1,  part2, i,  part3)
        r_i <-  data.frame(fromJSON(url_i))
        results[[i]] <- r_i

myvec_i <- sapply(results, NROW)

print(c(i, sum(myvec_i))) 
       
        ifelse(i %% 200 == 0, saveRDS(results, "results_index.RDS"), "" )
    }
}, error = function(e){})
}
final = do.call(rbind.data.frame, results)
saveRDS(final, "final.RDS")

The code runs - but I am looking for tips to increase the speed and efficiency of this code. For example, I have noticed that:

• Sometimes this code seems to take a really long time on certain iterations
• I also have a feeling that as the "list" grows in size and the global environment with R becomes more full, things are also slowing down.
• Sometimes, the webscraping stops collecting new results (i.e. I added a statement which shows the cumulative number of results that have been collected at each iteration - sometimes, this number stops updating)
• I used "tryCatch()" to skip errors to prevent the loop from crashing - but perhaps there might have been some way around this that could have potentially resulted in more Reddit comments being scraped?

Could someone please recommend some tips on how to optimize and speed this code up?

Answer 1

Sometimes this code seems to take a really long time on certain iterations

When? How long a time? Is it correlated to the amount of results you get on that iteration? How many total comments there are in an iteration? Something else? These are questions that you should be able to answer before you start optimizing your code.

If you can identify a particular query with a long execution time, check it in your browser. Does it take a long time in your browser? Then the problem is the request, not anything you can fix in your code.

You should not rely on feelings or appearances. Google gives results like this for measuring execution time. Benchmark these things. Then you can look for patterns.

Visit URLs that give odd results in your browser. That can help distinguish between problems with Pushshift and problems with your code.

For that matter, why are you iterating? Why not just "https://api.pushshift.io/reddit/search/comment/?q=trump&after=20000h&sort=asc&size=500"? As is, there are going to be several problems. For example, what happens if there are more than 500 comments in an hour that match? Since you have to figure out how to get more than 500 results for a query, you might as well just use one base query and update the after based on the UTC time of the result (while subtracting one and removing duplicate overlap between requests).

20,000 hours is 833 1/3 days or more than two years. Do you actually need that much data? If so, it seems like you could run this once and then just update it in the future. Because those two-year-old comments should not be changing.

You can read more about using Pushshift on its GitHub. Or read more about the Reddit API.

Note: I'm not an R user, so I'm not critiquing your R. Just how you are describing the problem and your API use. I will say that

for (i in 1:20000)
{tryCatch({

is going to be wrong in any language using the C-style {} blocks. Any of

for (i in 1:20000) {
    tryCatch({

or

for (i in 1:20000)
{
    tryCatch({

or

for (i in 1:20000)
{
    tryCatch(
    {

would be more consistent with general practice. I.e. don't put more code on the same line but after an opening { unless you are putting the closing } on the same line. Some styles will allow the { to go on the same line as the proceeding code.