1
\$\begingroup\$

How can I optimize my code to retrieve the data faster? I'm currently using nested foreach loops to ping API on 2 levels (for each KPI -> for each municipality).

API holds KPI data for each municipality in Sweden. Because the API is built with limitation of 5000 response limit per request i have to use iteration to ping the api. The API holds KPIs for municipalities in Sweden. There are some 2000-3000 KPIs and for each KPI there are 290 municipalities and each municipality holds data for 1 or several years for that KPI.

Link to API documentation: https://github.com/Hypergene/kolada

Packages I need to use:

  • httr \$\rightarrow\$ get the data from the API
  • jsonlite \$\rightarrow\$ converting JSON to dataframes
  • foreach \$\rightarrow\$ foreach loops
  • snow \$\rightarrow\$ running foreach loops as parallel

Some set up:

  • url \$\leftarrow\$ "http://api.kolada.se"
  • path \$\leftarrow\$ ("/v2/municipality?title") # path metadata municipality look up table.
  • path2 \$\leftarrow\$ ("/v2/kpi") #path metadata KPI lookup table

Function to ping the API with input KPI id and municipality id:

API_get <- function(kpi,municipality){
as.data.frame(fromJSON(
  content(
    GET(url = url, 
        path = str_c("/v2/data/kpi/",kpi,"/municipality/",municipality)),"text"))) %>%
  unnest() %>%
  select(kpi_id = values.kpi,
         municipality_id = values.municipality,
         period = values.period,
         gender,
         status,
         value) %>%
  type_convert()}

Alternative function that tries to simplify the function to speed up the process:

API_get2 <- function(kpi,municipality){
as.data.frame(fromJSON(
  content(
    GET(url = url, 
        path = str_c("/v2/data/kpi/",kpi,"/municipality/",municipality)),"text"))) %>%
  unnest()}

The speed difference between API_get2 runs 1-1, 5 secs faster than API_get on 1 KPI and all of the 290 municipalities.

Here is the nested foreach loop that I run:

system.time(
foreach(k = SKL_kpi_has_no_gender$kpi_id[1:1],
        .combine = rbind,
        .packages = c("tidyverse","jsonlite","httr","stringr")) %:% 
        foreach(m = as.character(SKL_municipality$municipality_id[1:290]),
        .combine = rbind,
        .packages = c("tidyverse","jsonlite","httr")) %dopar% API_get2(k,m) )

   user  system elapsed
   0.20    0.03    3.17

I have tried running the mapply function as the function i created has two inputs but it seem to be slower. It runs on one of my CPU cores so maybe that is why the foreach has a speed advantage. I am more wondering if there is a way to make the speed non-linear with the increase in KPIs that is iterated over.

\$\endgroup\$
  • \$\begingroup\$ Do you have a link to the API's documentation? \$\endgroup\$ – flodel Jul 30 '17 at 11:41
  • \$\begingroup\$ @flodel github.com/Hypergene/kolada \$\endgroup\$ – Acke Jul 30 '17 at 11:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.