4
\$\begingroup\$

I'm wondering if there's a more Pythonic way to keep paginating an API endpoint until there's no next key in the response. The thing that bothers me the most is the for item in json_resp.get("data") (I know this can be extracted as a function).

# some code is omitted, some obfuscated
# attachment of token in API calls omitted for code brevity
def get_object_list_for_user(user_service_token): 
    endpoint = f"{HOSTNAME}/me/library/objects/"
    params = {"limit": 100}

    response = _call_service_api(endpoint, params=params)
    json_resp = response.json()

    lists_of_objects_to_append = []

    # get first batch
    for item in json_resp.get("data"):
        object_dict = {"library_id": item["id"], "name": item["attributes"]["name"]}
        lists_of_objects_to_append.append(object_dict)

    offset = 0
    while json_resp.get("next"):
        offset += 100
        params.update({"offset": offset})
        response = _call_service_api(endpoint, params=params)
        json_resp = response.json()

        for item in json_resp.get("data"):
            object_dict = {"library_id": item["id"], "name": item["attributes"]["name"]}
            lists_of_objects_to_append.append(object_dict)

    return lists_of_objects_to_append
\$\endgroup\$
1
\$\begingroup\$

I attempted to reduce redundancy by simulating a do-while loop so that the first condition initializes the sequence and then the code can be factored out.

Instead of having a temporary list, it returns a generator which can be appended to the list.

I couldn't test this so I don't know if it works:

def get_object_list_for_user(user_service_token): 
    endpoint = f"{HOSTNAME}/me/library/objects/"
    params = {"limit": 100}
    offset = 0

    lists_of_objects_to_append = []

    while True:
      json_resp = _call_service_api(endpoint, params=params).json()
      lists_of_objects_to_append.extend(extract_items(json_resp))

      offset += 100
      params.update({"offset": offset})

      if not json_resp.get("next"):
          return lists_of_objects_to_append

def extract_items(json_resp):
    for item in json_resp.get("data"):
        yield({"library_id": item["id"], "name": item["attributes"]["name"]})
| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.