Paginating API with "next" Pythonically

Question

I'm wondering if there's a more Pythonic way to keep paginating an API endpoint until there's no next key in the response. The thing that bothers me the most is the for item in json_resp.get("data") (I know this can be extracted as a function).

# some code is omitted, some obfuscated
# attachment of token in API calls omitted for code brevity
def get_object_list_for_user(user_service_token): 
    endpoint = f"{HOSTNAME}/me/library/objects/"
    params = {"limit": 100}

    response = _call_service_api(endpoint, params=params)
    json_resp = response.json()

    lists_of_objects_to_append = []

    # get first batch
    for item in json_resp.get("data"):
        object_dict = {"library_id": item["id"], "name": item["attributes"]["name"]}
        lists_of_objects_to_append.append(object_dict)

    offset = 0
    while json_resp.get("next"):
        offset += 100
        params.update({"offset": offset})
        response = _call_service_api(endpoint, params=params)
        json_resp = response.json()

        for item in json_resp.get("data"):
            object_dict = {"library_id": item["id"], "name": item["attributes"]["name"]}
            lists_of_objects_to_append.append(object_dict)

    return lists_of_objects_to_append

Answer 1

I attempted to reduce redundancy by simulating a do-while loop so that the first condition initializes the sequence and then the code can be factored out.

Instead of having a temporary list, it returns a generator which can be appended to the list.

I couldn't test this so I don't know if it works:

def get_object_list_for_user(user_service_token): 
    endpoint = f"{HOSTNAME}/me/library/objects/"
    params = {"limit": 100}
    offset = 0

    lists_of_objects_to_append = []

    while True:
      json_resp = _call_service_api(endpoint, params=params).json()
      lists_of_objects_to_append.extend(extract_items(json_resp))

      offset += 100
      params.update({"offset": offset})

      if not json_resp.get("next"):
          return lists_of_objects_to_append

def extract_items(json_resp):
    for item in json_resp.get("data"):
        yield({"library_id": item["id"], "name": item["attributes"]["name"]})