I find myself often encountering a use case where I need to fetch paginated data, where I need all of the data. Currently I use something similar to below to handle these cases:
def fetch_paginated_data(api_base, query, good_citizen_rate):
encoded = urllib.parse.quote(query.encode('utf-8'))
initial = f'{api_base}?q={encoded}'
payload = {
'has_more': True,
'next_page': initial
}
storage = []
while payload['has_more']:
response = requests.get(payload['next_page'])
if response.status_code != 200:
raise ValueError('Status not 200')
payload['has_more'] = False
return
data = response.json()
if (data and data['data']):
storage.extend(data['data'])
payload['has_more'] = data['has_more']
if 'next_page' in data:
payload['next_page'] = data['next_page']
time.sleep(good_citizen_rate)
return storage
My question is how could I better make this convention more reusable? Should I use async
/ await
? yield
pages? Add a callback to transform the data prior to adding it to the collection variable?
requests.get(payload['next_page'])
if there is no next page? And why do you needtime.sleep(good_citizen_rate)
? \$\endgroup\$ – ades May 11 '20 at 12:38has_more
from the apis I often use.time.sleep(good_citizen_rate)
is to follow the providers of these apis guidelines \$\endgroup\$ – SumNeuron May 11 '20 at 13:03payload['has_more'] = False
has no effect, because it is written afterraise
(also thereturn
). And it seems not only related to 3.6+ (except for the f-string feature you're using, so this is only a side comment). \$\endgroup\$ – colidyre May 19 '20 at 13:10