Skip to main content

Python script to access genealogy api for list of users and attributes

I wrote the following code partly to learn api's in general, partly to learn the stackexchange api, and partly because I wanted the data from the genealogy se site to play with. Feedback welcome.


'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use pretty print for output
import pprint

# save output in file in format compatable with eval
#   I didn't use json or yaml since my intent is to continue on using 
#   the parsed output so saves a parse step
pytFilename = "users.pyt"

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
url='https://api.stackexchange.com/2.1/users'
site = 'genealogy'

urlParams = {}
urlParams['site'] = site
urlParams['pagesize'] = 100
urlParams['order'] = 'desc'
urlParams['sort'] = 'reputation'

page = 1
notDone = True
userList = []

# replies are paginated so loop thru until none left
while notDone:
    urlParams['page'] = page
    # get next page of users
    r = requests.get(url,params=urlParams)

    # pull the list of users out of the json answer
    userList += r.json()['items']

    # show progress each time thru loop, note only so many queries allowed per day
    print r.url
    print '\tquota remaining: %s' % r.json()['quota_remaining'] 
    print "\t%s users after page %s" % (len(userList),page)

    # prepare for next iteration if needed
    page += 1
    notDone = r.json()['has_more']


# output list of users to a file in a format readable by eval
open(pytFilename,"w").write( pprint.pformat(userList) )

# validate it wrote correctly 
#   note this  shoasw example of how to read 
readList = eval( open(pytFilename,"r").read() )


print 'wrote %s as python evaluatable list of users' % pytFilename

Duncan
  • 164
  • 5