Script to access genealogy API for list of users and attributes

Question

This is a script to access the Stack Exchange API for the genealogy site and create a list of user dictionaries. It contains a snippet at end to validate. The data obtained is stored in YAML format in a file for use by other programs to analyze the data.

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module 
#      - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use YAML to store output for use by other programs
#     see http://pyyaml.org/wiki/PyYAML
import yaml
OUTFILENAME = "users.yaml"

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
URL ='https://api.stackexchange.com/2.1/users'

url_params = { 
    'site'     : 'genealogy',
    'pagesize' : 100,
    'order'    : 'desc',
    'sort'     : 'reputation',
    }



page = 1
not_done = True
user_list = []

# replies are paginated so loop thru until none left
while not_done:
    url_params['page'] = page
    # get next page of users
    api_response = requests.get(URL,params=url_params)
    json_data = api_response.json()

    # pull the list of users out of the json answer
    user_list.extend( json_data['items'] )

    # show progress each time thru loop
    print api_response.url
    #note only so many queries allowed per day
    print '\tquota remaining: %s' % json_data['quota_remaining'] 
    print "\t%s users after page %s" % (len(user_list),page)

    # prepare for next iteration if needed
    page += 1
    not_done = json_data['has_more']


# output list of users to a file in yaml, a format easily readable by humans and parsed
#   note safe_dump is used for security reasons 
#   since dump allows executable python code 
#   Sidebenefit of safe_dump is cleaner text
outFile = open(OUTFILENAME,"w")
yaml.safe_dump(user_list, outFile)
outFile.close()

# validate it wrote correctly 
#   note this  shows example of how to read 
infile = open(OUTFILENAME)
readList = yaml.safe_load(infile)
infile.close()


print 'wrote list of %d users as yaml file %s' % (len(readList),OUTFILENAME)

I wrote this code because I wanted the data it gets from the genealogy site but also:

• to learn APIs in general
• to learn the Stack Exchange API
• to get back into programming which I hadn't done much lately
• I thought I'd try YAML

Since it's such a simple script I didn't bother with objects, but I would be interested in how it could be remade in the functional programming paradigm which I'm not as familiar with.

Answer 1

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use pretty print for output

That's a pretty useless comment. It doesn't really tell me anything I didn't already know from the import

import pprint



# save output in file in format compatable with eval
#   I didn't use json or yaml since my intent is to continue on using 
#   the parsed output so saves a parse step

eval also has a parse step. So you aren't saving anything by avoiding json.

pytFilename = "users.pyt"

python convention is for constants to be in ALL_CAPS

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
url='https://api.stackexchange.com/2.1/users'
site = 'genealogy'

urlParams = {}
urlParams['site'] = site
urlParams['pagesize'] = 100
urlParams['order'] = 'desc'
urlParams['sort'] = 'reputation'

Why not put all these parameters into a single dict literal?

page = 1
notDone = True
userList = []

Python convention is for local variables to be mixed_case_with_underscores.

# replies are paginated so loop thru until none left
while notDone:
    urlParams['page'] = page
    # get next page of users
    r = requests.get(url,params=urlParams)

avoid single letter variable names. It makes it harder to figure out what you are doing.

    # pull the list of users out of the json answer
    userList += r.json()['items']

I'd use userList.extend( r.json()['items']). Also given the number of times you access the json, I'd have ajson = r.json()` line and access the json data through that.

    # show progress each time thru loop, note only so many queries allowed per day
    print r.url
    print '\tquota remaining: %s' % r.json()['quota_remaining'] 
    print "\t%s users after page %s" % (len(userList),page)

    # prepare for next iteration if needed
    page += 1
    notDone = r.json()['has_more']

I'd do:

json = {'has_more' : True}
while json['has_more']:
     ...
     json = r.json()
     ...

As I think it's easier to follow then a boolean flag.

# output list of users to a file in a format readable by eval
open(pytFilename,"w").write( pprint.pformat(userList) )

This isn't recommended practice. In CPython this will close the file, but it won't if you are using one of the other pythons. Instead, it's recommend to do:

with open(pytFilename, 'w') as output:
     output.write( pprint.pformat(userList) )

And that will close the file in all implementations of python.

# validate it wrote correctly 
#   note this  shoasw example of how to read 
readList = eval( open(pytFilename,"r").read() )


print 'wrote %s as python evaluatable list of users' % pytFilename