(revised) Python script to access genealogy api for list of users and attributes

Question

The problem the code is solving

This is a script to access the stackexchange api for the genealogy site and create a list of user dictionaries. Ie each user dictionary contains the info about that user obtained from the api. This data can then be used to analyze user behavior on the beta genealogy site and look for things to improve.

The code works. It contains a snippet at end to validate. The data obtained is stored in YAML format in a file for use by other programs to analyze the data.

---

The Code

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module 
#      - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use YAML to store output for use by other programs
#     see http://pyyaml.org/wiki/PyYAML
import yaml
OUTFILENAME = "users.yaml"

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
URL ='https://api.stackexchange.com/2.1/users'

url_params = { 
    'site'     : 'genealogy',
    'pagesize' : 100,
    'order'    : 'desc',
    'sort'     : 'reputation',
    }



page = 1
not_done = True
user_list = []

# replies are paginated so loop thru until none left
while not_done:
    url_params['page'] = page
    # get next page of users
    api_response = requests.get(URL,params=url_params)
    json_data = api_response.json()

    # pull the list of users out of the json answer
    user_list.extend( json_data['items'] )

    # show progress each time thru loop
    print api_response.url
    #note only so many queries allowed per day
    print '\tquota remaining: %s' % json_data['quota_remaining'] 
    print "\t%s users after page %s" % (len(user_list),page)

    # prepare for next iteration if needed
    page += 1
    not_done = json_data['has_more']


# output list of users to a file in yaml, a format easily readable by humans and parsed
#   note safe_dump is used for security reasons 
#   since dump allows executable python code 
#   Sidebenefit of safe_dump is cleaner text
outFile = open(OUTFILENAME,"w")
yaml.safe_dump(user_list, outFile)
outFile.close()

# validate it wrote correctly 
#   note this  shows example of how to read 
infile = open(OUTFILENAME)
readList = yaml.safe_load(infile)
infile.close()


print 'wrote list of %d users as yaml file %s' % (len(readList),OUTFILENAME)

---

What looking for in review

I wrote this code because I wanted the data it gets from the genealogy site but also:

• to learn api's in general
• to learn the stackexchange api
• to get back into programming which I hadn't done much lately
• I thought I'd try YAML

I learned alot from comments on my original submission so I'd appreciate feedback on this version. Since it's such a simple script I didn't bother with objects, but I would be interested in how it could be remade in the functional programming paridigm which I'm not as familar with.

Answer 1

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use pretty print for output

That's a pretty useless comment. It doesn't really tell me anything I didn't already know from the import

import pprint



# save output in file in format compatable with eval
#   I didn't use json or yaml since my intent is to continue on using 
#   the parsed output so saves a parse step

eval also has a parse step. So you aren't saving anything by avoiding json.

pytFilename = "users.pyt"

python convention is for constants to be in ALL_CAPS

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
url='https://api.stackexchange.com/2.1/users'
site = 'genealogy'

urlParams = {}
urlParams['site'] = site
urlParams['pagesize'] = 100
urlParams['order'] = 'desc'
urlParams['sort'] = 'reputation'

Why not put all these parameters into a single dict literal?

page = 1
notDone = True
userList = []

Python convention is for local variables to be mixed_case_with_underscores.

# replies are paginated so loop thru until none left
while notDone:
    urlParams['page'] = page
    # get next page of users
    r = requests.get(url,params=urlParams)

avoid single letter variable names. It makes it harder to figure out what you are doing.

    # pull the list of users out of the json answer
    userList += r.json()['items']

I'd use userList.extend( r.json()['items']). Also given the number of times you access the json, I'd have ajson = r.json()` line and access the json data through that.

    # show progress each time thru loop, note only so many queries allowed per day
    print r.url
    print '\tquota remaining: %s' % r.json()['quota_remaining'] 
    print "\t%s users after page %s" % (len(userList),page)

    # prepare for next iteration if needed
    page += 1
    notDone = r.json()['has_more']

I'd do:

json = {'has_more' : True}
while json['has_more']:
     ...
     json = r.json()
     ...

As I think it's easier to follow then a boolean flag.

# output list of users to a file in a format readable by eval
open(pytFilename,"w").write( pprint.pformat(userList) )

This isn't recommended practice. In CPython this will close the file, but it won't if you are using one of the other pythons. Instead, it's recommend to do:

with open(pytFilename, 'w') as output:
     output.write( pprint.pformat(userList) )

And that will close the file in all implementations of python.

# validate it wrote correctly 
#   note this  shoasw example of how to read 
readList = eval( open(pytFilename,"r").read() )


print 'wrote %s as python evaluatable list of users' % pytFilename