Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.

This is a script to access the Stack Exchange API for the genealogy site and create a list of user dictionaries. It contains a snippet at end to validate. The data obtained is stored in YAML format in a file for use by other programs to analyze the data.

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module 
#      - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use YAML to store output for use by other programs
#     see http://pyyaml.org/wiki/PyYAML
import yaml
OUTFILENAME = "users.yaml"

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
URL ='https://api.stackexchange.com/2.1/users'

url_params = { 
    'site'     : 'genealogy',
    'pagesize' : 100,
    'order'    : 'desc',
    'sort'     : 'reputation',
    }



page = 1
not_done = True
user_list = []

# replies are paginated so loop thru until none left
while not_done:
    url_params['page'] = page
    # get next page of users
    api_response = requests.get(URL,params=url_params)
    json_data = api_response.json()

    # pull the list of users out of the json answer
    user_list.extend( json_data['items'] )

    # show progress each time thru loop
    print api_response.url
    #note only so many queries allowed per day
    print '\tquota remaining: %s' % json_data['quota_remaining'] 
    print "\t%s users after page %s" % (len(user_list),page)

    # prepare for next iteration if needed
    page += 1
    not_done = json_data['has_more']


# output list of users to a file in yaml, a format easily readable by humans and parsed
#   note safe_dump is used for security reasons 
#   since dump allows executable python code 
#   Sidebenefit of safe_dump is cleaner text
outFile = open(OUTFILENAME,"w")
yaml.safe_dump(user_list, outFile)
outFile.close()

# validate it wrote correctly 
#   note this  shows example of how to read 
infile = open(OUTFILENAME)
readList = yaml.safe_load(infile)
infile.close()


print 'wrote list of %d users as yaml file %s' % (len(readList),OUTFILENAME)

I wrote this code because I wanted the data it gets from the genealogy site but also:

  • to learn APIs in general
  • to learn the Stack Exchange API
  • to get back into programming which I hadn't done much lately
  • I thought I'd try YAML

Since it's such a simple script I didn't bother with objects, but I would be interested in how it could be remade in the functional programming paradigm which I'm not as familiar with.

share|improve this question
    
Post rolled back as it invalidated the answer. The original code should not be revised as such. –  Jamal May 9 at 2:21
add comment

1 Answer 1

'''
This is a script to access the stackexchange api
    for the genealogy site 
    and create a list of user dictionaries
'''

# use requests module - see http://docs.python-requests.org/en/latest/user/install/#install
import requests

# use pretty print for output

That's a pretty useless comment. It doesn't really tell me anything I didn't already know from the import

import pprint



# save output in file in format compatable with eval
#   I didn't use json or yaml since my intent is to continue on using 
#   the parsed output so saves a parse step

eval also has a parse step. So you aren't saving anything by avoiding json.

pytFilename = "users.pyt"

python convention is for constants to be in ALL_CAPS

# use se api and access genealogy site
#   see https://api.stackexchange.com/docs/users for api info
url='https://api.stackexchange.com/2.1/users'
site = 'genealogy'

urlParams = {}
urlParams['site'] = site
urlParams['pagesize'] = 100
urlParams['order'] = 'desc'
urlParams['sort'] = 'reputation'

Why not put all these parameters into a single dict literal?

page = 1
notDone = True
userList = []

Python convention is for local variables to be mixed_case_with_underscores.

# replies are paginated so loop thru until none left
while notDone:
    urlParams['page'] = page
    # get next page of users
    r = requests.get(url,params=urlParams)

avoid single letter variable names. It makes it harder to figure out what you are doing.

    # pull the list of users out of the json answer
    userList += r.json()['items']

I'd use userList.extend( r.json()['items']). Also given the number of times you access the json, I'd have ajson = r.json()` line and access the json data through that.

    # show progress each time thru loop, note only so many queries allowed per day
    print r.url
    print '\tquota remaining: %s' % r.json()['quota_remaining'] 
    print "\t%s users after page %s" % (len(userList),page)

    # prepare for next iteration if needed
    page += 1
    notDone = r.json()['has_more']

I'd do:

json = {'has_more' : True}
while json['has_more']:
     ...
     json = r.json()
     ...

As I think it's easier to follow then a boolean flag.

# output list of users to a file in a format readable by eval
open(pytFilename,"w").write( pprint.pformat(userList) )

This isn't recommended practice. In CPython this will close the file, but it won't if you are using one of the other pythons. Instead, it's recommend to do:

with open(pytFilename, 'w') as output:
     output.write( pprint.pformat(userList) )

And that will close the file in all implementations of python.

# validate it wrote correctly 
#   note this  shoasw example of how to read 
readList = eval( open(pytFilename,"r").read() )


print 'wrote %s as python evaluatable list of users' % pytFilename
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.