This is simple code that recursively downloads articles from the public WordPress API:

import requests

from pymongo import MongoClient
client = MongoClient()
db = client.wp

'number': 100
}

url = 'https://public-api.wordpress.com/rest/v1.1/sites/{site_id}/posts'
found = response['found']
posts = response['posts']
print 'found {number} posts'.format(number=found)
db.posts.insert_many(posts)

def get_next_page(offset):
print 'Inserted we have {left} posts left to go'.format(left=abs(found-offset))
response = requests.get(url, params= {
'number': 100,
'offset': offset
}).json()

db.posts.insert_many(response['posts'])

if int(response['found']) > offset:
get_next_page(offset + 100)

get_next_page(100)


The logic for getting 100 posts is duplicated for the case of the initial load and subsequent loads with an offset. It would be better to refactor in a way to eliminate the duplicated logic:

params = {'number': 100}
offset = 0

while True:
response = requests.get(url, params=params).json()
found = response['found']
posts = response['posts']
db.posts.insert_many(posts)
print 'Inserted {number} posts'.format(number=len(posts))
offset += 100
if offset >= found:
break
print 'We have {left} posts left to go'.format(left=found-offset)
params['offset'] = offset


Your code follows most of PEP8.

However:

• Lines should be a maximum of 79 characters. The exception to this is comments, at 72.
• Operators should have a space either side of them. 2 * 2. The exception to this is to show precedence: 2*2 + 2.
• You should not have any spaces around the = when passing arguments:

response = requests.get(url, params= {


Your code looks really good otherwise.

It may be worth removing the need to parse key-words to str.format:

print 'found {number} posts'.format(number=found)


It's good, but a bit long-winded.

Also, I would recommend not relying on globals in get_next_page. If you only want to pass found once you can do this:

def get_next_page(found):
def inner(offset):
# Same as before
if int(response['found']) > offset:
inner(offset + 100)
return inner

get_next_page(found)(100)


And finally, if you are planning on using this on a very large amount of data, where you would have to use get_next_page a lot, I would recommend not using recursion.

Have a look at sys.getrecursionlimit(). It tells you how many times you can recurse. My limit is 1000, for example.

• Can you explain why using inner? – toy Jul 13 '15 at 23:48