7
\$\begingroup\$

This is simple code that recursively downloads articles from the public WordPress API:

import requests

from pymongo import MongoClient
client = MongoClient()
db = client.wp

payload = {
    'number': 100
}

url = 'https://public-api.wordpress.com/rest/v1.1/sites/{site_id}/posts'
response = requests.get(url, params=payload).json()
found = response['found']
posts = response['posts']
print 'found {number} posts'.format(number=found)
db.posts.insert_many(posts)

def get_next_page(offset):
    print 'Inserted we have {left} posts left to go'.format(left=abs(found-offset))
    response = requests.get(url, params= {
        'number': 100,
        'offset': offset
    }).json()

    db.posts.insert_many(response['posts'])

    if int(response['found']) > offset:
        get_next_page(offset + 100)


get_next_page(100)
\$\endgroup\$
5
\$\begingroup\$

The logic for getting 100 posts is duplicated for the case of the initial load and subsequent loads with an offset. It would be better to refactor in a way to eliminate the duplicated logic:

params = {'number': 100}
offset = 0

while True:
    response = requests.get(url, params=params).json()
    found = response['found']
    posts = response['posts']       
    db.posts.insert_many(posts)
    print 'Inserted {number} posts'.format(number=len(posts))
    offset += 100
    if offset >= found:
        break
    print 'We have {left} posts left to go'.format(left=found-offset)       
    params['offset'] = offset
\$\endgroup\$
4
\$\begingroup\$

Your code follows most of PEP8.

However:

  • Lines should be a maximum of 79 characters. The exception to this is comments, at 72.
  • Operators should have a space either side of them. 2 * 2. The exception to this is to show precedence: 2*2 + 2.
  • You should not have any spaces around the = when passing arguments:

    response = requests.get(url, params= {
    

Your code looks really good otherwise.


It may be worth removing the need to parse key-words to str.format:

print 'found {number} posts'.format(number=found)

It's good, but a bit long-winded.


Also, I would recommend not relying on globals in get_next_page. If you only want to pass found once you can do this:

def get_next_page(found):
    def inner(offset):
        # Same as before
        if int(response['found']) > offset:
            inner(offset + 100)
     return inner

get_next_page(found)(100)

And finally, if you are planning on using this on a very large amount of data, where you would have to use get_next_page a lot, I would recommend not using recursion.

Have a look at sys.getrecursionlimit(). It tells you how many times you can recurse. My limit is 1000, for example.

For more info, look here.

\$\endgroup\$
  • \$\begingroup\$ Can you explain why using inner? \$\endgroup\$ – toy Jul 13 '15 at 23:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.