I'm interested in determining the best way to check a paginated website for new entries. I want to be able to scrape pages 1, 2, 3, ... as necessary to get all updates. However the scraping is fairly network intensive (e.g. I have to recurse into links to get data) so I only want to scrape what is necessary. Performing a diff or set comparison on the last ~1000 entries every N interval is not a good option.

Attempted solutions

  • My first solution has been to utilize urlwatch with a custom hook/filter (a CSS selector among other things) to extract the desired items. One serious problem so far as I can tell is that I'm only able to extract the first page. If recent updates happen to extend beyond the first page, they will be missed. This is a problem. There are some other issues with this approach (I only want new entries, not complete diff output, also I don't want to perform a bunch of time-consuming scraping of old entries) but I didn't bother attempting to address those.

  • Another option I looked into was using a Webpage --> RSS service on it's own or in combination with something like a RSS --> JSON Webpipe. However these were quickly ruled out as they suffer from the same issue, only one page of results.

  • Another option I recently found out about is Kimono Labs. It looked like it might address a lot of pain points (pagination, recurring events, notifications/webhooks for new entries, straightforward API) however one of the particular URLs I'm interested in scraping is not supported. It looked really promising if not for that. I also looked into Scraping Hub's offerings but couldn't figure it out in a timely manner.

  • Just to cover my bases, I've used Specto in the past for other purposes but it is not a good fit for this.

Working solution

What follows is a MVP, but I'm interested to know some better ways to solve this. It seems like it would be a recurring problem for which there would be libraries, no? Or some sort of data store that makes it particularly easy to identify new entries as soon as possible?

# -*- coding: utf-8 -*-

"""Poll websites for new entries"""

# standard imports
import itertools
import os
import shelve
from collections import deque
from hashlib import sha1

# third-party imports
import bs4
import requests

# dummy functions
do_deep_scrape = lambda x: x
do_something = print

class ScraperOne:
    """Scrape Hacker News."""

    def __init__(self):
        self.pattern = 'https://news.ycombinator.com/news?p=%d'

    def scrape(self):
        """Iterate over entries."""
        for page in itertools.count(1):
            url = self.pattern % page
            soup = get_soup(url)
            posts = soup.select('table > tr.athing > td.title > a')
            for post in posts:
                title = post.text.strip()
                path = post.get('href')
                if title and path is not None:
                    yield title, do_deep_scrape(path.strip())

class Config:
    """Store global script configuration values."""

    DATABASE = os.path.join(os.environ['HOME'], 'Desktop', 'hackerscraper.db')
    MAXLEN = 300
    SCRAPERS = [ScraperOne()]

def get_soup(url):
    """Return a BeautifulSoup instance for a given URL."""

    return bs4.BeautifulSoup(requests.get(url).text)

def main():
    """Start application."""

    with shelve.open(Config.DATABASE) as database:
        for scraper in Config.SCRAPERS:
            # get hash of URL pattern
            key = sha1(scraper.pattern.encode()).hexdigest()

            # get pre-existing entries stored in DB
            existing = database.setdefault(key, deque(maxlen=Config.MAXLEN))

            count = 0
            newentries = []
            for entry in itertools.islice(scraper.scrape(), Config.MAXLEN):
                # check whether entry has been seen before
                if entry in existing:
                    count += 1

                # break upon reaching threshold
                # i.e. seen N items in pre-existing queue
                if count >= 3:

            # do something with the new entries...
            # e.g. notify user or send off to other process
            for entry in newentries:

            # add new entries to pre-existing queue

            # write to database/"shelf"
            database[key] = existing

if __name__ == '__main__':

I'm not interested in scraping Hacker News or sites with public APIs. The aforementioned code was just an easy example. Also, the sites in question do not have upvote-style curation but are rather just chronological posts/entries akin to a blog.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.