Checking paginated website for new entries

I'm interested in determining the best way to check a paginated website for new entries. I want to be able to scrape pages 1, 2, 3, ... as necessary to get all updates. However the scraping is fairly network intensive (e.g. I have to recurse into links to get data) so I only want to scrape what is necessary. Performing a diff or set comparison on the last ~1000 entries every N interval is not a good option.

Attempted solutions

• My first solution has been to utilize urlwatch with a custom hook/filter (a CSS selector among other things) to extract the desired items. One serious problem so far as I can tell is that I'm only able to extract the first page. If recent updates happen to extend beyond the first page, they will be missed. This is a problem. There are some other issues with this approach (I only want new entries, not complete diff output, also I don't want to perform a bunch of time-consuming scraping of old entries) but I didn't bother attempting to address those.

• Another option I looked into was using a Webpage --> RSS service on it's own or in combination with something like a RSS --> JSON Webpipe. However these were quickly ruled out as they suffer from the same issue, only one page of results.

• Another option I recently found out about is Kimono Labs. It looked like it might address a lot of pain points (pagination, recurring events, notifications/webhooks for new entries, straightforward API) however one of the particular URLs I'm interested in scraping is not supported. It looked really promising if not for that. I also looked into Scraping Hub's offerings but couldn't figure it out in a timely manner.

• Just to cover my bases, I've used Specto in the past for other purposes but it is not a good fit for this.

Working solution

What follows is a MVP, but I'm interested to know some better ways to solve this. It seems like it would be a recurring problem for which there would be libraries, no? Or some sort of data store that makes it particularly easy to identify new entries as soon as possible?

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""Poll websites for new entries"""

# standard imports
import itertools
import os
import shelve
from collections import deque
from hashlib import sha1

# third-party imports
import bs4
import requests

# dummy functions
do_deep_scrape = lambda x: x
do_something = print

class ScraperOne:
"""Scrape Hacker News."""

def __init__(self):
self.pattern = 'https://news.ycombinator.com/news?p=%d'

def scrape(self):
"""Iterate over entries."""
for page in itertools.count(1):
url = self.pattern % page
soup = get_soup(url)
posts = soup.select('table > tr.athing > td.title > a')
for post in posts:
title = post.text.strip()
path = post.get('href')
if title and path is not None:
yield title, do_deep_scrape(path.strip())

class Config:
"""Store global script configuration values."""

DATABASE = os.path.join(os.environ['HOME'], 'Desktop', 'hackerscraper.db')
MAXLEN = 300
SCRAPERS = [ScraperOne()]

def get_soup(url):
"""Return a BeautifulSoup instance for a given URL."""

return bs4.BeautifulSoup(requests.get(url).text)

def main():
"""Start application."""

with shelve.open(Config.DATABASE) as database:
for scraper in Config.SCRAPERS:
# get hash of URL pattern
key = sha1(scraper.pattern.encode()).hexdigest()

# get pre-existing entries stored in DB
existing = database.setdefault(key, deque(maxlen=Config.MAXLEN))

count = 0
newentries = []
for entry in itertools.islice(scraper.scrape(), Config.MAXLEN):
# check whether entry has been seen before
if entry in existing:
count += 1
else:
newentries.append(entry)

# break upon reaching threshold
# i.e. seen N items in pre-existing queue
if count >= 3:
break

# do something with the new entries...
# e.g. notify user or send off to other process
for entry in newentries:
do_something(entry)

# add new entries to pre-existing queue
existing.extendleft(newentries)

# write to database/"shelf"
database[key] = existing

if __name__ == '__main__':
main()


I'm not interested in scraping Hacker News or sites with public APIs. The aforementioned code was just an easy example. Also, the sites in question do not have upvote-style curation but are rather just chronological posts/entries akin to a blog.