I'm interested in determining the best way to check a paginated website for new entries. I want to be able to scrape pages 1, 2, 3, ... as necessary to get all updates. However the scraping is fairly network intensive (e.g. I have to recurse into links to get data) so I only want to scrape what is necessary. Performing a
diff or set comparison on the last ~1000 entries every N interval is not a good option.
My first solution has been to utilize urlwatch with a custom hook/filter (a CSS selector among other things) to extract the desired items. One serious problem so far as I can tell is that I'm only able to extract the first page. If recent updates happen to extend beyond the first page, they will be missed. This is a problem. There are some other issues with this approach (I only want new entries, not complete
diffoutput, also I don't want to perform a bunch of time-consuming scraping of old entries) but I didn't bother attempting to address those.
Another option I looked into was using a Webpage --> RSS service on it's own or in combination with something like a RSS --> JSON Webpipe. However these were quickly ruled out as they suffer from the same issue, only one page of results.
Another option I recently found out about is Kimono Labs. It looked like it might address a lot of pain points (pagination, recurring events, notifications/webhooks for new entries, straightforward API) however one of the particular URLs I'm interested in scraping is not supported. It looked really promising if not for that. I also looked into Scraping Hub's offerings but couldn't figure it out in a timely manner.
Just to cover my bases, I've used Specto in the past for other purposes but it is not a good fit for this.
What follows is a MVP, but I'm interested to know some better ways to solve this. It seems like it would be a recurring problem for which there would be libraries, no? Or some sort of data store that makes it particularly easy to identify new entries as soon as possible?
#!/usr/bin/python3 # -*- coding: utf-8 -*- """Poll websites for new entries""" # standard imports import itertools import os import shelve from collections import deque from hashlib import sha1 # third-party imports import bs4 import requests # dummy functions do_deep_scrape = lambda x: x do_something = print class ScraperOne: """Scrape Hacker News.""" def __init__(self): self.pattern = 'https://news.ycombinator.com/news?p=%d' def scrape(self): """Iterate over entries.""" for page in itertools.count(1): url = self.pattern % page soup = get_soup(url) posts = soup.select('table > tr.athing > td.title > a') for post in posts: title = post.text.strip() path = post.get('href') if title and path is not None: yield title, do_deep_scrape(path.strip()) class Config: """Store global script configuration values.""" DATABASE = os.path.join(os.environ['HOME'], 'Desktop', 'hackerscraper.db') MAXLEN = 300 SCRAPERS = [ScraperOne()] def get_soup(url): """Return a BeautifulSoup instance for a given URL.""" return bs4.BeautifulSoup(requests.get(url).text) def main(): """Start application.""" with shelve.open(Config.DATABASE) as database: for scraper in Config.SCRAPERS: # get hash of URL pattern key = sha1(scraper.pattern.encode()).hexdigest() # get pre-existing entries stored in DB existing = database.setdefault(key, deque(maxlen=Config.MAXLEN)) count = 0 newentries =  for entry in itertools.islice(scraper.scrape(), Config.MAXLEN): # check whether entry has been seen before if entry in existing: count += 1 else: newentries.append(entry) # break upon reaching threshold # i.e. seen N items in pre-existing queue if count >= 3: break # do something with the new entries... # e.g. notify user or send off to other process for entry in newentries: do_something(entry) # add new entries to pre-existing queue existing.extendleft(newentries) # write to database/"shelf" database[key] = existing if __name__ == '__main__': main()
I'm not interested in scraping Hacker News or sites with public APIs. The aforementioned code was just an easy example. Also, the sites in question do not have upvote-style curation but are rather just chronological posts/entries akin to a blog.