# Scraping SEDE query results with caching

I use this script to scrape the results of a SEDE page and return as a BeautifulSoup object.

A small twist is that if I don't use a SEDE query manually in the browser for a few days, then non-interactive downloads get empty results. (I suspect there is a robot test there.)

import logging
import os

import requests
from bs4 import BeautifulSoup

BASE_DIR = os.path.dirname(__file__)
CACHE_DIR = os.path.join(BASE_DIR, '.cache')

def fetch_sede_soup(label, url):
def is_valid(soup):
for script in soup.findAll('script'):
if 'resultSets' in script.text:
return True
return False

if not os.path.isdir(CACHE_DIR):
os.mkdir(CACHE_DIR)

logging.info('fetching {} as {}'.format(label, url))
html = requests.get(url).text
soup = BeautifulSoup(html)

cache_path = os.path.join(CACHE_DIR, '{}.html'.format(label))
debug_cache_path = os.path.join(CACHE_DIR, '{}-debug.html'.format(label))

if is_valid(soup):
logging.info('updating cache')
with open(cache_path, 'w') as fh:
fh.write(html)
return soup

with open(debug_cache_path, 'w') as fh:
fh.write(html)

logging.warning('result not valid')
if os.path.exists(cache_path):
logging.info('using previous cache')
with open(cache_path) as fh:
return BeautifulSoup(fh)


Can this be written better?

In case it helps, here are some sample output files:

• Single responsibility concern – Your function is name fetch_sede_soup() and all actions are related to this. However it still creates the CACHE_DIR if needed, write output files, verifies validity in inner function. This could be separated into more functions, although it is not extremely large and hard to get the overview of.
One point I would at least add a new function is related to the return soup near the middle. Code after this seems to be error handling, but it is somewhat hidden by the return and as such it is not very clear. If you moved the remaining part into a function you would see a return soup followed by a else: clause calling the error handling.
• Add error handling around mkdir – Especially around external os calls, like mkdir, I would add try ... except code to handle potential error situations. This code related to creating the CACHE_DIR could possibly be done before this method is called as part of initialization.