Download and concatenate text files from multiple API endpoints

Question

Background

Chronicling America is an archive of digitized US newspapers hosted by the Library of Congress. The site allows programmatic access of its resources through an API, one part of which is JSON representations of each newspaper. Newspapers are composed of issues (denoted by date and edition number), and issues are composed of pages, which have several representations. I am interested in the ocr.txt representation.

Each newspaper is represented in the API by URL-linked JSON representations:

newspaper.json -> issue.json -> page.json -> ocr.txt`

Example URL sequence to arrive at a single txt page of a newspaper issue:

http://chroniclingamerica.loc.gov/lccn/sn84026994.json
http://chroniclingamerica.loc.gov/lccn/sn84026994/1865-08-21/ed-1.json
http://chroniclingamerica.loc.gov/lccn/sn84026994/1865-08-21/ed-1/seq-1.json
http://chroniclingamerica.loc.gov/lccn/sn84026994/1865-08-21/ed-1/seq-1/ocr.txt

where

• sn84026994 = newspaper lccn
• 1865-08-21/ed-1 = issue
• seq-1 = page
• ocr.txt = plain text representation of page

Design

This Python module is designed to traverse the url links for a given newspaper and range of issues (denoted by the date of issue).

The two functions designed to be imported and interact with calling code are disp_newspaper() and dwnld_newspaper():

1. disp_newspaper() retrieves some data about the newspaper and prints it on the terminal to assist the user in deciding on arguments for dwnld_newspaper()
2. dwnld_newspaper() downloads and assembles the ocr.txt files for each desired issue of a given newspaper into a dict {'date': 'text'}

Questions

As I am somewhat of a beginner in coding systems:

1. Is this code easy to understand? If not, how can it be refactored?
2. Does it follow style guides?
3. Have I missed a simple strategy for traversing the API endpoints and obtaining the end result of a single string of all .txt files for a given newspaper and issue?

Callable Code

import json
import os

from datetime import datetime
from urllib.request import Request, urlopen
from urllib.error import URLError

def disp_newspaper(url):
    """Displays information and issues available for a given newspaper

    Parameters: url -> url of JSON file for newspaper: str
    Returns:    newspaper_json -> dict representation of JSON from http
                request: dict"""

    try:
        newspaper_json = get_json(url)
    except ValueError as e:
        return e

    newspaper_string = ('{} | Library of Congress No.: {} | {}\nPublished '
                        'from {} to {} by {}').format(
                           newspaper_json['name'],
                           newspaper_json['lccn'],
                           newspaper_json['place_of_publication'],
                           newspaper_json['start_year'],
                           newspaper_json['end_year'],
                           newspaper_json['publisher'])

    issues_string = ('Number of Issues Downloadable: {}\nFirst issue: {}\n'
                     'Last Issue: {}\n').format(
                        len(newspaper_json['issues']),
                        newspaper_json['issues'][0]['date_issued'],
                        newspaper_json['issues'][-1]['date_issued'])
    print(newspaper_string)
    print('\n', end='')
    print(issues_string)

def dwnld_newspaper(url, start_date, end_date):
    """Downloads OCR text of a newspaper from chroniclingamerica.loc.gov by
    parsing the .json representation using the exposed API. Traverses
    the json from the newspaper .json url to each page and composes them into
    a dict of issues where {'date': 'issue text'}

    Params: url -> str: base url of newspaper. Ends in .json
            start_date -> date: date(year, month, day)
                          represents the first issue to download
            end_date   -> date: date(year, month, day)
                          represents the last issue to download
    Return: newspaper_issues -> dict: {'date': 'issue text'}"""

    newspaper_issues = {}

    # Terminal UI Print statements
    print('start date:', start_date)
    print('end date:', end_date)

    # Interface
    print('Getting issues:')

    # TODO: handle more than 2 issue editions for same date
    try:
        for issue in get_json(url)['issues']:
            if (parse_date(issue['date_issued']) >= start_date and
                    parse_date(issue['date_issued']) <= end_date):
                # Check for multiple issues with same date
                if issue['date_issued'] not in newspaper_issues:
                    print(issue['date_issued'])
                    newspaper_issues[issue['date_issued']] = \
                        assemble_issue(issue['url'])
                # append to differentiate second edition of same date
                else:
                    print(issue['date_issued'] + '-ed-2')
                    newspaper_issues[issue['date_issued'] + '-ed-2'] = \
                        assemble_issue(issue['url'])
        return newspaper_issues # dict {'date_issued': 'alltextforallpages'}

    except ValueError as e:
        return e

Supporting Functions

The above two functions rely on other functions which are designed to modularize the code.

def validate_chronam_url(url):
    """Naive check. Ensures that the url goes to a
    chroniclingamerica.loc.gov newspaper
    and references the .json representation

    Params: url -> url of JSON file for newspaper to download: str
    Return: Boolean"""

    domain_chk = 'chroniclingamerica.loc.gov/lccn/sn'
    json_chk = '.json'
    if domain_chk in url and json_chk in url:
        return True
    else:
        return False


def get_json(url):
    """Downloads json from url from chronliclingamerica.loc.gov 
    and saves as a Python dict.

    Parameters: url -> url of JSON file for newspaper 
                       to download: str
    Returns:    json_dict -> dict representation of 
                             JSON from http request: dict"""

    r = Request(url)

    # Catch non-chronam urls
    if validate_chronam_url(url) is not True:
        raise ValueError('Invalid url for chroniclingamerica.loc.gov'
                         'OCR newspaper (url must end in .json)')
    try:
        data = urlopen(r)
    except URLError as e:
        if hasattr(e, 'reason'):
            print('We failed to reach a server.')
            print('Reason: ', e.reason)
            print('url: ', url)
        elif hasattr(e, 'code'):
            print('The server couldn\'t fulfill the request.')
            print('Error code: ', e.code)
            print('url: ', url)
    else:
        # read().decode('utf-8') is necessary for Python 3.4
        json_dict = json.loads(data.read(), encoding='utf-8')
        return json_dict


def get_txt(url):
    """Downloads txt from url from chroniclingamerica.loc.gov 
       and saves as python str.

    Relies on valid url supplied by get_json()

    Parameters: url -> url for OCR text returned by get_json(): str
    Returns:    retrieved_txt -> OCR text: str"""

    # TODO: return lists of missing & failed pages
    missing_pages = []
    failed_pages = []

    r = Request(url)
    try:
        data = urlopen(r)
    except URLError as e:
        if hasattr(e, 'reason'):
            print('We failed to reach a server.')
            print('Reason: ', e.reason)
            print('url: ', url)
            retrieved_txt = ('Likely Missing Page: Not digitized,'
                            'published')
            missing_pages.append(url)

        elif hasattr(e, 'code'):
            print('The server couldn\'t fulfill the request.')
            print('Error code: ', e.code)
            print('url: ', url)
            retrieved_txt = 'Server didn\'t return any text'
            failed_pages.append(url)
    else:
        retrieved_txt = data.read().decode('utf-8')
    return retrieved_txt

def dwnld_page(url):  # url of page
    """Downloads the OCR text of a newspaper page. Relies on valid 
       url from assemble_issue()

    Params: url -> url of OCR text of page: str
    Return: txt -> OCR text of a newspaper page: str"""

    txt_url = get_json(url)['text']
    txt = get_txt(txt_url)
    return txt

def assemble_issue(url):  # url of issue
    """Assembles the OCR text for each page of a newspaper.
    Relies on valid url from dwnld_newspaper()

    Params: url -> url of newspaper issue: str
    Return: txt -> OCR text of all pages in newspaper: str"""

    issue_string = ''
    for page in get_json(url)['pages']:
        issue_string += dwnld_page(page['url'])
    return issue_string  # str 'alltextforallpages'


def parse_date(datestring):
    """Converts YYYY-MM-DD string into date object

    Params: date -> str: 'YYYY-MM-DD'
    Return: return_date -> date"""

    date_fmt_str = '%Y-%m-%d'
    return_date = datetime.strptime(datestring, date_fmt_str).date()
    return return_date

def lccn_to_disk(dir_name, downloaded_issue):
   """Saves a dict of downloaded issues to disk. Creates a directory:

   dir_name
     |--key1.txt
     |--key2.txt
     +--key3.txt

   Params: dir_name -> str: name of created directory for data
           downloaded_issue -> dict: {'YYYY-MM-DD': 'string 
                                                     of txt'}"""

   if not os.path.exists(dir_name):
       os.makedirs(dir_name)
   for date, text in downloaded_issue.items():
       with open(os.path.join(dir_name, date + '.txt'), 'w') as f:
           f.write(text)
   return

Answer 1

Some of the things that may make your code better:

• you can pass the whole newspaper object into the format string and use a multi-line string:

  newspaper_string = """
      {newspaper.name} | Library of Congress No.: {newspaper.lccn} | {newspaper.place_of_publication}
      Published from {newspaper.start_year} to {newspaper.end_year} by {newspaper.publisher}
  """.format(newspaper=newspaper_json)
  

  Or, unpack the dictionary into keyword arguments:

  newspaper_string = """
      {name} | Library of Congress No.: {lccn} | {place_of_publication}
      Published from {start_year} to {end_year} by {publisher}
  """.format(**newspaper_json)
  

  Same goes for the issues_string string definition.

• you can further improve that by using f-strings (Python 3.6+)

• there is not a lot of reasoning behind not using the full words instead of abbreviated for the function names - for instance, download_newspaper instead of dwnld_newspaper
• the parse_date() is called twice per single iteration of the for issue in get_json(url)['issues'] loop. You can do it once and save the result into a variable

• validate_chronam_url body may be simplified and improved to:

  domain_matches = 'chroniclingamerica.loc.gov/lccn/sn' in url
  is_json = '.json' in url
  
  return domain_matches and is_json
  

• you can use json.load(data, ...) instead of json.loads(data.read(), ...)

• you can use str.join() for the assemble_issue function:

  return ''.join(dwnld_page(page['url']) for page in get_json(url)['pages'])
  

• you don't need a return in the lccn_to_disk function

Some other thoughts:

• not sure that returning an exception instance in case things go wrong is a good idea. Either let them fail and propagate or handle - e.g. log the error and exit
• consider using requests which would not only simplify the JSON response parsing part (there is a response.json() built-in method), but would also make the subsequent requests faster if you would use a single session instance
• it feels like having a Newspaper class might be a good idea. At least you would have a nice way to hide the string representation of a newspaper representation string inside the __repr__ magic method
• if performance is a concern, aside from requests.Session, you may improve on JSON parsing by switching to ujson parser