Improved version of "Let's read a random Goodreads book..."

Question

Since my last post "Let's read a random Goodreads book in an optimal way," I have added new features to my program.

As far as this question is concerned: if you haven't already, it may help to read the last post for context.

Changes

Since the last post, I have made the following changes:

• In to_int(rating_count), added if-else to catch if rating count is already in integer form - fixing error caused by .split()[0] on an integer.
• In format_title(book_title), added code to return title with CSV formatting (double quotes around title, if title already had double quotes - "escaped" with ""), and added if-statement to remove occasional extra space in title (caused by removal of series identification).
• In get_book_rating_count(soup), added try-except to catch occasional AttributeError caused by missing rating count.
• Added command line parsing, here's the "help" print out:

usage: Booky-v4.py [-h] [-v {0,1,2,3}] [-mr MIN_RATING] filename

Gather random book data from Goodreads and append it to a file in CSV format, until the program is manually closed or until a connection issue.     

example command: Booky-v4.py -v3 -mr /path/to/file.csv

positional arguments:
  filename              output file, only supports FULL path (no tilde, etc.)

optional arguments:
  -h, --help            show this help message and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        specify verbosity level, default = 2        
                        0 = no output       
                        1 = continually print line count after appending to file
                        2 = after keyboard interrupt, print count of lines added since program start and program run time       
                        3 = both verbosity options 1 and 2
  -mr MIN_RATING, --min-rating MIN_RATING
                        specify the minimum accepted rating, default = 30

Another example of command parsing:

$ python3 -OO Booky-v4.py -v2 --min-rating 400 /path/to/file.csv

• Added option to print program run time (in real time) and total count of lines added - upon keyboard interrupt (manual exit) - if --verbosity is 2 or 3.
• Changed so program appends book data to a CSV file (in correct CSV format) instead of printing it in terminal.
• Added FileNotFoundError to main try-except block, along with os.path.isfile(FILE_PATH) inside of the while loop to catch file deletion while the program is running.
• Added option to continually print total line count after appendage (always prints on the same line thanks to sys.stdout.write("\033[F")) - if --verbosity is 1 or 3.

Code

import argparse
from bs4 import BeautifulSoup as bs
import os.path
import requests
import re
import sys
import time

URL = "https://www.goodreads.com/book/random"


def to_int(rating_count):
    """ Return rating count as a int """
    if isinstance(rating_count, int):
        return rating_count
    else:
        rating_count = rating_count.split()[0]
        if ',' in rating_count:
            return int(rating_count.replace(',', ''))
        return int(rating_count)


def format_title(book_title):
    """ 
        Return formatted book title:
            "Actual Book Title! (Series identifier #3)" -> "Actual Book Title!"
            "History of 4D Fish Slapping [Silly, Part 4]" -> "History of 4D Fish Slapping"

        Along with CSV formatting
    """
    book_title = ' '.join(book_title.split()).replace('&', '&')
    book_title = re.sub(r'(\(|\[)(.*)(\)|\])','', book_title)
    book_title = '"' + book_title.replace('"', '""') + '"'
    if book_title[-2:-1] == ' ':
        book_title = book_title[:-2] + '"'
    return book_title


def get_html_source():
    """ Return html source """
    html_source = requests.get(URL).text
    return bs(html_source, 'html.parser')


def get_book_rating_count(soup):
    """ Return book rating count """
    try:
        book_rating_count = soup.find('span', attrs={'class', 'value-title'}).get_text()
        return book_rating_count
    except AttributeError:
        """ Attribute error, the rating count is missing """
        return -1


def get_book_title(soup):
    """ Return book title"""
    return soup.find('h1', attrs={'class': 'bookTitle'}).get_text()


def get_book_pages(soup):
    """ Return book pages """
    book_pages = soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()
    if " pages" in book_pages:
        return book_pages.replace(' pages', '')
    elif " page" in book_pages:
        return book_pages.replace(' page', '')
    return book_pages


def get_book_rating(soup):
    """ Return book rating """
    return soup.find('span', attrs={'itemprop': 'ratingValue'}).get_text()


def main():
    """ Command line parsing, with specifiable verbosity, minimum rating, and required file name """
    parser = argparse.ArgumentParser(description="Gather random book data from Goodreads and append it to a file in CSV format, until the program is manually closed or until a connection issue.\
        \n\nexample command: Booky-v4.py -v3 -mr /path/to/file.csv",
        formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument("-v", "--verbosity", type=int, choices=[0,1,2,3],
        help="specify verbosity level, default = 2\
        \n0 = no output\
        \n1 = continually print line count after appending to file\
        \n2 = after keyboard interrupt, print count of lines added since program start and program run time\
        \n3 = both verbosity options 1 and 2", default=2)
    parser.add_argument("-mr", "--min-rating", type=int, default=30,
        help="specify the minimum accepted rating, default = 30")
    parser.add_argument("filename", help="output file, only supports FULL path (no tilde, etc.)")
    args = parser.parse_args()

    FILE_PATH = args.filename
    MIN_RATING_COUNT = args.min_rating

    try:
        if args.verbosity in (2, 3):
            start_time = time.time()
            with open(FILE_PATH, 'r') as file:
                init_filelength = sum(1 for i in file)

        while True:
            soup = get_html_source()
            book_rating_count = get_book_rating_count(soup)

            if to_int(book_rating_count) >= MIN_RATING_COUNT:
                try:
                    book_pages = get_book_pages(soup)
                    book_title = format_title(get_book_title(soup))
                    book_rating = get_book_rating(soup)
                    """ Re-check if file exists to catch deletion while program is running """
                    if book_pages != "0" and os.path.isfile(FILE_PATH):
                        with open(FILE_PATH, 'a') as file:
                            file.write(book_title + ','  + book_pages + ',' + book_rating + '\n')

                        if args.verbosity in (1, 3):
                            sys.stdout.write("\033[F")
                            with open(FILE_PATH, 'r') as file:
                                print("\nLine count:", sum(1 for i in file), end='')
                    elif not os.path.isfile(FILE_PATH):
                        raise FileNotFoundError

                except AttributeError:
                    continue

    except FileNotFoundError:
        print("File '", FILE_PATH, "' does not exist, or has been deleted.", sep='')

    except KeyboardInterrupt:
        if args.verbosity in (2, 3):
            with open(FILE_PATH, 'r') as file:
                lines_added = sum(1 for i in file) - init_filelength
            print("\nLines added: ", lines_added, " (since program start)", sep='')
            print("Run time: ", round(time.time() - start_time, 3), " seconds (real)", sep='')

if __name__ == '__main__':
    try:
        main()

    except ConnectionResetError:
        print("\nConnection reset by peer, exiting program.")
        pass

---

The code versions are on GitHub - version 2 (from last post) and version 4 (current code) - if that helps with anything.

---

I am new to this whole argparse stuff - let me know if I can improve anything in that department. I also attempted to document my code effectively, but I could always do better.

How can I optimize my code, both by improving the code's logic/structure or by improving performance?

Answer 1

Bug/Problem

When you extract a book rating, you are using the following method:

book_rating_count = soup.find('span', attrs={'class', 'value-title'}).get_text()

Here, you meant to find an element with value-title class - there should be a : instead of a , in the attrs keyword argument definition:

book_rating_count = soup.find('span', attrs={'class': 'value-title'}).get_text()

Switching to CSS selectors

I personally prefer to use CSS selectors in BeautifulSoup wherever possible - they are generally more concise and easy, especially if matching by class value(s). E.g.:

soup.find('h1', attrs={'class': 'bookTitle'}).get_text()

will be replaced with:

soup.select_one('h1.bookTitle').get_text()

This:

book_pages = soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()

will be replaced with:

book_pages = soup.select_one('span[itemprop=numberOfPages]').get_text()

Performance Improvements

There are couple of things that may be improved performance-wise:

• since you are issuing multiple requests to the same domain, reusing the same requests.Session() instance will result into performance boost:

  if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

• there are several parsers that can be used with BeautifulSoup. If you are interested in making HTML parsing faster, switch from html.parser to lxml:

  bs(html_source, 'lxml')
  

  Note: requires lxml to be installed

• you may also utilize the SoupStrainer class to parse only a part of the HTML document. This will help to save time on not parsing the things you don't need. In your case, I think you may only parse the "meta" part of the page:

  parse_only = SoupStrainer(id="metacol")  # don't forget to import SoupStrainer
  soup = bs(html_source, 'lxml', parse_only=parse_only)
  

Code Style Notes

• instead of using single line strings with \n, use a multi-line string:

  parser.add_argument("-v", "--verbosity", 
                      type=int, choices=[0,1,2,3], default=2
                      help="""
      specify verbosity level, default = 2
          0 = no output
          1 = continually print line count after appending to file
          2 = after keyboard interrupt, print count of lines added since program start and program run time
          3 = both verbosity options 1 and 2""")
  

• sum(1 for i in file) - here i is not used. In Python, there is a special "convention" for this kind of throw-away variables - an underscore:

  sum(1 for _ in file)
  

• don't use triple-quoted strings as comments, replace:

  """ Re-check if file exists to catch deletion while program is running """
  

  with:

  # Re-check if file exists to catch deletion while program is running 
  

• there are also some PEP8 code style violations. Running flake8 or/and pylint static code analysis tools against the code will help to catch most of them.

Code Organization Notes

Note how get_book_rating_count(), get_book_title(), get_book_pages() and get_book_rating() functions all accept the soup BeautifulSoup instance argument. This is a "code smell" indicating that having a class with the soup instance variable would probably be a better alternative. You can find more on this topic here:

• Start Writing More Classes

And, if you would run a complexity checker, like mccabe, against your script, you would see that main() function is overly complex and "heavy" - think of dividing it into logical pieces. Moving the "parsing command-line arguments" part can be the first candidate for extraction.