Since my last post "Let's read a random Goodreads book in an optimal way," I have added new features to my program.

As far as this question is concerned: if you haven't already, it may help to read the last post for context.

Changes

Since the last post, I have made the following changes:

  • In to_int(rating_count), added if-else to catch if rating count is already in integer form - fixing error caused by .split()[0] on an integer.
  • In format_title(book_title), added code to return title with CSV formatting (double quotes around title, if title already had double quotes - "escaped" with ""), and added if-statement to remove occasional extra space in title (caused by removal of series identification).
  • In get_book_rating_count(soup), added try-except to catch occasional AttributeError caused by missing rating count.
  • Added command line parsing, here's the "help" print out:
usage: Booky-v4.py [-h] [-v {0,1,2,3}] [-mr MIN_RATING] filename

Gather random book data from Goodreads and append it to a file in CSV format, until the program is manually closed or until a connection issue.     

example command: Booky-v4.py -v3 -mr /path/to/file.csv

positional arguments:
  filename              output file, only supports FULL path (no tilde, etc.)

optional arguments:
  -h, --help            show this help message and exit
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        specify verbosity level, default = 2        
                        0 = no output       
                        1 = continually print line count after appending to file
                        2 = after keyboard interrupt, print count of lines added since program start and program run time       
                        3 = both verbosity options 1 and 2
  -mr MIN_RATING, --min-rating MIN_RATING
                        specify the minimum accepted rating, default = 30

Another example of command parsing:

$ python3 -OO Booky-v4.py -v2 --min-rating 400 /path/to/file.csv
  • Added option to print program run time (in real time) and total count of lines added - upon keyboard interrupt (manual exit) - if --verbosity is 2 or 3.
  • Changed so program appends book data to a CSV file (in correct CSV format) instead of printing it in terminal.
  • Added FileNotFoundError to main try-except block, along with os.path.isfile(FILE_PATH) inside of the while loop to catch file deletion while the program is running.
  • Added option to continually print total line count after appendage (always prints on the same line thanks to sys.stdout.write("\033[F")) - if --verbosity is 1 or 3.

Code

import argparse
from bs4 import BeautifulSoup as bs
import os.path
import requests
import re
import sys
import time

URL = "https://www.goodreads.com/book/random"


def to_int(rating_count):
    """ Return rating count as a int """
    if isinstance(rating_count, int):
        return rating_count
    else:
        rating_count = rating_count.split()[0]
        if ',' in rating_count:
            return int(rating_count.replace(',', ''))
        return int(rating_count)


def format_title(book_title):
    """ 
        Return formatted book title:
            "Actual Book Title! (Series identifier #3)" -> "Actual Book Title!"
            "History of 4D Fish Slapping [Silly, Part 4]" -> "History of 4D Fish Slapping"

        Along with CSV formatting
    """
    book_title = ' '.join(book_title.split()).replace('&', '&')
    book_title = re.sub(r'(\(|\[)(.*)(\)|\])','', book_title)
    book_title = '"' + book_title.replace('"', '""') + '"'
    if book_title[-2:-1] == ' ':
        book_title = book_title[:-2] + '"'
    return book_title


def get_html_source():
    """ Return html source """
    html_source = requests.get(URL).text
    return bs(html_source, 'html.parser')


def get_book_rating_count(soup):
    """ Return book rating count """
    try:
        book_rating_count = soup.find('span', attrs={'class', 'value-title'}).get_text()
        return book_rating_count
    except AttributeError:
        """ Attribute error, the rating count is missing """
        return -1


def get_book_title(soup):
    """ Return book title"""
    return soup.find('h1', attrs={'class': 'bookTitle'}).get_text()


def get_book_pages(soup):
    """ Return book pages """
    book_pages = soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()
    if " pages" in book_pages:
        return book_pages.replace(' pages', '')
    elif " page" in book_pages:
        return book_pages.replace(' page', '')
    return book_pages


def get_book_rating(soup):
    """ Return book rating """
    return soup.find('span', attrs={'itemprop': 'ratingValue'}).get_text()


def main():
    """ Command line parsing, with specifiable verbosity, minimum rating, and required file name """
    parser = argparse.ArgumentParser(description="Gather random book data from Goodreads and append it to a file in CSV format, until the program is manually closed or until a connection issue.\
        \n\nexample command: Booky-v4.py -v3 -mr /path/to/file.csv",
        formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument("-v", "--verbosity", type=int, choices=[0,1,2,3],
        help="specify verbosity level, default = 2\
        \n0 = no output\
        \n1 = continually print line count after appending to file\
        \n2 = after keyboard interrupt, print count of lines added since program start and program run time\
        \n3 = both verbosity options 1 and 2", default=2)
    parser.add_argument("-mr", "--min-rating", type=int, default=30,
        help="specify the minimum accepted rating, default = 30")
    parser.add_argument("filename", help="output file, only supports FULL path (no tilde, etc.)")
    args = parser.parse_args()

    FILE_PATH = args.filename
    MIN_RATING_COUNT = args.min_rating

    try:
        if args.verbosity in (2, 3):
            start_time = time.time()
            with open(FILE_PATH, 'r') as file:
                init_filelength = sum(1 for i in file)

        while True:
            soup = get_html_source()
            book_rating_count = get_book_rating_count(soup)

            if to_int(book_rating_count) >= MIN_RATING_COUNT:
                try:
                    book_pages = get_book_pages(soup)
                    book_title = format_title(get_book_title(soup))
                    book_rating = get_book_rating(soup)
                    """ Re-check if file exists to catch deletion while program is running """
                    if book_pages != "0" and os.path.isfile(FILE_PATH):
                        with open(FILE_PATH, 'a') as file:
                            file.write(book_title + ','  + book_pages + ',' + book_rating + '\n')

                        if args.verbosity in (1, 3):
                            sys.stdout.write("\033[F")
                            with open(FILE_PATH, 'r') as file:
                                print("\nLine count:", sum(1 for i in file), end='')
                    elif not os.path.isfile(FILE_PATH):
                        raise FileNotFoundError

                except AttributeError:
                    continue

    except FileNotFoundError:
        print("File '", FILE_PATH, "' does not exist, or has been deleted.", sep='')

    except KeyboardInterrupt:
        if args.verbosity in (2, 3):
            with open(FILE_PATH, 'r') as file:
                lines_added = sum(1 for i in file) - init_filelength
            print("\nLines added: ", lines_added, " (since program start)", sep='')
            print("Run time: ", round(time.time() - start_time, 3), " seconds (real)", sep='')

if __name__ == '__main__':
    try:
        main()

    except ConnectionResetError:
        print("\nConnection reset by peer, exiting program.")
        pass

The code versions are on GitHub - version 2 (from last post) and version 4 (current code) - if that helps with anything.


I am new to this whole argparse stuff - let me know if I can improve anything in that department. I also attempted to document my code effectively, but I could always do better.

How can I optimize my code, both by improving the code's logic/structure or by improving performance?

up vote 5 down vote accepted

Bug/Problem

When you extract a book rating, you are using the following method:

book_rating_count = soup.find('span', attrs={'class', 'value-title'}).get_text()

Here, you meant to find an element with value-title class - there should be a : instead of a , in the attrs keyword argument definition:

book_rating_count = soup.find('span', attrs={'class': 'value-title'}).get_text()

Switching to CSS selectors

I personally prefer to use CSS selectors in BeautifulSoup wherever possible - they are generally more concise and easy, especially if matching by class value(s). E.g.:

soup.find('h1', attrs={'class': 'bookTitle'}).get_text()

will be replaced with:

soup.select_one('h1.bookTitle').get_text()

This:

book_pages = soup.find('span', attrs={'itemprop': 'numberOfPages'}).get_text()

will be replaced with:

book_pages = soup.select_one('span[itemprop=numberOfPages]').get_text()

Performance Improvements

There are couple of things that may be improved performance-wise:

  • since you are issuing multiple requests to the same domain, reusing the same requests.Session() instance will result into performance boost:

    if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

  • there are several parsers that can be used with BeautifulSoup. If you are interested in making HTML parsing faster, switch from html.parser to lxml:

    bs(html_source, 'lxml')
    

    Note: requires lxml to be installed

  • you may also utilize the SoupStrainer class to parse only a part of the HTML document. This will help to save time on not parsing the things you don't need. In your case, I think you may only parse the "meta" part of the page:

    parse_only = SoupStrainer(id="metacol")  # don't forget to import SoupStrainer
    soup = bs(html_source, 'lxml', parse_only=parse_only)
    

Code Style Notes

  • instead of using single line strings with \n, use a multi-line string:

    parser.add_argument("-v", "--verbosity", 
                        type=int, choices=[0,1,2,3], default=2
                        help="""
        specify verbosity level, default = 2
            0 = no output
            1 = continually print line count after appending to file
            2 = after keyboard interrupt, print count of lines added since program start and program run time
            3 = both verbosity options 1 and 2""")
    
  • sum(1 for i in file) - here i is not used. In Python, there is a special "convention" for this kind of throw-away variables - an underscore:

    sum(1 for _ in file)
    
  • don't use triple-quoted strings as comments, replace:

    """ Re-check if file exists to catch deletion while program is running """
    

    with:

    # Re-check if file exists to catch deletion while program is running 
    
  • there are also some PEP8 code style violations. Running flake8 or/and pylint static code analysis tools against the code will help to catch most of them.

Code Organization Notes

Note how get_book_rating_count(), get_book_title(), get_book_pages() and get_book_rating() functions all accept the soup BeautifulSoup instance argument. This is a "code smell" indicating that having a class with the soup instance variable would probably be a better alternative. You can find more on this topic here:

And, if you would run a complexity checker, like mccabe, against your script, you would see that main() function is overly complex and "heavy" - think of dividing it into logical pieces. Moving the "parsing command-line arguments" part can be the first candidate for extraction.

  • 2
    Nice review. I didn't know about SoupStrainer. +1 – яүυк May 31 '17 at 9:10

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.