I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I should spend with coding) I wrote it so I could read FanFictions offline. (For a Raspberry Pi E-Paper reader, but that's offtopic for this question)

I even spent more time making a GUI for it.

Here is the code:

import tkinter as tk
from tkinter import filedialog
from tkinter import ttk
from tkinter import messagebox
import requests    # pip install requests
import bs4         # pip install beautifulsoup4
import pathlib
from time import sleep
import re
# import logging

# Set logging level
# logging.basicConfig(level=logging.DEBUG)

# Define functions that takes care of the ID and URL entries
# https://bit.ly/tkinterautofillonso
after_ids = {}

def get_url(id_):
"""returns url from id."""
url = "https://www.fanfiction.net/s/{}/"
return url.format(id_.replace(" ", ""))

def get_id(url):
"""returns id from the url."""
l = url.split("/")
return l[4] if len(l) > 4 else ""

def autofill_entry(mode, delay=50):
"""Auto-fills Url/ID."""
for v in after_ids.values():
root.after_cancel(v)
if mode == "url":
id_ = get_id(fanfic_url.get())
after_ids[0] = root.after(delay, lambda: fanfic_id.set(id_))
elif mode == "id":
url = get_url(fanfic_id.get())
after_ids[1] = root.after(delay, lambda: fanfic_url.set(url))

# Define a function to have the user choose a directory and set the path accordingly
def get_directory():
global path
# Set selected directory if user did not cancel
if selected_directory:
path = pathlib.Path(selected_directory)
directory.set(path)

# Define a function to set the status
def set_status(string):
status.config(text = string + 100 * " ")
# Allows GUI to process and display events

num_chapter = 1
chapter = 0
base_url = fanfic_url.get()
progress_bar["value"] = 0
while not chapter == num_chapter:
# Set progress barmax value
progress_bar["maximum"] = 10 * num_chapter
chapter += 1
url = base_url + f"{chapter}/"
progress_bar["value"] = 1 * chapter
try:
response = requests.get(url)
except requests.exceptions.MissingSchema:
"Uh oh, an error has occurred!\n"
"That does not seem to a be a valid URL!")
except requests.exceptions.InvalidSchema:
"Uh oh, an error has occurred!\n"
"That does not seem to a be a valid URL!")
else:
set_status(f"Status code is {response.status_code}")
if response.status_code == 200:
set_status("Parsing FanFiction...")
# Parse HTML with html5lib
soup = bs4.BeautifulSoup(response.content, "html5lib")
progress_bar["value"] = 2 * chapter
# Check if we actually got a real chapter not an error message
if response.text.find("FanFiction.Net Message Type ") == -1 and \
set_status("Counting chapters...")
# This gets the number between the words "Chapters: " and " - Words: ")
num_chapter = int(
soup.find("span", class_="xgray xcontrast_txt").text[
soup.find(
"span", class_="xgray xcontrast_txt"
).text.find("Chapters: ") + 10:soup.find(
"span", class_="xgray xcontrast_txt"
).text.find(" - Words: ")
]
)
progress_bar["value"] = 3 * chapter
set_status("Finding title...")
text = soup.find("b", class_="xcontrast_txt").string + "\n"
progress_bar["value"] = 4 * chapter
# Find first link that looks like the author's name
set_status("Finding author...")
for author in soup.findAll("a", class_="xcontrast_txt"):
if author.parent.name == "div":
text += f"By: {author.string}\n\n"
# Break because we only want the first one
break
progress_bar["value"] = 5 * chapter
set_status("Finding synopsis...")
text += f"Synopsis: {soup.find('div', class_='xcontrast_txt').string}\n\n"
progress_bar["value"] = 6 * chapter
set_status("Finding FanFiction information...")
text += soup.find("span", class_="xgray xcontrast_txt").text + "\n\n"
progress_bar["value"] = 7 * chapter
set_status("Finding FanFiction content...")
for paragraph in soup.find_all("p"):
text += paragraph.text + "\n"
progress_bar["value"] = 8 * chapter
set_status("Signing FanFiction...")
progress_bar["value"] = 9 * chapter
# Make path to fanfiction
file_path = path / (
re.sub("[^\w\-_\. ]", "_", soup.title.string).replace(" ", "_") + ".txt"
)
set_status("Writing FanFiction to "
f"{str(file_path)[:80] + '...' if len(str(file_path)) > 80 else str(file_path)}")
# If a file with that name already exists
if file_path.exists():
"It looks like this file already exists! Overwrite?"):
with file_path.open("wt") as file:
file.write(text)
set_status("Sucessfully written FanFiction to "
f"{str(file_path)[:68] + '...' if len(str(file_path)) > 68 else str(file_path)}")
else:
set_status("Canceled writting FanFiction to "
f"{str(file_path)[:70] + '...' if len(str(file_path)) > 70 else str(file_path)}")
else:
with file_path.open("wt") as file:
file.write(text)
set_status("Sucessfully written FanFiction to "
f"{str(file_path)[:68] + '...' if len(str(file_path)) > 68 else str(file_path)}")
progress_bar["value"] = 10 * chapter
else:
# Chapter does not exist
if response.text.find("FanFiction.Net Message Type ") == -1:
"Uh oh, an error has occurred!\n"
"Story is unavailable for reading. (A)")
# Story does not exist
"Uh oh, an error has occurred!\n"
"New chapter/story can take up to 15 minutes to show up.")
else:
# Raise an error
"Uh oh, an error has occurred!\n"
f"Unexpected status code: {response.status_code}\n"
"No files have been written.")

path = pathlib.Path.cwd()

# Root window
root = tk.Tk()
# Set title of root window

# Define the labeled frame where we input stuff
input_frame = tk.LabelFrame(master=root, text="Input")

# Label for entering URL
# Entry field for URL
fanfic_url = tk.StringVar()
fanfic_url.trace_variable("w", lambda *a: autofill_entry("url"))
url_entry = ttk.Entry(master=input_frame, textvariable=fanfic_url)

# Label for entering fanfic ID
# Entry field for fanfic ID
fanfic_id = tk.StringVar()
fanfic_id.trace_variable("w", lambda *a: autofill_entry("id"))
id_entry = ttk.Entry(master=input_frame, textvariable=fanfic_id)

# Define the labeled frame where we output stuff
output_frame = tk.LabelFrame(master=root, text="Output")

ttk.Label(
directory = tk.StringVar()
directory.set(path)
directory_entry = ttk.Entry(master=output_frame, textvariable=directory)

# Define the labeled frame where we output status stuff
status_frame = tk.LabelFrame(master=root, text="Status")

# Progress bar for, well, progress
progress_bar = ttk.Progressbar(master=status_frame, orient=tk.HORIZONTAL, length=670, mode="determinate")
# Status bar for showing current operation
status = ttk.Label(master=status_frame, text="Idle", width=100)

# Start GUI event loop
root.mainloop()


Here is the Gist link too

When I implemented the progress bar, I did some stuff that doesn't look pretty and the progress bar likes to jump back to the beginning for like 0.1 seconds. The thing works, but that detail is driving me up the wall. I'd appreciate help in that department. I'd also appreciate general code optimizations/cleanups.

EDIT: After playing with it a bit more, I realized that Windows likes to mark the program as not responding if there are a lot of FanFictions to download. Is there a way to get rid of that too?

• +1 for side-project candidness. I know that feel. – Reinderien Jun 19 '20 at 16:44
• Yes. I was supposed to keep working on my RPi, but I wanted some more stories than the Bible on it. (I will admit the Bible can get boring - and I did mention that I made an E-paper reader with RPi) So you turn to Python to get FanFictions for you, because I am too lazy to look up a program to do it on code review stack exchange but I will work 4+ hours into making a program to do it for me. Perfectly sensible... – Unsigned_Arduino Jun 19 '20 at 17:05

## Requirements

Move these:

# pip install requests
# pip install beautifulsoup4


into a pip-compatible requirements.txt:

requests
beautifulsoup4


## Type hints

They will help. I don't know if id_ is an int or str here, but:

def get_url(id_: int) -> str:


## Path parsing

This:

l = url.split("/")
return l[4] if len(l) > 4 else ""


is a little risky. You're probably better off writing a regular expression that is explicit in where the ID is situated with respect to the element above it:

re.search(r'/s/(\d+)')


assuming that the ID is integral. Also, avoid using an empty string as the default if no ID is found; either return None or maybe raise an exception, depending on how much of a problem it is.

## Mode variables

Since mode only has two states, use a bool, maybe named get_url. If it had more than two states, or if you feel that the use of bool is too confusing, use an Enum - but avoid using strings for this purpose.

## Globals

def get_directory():
global path


should not declare a global. Simply return path.

## Separation of UI and logic

download_fanfic is a big old ball of yarn. You have calls to requests beside calls to messagebox. Separate out the actual downloading logic and parsing logic into their own separate functions that do not have any tk code in them at all. Errors can be signalled via exceptions, and progress can be signalled via a generic callback function.

## Logging

Try converting this:

set_status("Writing FanFiction to " ...


into a call to the standard logging framework, and adding your own logging handler to do what set_status currently does. It will make your application much more flexible - you could fairly easily flip one switch to have the entire thing run in console-only mode.

## Requests check

        if response.status_code == 200:


should be replaced with:

if response.ok:


or better yet

response.raise_for_status()


• Sorry if I'm bothering you, but I'm having trouble with trying not to use global, because I can't figure out how to assign the result of get_directory() in a lambda if the result is not None – Unsigned_Arduino Jun 19 '20 at 17:53