Web scraping and downloading manga

Question

I'm new to Python and decided to work on a personal project and make a program that web scrapes and downloads manga. It works as expected, however, I am sure that there is a better way to structure my code. I'd appreciate some guidance on how to better write this project. You can check it out on my GitHub here.

import threading
import requests
import os
import re
import queue

from bs4 import BeautifulSoup
from display import Display


def gen_title(link):
    """Finds title of manga and removes any illegal characters"""
    page = requests.post(link, data=dict(adult="true"))
    soup = BeautifulSoup(page.text, 'lxml')

    title = soup.find('h1', {"class": "hb dnone"}).find('a').get('title')

    # Search for invalid characters and remove them.
    return re.sub('[^A-Za-z0-9 ]+', '', title)


def get_volume(soup):
    """Finds volume num"""
    volume = soup.find('h1', {"class": "hb dnone"}).findAll('a')
    volume = volume[1].get('title')

    # Grab first number in title
    temp = re.findall('\d+', volume)
    return "Volume " + temp[0]


def write_image(image, title, volume, counter):
    """Writes image to a file"""
    # Write the image to a file in chapter directory
    imagefile = open(title + "/" + volume + "/" + str(counter) + ".png", 'wb')
    imagefile.write(image.content)
    imagefile.close()


def download_manga(start_link, end_link=""):
    """given a start link and end link from twistedhelscans.com, downloads all manga images"""
    next_link = start_link
    counter = 1

    # Deal with end link being first page
    if end_link.endswith('1'):
        end_link = end_link[:-6]

    # get title of manga
    try:
        title = gen_title(start_link)
    except:
        queue.put("Title not found")
        return

    while next_link != end_link:
        # Open initial page
        page = requests.post(next_link, data=dict(adult="true"))

        # check if end link is first page redirect
        if page.url == end_link:
            break

        queue.put(page.url)
        soup = BeautifulSoup(page.text, 'lxml')

        if not end_link:
            end_link = soup.find('h1', {"class": "hb dnone"}).find('a').get('href')

        # Find image link and vol. num
        try:
            volume = get_volume(soup)
            image = soup.find('div', {"class": "inner"}).find('img').get('src')
        except:
            queue.put("Could not find image link. Website is not Twisted Hel Scan page?")
            return

        # Download the image
        image = requests.get(image)

        # Make manga directory
        if not os.path.exists(title):
            os.mkdir(title)

        # Make volume directory
        if not os.path.exists(title + "/" + volume):
            os.mkdir(title + "/" + volume)
            counter = 1

        # Write image to file
        write_image(image, title, volume, counter)
        counter += 1

        # Find next link
        next_link = soup.find('div', {"class": "inner"}).find('a').get('href')
    queue.put("Done")


def on_click(window):
    """Fetches text from entries and calls download manga"""
    start_link = window.start_entry.get()
    end_link = window.end_entry.get()

    if not start_link.strip():
        queue.put("No start link given")
        return

    def callback():
        download_manga(start_link, end_link)

    t = threading.Thread(target=callback)
    t.start()
    display.progress.pack(fill='x')
    display.progress.start(15)


def periodic_call():
    """
    Check every 100 ms if there is something new in the queue.
    """
    display.process_incoming()
    display.root.after(100, periodic_call)


# Create window and bind go button
queue = queue.Queue()
display = Display(queue)
display.go_button.bind("<Button-1>", lambda x: on_click(display))
periodic_call()
display.root.mainloop()
os._exit(1)

Answer 1

Welcome to Code Review. That is pretty useful tool. :) From the GitHub link, I notice that you're using Python 3. There are a few things, I think you'd benefit from:

1. Reorder your imports so that Python's library is imported first, followed by 3rd party modules; followed by your local display module. Read also the PEP-8 guide for the same.
2. Since you're combining requests and beautifulsoup4, I would suggest looking into Kenneth's yet another awesome project requests-html. You can have a single session to download the entire chapter/volume :)
3. Place your display initialisation code block inside the if __name__ == "__main__" clause.
4. You send a request more than once for the same page. First to get the page title, and later to actually start downloading images. Doing an HTTP request is heavy work. Do it once.

5. This looks ugly: open(title + "/" + volume + "/" + str(counter) + ".png", 'wb'). Use the with context block, and string formatting (python 3.6+):

   with open(f"{title}/{volume}/{counter}.png", 'wb') as imagefile:
       imagefile.write(image.content)