Python script to download adult comics from 8muses

Question

I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the images.

Enter the the gallery url and download location to start downloading. Sample gallery urls:

https://www.8muses.com/comics/album/MilfToon-Comics/Milfage/Issue-1 https://www.8muses.com/comics/album/MilfToon-Comics/Lemonade/Lemonade-1

I would like to know improvements to the code or alternative methods to make it work faster. Thanks !

Code : app.py

import os
from multiprocessing.dummy import Pool
from queue import Queue
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
from threading import Thread
import urllib.request
import requests 
import shutil

options = Options()
options.headless = True
chrome_driver_path = r"C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe"
base_url = "https://www.8muses.com"

def fetch_image_url(url,filename,download_location):
    driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
    driver.get(url)
    page = driver.page_source
    soup = BeautifulSoup(page,"lxml")
    image_url = "http:"+soup.find("img",{"class":"image"})['src']
    download_image(image_url,filename,download_location)

def download_image(image_url,filename,download_location):
    r = requests.get(image_url,stream=True, headers={'User-agent': 'Mozilla/5.0'})
    if r.status_code == 200:
        with open(os.path.join(download_location,str(filename)+".png"), 'wb') as f:
            r.raw.decode_content = True
            shutil.copyfileobj(r.raw, f)
    print("Downloaded page {pagenumber}".format(pagenumber=filename))


if __name__=="__main__":
    print("Album Url : ")
    album_url = input()
    print("Download Location : ")
    download_location = input()
    driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
    print("Loading Comic...")
    driver.get(album_url)
    album_html = driver.page_source
    print("Comic successfully loaded")
    soup = BeautifulSoup(album_html,"lxml")
    comic_name = soup.find("title").text.split("|")[0].strip()
    download_location = os.path.join(download_location,comic_name)
    os.mkdir(download_location)
    print("Finding comic's pages")
    images = soup.find_all("a",{"class":"c-tile t-hover"})
    page_urls = []
    pages = []
    threads = []
    for image in images:
        page_urls.append(base_url + image['href'])
    print("Found {} pages".format(len(page_urls)))
    for i in  range(len(page_urls)):
            pages.append((page_urls[i],i,download_location))
    p = Pool(3)  # 3 threads in the pool
    p.starmap(fetch_image_url,pages)
    p.close()
    p.join()
    driver.quit()
    print ("DONE ! Happy Reading ")

Github for the project : https://github.com/ggrievous/8muser

Answer 1

There's no reason to use selenium here. Instead of reaching for selenium, first try the easier path. Load the page without javascript and see if you can find any helpful information about how the images get there. There must be something (perhaps AJAX) that gets the URLs for them. They don't just magically appear!

It turns out if you do this, you'll see there isn't any fancy JS stuff. The images are there, they just look like this:

<div class="image">
  <img class="lazyload" data-src="/image/th/QD-H-4F3JpxykaxnIIFbrixlkt4rwBphjoSmX2E8fvoJ8JanT-+S2MdyjKTADNm+SJdCVhXQwkdIZ0tQel-n8-y70M9EOTmeW06uA5ubLwnl2gi5X14+yw6GKhNbhj7S.jpg">
</div>

This means, you can extract all these URLs with a single line of BeautifulSoup:

urls = [img['data-src'] for img in doc.find_all('img', class_='lazyload')]

Now some comments about your code:

• PEP8 it! Your spacing is inconsistent. Make good use of vertical whitespace. Phrase things like you would paragraphs. That makes things much easier to read.
• You don't need selenium, but you definitely shouldn't hard code a driver path in a project that you open source. How many people have a selenium driver exactly at C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe on their computer?
• Nice use of functions to separate concerns
• You should probably use BeautifulSoup(page, 'html5lib') instead of lxml
• Your construction of image_url is a bit sloppy. Typically, we'd reach for urllib.path to build paths instead of just doing string concatenation.
• Use pathlib instead of os.path
• 'Mozilla/5.0' isn't a User-Agent that's going to fool anybody. If you're really trying to stay under the radar, use a real UA
• But none of that matters, because you appear to request pages as fast as possible. Add sleep()s in between downloading. Throttle your scraper.
• threading is a bit useless in Python. This is somewhat of an I/O bound task (for which threads are well suited), but the HTML parsing and extraction could definitely be done concurrently with web requests (but threading doesn't allow this). You almost always want to reach for multiprocessing.
• Use the context manager of a pool instead of manually calling close() and join():

with Pool() as pool:
    pool.imap_unorderd(fetch_image_url, pages)

• Also, don't pass a parameter to Pool. It defaults to the number of CPU cores, which is almost always what you want
• starmap is ordered and blocking. It only can process things in order. This is okay in this case, because you aren't actually returning anything, but if you were say doing math you probably want imap_unordered which yields results as they arrive (likely out of order).
• Don't print from a separate process. You want a single process writing to stdout, otherwise you can have write contention (which you'll luck out and probably never run into because your strings probably fit inside the stdout buffer, but it's possible they may not under certain circumstances).

But this all culminates in the following advice: don't use Python to download things!

Especially since this scraping doesn't appear to be useful as a library (instead, it seems like you just are providing a CLI utility for a human to download these things). Given this, it's much smarter and safer to not reinvent the wheel. There are tools that already do jobs like this well: namely, wget (it appears that you're on windows, you can and should use the Ubuntu subsystem, which will have wget). wget is particularly suited for this job and has tons of builtin functionality which will be super useful to you. This includes:

• Not redownloading things
• Throttling (including random delays)
• Restarting after a catastrophic (program crashing) failure
• Retrying requests per HTTP spec

All of these are things that your script doesn't do currently. In particular, in python it's very easy to do something like this:

pages = download_hundreds_of_pages()  # takes hours...
for page in paages:  # oops, this NameErrors and you lose everything you've downloaded
  pass

Mistakes like this are too easy to make. You can completely avoid them with the following workflow:

1. Build up a list of urls you want to download (perhaps with python)
2. Use wget -nc -i urls.txt to download them
3. Repeat as necessary

For you, that would involve making a list of urls containing the images. Then do wget -nc -i pages.txt. That will download all of the pages to the current directory. Then you can make a Python script which uses beautiful soup (and the line I mentioned above) to extract the image urls: python3 extract_image_urls.py > image_urls.txt. Then to download them do wget -nc -i image_urls.txt. If your python script fails at any point, you don't lose all of the downloads you've already done. You can wrap all of this in a convenient bash script.