I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the images.

I would like to know improvements to the code or alternative methods to make it work faster. Thanks !

Code : app.py

import os
from multiprocessing.dummy import Pool
from queue import Queue
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import urllib.request
import requests
import shutil

options = Options()
chrome_driver_path = r"C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe"
base_url = "https://www.8muses.com"

driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
driver.get(url)
page = driver.page_source
soup = BeautifulSoup(page,"lxml")
image_url = "http:"+soup.find("img",{"class":"image"})['src']

if r.status_code == 200:
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)

if __name__=="__main__":
print("Album Url : ")
album_url = input()
driver = webdriver.Chrome(chrome_driver_path, chrome_options=options)
driver.get(album_url)
album_html = driver.page_source
soup = BeautifulSoup(album_html,"lxml")
comic_name = soup.find("title").text.split("|")[0].strip()
print("Finding comic's pages")
images = soup.find_all("a",{"class":"c-tile t-hover"})
page_urls = []
pages = []
for image in images:
page_urls.append(base_url + image['href'])
print("Found {} pages".format(len(page_urls)))
for i in  range(len(page_urls)):
p = Pool(3)  # 3 threads in the pool
p.starmap(fetch_image_url,pages)
p.close()
p.join()
driver.quit()
print ("DONE ! Happy Reading ")


Github for the project : https://github.com/ggrievous/8muser

There's no reason to use selenium here. Instead of reaching for selenium, first try the easier path. Load the page without javascript and see if you can find any helpful information about how the images get there. There must be something (perhaps AJAX) that gets the URLs for them. They don't just magically appear!

It turns out if you do this, you'll see there isn't any fancy JS stuff. The images are there, they just look like this:

<div class="image">
</div>


This means, you can extract all these URLs with a single line of BeautifulSoup:

urls = [img['data-src'] for img in doc.find_all('img', class_='lazyload')]


• PEP8 it! Your spacing is inconsistent. Make good use of vertical whitespace. Phrase things like you would paragraphs. That makes things much easier to read.
• You don't need selenium, but you definitely shouldn't hard code a driver path in a project that you open source. How many people have a selenium driver exactly at C:\Users\NH\PycharmProjects\SeleniumTest\drivers\chromedriver.exe on their computer?
• Nice use of functions to separate concerns
• You should probably use BeautifulSoup(page, 'html5lib') instead of lxml
• Your construction of image_url is a bit sloppy. Typically, we'd reach for urllib.path to build paths instead of just doing string concatenation.
• Use pathlib instead of os.path
• 'Mozilla/5.0' isn't a User-Agent that's going to fool anybody. If you're really trying to stay under the radar, use a real UA
• But none of that matters, because you appear to request pages as fast as possible. Add sleep()s in between downloading. Throttle your scraper.
• threading is a bit useless in Python. This is somewhat of an I/O bound task (for which threads are well suited), but the HTML parsing and extraction could definitely be done concurrently with web requests (but threading doesn't allow this). You almost always want to reach for multiprocessing.
• Use the context manager of a pool instead of manually calling close() and join():
with Pool() as pool:
pool.imap_unorderd(fetch_image_url, pages)

• Also, don't pass a parameter to Pool. It defaults to the number of CPU cores, which is almost always what you want
• starmap is ordered and blocking. It only can process things in order. This is okay in this case, because you aren't actually returning anything, but if you were say doing math you probably want imap_unordered which yields results as they arrive (likely out of order).
• Don't print from a separate process. You want a single process writing to stdout, otherwise you can have write contention (which you'll luck out and probably never run into because your strings probably fit inside the stdout buffer, but it's possible they may not under certain circumstances).

Especially since this scraping doesn't appear to be useful as a library (instead, it seems like you just are providing a CLI utility for a human to download these things). Given this, it's much smarter and safer to not reinvent the wheel. There are tools that already do jobs like this well: namely, wget (it appears that you're on windows, you can and should use the Ubuntu subsystem, which will have wget). wget is particularly suited for this job and has tons of builtin functionality which will be super useful to you. This includes:

• Throttling (including random delays)
• Restarting after a catastrophic (program crashing) failure
• Retrying requests per HTTP spec

All of these are things that your script doesn't do currently. In particular, in python it's very easy to do something like this:

pages = download_hundreds_of_pages()  # takes hours...
for page in paages:  # oops, this NameErrors and you lose everything you've downloaded
pass


Mistakes like this are too easy to make. You can completely avoid them with the following workflow:

1. Build up a list of urls you want to download (perhaps with python)
2. Use wget -nc -i urls.txt to download them
3. Repeat as necessary

For you, that would involve making a list of urls containing the images. Then do wget -nc -i pages.txt. That will download all of the pages to the current directory. Then you can make a Python script which uses beautiful soup (and the line I mentioned above) to extract the image urls: python3 extract_image_urls.py > image_urls.txt. Then to download them do wget -nc -i image_urls.txt. If your python script fails at any point, you don't lose all of the downloads you've already done. You can wrap all of this in a convenient bash script.