13
\$\begingroup\$

Note: The topics of performance and Selenium/BS4 have not yet been addressed, so this question can still receive a better answer!

Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-discussion

This is a recreational script made to update my home server w/ the latest season of anime from HorribleSubs. I'd like to know if there are any obvious syntactical and performance improvements, details on my usage of Selenium and BS4 and whether or not this usage is proper.

Windows Setup

  1. Install Scoop to install any necessary programs.
  2. Run the following commands for whatever you don't have installed:
scoop bucket add extras
scoop install python geckodriver qbittorrent
pip install beautifulsoup4 selenium python-qbittorrent
  1. Enable the qBittorrent web interface.
  2. Enjoy, and don't forget to seed your favorites!

Le Code

import os
import traceback

from sys import platform
from shutil import rmtree

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import NoSuchElementException

import urllib.request as Web
from bs4 import BeautifulSoup as Soup
from qbittorrent import Client as qBittorrent
from wget import download
from collections import defaultdict


def get_dl_path():
    # TODO: Check if this drive has space, else check another drive
    # if there's no free space, crash
    return 'E:/Torrents/'


def get_addons_path():
    path = os.getcwd()

    if platform == 'win32':
        path += '\\addons\\'
    else:
        path += '/addons/'

    if not os.path.exists(path):
        os.mkdir(path)

    return path


dl_path = get_dl_path()
addons_path = get_addons_path()
profile = webdriver.FirefoxProfile()

# Run the browser in private mode
profile.set_preference('extensions.allowPrivateBrowsingByDefault', True)
profile.set_preference('browser.privatebrowsing.autostart', True)

# Privacy settings (https://www.privacytools.io/)
profile.set_preference('media.peerconnection.turn.disable', True)
profile.set_preference('media.peerconnection.use_document_iceservers', False)
profile.set_preference('media.peerconnection.video.enabled', False)
profile.set_preference('media.peerconnection.identity.timeout', 1)
profile.set_preference('privacy.firstparty.isolate', True)
profile.set_preference('privacy.resistFingerprinting', True)
profile.set_preference('privacy.trackingprotection.fingerprinting.enabled', True)
profile.set_preference('privacy.trackingprotection.cryptomining.enabled', True)
profile.set_preference('privacy.trackingprotection.enabled', True)
profile.set_preference('browser.send_pings', False)
profile.set_preference('browser.sessionstore.max_tabs_undo', 0)
profile.set_preference('browser.sessionstore.privacy_level', 2)
profile.set_preference('browser.urlbar.speculativeConnect.enabled', False)
profile.set_preference('dom.event.clipboardevents.enabled', False)
profile.set_preference('media.eme.enabled', False)
profile.set_preference('media.gmp-widevinecdm.enabled', False)
profile.set_preference('media.navigator.enabled', False)
profile.set_preference('network.cookie.cookieBehavior', 2)
profile.set_preference('network.cookie.lifetimePolicy', 2)
profile.set_preference('network.http.referer.XOriginPolicy', 2)
profile.set_preference('network.http.referer.XOriginTrimmingPolicy', 2)
profile.set_preference('network.IDN_show_punycode', True)
profile.set_preference('webgl.disabled', True)

# Settings unique to https://restoreprivacy.com/firefox-privacy/
profile.set_preference('geo.enabled', False)
profile.set_preference('media.peerconnection.enabled', False)
profile.set_preference('network.dns.disablePrefetch', True)
profile.set_preference('network.prefetch-next', False)

options = webdriver.FirefoxOptions()
options.headless = True

browser = webdriver.Firefox(firefox_profile=profile, options=options)

ext_prefix = 'https://addons.mozilla.org/en-US/firefox/addon/'
exts = [
    # 'ublock-origin',  # Blocks ads & such
    # 'https-everywhere',  # TODO: Figure out how to enable 'Encryt All Sites Eligble'
    # 'decentraleyes',  # Blocks Content Management Systems and handles their abilities locally
    'umatrix'  # Will block Disqus on HorribleSubs automatically
]

for ext in exts:
    browser.get(ext_prefix + ext)
    btn = browser.find_element_by_class_name('AMInstallButton')
    ref = btn.find_element_by_tag_name('a').get_attribute('href')
    url = ref.split('?')[0]
    addon = download(url, out=addons_path).replace('/', '')
    browser.install_addon(addon, temporary=True)

browser.get('https://horriblesubs.info/current-season/')
src = browser.page_source
parser = Soup(src, features='html.parser')
divs = parser.body.find_all('div', attrs={'class': 'ind-show'})
size = len(divs)
season = defaultdict(list)

print('\nDownloading', size, 'shows')

try:
    for i, div in enumerate(divs):
        browser.get('https://horriblesubs.info' + div.a['href'])

        # Wait to dodge `selenium.common.exceptions.ElementNotInteractableException: Message: Element could not be scrolled into view`
        WebDriverWait(browser, 15).until(EC.element_to_be_clickable((By.CLASS_NAME, 'more-button')))

        # Expand the whole listing to get all the episodes
        if not browser.find_elements_by_id('01'):
            try:
                while True:
                    browser.find_element_by_class_name('more-button').click()
            except NoSuchElementException:
                pass

        src = browser.page_source
        parser = Soup(src, features='html.parser')
        episodes = parser.body\
            .find('div', attrs={'class': 'hs-shows'})\
            .find_all('div', attrs={'class': 'rls-info-container'})

        for episode in episodes:
            links = [
                episode.find('div', attrs={'class': 'rls-link link-480p'}),
                episode.find('div', attrs={'class': 'rls-link link-720p'}),
                episode.find('div', attrs={'class': 'rls-link link-1080p'})
            ]
            magnet = None

            for link in links:
                if link is not None:
                    a = link.find('a', attrs={'title': 'Magnet Link'})
                    if a is not None:
                        magnet = a['href']

            if magnet is not None:
                season[dl_path + div.a.text].append(magnet)

        print('[%]', round(((i + 1) / size) * 100, 2))
except Exception:
    print(traceback.print_exc())
finally:
    browser.quit()
    rmtree(addons_path)

try:
    # Web UI -> 'Bypass authentication for hosts on localhost' should be enabled
    # Downloads -> 'Do not start download automatically' should be enabled
    qb = qBittorrent('http://127.0.0.1:8080/')

    # Use DP to decrease show fetch time
    for path, magnets in season.items():
        for magnet in magnets:
            qb.download_from_link(magnet, savepath=path, category='anime')

    qb.resume_all()
except ConnectionError:
    print('[!] qBittorrent not active!')

\$\endgroup\$
  • \$\begingroup\$ If this is a follow-up to your previous question, don't forget to include a link. See also How to post a follow-up question? on meta. \$\endgroup\$ – AlexV Oct 20 at 21:47
  • 1
    \$\begingroup\$ @AlexV I'd say it's related but not a follow-up, since a) the previous had no answers and b) this is a significantly updated version. \$\endgroup\$ – T145 Oct 20 at 21:48
  • 3
    \$\begingroup\$ @AlexV Best thing would've been for the original question to be edited really, since now there's a question out there that nobody cares about, doesn't need an answer anymore and has no valuable answers for future reference. \$\endgroup\$ – Mast Oct 21 at 9:48
  • \$\begingroup\$ Are you still looking for answers/feedback on this? \$\endgroup\$ – Alexander Cécile Oct 25 at 1:35
  • \$\begingroup\$ @AlexanderCécile Yes; I've left a note above stating what hasn't been addressed yet, if anyone is interested. \$\endgroup\$ – T145 Oct 25 at 1:44
9
\$\begingroup\$

I have no familiarity with any of the libraries used here, so I can't comment on their usage.

What I will mention though is the giant chunk of profile.set_preference calls in the middle of the script. It would be much cleaner and less repetitive to save the string/bool pairs of options as a dictionary (or another "paired" structure), then just iterate over it. Example (partial):

# Just so we can help prevent bad data entry into the dictionary
from typing import Dict, Any

# The \ is just so I can stick {} on the next line for neatness
profile_settings: Dict[str, Any] = \
    {'extensions.allowPrivateBrowsingByDefault': True,
     'browser.privatebrowsing.autostart': True,

     'media.peerconnection.turn.disable': True,
     'media.peerconnection.use_document_iceservers': False

     # And the rest of pairs 
     }

for setting_name, setting_value in profile_settings.items():
    profile.set_preference(setting_name, setting_value)

Now you don't need to copy and paste profile.set_preference a hundred times. This also allows you to easily save profile_settings into a config file so you can edit settings without needing to edit the code. When needed, you can just read the settings and iterate over them.

And to clarify why I'm specifying profile_settings as being of type Dict[str, Any] using type hints: telling the IDE what type your variable is can help it catch mistakes you make. Let's say you have a dictionary of strings to ints, and you accidentally pass it the wrong piece of data:

the_data_I_want = 1
the_data_I_dont_want = "some stuff"

d: Dict[str, int] = {"a": the_data_I_dont_want}  # Whoops

The last line will raise a warning

Expected type Dict[str, int], got Dict[str, str] instead.

With how you currently have it, it's unlikely that you'd accidentally give it a key of a type other than a string. If you start reading that data from elsewhere though, or begin pulling keys from variables, it's nice to have the IDE able to catch you when you've made a typo (like a bad auto-complete).


You also have at the bottom

except Exception:
    print(traceback.print_exc())

It's good that you're printing out a stack trace so at least you aren't muting any useful debugging information, but I don't see why you're catching in the first place.

If you just want to use the finally, you don't need to specify an except:

try:
    . . .

finally:
    browser.quit()
    rmtree(addons_path)
\$\endgroup\$
  • \$\begingroup\$ Hey, any input is better that none, so thank you! I'm curious as to what you mean by Just so we can help prevent bad data entry into the dictionary; may you elaborate? The try-catch block around the main gathering logic is just to fall into the finally if anything bad happens, just so I don't have to manually delete files. As for the listing expansion, it runs until the "more-button" element is gone. If you examine some of the HTML on a show listing page you'll get a better idea of what's happening. Or just remove the setting that makes it headless and watch :p \$\endgroup\$ – T145 Oct 21 at 0:02
  • \$\begingroup\$ By "Just so we can help prevent bad data entry into the dictionary", I meant that if you tell Python that profile_settings is a Dict[str, bool] (a dictionary mapping strings to bools), you'll get a warning if you accidentally try to put something other than a string or bool into it. I'll clarify that in the answer. And I'll just note that you don't need a except in a try. You only need either a except or a finally. try: ... finally: stuff() is fine. \$\endgroup\$ – Carcigenicate Oct 21 at 0:11
  • 1
    \$\begingroup\$ @T145 See my edit mid-to-bottom. I changed some stuff. \$\endgroup\$ – Carcigenicate Oct 21 at 0:19
  • 1
    \$\begingroup\$ @BruceWayne The : Dict[str, Any] part is type hinting. Unlike in a statically typed language, type hints are entirely for aesthetics, and for the IDE. It does not cause actual enforment of types. They're still very helpful for documentation purposes and to catch typos though. \$\endgroup\$ – Carcigenicate Oct 21 at 17:33
  • 1
    \$\begingroup\$ @Carcigenicate - Thanks so much for your comments! I'll start using these, I'm not fluent in Python but I'm pretty comfortable these days, so this type of information is much welcomed :) \$\endgroup\$ – BruceWayne Oct 21 at 18:30
1
\$\begingroup\$

Firstly good to learn about Scoop, very useful.

Secondly, as a big change, which you might not want to do now that you've already built it: Consider not using a full browser for this. I opened up the website with JavaScript blocked and it worked absolutely fine, in fact I imagine the HTML will be parseable by just about any library. Why make that change? Well, it will likely reduce your resource usage by quite a bit and very probably make it much faster, not to mention being able to do things concurrently, so you'll see results much quicker. Edit: I missed the bit where at least one button, "Show more", does run via JavaScript. Could probably still be worked around by generating the requested URLs programmatically, otherwise Selenium is probably required.

Right, so after that, the pathname handling should probably use pathlib to be more robust.

The querying for HTML content I'd rather suggest using XPath or CSS query syntax to make things more expressive. Like div[class~=hs-shows] div[class~=rls-info-container] etc. Fewer function calls, easier to understand if you already know XPath or CSS. Plus, you can easily try it in the browser first.

What else? Well once this gets bigger consider if __name__ == "__main__" and having a main function.

You could also consider some concurrency by immediately passing content to qBittorrent? But maybe that was also intentionally done later.

\$\endgroup\$
  • 1
    \$\begingroup\$ I'd certainly like to not use Selenium, but other scrapers don't seem to support dynamic page content. The big problem is the "show more" element, otherwise I'd just use pure BS4. Originally I had a main ftn & passed shows concurrently to QB. Ditched the first since this is relatively short and used DP to pass shows at the end to avoid having to wait on QB receiving shows in order to move to the next show. \$\endgroup\$ – T145 Oct 21 at 11:31
  • \$\begingroup\$ Ooops. Yes, sorry, in that case you'd obviously have to require a bit more knowledge about how the URLs are generated. Even though I'd guess that's still doable. Anyway, I'll mark that as such in the answer. \$\endgroup\$ – ferada Oct 21 at 12:15
1
\$\begingroup\$

Instead of:

if platform == 'win32':
    path += '\\addons\\'
else:
    path += '/addons/'

You could use os.sep or even better: os.path.join

\$\endgroup\$
  • \$\begingroup\$ Good to know more about relative addressing in Python; thank you! \$\endgroup\$ – T145 Oct 21 at 18:07
  • \$\begingroup\$ Should be noted that using this technique requires the path over @ addon to be normalized: docs.python.org/3/library/os.path.html#os.path.normpath \$\endgroup\$ – T145 Oct 21 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.