2
\$\begingroup\$

I am scraping a pool of URLs with a python/selenium framework using Firefox and Geckodriver.

For each URL, all I am doing is to find one element by class name and save it to disk. In order speed this up as much as possible, I am creating a custom Firefox profile with the specs defined below in firefox_profile.set_preference.

The challenge is that testing all combinations of all Firefox profile parameters in order to know which specs are best for speed is laborious. Therefore, I am hoping to get some input.

  1. Are there any preferences parameters beyond the ones mentioned below that also would speed up Firefox?

  2. Is any of those preference parameters mentioned below not suitable to speed up Firefox?

def create_driver():
        firefox_profile = webdriver.FirefoxProfile()
        firefox_profile.set_preference('browser.download.animateNotifications', False)
        firefox_profile.set_preference('browser.fullscreen.animate', False)
        firefox_profile.set_preference('browser.preferences.animateFadeIn', False)
        firefox_profile.set_preference('browser.tabs.animate', False)
        firefox_profile.set_preference('browser.cache.use_new_backend', 1)
        firefox_profile.set_preference('browser.sessionhistory.max_total_viewers', 0)
        firefox_profile.set_preference('browser.safebrowsing.enabled', False)
        firefox_profile.set_preference('browser.shell.checkDefaultBrowser', False)
        firefox_profile.set_preference('browser.startup.page', 0)
        firefox_profile.set_preference('layout.animated-image-layers.enabled', False)
        firefox_profile.set_preference('extensions.checkCompatibility', False)
        firefox_profile.set_preference('extensions.checkUpdateSecurity', False)
        firefox_profile.set_preference('extensions.logging.enabled', False)
        firefox_profile.set_preference('extensions.update.autoUpdateEnabled', False)
        firefox_profile.set_preference('extensions.update.enabled', False)
        firefox_profile.set_preference('print.postscript.enabled', False)
        firefox_profile.set_preference('toolkit.storage.synchronous', 0)
        firefox_profile.set_preference('image.animation_mode', 'none')
        firefox_profile.set_preference('images.dither', False)
        firefox_profile.set_preference('content.notify.interval', 1000000)
        firefox_profile.set_preference('content.switch.treshold', 100000)
        firefox_profile.set_preference('nglayout.initialpaint.delay', 1000000)
        firefox_profile.set_preference('network.dnscacheentries', 200)
        firefox_profile.set_preference('network.dnscacheexpiration', 600)
        firefox_profile.set_preference('network.prefetch-next', False)
        firefox_profile.set_preference('permissions.default.image', False)
        firefox_profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', False)
        firefox_profile.set_preference('dom.ipc.plugins.flash.disable-protected-mode', False)
        firefox_profile.set_preference('app.update.enabled', False)
        firefox_profile.set_preference('app.update.service.enabled', False)
        firefox_profile.set_preference('app.update.auto', False)
        firefox_profile.set_preference('app.update.staging', False)
        firefox_profile.set_preference('app.update.silent', False)
        firefox_profile.set_preference("javascript.enabled", False)
        driver = webdriver.Firefox(firefox_profile = firefox_profile)
    return driver
Improve this question
\$\endgroup\$
4
  • \$\begingroup\$ Are you sure, your pages are going to properly load with these settings? E.g. with javascript disabled? If yes, you might not even need a browser altogether.. \$\endgroup\$
    – alecxe
    Dec 14, 2017 at 4:03
  • \$\begingroup\$ yes it is loading, and I need the browser here since all I get with other methods is the content of robots.txt (i.e. beautiful soup) \$\endgroup\$
    – sudonym
    Dec 14, 2017 at 4:06
  • 2
    \$\begingroup\$ Okay. Have you tried using, say, requests and specifying a User-Agent header (pretending to be a browser)? Thanks. \$\endgroup\$
    – alecxe
    Dec 14, 2017 at 4:06
  • \$\begingroup\$ no, thanks for your hint - will try right now \$\endgroup\$
    – sudonym
    Dec 14, 2017 at 4:07

1 Answer 1

Reset to default
3
\$\begingroup\$

Regardless of if these settings are correct, or if you could get by without a webbrowser, this is not a very good way to set all these values. At the very least extract the key, value pairs to a dictionary, which you ideally read from a config file.

import webdriver

SETTINGS = {'browser.download.animateNotifications': False,
            ...}

def create_driver():
    firefox_profile = webdriver.FirefoxProfile()
    for key, value in SETTINGS.items():
        firefox_profile.set_preference(key, value)
    driver = webdriver.Firefox(firefox_profile = firefox_profile)
    return driver
Improve this answer
\$\endgroup\$
3
  • \$\begingroup\$ true - apologies - just edited the type and thanks for the improved syntax \$\endgroup\$
    – sudonym
    Dec 14, 2017 at 4:14
  • \$\begingroup\$ I just changed the syntax accordingly and can't confirm that this has any impact on scraping performance speed-wise \$\endgroup\$
    – sudonym
    Dec 14, 2017 at 9:26
  • \$\begingroup\$ @suonym And it should not. It was a comment on coding style, not a speed improvement. \$\endgroup\$
    – Graipher
    Dec 14, 2017 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.