3
\$\begingroup\$

I have this small script that needs to make a reservation at a DMV office.

The problem is that the reservation window opens only every few hours for a few minutes (it might show up in one of these months: December, January, February or March of next year). My code is too slow to even "catch" the "open day".

Any ideas on how I can speed up the whole process? Maybe a different approach?

from selenium import webdriver
import time

firstName = "John"
lastName = "Smith"
yob = "1929"
email = "john.smith29@yolol.com"
phone = "555-123-4567"

browser = webdriver.PhantomJS()
browser.get("https://page/page2/ACFCustom/Se.aspx")

def check_date_avaialbility():
    # Choose a service
    browser.find_element_by_name("ct$cphBy$rS$ctl$btn1").click()
    # Choose an office
    browser.find_element_by_name("cphBy_rpt_btn1").click()

    # Loop through the next 4 months and search for an open window (The day will be highlighted in green)
    for i in range(1, 5):
        try:
            print("Searching for available day... ")
            browser.find_element_by_xpath("//*[contains(@style, 'background-color:#4A0889;')]").click()
            browser.find_element_by_xpath("//*[@id='cphBy_rblAT']").click()
            browser.find_element_by_id("cphBy_Nt").click()
            form_data()
        except:
            print("There are no available dates. Next page... \n")
            browser.find_element_by_name("ctl_cphBy_rdcAD_NN").click()
            time.sleep(1)

    # If you couldn't find anything three. Go back to the "home" page 
    browser.find_element_by_name("cphBy_lnkbnK").click()
    time.sleep(1)
    # Start going through all the next 4 months all over again
    check_date_avaialbility()

def form_data():
    print("Filling up the form...")
    browser.find_element_by_id("cphBy_txbFN").send_keys(firstName)
    browser.find_element_by_id("cphBy_txbLN").send_keys(lastName)
    browser.find_element_by_id("cphBy_txbYB").send_keys(yob)
    browser.find_element_by_id("cphBy_txbE").send_keys(email)
    browser.find_element_by_id("cphBy_txbPN").send_keys(phone)
    browser.find_element_by_id("cphBy_S").click()

check_date_avaialbility()
\$\endgroup\$
  • 1
    \$\begingroup\$ Maybe pay someone to stand in line? (Kidding) bit to be more serious, I am sure the script is already stretching what might be acceptable use, I don’t think cranking it up by hitting the site from multiple bots in parallel will make it less suspicious... \$\endgroup\$ – eckes Nov 11 '17 at 3:42
4
\$\begingroup\$

You can try approaching the problem without using an actual browser and sending HTTP requests using requests and parsing HTML using, say, BeautifulSoup. This though would involve diving into how the page is formed, is more error-prone and would be far more suspicious and easier to detect in general. Of course, with no browser overhead, this would be much faster.

There are some other relevant tools like the Scrapy web-scraping framework. Which, if needed, can be combined with a JavaScript engine - see scrapy-splash.

There are though some things you can try to speed things up in your current approach:

  • remove time.sleep() hardcoded delays and switch to using Explicit Waits based on WebDriverWait and Expected Conditions
  • avoid using XPaths as it is generally the slowest element location technique (of course, that's not a bottleneck here)
  • you can pass certain flags to PhantomJS to help it load pages faster - like --load-images
  • you can try switching to headless Chrome instead of PhantomJS - I've seen benchmarks showing that the headless Chrome beats PhantomJS in terms of both speed and memory usage
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.