Making a reservation using Selenium/PhantomJS

Question

I have this small script that needs to make a reservation at a DMV office.

The problem is that the reservation window opens only every few hours for a few minutes (it might show up in one of these months: December, January, February or March of next year). My code is too slow to even "catch" the "open day".

Any ideas on how I can speed up the whole process? Maybe a different approach?

from selenium import webdriver
import time

firstName = "John"
lastName = "Smith"
yob = "1929"
email = "[email protected]"
phone = "555-123-4567"

browser = webdriver.PhantomJS()
browser.get("https://page/page2/ACFCustom/Se.aspx")

def check_date_avaialbility():
    # Choose a service
    browser.find_element_by_name("ct$cphBy$rS$ctl$btn1").click()
    # Choose an office
    browser.find_element_by_name("cphBy_rpt_btn1").click()

    # Loop through the next 4 months and search for an open window (The day will be highlighted in green)
    for i in range(1, 5):
        try:
            print("Searching for available day... ")
            browser.find_element_by_xpath("//*[contains(@style, 'background-color:#4A0889;')]").click()
            browser.find_element_by_xpath("//*[@id='cphBy_rblAT']").click()
            browser.find_element_by_id("cphBy_Nt").click()
            form_data()
        except:
            print("There are no available dates. Next page... \n")
            browser.find_element_by_name("ctl_cphBy_rdcAD_NN").click()
            time.sleep(1)

    # If you couldn't find anything three. Go back to the "home" page 
    browser.find_element_by_name("cphBy_lnkbnK").click()
    time.sleep(1)
    # Start going through all the next 4 months all over again
    check_date_avaialbility()

def form_data():
    print("Filling up the form...")
    browser.find_element_by_id("cphBy_txbFN").send_keys(firstName)
    browser.find_element_by_id("cphBy_txbLN").send_keys(lastName)
    browser.find_element_by_id("cphBy_txbYB").send_keys(yob)
    browser.find_element_by_id("cphBy_txbE").send_keys(email)
    browser.find_element_by_id("cphBy_txbPN").send_keys(phone)
    browser.find_element_by_id("cphBy_S").click()

check_date_avaialbility()

Answer 1

You can try approaching the problem without using an actual browser and sending HTTP requests using requests and parsing HTML using, say, BeautifulSoup. This though would involve diving into how the page is formed, is more error-prone and would be far more suspicious and easier to detect in general. Of course, with no browser overhead, this would be much faster.

There are some other relevant tools like the Scrapy web-scraping framework. Which, if needed, can be combined with a JavaScript engine - see scrapy-splash.

There are though some things you can try to speed things up in your current approach:

• remove time.sleep() hardcoded delays and switch to using Explicit Waits based on WebDriverWait and Expected Conditions
• avoid using XPaths as it is generally the slowest element location technique (of course, that's not a bottleneck here)
• you can pass certain flags to PhantomJS to help it load pages faster - like --load-images
• you can try switching to headless Chrome instead of PhantomJS - I've seen benchmarks showing that the headless Chrome beats PhantomJS in terms of both speed and memory usage