Improving performance in a Webdriver method

Question

I'm testing an application which is not exactly an e-commerce application but which behaves enough like one that you can think of it as an e-commerce application for the purposes of understanding this code :)

This code relies heavily on the Selenium Webdriver library.

The code comes from the Product page, where you have searched and found a list of products. This is a utility method to get the list of products being displayed so I can verify correctness along any axis. This method only gets those products that are visible, which may differ from the total list of products if filtering is involved.

I maintain a list of WebElements representing the container for the product display, one for each product on the page, which is initialized in initLists() (along with some other non-important lists) so that we don't have stale references. That's what's stored in allProducts.

ProductView is an inner class that encapsulates a specific Product record in the list; interactions with a given product (such as clicking any of the links for more information) are delegated to the ProductView for that Product.

public ArrayList<Product> getAllVisibleProducts() {
    initLists(); //Reinitialize lists

    Utils.WaitForAjax(driver);
    ArrayList<Product> retval = new ArrayList<Product>();
    for (WebElement rProduct : allProducts) {
        try {
            if (rProduct.isDisplayed()) retval.add(new ProductView(rProduct).getProduct());
        } catch (StaleElementReferenceException e) {
            continue; //We just ignore that element
        }
    }
    return retval;
}

I'm getting hangs of sometimes multiple minutes where the test appears to do nothing on the product page. I'm currently still trying to reproduce the super-long lag, but at least part of my problem is that this method takes quite a long time to return. I put log statements at various portions of the app, and between starting the loop and returning, it was something like 44 seconds for 8 products. Am I doing too much web scraping here?

(ProductView stores references to the elements that are interactable for the given product and attempts to build an in-memory representation of that product by scraping the page using Webdriver. The code to build the product is the vast majority of the constructor and clocks in about 70-odd lines, so I omitted it).

Ideally I think I need to hear from someone who has used WebDriver before, since this probably boils down to WebDriver best practices.

Answer 1

You claim "ProductView stores references to the elements that are interactable for the given product and attempts to build an in-memory representation of that product by scraping the page using Webdriver", could the lag simply be due to the time it takes to scrape the page then?

Is this web scraping taking place over the public Internet, or can you set up a simple caching mechanism on top of your application to reduce load times? An internal web server just to facilitate testing should work too. Ideally, you should have a stub version of ProductView that you can test without having to perform the web scraping.

The one final minor optimization I can think of (a really minor point, if I haven't emphasized enough) is to make initialize retval to the size of allProducts first, so that there is definitely no need to resize it later.

Otherwise, stick with @Marc-Andre's answer, and look towards tackling the web scraping mechanism.

Answer 2

Bunch of small things not related to your performance problem:

• Utils.WaitForAjax should be Utils.waitForAjax. Methods in Java should start with a lower letter.
• Don't return ArrayList; it's an implementation. You should use the interface List. The day you want to change to a LinkedList, it will be easier.
• I recommend to always use {} for if, even if you only have just one line. But that is up to you.
• I like to name ignoredException an exception that I'm ignoring. Make it more clear.

---

By looking at your code, I can only guess that your performance bottleneck probably is new ProductView(rProduct) or rProduct.isDisplayed(). If in ProductView you're scraping the page, it could take a lot of time depending on the driver and the method you're doing it.

But with the code you're showing us right now, there is nothing particularly wrong.