2
\$\begingroup\$

I'm using Selenium to test a PHP site that goes through the following process:

  1. Register user
  2. Log in
  3. Fill out Form 1 and submit
  4. Fill out Form 2 and submit
  5. Fill out Form 3 and submit
  6. (tear down) Delete user

A real user would go through these steps in this order.

I've read in places that tests should be bite-sized, and to avoid asserting a lot of things in each test. So I've written my tests like this:

function testRegister() {
    //...
}

/**
 * @depends testRegister
 */
function testLogin() {
    //...
}

/**
 * @depends testLogin
 */
function testFillForm1() {
    //...
}

/**
 * @depends testFillForm1
 */
function testFillForm2() {
    //...
}

/**
 * @depends testFillForm2
 */
function testFillForm3() {
    //...
}

This way my tests execute in the correct order.

However, splitting up my tests is merely cosmetic. Since each test is dependent on the success of the previous one, it may as well be one long test.

It also causes another problem. If testFillForm2 fails, I have no way of re-running the test on its own, I have to wait for all the other tests to complete each time because of the dependencies.

Is there a better approach I can take?

\$\endgroup\$
  • 2
    \$\begingroup\$ The "bite sized tests" remark is certainly true for unit tests (which test a single chunk of code in isolation), but I don't think it's necessarially so for a functional test (which tests several components at once to make sure they interact as intended). \$\endgroup\$ – GordonM May 11 '12 at 7:54
3
\$\begingroup\$

@depends does not define order

[@depends] do not define the order in which the test methods are to be executed

Using @depends and not returning/passing arguments is an indicator that @depends isn't appropriate for what you're currently doing. You need to have a consistent application state before running each individual test method so that they do not depend on execution order. Otherwise tests easily become quite unwieldy and ultimately of little value.

If you want to do that via @depends, you need to return from a test method and the following method should expect it as the first argument (and literally depend on whatever it receives).

Reduce dependencies

In tests you should aim to reduce, not add dependencies, otherwise you can be in quite a fragile position. If for example your register function breaks, it wouldn't affect existing users, or therefore all other tests, at all - but using cascading tests as shown you would be unable to test/demonstrate that.

A broken register function would certainly be a problem - but is it a "100% of the site is broken" problem? It makes test results un-representative, and potentially harder to identify the root cause. If you have one failing test - it's very obvious where to focus your attention just from the test results. If all tests fail, it is not.

Of course anything more than of 0% fails warrants developer attention.

Achieve consistent application state

So, for example, each test should:

  • Establish a consistent application state (setUp functionality)
  • Use the concept of fixtures to e.g. populate the session, cookies, db - whatever is required
  • test what it needs to test
  • cleanup after itself (tearDown functionality)

In Context, I would say that means:

Create a test user to be able to test logging in without first being forced in the test case to "test" your register function:

public funtion testLogin() {
    // Insert a user directly into the users table - simulate anything else from testRegister
    ...
    $this->assertSame("UserName", $_SESSION['user']['name']);
}

You setup the session/cookie/whatever state for each subsequent test so that you don't need to "test" your login function to be able to test other authorized functions:

public funtion testFillForm1() {
    $this->_login();
    ...
}

public funtion testFillForm2() {
    $this->_login();
    ...
}

public funtion testFillForm3() {
    $this->_login();
    ...
}

protected function _login($user = array()) {
    if (!$user) {
        $user = array('name' => 'Default');
    }
    $_SESSION['user'] = $user;
}

If any of your setup steps are generic enough to apply to all tests, put them in the setUp function. E.g. loading 10 test users, with various permissions - put that in setUp and therefore testing login, registration-"that username already exists" would need no in-test setup at all.

I'm not exactly sure how that applies to using selenium, but hopefully this gives you enough of a hint to remove some of your inter-test dependencies.

\$\endgroup\$
  • \$\begingroup\$ Isn't inserting a user manually into the database before the test a bit dodgy? I can't guarantee that the app would be in the same state as if I went through the actual login/registration process in Selenium (particularly as it's built on a CMS and versions might change). \$\endgroup\$ – waitinforatrain May 11 '12 at 17:22
  • 1
    \$\begingroup\$ If, as is now apparent, it's code that you need to treat as a black box - depends. You really, really don't want false test dependencies. E.g. what if the register functionality changed but was not broken and it took a week to figure out how to test it. You'd be completely blind to other test failures for a week. Inserting a user into the db was merely a hint - do whatever you need to have a consistent application state. Insert into the db, delete stuff, restore a db dump - just put yourself in the position to reduce your test dependencies. \$\endgroup\$ – AD7six May 15 '12 at 7:46
  • 1
    \$\begingroup\$ As an example: On a long-past project I would restore the whole disk image to have a consistent state before running the test suite (on another machine) because the application made changes all over the machine - db, windows registry, tmp files etc. But here you're talking about a typical php app - that's certainly not necessary. The principle is the same though: achieve a consistent application state. \$\endgroup\$ – AD7six May 15 '12 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.