Controlling the order of unittest.TestCases

Question

Apparently everyone gets burned by their Python unittests not running in the order they want.

I am not in the business of telling people not to do perfectly reasonable things they want to do, so I consider "write your tests differently" Not An Answer to the question "how do I control the order of my TestCase subclasses".

With that in mind, I also consider "Why do you want to control the order of unittests? Just write them differently" as a lame, non-answer to this CR post.

I do, however, consider the following (more than) an answer to the above question:

import unittest

def suiteFactory(
        *testcases,
        testSorter   = None,
        suiteMaker   = unittest.makeSuite,
        newTestSuite = unittest.TestSuite
    ):
    """
    make a test suite from test cases, or generate test suites from test cases.

    *testcases     = TestCase subclasses to work on
    testSorter     = sort tests using this function over sorting by line number
    suiteMaker     = should quack like unittest.makeSuite.
    newTestSuite   = should quack like unittest.TestSuite.
    """

    if testSorter is None:
        ln         = lambda f:    getattr(tc, f).__code__.co_firstlineno
        testSorter = lambda a, b: ln(a) - ln(b)

    test_suite = newTestSuite()
    for tc in testcases:
        test_suite.addTest(suiteMaker(tc, sortUsing=testSorter))

    return test_suite


def caseFactory(
        scope        = globals().copy(),
        caseSorter   = lambda f: __import__("inspect").findsource(f)[1],
        caseSuperCls = unittest.TestCase,
        caseMatches  = __import__("re").compile("^Test")
    ):
    """
    get TestCase-y subclasses from frame "scope", filtering name and attribs

    scope        = iterable to use for a frame; preferably a hashable (dictionary).
    caseMatches  = regex to match function names against; blank matches every TestCase subclass
    caseSuperCls = superclass of test cases; unittest.TestCase by default
    caseSorter   = sort test cases using this function over sorting by line number
    """

    from re import match

    return sorted(
        [
            scope[obj] for obj in scope
                if match(caseMatches, obj)
                and issubclass(scope[obj], caseSuperCls)
        ],
        key=caseSorter
    )


if __name__ == '__main__':

    cases = suiteFactory(*caseFactory())
    runner = unittest.TextTestRunner(verbosity=2)
    runner.run(cases)

A gist.

For reference, here're some example tests:

import unittest

class Test_MyTests(unittest.TestCase):
    def test_run_me_first(self): pass
    def test_2nd_run_me(self):   pass
    def test_and_me_last(self):  pass

class Test_AnotherClass(unittest.TestCase):
    def test_first(self):        pass
    def test_after_first(self):  pass
    def test_de_last_ding(self): pass

if __name__ == "__main__": unittest.main(verbosity=2)    

(The names are all unittest cares about, and all I need for demonstration.)

Here's what running that looks like:

test_after_first (__main__.Test_AnotherClass) ... ok
test_de_last_ding (__main__.Test_AnotherClass) ... ok
test_first (__main__.Test_AnotherClass) ... ok
test_2nd_run_me (__main__.Test_MyTests) ... ok
test_and_me_last (__main__.Test_MyTests) ... ok
test_run_me_first (__main__.Test_MyTests) ... ok

Oh no! My tests weren't run in the order I thought they'd be wanted.

Running the content of the gist, aka same tests, but replacing the ifmain with the full code from above:

test_run_me_first (__main__.Test_MyTests) ... ok
test_2nd_run_me (__main__.Test_MyTests) ... ok
test_and_me_last (__main__.Test_MyTests) ... ok
test_first (__main__.Test_AnotherClass) ... ok
test_after_first (__main__.Test_AnotherClass) ... ok
test_de_last_ding (__main__.Test_AnotherClass) ... ok

Success! The tests were run based on where in the file they were defined.

I think this is pretty useful, and quite optimal. But last time I thought that, I was really wrong.

---

^{Incidentally, if you don't want the TestCases to run all as a single suite, but as individual suites with individual runners, just change suiteFactory to be a generator, and change the ifmain to iterate over said generator. I way prefer when my tests run all together, and functions are either generators or they aren't, hence the way it's written.}

Answer 1

A search for "python sort unit test order in method definition order" got me here, and the documentation isn't particularly useful either ("Function" it says, what is it passed, what is it expected to return?)

Reading the four questions linked to at the start of this one yields only three unique answers:

1. Don't sort your test methods (not an answer).
2. Put numbers in your test method names (ugly, and means you need to renumber them all if you want to put new tests in the middle, or use ever-growing strings, etc.).
3. Go through all this rigmarole, which it looks like the poster had success with but isn't exactly easy to understand and digs into Python 3's internals for the function line number.

sortTestMethodsUsing expects a function like Python 2's cmp, which has no equivalent in Python 3 (I went to check if Python 3 had a <=> spaceship operator yet, but apparently not; they expect you to rely on separate comparisons for < and ==, which seems much a backwards step...). The function takes two arguments to compare, and must return a negative number if the first is smaller. Notably in this particular case, the function may assume that the arguments are never equal, as unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated functions appear in the file. As far as I know, this is intended, and I'd be surprised if it changed, but I don't actually know if it's guaranteed behavior. I think this solution will even work in Python 2 as well, for those who are unfortunately stuck with it, though I haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class definition, though how this can be used with sortTestMethodsUsing will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test order being fixed - and I fully understand that you shouldn't rely on test order, and this is the reason people use to avoid answering this question. The order of my tests could be randomised and it'd work just as well. However, there is one very good reason I'd like the order to be fixed to the order they're defined in the file: it makes it so much easier to see at a glance which tests failed.

Answer 2

Here is a simpler method that has the following advantages:

• No need to create a custom TestCase class.
• No need to decorate every test method.
• Use the unittest standard load test protocol. See the Python docs here.

The idea is to go through all the test cases of the test suites given to the test loader protocol and create a new suite but with the tests ordered by their line number.

Here is the code:

import unittest

def load_ordered_tests(loader, standard_tests, pattern):
    """
    Test loader that keeps the tests in the order they were declared in the class.
    """
    ordered_cases = []
    for test_suite in standard_tests:
        ordered = []
        for test_case in test_suite:
            test_case_type = type(test_case)
            method_name = test_case._testMethodName
            testMethod = getattr(test_case, method_name)
            line = testMethod.__code__.co_firstlineno
            ordered.append( (line, test_case_type, method_name) )
        ordered.sort()
        for line, case_type, name in ordered:
            ordered_cases.append(case_type(name))
    return unittest.TestSuite(ordered_cases)

You can put this in a module named order_tests and then in each unittest Python file, declare the test loader like this:

from order_tests import load_ordered_tests

# This orders the tests to be run in the order they were declared.
# It uses the unittest load_tests protocol.
load_tests = load_ordered_tests