C++ Minimalistic Unit Testing Library

Question

I was looking for a unit testing library for C++, and I found many. Unfortunately, I decided I didn't want to use any of them. I decided to make my own.

This is the result. I made heavy use of the preprocessor, and followed the convention of leading underscore to mean non-published. Everything is done in all-caps, to try to follow good conventions and to avoid name collisions.

Please note that this code is published under the MIT license.

unittest.h

#pragma once

#include <vector>
#include <iostream>

// Helper Macros
#define _STR_HELP(x) #x
#define _STR(x) _STR_HELP(x)
#define ESCAPE(...) __VA_ARGS__ // TEST_CASE(std::map<int, int>) would fail. Wrap in this Macro to not fail.

#define _FILE_LINE __FILE__ ":" _STR(__LINE__)

// Assertion Macros
#define _GENERATE_MESSAGE(test) _FILE_LINE " Failed " #test "." // Message in almost every assertion

/*
 * Assertions are intended to be used within a TEST_CASE macro. We return EXIT_FAILURE if we fail the assertion,
 * meaning that the TEST_CASE fails. Assertions can also be used straight inside the main function, in which case
 * the main function will exit. They are intended to be used within the TEST_CASE macro, so it's not recommended to do
 * that.
 */
#define _ASSERT(test, message)  if (!(test)) {\
                                    std::cerr << message << std::endl;\
                                    return EXIT_FAILURE;\
                                }
// One argument assertion. This generates a message to display.
#define _ASSERT_1(test) _ASSERT(test, _GENERATE_MESSAGE(test))
// Two argument assertion, which appends the user-defined message to the generated one.
#define _ASSERT_2(test, additional) _ASSERT(test, _GENERATE_MESSAGE(test) " " additional)

// Helper macro to get the correct assertion (out of _ASSERT_1 or _ASSERT_2)
#define _GET_ASSERTION(_1, _2, NAME, ...) NAME

// Public Macro for actually calling the ASSERT
#define ASSERT(...) _GET_ASSERTION(__VA_ARGS__, _ASSERT_2, _ASSERT_1)(__VA_ARGS__)
// More advanced macro than ASSERT; it tells the values of each of the arguments as part of the message.
#define ASSERT_EQ(a, b) {\
                            const auto& _VAL_1 = a;\
                            const auto& _VAL_2 = b;\
                            if (_VAL_1 != _VAL_2) {\
                                std::cerr << _FILE_LINE " Failed " #a " == " #b ". Actual Values:\n" \
                                    #a ": " << _VAL_1 << "\n" \
                                    #b ": " << _VAL_2 << std::endl;\
                                return EXIT_FAILURE;\
                            }\
                        }

/*
 * Convenience struct for storing test cases and all related data into a vector.
 */
struct _TEST_CASE
{
    const char *name;
    int (*function)();
    bool result;
};
std::vector<_TEST_CASE> _TEST_CASES; // All test cases are added to this global vector.

// Test Execution Macros
#define TEST_CASE(name, code) _TEST_CASES.push_back({ name, [](){ code; return EXIT_SUCCESS; }, false });

#define RUN_TESTS() std::cout << std::endl; \
                    for (_TEST_CASE& testCase : _TEST_CASES) {\
                        testCase.result = EXIT_SUCCESS == (*testCase.function)();\
                        if (!testCase.result) {\
                            std::cerr << "Failed test case " << testCase.name << "\n";\
                        }\
                    }\
                    std::cout << "Results: ";\
                    for (_TEST_CASE testCase : _TEST_CASES) {\
                        std::cout << (testCase.result ? '.' : 'F');\
                    }\
                    std::cout << std::endl << std::endl;

Answer 1

Unfortunately, I decided I didn't want to use any of them. I decided to make my own.

I took the same way with my unit testing (for personal projects I develop at home) but for production code, this is a bad decision to make.

I stated my unit testing lib roughly two years ago, and every two or three weeks, I keep adding features to it (and it is still not complete).

Here are some things I would not do (and why):

I made heavy use of the preprocessor [...]

That's a bad call. Ideally, you should only use the preprocessor when no other alternative exists. In this case, many many alternatives exist.

and followed the convention of leading underscore to mean non-published.

This potentially causes your code to exhibit UB because leading underscore followed by capital letter is reserved for standard library impleemnters (I think).

You also used the same coding convention for code and macros (please don't).

The way you use macros ensures client code cannot avoid using them to write tests. If you redesign your API to not rely on macros, you can then add the macros later with minimal effort). This will make your code maintainable (it's easier to maintain C++ functions than macros) and will not impose macros on the client code.

Some features you may wish to add (complementing the list provided by Loki):

• test suite support
• automatic processing of exceptions in your unit tests:
  • expected exceptions (testing that your code correctly identifies and reacts to error scenarios)
  • unexpected exceptions (should cause your tests to fail gracefully and report the errors)

• code checkpoints: this is a (usually transparent) feature, that marks the last executed line in a test (last unit test API file and line, or last _ASSERT macro call for example); if an unexpected exception occurs, that location is reported, automatically restricting the range of code you have to check to fix the issue.

• disconnected/customized reporting of results; Ideally, you should be able to plug in a file writer, an XML logger or anything else into a unit test suite and generate the same test output report in various formats.

Other problems:

• the code is monolythic (you cannot choose to use something else than std::cerr in the macros, because it is hard-coded - instead of being injected into the code).

• the code is difficult to maintain (this is a classic problem of abusing macros)

As a point of comparison, here's the how unit tests look with my (custom) library:

void bad_command(unittest::test_context& ctx)
{
    // tested scenario here

    ctx.check_equal(1, 2); // will fail: 1 != 2
}

int main(int argc, char* argv[])
{
    unittest::runtime_args args{ argv, argv + argc };
    auto suite = unittest::make_test_suite("test-utility-apis",
         std::cout, args);

    suite.add_test("bad_command", bad_command); // one call per unit test

    return suite.run();
}

This code contains no macros.

The passing of runtime args. to the test suite allows for: selection of output format, filtering of executed tests based on args and (probably in the future) more runtime arguments (parallel execution, etc).

Answer 2

I decided I didn't want to use any of them. I decided to make my own.

That's a mistake.

and followed the convention of leading underscore to mean non-published.

A leading underscore followed by an uppercase letter is reserved identifier reserved for the implementation.

Please note that this code is published under the MIT license.

Also note that this code is not published under the creative commons license. See the bottom of the page. Everything published on any stack-exchange site is available under creative commons.

Design

There are a lot of missing features that I would expect in unit test framework.

• Ability to run tests in isolation
• Ability to run groups of tests.
• Ability to set up state for tests
• Ability to tear down state and reset the environment.
• Some facility for mocking other classes.