Generic test case templates

Question

I often find that when I am writing, refactoring, or reviewing code that I want to do some simple testing. There are many existing test frameworks such as gtest and cppunit but my desire was to create something much simpler and with fewer features.

Specifically, I have two use cases in mind:

1. I have some test input values with known desired outputs
2. I have an existing function that I want to optimize but keep correct

For both of these, I've create a very simple pair of templated objects after studying my own use of such techniques in existing code. In looking at how I use such code, I find that I have three general uses:

1. I want pretty color printing to the screen with good values in green and bad ones in red
2. I want to redirect the output to a file without the pretty colors
3. I want to run all the tests and just silently get a bool that indicates whether all tests passed

Here are the templates I'd like to get reviewed.

testcase.h

#ifndef EDWARD_TEST_CASE
#define EDWARD_TEST_CASE

#include <string>
#include <string_view>
#include <vector>
#include <iomanip>
#include <iostream>

namespace edward {

static const std::string CSI{"\x1b["};
static const std::string RED{CSI + "31m"};
static const std::string GREEN{CSI + "32m"};
static const std::string RESET{CSI + "0m"};

static const std::string badgood[2][2]{ 
    { "[BAD] ", "[OK] " },
    { RED + "[BAD] " + RESET, GREEN + "[OK]  "+ RESET },
}; 

template <class InputType, class OutputType>
class TestCollection {
    struct TestCase {
        InputType input;
        OutputType expected;
    };
public:
    TestCollection(OutputType (*testfunc)(InputType), std::vector<TestCase> tests) 
        : testfunc{testfunc}
        , tests{tests}
    {}
    bool testAll(bool verbose = true, bool color = true) const {
        return verbose ? verboseTest(color) : quietTest();
    }
private:
    bool quietTest() const {
        bool good{true};
        for (const auto& t : tests) {
            good &= (testfunc(t.input) == t.expected);
        }
        return good;
    }
    bool verboseTest(bool color = true) const {
        bool good{true};
        std::size_t i{0};
        for (const auto& t : tests) {
            auto result = testfunc(t.input);
            bool isOK = result == t.expected;
            good &= isOK;
            std::cout << badgood[color][isOK]
                << "Test #" << i << ": "
                << std::boolalpha << "got \"" << result 
                << "\", expected \"" << t.expected 
                << "\" from \"" << t.input << "\"\n";
        }
        return good;
    }
    OutputType (*testfunc)(InputType);
    std::vector<TestCase> tests;
};

template <class InputType, class OutputType>
class DynamicTest {
public:
    DynamicTest(OutputType (*testfunc)(InputType), OutputType (*trustedfunc)(InputType))
        : testfunc{testfunc}
        , trustedfunc{trustedfunc}
    {}
    bool test(InputType in, bool verbose = true, bool color = true) const {
        return verbose ? verboseTest(in, color) : quietTest(in);
    }
private:
    bool quietTest(InputType in) const {
        return testfunc(in) == trustedfunc(in);
    }
    bool verboseTest(InputType in, bool color = true) const {
        OutputType result{testfunc(in)};
        OutputType expected{trustedfunc(in)};
        bool isOK{result == expected};
        std::cout << badgood[color][isOK]
            << std::boolalpha << "got \"" << result 
            << "\", expected \"" << expected 
            << "\" from \"" << in << "\"\n";
        return isOK;
    }

    OutputType (*testfunc)(InputType);
    OutputType (*trustedfunc)(InputType);
};

}; // end of namespace EDWARD

#endif // EDWARD_TEST_CASE

Here is some sample test code that exercises both templates and illustrates the intended use. Note that these use this idiom to decide whether or not to print color:

bool color{static_cast<bool>(isatty(STDOUT_FILENO))};

This works on Linux (or any POSIX compliant) machines but is not, to my knowledge, portable to Windows. Rather than make the template non-portable, I simply omit it from the templates and rely on the caller. Also note that the printing is only done to std::cout in the template. Here again, I found that it's the only way I used existing code, so I didn't incorporate features I've never used.

main.cpp

#include "testcase.h"
#include <iostream>
#include <string>
#include <algorithm>
#include <cmath>
#include <functional>
#include <unistd.h>

bool isInteger(const std::string& n) {
    if (n.size() > 1) {
        return std::all_of(n.begin() + 1, n.end(), isdigit)
            && (isdigit(n[0]) || (n[0] == '-') || (n[0] == '+'));
    }
    return isdigit(n[0]);
}

void one() {
    bool verbose{true};
    bool color{static_cast<bool>(isatty(STDOUT_FILENO))};

    static const edward::TestCollection<const std::string&, bool> tc{ isInteger, {
        {"+", false},
        {"-", false},
        {"0", true},
        {"3", true},
        {"9", true},
        {"a", false},
        {"99a9", false},
        {"9909", true},
        {"", false},
        {"this test lies", true},  // deliberately bad test
        {"-3.14", false},
        {"+32768", true},
        {"-32768", true},
    }};
    auto result{tc.testAll(verbose, color)};
    std::cout << "All tests " << (result ? "passed" : "did NOT pass") << "\n";
}

int square_it(int x) {
    return std::pow(x, 2);
}

void two() {
    bool verbose{true};
    bool color{static_cast<bool>(isatty(STDOUT_FILENO))};

    edward::DynamicTest<int, int> dt{[](int x)->int{return x*x;}, square_it };
    bool result{true};
    for (int i{-5}; i < 5; ++i) {
        result &= dt.test(i, verbose, color);
    }
    std::cout << "All tests " << (result ? "passed" : "did NOT pass") << "\n";
}

int main() {
    one();
    two();
}

I'm interested in a general review.

Answer 1

C++ 17

Why limit yourself to C functions?

DynamicTest(OutputType (*testfunc)(InputType), OutputType (*trustedfunc)(InputType))

I would declare this as:

DynamicTest(std::function<OutputType(InputType)>&& testfunc, std::function<OutputType(InputType)>&& trustedfunc)

Also the C++ versions for function declarations are easier to read.

---

When passing your test pass by reference and allow both copy and move semantics. Also do you want to force a vetor. Might be better to use an initializer list:

TestCollection(OutputType (*testfunc)(InputType), std::vector<TestCase> tests)
//                                                                      ^^^^^

Pass by value adds an unneeded copy.

TestCollection(OutputType (*testfunc)(InputType), std::vector<TestCase> const& tests);
TestCollection(OutputType (*testfunc)(InputType), std::vector<TestCase>&& tests);
TestCollection(OutputType (*testfunc)(InputType), std::initializer_list<TestCase> const& tests);
// The last one allows you to build the test cases in place in the constructor.

---

With only minor modification to the code you factor out the following methods into a base class:

        bool quietTest() const;
        bool verboseTest(bool color = true) const;

I would create a common base class with the following interface:

          template<typename C>
          bool quietTest(C const& tests) const;
          template<typename C>
          bool verboseTest(C const& tests, bool color = true) const;

Then they can be called like this:

return verbose ? this->verboseTest(tests, color) : this->quietTest(tests);

// and

std::vector<TestCase>   data{TestCase{in, trustedfunc(in)}};
return verbose ? this->verboseTest(data, color) : this->quietTest(data);

---

Be consistent.

namespace edward {
// STUFF
}; // end of namespace EDWARD

 ^ Don't need the semi-colon.

Be consistent in naming.

edward or EDWARD

I don't mind the lowercase edward but would avoid EDWARD but they should be the same. Somebody using a less powerful editor may copy and paste the EDWARD into a search to find the begining of the namespace.