Testing framework with a single assertion macro

Question

I usually use the Catch2 testing framework in my C++ projects. Unlike other frameworks, it has only one core assertion macro for all comparisons. Instead of writing REQUIRE_EQ(value, correct); one could write REQUIRE(value == correct);.

I looked up how it was implemented and replicated its core feature. I concluded that I could use macros to extend the TEST(value == correct) to the verify(Decomposer {} <= value == correct, "value == correct") and the TEST(value) to the verify(Decomposer {} <= value). In case of failure now both value and correct would be saved in variables for printing the values instead of the result of comparison.

The expression Decomposer {} <= value == correct first evaluates Decomposer {} <= value which becomes of type UnaryExpression. It has value as its member. Then unaryExpression == correct becomes of the type BinaryExpression. It can compare the value to the correct while having both values as a member. In case of failure, both values can be printed.

I deliberately kept the features to a minimum and probably won't be continuing this project further. I would like to get feedback on

• Use of macros. I usually don't use any macros in my programming. The TEST macro is needed when printing variable names. Also like in Catch2 I use macro when generating all operations.
• Use of templates and polymorphism. Similar things can be achieved with templates and polymorphism. The other one is static and the other is dynamic. I used expression as a base class instead of having it as a template parameter and I used templates to have any types given into my expression classes.
• What would be the next small features that would be important for testing the library? How would you implement them?

Here is the result:

#include <iostream>
#include <sstream>
#include <type_traits>


const std::string unprintable = "{?}";

template<typename T>
concept Writable = requires(T a) {
  { std::cout << a } -> std::same_as<std::ostream&>;
};


template<typename T>
std::string stringify(const T& value) {
  return unprintable;
}

template<Writable T>
std::string stringify(const T& value) {
  std::ostringstream out;
  out << value;
  return out.str();
}


class Expression {
  public:
    virtual ~Expression() = default;
    virtual bool result() const = 0;
    virtual std::string expand() const = 0;
};

template<typename Left, typename Right>
class BinaryExpression: public Expression {
  public:
    explicit BinaryExpression(bool result, Left left, std::string op, Right right)
      : comparison_result(result), left(left), op(op), right(right)
    {}

    bool result() const override {
      return comparison_result;
    }

    std::string expand() const override {
      return stringify(left) + ' ' + op + ' ' + stringify(right);
    };

  private:
    bool comparison_result;
    Left left;
    std::string op;
    Right right;
};

template<typename Left>
class UnaryExpression: public Expression {
  public:
    explicit UnaryExpression(Left expression)
      : expression(expression)
    {}

#define OPERATOR(op) \
    template<typename Right> \
    friend BinaryExpression<Left, Right> \
    operator op(UnaryExpression l, Right r) { \
      return BinaryExpression { l.expression op r, l.expression, #op, r }; \
    }

    OPERATOR(==);
    OPERATOR(!=);
    OPERATOR(<);
    OPERATOR(<=);
    OPERATOR(>);
    OPERATOR(>=);

#undef OPERATOR

    bool result() const override {
      if constexpr (std::is_convertible_v<Left, bool>) {
        return static_cast<bool>(expression);
      }
      return false;
    }

    std::string expand() const override {
      return stringify(expression);
    }

  private:
    Left expression;
};


struct Decomposer {
  template<typename T>
    friend UnaryExpression<T> operator<=(Decomposer, T left) {
      return UnaryExpression<T> { left };
    };
};

void verify(const Expression& result, std::string code) {
  if (result.result()) {
    std::cerr << "pass\n";
  } else {
    std::cerr << "fail:\n"
      << "  got  " << code << '\n'
      << "  with " << result.expand() << '\n';
  }
}

#define TEST(...) verify((Decomposer {} <= __VA_ARGS__), #__VA_ARGS__)


struct Point {
  friend auto operator<=>(const Point& l, const Point& r) = default;
  friend std::ostream& operator<<(std::ostream& out, const Point& point) {
    return out << '(' << point.x << ',' << point.y << ')';
  }
  int x, y;
};

struct Point3d {
  friend auto operator<=>(const Point3d& l, const Point3d& r) = default;
  int x, y, z;
};

int main() {
  TEST(true);
  TEST(1 == 1);
  TEST(1 != 2);
  TEST(1 < 2);
  TEST(1 <= 1);
  TEST(2 > 1);
  TEST(2 >= 2);
  TEST(Point { 1, 2 } == Point { 1, 2 });
  TEST(Point { 1, 2 } >=  Point { 1, 1 });
  TEST(Point3d { 1, 2, 3 } >=  Point3d { 1, 1, 1 });

  TEST(false);
  TEST(1 == 2);
  TEST(1 != 1);
  TEST(2 <= 1);
  TEST(2 < 2);
  TEST(1 >= 2);
  TEST(1 > 1);
  TEST(Point { 1, 2 } == Point { 1, 1 });
  TEST(Point { 1, 2 } <=  Point { 1, 1 });
  TEST(Point3d { 1, 2, 3 } <  Point3d { 1, 1, 1 });
}

Code can be run with g++ -Wall -Wno-parentheses -g -std=c++2a -o testing main.cpp && ./testing to avoid warnings from macros.

Answer 1

This is nice - good job!

I'd move the C++ identifiers into a namespace so that they don't conflict with names from the code under test.

I get a warning from the unused identifier here:

template<typename T>
std::string stringify(const T& value) {
  return unprintable;
}

It's easy to re-write as std::string stringify(const T&).

I would add a couple of extra arguments to verify() so that TEST() can pass __FILE__ and __LINE__ for a properly formatted failure message (so we can jump straight to it in compilation-mode, for example).

We're missing important functionality - there's no way to know whether all the tests passed, so our demo program exits with success code even though some tests failed. That makes this harder to use in a build system.