Function composition in the context of data processing pipelines

Question

Prior Notification

• This follows a previous review of mine that addressed the core helper function named make_skippable.
• The composition implementation presented here is heavily inspired by another review that already introduced the basic concept: Function composition in C++. Thanks to Nestor for providing the pattern.

The Problem

This concerns the construction of data processing pipelines. Given a series of callables, each designed to transform input into output, the goal is to find an intuitive and straightforward method to link these callables together in a chain, by creating a single, unified callable, the composition. The composition should be capable of accepting the same argument as the first callable in the chain and should return the output of the last callable in the chain. That‘s basically it. Below is a detailed list of requirements that any effective solution should satisfy.

• Language Standard: c++17
• Ease of Use: constructing a composition out of given callables should be as straightforward as possible
• Supported Callable Types: functions, lambdas, and functors
• Callable Signatures: All callables to be composed must accept a single argument and return a non-void type. Callables that return void are excluded as they cannot be chained in data processing flows.
• Composition Object: The resulting composition is a callable itself, as such it it assignable to std::function.
• Breakable Chain Mechanism: Any callable may break the chain by not providing a result. In consequence the subsequent callables are not executed (skipped). Not providing a result can be considered a usual outcome and does not need to be an error.
• Error Handling: The implementation should be exception-safe. Exceptions thrown by any callable in the chain should be propagated to the caller of the composition.
• Generic and Overloaded Callable Support: Generic lambdas, generic functors and overloaded functors shall be supported as callables to be composed. This means a single composed chain might take different input argument types that might even result in different output types. Thus, the composition signature is as flexibel as the signatures of the composed callables itself (see Generic Lambda Example below).
• Compile-Time Signature Validation: In case of mismatching callable signatures, meaningful compile time error messages should get provided.
• Move Semantics and Perfect Forwarding: The implementation must fully support move semantics for callables and their arguments.

My Solution

Basically my solution provides a function named compose that can be used as follows.

Basic Usage Example

auto lambda_1 = [](bool flag) -> int { return flag ? 7 : 0; };
auto lambda_2 = [](int value) -> std::string { return std::to_string(value); };
auto lambda_3 = [](const std::string& string) -> std::string { return string + string; };

auto composition = compose(lambda_1, lambda_2, lambda_3);

std::optional<std::string> result = composition(true);
assert(result = “77”)

Generic Lambda Example

auto generic_lambda = DisablingOptionalFn{[](auto arg) { return arg; }};

auto composition = compose(generic_lambda, generic_lambda);

auto result1 = composition(true);
assert(result1 == true);

auto result2 = composition(“string”);
assert(result2 == "string");

Chain Breaking Example

auto breaking_lambda = [](bool) -> std::optional<int> { return std::nullopt; };
auto subsequent_lambda = [](int) -> int { return 0; };

auto composition = compose(breaking_lambda, subsequent_lambda, subsequent_lambda);

std::optional<int> result = composition(true);
assert(!result.has_value());

The Implementation

// A type trait that checks if a given type is std::optional.
template <typename>
struct IsOptional : std::false_type {};
template <typename T>
struct IsOptional<std::optional<T>> : std::true_type {};

// A type trait that wraps a given type in std::optional if it isn't already.
template <typename T>
struct EnsureOptional {
  using Type = std::optional<T>;
};
template <typename U>
struct EnsureOptional<std::optional<U>> {
  using Type = std::optional<U>;
};

// A function that takes any value and ensures it is wrapped in std::optional.
template <typename TArg>
auto ensure_optional(TArg&& arg) {
  using OptionalType = typename EnsureOptional<std::decay_t<TArg>>::Type;
  return OptionalType{std::forward<TArg>(arg)};
}

// A type trait providing the return type of TFn including compile-time checks
template <typename TFn, typename TArg>
struct InvokeResult {
  static_assert(std::is_invocable_v<TFn, TArg>, "Callable TFn does not support arguments of type TArg");
  static_assert(not std::is_invocable_v<TFn, std::optional<TArg>>,
                "Callable TFn may not support an argument of type std::optional");
  using Type = typename std::invoke_result<TFn, TArg>::type;
  static_assert(not std::is_void_v<Type>, "Result of TFn may not be void");
};

// Helper template function that transforms a given function (Fn) into another one (FnSkippable). FnSkippable expectes
// Fn's argument wrapped in a std::optional. Moreover, FnSkippable returns Fn's result wrapped into std::optional,
// unless it is not already. When calling FnSkippable, Fn is just executed unless FnSkippable is called with a
// std::nullopt. In this case, Fn is not executed (skipped) and FnSkippable simply returns a std::nullopt.
template <typename TFn>
auto make_skippable(TFn&& fn) {
  return [fn = std::forward<TFn>(fn)](auto&& optional_arg) mutable {
    using OptionalArg = std::decay_t<decltype(optional_arg)>;
    using ValueArg = typename OptionalArg::value_type;
    using FnResult = typename InvokeResult<TFn, ValueArg>::Type;
    using OptionalFnResult = typename EnsureOptional<FnResult>::Type;

    if (optional_arg.has_value()) {
      auto&& unwrapped_value = std::forward<OptionalArg>(optional_arg).value();
      return OptionalFnResult{fn(std::forward<decltype(unwrapped_value)>(unwrapped_value))};
    }

    // skip fn
    return OptionalFnResult{std::nullopt};
  };
}

// Helper function template overload that terminates the recursion. The given callable essentially represents the
// composition. To make sure that the composition is always called with an optional argument (as expected by the
// skippable callables) ensure_optional is applied to the argument.
template <typename TFn>
auto compose_skippables(TFn&& fn) {
  return [fn = std::forward<TFn>(fn)](auto&& arg) mutable {
    return fn(ensure_optional(std::forward<decltype(arg)>(arg)));
  };
}

// Helper function template overload that creates a composition out of an arbitrary number of given callables. It
// composes the first two callables into a new lambda that, when called, executes the first callable and passes its
// result to the second callable. The function then recursively composes this combined lambda with the rest of the
// provided callables.
template <typename TFn1, typename TFn2, typename... TFnOthers>
auto compose_skippables(TFn1&& fn_1, TFn2&& fn_2, TFnOthers&&... fn_others) {
  auto chained_fn = [fn_1 = std::forward<TFn1>(fn_1), fn_2 = std::forward<TFn2>(fn_2)](auto&& arg) mutable {
    return fn_2(fn_1(std::forward<decltype(arg)>(arg)));
  };
  return compose_skippables(std::move(chained_fn), std::forward<TFnOthers>(fn_others)...);
}

// Function template that creates a composition out of the given callables. It first makes each callable "skippable" and
// then composes them into a single callable chain using compose_skippables.
template <typename... TFns>
auto compose(TFns&&... fns) {
  return compose_skippables(make_skippable(std::forward<TFns>(fns))...);
}

// Helper template to remove support for optional arguments for TFn
template <typename TFn>
struct DisablingOptionalArgumentFn {
  explicit DisablingOptionalArgumentFn(TFn&& fn) : m_fn{std::forward<TFn>(fn)} {}

  template <typename TArg, typename = std::enable_if_t<not IsOptional<TArg>::value>>
  auto operator()(TArg&& arg) {
    return m_fn(std::forward<TArg>(arg));
  }

private:
  TFn m_fn;
};

Additional Remarks

• Decision for std::optional: To enable callables to not return a result std::optional was choosen, even though std::expect being the preferable option, which is not available in c++17.
• No Monadic Operations: It's recognized that c++23 introduced monadic operations for std::optional and std::except, enabling the construction of pipelines through their application. However, it's important to note that these do not offer the straightforward composition functionality desired in this context.
• No Support for Callables Accepting std::optional: As pointed out in the previous review by indi, supporting callables that accept a std::optional as argument raises the question how to make them skippable in a reasonable way (see his question “What should I get in that last quadrant?”). Ease of use is a core goal here, so to keep it simple I decided to even not raise this question and therefore dropped support for callables accepting a std::optional. A static_assert was added accordingly. To address this, the DisablingOptionalArgumentFn helper template has been introduced, enabling users to adapt generic lambdas for use with this implementation, thereby raising awareness of this design choice. See Generic Lambda Example above.

How do you feel about this implementation? I'm open to any feedback you might have :-)

Please find the implementation including tests at godbolt.

Answer 1

Issues:

1. There's a technical issue in the implementation is the design of the type trait InvokeResult. The problem lies in the usage of static_assert. While it seems like a nice feature, as it provides feedback directly, it also comes with a hidden problem that may cause problems for users of the library.

   The issue is that, as implementation stands, users cannot inspect the type trait. Say, if they try to test whether InvokeResult<A,B> is valid for some types A and B, then do some different operations based on whether or not it is valid. Say the user writes a generic code and has several compile-time options on what to do with the input types, and in one of them, he checks if they can be composed. Unfortunately, by relying on static_assert your type trait will raise a compiler error upon inspection, so the class is not inspectable. Which is not good for C++17 and would be much worse for C++20 and later, where such inspections are far more common due to the presence of concepts. It might not be that relevant for the current scenario, but it is a general design issue.

2. The function make_skippable(TFn&& fn) feels like a fast-fix implementation. The problem is with the output callable [...](auto&& optional_arg) as it technically accepts any input. I'd recommend writing a dedicated class that restricts input as much as possible according to the input callable type TFn. Normally, you'd want to write the restriction on the arguments in the function declaration rather than inside the function. Failing to meet such requirement is the reason why std::is_copy_assignable_v<std::vector<std::unique_ptr<int> > > returns true despite it being clearly false - the is_copy_assignable_v and other type traits and concepts perform shallow checks (they check if the function is properly declared, they don't check if it compiles successfully as that would be perceived as an implementation bug).

3. There's a minor design inefficiency. I advise writing more general-purpose templates and utilizing them rather than writing many overly specialized ones. The compose_skippables template is unnecessary. You don't need a dedicated function for that. Just write a simple general compose and wrap the result in a function that converts input into the optional and calls the composed function.

4. If you don't want users to use certain functions/classes, let them know by writing them inside the namespace "detail". It feels to me that some of the written functions weren't intended for end-users. There's a lot less scrutiny over functions not intended for end-users.

Design Choices:

1. "No Support for Callables Accepting std::optional" I believe not accepting functions with an optional input type is not a good design choice. I'd solve it by making such functions inherently unskippable. Otherwise, you'd need to either write a specialized class used only for the implementation whose purpose is to indicate whether to skip the next call or use two layered std::optional; neither option is easy or convenient to use.

2. I think you overuse lambda functions. They are convenient callables but limited in functionality. Consider writing a dedicated class when that offers an advantage. For instance, your output and intermediate callables technically accept any input type, which is not ideal as it may lead to bugs and confusion.

   If you had written a custom class, you could've specified what input parameter it accepts via a declaration using input_type = ...; which permits inspection (which is impossible in lambdas as it is difficult to inspect such properties), and then you can propagate it to all intermediate callable types what is input. Ultimately, the composed functions will have clear and type-safe input/output types. (The only problem is needing to figure out the initial input type, which we might require the user to declare).

3. I believe the composition order is not ideal. You have compose(A, B, C)(x) to be equal to C(B(A(x))) but I believe it should be A(B(C(x))).

Ideas To Improve Functionality:

1. As it is now, all functions accept a single argument. What about accepting several arguments? It might not be a big difference but having a tuple of a bunch of types as input is never nice. But how do you deal with the problem that your intermediate callables can return only a single argument? Simple, when it returns a tuple-like object, you check if calling the following function on the tuples' elements is an option and perform it in such case.

Answer 2

I would like to share what I have done with the feedback from ALX23z:

• Issue 2: I introduced a dedicated functor implementing the make_skippable functionality that explicitly states std::optional as parameter. I feel this also improves the readability a lot.
• Issue 3: Yes, aiming for proper modularity is always a great advice. I extracted the raw composing in its own details header. Later, I was able to clean it up even more.
• Issue 4: I introduced a details namespace which contains everything that is not intended to be part of the public interface. This is also reflected by the directory structure.
• Design Choices 1: After second thoughts I decided to discard the static_assert ensuring that a callable does not support std::optional argument. Still, I don't think it makes sense to forward a std::nullopt but enforcing it makes usage of generic callables way too cumbersome.
• Design Choices 2: Yes, more functors. I introduced many dedicated ones in the meanwhile, like here or here.
• Design Choices 3: Indeed, in functional programming context compose means right to left. Since I find the reverse order more readable, I now use the term pipe instead.
• Ideas To Improve Functionality 1: I introduced another layer that can unpack std::tuple. It works like a charm.

In conclusion, I am happy about the progress. Thanks again ALX23z for providing the helpful feedback!