Adjusting business logic conveniently through JSON and expression trees

Question

I need to evaluate some data. The rules how it should be done are changing frequently (it's an evolving model) so I don't want to rewrite my application each time such a change comes. I'd rather do it quickly via a config file.

In order to make this possible I've designed a system of components that are very similar to C#'s expression trees and LINQ extensions. They can be put together to be used as decision trees or to calculate other results. It provides a couple of standard operations and needs to be extended by buisiness specific components.

---

Core

The Expression type is the core. All other components are built from here. It's an interface and a class that provides the basic functionality.

public interface ISwitchable
{
    [DefaultValue(true)]
    bool Enabled { get; }
}

[UsedImplicitly]
public interface IExpression : ISwitchable
{
    [NotNull]
    string Name { get; }

    [NotNull]
    IExpression Invoke([NotNull] IExpressionContext context);
}

public abstract class Expression : IExpression
{
    protected Expression(string name) => Name = name;

    public virtual string Name { get; }

    public bool Enabled { get; set; } = true;

    public abstract IExpression Invoke(IExpressionContext context);
}

The next level is represented by types that help me to implement standard logical operations or calculations:

public abstract class PredicateExpression : Expression
{
    protected PredicateExpression(string name) : base(name) { }

    public override IExpression Invoke(IExpressionContext context)
    {
        using (context.Scope(this))
        {
            return Constant.Create(Name, Calculate(context));
        }
    }

    protected abstract bool Calculate(IExpressionContext context);
}    

public abstract class AggregateExpression : Expression
{
    private readonly Func<IEnumerable<double>, double> _aggregate;

    protected AggregateExpression(string name, [NotNull] Func<IEnumerable<double>, double> aggregate) : base(name) => _aggregate = aggregate;

    [JsonRequired]
    public IEnumerable<IExpression> Expressions { get; set; }

    public override IExpression Invoke(IExpressionContext context) => Constant.Create(Name, _aggregate(Expressions.InvokeWithValidation(context).Values<double>().ToList()));
}

public abstract class ComparerExpression : Expression
{
    private readonly Func<int, bool> _predicate;

    protected ComparerExpression(string name, [NotNull] Func<int, bool> predicate) : base(name) => _predicate = predicate;

    [JsonRequired]
    public IExpression Left { get; set; }

    [JsonRequired]
    public IExpression Right { get; set; }

    public override IExpression Invoke(IExpressionContext context)
    {
        var result1 = Left.InvokeWithValidation(context);
        var result2 = Right.InvokeWithValidation(context);

        // optimizations

        if (result1 is Constant<double> d1 && result2 is Constant<double> d2) return Constant.Create(Name, _predicate(d1.Value.CompareTo(d2.Value)));
        if (result1 is Constant<int> i1 && result2 is Constant<int> i2) return Constant.Create(Name, _predicate(i1.Value.CompareTo(i2.Value)));

        // fallback to weak comparer
        var x = (result1 as IConstant)?.Value as IComparable ?? throw new InvalidOperationException($"{nameof(Left)} must return an {nameof(IConstant)} expression with an {nameof(IComparable)} value.");
        var y = (result2 as IConstant)?.Value as IComparable ?? throw new InvalidOperationException($"{nameof(Right)} must return an {nameof(IConstant)} expression with an {nameof(IComparable)} value."); ;
        return Constant.Create(Name, _predicate(x.CompareTo(y)));
    }
}

---

Expressions

I use the above base classes to create the actual components with very few lines of code. They mostly use LINQ internally.

public class All : PredicateExpression
{
    public All() : base(nameof(All)) { }

    [JsonRequired]
    public IEnumerable<IExpression> Expressions { get; set; }

    protected override bool Calculate(IExpressionContext context)
    {
        return 
            Expressions
                .Enabled()
                .InvokeWithValidation(context)
                .Values<bool>()
                .All(x => x);
    }
}

public class Any : PredicateExpression
{
    public Any() : base(nameof(Any)) { }

    [JsonRequired]
    public IEnumerable<IExpression> Expressions { get; set; }

    protected override bool Calculate(IExpressionContext context)
    {
        return
            Expressions
                .Enabled()
                .InvokeWithValidation(context)
                .Values<bool>()
                .Any(x => x);

    }
}

public class IIf : Expression
{
    public IIf() : base(nameof(IIf)) { }

    [JsonRequired]
    public IExpression Predicate { get; set; }

    public IExpression True { get; set; }

    public IExpression False { get; set; }

    public override IExpression Invoke(IExpressionContext context)
    {
        using (context.Scope(this))
        {
            var expression =
                (Predicate.InvokeWithValidation(context).Value<bool>() ? True : False)
                    ?? throw new InvalidOperationException($"{nameof(True)} or {nameof(False)} expression is not defined."); ;

            return expression.InvokeWithValidation(context);
        }
    }
}

public class Min : AggregateExpression
{
    public Min()
    : base(nameof(Min), Enumerable.Min)
    { }
}

public class Max : AggregateExpression
{
    public Max()
        : base(nameof(Max), Enumerable.Max)
    { }
}

public class Sum : AggregateExpression
{
    public Sum()
    : base(nameof(Sum), Enumerable.Sum)
    { }
}

public class Equals : PredicateExpression
{
    public Equals() : base(nameof(Equals)) { }

    [DefaultValue(true)]
    public bool IgnoreCase { get; set; } = true;

    public IExpression Left { get; set; }

    public IExpression Right { get; set; }

    protected override bool Calculate(IExpressionContext context)
    {
        var x = Left.InvokeWithValidation(context).ValueOrDefault();
        var y = Right.InvokeWithValidation(context).ValueOrDefault();

        if (x is string str1 && y is string str2 && IgnoreCase)
        {
            return StringComparer.OrdinalIgnoreCase.Equals(str1, str2);
        }

        return x.Equals(y);
    }
}

public class Matches : PredicateExpression
{
    protected Matches() : base(nameof(Matches)) { }

    [DefaultValue(true)]
    public bool IgnoreCase { get; set; } = true;

    public IExpression Expression { get; set; }

    public string Pattern { get; set; }

    protected override bool Calculate(IExpressionContext context)
    {
        var x = Expression.InvokeWithValidation(context).Value<string>();

        return !(x is null) && Regex.IsMatch(x, Pattern, IgnoreCase ? RegexOptions.IgnoreCase : RegexOptions.None);
    }
}

public class GreaterThan : ComparerExpression
{
    public GreaterThan()
        : base(nameof(GreaterThan), x => x > 0)
    { }
}

public class GreaterThanOrEqual : ComparerExpression
{
    public GreaterThanOrEqual()
        : base(nameof(GreaterThanOrEqual), x => x >= 0)
    { }
}

public class LessThan : ComparerExpression
{
    public LessThan()
        : base(nameof(LessThan), x => x < 0)
    { }
}

public class LessThanOrEqual : ComparerExpression
{
    public LessThanOrEqual()
        : base(nameof(LessThanOrEqual), x => x <= 0)
    { }
}

public class Not : PredicateExpression
{
    public Not() : base(nameof(Not)) { }

    public IExpression Expression { get; set; }

    protected override bool Calculate(IExpressionContext context) => !Expression.InvokeWithValidation(context).Value<bool>();
}

---

Constant expression

There is also one very special expression which is the Constant<T>. No expression is allowed to return a null so they all must return either another expression or a Constant<T> expression. A constant is a type that can have a name and must have a Value. It also provides a bunch of helper factory methods to reduce the ammount of typing necessary to create them.

public interface IConstant
{
    string Name { get; }

    object Value { get; }
}

public class Constant<TValue> : Expression, IEquatable<Constant<TValue>>, IConstant
{
    public Constant(string name) : base(name) { }

    [JsonConstructor]
    public Constant(string name, TValue value) : this(name) => Value = value;

    [AutoEqualityProperty]
    [CanBeNull]
    public TValue Value { get; }

    [CanBeNull]
    object IConstant.Value => Value;

    public override IExpression Invoke(IExpressionContext context)
    {
        using (context.Scope(this))
        {
            return this;
        }
    }

    public override string ToString() => $"\"{Name}\" = \"{Value}\"";

    public static implicit operator Constant<TValue>((string name, TValue value) t) => new Constant<TValue>(t.name, t.value);

    public static implicit operator TValue(Constant<TValue> constant) => constant.Value;

    #region IEquatable

    public override int GetHashCode() => AutoEquality<Constant<TValue>>.Comparer.GetHashCode(this);

    public override bool Equals(object obj) => obj is Constant<TValue> constant && Equals(constant);

    public bool Equals(Constant<TValue> other) => AutoEquality<Constant<TValue>>.Comparer.Equals(this, other);

    #endregion
}

public class One : Constant<double>
{
    public One(string name) : base(name, 1.0) { }
}

public class Zero : Constant<double>
{
    public Zero(string name) : base(name, 0.0) { }
}

public class True : Constant<bool>
{
    public True(string name) : base(name, true) { }
}

public class False : Constant<bool>
{
    public False(string name) : base(name, false) { }
}

public class String : Constant<string>
{
    [JsonConstructor]
    public String(string name, string value) : base(name, value) { }
}

/// <summary>
/// This class provides factory methods.
/// </summary>
public class Constant
{
    private static volatile int _counter;

    public static Constant<TValue> Create<TValue>(string name, TValue value) => new Constant<TValue>(name, value);

    public static Constant<TValue> Create<TValue>(TValue value) => new Constant<TValue>($"{typeof(Constant<TValue>).ToPrettyString()}{_counter++}", value);

    public static IList<Constant<TValue>> CreateMany<TValue>(string name, params TValue[] values) => values.Select(value => Create(name, value)).ToList();

    public static IList<Constant<TValue>> CreateMany<TValue>(params TValue[] values) => values.Select(Create).ToList();
}

---

Unit-testing

The Constant expression is also a great help for testing. Here are a couple of exmpales (the actual list is much longer):

    [TestMethod]
    public void All_ReturnsTrueWhenAllTrue() => Assert.That.ExpressionsEqual(true, new All { Expressions = Constant.CreateMany(true, true, true) });

    [TestMethod]
    public void All_ReturnsFalseWhenSomeFalse() => Assert.That.ExpressionsEqual(false, new All { Expressions = Constant.CreateMany(true, false, true) });

    [TestMethod]
    public void All_ReturnsFalseWhenAllFalse() => Assert.That.ExpressionsEqual(false, new All { Expressions = Constant.CreateMany(false, false, false) });

    [TestMethod]
    public void Any_ReturnsTrueWhenSomeTrue() => Assert.That.ExpressionsEqual(true, new Any { Expressions = Constant.CreateMany(false, false, true) });

    [TestMethod]
    public void Any_ReturnsFalseWhenAllFalse() => Assert.That.ExpressionsEqual(false, new Any { Expressions = Constant.CreateMany(false, false, false) });

They use my helper extension to reduce code repetition:

internal static class Helpers
{
    public static void ExpressionsEqual<TValue, TExpression>(this Assert _, TValue expectedValue, TExpression expression, IExpressionContext context = null) where TExpression : IExpression
    {
        context = context ?? new ExpressionContext();
        var expected = Constant.Create(expression.Name, expectedValue);
        var actual = expression.Invoke(context);

        if (!expected.Equals(actual))
        {
            throw new AssertFailedException(CreateAssertFailedMessage(expected, actual));
        }
    }

    private static string CreateAssertFailedMessage(object expected, object actual)
    {
        return
            $"{Environment.NewLine}" +
            $"» Expected:{Environment.NewLine}{expected}{Environment.NewLine}" +
            $"» Actual:{Environment.NewLine}{actual}" +
            $"{Environment.NewLine}";
    }
}

---

Invoking expressions

To run an expression you Invoke it by passing the IExpressionContext

public interface IExpressionContext
{
    [NotNull]
    IDictionary<object, object> Items { get; }        

    [NotNull]
    ExpressionMetadata Metadata { get; }
}

public class ExpressionContext : IExpressionContext
{
    public IDictionary<object, object> Items { get; } = new Dictionary<object, object>();        

    public ExpressionMetadata Metadata { get; } = new ExpressionMetadata();
}

public class ExpressionMetadata
{
    public string DebugView => ExpressionContextScope.Current.ToDebugView();
}

I usually use this context as a base class for a business context adding other properties to it (e.g. CarName)

I borrowed also the idea of Items from ASP.NET Core's HttpContext.Items and the Metadata from EF Core. I use the metadata to create a DebugView and to see where I am while testing the tree:

---

Debug helpers

The ExpressionContextScope is inspired by the logger scope used in ASP.NET Core. Here it maintains the scope of expressions and the extension is used to build a string showing the position in the tree. (This is going to be more complex later and will render more information in to DebugView.)

[DebuggerDisplay("{" + nameof(DebuggerDisplay) + ",nq}")]
public class ExpressionContextScope : IDisposable
{
    // ReSharper disable once InconsistentNaming - This cannot be renamed because it'd confilict with the property that has the same name.
    private static readonly AsyncLocal<ExpressionContextScope> _current = new AsyncLocal<ExpressionContextScope>();

    private ExpressionContextScope(IExpression expression, IExpressionContext context, int depth)
    {
        Expression = expression;
        Context = context;
        Depth = depth;
    }

    private string DebuggerDisplay => this.ToDebuggerDisplayString(builder =>
    {
        builder.Property(x => x.Depth);
    });

    public ExpressionContextScope Parent { get; private set; }

    public static ExpressionContextScope Current
    {
        get => _current.Value;
        private set => _current.Value = value;
    }

    public IExpression Expression { get; }

    public IExpressionContext Context { get; }

    public int Depth { get; }

    public static ExpressionContextScope Push(IExpression expression, IExpressionContext context)
    {
        var scope = Current = new ExpressionContextScope(expression, context, Current?.Depth + 1 ?? 0)
        {
            Parent = Current
        };
        return scope;
    }

    public void Dispose() => Current = Current.Parent;
}

public static class ExpressionContextScopeExtensions
{
    private const int IndentWidth = 4;

    public static string ToDebugView(this ExpressionContextScope scope)
    {
        var scopes = new Stack<ExpressionContextScope>(scope.Flatten());

        var debugView = new StringBuilder();

        foreach (var inner in scopes)
        {
            debugView
                .Append(IndentString(inner.Depth))
                .Append(inner.Expression.Name)
                .Append(inner.Expression is IConstant constant ? $": {constant.Value}"  : default)
                .AppendLine();
        }

        return debugView.ToString();
    }

    private static string IndentString(int depth) => new string(' ', IndentWidth * depth);

    public static IEnumerable<ExpressionContextScope> Flatten(this ExpressionContextScope scope)
    {
        var current = scope;
        while (current != null)
        {
            yield return current;
            current = current.Parent;
        }
    }
}

---

Using data from other expressions

There also a couple of cases where expressions like TryGetCarColor are not only used to determine whether a property exists but also should return a value that is used by other expression later.

To make the framework more robust I decorate such expressions with In and/or Out attibutes that specify which values they expect or return. The in/out data is stored inside Items.

[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
public class InAttribute : Attribute, IParameterAttribute
{
    public InAttribute(string name) => Name = name;

    public string Name { get; }

    public bool Required { get; set; } = true;
}

[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
public class OutAttribute : Attribute, IParameterAttribute
{
    public OutAttribute(string name) => Name = name;

    public string Name { get; }
    
    public bool Required { get; set; } = true;
}

The framework validates their existence via

.InvokeWithValidation(context)

that checks whether all required in/out items exist:

    public static IEnumerable<IExpression> InvokeWithValidation(this IEnumerable<IExpression> expressions, IExpressionContext context)
    {
        return
            from expression in expressions
            select
                expression
                    .ValidateInItems(context)
                    .Invoke(context)
                    .ValidateOutItems(context);
    }

This way (if everything is properly decorated) I can be sure that each expression will receive it's data and doesn't need any additional checks.

---

Example

Here's an example of a real-world epxpression tree. (I've anonymized it by only changing the name of the business specific expressions like HasColor etc. the tree by itself is the same.)

As you can see I use it to evaluate a couple of conditions and then based on them perfom a calculation. Business specific expressions such as HasColor or SeatCount are also derived from Expression but they evaluate the business data.

{
    "$t:": "IIf",
    "Predicate": {    
        "$t": "Not",
        "Expression": {
            "$t": "Any",
            "Expressions": [
                {
                    "$t": "All",
                    "Expressions": [
                        {
                            "$t": "HasColor",
                            "Values": [ "Red", "Blue" ]
                        },
                        {
                            "$t": "HasFeature",
                            "Values": [ "PowerSteering" ]
                        }
                    ]
                },
                {
                    "$t": "IIf",
                    "Predicate": {
                        "$t": "HasColor",
                        "Values": [ "Red" ]
                    },
                    "True": {
                        "$t": "Not",
                        "Expression": {
                            "$t": "HasFeature",
                            "Values": [ "PowerBrake" ]
                        }
                    },
                    "False": {
                        "$t": "Constant<double>",
                        "Value": 1
                    }
                }
            ]
        }
    },
    "True": {
        "$t": "Sum",
        "Expressions": [
            {
                "$t": "Color"
            },
            {
                "$t": "SeatCount"
            },
            {
                "$t": "IIf",
                "Predicate":{
                    "$t": "HasFeature",
                    "Values": [ "PowerBrake" ]                
                },
                "True": {
                    "$t": "Constant<double>",
                    "Value": 3,
                },
                "False": null
            }
        ]
    },
    "False": null    
}

This means that in code you'd have:

var result = carValueExpression.Invoke(new CarStockExpressionContext
{
    // ... general car data
    // other data can be pulled from a db by any business expression
}).Value<double>();

---

I find this is very easy to test and to extend because everything can be covered by unit-tests. Knowing that all components work as expected, it's a piece of cake to put them together so that they can do much bigger things.

---

In case you are wondering what those $t are and why the types are not named by their full names, I'm using here my json.net helper for more friendly type handling.

---

What do you think about this framework? Did I forget to implement anything important or could I have done it better?

Answer 1

I think there's a lot of room for improvement here. The 'syntax' could be made less verbose and extending the system could be made easier with a better selection of core expressions, among other things.

Syntax

Using JSON as a serialization format saves a lot of work, but the resulting syntax is, in my opinion, quite terrible. 60+ lines of JSON, littered with repetitive $t's and property names, for just a couple of lines of code. That makes these scripts difficult to write and read, increasing the likelyhood of bugs and the cost of fixing them. For example, let's translate that example to C#:

if (!(HasColor("Red", "Blue") && HasFeature("PowerSteering")) ||
    HasColor("Red") ? !HasFeature("PowerBrake") : 1.0)
{
    if (Color + SeatCount + HasFeature("PowerBrake"))
    {
        return 3.0;
    }
}

It's now much more obvious that the code is broken: it's adding a color, a number and a boolean together and treating it as a boolean. It's also missing else branches so it's not always returning a result, and there's an if-statement that returns either a boolean or a number, depending on its condition. The logic itself is also somewhat complicated - a few descriptively named local variables would be helpful.

Alternatives

I'd suggest taking some inspiration from Lisp, in this case the s-expression format it uses:

(if (not (any (all (HasColor "Red" "Blue")
                   (HasFeature "PowerSteering"))
              (if (HasColor "Red")
                  (not (HasFeature "PowerBrake"))
                  1.0)))
    (sum Color
         SeatCount
         (if (HasFeature "PowerBrake")
             3.0)))

Quite readable if you ask me, and fairly easy to parse as well. Parenthesis denote lists, and the first item in a list is either a function or a special form (a keyword), with the rest of the items being arguments (or keyword-specific parts). You could do something similar with JSON arrays. It'll be less terse, and you may have to get creative to distinguish between identifiers and strings (I've prefixed identifiers with a ' below), but it could be a reasonable trade-off between development time and usability:

["'if", ["'not", ["'any", ["'all", ["'HasColor", "Red", "Blue"],
                                   ["'HasFeature", "PowerSteering"]],
                          ["'if", ["'HasColor", "Red"],
                                  ["'not", ["'HasFeature", "PowerBrake"]],
                                  1.0]]],
        ["'sum", "'Color",
                 "'SeatCount",
                 ["'if", ["'HasFeature", "PowerBrake"],
                         3.0]]]

---

Expressions

If you look at the types in System.Linq.Expressions, you'll notice that all of them represent a language construct. Most of your expression types however represent standard library functions. This means that in C#, you only need to extend the language if you want to introduce a new syntactical construct. But in your language, every function and variable you wish to expose requires an extension.

I would replace all of Min, Max, Sum, Equals, Matches, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, Not and other application-specific expression classes with just two expression types: FunctionCall and Identifier. That enables a more data-driven extension approach:

// Initialize the context with bindings to standard library functions:
context.Items["Min"] = StandardFunctions.Min;

// Which can then be referenced via identifiers:
new FunctionCall(
    function: new Identifier("Min"),
    arguments: new IExpression[] {
        new Constant<double>(4.0),
        new Identifier("height")
    });

Other useful expression types would be MemberAccess and Index, and perhaps a Scope expression that allows you to introduce local variables:

// [scope, [[local-identifier, value-expression], ...], body-expression]:
// [member, object-expression, member-identifier]:
// [index, indexable-expression, index-expression]:
["'scope", [["'minHeight", 12.5],
            ["'maxHeight", 37.5],
            ["'firstCarHeight", ["'member", ["'index", "'cars", 0], "'height"]]],
           ["'all", ["'>=", "'firstCarHeight", "'minHeight"],
                    ["'<=", "'firstCarHeight", "'maxHeight"]]]

---

Usability

Other (mostly usability) issues:

• A lack of documentation. There are almost no comments in the code, and there's no high-level explanation of how the system is meant to be used. How does each expression work? Are all their parts required or are some optional? What types does each expression expect and return? How should the system be extended? How are those In/Out attributes supposed to work?
• A confusing 'entry-point': ExpressionSerializer.Deserialize takes a generic parameter, but the most obvious choice, IExpression, failed (due to a typo in your example, I later found out). I would've expected a more simple signature like IExpression Parse(...) instead.
• Error reporting could be better. A few examples:
  • A "$t:" typo in your example resulted in a serialization exception that did point out the location, but the rest of the message wasn't very helpful (Type is an interface or abstract class and cannot be instantiated.).
  • Trying to sum two incompatible types gives an invalid-expression exception that says Invalid expression type. Expected: Constant`1; Actual: Constant`1.. That doesn't show the actual types or the location of the problem.
  • Evaluating an empty IIf expression resulted in a null-reference exception. I would've expected a parsing failure instead.
• What's with the typo in IIf?
• ISwitchable's Enabled only affects child expressions within Any and All expressions, yet every expression is switchable regarless of their parent. How exactly are expressions disabled in practice, and shouldn't this be taken care of by Any and All instead?
• Why does ExpressionContextScope have a static Current property? That seems brittle.