Repetitive unit tests for a programming language

Question

I've got a recreational regex-based programming language called Retina. Before the next big refactoring of the interpreter, I want to rework how I test the code. The interpreter class works such that it's instantiated once with the program (provided as a list of strings, which usually but not necessarily are all single lines) and can then be run several times on different inputs.

I'd like to make use of this in the new test suite so that I can specify tests by defining a program once and then testing a bunch of input/output pairs on it. Virtually all tests will be written this way, and there will be a lot of them, so I'd like to be able to set this up with as little syntactic overhead as possible which might distract from the programs and I/O pairs. Here's what I'm currently doing. I've got a base class for my test cases:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Retina;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace RetinaTest
{
    [TestClass]
    public class RetinaTestBase
    {
        protected void AssertProgram(List<string> sources, List<(string input, string output)> ioPairs)
        {
            var interpreter = new Interpreter(sources);

            foreach ((var input, var expectedOutput) in ioPairs)
            {
                var actualOutput = new StringWriter();
                interpreter.Execute(input, actualOutput);

                Assert.AreEqual(expectedOutput, actualOutput.ToString());
            }
        }
    }
}

In the derived test classes I can then simply do:

namespace RetinaTest
{
    [TestClass]
    public class ReplaceStageTest : RetinaTestBase
    {
        [TestMethod]
        public void TestBasicReplacement()
        {
            AssertProgram(
                new List<string> { 
                    "a", 
                    "x"
                }, 
                new List<(string, string)> { 
                    ( "abc", "xbc" ),
                    ( "Hello, World!", "Hello, World!" ),
                    ( "banana", "bxnxnx" )
                }
            );
            /* etc */
        }
    }
}

A List of string tuples seemed like the natural type for the I/O pairs, but I find the constructor is almost too much in terms of overhead. I'm tempted to turn them both into arrays and let the compiler infer the type with new[], or even provide the I/O pairs as an even-length variadic argument list. That seems like bad style to me though because I could easily mess up the pairing if I forget a string somewhere. Not using C# very regularly I'm also not too clear on the exact benefits and shortcomings of the various list types (in this case mainly lists and plain arrays).

Is there a way to set this up which is both idiomatic and requires very little visual overhead beyond specifying a program (as a list of strings) and a list of I/O pairs? I'd be happy about any other criticism of the code as well, of course.

Answer 1

You can make your code less verbose and more expressive at the same time. Now let me explain what I mean.

Make creating test cases easier without all those new lists and types like this

AssertProgram(new TestData
{
    Sources = { "a", "x" },
    TestCases =
    {
        { "abc", "xbc" },
        { "Hello, World!", "Hello, World!" },               
    }
});

It's at the same time less verbose because there is only one new for TestData and it's easier to understand what all the strings are.

In order for this to work you'll need three helpers with lists that are already instantiated so that you can just use the collection initializer.

class TestData
{
    public List<string> Sources { get; set; } = new List<string>();
    public TestCaseList TestCases { get; set; } = new TestCaseList();
}

class TestCase
{
    public string Input { get; set; }
    public string Output { get; set; }
}

class TestCaseList : List<TestCase>
{
    public void Add(string input, string output) => Add(new TestCase { Input = input, Output = output });
}

The last class is necessary so that you can add a new Add overload to the list and create your test case easily.

---

protected void AssertProgram(TestData testData)
{
    var interpreter = new Interpreter(testData.Sources);

    foreach (var testCase in testData.TestCases)
    {
        var actualOutput = new StringWriter();
        interpreter.Execute(testCase.Input, actualOutput);

        Assert.AreEqual(expectedOutput, actualOutput.ToString());
    }
}

If we now rewrite the AssertProgram method to work with the TestData class, it'll be much easier to use the properties than simple strings or repeat the tuple names everywhere.

---

BTW: you are not using the Output anywhere... yet?

Answer 2

If you manually specify all the test cases as in your example above, then I don't really see the advantage of passing multiple pairs to your method. Personally I find this version a bit easier to read:

var program = new List<string> { "a", "x" };
AssertProgram(program, input: "abc", output: "xbc");
AssertProgram(program, input: "Hello, World!", output: "Hello, World!");
AssertProgram(program, input: "banana", output: "bxnxnx");

But only a bit, both versions look fine. The only thing I don't like is etc. in the end. You should probably keep your tests as small as possible and create separate test for etc. part(s).

P.s. Also it's worth mentioning, that most unit-testing frameworks have some support for test cases. In NUnit, for example, you can write:

[TestCase("abc", ExpectedResult = "xbc")]
[TestCase("Hello, World!", ExpectedResult = "Hello, World!")]
public string TestBasicReplacement(string input)
{
    ....
}