Full integration test for a Console application

Question

I'm sill experimenting with different design patterns for full integration tests for Console applications (and later also Windows Services) and I wasn't quite happy with the result of the refactoring of my last question. I've changed a few things and this is what I've come up this time.

The main application code that starts an application is now divided into two files Program.Entry.cs and Program.Instance.cs that are partial classes of the Program (otherwise I needed a new name for the other file and this isn't actually necessary). The Entry file contains only the Main method and redirects this call to the Start method that besides the args parameter also takes a few other ones that I'm using for testing. It also contains other static methods necessary for the initialization. The Instance file contains only instance code that runs the application. Having them both split allows me to better separate the two tasks. The new code:

Program.Entry.cs

internal partial class Program
{
    internal static int Main(string[] args)
    {
        return Start(
            args, 
            InitializeLogging, 
            InitializeConfiguration, 
            configuration => InitializeContainer(configuration, Enumerable.Empty<Autofac.Module>()));
    }

    public static int Start(
        string[] args, 
        Action initializeLogging, 
        Func<Configuration> initializeConfiguration, 
        Func<Configuration, IContainer> initializeContainer)
    {
        initializeLogging();

        var mainLogger = LoggerFactory.CreateLogger(nameof(Program)); LogEntry.New().Debug().Message("Logging initialized.").Log(mainLogger);
        var mainLogEntry = LogEntry.New().Stopwatch(sw => sw.Start());

        try
        {
            var configuration = initializeConfiguration(); LogEntry.New().Debug().Message("Configuration initialized.").Log(mainLogger);
            var container = initializeContainer(configuration); LogEntry.New().Debug().Message("IoC initialized.").Log(mainLogger);

            using (var scope = container.BeginLifetimeScope())
            {
                var program = scope.Resolve<Program>();
                LogEntry.New().Info().Message($"Created {Name} v{Version}").Log(mainLogger);
                program.Start(args);
            }

            mainLogEntry.Info().Message("Completed.");
            return 0;
        }
        catch (Exception ex)
        {
            mainLogEntry.Fatal().Message("Crashed.").Exception(ex);
            return 1;
        }
        finally
        {
            mainLogEntry.Log(mainLogger);
            LogEntry.New().Info().Message("Exited.").Log(mainLogger);
        }
    }

    #region Initialization

    internal static void InitializeLogging()
    {
        Reusable.Logging.NLog.Tools.LayoutRenderers.InvariantPropertiesLayoutRenderer.Register();

        Reusable.Logging.Logger.ComputedProperties.Add(new Reusable.Logging.ComputedProperties.AppSetting(name: "Environment", key: $"Gunter.Program.Config.Environment"));
        Reusable.Logging.Logger.ComputedProperties.Add(new Reusable.Logging.ComputedProperties.ElapsedSeconds());

        Reusable.Logging.LoggerFactory.Initialize<Reusable.Logging.Adapters.NLogFactory>();
    }

    internal static Configuration InitializeConfiguration()
    {
        try
        {
            return new Configuration(new AppSettings());
        }
        catch (Exception ex)
        {
            throw new InitializationException("Could not initialize configuration.", ex);
        }
    }

    internal static IContainer InitializeContainer(Configuration configuration, IEnumerable<Autofac.Module> moduleOverrides)
    {
        try
        {
            var builder = new ContainerBuilder();

            builder.RegisterInstance(configuration.Load<Program, Workspace>());

            builder.RegisterModule<SystemModule>();
            builder.RegisterModule<DataModule>();
            builder.RegisterModule<ReportingModule>();
            builder.RegisterModule<HtmlModule>();

            builder
                .RegisterType<TestRunner>()
                .WithParameter(new TypedParameter(typeof(ILogger), LoggerFactory.CreateLogger(nameof(TestRunner))));

            builder
                .RegisterType<Program>()
                .WithParameter(new TypedParameter(typeof(ILogger), LoggerFactory.CreateLogger(nameof(Program))))
                .PropertiesAutowired();

            foreach (var module in moduleOverrides)
            {
                builder.RegisterModule(module);
            }

            return builder.Build();
        }
        catch (Exception ex)
        {
            throw new InitializationException("Could not initialize container.", ex);
        }
    }

    #endregion
}

Program.Instance.cs

internal partial class Program
{
    public static readonly string Name = Assembly.GetAssembly(typeof(Program)).GetName().Name;
    public static readonly string Version = "2.0.0";
    private static readonly string GlobalFileName = "_Global.json";

    private readonly ILogger _logger;
    private readonly IPathResolver _pathResolver;
    private readonly IFileSystem _fileSystem;
    private readonly IVariableBuilder _variableBuilder;
    private readonly AutofacContractResolver _autofacContractResolver;
    private readonly TestRunner _testRunner;

    public Program(
        ILogger logger,
        IPathResolver pathResolver,
        IFileSystem fileSystem,
        IVariableBuilder variableBuilder,
        AutofacContractResolver autofacContractResolver,
        TestRunner testRunner)
    {
        _logger = logger;
        _pathResolver = pathResolver;
        _fileSystem = fileSystem;
        _variableBuilder = variableBuilder;
        _autofacContractResolver = autofacContractResolver;
        _testRunner = testRunner;
    }

    public Workspace Workspace { get; set; }

    public void Start(string[] args)
    {
        var globalFile = LoadGlobalFile();

        var globals = VariableResolver.Empty
            .MergeWith(globalFile.Globals)
            .MergeWith(_variableBuilder.BuildVariables(Workspace));

        var testFiles = LoadTestFiles().ToList();

        LogEntry.New().Debug().Message($"Test files ({testFiles.Count}) loaded.").Log(_logger);
        LogEntry.New().Info().Message($"*** {Name} v{Version} started. ***").Log(_logger);

        _testRunner.RunTestFiles(testFiles, args, globals);
    }

    private GlobalFile LoadGlobalFile()
    {
        var targetsDirectoryName = _pathResolver.ResolveDirectoryPath(Workspace.Targets);
        var fileName = Path.Combine(targetsDirectoryName, GlobalFileName);

        if (!File.Exists(fileName)) { return new GlobalFile(); }

        try
        {
            var globalFileJson = _fileSystem.ReadAllText(fileName);
            var globalFile = JsonConvert.DeserializeObject<GlobalFile>(globalFileJson);

            VariableValidator.ValidateNamesNotReserved(globalFile.Globals, _variableBuilder.Names);

            LogEntry.New().Debug().Message($"{Path.GetFileName(fileName)} loaded.").Log(_logger);

            return globalFile;
        }
        catch (Exception ex)
        {
            throw new InitializationException($"Could not load {Path.GetFileName(fileName)}.", ex);
        }
    }

    [NotNull, ItemNotNull]
    private IEnumerable<TestFile> LoadTestFiles()
    {
        LogEntry.New().Debug().Message("Initializing tests...").Log(_logger);

        return
            GetTestFileNames()
                .Select(LoadTest)
                .Where(Conditional.IsNotNull);
    }

    [NotNull, ItemNotNull]
    private IEnumerable<string> GetTestFileNames()
    {
        var targetsDirectoryName = _pathResolver.ResolveDirectoryPath(Workspace.Targets);

        return
            from fullName in _fileSystem.GetFiles(targetsDirectoryName, "*.json")
            where !Path.GetFileName(fullName).StartsWith("_", StringComparison.OrdinalIgnoreCase)
            select fullName;
    }

    [CanBeNull]
    private TestFile LoadTest(string fileName)
    {
        var logEntry = LogEntry.New().Info();
        try
        {
            var json = _fileSystem.ReadAllText(fileName);
            var testFile = JsonConvert.DeserializeObject<TestFile>(json, new JsonSerializerSettings
            {
                ContractResolver = _autofacContractResolver,
                DefaultValueHandling = DefaultValueHandling.Populate,
                TypeNameHandling = TypeNameHandling.Auto,
            });
            testFile.FullName = fileName;

            VariableValidator.ValidateNamesNotReserved(testFile.Locals, _variableBuilder.Names);

            logEntry.Message($"Test initialized: {fileName}");
            return testFile;
        }
        catch (Exception ex)
        {
            logEntry.Error().Message($"Could not initialize test: {fileName}").Exception(ex);
            return null;
        }
        finally
        {
            logEntry.Log(_logger);
        }
    }
}

---

Testing

Now I am able to override the default modules with my test versions to actually use other files that I store in a test project as Embeded Resources. This way I can simulate various scenarios with non-existing or invalid files, not working database connections etc and verfiy that the application behaves as I want it to. This is, it should exit in some situations and survive in others or log something etc.

Here's one of the first tests I wrote for it and a new TestFileSystem that I use to fake files. Additionaly I can now test it with different command line arguments.

[TestClass]
public class ProgramTest
{
    [TestMethod]
    public void Start_NoArguments_RunsAllTests()
    {
        var exitCode = Program.Start(
            new string[0],
            Program.InitializeLogging,
            Program.InitializeConfiguration,
            configuration => Program.InitializeContainer(configuration, new[]
            {
                new TestModule
                {
                    FileSystem = new TestFileSystem
                    {
                        Files = { @"t:\tests\assets\targets\single-test.json" }
                    }
                }
            }));

        Assert.AreEqual(0, exitCode);
        Assert.AreEqual(1, TestAlert.GetReports("single-test.json").Count);
    }
}

internal class TestModule : Autofac.Module
{
    public TestFileSystem FileSystem { get; set; }

    protected override void Load(ContainerBuilder builder)
    {
        builder
            .RegisterType<TestAlert>();

        builder
            .RegisterType<TestPathResolver>()
            .As<IPathResolver>();

        builder
            .RegisterInstance(FileSystem)
            .As<IFileSystem>();
    }
}

internal class TestFileSystem : IFileSystem
{
    public List<string> Files { get; } = new List<string>();

    public string ReadAllText(string fileName)
    {
        switch (fileName.ToLower())
        {
            case @"t:\tests\_Global.json": return ResourceReader.ReadEmbeddedResource<ProgramTest>("Resources.assets.targets._Global.json");
            case @"t:\tests\single-test.json": return ResourceReader.ReadEmbeddedResource<ProgramTest>("Resources.assets.targets.single-test.json");
            default: throw new FileNotFoundException($"File \"{fileName}\" does not exist.");
        }
    }

    public string[] GetFiles(string path, string searchPattern)
    {
        return Files.ToArray();
    }
}

---

What do you think of this design? Can it still be improved in any way?

Answer 1

var mainLogger = LoggerFactory.CreateLogger(nameof(Program)); LogEntry.New().Debug().Message("Logging initialized.").Log(mainLogger);

This should really be split into two lines. Not sure what happened, but I'm sure it was a mistake.

var mainLogger = LoggerFactory.CreateLogger(nameof(Program)); 
LogEntry.New().Debug().Message("Logging initialized.").Log(mainLogger);

Same with the very next line.

Maybe it wasn't a mistake. Maybe you did this intentionally because of all the noisy daisy chaining your logger needs. Try extracting a few small helpers to reduce the noise.

private void logDebug(string message, ILogger logger)
{
    LogEntry.New().Debug().Message(message).Log(logger);
}

Otherwise, I find your entry point well structured and readable. I like that you ensure the program returns a non-zero exit code on failure. We windows devs tend to forget these kinds of things, but it's important if you ever program ever gets used as part of some shell script. +1

---

Using a couple of variables here could clean this up.

internal static void InitializeLogging()
{
    Reusable.Logging.NLog.Tools.LayoutRenderers.InvariantPropertiesLayoutRenderer.Register();

    Reusable.Logging.Logger.ComputedProperties.Add(new Reusable.Logging.ComputedProperties.AppSetting(name: "Environment", key: $"Gunter.Program.Config.Environment"));
    Reusable.Logging.Logger.ComputedProperties.Add(new Reusable.Logging.ComputedProperties.ElapsedSeconds());

    Reusable.Logging.LoggerFactory.Initialize<Reusable.Logging.Adapters.NLogFactory>();
}

internal static void InitializeLogging()
{
    Reusable.Logging.NLog.Tools.LayoutRenderers.InvariantPropertiesLayoutRenderer.Register();

    var properties = Reusable.Logging.Logger.ComputedProperties;
    properties.Add(new Reusable.Logging.ComputedProperties.AppSetting(name: "Environment", key: $"Gunter.Program.Config.Environment"));
    properties.Add(new Reusable.Logging.ComputedProperties.ElapsedSeconds());

    Reusable.Logging.LoggerFactory.Initialize<Reusable.Logging.Adapters.NLogFactory>();
}

If I got that wrong and those are namespaces, not static properties, for goodness sake use some imports.

---

One last thought...

If your entry point has become so large and complex that you're worried about testing the code directly, or are tempted to use a partial class, then your domain is likely missing some concepts.

I think it would be useful to ask yourself why you want to test these conditions, and why is it hard to test them? Could you test them by just running the program under different contexts? Could you use a shell script to just run it with different inputs? Why not? Because you're testing embedded resources? Why not pass those files via args instead? Why don't your other existing tests inside of the actual domain logic test those embedded resources efficiently? Have you put a ton of time and effort into a problem that didn't need to be solved out of some dogmatic reasoning that everything must be tested extremely thoroughly? What benefit did it give you? Sometimes simpler is better and we need to know when enough is enough. I'm not saying that you were wrong to do any of this, I'm just asking you to reflect on whether or not you were.