Benchmarking things in C#

Question

I needed a better way to benchmark code, because, well, rewriting the same benchmarking code every time I need it is just...well...unpleasant.

So, here's a class which does just that, it runs an Action over a specific number of rounds, and calculates certain stats on it.

Another nice feature is that it doesn't store the run times as it calculates the stats. So you can literally supply any value for rounds and it should work. (Not tested for rounds values greater than 10,000,000.)

The latest version is on GitHub.

It's pretty simple. Two static methods on a simple class that run the benchmark.

The nice thing about this class is it includes a version for Func<T> as well, which will also verify the output of the function. This means you can benchmark and verify your code at the same time, to make sure that nothing weird happens.

/// <summary>
/// Represents the result of a benchmarking session.
/// </summary>
public class BenchmarkResult
{
    /// <summary>
    /// The total number of rounds ran.
    /// </summary>
    public ulong RoundsRun { get; set; }

    /// <summary>
    /// The average time for all the rounds.
    /// </summary>
    public TimeSpan AverageTime { get; set; }

    /// <summary>
    /// The maximum time taken for a single round.
    /// </summary>
    public TimeSpan MaxTime { get; set; }

    /// <summary>
    /// The minimum time taken for a single round.
    /// </summary>
    public TimeSpan MinTime { get; set; }

    /// <summary>
    /// The variance (standard deviation) of all the rounds.
    /// </summary>
    public TimeSpan Variance { get; set; }

    /// <summary>
    /// The number of rounds that passed testing. (Always equivalent to <see cref="RoundsRun"/> for <see cref="Benchmark(ulong, Action)"/>.)
    /// </summary>
    public ulong RoundsPassed { get; set; }

    /// <summary>
    /// The total amount of time taken for all the benchmarks. (Does not include statistic calculation time, or result verification time.)
    /// </summary>
    /// <remarks>
    /// Depending on the number of rounds and time taken for each, this value may not be entirely representful of the actual result, and may have rounded over. It should be used with caution on long-running methods that are run for long amounts of time, though that likely won't be a problem as that would result in the programmer having to wait for it to run. (It would take around 29,247 years for it to wrap around.)
    /// </remarks>
    public TimeSpan TotalTime { get; set; }

    /// <summary>
    /// Runs a benchmark of a method.
    /// </summary>
    /// <param name="rounds">The number of rounds to run.</param>
    /// <param name="method">The code to run.</param>
    /// <returns>A <see cref="BenchmarkResult"/> representing the result of the session.</returns>
    public static BenchmarkResult Benchmark(ulong rounds, Action method)
    {
        var sw = new Stopwatch();

        double m2 = 0;
        double averageTicks = 0;
        double totalValues = 0;
        long maxTicks = 0;
        long minTicks = 0;
        long totalTime = 0;

        for (ulong i = 0; i < rounds; i++)
        {
            sw.Start();
            method.Invoke();
            sw.Stop();

            if (totalValues == 0)
            {
                maxTicks = sw.ElapsedTicks;
                minTicks = sw.ElapsedTicks;
            }

            totalValues++;

            maxTicks = Math.Max(sw.ElapsedTicks, maxTicks);
            minTicks = Math.Min(sw.ElapsedTicks, minTicks);

            // We need to store `delta` here as the `averageTicks` will change on the next calculation, and we need this previous `delta` for the calculation after it.
            double delta = sw.ElapsedTicks - averageTicks;
            averageTicks = averageTicks + delta / totalValues;
            m2 += delta * (sw.ElapsedTicks - averageTicks);

            totalTime += sw.ElapsedTicks;

            sw.Reset();
        }

        double variance = m2 / (totalValues - 1);

        return new BenchmarkResult
        {
            AverageTime = new TimeSpan(Convert.ToInt64(averageTicks)),
            MaxTime = new TimeSpan(maxTicks),
            MinTime = new TimeSpan(minTicks),
            RoundsPassed = rounds,
            RoundsRun = rounds,
            TotalTime = new TimeSpan(totalTime),
            Variance = new TimeSpan(Convert.ToInt64(variance))
        };
    }

    /// <summary>
    /// Runs a benchmark of a function and returns the results of the session.
    /// </summary>
    /// <typeparam name="T">The type of the output of the function.</typeparam>
    /// <param name="rounds">The number of rounds to run.</param>
    /// <param name="method">The code to run.</param>
    /// <param name="expectedResult">The expected result of the function. This will be compared to the actual result and used for <see cref="BenchmarkResult.RoundsPassed"/>. This uses the default <code>object.Equals(object)</code> method.</param>
    /// <returns>A <see cref="BenchmarkResult"/> representing the result of the session.</returns>
    public static BenchmarkResult Benchmark<T>(ulong rounds, Func<T> method, T expectedResult)
    {
        var sw = new Stopwatch();

        double m2 = 0;
        double averageTicks = 0;
        double totalValues = 0;
        long maxTicks = 0;
        long minTicks = 0;
        long totalTime = 0;
        ulong roundsPassed = 0;

        for (ulong i = 0; i < rounds; i++)
        {
            sw.Start();
            var result = method.Invoke();
            sw.Stop();

            if (expectedResult.Equals(result))
            {
                roundsPassed++;
            }

            if (totalValues == 0)
            {
                maxTicks = sw.ElapsedTicks;
                minTicks = sw.ElapsedTicks;
            }

            totalValues++;

            maxTicks = Math.Max(sw.ElapsedTicks, maxTicks);
            minTicks = Math.Min(sw.ElapsedTicks, minTicks);

            // We need to store `delta` here as the `averageTicks` will change on the next calculation, and we need this previous `delta` for the calculation after it.
            double delta = sw.ElapsedTicks - averageTicks;
            averageTicks = averageTicks + delta / totalValues;
            m2 += delta * (sw.ElapsedTicks - averageTicks);

            totalTime += sw.ElapsedTicks;

            sw.Reset();
        }

        double variance = m2 / (totalValues - 1);

        return new BenchmarkResult
        {
            AverageTime = new TimeSpan(Convert.ToInt64(averageTicks)),
            MaxTime = new TimeSpan(maxTicks),
            MinTime = new TimeSpan(minTicks),
            RoundsPassed = roundsPassed,
            RoundsRun = rounds,
            TotalTime = new TimeSpan(totalTime),
            Variance = new TimeSpan(Convert.ToInt64(variance))
        };
    }
}

It's literally a one-liner:

var result = BenchmarkResult.Benchmark((ulong)1e7, () => Thread.Sleep(0));
result = BenchmarkResult.Benchmark(10000000, () => true, true);

That will benchmark the Thread.Sleep(0) method 10,000,000 times, and the return true method 10,000,000 times while verifying it always returns true.

Answer 1

Small issues before I get into the big one:

• Please make those setters private. The caller of this code has no business changing any of those values.

• Don't use ulong unless you are interoperating with unmanaged code. long has plenty of range. .NET uses signed quantities even for quantities that are logically always positive. It makes it easier to do things like take the difference of two quantities.

---

Indeed, benchmarking is hard. I wrote a long series of articles on this for a developer site which unfortunately I think went out of business. I'll have to see if I can find those and repost them.

So today just a quick note.

Suppose the code under test allocates a lot of objects, but not so many that the GC gets invoked. You then run a second test in the same process which allocates a small number of objects, just enough to push the collection pressure up higher, and boom, the GC runs. Who gets charged the cost of that GC? The second test. Who was to blame for most of the cost of that GC? The first test. So you can easily charge GC costs to the wrong thing.

The net effect of this is not just that the cost gets charged to the wrong code, but also that your results may vary widely from run to run, depending on the state of the heap at any particular time.

Now, one might say, OK, we'll just do a forced collection (don't forget to wait for pending finalizers!) after every test, and charge that cost to the code under test. This will certainly remove variability from the test, which is good. But now we are benchmarking code in an unrealistic environment: an environment where the GC is fully collected. Real users running your code will not be running in that world, so now we've made the test less variable by making it less realistic. That variability is part of the user's experience of the code, and you're removing that for your testing purposes.

I don't have a good one-size-fits-all solution. What I have is the knowledge that when I'm designing benchmarks, I have to think about deferred costs like GCs and make a policy for how I'm going to measure them.

Answer 2

The problem with an approach like that is that you you don't take into account that switching processors/cores is taking some time which will fake the results. This can be avoided by setting the ProcessorAffinity to use only a specific processor/core like so

Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2); // Use only the second core 

To ensure that we get priority on that core we need to set the PriorityClass of the current process to High and the priority of the current thread to Highest like

Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;
Thread.CurrentThread.Priority = ThreadPriority.Highest;  

Another thing what could fake the results is that you aren't using a warm up phase to stabilize the results. That means calling the desired Action/Function<T> until the timing seems to be the same. This is different from computer to computer so you should evaluate how long it takes to do this for yourself. I usually use 5 seconds of warm up to be on the safe side.

Remember that the measurement is only correct if you use run without a debugger attached.

A good read about this (where most of this information is from) is http://www.codeproject.com/Articles/61964/Performance-Tests-Precise-Run-Time-Measurements-wi

Answer 3

You've got some pretty poor variable names there. For example, what's m2 supposed to mean?

---

I don't like the fact, that result generates itself. I would rather have two separate classes: one to run a benchmark and one to represent benchmark result. Also calling some Benchmark.Run method makes much more sense semantically than calling BenchmarkResult.Benchmark.

---

You should replace two methods with one. Your first method is a special case of your second method, so one way to remove code duplication is to call:

public static BenchmarkResult Benchmark(ulong rounds, Action method)
{
    Benchmark(rounds, () => { method(); return true; }, true);
}

It adds an overhead of returning bool value, but if the method you benchmark does any meaningful work whatsoever, it shouldn't really affect the result given the fact that benchmarks are not that accurate to begin with. But it is probably something, that you might want to... benchmark.

There are definitely other (probably less pretty) ways to remove copy-pasted code though.