44
\$\begingroup\$

I needed a better way to benchmark code, because, well, rewriting the same benchmarking code every time I need it is just...well...unpleasant.

So, here's a class which does just that, it runs an Action over a specific number of rounds, and calculates certain stats on it.

Another nice feature is that it doesn't store the run times as it calculates the stats. So you can literally supply any value for rounds and it should work. (Not tested for rounds values greater than 10,000,000.)

The latest version is on GitHub.

It's pretty simple. Two static methods on a simple class that run the benchmark.

The nice thing about this class is it includes a version for Func<T> as well, which will also verify the output of the function. This means you can benchmark and verify your code at the same time, to make sure that nothing weird happens.

/// <summary>
/// Represents the result of a benchmarking session.
/// </summary>
public class BenchmarkResult
{
    /// <summary>
    /// The total number of rounds ran.
    /// </summary>
    public ulong RoundsRun { get; set; }

    /// <summary>
    /// The average time for all the rounds.
    /// </summary>
    public TimeSpan AverageTime { get; set; }

    /// <summary>
    /// The maximum time taken for a single round.
    /// </summary>
    public TimeSpan MaxTime { get; set; }

    /// <summary>
    /// The minimum time taken for a single round.
    /// </summary>
    public TimeSpan MinTime { get; set; }

    /// <summary>
    /// The variance (standard deviation) of all the rounds.
    /// </summary>
    public TimeSpan Variance { get; set; }

    /// <summary>
    /// The number of rounds that passed testing. (Always equivalent to <see cref="RoundsRun"/> for <see cref="Benchmark(ulong, Action)"/>.)
    /// </summary>
    public ulong RoundsPassed { get; set; }

    /// <summary>
    /// The total amount of time taken for all the benchmarks. (Does not include statistic calculation time, or result verification time.)
    /// </summary>
    /// <remarks>
    /// Depending on the number of rounds and time taken for each, this value may not be entirely representful of the actual result, and may have rounded over. It should be used with caution on long-running methods that are run for long amounts of time, though that likely won't be a problem as that would result in the programmer having to wait for it to run. (It would take around 29,247 years for it to wrap around.)
    /// </remarks>
    public TimeSpan TotalTime { get; set; }

    /// <summary>
    /// Runs a benchmark of a method.
    /// </summary>
    /// <param name="rounds">The number of rounds to run.</param>
    /// <param name="method">The code to run.</param>
    /// <returns>A <see cref="BenchmarkResult"/> representing the result of the session.</returns>
    public static BenchmarkResult Benchmark(ulong rounds, Action method)
    {
        var sw = new Stopwatch();

        double m2 = 0;
        double averageTicks = 0;
        double totalValues = 0;
        long maxTicks = 0;
        long minTicks = 0;
        long totalTime = 0;

        for (ulong i = 0; i < rounds; i++)
        {
            sw.Start();
            method.Invoke();
            sw.Stop();

            if (totalValues == 0)
            {
                maxTicks = sw.ElapsedTicks;
                minTicks = sw.ElapsedTicks;
            }

            totalValues++;

            maxTicks = Math.Max(sw.ElapsedTicks, maxTicks);
            minTicks = Math.Min(sw.ElapsedTicks, minTicks);

            // We need to store `delta` here as the `averageTicks` will change on the next calculation, and we need this previous `delta` for the calculation after it.
            double delta = sw.ElapsedTicks - averageTicks;
            averageTicks = averageTicks + delta / totalValues;
            m2 += delta * (sw.ElapsedTicks - averageTicks);

            totalTime += sw.ElapsedTicks;

            sw.Reset();
        }

        double variance = m2 / (totalValues - 1);

        return new BenchmarkResult
        {
            AverageTime = new TimeSpan(Convert.ToInt64(averageTicks)),
            MaxTime = new TimeSpan(maxTicks),
            MinTime = new TimeSpan(minTicks),
            RoundsPassed = rounds,
            RoundsRun = rounds,
            TotalTime = new TimeSpan(totalTime),
            Variance = new TimeSpan(Convert.ToInt64(variance))
        };
    }

    /// <summary>
    /// Runs a benchmark of a function and returns the results of the session.
    /// </summary>
    /// <typeparam name="T">The type of the output of the function.</typeparam>
    /// <param name="rounds">The number of rounds to run.</param>
    /// <param name="method">The code to run.</param>
    /// <param name="expectedResult">The expected result of the function. This will be compared to the actual result and used for <see cref="BenchmarkResult.RoundsPassed"/>. This uses the default <code>object.Equals(object)</code> method.</param>
    /// <returns>A <see cref="BenchmarkResult"/> representing the result of the session.</returns>
    public static BenchmarkResult Benchmark<T>(ulong rounds, Func<T> method, T expectedResult)
    {
        var sw = new Stopwatch();

        double m2 = 0;
        double averageTicks = 0;
        double totalValues = 0;
        long maxTicks = 0;
        long minTicks = 0;
        long totalTime = 0;
        ulong roundsPassed = 0;

        for (ulong i = 0; i < rounds; i++)
        {
            sw.Start();
            var result = method.Invoke();
            sw.Stop();

            if (expectedResult.Equals(result))
            {
                roundsPassed++;
            }

            if (totalValues == 0)
            {
                maxTicks = sw.ElapsedTicks;
                minTicks = sw.ElapsedTicks;
            }

            totalValues++;

            maxTicks = Math.Max(sw.ElapsedTicks, maxTicks);
            minTicks = Math.Min(sw.ElapsedTicks, minTicks);

            // We need to store `delta` here as the `averageTicks` will change on the next calculation, and we need this previous `delta` for the calculation after it.
            double delta = sw.ElapsedTicks - averageTicks;
            averageTicks = averageTicks + delta / totalValues;
            m2 += delta * (sw.ElapsedTicks - averageTicks);

            totalTime += sw.ElapsedTicks;

            sw.Reset();
        }

        double variance = m2 / (totalValues - 1);

        return new BenchmarkResult
        {
            AverageTime = new TimeSpan(Convert.ToInt64(averageTicks)),
            MaxTime = new TimeSpan(maxTicks),
            MinTime = new TimeSpan(minTicks),
            RoundsPassed = roundsPassed,
            RoundsRun = rounds,
            TotalTime = new TimeSpan(totalTime),
            Variance = new TimeSpan(Convert.ToInt64(variance))
        };
    }
}

It's literally a one-liner:

var result = BenchmarkResult.Benchmark((ulong)1e7, () => Thread.Sleep(0));
result = BenchmarkResult.Benchmark(10000000, () => true, true);

That will benchmark the Thread.Sleep(0) method 10,000,000 times, and the return true method 10,000,000 times while verifying it always returns true.

\$\endgroup\$
  • 4
    \$\begingroup\$ Practically speaking, the issue with benchmarking is not so much computing the variance as reducing it, in particular by detecting and removing outliers caused by effects like: initial jitting, faulting lots of pages from disk to memory, task switches and so on... With languages like the hashish ones you also need to tame the garbage collector. Also, averaging destroys some information that would still be available if you used something like median selection; it mixes everything up to the point where you have no way of knowing how far to trust the data. \$\endgroup\$ – DarthGizka Apr 13 '16 at 8:44
  • 1
    \$\begingroup\$ Have you seen this? BenchmarkDotNet \$\endgroup\$ – RobH Apr 13 '16 at 9:04
  • 2
    \$\begingroup\$ @EBrown: Having a canned solution like the one you put up for review makes it more likely that people will use the results uncritically, without reflecting about the degree of their validity or the appropriateness of the computation scheme. Unreflected, uncritical use of simple results that someone else dishes up seems somewhat endemic in modern culture... I'm certain that you are well aware of the limits of the approach underlying your class; I'm equally certain that a young padawan who sees this nice class by an experienced programmer like you will never even think about fundamentals. \$\endgroup\$ – DarthGizka Apr 13 '16 at 9:08
  • 2
    \$\begingroup\$ @Mast: =8O Nothing malicious in this topic anywhere - my comment was only intended to put things into perspective, to provide a more balanced view... And I put it in a comment instead of a full review because it's such a small thing. Somehow it got blown up way out of proportion by the discussion. \$\endgroup\$ – DarthGizka Apr 13 '16 at 11:38
  • 2
    \$\begingroup\$ Take a look at BenchmarkDotNet. They seem to have thought of everything, and if they haven't I'm sure they'll welcome feedback to drive future improvments. \$\endgroup\$ – redcalx Apr 13 '16 at 13:48
41
\$\begingroup\$

Small issues before I get into the big one:

  • Please make those setters private. The caller of this code has no business changing any of those values.

  • Don't use ulong unless you are interoperating with unmanaged code. long has plenty of range. .NET uses signed quantities even for quantities that are logically always positive. It makes it easier to do things like take the difference of two quantities.


Indeed, benchmarking is hard. I wrote a long series of articles on this for a developer site which unfortunately I think went out of business. I'll have to see if I can find those and repost them.

So today just a quick note.

Suppose the code under test allocates a lot of objects, but not so many that the GC gets invoked. You then run a second test in the same process which allocates a small number of objects, just enough to push the collection pressure up higher, and boom, the GC runs. Who gets charged the cost of that GC? The second test. Who was to blame for most of the cost of that GC? The first test. So you can easily charge GC costs to the wrong thing.

The net effect of this is not just that the cost gets charged to the wrong code, but also that your results may vary widely from run to run, depending on the state of the heap at any particular time.

Now, one might say, OK, we'll just do a forced collection (don't forget to wait for pending finalizers!) after every test, and charge that cost to the code under test. This will certainly remove variability from the test, which is good. But now we are benchmarking code in an unrealistic environment: an environment where the GC is fully collected. Real users running your code will not be running in that world, so now we've made the test less variable by making it less realistic. That variability is part of the user's experience of the code, and you're removing that for your testing purposes.

I don't have a good one-size-fits-all solution. What I have is the knowledge that when I'm designing benchmarks, I have to think about deferred costs like GCs and make a policy for how I'm going to measure them.

\$\endgroup\$
  • \$\begingroup\$ Since the values only need to be set once, I would even be inclined to create a private constructor and make the properties immutable. \$\endgroup\$ – Dan Lyons Apr 13 '16 at 17:14
  • 7
    \$\begingroup\$ @EricLippert It would be interesting to see those articles, I'm quite curious now. \$\endgroup\$ – Der Kommissar Apr 13 '16 at 18:15
  • \$\begingroup\$ With regards to the cost of GC: I think it's worth leaving that interpretation to the dev. If the library cleans up after benchmark but includes things like allocations, gen distribution, etc then that would solve the issue somewhat. You're no longer looking at a pure time-based benchmark but the more info, the better anyway. \$\endgroup\$ – Jeroen Vannevel Apr 13 '16 at 18:41
  • 1
    \$\begingroup\$ @SirPython: OK, that's a complicated question. First off: remember, the first time you run the code you are paying the price of jitting it, so its cost is already going to be different than the cost of any other run. That effect could be much larger than the effect of running the GC. Second, yes in C# you can force a GC, and you can force the current thread to wait until the finalizer thread finishes. \$\endgroup\$ – Eric Lippert Apr 14 '16 at 0:02
  • 3
    \$\begingroup\$ @Ebrown see part one here: web.archive.org/web/20130607115719/http://tech.pro/blog/1293/… \$\endgroup\$ – Heslacher Apr 14 '16 at 5:13
38
\$\begingroup\$

The problem with an approach like that is that you you don't take into account that switching processors/cores is taking some time which will fake the results. This can be avoided by setting the ProcessorAffinity to use only a specific processor/core like so

Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2); // Use only the second core 

To ensure that we get priority on that core we need to set the PriorityClass of the current process to High and the priority of the current thread to Highest like

Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;
Thread.CurrentThread.Priority = ThreadPriority.Highest;  

Another thing what could fake the results is that you aren't using a warm up phase to stabilize the results. That means calling the desired Action/Function<T> until the timing seems to be the same. This is different from computer to computer so you should evaluate how long it takes to do this for yourself. I usually use 5 seconds of warm up to be on the safe side.

Remember that the measurement is only correct if you use run without a debugger attached.

A good read about this (where most of this information is from) is http://www.codeproject.com/Articles/61964/Performance-Tests-Precise-Run-Time-Measurements-wi

\$\endgroup\$
  • \$\begingroup\$ Calling the function once should be enough for the JIT to compile it, which is the reason a warmup phase is needed. Also, see the related question Benchmarking small code samples in C# \$\endgroup\$ – BlueRaja - Danny Pflughoeft Apr 13 '16 at 20:08
  • 1
    \$\begingroup\$ It's not like the real production code is going to run in a significantly different environment (if you're doing your testing right). You're hiding potential issues with thread switching and concurrency. There's not much value in knowing how your code performs in a completely artificial environment that will never be reproducible in practice. Keep your tests as close to reality as possible - you're making a product, not a white paper. Performance measurements are highly contextual, that's one of the main reasons why guessing around performance is extremely unreliable. \$\endgroup\$ – Luaan Apr 15 '16 at 10:20
13
\$\begingroup\$

You've got some pretty poor variable names there. For example, what's m2 supposed to mean?


I don't like the fact, that result generates itself. I would rather have two separate classes: one to run a benchmark and one to represent benchmark result. Also calling some Benchmark.Run method makes much more sense semantically than calling BenchmarkResult.Benchmark.


You should replace two methods with one. Your first method is a special case of your second method, so one way to remove code duplication is to call:

public static BenchmarkResult Benchmark(ulong rounds, Action method)
{
    Benchmark(rounds, () => { method(); return true; }, true);
}

It adds an overhead of returning bool value, but if the method you benchmark does any meaningful work whatsoever, it shouldn't really affect the result given the fact that benchmarks are not that accurate to begin with. But it is probably something, that you might want to... benchmark.

There are definitely other (probably less pretty) ways to remove copy-pasted code though.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.