Measure a given method's execution time - follow-up

Question

A long time ago, I've written a method that would allow me to measure the amount of time a method takes to run. I've decided to tweak this method and make use of TPL since measuring a method may take some time.

public static class ComparableUtils
{
    public static T Min<T>(this T t1, T t2)where T:IComparable
    {
        if (t1.CompareTo(t2) <= 0)
        {
            return t1;
        }
        return t2;
    }

    public static T Max<T>(this T t1, T t2) where T : IComparable
    {
        if (t1.CompareTo(t2) >= 0)
        {
            return t1;
        }
        return t2;
    }
}
public static class TestUtils
{
    private static readonly Stopwatch Watch = new Stopwatch();

    //This determines a minimum amount of iterations needed to perform to get a significative time measurment
    private static Task<int> GetIterations(Action action)
    {
        return Task.Run(() =>
        {
            const int minimumTime = 20;
            int iterations = 0;
            Watch.Start();
            while (Watch.Elapsed.Milliseconds < minimumTime)
            {
                action();
                ++iterations;
            }
            return iterations;
        });
    }

    private static TimeSpan MeasureTime(Action action, int iterations)
    {
        Watch.Restart();
        while (iterations-- > 0)
        {
            action();
        }
        Watch.Stop();
        return Watch.Elapsed;
    }

    public static Task<Time> Measure(Action action, int? iterations = null, int times = 1)
    {
        return Task.FromResult(iterations)
            .ContinueWith(async t => t.Result.GetValueOrDefault(await GetIterations(action)))
            .Unwrap()
            .ContinueWith(t =>
            {
                var iters = t.Result;
                TimeSpan timeTaken = TimeSpan.MaxValue;
                for (int i = 0; i < times; ++i)
                {
                    timeTaken = MeasureTime(action, iters).Min(timeTaken);
                }
                return new Time(timeTaken.Ticks*100/iters);
            });
    }
}

public class Time
{
    private readonly long _nanoSeconds;
    public Time(long nanoSeconds)
    {
        _nanoSeconds = nanoSeconds;
    }

    public override string ToString()
    {
        if (_nanoSeconds < 9999)
        {
            return _nanoSeconds.ToString()+"ns";
        }
        return new TimeSpan(_nanoSeconds / 100).ToString("g");
    }
}

Is this a well conducted measurement algorithm? Is there something that is amiss?

Answer 1

while (Watch.Elapsed.Milliseconds < minimumTime)

It probably won't affect this code, but Milliseconds doesn't do what you probably think it does. I think you want TotalMilliseconds.

---

public static Task<Time> Measure(Action action, int? iterations = null, int times = 1)
{
    return Task.FromResult(iterations)
        .ContinueWith(async t => t.Result.GetValueOrDefault(await GetIterations(action)))
        .Unwrap()
        .ContinueWith(t =>
        {
            var iters = t.Result;
            TimeSpan timeTaken = TimeSpan.MaxValue;
            for (int i = 0; i < times; ++i)
            {
                timeTaken = MeasureTime(action, iters).Min(timeTaken);
            }
            return new Time(timeTaken.Ticks*100/iters);
        });
}

1. Generally speaking, if you can use await, you don't need ContinueWith().
2. What's the point of using Task.FromResult().ContinueWith()? Task.Run() would make much more sense here.
3. There is actually no reason to use Task and async-await here. You should use that only for truly asynchronous IO operations. If the caller wants to run your long-running CPU-bound code on a separate thread, they can easily do it themselves.
4. For performance-related code, it feels weird that you're calling GetIterations() even when it's not needed.

Answer 2

I guess you prefer Min() over an Average() method, because you sometimes get higher than normal values from measuring the elapsed time. This can have many causes and to prevent most of them you should set ProcessorAffinity and PriorityClass of the current process and also the Priority property of the current thread.

This in addition with a warmup phase will return results which are more precise.

See also this Note of the Stopwatch documentation

On a multiprocessor computer, it does not matter which processor the thread runs on. However, because of bugs in the BIOS or the Hardware Abstraction Layer (HAL), you can get different timing results on different processors. To specify processor affinity for a thread, use the ProcessThread.ProcessorAffinity method.

And a good article here: http://www.codeproject.com/Articles/61964/Performance-Tests-Precise-Run-Time-Measurements-wi

The most important thing is to prevent switching between CPU cores or processors. Switching dismisses the cache, etc. and has a huge performance impact on the test. This can be done by setting the ProcessorAffinity mask of the process

To get the CPU core more exclusively, we must prevent that other threads can use this CPU core. We set our process and thread priority to achieve this

---

You should change the condition of the if statement inside the ToString() method of the Time class to also show a value of 9999 as nanoseconds like:

public override string ToString()
{
    if (_nanoSeconds < 10000)
    {
        return _nanoSeconds.ToString()+"ns";
    }
    return new TimeSpan(_nanoSeconds / 100).ToString("g");
}

---

Your IDE will have a keyboard shortcut for formatting the code, which you should use to change these lines

public static T Min<T>(this T t1, T t2)where T:IComparable

return new Time(timeTaken.Ticks*100/iters);

return _nanoSeconds.ToString()+"ns";

to

public static T Min<T>(this T t1, T t2) where T : IComparable

return new Time(timeTaken.Ticks * 100 / iters);

return _nanoSeconds.ToString() + "ns";

which is more readable.