Growing more and more concerned about the real-world performance, especially the execution time, of my code (not just from the perspective of Complexity theory) I decided to build a performance metrics library to get some numbers about the execution time of code.
I basically want a function which takes another function with its parameters as argument, executes it and measures the execution time.
getPerformance(function, function_arg1, function_arg2, ...)
Here is the code;
// Compiled with MinGW gcc version 5.1.0 (tdm64-1), tested on Windows 10 Edu 17134
#include <algorithm>
#include <chrono>
#include <iostream>
#include <time.h>
#include <utility>
#include <windows.h>
#define currentTime() std::chrono::high_resolution_clock::now()
//#define ERROR_GENERAL 0x1
#define ERROR_NULL_POINTER 0x2
typedef std::chrono::high_resolution_clock::time_point ChronoTime;
// Uses Windows Performance Counters
template<typename F, typename... Args>
double performanceMethod1(F function, Args&&... args);
long long performanceMethod1Time();
// Uses GetSystemTimeAsFileTime
template<typename F, typename... Args>
long long performanceMethod2(F function, Args&&...args);
unsigned long long getTime();
unsigned long long WindowsTickToUnixSeconds(long long windowsTicks);
// Uses C/C++ std lib time()
template<typename F, typename... Args>
double performanceMethod3(F function, Args&&... args);
// Uses C++ Chrono
template<typename F, typename... Args>
double performanceMethod4(F function, Args&&... args);
// Test the performance of these algorithms, more to come
void testAlgoFibonacci(unsigned int upperBound);
//void testAlgoMarkov(struct *pMarkov, char *pStr);
int main(void)
{
std::cout << "Test performance with Fibonacci numbers." << std::endl;
std::cout << "\nExecuting Fibonacci algorithm, retrieving time with QueryPerformanceFrequency." << std::endl;
std::cout << "Algorithm time:" << performanceMethod1(testAlgoFibonacci, 2000000000) << "ms" << std::endl;
std::cout << "\nExecuting Fibonacci algorithm, retrieving time with GetSystemTimeAsFileTime." << std::endl;
std::cout << "Algorithm time:" << performanceMethod2(testAlgoFibonacci, 2000000000) << "ms" << std::endl;
std::cout << "\nExecuting Fibonacci algorithm, retrieving time with C clock function." << std::endl;
std::cout << "Algorithm time:" << performanceMethod3(testAlgoFibonacci, 2000000000) << "ms" << std::endl;
std::cout << "\nExecuting Fibonacci algorithm, retrieving time with C++ chrono." << std::endl;
std::cout << "Algorithm time:" << performanceMethod4(testAlgoFibonacci, 2000000000) << "s" << std::endl;
return 0;
}
template<typename F, typename... Args>
double performanceMethod1(F function, Args&&... args)
{
if(function == NULL)
{
std::cout << "NULL pointer reference for 'function' in performanceMethod1" << std::endl;
return ERROR_NULL_POINTER;
}
long long start = performanceMethod1Time();
function(std::forward<Args>(args)...);
long long end = performanceMethod1Time();
return (end - start);
}
long long performanceMethod1Time()
{
static LARGE_INTEGER largeIntegerX;
static BOOL qpf = QueryPerformanceFrequency(&largeIntegerX);
if (qpf)
{
LARGE_INTEGER largeIntegerY;
QueryPerformanceCounter(&largeIntegerY);
return (1000LL * largeIntegerY.QuadPart) / largeIntegerX.QuadPart;
}
return GetTickCount();
}
template<typename F, typename... Args>
long long performanceMethod2(F function, Args&&...args)
{
if(function == NULL)
{
std::cout << "NULL pointer reference for 'function' in performanceMethod2" << std::endl;
return ERROR_NULL_POINTER;
}
unsigned long long start = getTime();
function(std::forward<Args>(args)...);
unsigned long long end = getTime();
return (end - start);
}
unsigned long long getTime()
{
FILETIME fileTime;
LARGE_INTEGER largeIntegerX;
//There is also GetSystemTimePreciseAsFileTime for UTC sync time stamps
//https://msdn.microsoft.com/en-us/library/windows/desktop/hh706895(v=vs.85).aspx
GetSystemTimeAsFileTime(&fileTime);
largeIntegerX.LowPart = fileTime.dwLowDateTime;
largeIntegerX.HighPart = fileTime.dwHighDateTime;
unsigned long long ret = largeIntegerX.QuadPart;
return WindowsTickToUnixSeconds(ret);
}
unsigned long long WindowsTickToUnixSeconds(long long windowsTicks)
{
windowsTicks = windowsTicks - 116444736000000000LL;
windowsTicks = windowsTicks / 10000;
return windowsTicks;
}
template<typename F, typename... Args>
double performanceMethod3(F function, Args&&... args)
{
if(function == NULL)
{
std::cout << "NULL pointer reference for 'function' in performanceMethod3" << std::endl;
return ERROR_NULL_POINTER;
}
clock_t start, end;
start = clock();
function(std::forward<Args>(args)...);
end = clock();
return (end - start);
}
template<typename F, typename... Args>
double performanceMethod4(F function, Args&&... args)
{
if(function == NULL)
{
std::cout << "NULL pointer reference for 'function' in performanceMethod4" << std::endl;
return ERROR_NULL_POINTER;
}
ChronoTime timeBeforeFunction = currentTime();
function(std::forward<Args>(args)...);
ChronoTime timeAfterFunction = currentTime();
return std::chrono::duration_cast<std::chrono::nanoseconds>(timeAfterFunction - timeBeforeFunction).count();
}
void testAlgoFibonacci(unsigned int upperBound)
{
size_t t1, t2, sum;
t1 = 0;
t2 = 1;
sum = 0;
for (unsigned int i = 1; i <= upperBound; ++i)
{
sum = t1 + t2;
t1 = t2;
t2 = sum;
}
return;
}
Each performanceMethod function implements a different way to get the Wall time of the function to be tested, in this case the Fibonacci algorithm. All four performance methods simply expect the function with its parameters as arguments.
performanceMethod1(testAlgoFibonacci, 2000000000)
testAlgoFibonacci is the code to be measured, 2000000000 its only parameter.
I compiled the source with TDM MinGW 5.1
g++ -Wall -std=c++11 benchmark.cpp -o benchmark.exe
An example output with a low CPU load around 30% looks like this
Test performance with Fibonacci numbers.
Executing Fibonacci algorithm, retrieving time with QueryPerformanceFrequency.
Algorithm time:6014ms
Executing Fibonacci algorithm, retrieving time with GetSystemTimeAsFileTime.
Algorithm time:6263ms
Executing Fibonacci algorithm, retrieving time with C clock function.
Algorithm time:6619ms
Executing Fibonacci algorithm, retrieving time with C++ chrono.
Algorithm time:6.36697e+009s
Mayor concerns are:
Variance between measurements when determining the execution time of a multithreaded function.
Intuitive function names. "performanceMethodX" sounds bad, but "performanceMethodChrono" or "performanceMethodGetSystemTimeAsFileTime" even worse.
Graphical representation by a chart when measuring the code performance of a function over longer periods of time.
Logging of hardware specific's, how necessary is it ?
Should there be a wrapper function e.g.
getPerformance(method="CHRONO", function, **args)
with a default selection (For example QueryPerformanceFrequency as recommended by Microsoft) for performance metrics ?