3
\$\begingroup\$

I would like feedback on my benchmark script. What's another method for benchmarking that might be better/easier?

Overhead

I've found that subprocess.run has an overhead of about 0.01 seconds by using this test function.

import time
import subprocess

def overhead():
     start = time.time()
     process = subprocess.run("", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, executable="/bin/bash")
     print(time.time() - start)

overhead()

To reduce this I'm running the programs I want to benchmark in batches using a Bash loop like below. The $iterations variable is the batch size and $command is the program I want to benchmark.

loop = """
iterations={} command=\"{}\"
for ((i = 0; i < $iterations; ++i))
do 
    $command
done
"""

How it works

My benchmark program is run like python3 benchmark.py "[executable name] [arguments separated by a space]". Each program must print its own runtime as the only output to stdout. The executable name is expected to have a source file name that's the same but ends in .cpp. Here are test files and the benchmark script.

count_down.cpp

#include "./Clock.hpp"
#include <iostream>

int main(int argc, char ** argv){
    unsigned long iterations = 10000000;
    Clock clock;

    clock.tick();
    unsigned long sum = 0;
    for(unsigned long i = iterations; i > 0 ; --i){
        sum += i % 10 == 0;
    }
    float elapsed = clock.tock();
    std::cout << elapsed << std::endl;
    return sum;
}

count_up.cpp

#include "./Clock.hpp"
#include <iostream>

int main(int argc, char ** argv){
    unsigned long iterations = 10000000;
    Clock clock;

    clock.tick();
    unsigned long sum = 0;
    for(unsigned long i = 0; i < iterations ; ++i){
        sum += i % 10 == 0;
    }
    float elapsed = clock.tock();
    std::cout << elapsed << std::endl;
    return sum;
}

Clock.hpp

#ifndef _CLOCK_HPP
#define _CLOCK_HPP

#include <iostream>
#include <iomanip>
#include <chrono>

class Clock {
    protected:
        std::chrono::steady_clock::time_point start_time;

    public:
        Clock() {
            tick();
        }

    void tick() {
        start_time = std::chrono::steady_clock::now();
    }

    float tock() {
        std::chrono::steady_clock::time_point end_time = std::chrono::steady_clock::now();

        return float(
            std::chrono::duration_cast<std::chrono::microseconds>(
                end_time - start_time
            ).count()
        ) / 1e6f;
    }

    // Print elapsed time with newline
    void ptock() {
        float elapsed = tock();
        std::cout << "Took " << elapsed << " seconds" << std::endl;
    }
};

#endif

benchmark.py

import numpy as np
import subprocess
import sys

import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

class Benchmark(object):
    def __init__(self):
        self.shell = "/bin/bash"
        self.batch_size = 10

        self.programs = self.parse_args()
        self.compile()
        self.make_scripts()
        self.animate()

    def animate(self):
        figure = plt.figure()
        delay_ms = 1
        animation = FuncAnimation(figure, self.run_and_graph, interval=delay_ms, init_func=self.setup_plot)
        plt.show()

    def setup_plot(self):
        one_plot = 111
        axis = plt.subplot(one_plot)
        sides = ["top", "bottom", "left", "right"]
        for side in sides:
            axis.spines[side].set_visible(False)

        x_label = "Programs"
        y_label = "Relative Runtimes"
        plt.xlabel(x_label)
        plt.ylabel(y_label)

    def run_and_graph(self, i):
        unicode = "utf8"
        for program in self.programs:
            script = program["script"]
            process = subprocess.run(script, shell=True, executable=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            return_code = process.returncode
            stderr = process.stderr.decode(unicode).split()
            stdout = process.stdout.decode(unicode).split()
            runtimes = np.loadtxt(stdout)
            program["runtime"] += np.sum(runtimes)

        by_runtime = lambda program: program["runtime"]
        self.programs.sort(key=by_runtime)
        max_time = self.programs[-1]["runtime"]
        min_time = self.programs[0]["runtime"]

        plt.cla()
        for program in self.programs:
            name = program["name"]
            x_label = program["label"]
            command = program["command"]
            legend = "{}\n{}".format(x_label, command)
            runtime = program["runtime"]
            ratio = runtime / max_time
            plt.bar(x_label, ratio, label=legend)

        iterations = i * self.batch_size + self.batch_size
        title = "{} Iterations".format(iterations)
        plt.title(title, loc="left")

        legend_coordinates = 1, 1
        plt.legend(bbox_to_anchor=legend_coordinates, frameon=False)

        min_ratio = min_time / max_time
        y_range = 1 - min_ratio
        scale = 0.1
        buffer = y_range * scale
        y_min = min_ratio - buffer
        y_min = max(0, y_min)
        y_max = 1 + buffer
        plt.ylim(y_min, y_max) 

        plt.tight_layout()

    def make_scripts(self):
        script_template = """
        iterations={} command=\"{}\"
        for ((i = 0; i < $iterations; ++i))
        do 
            $command
        done
        """
        for program in self.programs:
            command = program["command"]
            program["script"] = script_template.format(self.batch_size, command)

    def compile(self):
        for program in self.programs:
            name = program["name"]
            command = "g++ -O3 -march=native -std=c++11 -o {0} {0}.cpp".format(name)
            subprocess.run(command, shell=True, executable=self.shell)

    def parse_args(self):
        arguments = sys.argv[1:]
        current_folder = "./"

        commands = [command.strip(current_folder) for command in arguments]
        names = [command.split()[0] for command in commands]
        commands = [current_folder + command for command in commands]

        programs = [
            {
                "name": name,
                "label": "{} {}".format(name, i + 1),
                "command": command,
                "runtime": 0,
            }
            for i, (name, command) in enumerate(zip(names, commands))
        ]
        return programs

if __name__ == "__main__":
    Benchmark()

Output

The output is a live animation of the cumulative runtime of all programs specified. The legend shows the program name, a number that allows the same program with different arguments to be used, and the command sent to Bash including any arguments given. This graph was made by running python3 benchmark.py "count_up dummy_arg" "count_down dummy_arg dummy arg_2" count_down.

enter image description here

\$\endgroup\$
5
\$\begingroup\$

Use differential measurements to eliminate the contribution of overhead

By starting a bash script that runs the desired command multiple times, you are adding even more overhead (namely, starting the shell, which has to interpret the shell script, which then still has some overheads each time it executes the command). You are getting rid of only parts of the overhead by running many iterations in a batch, and only if the number of iterations is very high.

The easiest way to remove the overhead of starting a subprocess from the measurement is to measure twice, once with the actual process you want to benchmark, and once with a dummy process that doesn't do anything, and then use the difference of the two measurements.

Avoid adding more programming languages

Adding bash scripts doesn't help solve your problem, it only increases the number of programming languages and runtime dependencies, making your benchmark more complex than it needs to be.

If the goal was just to perform benchmarks of C++ code, I would recommend that you just use a C++ benchmark library, such as Google Benchmark. It won't give you fancy live graphs, but if you are just interested in the numbers, it's more than enough. Alternatively:

Split the problem into smaller pieces

Your Python code tries to do multiple things at once:

  1. It compiles the C++ code.
  2. It benchmarks the resulting executables.
  3. It displays a (live) graph of the results.

I would try to split your problem into these parts, and solve them independently. In particular, compiling the code is typically something you would let a build system solve. It can be something as simple as make in this case. Just write a Makefile that compiles the C++ code. You can even add a target to the Makefile that starts the benchmark if you want.

Second, use an existing benchmarking library to do the actual measurements. Google Benchmark for example will take care of running multiple iterations while removing unwanted overhead from the measurement results.

Finally, the Python script can be reduced to just starting the benchmarks and drawing the results, without having to worry about anything else.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.