3
\$\begingroup\$

I wrote my first benchmark using Google Benchmark to check what is faster between the use of std::string::length and strlen to compute the length of a std::string.

I would like to know what I can improve in my code to use it as a base for future benchmarks. My main concerns are about:

  • the creation of the std::string at the beginning of my functions: is it the right place to do it? Should I use fixtures? Or create a global std::vector<std::string> that contains every strings I'll use for my benchmark?
  • the use of benchmark::DoNotOptimize and benchmark::ClobberMemory: I'm not sure I use these functions as I should. The idea is the following: I want to be sure the std::string s is created and stored into memory before entering in my loop and I want to be sure the functions std::string::length and strlen are really called and not optimized. Is it the good way to do it?
  • the use of DenseRange to create a benchmark with multiple length of string. At first, I wanted to pass an std::string directly to my function but, apparently, only int can be returned by state.range. Is there another way to pass a string to my functions?

Here is my code:

#include <benchmark/benchmark.h>

#include <string>
#include <cstring>

static void BM_InneficientBasicStringLength(benchmark::State& state)
{
    std::string s;

    for (int i = 0; i < state.range(0); i++)
        s += 'a';

    benchmark::DoNotOptimize(s.data());
    benchmark::ClobberMemory();
    size_t size;

    for (auto _ : state)
    {
        benchmark::DoNotOptimize(size = strlen(s.c_str()));
        benchmark::ClobberMemory();
    }
}

BENCHMARK(BM_InneficientBasicStringLength)->DenseRange(0, 100, 10);

static void BM_InneficientBasicStringLength_Fix(benchmark::State& state)
{
    std::string s;

    for (int i = 0; i < state.range(0); i++)
        s += 'a';

    benchmark::DoNotOptimize(s.data());
    benchmark::ClobberMemory();
    size_t size;

    for (auto _ : state)
    {
        benchmark::DoNotOptimize(size = s.length());
        benchmark::ClobberMemory();
    }
}
BENCHMARK(BM_InneficientBasicStringLength_Fix)->DenseRange(0, 100, 10);

BENCHMARK_MAIN();

It compiles with: g++ -o example example.cpp -lbenchmark -lpthread.


EDIT: For your information, the results are better with std::string::length which have a constant complexity while strlen have a linear complexity:

> ./example 
2020-08-14T14:48:17+02:00
Running ./example
Run on (12 X 3900 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 1024 KiB (x6)
  L3 Unified 8448 KiB (x1)
Load Average: 5.18, 6.05, 6.32
----------------------------------------------------------------------------------
Benchmark                                        Time             CPU   Iterations
----------------------------------------------------------------------------------
BM_InneficientBasicStringLength/0             4.79 ns         4.79 ns    150060775
BM_InneficientBasicStringLength/10            4.66 ns         4.66 ns    151148682
BM_InneficientBasicStringLength/20            5.39 ns         5.39 ns    136288403
BM_InneficientBasicStringLength/30            5.36 ns         5.35 ns    135353692
BM_InneficientBasicStringLength/40            5.19 ns         5.18 ns    137059957
BM_InneficientBasicStringLength/50            5.22 ns         5.22 ns    134737400
BM_InneficientBasicStringLength/60            5.32 ns         5.32 ns    134098029
BM_InneficientBasicStringLength/70            6.65 ns         6.65 ns    107061965
BM_InneficientBasicStringLength/80            6.85 ns         6.85 ns    109856173
BM_InneficientBasicStringLength/90            6.78 ns         6.78 ns    106762788
BM_InneficientBasicStringLength/100           6.66 ns         6.66 ns     98081332
BM_InneficientBasicStringLength_Fix/0         3.01 ns         3.01 ns    231875548
BM_InneficientBasicStringLength_Fix/10        3.06 ns         3.06 ns    221447369
BM_InneficientBasicStringLength_Fix/20        2.96 ns         2.96 ns    237989703
BM_InneficientBasicStringLength_Fix/30        2.98 ns         2.98 ns    234755616
BM_InneficientBasicStringLength_Fix/40        2.99 ns         2.98 ns    231015140
BM_InneficientBasicStringLength_Fix/50        2.94 ns         2.94 ns    223906062
BM_InneficientBasicStringLength_Fix/60        3.04 ns         3.04 ns    225199556
BM_InneficientBasicStringLength_Fix/70        3.07 ns         3.07 ns    230208201
BM_InneficientBasicStringLength_Fix/80        3.02 ns         3.02 ns    219373040
BM_InneficientBasicStringLength_Fix/90        3.10 ns         3.10 ns    219921248
BM_InneficientBasicStringLength_Fix/100       3.10 ns         3.10 ns    238076969
\$\endgroup\$
8
  • 2
    \$\begingroup\$ What did you find, could you post your results? I would expect string.length() to be much faster since it may only need to access an internal count that is made while the string is created. \$\endgroup\$
    – pacmaninbw
    Aug 14, 2020 at 13:25
  • 1
    \$\begingroup\$ Are you trying to imitate perl? for (auto _ : state). Don't be obtuse in your code it is supposed to be human readable. \$\endgroup\$ Aug 14, 2020 at 17:08
  • 1
    \$\begingroup\$ @MartinYork Almost all of the examples given on the GitHub of the project follow this pattern: github.com/google/benchmark#usage Whatever, what alternatives could I use? \$\endgroup\$
    – Pierre
    Aug 15, 2020 at 8:11
  • \$\begingroup\$ Benchmarking calculating the string length is not representative of real world code. Compilers will be able to optimize this if they know anything about the string. See this example. I also wouldn't trust any results that only take a few nanoseconds, it's likely that despite Google Benchmark's best efforts to avoid that, these low numbers will include overhead from benchmarking itself. \$\endgroup\$
    – G. Sliepen
    Aug 15, 2020 at 16:09
  • \$\begingroup\$ @G.Sliepen One of my concern is to avoid optimizations of the compiler. That's why I use benchmark::DoNotOptimize and benchmark::ClobberMemory. However, this is one of my questions: am I using this functions correctly in order to have a correct measure of strlen and std::string::length. About the duration of my benchmark, that's all the problem with microbenchmarking. The idea is to take a very little piece of code outside of its context to measure its efficiency. To prevent any issue about the reproducibility, we can see each measures are performed at least 98081332 times. \$\endgroup\$
    – Pierre
    Aug 15, 2020 at 18:54

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.