Better to use string concatenation using std::string::operator+() or std::string::reserve() and std::string::append()? [closed]

Question

I'm doing some code review and I saw something like the following:

std::ifstream dataFile(filename);
if (dataFile.is_open())
{
    std::deque<std::string> lines;
    std::string line, line_batch;
    while (std::getline(dataFile, line))
    {
        if (!line.empty())
            lines.push_back(line);
    }
    dataFile.close();

    for (size_t i = 0; i < lines.size();)
    {
        if (i < MAX_BATCH_SIZE)
        {
            line_batch += "{" + lines.at(i) + "},";
            i++;
        }

I'm thinking that having many small strings (1000 lines equating to around 3000 strings) that will be concatenated together, that it would be beneficial to calculate the final string size ahead of time, reserve it and then do a concatenation using string::append as this would reduce fragmentation, reallocation and increase speed (the last one I don't think would be that big of a deal).

A read in line would, at a guess, be between 10-1000 characters. Max size of line_batch should be under 20MB. The application can get pretty big during execution. Big enough to force us to move from a 32 to a 64 bit memory scheme. The amount of times that this will happen depends on the user's environment, but if I were to guess, maybe between 0 and 100 times in a session. Could be more.

I'm a little on the fence on this. Am I being too nit picky? Is the added complexity worth it? If it matters, we're compiling with VS2019, though I don't think we're using it as a fully compliant c++17 compiler.

Answer 1

This question is a bit too general. And I believe you are nitpicking but not for the reason you mentioned.

In the code it first reads all 1000+ lines and stores them into a deque. Each line is a potential allocation and deque also allocates linearly in the number of objects (one allocation for each 4k bytes and each strings takes 32 bytes... surely these numbers depend on platform and implementation and it works differently for large objects). Wouldn't that be a much much bigger source of memory fragmentation than just resizing 10 times one poor big string? IIRC string's capacity is in increased via the same policy as vectors, i.e., exponentially - thus it is just a few resizes.

I find the issue of storing the string as separate entities to be a much bigger problem than the big string. To deal with it, one needs to read each line and process it immediately (put it into the batchline and hopefully getline will reuse reserved data). At least I hope its enough. In this case one cannot figure out the final size of the string batch without reading the file twice... and nobody wants that. So you can either increase its size by appending or reserve at the start its maximal size and add elements till it hits the capacity and then forward it. Ofc, there is a small nuance of dealing with extra large strings bigger than the reserved size - but it is never too hard to fix. Just make sure you deal with it.

Disclaimer: I don't fully understand what brings more problems to memory fragmentation - it might be that deleting those strings at the end fixes most problems but if it so then why worry about the batch_string?