4
\$\begingroup\$

I have 10 large (34 million cell) 2D gridded vectors storing doubles. When written they are over 200 MB in size. I use an ofstream object to write them to a text file (.csv format), one element at a time, using two for loops (one for my rows, one for my columns). They take way to long to write. Is there a faster way to write from these vectors?

(I am using Visual Studio 2012.)

// Resizing of vectors
flowDirGrid.resize(rows, std::vector<double>(cols, NODATA));

// Do some processing
....

// Outputting processed data
ofstream outfile3("FlowDir.dat");
if(!outfile3.good())
    return;

for(int i=0; i<rows; i++)        
    {
        for (int j=0; j<cols; j++)
        {
            if(elevation[i][j]!=NODATA)
                outfile3 << flowDirGrid[i][j]<<" ";
            else
                outfile3 << NODATA<<" ";
        }
        outfile3 << std::endl;
    }

outfile3.close();
\$\endgroup\$
1
\$\begingroup\$

The output-writing code should reside in a function that takes two parameters: the flowDirGrid and a filename.

Don't omit braces on multi-line conditionals. It's a filthy habit, and someday you will contribute to a coding accident like this.

A performance problem is outputting std::endl at the end of every line, which causes a flush. Output '\n' instead.

\$\endgroup\$
0
\$\begingroup\$

Looking at your conditional:

        if(elevation[i][j]!=NODATA)
            outfile3 << flowDirGrid[i][j]<<" ";
        else
            outfile3 << NODATA<<" ";

I rewrite as a ternary

 outfile3 << elevation[i][j] == NODATA ? NODATA : flowDirGrid[i][j] << " ";

The ternary poses importance on the fact that only the value changes. if should be used when also the action changes.

\$\endgroup\$
  • \$\begingroup\$ Will this really speed up the code? What's the typical performer improvement? \$\endgroup\$ – traggatmot Aug 15 '15 at 8:18
  • \$\begingroup\$ @traggatmot This may speed up the code a tiny bit but mainly enhances clarity. \$\endgroup\$ – Caridorc Aug 15 '15 at 8:22
  • \$\begingroup\$ That's funny. I find ternaries more confusing. \$\endgroup\$ – traggatmot Aug 15 '15 at 8:22
  • \$\begingroup\$ @traggatmot The result of a ternary can only be a value. You may find anything (an expression) in an if block. If the action (print) is the same, a ternary is more limited and therefore more easy to understand. \$\endgroup\$ – Caridorc Aug 15 '15 at 8:24
0
\$\begingroup\$

It might be worth considering using std::basic_streambuf as apposed to std::basic_ostream. You'd need to do the double formatting yourself, but this may still be quicker. Here's an example which shows how to do this for a single std::vector<double>, it should be fairly easy to generalise to a 2D version.

// streambuf.cpp

#include <fstream>
#include <iostream>
#include <iterator>
#include <vector>
#include <string>
#include <algorithm>
#include <cstddef>
#include <random>
#include <chrono>

void write_basic_ostream(const std::vector<double>& v, std::ofstream& of)
{
    std::copy(v.cbegin(), v.cend(), std::ostream_iterator<double> {of, " "});
}

void write_basic_streambuf(const std::vector<double>& v, std::ofstream& of)
{
    std::string str {};
    str.reserve(v.size() * (1 + sizeof(double)));

    for (const auto& e : v) {
        str += std::to_string(e);
        str += " ";
    }

    std::copy(str.cbegin(), str.cend() - 1, std::ostreambuf_iterator<char>(of));
}

int main()
{
    std::size_t n {10000000};

    std::default_random_engine generator {};
    std::uniform_real_distribution<double> distribution {0.0, 1.0};

    std::vector<double> v(n);
    std::generate_n(v.begin(), n, [&generator, &distribution] { return distribution(generator); });

    std::ofstream of1 {"/Users/you/test_basic_ostream.txt"};
    std::ofstream of2 {"/Users/you/test_basic_streambuf.txt"};

    auto start = std::chrono::system_clock::now();

    write_basic_ostream(v, of1);

    auto end = std::chrono::system_clock::now();
    auto duration = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start);

    std::cout << "basic_ostream: "  << duration.count() << std::endl;

    start = std::chrono::system_clock::now();

    write_basic_streambuf(v, of2);

    end = std::chrono::system_clock::now();
    duration = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start);

    std::cout << "write_basic_streambuf: "   << duration.count() << std::endl;

    return 0;
}

Some quick benchmarks has write_basic_ostream about 20% slower than write_basic_streambuf on my system (g++ streambuf.cpp -std=c++14 -O3 -o test).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.