# Large ASCII file data read

I am moving a project from Python to C++, partly in order to achieve a speed up.

This part of the code reads large .txt data files (well only about 2MB per file, but quite a lot of files), and needs to convert the data to floating point.

This C++ code does is not faster than Python bytecode (.pyc). Regardless, my application requires faster processing. What can you see are the main things I am doing wrong?

Below is a complete standalone representative example that will compile with:

cl.exe turtlereader.cpp (or other compiler I believe)

#include <iostream>
#include <string>
#include <vector>
#include <sstream>
#include <fstream>
#include <algorithm>
#include <iterator>

public:

// pointless constructor for this example

// Turtle read looks for zones in the input file and directs the fileread process.

// records line we are reading on:
int linenumber =  0;
int data_starts_on_line = 0;  // init to 0

// find first zone = line
std::string line;
linenumber += 1;
if ( line.find("ZONE") < line.size() ) {

std::string num_zones = line.substr( line.find("I=")+2 ,
(line.find_first_of(",") -
(line.find("I=")+2))
);

std::stringstream zone_to_int_ss(num_zones);
int total_z_values = 0;
zone_to_int_ss >> total_z_values;

// at this point, we have come across a zone line in the file. And,
// we know how many data lines to read next.
for (size_t i = 0; i < (total_z_values); ++i) {
linenumber += 1;
if (check !=0) {
return check;
}
}
}
}
return 0;
}

private:

// The purpose of this method is to take a line of the file as a string,
// and direct the values to the correct member variable, with checks.
int ReadOneLine(std::string line, int linenumber) {
bool error_found = false;
std::vector<double> vec = LineSplit<double>(line);
if (vec.size() == 9) {
a1_.push_back(vec[0]);
a2_.push_back(vec[1]);
a3_.push_back(vec[2]);
a4_.push_back(vec[3]);
a5_.push_back(vec[3]);
a6_.push_back(vec[4]);
a7_.push_back(vec[5]);
a8_.push_back(vec[6]);
a9_.push_back(vec[5]);

// Do some check on the data here:
if (vec[0] > 10 || vec[0] < 0) {
error_found = true;
}
} else {
std::cout << " * ERROR: Expected 9 data elements" << std::endl;
error_found = true;
}

if (error_found) {
std::cout << " file has possible error on line " << linenumber << std::endl;
return 1;
}
return 0;
}

// this splits a string of values and puts into a vector:
template<typename T>
std::vector<T> LineSplit(const std::string& line) {
std::istringstream is(line);
return std::vector<T>(std::istream_iterator<T>(is), std::istream_iterator<T>());
}

// data storage variables:
std::vector<double> a1_;
std::vector<double> a2_;
std::vector<double> a3_;
std::vector<double> a4_;
std::vector<double> a5_;
std::vector<double> a6_;
std::vector<double> a7_;
std::vector<double> a8_;
std::vector<double> a9_;
};
int main() {
return check;
}


With this sample text file that has to be named "sampleinput.txt" for this example:

# Turtleread input file example
TITLE="DATASET1"
VARIABLES="a" "b" "c" "d" "e" "f" "g" "h" "i"
ZONE I=2, F=POINT, T="DATASET1a"
1.2       -0.3        0.01        0.01        0.0        0.3        0.3        0.04  10
0.1       -0.3        0.01        0.01        0.0        0.3        0.3        0.04  10
ZONE I=5, F=POINT, T="DATASET1b"
1.2       -0.3        0.01        0.01        0.0        0.3        0.3        1.23  100
0.1       -0.3        0.01        0.01        0.0        0.3        0.3        2.04  100
1.2       -0.3        0.01        0.01        0.0        0.3        0.3        2.04  100
0.1       -0.3        0.01        0.01        0.0        0.3        0.3        2.01  100
0.5       -0.3        0.01        0.01        0.0        0.3        0.3        3.01  100

• First thing to do is to profile the code. What you'll probably find is that almost all the time is spent in I/O operations. Reading data off a hard-drive is almost always going to be a bottleneck, regardless of language. – Yuushi May 22 '14 at 7:52
• @Yuushi I have added profiling results. What can you make of them? – windenergy May 22 '14 at 9:17

The first principle of optimization is: "measure don't guess". So the first step is to use a profiler on your platform to measure the most consuming steps in your algorithm. It may depend on compilation options (optimization turned on/off). On my platform (x86-64 Linux/g++ 4.8.1 with -O3), the most consuming operation is:

template<typename T>
std::vector<T> LineSplit(const std::string& line) {
std::istringstream is(line);
return std::vector<T>(std::istream_iterator<T>(is), std::istream_iterator<T>());
}


I would first try to write a specialization of this method for double and parse line manually (using pointer arithmetic and the strtod() function from the STL), then measure and optimize the next bottleneck.

• Thanks for the suggestion. I have access to a profiler now which I will use for help. I am going to try your suggestion and see what comes of it. – windenergy May 22 '14 at 9:18

These variables don't really need to be separated:

std::vector<double> a1_;
std::vector<double> a2_;
std::vector<double> a3_;
std::vector<double> a4_;
std::vector<double> a5_;
std::vector<double> a6_;
std::vector<double> a7_;
std::vector<double> a8_;
std::vector<double> a9_;


Since you have 9 of these, just fill-construct a 2D std::vector:

std::vector<std::vector<double> > data_storage(9, std::vector<double>());


Accessing each one would be a bit different, but this is still more concise.

Also, you may use a typedef to shorten the type name, otherwise you would need to type it out in its entirely each time it's used:

typedef std::vector<std::vector<double> > DataStorage;


Since the vectors are private, the typedef should be private as well.

Recently I also had massive performance problems with loading numbers from text files and with parsing big EDI files, which is how I found out that C++ streams are orders of magnitude slower than plain code using the C library functions, for the actual file I/O as well as for parsing/conversion.

I defined an abstract interface to isolate my code from the underlying file access logic, and made three different implementations. One with an istream (using getline()), one with fgets(), and one that used fread() in conjunction with a buffer and EOL parsing. A fourth implementation using raw OS calls for file I/O (with fine-grained control over things like read-ahead and caching) was planned but turned out to be unnecessary since fread() was already fast enough.

Here are the timings with warm caches:

getline  242074 lines, 7335915 bytes,  0.149 MLines/s,    4.5 MByte/s, 1549.3 ms
getline* 242074 lines, 6851767 bytes,  2.426 MLines/s,   68.7 MByte/s,   95.2 ms
fgets    242074 lines, 7335915 bytes,  8.580 MLines/s,  260.0 MByte/s,   26.9 ms
fread    242074 lines, 7335915 bytes, 35.827 MLines/s, 1085.7 MByte/s,    6.4 ms


getline* refers to an implementation that elides calls to tellg(), which means that file offsets and line endings are not available (as can be seen from the wrong byte count). That makes this implementation useless for my purposes but at least it doesn't look quite as bad as the version that actually works...

The picture is exactly the same when it comes to parsing text and converting numbers to and from text. See my post in C++ most efficient way to convert string to int (faster than atoi), for example. The topic concerns integers but with floats it's exactly the same problem, and the performance ratios are even worse than for the getline() problem.

For your small files getline() might still be sufficient but for parsing and converting the contents of the text lines you're probably better off using the C library (<cstring> and friends) and a couple of simple helpers like this:

char const *skip_white (char const *s)
{
while (*s == ' ' || *s == '\t')
++s;

return s;
}


That way your parser will be even cleaner and easier to understand - and orders of magnitude faster.