# Reading binary file of doubles

I'm trying to read doubles from a relatively small binary file. This currently reads a 100 KB file in about 6 milliseconds in my system. I would like to reduce that if possible.

void readNParseData(const char* filePath, vector<double> *&data){

ifstream ifs(filePath, ios::in | ios::binary);

// If this is a valid file
if (ifs) {
// Temporary Variables
std::streampos fileSize;
double *fileBuffer;
size_t sizeOfBuffer;

// Check whether the parameter is already full
if (data != 0){
// Reset the output
data->clear();
data = 0;
}

// Get the size of the file
ifs.seekg(0, std::ios::end);
fileSize = ifs.tellg();
ifs.seekg(0, std::ios::beg);

sizeOfBuffer = fileSize / sizeof(double);
fileBuffer = new double[sizeOfBuffer];

// Now convert the double array into vector
data = new vector<double>(fileBuffer, fileBuffer + sizeOfBuffer);

free(fileBuffer);
}
}


As you can see there is a redundant copy of a double * array to a vector. Perhaps reading to the vector directly might speed it up, but I don't know how.

• Depending how your file is produced you could try using memory mapped access rather than the io library. – Harald Scheirich Sep 25 '18 at 13:58

Suggestion 1

This block of code does not seem clean:

    // Check whether the parameter is already full
if (data != 0){
// Reset the output
data->clear();
data = 0;
}


If data used to point to non-NULL, then you are just making it NULL. You should delete data before pointing it to NULL:

    if (data != 0){
// Reset the output
data->clear();
delete data;
data = 0;
}


Suggestion 2

Still better, if you have the option, change the interface to

void readNParseData(const char* filePath, vector<double>& data);


Suggestion 3

There is nothing in your code to indicate to the calling function that you were able or unable to read the data from the file. There is no else to go with

if (ifs) {


One way of indicating whether the function was successful in reading the data is to change the return type of the function to bool. Then, you can add

return true;


at the end of the if block and then add an else block:

else {
return false;
}


Suggestion 4

To remove the redundant memory allocation and deallocation to fileBuffer, simply use std::vector::data:

data.resize(sizeOfBuffer);


Suggestion 5

Add a check to make sure that ifs.read was successful:

ifs.read(reinterpret_cast<char*>(data.data()), fileSize);
if ( !ifs )
{
return false;
}

• Great suggestions, should I do something like if(data.size() != 0) instead of the null check? For your 2nd suggestion, did you suggest it because of the overhead of dereferencing operator? Also would you recommend C-style File pointer i/o or would you still recommend istream in terms of speed? – JohnJohn Dec 29 '14 at 20:54
• I think you don't need the check if (data.size() != 0) at all. You are going to overwrite the contents of data anyway. – R Sahu Dec 29 '14 at 23:00
• Using a reference is better than a pointer for an argument. The function doesn't have to worry about whether the argument is NULL or not. It can assume that it has a valid object to work with. It also has the other benefit of not having to worry about allocating memory on one function and deallocating memory in another function. Construction and destruction of the object can happen in the calling function. – R Sahu Dec 29 '14 at 23:02
• Regarding FILE* and istream, I haven't experimented to determine the relative performance. I would guess that they will be comparable for your usage. I'll be extremely surprised if they are not. – R Sahu Dec 29 '14 at 23:03
• I've tried using File* but it worked slower in my implementation. Was wondering whether it was my implementation or not. Thanks for the clarifications. – JohnJohn Dec 30 '14 at 5:39
• It would probably be more readable to check for an invalid file instead:

if (!ifs) {
return;
}


This can especially avoid having one large nested block.

• If you use new, you should only use delete:

delete[] fileBuffer;


Only use free() with malloc().

• Yeah great suggestions. I've used malloc in my previous implementation, forgot to replace it. Still didn't cause mem leak for some reason. – JohnJohn Dec 29 '14 at 20:43

Depending on how you are running your tests, here are a few things to consider:

1. Startup time: Which portion of the code are you timing? Opening the file, getting the file size, etc, and other operations can take time. These operations will be take a relatively fixed amount of time, and won't slow down larger files.
2. Hard drive latency: According to Wikipedia, the average seek latency is ~4ms. If the file isn't already cached by the OS, it will take (on average) that long to even begin to read.
3. Data transfer limits: According to the same article, the average HD transfer rate is ~1000Mbit/s. If actually reading the data is taking 1ms (see #2) that's a data transfer rate of 800MBits which doesn't leave a lot of room for improvement (assuming you're using a 7200rpm drive). If you're using a 5400rpm drive the data transfer rate can be expected to be lower.
4. Memory allocation/copying: Probably not going to be the main time sink, but could be optimized by not allocating memory for each file. If you have an upper limit to the size of the files, you can pre-allocate a buffer of that size, fill it in your function, and return the size. Depending on the rest of your architecture, you could get away with a single buffer, or you might have to use a pool.
5. Avoid the HD: If the files are being produced on the local machine, can they be written to shared memory or a ramdisk? That would significantly increase the seek times and data transfer rates.
• Well I have to take an average for the results as they just aren't the same for each execution. I am including the code given above (opening the file, getting the file size, reading and storing the file into a vector + a reinterpret_cast). Besides that I make a single iteration over the vector with a iterator. There are times that the code executed for only 2.4 milliseconds but there were also times it took 5 milliseconds. Tests that measure small intervals aren't as reliable as the ones which take longer. Thank you for the information. – JohnJohn Dec 29 '14 at 22:40
• You've measured the average time for your operation at 6ms. That really isn't a lot of time, yet you still want to reduce that. That suggests you are either planning on performing the entire operation many times repeatedly, you intend to drastically increase the size of the data file, or you wish to perform micro-optimization for it's own sake. The most appropriate optimizations will be different for each scenario. Can you provide more information? – tombrown52 Dec 29 '14 at 23:16
• Sure. I'm reading an offline analysis data for a binaural sound engine. The data that I'm reading is a result of series of signal processing calculations that I perform on a 512 window length FFT samples. The sampling rate is 44100 so each window equates to about 11.6 ms. So I need to process these data files under about 9ms or so for this to work on other systems. My other computer has SSD so I think it'll be less of an issue there but on HDD drives, hard disk access takes half of my time. So it's more about speed than volume of processing. – JohnJohn Dec 30 '14 at 5:36
• I also considered having multiple threads reading through the same file but it has more overall cost for the CPU then reading through with a single process / thread. I hope that answers your question. – JohnJohn Dec 30 '14 at 5:37
• Getting the size of file via seek is slow: it involves an extra access to a disk (possibly a number of them). Call stat instead.

• Copying an array into a vector is redundant indeed. You may read directly into std::vector::data.

• I am not 100% sure, but I don't think that seek actually accesses the disk. It just updates the offset in the file descriptor. – Martin R Dec 29 '14 at 18:37

This is how you get both text and numbers, floats, ints, doubles separated by , c++ using string stream to read each line.

if ( myFile.is_open( ) ) {
while ( getline( myFile, reading ) )
{