Reading binary file of doubles

Question

I'm trying to read doubles from a relatively small binary file. This currently reads a 100 KB file in about 6 milliseconds in my system. I would like to reduce that if possible.

void readNParseData(const char* filePath, vector<double> *&data){

    ifstream ifs(filePath, ios::in | ios::binary);

    // If this is a valid file
    if (ifs) {
        // Temporary Variables
        std::streampos fileSize;
        double *fileBuffer;
        size_t sizeOfBuffer;

        // Check whether the parameter is already full
        if (data != 0){
            // Reset the output
            data->clear();
            data = 0;
        }

        // Get the size of the file
        ifs.seekg(0, std::ios::end);
        fileSize = ifs.tellg();
        ifs.seekg(0, std::ios::beg);

        sizeOfBuffer = fileSize / sizeof(double);
        fileBuffer = new double[sizeOfBuffer];

        ifs.read(reinterpret_cast<char*>(fileBuffer), fileSize);

        // Now convert the double array into vector
        data = new vector<double>(fileBuffer, fileBuffer + sizeOfBuffer);

        free(fileBuffer);
    }
}

As you can see there is a redundant copy of a double * array to a vector. Perhaps reading to the vector directly might speed it up, but I don't know how.

Answer 1

Suggestion 1

This block of code does not seem clean:

    // Check whether the parameter is already full
    if (data != 0){
        // Reset the output
        data->clear();
        data = 0;
    }

If data used to point to non-NULL, then you are just making it NULL. You should delete data before pointing it to NULL:

    if (data != 0){
        // Reset the output
        data->clear();
        delete data;
        data = 0;
    }

Suggestion 2

Still better, if you have the option, change the interface to

void readNParseData(const char* filePath, vector<double>& data);

Suggestion 3

There is nothing in your code to indicate to the calling function that you were able or unable to read the data from the file. There is no else to go with

if (ifs) {

One way of indicating whether the function was successful in reading the data is to change the return type of the function to bool. Then, you can add

return true;

at the end of the if block and then add an else block:

else {
   return false;
}

Suggestion 4

To remove the redundant memory allocation and deallocation to fileBuffer, simply use std::vector::data:

data.resize(sizeOfBuffer);
ifs.read(reinterpret_cast<char*>(data.data()), fileSize);

Suggestion 5

Add a check to make sure that ifs.read was successful:

ifs.read(reinterpret_cast<char*>(data.data()), fileSize);
if ( !ifs )
{
   return false;
}

Answer 2

• It would probably be more readable to check for an invalid file instead:

  if (!ifs) {
      return;
  }
  

  This can especially avoid having one large nested block.

• If you use new, you should only use delete:

  delete[] fileBuffer;
  

  Only use free() with malloc().

Answer 3

Depending on how you are running your tests, here are a few things to consider:

1. Startup time: Which portion of the code are you timing? Opening the file, getting the file size, etc, and other operations can take time. These operations will be take a relatively fixed amount of time, and won't slow down larger files.
2. Hard drive latency: According to Wikipedia, the average seek latency is ~4ms. If the file isn't already cached by the OS, it will take (on average) that long to even begin to read.
3. Data transfer limits: According to the same article, the average HD transfer rate is ~1000Mbit/s. If actually reading the data is taking 1ms (see #2) that's a data transfer rate of 800MBits which doesn't leave a lot of room for improvement (assuming you're using a 7200rpm drive). If you're using a 5400rpm drive the data transfer rate can be expected to be lower.
4. Memory allocation/copying: Probably not going to be the main time sink, but could be optimized by not allocating memory for each file. If you have an upper limit to the size of the files, you can pre-allocate a buffer of that size, fill it in your function, and return the size. Depending on the rest of your architecture, you could get away with a single buffer, or you might have to use a pool.
5. Avoid the HD: If the files are being produced on the local machine, can they be written to shared memory or a ramdisk? That would significantly increase the seek times and data transfer rates.

Answer 4

• Getting the size of file via seek is slow: it involves an extra access to a disk (possibly a number of them). Call stat instead.

• Copying an array into a vector is redundant indeed. You may read directly into std::vector::data.

Answer 5

This is how you get both text and numbers, floats, ints, doubles separated by , c++ using string stream to read each line.

if ( myFile.is_open( ) ) {
    while ( getline( myFile, reading ) )
    {
        stringstream eachline( reading ) ;
        while ( eachline( ) )
        {
            string x = ',';
            getline( eachline, StoreData, x ) ;

    //    read if float or read else word from file

            Increment++ here    
        }
    }
}