# Fast import of numbers stored in a CSV file

I would like to efficiently import real numbers stored in CSV format into an array. Is there a way to modify my code to make it faster? Also, is there a way to scan the file and compute the number of rows and columns rather than having to provide these directly?

double * getcsv(string fname, int m, int n){
double *a;
a = (double *)malloc(n * m * sizeof(double));

ifstream fs(fname.c_str());
char buf[100];

for(int i=0;i<m;i++){
for(int j=0;j<n;j++){
if (j < (n-1) ){
fs.getline(buf,100,',');}
else{
fs.getline(buf,100);}
stringstream data(buf);
data>>a[(i*n)+j];

}
}

return(a);
}

• I am not sure how to do the formatting to indicate my code. If someone knows how, I will fix this quickly. May 19 '11 at 20:57

• In C++, we usually avoid malloc - in this case, you could use a std::vector< double > in stead. Doing this, you will allow the user of your code to use RAII in stead of manually managing the allocated memory. You cal tell the vector how much to allocate by initializing it with the right size. The compiler will apply return value optimization to avoid copying the vector (by constructing it into the return value) so you shouldn't worry too much about that.
• You don't preserve the values of m and n. I don't know if this is intentional but if you don't want to burden your client code with lugging it around, consider a struct or a class to carry it, and the values you read, around.
• You don't check the return values of the methods you call on the fstream - you should.
• You'd probably be better off reading the values from the fstream directly into the array, rather than reading them into a buffer, copying that buffer into a stringstream and then reading them from the stringstream to convert. You can save the use of the stringstream (and therefore eliminate it altogether).

Example program:

#include <iostream>
#include <iterator>
#include <sstream>
#include <vector>

using namespace std;

struct Data
{
Data(unsigned int width, unsigned int height)
: width_(width)
, height_(height)
, data_(width * height)
{ /* no-op */ }
unsigned int width_;
unsigned int height_;
vector< double > data_;
};

Data split(istream &is, unsigned int width, unsigned int height)
{
Data retval(width, height);
double d;
int i(0);

while (is >> d)
{
retval.data_[i++] = d;
is.ignore(1, ',');
}

return retval;
}

int main()
{
const char *test_string = ""
"2.83, 31.26, 2354.3262, 0.83567\n"
"12.3, 35.236, 2854.3262, 0.83557\n"
"32.3, 33.26, 2564.3262, 0.83357\n"
"27.3, 3.2869, 2594.3262, 0.82357\n";
const unsigned int width = 4;
const unsigned int height = 4;
stringstream ss(test_string);
Data data(split(ss, width, height));
copy(data.data_.begin(), data.data_.end(), ostream_iterator< double >(cout, " "));
}

• Thanks for the feedback. Can you give me a sample of the code I need to write to read the values from fstream directly into an array? May 20 '11 at 2:30
• @HenryB I've edited my answer to add sample code. The example is self-sufficient, but having similar input in a file and using fstream should yield the same results. Note that only the split function is the function you're looking for
– rlc
May 21 '11 at 0:04

rlc has covered all the important parts in his answer. I merely want to mention boost::iostreams::mapped_file since you specifically asked for performance improvements. Memory-mapping files can give you great speed boosts. Note that unlike std::fstream, the file must exist before being opened. Also, the memory-mapped file is opened in binary mode so you must alter you code accordingly.

Memory-mapped files are much more low-level than your current approach. Only pursue this option if you have profiled your current code and found it to be I/O bound.