I am trying to read a single-columned CSV of doubles into Java
with a string header. It is 11 megabytes and takes over 15 minutes to read, which is clearly unacceptable. In R
this CSV would take about 3 seconds to load.
This CSV file may contain strings so I am parsing it with this in mind.
The CSV reading method needs to return Vector<Double>
due to reliance on this output by other parts of the application.
The issue is not due to the isNumber
static method, since each call to that is taking 200 nanoseconds
, thus contributing approximately 0.2 seconds to the 15 minutes of parsing time.
The Double.valueOf()
only takes about 500 nanoseconds, so it is not that either.
csvData.add()
is only taking 80 nanoseconds so it is not that.
private static Vector<Double> readTXTFileSingle(String csvFileName) throws IOException {
String line = null;
BufferedReader stream = null;
Vector<Double> csvData = new Vector<Double>();
try {
stream = new BufferedReader(new FileReader(csvFileName));
while ((line = stream.readLine()) != null) {
String[] splitted = line.split(",");
if( ! NumberUtils.isNumber(splitted[0])) {
continue;
}
Double dataLine = Double.valueOf(splitted[0]);
csvData.add(dataLine);
}
} finally {
if (stream != null)
stream.close();
}
return csvData;
}
NumberUtils.isNumber
implemented? \$\endgroup\$