I am trying to read entries of my CSV file into an entries vector whose data type is object OrderBookEntry
, which is an object that stores the tokenized variables from the CSV line which itself is converted through the function tokenise
. My problem is that my CSV file has around a million lines and it takes over 30 seconds to load; how can I reduce this?
This is my code that actually reads lines from the file and converts them into entries.
std::vector<OrderBookEntry> CSVReader::readCSV(std::string csvFilename)
{
std::vector<OrderBookEntry> entries;
std::ifstream csvFile{csvFilename};
std::string line;
if (csvFile.is_open())
{
while(std::getline(csvFile, line))
{
try {
OrderBookEntry obe = stringsToOBE(tokenise(line, ','));
entries.push_back(obe);
//std::cout << "pushing entry" << std::endl;
}catch(const std::exception& e)
{
//std::cout << "CSVReader::readCSV bad data" << std::endl;
}
}// end of while
}
std::cout << "CSVReader::readCSV read " << entries.size() << " entries" <<
std::endl;
return entries;
}
This is the part of my code that tokenises the read line:
std::vector<std::string> CSVReader::tokenise(std::string csvLine, char separator)
{
std::vector<std::string> tokens;
signed int start, end;
std::string token;
start = csvLine.find_first_not_of(separator, 0);
do{
end = csvLine.find_first_of(separator, start);
if (start == csvLine.length() || start == end) break;
if (end >= 0) token = csvLine.substr(start, end - start);
else token = csvLine.substr(start, csvLine.length() - start);
tokens.push_back(token);
start = end + 1;
}while(end > 0);
return tokens;
}
This is the part of my code that converts the tokens into a suitable object to work with:
OrderBookEntry CSVReader::stringsToOBE(std::vector<std::string> tokens)
{
double price, amount;
if (tokens.size() != 5) // bad
{
//std::cout << "Bad line " << std::endl;
throw std::exception{};
}
// we have 5 tokens
try {
price = std::stod(tokens[3]);
amount = std::stod(tokens[4]);
}catch(const std::exception& e){
std::cout << "Bad float! " << tokens[3]<< std::endl;
std::cout << "Bad float! " << tokens[4]<< std::endl;
throw;
}
OrderBookEntry obe{price,
amount,
tokens[0],
tokens[1],
OrderBookEntry::stringToOrderBookType(tokens[2])};
return obe;
}
Do advise on how I can make the code faster.
string_view
use methods from<charconv>
likefrom_chars
they convert numbers much faster thanstod
. Assuming, your compiler supports them ofc. Also, whyamount
is a double? Shouldn't it be integer? \$\endgroup\$