At the time I'm trying to read a quite big file into a C program for later user. The file size is in the range of 800 megabytes containing around 20 million lines of data of the following format:
YYYYMMDD HHMMSSMMM,X.XXXXX0,X,XXXXX0,0
- Y: Year
- M: Month
- D: Day
- H: Hour
- M: Minute
- S: Second
- M: Milliseconds
- X: floating point number
Here are a few examples:
20150101 130021493,1.209650,1.210070,0 20150101 130044493,1.209720,1.210140,0 20150101 130044743,1.209650,1.210070,0 20150101 130045493,1.209720,1.210140,0 20150101 130045743,1.209670,1.210090,0
I want to read the data of a single line into the following structure:
struct forexData {
struct tm timestamp;
uint32_t bidQuote;
uint32_t askQuote;
};
As you can already imagine, this data represents the bid/ask quotes at a specific point in time. The bid/ask quotes are stored in uint32_t, since I prefer an integer value over floats for later use. The structures of each line are placed in another structure, which will contain all data.
struct forexDataSet {
struct forexData **data;
uint32_t capacity;
uint32_t cardinality;
};
The data-property points to an array of pointers to forexData structures. The capacity-property describes how many pointers the array can hold and cardinality holds, how many pointers are currently used.
Here are the routines to read in the file:
#include <time.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <unistd.h>
#include <stdint.h>
#include <stdlib.h>
#define INITIAL_SET_SIZE 10000000
#define SET_INCREASE_STEP 1000000
// fileDescriptor returned by open(fileName, O_RDONLY, 0);
struct forexDataSet *createForexDataSetFromFile(int fileDescriptor) {
char buf[39000];
ssize_t nBytesRead;
struct forexDataSet *set = calloc(1, sizeof(struct forexDataSet));
set->data = malloc(INITIAL_SET_SIZE * sizeof(struct forexData *));
if (!set->data) {
return NULL;
}
set->capacity = INITIAL_SET_SIZE;
do {
nBytesRead = read(fileDescriptor, buf, sizeof(buf));
size_t lineStart;
for (lineStart = 0; lineStart < nBytesRead; lineStart += 39) {
struct forexData *tmp = createForexDataFromString(buf + lineStart);
if (!tmp) {
freeForexDataSet(set);
return NULL;
} else {
if (set->cardinality == set->capacity) {
struct forexData **increasedData = realloc(set->data, sizeof(struct forexData *) * (set->capacity + SET_INCREASE_STEP));
if (increasedData == NULL) {
freeForexDataSet(set);
free(tmp);
return NULL;
}
set->data = increasedData;
set->capacity += SET_INCREASE_STEP;
}
set->data[set->cardinality++] = tmp;
}
}
} while(nBytesRead == sizeof(buf));
return set;
}
struct forexData *createForexDataFromString(const char *str) {
struct forexData *tmp = calloc(1, sizeof(struct forexData));
if (!tmp) {
return NULL;
}
strptime(str, "%Y%m%d %H%M%S", &tmp->timestamp);
tmp->bidQuote = (uint32_t) (atof(str + 19) * 1000000);
tmp->askQuote = (uint32_t) (atof(str + 28) * 1000000);
return tmp;
}
void freeForexDataSet(struct forexDataSet *set) {
size_t dataIndex;
for (dataIndex = 0; dataIndex < set->cardinality; dataIndex++) {
free(set->data[dataIndex]);
}
free(set->data);
free(set);
}
The code compile with clang flags -Weverything -Wextra without warnings except one warning (comparing size_t to ssize_t in a for-loop).
What I'd like to get reviewed / brainstormed:
- Performance. Currently the process takes around 7 seconds to complete and I was wondering if there is anything major I did wrong.
- Buffer size. 39000 (exactly 1000 lines, since one line contains 39 characters). I only read in whole lines at a time, otherwise I'd had a problem with overlapping lines and parsing, but I chose 39000 without a specific reason. I once even tried to allocate a buffer sized as big as the file and it worked pretty well (I have 16GB of RAM), but I would prefer to not suck up all the RAM while parsing.
- Hard-Coded-Pieces. As you probably already saw a lot of this is hard coded
atof(str + 19),39000bytes buffer size,strptime(str, "%Y%m%d %H%M%S", &tmp->timestamp);and I was wondering how to improve these without to implement a fairly complex formatting string parser. Maybe there is a clever way to determine the buffer size of something like this. - Error handling and freeing. Is everything freed correctly, when something could not get allocated, etc.
