I am looking for the fastest way to parse an 80GB csv file with 300 columns in Python.
The csv file does not have backquote commas, e.g
I have tried Python's built-in csv module which gives me ~50MB/s speed.
with open(file_name) as csvfile: reader = csv.reader(csvfile,delimiter=',') for row in reader: pass
line.split() I get ~55MB/s:
with open(file_name) as csvfile: for line in csvfile: row = line.split(',')
Without parsing, I get ~400MB/s:
with open(file_name) as csvfile: for line in csvfile: pass
Using cperf I see that most of time time is used for
re.split(), which I have also used.
Can I make this faster? If a plain read from disk is 400MB/s, I would like to get ~200MB/s while parsing the csv file if possible.