# Read from csv file, transform to dictionnary

First of all, here's the code I'm working on:

def get_data(filename, cols):
with codecs.open(filename, 'r') as csvFile:
# build data dictionnary
data = {}
for i in range(len(cols)):
key = cols[i]

# update
value = row[i].strip()

# because some columns are splited in the csv file
if key in data:
data[key] += value
else:
data[key] = value

# yield data, for each row
yield data


So, depending of the size of the file, the time to process can be very long.

I tried testing csv.reader with custom dictionnary construction VS basic csv. DictReader, and for some reason, the first one is faster...

I have some long csv files with rows like this one:

"152Q694     ","892-000357          ",       0,       0,"        "


In some files, I have this (see the product name split in two columns?):

"A","COMPANY NAME         ","1234","987654321     ","I AM A PRODUCT NAME     ","WITH SOME EXTRA INFO          ","AB    ","12345               ",0000000000000001.23,0000000000000003.45,"A","            ","Z","01234567891234058","1234","EN",000000.01,"ABC","D","        ","        ",000000,000000,"            "


All in all, there will always be n rows and x columns, so the time of the process is a multiple of n*x. Am I wrong?

What can I do to speed things up?

• Can we get a sample CSV file? Did you run a profiler? If this is Python 2, consider using codecs.open() to always work on unicode, and not having to decode/encode in your own code. If this is Python 3, why decode/encode? – Quentin Pradet Sep 24 '14 at 9:17
• Hey, thanks for your comment. It's a shame, but I'm not comfortable with codecs ^^ I will try codecs.open (it's Python 2.7.6). I will edit my question for the sample. – Yann Sep 24 '14 at 9:26
• In Python 2 the csv module does not support Unicode input, so don't use codecs.open. – Janne Karila Sep 24 '14 at 10:11
• How large are the files? – Janne Karila Sep 24 '14 at 10:28
• More than 62000 lines for the longest. – Yann Sep 24 '14 at 10:47

I have found a great piece of code to decode/encode on the fly: https://stackoverflow.com/a/9177937/2659902

So, I wrote this class, based on that:

class CSVRecoder(object):
def __init__(self, filename, decoder, encoder='utf-8', eol='\r\n',
dialect='excel', **fmtparams):
"""
filename: to open the file object
decoder, encoder, eol: to instanciate the recoder object
dialect, **fmtparams: are passed to the csv.reader function
"""
self.filename = filename
self.decoder = decoder
self.encoder = encoder
self.eol = eol
self.dialect = dialect
self.fmtparams = fmtparams

def __enter__(self):
self.f = open(self.filename, 'rb')
sr = Recoder(self.f, self.decoder, self.encoder, self.eol)

def __exit__(self, type, value, traceback):
self.f.close()

with CSVRecoder(*args, **kwargs) as csv_recoder:
for row in csv_recoder:
yield row


Now, I can do things like:

def get_data(fname, cols, encoding):
def foo(row):
data = {}
get = data.get
for i in xrange(len(cols)):
key = cols[i]
data[key] = get(key, '') + row[i].strip()
yield data