Read from csv file, transform to dictionnary

Question

First of all, here's the code I'm working on:

def get_data(filename, cols):
    with codecs.open(filename, 'r') as csvFile:
        for row in csv.reader(csvFile):
            # build data dictionnary
            data = {}
            for i in range(len(cols)):
                key = cols[i]

                # update
                value = row[i].strip()

                # because some columns are splited in the csv file
                if key in data:
                    data[key] += value
                else:
                    data[key] = value

            # yield data, for each row
            yield data

So, depending of the size of the file, the time to process can be very long.

I tried testing csv.reader with custom dictionnary construction VS basic csv. DictReader, and for some reason, the first one is faster...

I have some long csv files with rows like this one:

"152Q694     ","892-000357          ",       0,       0,"        "

In some files, I have this (see the product name split in two columns?):

"A","COMPANY NAME         ","1234","987654321     ","I AM A PRODUCT NAME     ","WITH SOME EXTRA INFO          ","AB    ","12345               ",0000000000000001.23,0000000000000003.45,"A","            ","Z","01234567891234058","1234","EN",000000.01,"ABC","D","        ","        ",000000,000000,"            "

All in all, there will always be n rows and x columns, so the time of the process is a multiple of n*x. Am I wrong?

What can I do to speed things up?

Answer 1

I have found a great piece of code to decode/encode on the fly: https://stackoverflow.com/a/9177937/2659902

So, I wrote this class, based on that:

class CSVRecoder(object):
    def __init__(self, filename, decoder, encoder='utf-8', eol='\r\n', 
                 dialect='excel', **fmtparams):
        """
            filename: to open the file object
            decoder, encoder, eol: to instanciate the recoder object
            dialect, **fmtparams: are passed to the csv.reader function
        """
        self.filename = filename
        self.decoder = decoder
        self.encoder = encoder
        self.eol = eol
        self.dialect = dialect
        self.fmtparams = fmtparams

    def __enter__(self):
        self.f = open(self.filename, 'rb')
        sr = Recoder(self.f, self.decoder, self.encoder, self.eol)
        return csv.reader(sr, self.dialect, **self.fmtparams)

    def __exit__(self, type, value, traceback):
        self.f.close()

def reader(*args, **kwargs):
    with CSVRecoder(*args, **kwargs) as csv_recoder:
        for row in csv_recoder:
            yield row

Now, I can do things like:

def get_data(fname, cols, encoding):
    def foo(row):
        data = {}
        get = data.get
        for i in xrange(len(cols)):
            key = cols[i]
            data[key] = get(key, '') + row[i].strip()
        yield data
    res = list(map(foo, csv_recoder.reader(fname, encoding)))

Perhaps I can optimise the dictionnary construction a bit more, but I doubt it would make a big difference (performance wise). Thanks anyway. Feel free to comment if you think I missed something somewhere, I'm always happy to learn ! :)