3
\$\begingroup\$

First of all, here's the code I'm working on:

def get_data(filename, cols):
    with codecs.open(filename, 'r') as csvFile:
        for row in csv.reader(csvFile):
            # build data dictionnary
            data = {}
            for i in range(len(cols)):
                key = cols[i]

                # update
                value = row[i].strip()

                # because some columns are splited in the csv file
                if key in data:
                    data[key] += value
                else:
                    data[key] = value

            # yield data, for each row
            yield data

So, depending of the size of the file, the time to process can be very long.

I tried testing csv.reader with custom dictionnary construction VS basic csv. DictReader, and for some reason, the first one is faster...

I have some long csv files with rows like this one:

"152Q694     ","892-000357          ",       0,       0,"        "

In some files, I have this (see the product name split in two columns?):

"A","COMPANY NAME         ","1234","987654321     ","I AM A PRODUCT NAME     ","WITH SOME EXTRA INFO          ","AB    ","12345               ",0000000000000001.23,0000000000000003.45,"A","            ","Z","01234567891234058","1234","EN",000000.01,"ABC","D","        ","        ",000000,000000,"            "

All in all, there will always be n rows and x columns, so the time of the process is a multiple of n*x. Am I wrong?

What can I do to speed things up?

\$\endgroup\$
  • \$\begingroup\$ Can we get a sample CSV file? Did you run a profiler? If this is Python 2, consider using codecs.open() to always work on unicode, and not having to decode/encode in your own code. If this is Python 3, why decode/encode? \$\endgroup\$ – Quentin Pradet Sep 24 '14 at 9:17
  • \$\begingroup\$ Hey, thanks for your comment. It's a shame, but I'm not comfortable with codecs ^^ I will try codecs.open (it's Python 2.7.6). I will edit my question for the sample. \$\endgroup\$ – Yann Sep 24 '14 at 9:26
  • \$\begingroup\$ In Python 2 the csv module does not support Unicode input, so don't use codecs.open. \$\endgroup\$ – Janne Karila Sep 24 '14 at 10:11
  • \$\begingroup\$ How large are the files? \$\endgroup\$ – Janne Karila Sep 24 '14 at 10:28
  • \$\begingroup\$ More than 62000 lines for the longest. \$\endgroup\$ – Yann Sep 24 '14 at 10:47
0
\$\begingroup\$

I have found a great piece of code to decode/encode on the fly: https://stackoverflow.com/a/9177937/2659902

So, I wrote this class, based on that:

class CSVRecoder(object):
    def __init__(self, filename, decoder, encoder='utf-8', eol='\r\n', 
                 dialect='excel', **fmtparams):
        """
            filename: to open the file object
            decoder, encoder, eol: to instanciate the recoder object
            dialect, **fmtparams: are passed to the csv.reader function
        """
        self.filename = filename
        self.decoder = decoder
        self.encoder = encoder
        self.eol = eol
        self.dialect = dialect
        self.fmtparams = fmtparams

    def __enter__(self):
        self.f = open(self.filename, 'rb')
        sr = Recoder(self.f, self.decoder, self.encoder, self.eol)
        return csv.reader(sr, self.dialect, **self.fmtparams)

    def __exit__(self, type, value, traceback):
        self.f.close()

def reader(*args, **kwargs):
    with CSVRecoder(*args, **kwargs) as csv_recoder:
        for row in csv_recoder:
            yield row

Now, I can do things like:

def get_data(fname, cols, encoding):
    def foo(row):
        data = {}
        get = data.get
        for i in xrange(len(cols)):
            key = cols[i]
            data[key] = get(key, '') + row[i].strip()
        yield data
    res = list(map(foo, csv_recoder.reader(fname, encoding)))

Perhaps I can optimise the dictionnary construction a bit more, but I doubt it would make a big difference (performance wise). Thanks anyway. Feel free to comment if you think I missed something somewhere, I'm always happy to learn ! :)

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.