4
\$\begingroup\$

I am looking for the fastest way to parse an 80GB csv file with 300 columns in Python.

The csv file does not have backquote commas, e.g a,"blah,blah,blah",c

I have tried Python's built-in csv module which gives me ~50MB/s speed.

with open(file_name) as csvfile:
    reader = csv.reader(csvfile,delimiter=',')
    for row in reader:
        pass

With plain line.split() I get ~55MB/s:

with open(file_name) as csvfile:
    for line in csvfile:
        row = line.split(',')

Without parsing, I get ~400MB/s:

with open(file_name) as csvfile:
    for line in csvfile:
        pass

Using cperf I see that most of time time is used for split() or csv.reader() or re.split(), which I have also used.

Can I make this faster? If a plain read from disk is 400MB/s, I would like to get ~200MB/s while parsing the csv file if possible.

\$\endgroup\$

closed as too broad by Vogel612, RubberDuck, Pimgd, rolfl Feb 3 '15 at 14:45

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    \$\begingroup\$ Perhaps one of the libraries mentioned here could help: Fastest Python library to read a CSV file \$\endgroup\$ – Janne Karila Feb 3 '15 at 11:48
  • \$\begingroup\$ You may want to look at whitedb.org - it has CSV import and it has Python bindings. Whether the import is as fast as the database itself - I don't know, still might be worth a shot. \$\endgroup\$ – wvxvw Feb 3 '15 at 12:17
  • 3
    \$\begingroup\$ I'm voting to close this question as off-topic because not enough context has been provided, and currently it's a library/algorithm recommendation question. Code Reviews are done over more than just 3 lines of code. Perhaps this should get migrated to StackOverflow instead. Alternatively, Softwarerecs.stackexchange.com, with a library tag. \$\endgroup\$ – Pimgd Feb 3 '15 at 13:58
  • \$\begingroup\$ I have tried pandas briefly. It has tried to load all to memory but failed with memory error when RAM became short. Not yet tried whitedb.org but they say they try to keep all in RAM. So no option for me. \$\endgroup\$ – kotlet schabowy Feb 3 '15 at 20:57
3
\$\begingroup\$

I would suggest you to create a thread pool and do line.split() in separate thread. Playing with thread pool size can give you balance between processing speed and consumed resources.

\$\endgroup\$
  • \$\begingroup\$ There's the GIL, though. \$\endgroup\$ – Janne Karila Feb 3 '15 at 11:38
  • \$\begingroup\$ if GIL would have much influence on such task (and I'm not sure it will) then process pool can be used instead \$\endgroup\$ – iwlagn Feb 3 '15 at 13:00
  • \$\begingroup\$ Have you tried pypy ? \$\endgroup\$ – Marcin Feb 3 '15 at 15:46
  • \$\begingroup\$ I am using Cython. PyPy would be an option to try. Today I have tried multiprocessing.pool, map_async() but without any performance gain. \$\endgroup\$ – kotlet schabowy Feb 3 '15 at 20:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.