11
\$\begingroup\$

I recently suggested this method for emulating the Unix utility split in Python.

Is there a more elegant way of doing it?

Assume that the file chunks are too large to be held in memory. Assume that only one line can be held in memory.

import contextlib

def modulo(i,l):
    return i%l

def writeline(fd_out, line):
    fd_out.write('{}\n'.format(line))

file_large = 'large_file.txt'
l = 30*10**6  # lines per split file
with contextlib.ExitStack() as stack:
    fd_in = stack.enter_context(open(file_large))
    for i, line in enumerate(fd_in):
        if not modulo(i,l):
           file_split = '{}.{}'.format(file_large, i//l)
           fd_out = stack.enter_context(open(file_split, 'w'))
        fd_out.write('{}\n'.format(line))

I ran the Unix utility time and the Python module cProfile. Here is what I found (methods not comparable, as I was running other processes, but gives a good indication of slow parts of code):

Ugo's method:

tottime filename:lineno(function)
473.088 {method 'writelines' of '_io._IOBase' objects}

485.36 real       362.04 user        58.91 sys

My code:

tottime function
243.532 modulo
543.031 writeline
419.366 {method 'format' of 'str' objects}
1169.735 {method 'write' of '_io.TextIOWrapper' objects}

3207.60 real      2291.42 user        44.64 sys

The Unix utility split:

1676.82 real       268.92 user      1399.16 sys
| improve this question | |
\$\endgroup\$
  • \$\begingroup\$ the use of contextlib makes things a bit complexe to me. You could keep the same code and simply close fd_out before opening a new file. This would make the code a lot clearer. \$\endgroup\$ – Ugo Jul 18 '14 at 15:16
  • \$\begingroup\$ But how can I open a file using the with statement without using contextlib? Maybe I'm misunderstanding you. Can you post your solution? \$\endgroup\$ – tommy.carstensen Jul 18 '14 at 18:22
  • \$\begingroup\$ yes I was speaking about getting rid of the with statement for fd_outs. It is a bit ugly, you open many files and close them all at the end while you could just open and close on the fly. \$\endgroup\$ – Ugo Jul 19 '14 at 10:09
8
\$\begingroup\$

Unfortunately, as far as I know, there is no chunks methods in the standard library. But this makes things rather neat.

from itertools import chain, islice

def chunks(iterable, n):
   "chunks(ABCDE,2) => AB CD E"
   iterable = iter(iterable)
   while True:
       yield chain([next(iterable)], islice(iterable, n-1))

l = ...
file_large = 'large_file.txt'
with open(file_large) as bigfile:
    for i, lines in enumerate(chunks(bigfile, l)):
        file_split = '{}.{}'.format(file_large, i)
        with open(file_split, 'w') as f:
            f.writelines(lines)
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ I forgot to mention that the file chunks are too large to be held in memory. \$\endgroup\$ – tommy.carstensen Jul 18 '14 at 18:18
  • 1
    \$\begingroup\$ @tommy.carstensen ok that should be fine then lines is a generator only holding one line plus an iterator on the rest of the chunk print type(lines) <type 'itertools.chain'>. \$\endgroup\$ – Ugo Jul 19 '14 at 10:04
  • 2
    \$\begingroup\$ I finally had time to have a proper look at your answer. I really appreciate the beauty of your solution! As you mention yourself it is memory efficient, because lines is an iterator yielded by the generator chunks. Your solution is also fast. I will edit my answer and show a comparison between the unix utility split, your solution and my solution. Your answer made me further aware of the power of iterators! Thanks! \$\endgroup\$ – tommy.carstensen Jul 20 '14 at 20:23
  • 1
    \$\begingroup\$ I think you should leave this as an answer to the original question on SO. \$\endgroup\$ – tommy.carstensen Jul 21 '14 at 10:08
  • 1
    \$\begingroup\$ Answering my own question. The trick on Python 3 is to open in the files in binary mode with 'rb' and 'wb', respectively. \$\endgroup\$ – Harald Nordgren Mar 4 '17 at 14:52
1
\$\begingroup\$

Now, there is a pypi module available that you can use to split files of any size into chunks with optimum use of memory. Check out filesplit.

| improve this answer | |
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.