5
\$\begingroup\$

This is a script I wrote to find the n biggest files in a given directory (recursively):

import heapq
import os, os.path
import sys
import operator

def file_sizes(directory):
    for path, _, filenames in os.walk(directory):
        for name in filenames:
            full_path = os.path.join(path, name)
            yield full_path, os.path.getsize(full_path)

num_files, directory = sys.argv[1:]
num_files = int(num_files)

big_files = heapq.nlargest(
        num_files, file_sizes(directory), key=operator.itemgetter(1))
print(*("{}\t{:>}".format(*b) for b in big_files))

It can be run as, eg: bigfiles.py 5 ~.

Ignoring the complete lack of error handling, is there any obvious way to make this clearer, or at least more succinct? I am thinking about, eg, using namedtuples in file_sizes, but is there also any way to implement file_sizes in terms of a generator expression? (I'm thinking probably not without having two calls to os.path, but I'd love to be proven wrong :-)

\$\endgroup\$
4
\$\begingroup\$

You could replace your function with:

file_names = (os.path.join(path, name) for path, _, filenames in os.walk(directory)
        for name in filenames)

file_sizes = ((name, os.path.getsize(name)) for name in file_names)

However, I'm not sure that really helps the clarity.

I found doing this:

big_files = heapq.nlargest(
        num_files, file_names, key=os.path.getsize)
print(*("{}\t{:>}".format(b, os.path.getsize(b)) for b in big_files))

Actually runs slightly quicker then your version.

\$\endgroup\$
  • \$\begingroup\$ That's an interesting result. How did you time it? You wouldn't think that twice as many syscalls would speed it up! \$\endgroup\$ – lvc Feb 15 '12 at 2:47
  • \$\begingroup\$ @lvc, I used time python script.py. Its not making twice as many sys calls because I'm only making the second call for the top 5 or so files. \$\endgroup\$ – Winston Ewert Feb 15 '12 at 3:59
  • \$\begingroup\$ Ah, indeed, it will only be num_files extra syscalls. Presumably, what's happening is that constructing multiple tuples is more expensive - if that's the case, we could expect the difference to shrink and eventually go away as num_files approaches the number of files searched. But, when I look at it, I think I do prefer key=os.path.getsize over my many tuples for clarity, anyway. \$\endgroup\$ – lvc Feb 15 '12 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.