2
\$\begingroup\$

I've written a small package that counts the files and the size by file type in a directory tree. It produces the result as a markdown file. Here is a sample:

| directory | type | count | size |
| --- | --- | ---: | ---: |
| 0_0_root | ALL | 26 | 29.914 MiB |
| 0_0_root | .mp3 | 4 | 3.901 MiB |
| 0_0_root | .jpg | 5 | 211.019 KiB |
| 0_0_root | .blu | 5 | 9.829 MiB |
TRUNCATED
| 0_0_root/1_2_node | ALL | 6 | 12.852 MiB |
| 0_0_root/1_2_node | .jpg | 2 | 53.018 KiB |
| 0_0_root/1_2_node | .mp4 | 1 | 7.573 MiB |
TRUNCATED

I would be interested on how you would have iterated over the files. I'd be especially interested on how you would gain performance (time wise, I don't care about memory).

I've decided to use os.walk with the option topdown=False because that way I can iterate only once on each node. The idea is that it iterates over a tree in a way a that a directory is encountered after and only after it has already encountered all of its childs. And in order to include the size of the childs in the parent stats, I store them in a stack.

The stack can be modified in three way:

  1. A node add a clone of itself to the stack, waiting to be eaten by its parent
  2. A node get itself eaten by the top member of the stack, because it shares the same parent, the node from the stack continue to wait to be eaten by its parent
  3. A node eats the top member of the stack because it is its child

He is a first module that I defined to hold the data.

from collections import defaultdict
from os.path import splitext, join, dirname
from os import stat
import logging

log = logging.getLogger('tree_stat.dm')


class DirectoryMeasure:
    def __init__(self, files, path=None, parent=None):
        self.path = path
        self.parent = parent or dirname(path)
        self.file_type_measures = defaultdict(FilesMeasure)

        for f in files:
            ext = splitext(f)[1]
            file_size = stat(join(path, f)).st_size
            self.file_type_measures[ext].volume += file_size
            self.file_type_measures[ext].count += 1

    @property
    def total(self):
        it = FilesMeasure()
        for ext in self.file_type_measures.keys():
            it.volume += self.file_type_measures[ext].volume
            it.count += self.file_type_measures[ext].count
        return it

    def eat(self, child):
        for ext in child.file_type_measures.keys():
            self.file_type_measures[ext].volume += child.file_type_measures[ext].volume
            self.file_type_measures[ext].count += child.file_type_measures[ext].count

    def edible_clone(self):
        clone = DirectoryMeasure([], parent=dirname(self.path))
        clone.eat(self)
        return clone

    def __repr__(self):
        return 'DirectoryMeasure({})' \
            .format(', '.join(['{v}={{{v}}}'
                              .format(v=v) for v in vars(self).keys()])) \
            .format(**vars(self))


class FilesMeasure:
    def __init__(self):
        self.volume = 0
        self.count = 0

    def __repr__(self):
        return 'FilesMeasure({})' \
            .format(', '.join(['{v}={{{v}}}'
                              .format(v=v) for v in vars(self).keys()])) \
            .format(**vars(self))

And here is the main logic:

def take_measures(directory):
    dir_tree = dict()

    stack = []
    for current, sub_dirs, files in os.walk(directory, topdown=False):
        log.debug('working in {}'.format(current))

        measure = dm.DirectoryMeasure(files, path=current)
        dir_tree[current] = measure

        log.debug('own measure: {}'.format(measure))

        if stack and stack[-1].parent == measure.path:
            measure.eat(stack.pop())
            log.debug('child fed measure: {}'.format(measure))

        if stack and stack[-1].parent == measure.parent:
            stack[-1].eat(measure)
        else:
            stack.append(measure.edible_clone())

    return dir_tree

That's pretty much everything but if you want to play around with it, you're welcome: https://github.com/AdrienHorgnies/tree_stat.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.