Checks size of and number of files in each subdirectory

Question

The code snippet here checks in every folder for subfolders, and stores the sub-sub folders This is my sample directory structure. I want to read through every sub-folders to find the number of files and total size of the sub folder.

--Parent
    |---FolderA
    |   |__subFolder1 
    |   |__subFolder2
    |
    |---FolderB
        |__subFolder3
        |__subFolder4
        |__subFolder5

import glob
import os
from IPython import embed 
import subprocess
import humanize

list_of_files = os.listdir(os.getcwd())
total_stats =[] 
path = os.getcwd()

for files in list_of_files:  #check FolderA, Folder B etc.,
        print files
        while (os.walk(path+"/"+files).next()):
            curr_path ,dirs,_ = os.walk(path+"/"+files).next()
            data_dict = {}                     
            flag = "_hd" 

            for d in dirs:  #check subFolder1,2,3,4...
                 sub_p,sub_d,_ = os.walk(curr_path+"/"+d).next() 
                 for d  in sub_d:         #reads through files inside subfolder
                   p,_,f =os.walk(sub_p+"/"+d).next()
                   total_size = 0 
                   for fn in f:          #Finds total size of all files
                       total_size += os.path.getsize(p+"/"+fn)  
                   data_dict[d+flag] =  [len(f),humaize.naturalsize(total_size)] #prints file size in human readable form Ex: '2.3MB'
                 flag = "_rd"
            break     
    total_stats.append([files,data_dict])        #updates no. of files and sizes 

How can I optimize this code to avoid these many ridiculous amounts of for loops?

Answer 1

edit: I just realized, that you're using python2. I'm very unfamiliar with that and if you'll be able to use yield from, pathlib and format string literals. If you like the solution, I think I could rewrite it to be usable for python2.

Solution for python 3.6 and up (of that, I am very confident that it will produce no error):

Instead of creating all your information within one function, you can reduce the nesting depth by pulling up some of the functionality.

from pathlib import Path

def folders_in_path(path):
    if not Path.is_dir(path):
        raise ValueError("argument is not a directory")
    yield from filter(Path.is_dir, path.iterdir())

def folders_in_depth(path, depth):
    if 0 > depth:
        raise ValueError("depth smaller 0")
    if 0 == depth:
        yield from folders_in_path(path)
    else:
        for folder in folders_in_path(path):
            yield from folders_in_depth(folder, depth-1)

def files_in_path(path):
    if not Path.is_dir(path):
        raise ValueError("argument is not a directory")
    yield from filter(Path.is_file, path.iterdir())

def sum_file_size(filepaths):
    return sum([filep.stat().st_size for filep in filepaths])

if __name__ == '__main__':
    for folder in folders_in_depth(Path.cwd(),1):
        #       vvvv quick hack to to use len(), does not perform well
        files = list(files_in_path(folder))
        total_size = sum_file_size(files)
        print(f'{folder}: filecount:{len(files)}, total size:{total_size}')

which produces:

/tmp/Parent/FolderB/subfolder3: filecount:10, total size:50
/tmp/Parent/FolderB/subfolder2: filecount:10, total size:50
/tmp/Parent/FolderB/subfolder1: filecount:10, total size:50
/tmp/Parent/FolderA/subfolder3: filecount:10, total size:50
/tmp/Parent/FolderA/subfolder2: filecount:10, total size:50
/tmp/Parent/FolderA/subfolder1: filecount:10, total size:50

if your cwd is Parent. This will only count files in that are in a subdirectory at depth == 1 and nothing else.

There might be some neater solutions, that's just what I came up with in a few minutes.

---

edit2: I was curious, so I did a 2.7 version. os.walk is using depth first search. From what I gather, it could be done like this:

import os

def folders_in_depth(path, depth, walk_iter=None):
    if walk_iter is None:
        walk_iter = os.walk(path)
    if 0 > depth:
        raise ValueError("depth smaller 0")
    if 0 == depth:
        dirpath, dirnames, filenames = next(walk_iter)
        for i in range(len(dirnames)):
            dirpath, dirnames, filenames = next(walk_iter)
            yield dirpath, filenames
    else:
        dirpath, dirnames, filenames = next(walk_iter)
        for i in range(len(dirnames)):
            for result in folders_in_depth(path, depth-1, walk_iter=walk_iter):
                yield result

def sum_file_size(folder, filepaths):
    return sum([os.path.getsize(folder + '/' + filep) for filep in filepaths])

if __name__ == '__main__':
    for folder in folders_in_depth(os.getcwd(),1):
        foldername, files = folder
        print foldername,': filecount:', len(files), ', total size:', sum_file_size(foldername,files)

which results in

/tmp/Parent/FolderB/subFolderB3 : filecount: 10 , total size: 50
/tmp/Parent/FolderB/subFolderB2 : filecount: 10 , total size: 50
/tmp/Parent/FolderB/subFolderB1 : filecount: 10 , total size: 50
/tmp/Parent/FolderA/subFolderA3 : filecount: 10 , total size: 50
/tmp/Parent/FolderA/subFolderA2 : filecount: 10 , total size: 50
/tmp/Parent/FolderA/subFolderA1 : filecount: 10 , total size: 50

Not sure, where the extra spaces come from.

Here, the iterator is passed down in the recursion and has to be forwarded with next to produce the correct result.

Caveat: I think, this 2.7 solution will not work if the 'subFolder*' contain more folders on their own. Judging from how os.walk is implemented, modifying that solution might be the best option, so you don't have to worry about that. (sadly I was looking at how 3.6 implements os.walk, which is a little more complicated)

So for this to work correctly, you'd have to forward the iterator through the complete sub-tree, which is completely unneccesary. os.walk allows to modify the returned dirname to control where it should recurse via side effects. So clearing that might lead to the wanted result, you'd need to test that.