Listing first 50 big files in directory and print relative paths and sizes

Question

I use this Powershell command to recursively list first 50 big files in current directory and print relative paths and sizes in MiB. :

$dirName = $args[0]
if( $args.Count -eq 0 ) { $dirName = "." }
dir -Force -r $dirName -file | sort Length -descending | select @{ n= "RelativePath"; e={ Resolve-Path -Relative $_.fullname } } , @{ n="Size"; e={ "{0,-5:n3} MiB" -f ($_.length / 1mb) } } -f 50

EDIT0 : I tried this but it is slower :

dir -Force -r -file $dirName | select -Property fullName,Length | ? Length -gt 1mb | sort Length -desc | select -f 50 | select @{ n= "RelativePath"; e={ Resolve-Path -Relative $_.fullname } } , @{ n="Size"; e={ "{0,5:n3} MiB" -f ($_.length / 1mb) } }

Can this be simplified or optimised ?

Answer 1

filter

You do: dir | sort | select50

Can this be simplified or optimised ?

Yes, by sorting fewer entries: dir | filter | sort | select50

That is, pick an arbitrary cutoff like 1 MiB which you expect will be smaller than the top 50 selected entries, and filter out those tiny files early on. Now the sort stage has fewer input rows to worry about. Since dir accounts for most of the I/O cost, I would expect the savings to be quite minimal.

early formatting

The select is actually doing two things: formatting each row, and choosing the first head -50 rows. You might perform those two tasks at different points in the pipeline:

dir | fmt | sort | select50

If the fmt stage cuts the length of each input record in half, then the sort stage has half as much data to process. Again, I would expect the savings to be minimal. It is certainly possible to combine the "early formatting" and "early filtering" approaches in a single pipeline.

---

random I/O

Querying metadata from all over the disk takes more time than sequentially streaming a giant text file of similar size.

If you have subtrees which seldom change, consider caching dir output on a per-subtree or per-directory level. Combine those cache files to obtain the same dir -r output. Now the problem becomes one of deciding when to do cache invalidates and then reconstruct such files.

If you run this command "often", say several times per hour, and the giant files change "seldom", such as daily, then this can be a big win.

Maybe you have one or two "giant file generators" of interest, which could cooperate by doing a cache invalidate each time they run. A build script might fit this pattern.

only record the big fish

A subtree could have a million tiny files, yet we only report the top 50. Focus your caching efforts on, say, the top 60 or top 100 files. On a subsequent run, use dir to retrieve current metadata for those 60 or 100 entries. If no change, then "make a leap of faith" that no new giant file was created a moment ago, and report the top 50. So the I/O work is \$O(60)\$ rather than \$O(10^6)\$.

There may be a handful of folders where "new winners" appear, such as an obj/ or bin/ directory in which re-compiling source code will sometimes produce a new giant.exe file. Focus your cache checking / invalidation efforts on such folders.

free space

On each run, cache the number of free blocks. Be more aggressive about looking for fresh giant files when you notice that total free space went down a lot since the previous run.

cron job

Asynchronously run dir -r every hour or so, caching results to a text file in the background. No interactive user is impatiently awaiting its results.

Foreground queries can now do sequential I/O against a very large text file, without need of dir slowly doing random I/O. Additionally, we can sort those cached results, so that foreground queries only read a small portion of the cache.