In various projects I need to enumerate all files and folders from a specific root folder, either on a local drive or across a network. I've broken the task down into two IEnumerable
implementations as follows:
static IEnumerable<string> EnumeratePaths(string root)
{
if (root == null)
throw new ArgumentNullException("root");
if (!Directory.Exists(root))
throw new ArgumentException("Invalid root path", "root");
if (root.Length > 3)
root = Path.GetDirectoryName(root + "\\");
Queue<string> queue = new Queue<string>();
queue.Enqueue(root);
while (queue.Count > 0)
{
string curr = queue.Dequeue();
bool failed = false;
try
{
foreach (var path in Directory.GetDirectories(curr))
queue.Enqeue(path);
}
catch
{
failed = true;
}
if (!failed)
yield return curr;
}
}
static IEnumerable<string> EnumerateFiles(string root)
{
var paths = EnumeratePaths(root);
foreach (var nxt in paths)
{
foreach (var filename in Directory.GetFiles(nxt))
yield return filename;
}
}
From these I can build LINQ expressions to do filtering, etc. When I'm scanning a folder structure containing a few million files in several thousand folders this saves me from filling up memory with data that I only need once.
However in certain cases I have to process folders that contain tens of thousands of files, and the entire process stalls at the call to GetDirectories
or GetFiles
while it processes the folder... and bulks up my process memory significantly with the resultant arrays. Worse, this happens twice in EnumerateFiles
- once when EnumeratePaths
calls GetDirectories
, then again when I call GetFiles
.
So my questions are:
Is there anything overtly inefficient in the implementation as it stands (not related to the file system problems mentioned).
How can I best reduce the memory bloat and so on when processing folders containing huge numbers of files?
Update 1: Directory.EnumerateFiles(path, "*", SearchOption.AllDirectories)
As MarcinJuraszek pointed out there is already a way to do full-depth file enumeration using Directory.EnumerateFiles
that doesn't require me to fetch a full array of results in one hit. That's good because one of the folders I'm scanning has 150,000+ files in it (don't ask).
Here's some code I wrote to test it, using a drive mapped to a network share:
Stopwatch sw = new Stopwatch();
for (int i = 0; i < 10; i++)
{
sw.Restart();
var enumfiles = Directory.EnumerateFiles("I:\\", "*", SearchOption.AllDirectories);
int maxlen = enumfiles.Max(fname => fname.Length);
sw.Stop();
Console.WriteLine("{0}: {1:0.00}s", i, sw.ElapsedMilliseconds / 1000.0);
}
The average time over 10 runs was 73.942 seconds. Substituting the EnumerateFiles
method above gave an average of 72.264 seconds - close enough that network traffic variations could account for the difference. As for memory usage... Directory.EnumerateFiles()
gives a peak working set (from Process.GetCurrentProcess().PeakWorkingSet64
) of 35,896,786
bytes after 10 runs, compared to 80,711,680
bytes - a significant improvement.
Still feels like there is a lot of room for improvement though.
Update 2: FindFirstFile()
API
After much googling and testing I managed to get the kernel32 FindFirstFile()
API working in C#, hoping that it would provide a bit more of a boost. For a few of my projects it might do, since it gives me all sorts of useful information in the WIN32_FIND_DATA
structure during enumeration... but the overall performance when just enumerating file name is approximately equivalent to Directory.EnumerateFiles
, both in time and memory usage.
Thanks Marcin - answer accepted.
Update 3: FindFirstFile
revisited
Final update. Using some of the code from P/Invoke I've rewritten my FindFile enumeration code as follows:
public static IEnumerable<string> EnumerateFiles(string path)
{
WIN32_FIND_DATA finddata;
Queue<string> paths = new Queue<string>();
paths.Enqueue(path);
while (paths.Count > 0)
{
var nxtpath = paths.Dequeue();
using (var fh = FindFirstFile(Path.Combine(nxtpath, "*"), out finddata))
{
if (fh.IsInvalid)
continue;
bool ok = true;
while (ok)
{
if ((finddata.dwFileAttributes & FileAttributes.Directory) == FileAttributes.Directory)
{
if (finddata.cFileName != "." && finddata.cFileName != "..")
paths.Enqueue(Path.Combine(nxtpath, finddata.cFileName));
}
else
yield return Path.Combine(nxtpath, finddata.cFileName);
ok = FindNextFile(fh, out finddata);
}
}
}
}
This performs almost twice as fast as Directory.EnumerateFiles
in my test case - network mapped drive containing numerous files and folders including one folder with over 150,000 files. I think this one is going to end up in my library :)