Need advice on file/directory enumeration code

Question

In various projects I need to enumerate all files and folders from a specific root folder, either on a local drive or across a network. I've broken the task down into two IEnumerable implementations as follows:

static IEnumerable<string> EnumeratePaths(string root)
{
    if (root == null)
        throw new ArgumentNullException("root");
    if (!Directory.Exists(root))
        throw new ArgumentException("Invalid root path", "root");

    if (root.Length > 3)
        root = Path.GetDirectoryName(root + "\\");

    Queue<string> queue = new Queue<string>();
    queue.Enqueue(root);

    while (queue.Count > 0)
    {
        string curr = queue.Dequeue();
        bool failed = false;
        try
        {
            foreach (var path in Directory.GetDirectories(curr))
                queue.Enqeue(path);
        }
        catch
        {
            failed = true;
        }
        if (!failed)
            yield return curr;
    }
}

static IEnumerable<string> EnumerateFiles(string root)
{
    var paths = EnumeratePaths(root);
    foreach (var nxt in paths)
    {
        foreach (var filename in Directory.GetFiles(nxt))
            yield return filename;
    }
}

From these I can build LINQ expressions to do filtering, etc. When I'm scanning a folder structure containing a few million files in several thousand folders this saves me from filling up memory with data that I only need once.

However in certain cases I have to process folders that contain tens of thousands of files, and the entire process stalls at the call to GetDirectories or GetFiles while it processes the folder... and bulks up my process memory significantly with the resultant arrays. Worse, this happens twice in EnumerateFiles - once when EnumeratePaths calls GetDirectories, then again when I call GetFiles.

So my questions are:

1. Is there anything overtly inefficient in the implementation as it stands (not related to the file system problems mentioned).

2. How can I best reduce the memory bloat and so on when processing folders containing huge numbers of files?

---

Update 1: Directory.EnumerateFiles(path, "*", SearchOption.AllDirectories)

As MarcinJuraszek pointed out there is already a way to do full-depth file enumeration using Directory.EnumerateFiles that doesn't require me to fetch a full array of results in one hit. That's good because one of the folders I'm scanning has 150,000+ files in it (don't ask).

Here's some code I wrote to test it, using a drive mapped to a network share:

Stopwatch sw = new Stopwatch();
for (int i = 0; i < 10; i++)
{
    sw.Restart();
    var enumfiles = Directory.EnumerateFiles("I:\\", "*", SearchOption.AllDirectories);
    int maxlen = enumfiles.Max(fname => fname.Length);
    sw.Stop();
    Console.WriteLine("{0}: {1:0.00}s", i, sw.ElapsedMilliseconds / 1000.0);
}

The average time over 10 runs was 73.942 seconds. Substituting the EnumerateFiles method above gave an average of 72.264 seconds - close enough that network traffic variations could account for the difference. As for memory usage... Directory.EnumerateFiles() gives a peak working set (from Process.GetCurrentProcess().PeakWorkingSet64) of 35,896,786 bytes after 10 runs, compared to 80,711,680 bytes - a significant improvement.

Still feels like there is a lot of room for improvement though.

---

Update 2: FindFirstFile() API

After much googling and testing I managed to get the kernel32 FindFirstFile() API working in C#, hoping that it would provide a bit more of a boost. For a few of my projects it might do, since it gives me all sorts of useful information in the WIN32_FIND_DATA structure during enumeration... but the overall performance when just enumerating file name is approximately equivalent to Directory.EnumerateFiles, both in time and memory usage.

Thanks Marcin - answer accepted.

---

Update 3: FindFirstFile revisited

Final update. Using some of the code from P/Invoke I've rewritten my FindFile enumeration code as follows:

public static IEnumerable<string> EnumerateFiles(string path)
{
    WIN32_FIND_DATA finddata;
    Queue<string> paths = new Queue<string>();
    paths.Enqueue(path);

    while (paths.Count > 0)
    {
        var nxtpath = paths.Dequeue();

        using (var fh = FindFirstFile(Path.Combine(nxtpath, "*"), out finddata))
        {
            if (fh.IsInvalid)
                continue;
            bool ok = true;
            while (ok)
            {
                if ((finddata.dwFileAttributes & FileAttributes.Directory) == FileAttributes.Directory)
                {
                    if (finddata.cFileName != "." && finddata.cFileName != "..")
                        paths.Enqueue(Path.Combine(nxtpath, finddata.cFileName));
                }
                else
                    yield return Path.Combine(nxtpath, finddata.cFileName);

                ok = FindNextFile(fh, out finddata);
            }
        }
    }
}

This performs almost twice as fast as Directory.EnumerateFiles in my test case - network mapped drive containing numerous files and folders including one folder with over 150,000 files. I think this one is going to end up in my library :)

Answer 1

Why don't you use Directory.EnumerateFiles and Directory.EnumerateDirectories methods? There is an overload which takes SearchOption enum and lets you query subdirectiries too.

static IEnumerable<string> EnumerateFiles(string root)
{
    return Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories);
}

static IEnumerable<string> EnumeratePaths(string root)
{
    return Directory.EnumerateDirectories(root, "*", SearchOption.AllDirectories);
}

You can see, that GetDirectories and GetFiles uses the same logic as Enumerate* methods, just instantiate a list, instead of returning one element at the time.