Fastest way searching specific files

Question

I got thousands of files with a specific file extension in thousands of sub folders. Now, what is the fastest way to search with a pattern? I tried the method DirectoryInfo.GetFiles(rootfolder) (~8 minutes) and a recursive custom method (~5 minutes).

 private void WalkDirectoryTree(DirectoryInfo dr, string searchname)
    {
        System.IO.FileInfo[] files = null;
        System.IO.DirectoryInfo[] subDirs = null;
        try
        {
            files = dr.GetFiles(searchname + ".*");
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message);
        }

        if (files != null)
        {
            foreach (FileInfo fi in files)
            {
                allFiles.Add(fi);
            }
            subDirs = dr.GetDirectories();

            foreach (DirectoryInfo di in subDirs)
            {
                WalkDirectoryTree(di, searchname);
            }
        }
    }

Is there any faster way to do it?

Answer 1

Try parallelization. Instead of:

foreach (DirectoryInfo di in subDirs)
{
    WalkDirectoryTree(di, searchname);
}

Do

Parallel.ForEach(subDirs, dir => WalkDirectoryTree(dir, searchname));

Notice that by doing this allFiles will be accessed concurrently so change your collection to a ConcurrentBag.

Answer 2

You can just use DirectoryInfo.EnumerateFiles() method which returns an IEnumerable<FileInfo> and therefor if you access them by the enumerator they will be evaluated when they are accessed.

private void WalkDirectoryTree(DirectoryInfo dr, string searchname)
{
    foreach (FileInfo file in FindFiles(dr, searchname + ".*"))
    {
        // process file
        allFiles.Add(file);
    }
}
public IEnumerable<FileInfo> FindFiles(DirectoryInfo startDirectory, string pattern)
{
     return startDirectory.EnumerateFiles(pattern, SearchOption.AllDirectories);
}