Console application to search for files that contain a match for a regex

Question

I created a small console application to recursively search a folder and all subfolders for files that contain a match for a regex.

The command line arguments are the path you're looking to search followed by the regex (in .NET regex format).

I break this up into two threads: the main thread loops through the files and puts file data (their content, path, and name) on a concurrent queue. The second thread processes the queue, looking for regex matches.

Would using a Parallel.ForEach loop be better here? The main issue I had with this is that the queue is being modified while we're looping over it, but a Parallel.ForEach loop assumes that you already have a complete collection. The two possibilities I came up with this:

• Just read the content of the files, and then do a Parallel.ForEach loop to do the regex.
• Process it on the background thread like I'm doing until we're done reading the file contents, then do a Parallel.ForEach loop over any remaining items once we're done.

What I have here works but is slower than I'd like. I'd definitely like feedback on the multithreading I'm doing here, especially if one of the possibilities I mentioned above would be better than what I'm doing (or if there's another possibility that I don't mention above).

namespace SearchByRegex
{
class Program
{
    // I should probably just move this out to its own file
    private class FilePair
    {
        public string Content { get; set; }

        public string FilePath { get; set; }

        public string FileName { get; set; }
    }

    private static ConcurrentQueue<FilePair> fileInformation = new ConcurrentQueue<FilePair>();

    static void Main(string[] args)
    {
        Console.WriteLine("Working...");

        // Command lines are a path followed by the regular expression
        var regex = new Regex(args[1], RegexOptions.Multiline);

        // Process
        Thread thread = new Thread(PrintInformationOnMatches);
        thread.Start(regex);

        SearchDirectory(args[0]);            

        while (fileInformation.Any())
        {
            Thread.Sleep(1000);
        }

        thread.Abort();

        Console.WriteLine("Press any key to exit");
        Console.ReadLine();
    }

    /// <summary>
    /// Print information on all files whose content matches the <paramref name="regex"/>
    /// </summary>
    /// <param name="regex">Regex we're looking for</param>
    private static void PrintInformationOnMatches(object regex)
    {
        var castRegex = (Regex)regex;
        while (true)
        {
            // Process the entire queue
            while (fileInformation.Any())
            {
                FilePair pair;
                fileInformation.TryDequeue(out pair);

                if (castRegex.IsMatch(pair.Content))
                {
                    Console.WriteLine($"File name: {pair.FileName}. Path: {pair.FilePath}.");
                }
            }

            // Wait for more stuff on the queue
            Thread.Sleep(1000);
        }
    }

    /// <summary>
    /// Get the contents of the file and put them on the queue
    /// </summary>
    /// <param name="path">Path we are recursively searching</param>
    private static void SearchDirectory(string path)
    {
        string[] files = Directory.GetFiles(path);

        //Console.WriteLine("Path: " + path);
        foreach (string file in files)
        {
            fileInformation.Enqueue(new FilePair { Content = File.ReadAllText(file), FilePath = path, FileName = file });
        }

        foreach (string directory in Directory.GetDirectories(path))
        {
            SearchDirectory(directory);
        }
    }
}
}

Answer 1

private static void SearchDirectory(string path)

This method does not need to be recursive. You could use the Directory.EnumerateFiles method

Directory.EnumerateFiles Method (String, String, SearchOption)

where

One of the (SearchOption) enumeration values that specifies whether the search operation should include only the current directory or should include all subdirectories.

I don't know how your regex looks like but for simple pattern with wildcards you can use the second parameter

The search string to match against the names of files in path. This parameter can contain a combination of valid literal path and wildcard (* and ?) characters (see Remarks), but doesn't support regular expressions.

---

As far as the multithreading is concered you should definitely use the new TPL (Parallel.ForEach) instead of creating your own threads.

Answer 2

Cool question! In addition to t3chb0t's great comments, I would suggest using a BlockingCollection<T> instead of a ConcurrentQueue<T>. It has additional support for blocking and bounding so you don't need to poll the queue manually, and you can limit the number of "in-flight" files. This could be used to make sure you don't read too many files into memory at once.

private static readonly BlockingCollection<FilePair> FilePairs =
    new BlockingCollection<FilePair>(10); // Limited to 10 files being processed at once.

Once you have finished adding all the files to the collection, call FilePairs.CompleteAdding(), and from the other thread, your while loop becomes:

foreach (var pair in FilePairs.GetConsumingEnumerable())
{
    ...
}

Another thing is that you are performing blocking IO when you call File.ReadAllText(), and you may want to consider using an async equivalent instead, freeing up more threads as you are reading. Also Task.Run(() => PrintInformationOnMatches()) can be used instead of creating a Thread.

The TPL Dataflow library could be used to achieve something similar, or for a simpler approach that kicks off parallel tasks for each file, you could use something like this:

public static void Main(string[] args)
{
    var matchTasks = Directory.EnumerateFiles(args[0], "*", SearchOption.AllDirectories)
        .Select(async filePath =>
        {
            using (var reader = File.OpenText(filePath))
            {
                var fileContent = await reader.ReadToEndAsync();
                if (Regex.IsMatch(fileContent, args[1], RegexOptions.Multiline))
                    Console.WriteLine($"Found a match in {filePath}");
            }
        });

    Task.WaitAll(matchTasks.ToArray());
    Console.WriteLine("Done");
}