Find the newest file that contains a specific string in every subdirectory of the path using the least resources

Question

I have a server side logs folder that contains many hundreds of logs most of which are in subdirectories according to the machine the logs have come from. The task is to extract the full path of the latest file in each directory where the file contains a particular string (not all files have this string) so that analysis can be done per machine. I have included my attempt below but it seems rather clunky and long-winded and I wonder if there is an easier/better/faster/more efficient way of doing this maybe with linq?

void Main()
{
    string SourcePath = @"L:\machinelogs";
    string filemask = "*.log";
    string searchitem = @"cannot access server data";
    
    List<string> fileswithsearchitem = new List<string>();
    
    DirectoryInfo directory = new DirectoryInfo(SourcePath);
    IEnumerable<DirectoryInfo> dirs = directory.EnumerateDirectories("*",new EnumerationOptions() { RecurseSubdirectories = true, IgnoreInaccessible = true });
    dirs.Append(directory);
    foreach (var dir in dirs)
    {
        var found = false;
        var files = dir.EnumerateFiles(filemask);
        foreach(var file in files.OrderByDescending(f => f.CreationTime).ToList())
        {
            foreach (var line in File.ReadLines(file.FullName))
            {
                if(line.Contains(searchitem))
                {
                    fileswithsearchitem.Add(file.FullName + " : " + line);
                    found = true;
                    break;
                }
            }
            if(found)
            {
                break;
            }
        }
    }
    
    foreach (string item in fileswithsearchitem)
    {
        Console.WriteLine(item);
    }
}
```

Answer 1

• Use the proper Coding Conventions such as camel casing your local variables.
• When working with IEnumerable, try to Enumerate and not to Create unless you need to.
• try to give your variables a better naming that would be easy to understand and memorize.
• use SearchOption when searching in directories.
• use StringComparison when comparing strings.

Here is a revised version :

void Main()
{
    string directoryPath = @"L:\machinelogs";
    string searchPattern = "*.log";
    string searchTerm = @"cannot access server data";
    
    List<string> filesWithSearchResults = new List<string>();
    
    var directory = new DirectoryInfo(directoryPath);

    foreach (var file in directory.EnumerateFiles(searchPattern, SearchOption.AllDirectories)
        .OrderByDescending(x => x.CreationTime))
    {
        foreach (var line in File.ReadLines(file.FullName))
        {
            if (line.Contains(searchTerm, StringComparison.OrdinalIgnoreCase))
            {
                var result = $"{file.FullName} : {line}";       
                filesWithSearchResults.Add(result);
                Console.WriteLine(result);
            
            }
        }
    }
}

you could expand your enumeration by using a method such as :

public IEnumerable<string> SearchInLogFiles(string directoryPath, string searchTerm, SearchOption searchOption)
{
    if (string.IsNullOrWhiteSpace(directoryPath))
        throw new ArgumentNullException(nameof(directoryPath));

    if (string.IsNullOrWhiteSpace(searchTerm))
        throw new ArgumentNullException(nameof(searchTerm));

    if (!Directory.Exists(directoryPath)) yield break;          

    var directory = new DirectoryInfo(directoryPath);

    foreach (var file in directory.EnumerateFiles("*.log", searchOption)
        .OrderByDescending(x => x.CreationTime))
    {
        foreach (var line in File.ReadLines(file.FullName))
        {
            if (line.Contains(searchTerm, StringComparison.OrdinalIgnoreCase))
            {
                yield return $"{file.FullName} : {line}";
            }
        }
    }

}

Now you can do this :

var filesWithSearchResults = SearchInLogFiles(directoryPath, searchTerm, SearchOption.AllDirectories).ToList()

or you could enumerate it (print the results) :

foreach(var item in SearchInLogFiles(directoryPath, searchTerm, SearchOption.AllDirectories))
{
    Console.WriteLine(item);
}

if you only need the full path of the file that contains the search term, then you could do this :

var filesWithSearchResults = directory.EnumerateFiles(searchPattern, SearchOption.AllDirectories)
    .OrderByDescending(x=> x.CreationTime)
    .Where(x=> File.ReadLines(x.FullName).Contains(searchTerm, StringComparer.OrdinalIgnoreCase))
    .Select(x=> x.FullName)
    .ToList();

Answer 2

iSR5 answer has some great points (+1). Another thing you could try using PLINQ. Depending on how many directories exits could make a difference but would need to test it.

From MS Introduction to PLINQ

Measuring PLINQ Performance

In many cases, a query can be parallelized, but the overhead of setting up the parallel query outweighs the performance benefit gained. If a query does not perform much computation or if the data source is small, a PLINQ query may be slower than a sequential LINQ to Objects query.

Microsoft even has example of iterating over file directories with PLINQ

I personally prefer the method syntax over the query syntax of LINQ. An example of method syntax using PLINQ with your question would look like

public static IEnumerable<string> SearchFiles(string directoryPath, string fileMask, string searchItem)
{
    var directory = new DirectoryInfo(directoryPath);
    return directory.EnumerateDirectories("*", new EnumerationOptions()
        {
            RecurseSubdirectories = true,
            IgnoreInaccessible = true
        })
        .AsParallel() // Search directories in parallel
        .Select(dirInfo => dirInfo.EnumerateFiles(fileMask)
            .OrderByDescending(fileInfo => fileInfo.CreationTime)
            .Select(fileInfo => fileInfo.FullName)
            .FirstOrDefault(fileName => File.ReadLines(fileName).Any(line => line.Contains(searchItem))))
        .Where(fileName => fileName != null);
}


static void Main(string[] args)
{
    foreach (var file in SearchFiles(@"L:\machinelogs", "*.log", @"cannot access server data"))
    {
        Console.WriteLine(file);
    }
}