Searching for files across multiple code projects

Question

I was tasked with conducting a large number of text searches in the code files of multiple projects. Initially, I just used the Find in Files feature of Visual Studio in a couple of the running instances, but then I realized I needed to search other solutions as well. So I quickly created an F# Script (below) to conduct the searches across all code files async.

For all matching results, the script prints:

C:\path-to-file
linenumber: line of code

How can this be improved? The script runs extremely fast and did its job well. Now, out of curiosity, I would like to know what improvements could be made to optimize performance?

open System
open System.IO

let directory = @"C:\path-to-top-level-directory" 

// this just keeps async operations from writing to the console concurrently (garbled result)
let agent = MailboxProcessor.Start(fun inbox -> 
    // the message processing function
    let rec messageLoop () = async {
        let! msg = inbox.Receive()
        printfn "%s" msg
        return! messageLoop()
    }
    // start the loop
    messageLoop ()
)

let checkIfContains value file = async {
    use reader = new StreamReader(File.OpenRead(file))
    let lineNum = ref -1
    let postResult (i, s) = sprintf "%s\r\n%i: %s\r\n" file i s |> agent.Post

    seq { while not reader.EndOfStream do 
            incr lineNum 
            yield (!lineNum, reader.ReadLine()) } 
        |> Seq.filter (fun (_, s) -> s.Contains(value))
        |> Seq.iter postResult
}

let files = 
    Directory.EnumerateFiles(directory, "*.vb", SearchOption.AllDirectories)
        |> Seq.append (Directory.EnumerateFiles(directory, "*.cs", SearchOption.AllDirectories))

while true do
    agent.Post "Search string: "
    let value = Console.ReadLine()
    agent.Post "Searching...\r\n"

    files
        |> Seq.map (checkIfContains value)
        |> Async.Parallel
        |> Async.RunSynchronously
        |> ignore

    agent.Post "\r\nDone.\r\n\r\n"

Answer 1

For a base line, I modified your script so that it is not dominated by printing to the console:

open System
open System.IO

let directory = @"C:\Dev" 

// this just keeps async operations from writing to the console concurrently (garbled result)
let agent = MailboxProcessor.Start(fun inbox -> 
    // the message processing function
    let count = ref 0
    let rec messageLoop () = async {
        let! msg = inbox.Receive()
        count :=  !count + 1
        return! messageLoop()
    }
    // start the loop
    messageLoop ()
)

let checkIfContains value file = async {
    use reader = new StreamReader(File.OpenRead(file))
    let lineNum = ref -1
    let postResult (i, s) = sprintf "%s\r\n%i: %s\r\n" file i s |> agent.Post

    seq { while not reader.EndOfStream do 
            incr lineNum 
            yield (!lineNum, reader.ReadLine()) } 
        |> Seq.filter (fun (_, s) -> s.Contains(value))
        |> Seq.iter postResult
}

let files = 
   Directory.EnumerateFiles(directory, "*.cs", SearchOption.AllDirectories)

let value = "papa"

files
    |> Seq.map (checkIfContains value)
    |> Async.Parallel
    |> Async.RunSynchronously
    |> ignore


agent.Post "\r\nDone.\r\n\r\n"

This takes ~10 seconds on my machine. Like Carsten said, there's no need for a MailboxProcessor. Removing it takes me down to ~7.5 seconds. Next, there's no need for Async. There is no ! in that async comp expression. We could use Seq.AsParallel instead (in System.Linq namespace). Down to ~6.3 secs.

Now this task is dominated by IO, and AsParallel shouldn't help here. In fact, removing it yields the same result, ~6.3 secs.

Now, that seq inside of checkIfContains looks wasteful. Getting rid of that yields ~6.1 secs, so no big win.

I wouldn't consider this idiomatic code. I just took out some unused stuff, and wound up with this:

open System
open System.IO
open System.Linq
let directory = @"C:\Dev"
let count = ref 0 
let print s = lock count (fun () -> count := !count + 1)

let checkIfContains value file = 
    use reader = new StreamReader(File.OpenRead(file))
    let lineNum = ref -1
    let postResult i s = sprintf "%s\r\n%i: %s\r\n" file i s |> print
    while not reader.EndOfStream do 
      incr lineNum 
      let line = reader.ReadLine()
      if line.Contains(value) then postResult !lineNum line

let files = Directory.EnumerateFiles(directory, "*.cs", SearchOption.AllDirectories)
let value = "papa"
let f = files
              |> Seq.map (checkIfContains value)
              |> Seq.iter id