Recursive filename search

Question

I have created a little program to search a set of folders holding documents scanned.

The folder structure is as follows:

c:\images\year\month\date\documenttype\firstpartofdocumentNo.\

the year folder contains years from 2005 - 2015

the month folder contains the months of the year (Obviously) same with date

the documenttype folder can contain between 1 and 5 folders

the firstpartofdocumentno. can contain between 1 and 3 folders

The code I am using at the moment is:

CompName = Environment.MachineName

TicketNo = TxtTicketNo.Text

If CompName = "Comp1" Then
    ImageDir = "C:\Images\"
Else
    ImageDir = "\\Comp2\Images\"
End If

For Each DirYear As String In Directory.GetDirectories(ImageDir)
    Dim YearInfo As New DirectoryInfo(DirYear)

    For Each DirMonth As String In Directory.GetDirectories(DirYear)
        Dim MonthInfo As New DirectoryInfo(DirMonth)

        For Each DirDate As String In Directory.GetDirectories(DirMonth)
            Dim DateInfo As New DirectoryInfo(DirDate)

            For Each DirType As String In Directory.GetDirectories(DirDate)
                Dim TypeInfo As New DirectoryInfo(DirType)

                For Each DirStart As String In Directory.GetDirectories(DirType)
                    Dim StartInfo As New DirectoryInfo(DirStart)

                    MainDirectory = ImageDir & YearInfo.Name & "\" & MonthInfo.Name & "\" & DateInfo.Name & "\" & TypeInfo.Name & "\" & StartInfo.Name & "\"
                                                                                            For Each Ticket As String In Directory.GetFiles(MainDirectory, TicketNo & "*")
                        LstFiles.Items.Add(Ticket)
                    Next

                Next

             Next

        Next
    Next
   Next

I have a textbox on the form which is used to enter the last four numbers of the ticketno and then this code runs when the button is clicked.

The problem is it can take up to five minutes to search, so I was wondering if there is a way to optimize this code to speed it up a bit or does this sound about right for searching that many folders.

Answer 1

This might sound like a goofy answer, but if you already have Windows' file indexing service set up and running on your system, you can search the database contents of the index results using simple SQL terminology.

Here is an article that shows you how

In this way, you're not traversing the directory tree each time you search, but rather you're letting Windows Indexing Service log each new file when it is entered, and then searching the resulting database of information.

Answer 2

I've been looking at this question for a while and I don't see many opportunities to improve performance. Nested loops 6 levels deep is going to perform poorly. It doesn't seem that there is any case in which you can return early either.

Not all hope is lost though. Have you considered using a Parallel.ForEach? From the documentation:

A ForEach loop works like a For loop. The source collection is partitioned and the work is scheduled on multiple threads based on the system environment. The more processors on the system, the faster the parallel method runs. For some source collections, a sequential loop may be faster, depending on the size of the source, and the kind of work being performed.

I wouldn't go crazy, because the number of threads it will spin up is limited, but it seems reasonable to me to run each year in parallel. It's possible you'll see a good boost.

Answer 3

CompName = Environment.MachineName

TicketNo = TxtTicketNo.Text

If CompName = "Comp1" Then
    ImageDir = "C:\Images\"
Else
    ImageDir = "\\Comp2\Images\"
End If

This snippet has been bugging me to no end already since you're making two quite problematic assumptions here:

1. Comp1's name will always remain the same
2. You assume having network access to a Comp2, which exposes an Images directory for "public" access.

The latter can't be helped much. But things like these paths should be done with application settings. The resulting code would possibly look somewhat cleaner:

CompName = Environment.MachineName

TicketNo = TxtTicketNo.Text

If CompName = My.Settings.MachineName Then
    ImageDir = My.Settings.LocalDir
Else
    ImageDir = My.Settings.NetworkDir
End If

I find this code being somewhat clearer and cleaner as well as easier to maintain. Especially when you want to stay open for simple modifications like moving the Images between hosts.

Answer 4

I'm wondering why the existing .net classes won't work? IO.DirectoryInfo.EnumerateFiles has an AllDirectories search option that should do the trick.

CompName = Environment.MachineName

TicketNo = TxtTicketNo.Text

If CompName = "Comp1" Then
    ImageDir = "C:\Images\"
Else
    ImageDir = "\\Comp2\Images\"
End If 



Dim LstFiles = New IO.DirectoryInfo(ImageDir) _
                .EnumerateFiles(TicketNo + "*", IO.SearchOption.AllDirectories)

Answer 5

I had a similar problem finding MP3 files in a similar folder hierarchy.

My solution was to use recursion as opposed to iteration, which made for cleaner looking code. I just rewrote it to refresh my memory.. Scanning an entire drive containing 15,300 directories and 240,000 files for filenames containing the string "ab" (5500 matches) took 2.75 seconds. I've tried searching for files using indexing.. I don't understand WHY, but this is FAR faster

Imports System.IO

Public Class Form1
Dim MatchingList As New List(Of String)
Dim TotalDirs As Integer = 0
Dim TotalFiles As Integer = 0

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load


End Sub

Public Sub Start()
    Dim startpath As String = "f:\"
    RecurseDirectories(startpath, "ab")
End Sub

Public Sub RecurseDirectories(ByVal Path As String, ByVal SearchString As String)
    TotalDirs += 1
    Try
        For Each Dir As String In IO.Directory.EnumerateDirectories(Path)
            RecurseDirectories(Dir, SearchString)
        Next
    Catch ex As Exception
        'exception handling here
    End Try

    For Each File As String In IO.Directory.EnumerateFiles(Path)
        TotalFiles += 1
        CheckFiles(File, SearchString)
    Next
End Sub

Public Sub CheckFiles(ByVal FilePath As String, ByVal SearchString As String)
    If IO.Path.GetFileName(FilePath).Contains(SearchString) Then
        MatchingList.Add(FilePath)
    End If
End Sub

Public Sub ShowMatches()
    If IsNothing(MatchingList) Or MatchingList.Count = 0 Then
        Debug.WriteLine("No matches found")
        Exit Sub
    End If

    For Each s As String In MatchingList
        Debug.WriteLine(s)
    Next
End Sub

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
    Dim starttime As Date = Now
    Button1.Enabled = False
    Start()
    Button1.Enabled = True
    Dim endtime As Date = Now
    ShowMatches()

    Label1.Text = New TimeSpan(endtime.Ticks - starttime.Ticks).TotalSeconds.ToString
    Debug.WriteLine("Directories searched = " & TotalDirs & vbTab & "Files searched = " & TotalFiles)
    Debug.WriteLine("found " & MatchingList.Count & " filenames containing search string")
End Sub
End Class