Searching of Word documents

Question

I have a website that offers to search documents from local hearings conducted, stored on a network file server. I need to take in the search term and search a bunch of .docx (roughly 4500) files. They are not large, < 150 kb mostly, but it runs very slow downloading the files into the stream. I'm sure there's a better way to write the search, (perhaps multi processing) but I don't know how to tune it up and speed the search up. The search itself is taking over 3 minutes.

bool found = false;
Hearing h = new Hearing();
Stream str = null;
MemoryStream str2 = new MemoryStream();
HttpWebRequest fileRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse fileResponse = (HttpWebResponse)fileRequest.GetResponse();
str = fileResponse.GetResponseStream();
str.CopyTo(str2);
str2.Position = 0;
using (WordprocessingDocument wpd = WordprocessingDocument.Open(str2, true))
{
    string docText = null;
    using (StreamReader sr = new StreamReader(wpd.MainDocumentPart.GetStream()))
    {
        docText = sr.ReadToEnd();
        found = docText.ToUpper().Contains(txtBasicSearch.Text.ToUpper());
        if (found)
        {
            hearingArrayList.Add(h);
            foundCount++;
        }
    }
}

Answer 1

This is really the exact use-case for indexed full-text search engines.

Since you're running this code server-side on a website, I'd suggest you seriously consider writing a simple worker that polls your FS for new documents and adds them to a database that has full text searching enabled.

If you're using SQL Server: https://docs.microsoft.com/en-us/sql/relational-databases/search/get-started-with-full-text-search

If you're using MySQL: http://www.w3resource.com/mysql/mysql-full-text-search-functions.php

This way, you'd not only get your results back much more quickly than scanning each document manually, you'd also avoid the onerous network traffic involved in streaming every file from the FS for every request.

To do this, you can pretty easily either write a page in your site or a new console app (preferable) that is called by a cron job (linux) or scheduled task (windows) on the server every so often. That interval would be however often you expect there to be new documents added to the FS or whatever your tolerance is for stale data.

At that point, the page/app would pull the list of documents already cached in the database, query the FS for its contents, and compare the lists of filenames or file dates to see what needs to be added/updated. At that point, you can only stream in the files you actually need to add and you don't really care how long it takes.

The database would then take care of the indexing of the new documents. Your webpage then becomes a dumb pipe for searching those indexed documents.

---

If storing the texts in a database isn't an option, you might consider mirroring the files on your own server. It'd still remove the slowest part of your algorithm (the network traffic).

You'd still need your cron/scheduled task worker to do that mirroring but it'd be a simple matter of copying the new files from the FS to your local disk.

---

If you mirror locally or can't do either, your best bet is parallelization. You can do some refactoring but your local operations aren't your real bottleneck.

For instance, if you can mirror locally, you could use this in place of your existing code:

// ToUpper() your search string outside of the loop,
// rather than in each passs.
string txtBasicSearch = "My Search String".ToUpper();

// Use Parallel.ForEach over every docx file in our directory. 
Parallel.ForEach(Directory.EnumerateFiles(directoryPath, "*.docx"), (string file) =>
{
    string docText = string.Empty;

    try
    {
        // Try to dispose of our streams as soon as possible to avoid
        // holding memory unecessarily. Also, avoid copying Streams
        // to different types. A generic Stream works just fine.
        //
        // As well, only open with read perms to avoid unecessary locks and
        // any delays that may cause.
        using (Stream str = File.OpenRead(file))
        {
            using (WordprocessingDocument wpd = WordprocessingDocument.Open(str, false))
            {
                using (StreamReader sr = new StreamReader(wpd.MainDocumentPart.GetStream()))
                {
                    docText = sr.ReadToEnd();
                }
            }
        }

        // Search the haystack for the needle.
        if (docText.ToUpper().Contains(txtBasicSearch))
        {
            // No need for a counter variable. Just user
            // hearingArrayList.Count() at the end.
            hearingArrayList.Add(file);
        }
    }
    catch (Exception ex)
    {
        // Do whatever error handling here.
        return;
    }
});

Timing that parallel version against the same version using a regular foreach loop with a small directory on my local NAS showed that the parallel version was typically 3-6 times faster.

If you can't mirror locally, you can still parallelize the file streaming but will need to be cognizant of the limits the server may place on the number of connections you can open at once.

The HttpClient class will probably serve you better here than the WebRequest class. https://msdn.microsoft.com/en-us/library/hh696703(v=vs.110).aspx

There, you can query the files within the remote directory then iterate through them, making an async call with HttpClient.

So, that might look like:

string txtBasicSearch = "My Search String".ToUpper();

HttpClient client = new HttpClient();

// Use client to populate myFileList with the remote files.

foreach (string file in myFileList)
{
    client.GetStreamAsync(file).ContinueWith((Task<Stream> result) =>
    {
        if (result.Status != TaskStatus.RanToCompletion)
        {
            // Error handling.
            return;
        }

        string docText = string.Empty;

        try
        {
            using (WordprocessingDocument wpd = WordprocessingDocument.Open(result.Result, false))
            {
                using (StreamReader sr = new StreamReader(wpd.MainDocumentPart.GetStream()))
                {
                    docText = sr.ReadToEnd();
                }
            }

            if (docText.ToUpper().Contains(txtBasicSearch))
            {
                hearingArrayList.Add(file);
            }
        }
        catch (Exception ex)
        {
            // Do whatever error handling here.
            return;
        }
    }
}

The HttpClient class will take care of rate limiting you. By default, I believe it allows three connections at any time, but you can easily change that to your liking.

Enumerating the files on the remote server would be a different topic, depending on how that remote is being accessed. I'd suggest searching other SO answers like https://stackoverflow.com/questions/124492/c-sharp-httpwebrequest-command-to-get-directory-listing

(If your file server is just a NAS on your intranet, save yourself the pain an just use the System.IO.Directory and .File classes to query the files)