Web crawler that uses Task Parallel Library

Question

This is the first time using the Task Parallel Library and it seems to be performing much slower than I was expecting. The application is simply a web crawler that returns all URLs for a given URL.

Here is my method that, depending on the level param, will either download single page links (getSinglePageLinks()) or it will make use of my threaded method (getManyPageLinks()):

public static IEnumerable<string> getLinks(string url, bool hostMatch=true, bool validatePages=true, int level=0)
    {
        string formattedUrl = urlFormatValidator(url);
        if (string.IsNullOrEmpty(formattedUrl)) return Enumerable.Empty<string>();
        //download root url's
        IEnumerable<string> rootUrls = getSinglePageLinks(formattedUrl, hostMatch, validatePages);
        //download url's for each level
        for (int i=0; i<level; i++)
        {
            rootUrls = rootUrls.Union(getManyPageLinks(rootUrls, hostMatch, validatePages));
        }
        return rootUrls;
    }

getSinglePageLinks() makes use of the HTML agility pack, and it simply downloads and parses the given URL:

private static IEnumerable<string> getSinglePageLinks(string formattedUrl, bool hostMatch = true, bool validatePages = true)
    {
        try
        {
            HtmlDocument doc = new HtmlWeb().Load(formattedUrl);
            var linkedPages = doc.DocumentNode.Descendants("a")
                                              .Select(a => a.GetAttributeValue("href", null))
                                              .Where(u => !String.IsNullOrEmpty(u))
                                              .Distinct();

             //hostMatch and validatePages left out
             return linkedPages;
        }catch(...){...}

And my getManyPageLinks():

   private static IEnumerable<string> getManyPageLinks(IEnumerable<string> rootUrls, bool hostMatch, bool validatePages)
    {
        List<Task> tasks = new List<Task>();
        List<List<string>> allLinks = new List<List<string>>();

        foreach (string rootUrl in rootUrls)
        { 
            string rootUrlCopy = rootUrl; //required
            var task = Task.Factory.StartNew(() =>
            {
                IEnumerable<string> taskResult = getSinglePageLinks(rootUrlCopy, hostMatch, validatePages);
                return taskResult;
            });

            tasks.Add(task);
            allLinks.Add(task.Result.ToList());
        }

        Task.WaitAll(tasks.ToArray());
        return allLinks.SelectMany(x => x).Distinct();            
    }

The app works OK if the level is set to 0, but if I set the level to 1 so it gets all links for all of the roots URL, the CPU/Network usage does not go above 1-3%. How can I improve performance?

Answer 1

From MSDN on Task.Wait:

If the current task has not started execution, the Wait method attempts to remove the task from the scheduler and execute it inline on the current thread.

I have a feeling that something similar might be happening with WaitAll, killing performances. I'll have a look and see if I can find documentation about it. As you are using StartNew some of the tasks might be started already so they won't be in-lined.

I would refactor it using async/await so you are sure you are not using and blocking only one thread from the pool:

async static Task<IEnumerable<string>> GetAllPagesLinks(IEnumerable<string> rootUrls, bool hostMatch, bool validatePages)
{
    var result = await Task.WhenAll(rootUrls.Select(url => GetPageLinks(url, hostMatch, validatePages)));

    return result.SelectMany(x => x).Distinct();
}

static async Task<IEnumerable<string>> GetPageLinks(string formattedUrl, bool hostMatch = true, bool validatePages = true)
{
    var htmlDocument = new HtmlDocument();

    try
    {
        using (var client = new HttpClient())
            htmlDocument.Load(await client.GetStringAsync(formattedUrl));

        return htmlDocument.DocumentNode
                           .Descendants("a")
                           .Select(a => a.GetAttributeValue("href", null))
                           .Where(u => !string.IsNullOrEmpty(u))
                           .Distinct();
    }
    catch
    {
        return Enumerable.Empty<string>();
    }
}

async static Task<IEnumerable<string>> GetLinks(string url, bool hostMatch = true, bool validatePages = true, int level = 0)
{
    if (level < 0)
        throw new ArgumentOutOfRangeException(nameof(level));

    string formattedUrl = FormatAndValidateUrl(url);

    if (string.IsNullOrEmpty(formattedUrl))
        return Enumerable.Empty<string>();

    var rootUrls = await GetPageLinks(formattedUrl, hostMatch, validatePages);

    if (level == 0)
        return rootUrls;

    var links = await GetAllPagesLinks(rootUrls, hostMatch, validatePages);

    var tasks = await Task.WhenAll(links.Select(link => GetLinks(link, hostMatch, validatePages, --level)));

    return tasks.SelectMany(l => l);
}

Haven't had a chance to test it, but just get the gist.

Answer 2

Blocking code in a loop

You are starting your tasks asynchronously..

Task.Factory.StartNew(()

only to block synchronously inside the loop..

allLinks.Add(task.Result.ToList());

resulting in sequentially starting and awaiting the intermediate results for each cycle in the loop.

    foreach (string rootUrl in rootUrls)
    { 
        // .. code omitted

        // start an asynchronous task
        var task = Task.Factory.StartNew(() =>
        {
            // .. code omitted
        });

        tasks.Add(task);

        // BOTTLE-NECK:
        // synchronously await its result before starting the next task
        allLinks.Add(task.Result.ToList());
    }

    // super fast! but only because you have already awaited all results synchronously
    Task.WaitAll(tasks.ToArray());

    return allLinks.SelectMany(x => x).Distinct(); 

You could solve this by extracting the line allLinks.Add(task.Result.ToList()); from the loop and returning return tasks.SelectMany(x => x.Result).Distinct(); .

private static IEnumerable<string> getManyPageLinks(IEnumerable<string> rootUrls, bool hostMatch, bool validatePages)
{
    var tasks = new List<Task>();

    foreach (var rootUrl in rootUrls)
    { 
        string rootUrlCaptured = rootUrl;
        var task = Task.Run(() =>
        {
            var taskResult = getSinglePageLinks(rootUrlCaptured, hostMatch, validatePages);
            return taskResult;
        });

        tasks.Add(task);
    }

    Task.WaitAll(tasks.ToArray());
    return tasks.SelectMany(task => task.Result).Distinct();            
}