A faster way to look up for a url in a WebRequest/WebResponse html

Question

What I'm trying to do here is get an RSS feed and append an Enclosure XML node to each item, which has a link to a video file (wmv).

Try the below code with

url = "http://www.microsoft.com/events/series/digitalblackbelt.aspx?tab=rss"

for eg to get the point

The performance hit here consists of two parts: the LONG "foreach" loop, along with the HUGE number of requests done to retrieve the media link (marked in the code by the comment "This part needs attention"). Any advice concerning how to get the media link a much faster manner would really be appreciated!

You can get a glimpse of what that code do, by comparing this feed (the URL):

http://www.microsoft.com/events/series/digitalblackbelt.aspx?tab=rss

to this feed (created by the code below), note that it will be slow and might give error, if it did just refresh the page:

http://mshady.apphb.com/feeds/index?url=http://www.microsoft.com/events/series/digitalblackbelt.aspx?tab=rss

public class FeedsController : Controller
{
    private XmlDocument _Xml;

    [OutputCache(VaryByParam = "url", Duration=86400)]    
    public string Index(string url)
    {
        string feed = "";

        _Xml = new XmlDocument();
        _Xml.Load(url);

        foreach (XmlNode node in _Xml.ChildNodes[0].ChildNodes[0].ChildNodes)
        {
            if (node.Name == "item")
            {
                XmlNode childnode = GetEnclosureXmlNode(node);
                node.AppendChild(childnode);
            }
        }

        feed =  _Xml.OuterXml;

        return feed;
    }

    private string GetMediaUrl(string articleUrl)
    {
        string mediaUrl = "";

        ////////////////////////////////////////////    
        // This part needs attention
        ////////////////////////////////////////////    
        Regex regex = new Regex(@"<a href=""(.*?)"">WMV Download</a>");
        var request = HttpWebRequest.Create(articleUrl);
        var response = request.GetResponse();
        string responseHtml = new StreamReader(response.GetResponseStream()).ReadToEnd();
        var match = regex.Match(responseHtml);
        ////////////////////////////////////////////    
        // This part needs attention
        ////////////////////////////////////////////    


        if (match.Success)
            mediaUrl = match.Groups[1].Value;

        return mediaUrl;
    }

    private XmlNode GetEnclosureXmlNode(XmlNode node)
    {
        string articleUrl = node["link"].InnerText;
        string mediaUrl = GetMediaUrl(articleUrl);

        XmlElement childnode = CreateEnclosureXmlNode(mediaUrl);
        return childnode;
    }

    private XmlElement CreateEnclosureXmlNode(string mediaUrl)
    {
        XmlElement childnode = _Xml.CreateElement("enclosure");
        childnode.SetAttribute("url", mediaUrl);
        childnode.SetAttribute("type", "video/wmv");
        return childnode;
    }
}

Answer 1

I would bet that the regex.Match is the performance problem, as Jeff Atwood describes here.

One thing that I would do is move regex definition to a static variable outside of the function defined as below:

static Regex regex = new Regex(@"<a href=""(.*?)"">WMV Download</a>", RegexOptions.Compiled);

That's the only real thing that jumps out at me. Try taking a look at this blog post for more ideas and information.

Should give you something to try until guys with more experience glance their eyes over it :)

Answer 2

I'm not sure there is too great a performance hit on the foreach loop, but you could isolate out all the items with xpath first, such as:

var itemNodes = _Xml.SelectNodes(@"//channel/item");
foreach (XmlNode node in itemNodes)
{
    XmlNode childnode = GetEnclosureXmlNode(node);
    node.AppendChild(childnode);
}

The problem I see is the time to go and bring back a page not knowing if it will even have the media link you're looking for. If there is no other way than to get each rss url, bring back the bits and then look for the media file url, then adding some form of multi-threading to this and processing several at a time should cut down on the overall length. Finding the balance of how many at one time is then the bottleneck.