Parse and extract synonym data from a website

Question

I made a method that finds synonyms to words using thesaurus.com and I'm looking for comments and feedback to it. In what ways can I improve it, both when it comes to speed, security, reliability (regardless of how "reliable" it is to rely on a third-party website for look-ups), etc.

    /// <summary>
    /// This method relies heavily on thesaurus.com for synonym lookups. It is not completely reliable, but is deemed reliable enough in instances where you dont have your own thesaurus
    /// </summary>
    public static string[] GetSynonyms(string word)
    {
        string url = string.Format("http://thesaurus.com/search?q={0}", word);

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        if (response.StatusCode == HttpStatusCode.OK)
        {
            List<string> synonyms = new List<string>();
            StringBuilder data = new StringBuilder();
            string line;

            using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            {

                //we know that the synonyms is in the upper-part of the html stream so we do not want to read the entire stream.
                while((line = reader.ReadLine()) != null) {
                    var index = line.IndexOf("<span class=\"text\">");

                    if(index > 0) 
                    {
                        index = index + "<span class=\"text\">".Length;
                        synonyms.Add(line.Substring(index).Replace("</span>", ""));
                    }

                    //break when we come to the Antonyms section of the page
                    if (line.Contains("container-info antonyms"))
                    {
                        break;
                    }
                }
            }
            return synonyms.ToArray<string>();
        }
        else
        {
            return null;
        }
    }

Edit: As an example, it now takes about 3.5 seconds to find the synonyms for the word "old".

Answer 1

You're doing things the hard way. By that I mean, why build a web crawler for this? Is it a requirement that you only use thesaurus.com? There are several quality thesaurus services out there with APIs that you could utilize.

I've never used any of them, but a quick web search and a bit of sifting results and these ones appear to be at the head of the thesaurus API ballgame.

• Altervista Thesaurus: Thesaurus is a web service providing search capability for synonyms in different languages. The synonyms are retrieved from the thesauri dictionaries of OpenOffice according to the relevant licenses.
• Big Huge Thesaurus: This site sports a very simple API for retrieving the synonyms for any word. You can access the API by making a GET request...

I also found another Stack Exchange post on the topic, which lists some more options: https://webapps.stackexchange.com/questions/9156/database-of-english-synonyms

If thesaurus.com is absolutely a requirement, then you're doing the best you can. You have to send the request, crawl the page, extract the information you want, and format it into something you can use.

thesaurus.com does have an API, but from how they describe it, its difficult to get them to give you access. You can read more about that here:

thesaurus.com API