1
\$\begingroup\$

I made a method that finds synonyms to words using thesaurus.com and I'm looking for comments and feedback to it. In what ways can I improve it, both when it comes to speed, security, reliability (regardless of how "reliable" it is to rely on a third-party website for look-ups), etc.

    /// <summary>
    /// This method relies heavily on thesaurus.com for synonym lookups. It is not completely reliable, but is deemed reliable enough in instances where you dont have your own thesaurus
    /// </summary>
    public static string[] GetSynonyms(string word)
    {
        string url = string.Format("http://thesaurus.com/search?q={0}", word);

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        if (response.StatusCode == HttpStatusCode.OK)
        {
            List<string> synonyms = new List<string>();
            StringBuilder data = new StringBuilder();
            string line;

            using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            {

                //we know that the synonyms is in the upper-part of the html stream so we do not want to read the entire stream.
                while((line = reader.ReadLine()) != null) {
                    var index = line.IndexOf("<span class=\"text\">");

                    if(index > 0) 
                    {
                        index = index + "<span class=\"text\">".Length;
                        synonyms.Add(line.Substring(index).Replace("</span>", ""));
                    }

                    //break when we come to the Antonyms section of the page
                    if (line.Contains("container-info antonyms"))
                    {
                        break;
                    }
                }
            }
            return synonyms.ToArray<string>();
        }
        else
        {
            return null;
        }
    }

Edit: As an example, it now takes about 3.5 seconds to find the synonyms for the word "old".

\$\endgroup\$
1
  • \$\begingroup\$ I believe that HttpWebRequest gets the entire response, so a possible improvement would be to use sockets to read the data directly and then stop when finished. \$\endgroup\$
    – jmoreno
    Oct 17, 2013 at 5:18

1 Answer 1

3
\$\begingroup\$

You're doing things the hard way. By that I mean, why build a web crawler for this? Is it a requirement that you only use thesaurus.com? There are several quality thesaurus services out there with APIs that you could utilize.

I've never used any of them, but a quick web search and a bit of sifting results and these ones appear to be at the head of the thesaurus API ballgame.

  • Altervista Thesaurus: Thesaurus is a web service providing search capability for synonyms in different languages. The synonyms are retrieved from the thesauri dictionaries of OpenOffice according to the relevant licenses.
  • Big Huge Thesaurus: This site sports a very simple API for retrieving the synonyms for any word. You can access the API by making a GET request...

I also found another Stack Exchange post on the topic, which lists some more options: https://webapps.stackexchange.com/questions/9156/database-of-english-synonyms

If thesaurus.com is absolutely a requirement, then you're doing the best you can. You have to send the request, crawl the page, extract the information you want, and format it into something you can use.

thesaurus.com does have an API, but from how they describe it, its difficult to get them to give you access. You can read more about that here:

thesaurus.com API

\$\endgroup\$
1
  • \$\begingroup\$ It was more of a fun thing to do to create a thesaurus/dictionary API without my own data source, and it will most likely never be published. @jmoreno suggested using sockets as a possible improvement but I might leave it as is seeing that a webscraper is greatly inferior to a real thesaurus API \$\endgroup\$
    – Max
    Oct 18, 2013 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.