4
\$\begingroup\$

I have a multithreaded application for web scraping from an automobile website. While performing web scraping, there are many links that gives the same result, so I have to check for data redundancy.

I have maintained a BlockingCollection<String> that adds vehicle VIN number in it and performs a check if link to be scraped has VIN number in the list. If it doesn't have it, then the application can continue to scrap data from that link.

This is the code used to perform data extraction. It takes arguments as cities links and state name.

private void CityCarListExtraction(string statename, List<CityLinkMap> cities)
    {
        Notify(new DataArgs { Datum = "--- Data Extraction Start ---" });

        Action[] tsk = new Action[cities.Count];
        for (int ij = 0; ij < tsk.Length; ij++)
        {
            var city = cities[ij];
            tsk[ij] = new Action(() =>
            {
                try
                {
                    #region data
                    string cityName = city.Link.Substring(0, city.Link.IndexOf(".auto")).Replace("http://", "");
                    File.Delete(Path.Combine(Helper.DirectoryPath, cityName + ".txt"));

                    ///Request Html
                    string dataHtml = GetHtml(city.Link);
                    if (String.IsNullOrEmpty(dataHtml))
                        throw new ArgumentException("Remote Server returned empty response");
                    Notify(new DataArgs { Datum = "Extract Url : " + city.Link });

                    ///Load Html
                    HtmlDocument hd = new HtmlDocument();
                    hd.LoadHtml(dataHtml);


                    ///Get Total Pages and Url
                    var htmlNode = hd.DocumentNode.SelectNodes("//html[1]").FirstOrDefault();
                    int totalPage = 0;
                    string detailUrl = "";
                    if (htmlNode != null)
                    {
                        var pagerNode = htmlNode.Descendants().Where(x => x.Name == "div"
                            && (x.HasAttributes && x.Attributes["class"] != null && x.Attributes["class"].Value == "pagination"))
                            .FirstOrDefault();

                        if (pagerNode != null)
                        {
                            var pagesClass = pagerNode.Descendants().Where(x => x.Name == "div" && x.HasAttributes
                                                                                        && x.Attributes["class"] != null
                                                                                        && x.Attributes["class"].Value == "pages").FirstOrDefault();
                            if (pagesClass != null)
                            {
                                var localarr = pagesClass.Descendants().Where(x => x.Name == "a" && x.HasAttributes).ToList();
                                if (localarr.Count > 0)
                                {
                                    var nextLinkNode = localarr.FirstOrDefault(x => x.HasAttributes
                                                                && x.Attributes["class"] != null && x.Attributes["class"].Value == "next_page");
                                    if (nextLinkNode != null)
                                        localarr.Remove(nextLinkNode);
                                    totalPage = city.DataLimit > 0 ? Convert.ToInt32(Math.Ceiling((decimal)city.DataLimit / cAutoSitePageSize)) : Convert.ToInt32(localarr.Last().InnerHtml);
                                    detailUrl = city.Link + localarr.Last().Attributes["href"].Value.Split('=')[0] + "=";
                                }
                            }
                        }
                    }

                    Parallel.For(1, totalPage, new ParallelOptions { MaxDegreeOfParallelism = cMaxTask }, (i) =>
                    {
                        try
                        {
                            HtmlNode localPageHtml = null;
                            if (i > 0)
                            {
                                string detailHtml = GetHtml(detailUrl + i);
                                if (String.IsNullOrEmpty(detailHtml))
                                    throw new ArgumentException("Remote Server returned empty response");
                                HtmlDocument hloc = new HtmlDocument();
                                hloc.LoadHtml(detailHtml);
                                localPageHtml = hloc.DocumentNode.SelectNodes("//html[1]").FirstOrDefault();
                            }
                            else
                                localPageHtml = htmlNode;

                            var liNodes = localPageHtml.Descendants().Where(x => x.Name == "li"
                                                                    && x.Attributes["class"] != null
                                                                    && x.Attributes["class"].Value == "car");
                            if (liNodes != null)
                            {
                                var hrefs = liNodes.Select(x =>
                                                    x.SelectNodes(x.XPath + "//a[@class='summary']") == null ? "" :
                                                x.SelectNodes(x.XPath + "//a[@class='summary']")
                                            .FirstOrDefault()
                                         .Attributes["href"].Value.Trim()
                                     );

                                List<String> links = new List<string>();
                                foreach (var link in hrefs.Where(x => !String.IsNullOrEmpty(x)))
                                {
                                    string vin = link.Substring(link.LastIndexOf("-") + 1);

                                    if (!VinNoCollection.Contains(vin))
                                    {
                                        VinNoCollection.Add(vin);
                                        links.Add(link);
                                    }

                                    //else
                                    //{
                                    //    VinNoCollectionDUP.Add(vin);
                                    //}
                                }
                                ScrapData(statename, cityName, links.ToArray());
                            }
                        }
                        catch (Exception ex)
                        {
                            Notify(new DataArgs { Datum = "City Car List Page Extraction : " + ex.ToString(), IsError = true });
                        }
                    });
                    #endregion
                }
                catch (Exception ex)
                {
                    Notify(new DataArgs { Datum = "City Car List Extraction : " + ex.ToString(), IsError = true });
                }
            });
        }

        Parallel.Invoke(new ParallelOptions { MaxDegreeOfParallelism = cCityTask }, tsk);
    }

In the above function, I have to check for the VIN number existence in the list VinNoCollection. The problem is that for every vehicle, the relevant link is added to this list. There are near about 3 million vehicle records on the website that has to be extracted, which will increase the size of the list to millions as well.

Is this the right way to do this ? Can this code be optimized more ?

\$\endgroup\$
0
1
\$\begingroup\$

What do you ultimately intend to do with this data?

It may not be ideal to keep it all in memory. You may want to use an actual database, which will index and cache the data in an efficient way. Since you're doing things with tasks, the extra overhead of going to the database and back won't hurt you much.

Also I have a suggested refactor. Instead of building a separate array of tasks, use PLINQ's Parallel.ForEach():

    Parallel.ForEach(cities, new ParallelOptions { MaxDegreeOfParallelism = cCityTask }, city => ... );

You don't have to specifically construct the Action; the lambda expression already does that.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.