Timeline for Class to encapsulate and manage multiple background web crawlers
Current License: CC BY-SA 3.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 12, 2014 at 22:16 | history | edited | Jamal | CC BY-SA 3.0 |
deleted 7 characters in body; edited title
|
| Mar 12, 2014 at 22:13 | answer | added | Jeff Vanzella | timeline score: 6 | |
| Jul 17, 2013 at 2:10 | comment | added | Mouhong Lin | @almaz .NET 4.0. I crawl pages from one website, once a week. Each time it need to crawl 30,000 pages around. Cos the server will detect DOS attack. So I need to "slow down" the crawl speed. Yesterday I made some change to let it randomly pick a proxy server to crawl the page. But that's not important for the code review. Any feedback is welcome: scalability, robustness, code quality, etc :P Thanks | |
| Jul 16, 2013 at 21:45 | comment | added | almaz | @GarryVass my question was to original author | |
| Jul 16, 2013 at 20:52 | comment | added | Gayot Fow | @almaz, was that q for Mouhong or me? My stuff is 4.5 and uses async and Reactive Extensions and the Concurrent Collection namespace. The crawling strategy is implemented via polymorphism, but that's actually a separate question altogether :) | |
| Jul 16, 2013 at 20:42 | comment | added | almaz |
Is it a .NET 4.0 or 4.5 (async/await can reduce complexity of the code)? Do you crawl pages from one web site or multiple sites (affects how cool off period should be implemented)?
|
|
| Jul 16, 2013 at 16:26 | history | edited | Mouhong Lin | CC BY-SA 3.0 |
added 43 characters in body
|
| Jul 16, 2013 at 11:12 | history | edited | Mouhong Lin | CC BY-SA 3.0 |
added 233 characters in body
|
| Jul 16, 2013 at 9:36 | history | edited | Mouhong Lin | CC BY-SA 3.0 |
deleted 37 characters in body
|
| Jul 15, 2013 at 20:10 | history | tweeted | twitter.com/#!/StackCodeReview/status/356868432513351680 | ||
| Jul 15, 2013 at 18:40 | history | edited | Mouhong Lin | CC BY-SA 3.0 |
[Edit removed during grace period]
|
| Jul 15, 2013 at 18:35 | comment | added | Gayot Fow | I have several programs doing the same thing and also avoiding DOS detection on the server. For throttling, I use a Semaphore; and for downloading and sleeping I use a Task<T> instance that has a ContinueWith method. | |
| Jul 15, 2013 at 18:31 | history | edited | Mouhong Lin | CC BY-SA 3.0 |
added 65 characters in body
|
| Jul 15, 2013 at 18:09 | review | First posts | |||
| Jul 15, 2013 at 18:25 | |||||
| Jul 15, 2013 at 17:54 | history | asked | Mouhong Lin | CC BY-SA 3.0 |