Timeline for Scrapy spider for products on a site
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 10, 2020 at 13:24 | history | edited | CommunityBot |
Commonmark migration
|
|
| Oct 24, 2015 at 8:10 | comment | added | 301_Moved_Permanently |
@trendsetter37 Also I have the feeling that the PRODUCTS_TYPES collection should be built out of a query to avoid the need of maintaining it when the site adds new categories for instance…
|
|
| Oct 24, 2015 at 8:03 | comment | added | 301_Moved_Permanently | @trendsetter37 That makes sense. I added url rules to filter out pages that obviously are not products. You can extend it at will to get as close as 100% products requests ratio as possible. | |
| Oct 24, 2015 at 7:57 | history | edited | 301_Moved_Permanently | CC BY-SA 3.0 |
Included url rules
|
| Oct 24, 2015 at 7:47 | history | edited | 301_Moved_Permanently | CC BY-SA 3.0 |
Improved the draft a bit
|
| Oct 24, 2015 at 0:48 | comment | added | trendsetter37 | Also in regards to checking the page type (product or category) I tried to determine page type on the front end before it is passed through to the parse function. One of their grading parameters dealt with the ratio of scrapy.Requests to parsed items. I was under the impression that catching it before hand, i.e. using the regex for the sitemap url rules, would help cut down on initial Requests. I am not sure that I actually accomplished what was intended though. Realistically I managed to scrape 438 products out of 490 requests. | |
| Oct 24, 2015 at 0:32 | comment | added | trendsetter37 | Wow thank you @Mathias. I guess I still have a lot to learn. Never seen @staticmethod decorators before. Also your other points mentioned in your answer make sense. | |
| Oct 23, 2015 at 23:24 | history | answered | 301_Moved_Permanently | CC BY-SA 3.0 |