# Parsing different categories using Scrapy from a webpage

I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the required info.

The below scraper first starts from the main page then track each links within class art-indexhmenu then going to one layer deep it again tracks the links within class niveau2 then again follow the links within class niveau3 then tracking the links within class art-indexbutton-wrapper it reaches the target page. Then it scrapes "model", "country" and "years" of each products.

My scraper is doing its job errorlessly. However, although it is working nice, the way I've created this scraper is very repetitive to look at. As there are always room for improvement, I suppose there should be any way to make it more robust by getting rid of banality.

This is the spider (website included):

import scrapy
from scrapy.spiders import CrawlSpider
from scrapy.http.request import Request
from scrapy.crawler import CrawlerProcess

class BikePartsSpider(scrapy.Spider):
name = 'honda'

def start_requests(self):
yield Request(url = "https://www.bike-parts-honda.com/", callback = self.parse_links)

yield response.follow(link, callback = self.parse_inner_links) #going to one layer deep from landing page

yield response.follow(link, callback = self.parse_cat_links) # digging deep to go another layer

def parse_docs(self, response):
items = [item for item in response.css('.titre_12_red::text').extract()]
yield {"categories":items} #this is where the scraper parses the info

c = CrawlerProcess({               #using CrawlerProcess() method to be able to run from the IDE
'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(BikePartsSpider)
c.start()

• "My scraper is doing it's job errorlessly." In how much time, usually? And what's the expected output? – Mast Apr 11 '18 at 15:20
• Do you pipe the output into something useful with a secondary program by chance? – Mast Apr 11 '18 at 16:00
• In case of output, the categories defined within my scraper is enough @Mast. I'm not worried about output. The thing is I wish to know any better way other than what i did above cause it looks so repetitive. – SIM Apr 11 '18 at 17:21
• But you don't appear to do anything with those categories. Is that correct? – Mast Apr 11 '18 at 17:25
• Yep, right you are. I can see the results in the IDE and I checked whether the output I'm having is accurate. Btw, why the output is so important here as I've stated in the first place that i would like to go for any better design. Thanks. – SIM Apr 11 '18 at 17:29