Parsing different categories using Scrapy from a webpage

Question

I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the required info.

The below scraper first starts from the main page then track each links within class art-indexhmenu then going to one layer deep it again tracks the links within class niveau2 then again follow the links within class niveau3 then tracking the links within class art-indexbutton-wrapper it reaches the target page. Then it scrapes "model", "country" and "years" of each products.

My scraper is doing its job errorlessly. However, although it is working nice, the way I've created this scraper is very repetitive to look at. As there are always room for improvement, I suppose there should be any way to make it more robust by getting rid of banality.

This is the spider (website included):

import scrapy
from scrapy.spiders import CrawlSpider
from scrapy.http.request import Request
from scrapy.crawler import CrawlerProcess

class BikePartsSpider(scrapy.Spider):
    name = 'honda'

    def start_requests(self):
        yield Request(url = "https://www.bike-parts-honda.com/", callback = self.parse_links)

    def parse_links(self, response):
        for link in response.css('.art-indexhmenu a::attr(href)').extract():
            yield response.follow(link, callback = self.parse_inner_links) #going to one layer deep from landing page

    def parse_inner_links(self, response):
        for link in response.css('.niveau2 .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_cat_links) # digging deep to go another layer

    def parse_cat_links(self, response):
        for link in response.css('.niveau3 .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_target_links) ## go inside another layer

    def parse_target_links(self, response):
        for link in response.css('.art-indexbutton-wrapper .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_docs) # tracking links leading to the target page

    def parse_docs(self, response):
        items = [item for item in response.css('.titre_12_red::text').extract()]
        yield {"categories":items} #this is where the scraper parses the info

c = CrawlerProcess({               #using CrawlerProcess() method to be able to run from the IDE
    'USER_AGENT': 'Mozilla/5.0',   
})
c.crawl(BikePartsSpider)
c.start()

Answer 1

this scraper is very repetitive

Yeah, I get that motivation. I'm glad that's a thing you're worried about.

Offhand? No, I don't see any obvious improvements to the code.

But I will observe that bs4 offers recursive methods such as find_all.

There's more than four levels of "let's parse at one level deeper" in this codebase. I suspect that recursing down, in a single function call, would go a long way toward tightening up the source code.