I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the required info.

The below scraper first starts from the main page then track each links within class art-indexhmenu then going to one layer deep it again tracks the links within class niveau2 then again follow the links within class niveau3 then tracking the links within class art-indexbutton-wrapper it reaches the target page. Then it scrapes "model", "country" and "years" of each products.

My scraper is doing its job errorlessly. However, although it is working nice, the way I've created this scraper is very repetitive to look at. As there are always room for improvement, I suppose there should be any way to make it more robust by getting rid of banality.

This is the spider (website included):

import scrapy
from scrapy.spiders import CrawlSpider
from scrapy.http.request import Request
from scrapy.crawler import CrawlerProcess

class BikePartsSpider(scrapy.Spider):
    name = 'honda'

    def start_requests(self):
        yield Request(url = "https://www.bike-parts-honda.com/", callback = self.parse_links)

    def parse_links(self, response):
        for link in response.css('.art-indexhmenu a::attr(href)').extract():
            yield response.follow(link, callback = self.parse_inner_links) #going to one layer deep from landing page

    def parse_inner_links(self, response):
        for link in response.css('.niveau2 .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_cat_links) # digging deep to go another layer

    def parse_cat_links(self, response):
        for link in response.css('.niveau3 .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_target_links) ## go inside another layer

    def parse_target_links(self, response):
        for link in response.css('.art-indexbutton-wrapper .art-indexbutton::attr(href)').extract():
            yield response.follow(link, callback = self.parse_docs) # tracking links leading to the target page

    def parse_docs(self, response):
        items = [item for item in response.css('.titre_12_red::text').extract()]
        yield {"categories":items} #this is where the scraper parses the info

c = CrawlerProcess({               #using CrawlerProcess() method to be able to run from the IDE
    'USER_AGENT': 'Mozilla/5.0',   
  • \$\begingroup\$ "My scraper is doing it's job errorlessly." In how much time, usually? And what's the expected output? \$\endgroup\$
    – Mast
    Commented Apr 11, 2018 at 15:20
  • \$\begingroup\$ Do you pipe the output into something useful with a secondary program by chance? \$\endgroup\$
    – Mast
    Commented Apr 11, 2018 at 16:00
  • \$\begingroup\$ In case of output, the categories defined within my scraper is enough @Mast. I'm not worried about output. The thing is I wish to know any better way other than what i did above cause it looks so repetitive. \$\endgroup\$
    – SIM
    Commented Apr 11, 2018 at 17:21
  • \$\begingroup\$ But you don't appear to do anything with those categories. Is that correct? \$\endgroup\$
    – Mast
    Commented Apr 11, 2018 at 17:25
  • \$\begingroup\$ Yep, right you are. I can see the results in the IDE and I checked whether the output I'm having is accurate. Btw, why the output is so important here as I've stated in the first place that i would like to go for any better design. Thanks. \$\endgroup\$
    – SIM
    Commented Apr 11, 2018 at 17:29

1 Answer 1


this scraper is very repetitive

Yeah, I get that motivation. I'm glad that's a thing you're worried about.

Offhand? No, I don't see any obvious improvements to the code.

But I will observe that bs4 offers recursive methods such as find_all.

There's more than four levels of "let's parse at one level deeper" in this codebase. I suspect that recursing down, in a single function call, would go a long way toward tightening up the source code.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.