3
\$\begingroup\$

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/
This project is for personal learning & practice. The following Spider works as expected; however, it is very, very slow. It took several minutes to complete:

import scrapy


class DedeyneSpider(scrapy.Spider):
    name = 'dedeyne'

    custom_settings = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0',
         'FEEDS': {'dedeyne.csv': {'format': 'csv', 'overwrite': True}}
    }

    allowed_domains = ['apotheekdedeyne.be']
    url = 'https://apotheekdedeyne.be/pharmareserve/?per_page={}'


    def start_requests(self):
        for page in range(1, 2365):
            yield scrapy.Request(url=self.url.format(page))

    def parse(self, response):
        links = response.css('.productbox').xpath('.//a[2]/@href').getall()
        for link in links:
            yield scrapy.Request(
                url=response.urljoin(link),
                callback=self.parse_product_page,
            )

    def parse_product_page(self, response):
        product = response.xpath('//h2[@id="prod_title"]/text()').get()
        price = response.css('.price-new::text').get()
        cnk = response.xpath('//div').re_first(r'CNK\s\d*')

        if response.xpath('//div[@class="model"]'):
            manufacturer = response.xpath('//div[@class="model"]/a/text()').get()
        else:
            manufacturer = 'None'

        yield {
            'product': product,
            'price': price,
            'cnk': cnk,
            'manufacturer': manufacturer
        }

Question: Is there a more efficient way to write this spider? Perhaps parse all product_links asynchronously then parsing_product_page, or any other way? If so, would you kindly demonstrate with a code example ~ Thanks.

Improve this question
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Scraping is slow. Was that "several minutes" to scrape 2300+ pages? If so I'd say it's doubtful you're going to get better than that \$\endgroup\$
    – Reinderien
    Jan 17 at 2:20
  • \$\begingroup\$ I just thought Scrapy could scrape a lot faster than this spider performed. However, I do appreciate your feedback! \$\endgroup\$
    – Seraph776
    Jan 17 at 3:07
  • 1
    \$\begingroup\$ Check out the grequests library. GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. \$\endgroup\$
    – iohans
    Jan 20 at 15:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.