3
\$\begingroup\$

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/
This project is for personal learning & practice. The following Spider works as expected; however, it is very, very slow. It took several minutes to complete:

import scrapy


class DedeyneSpider(scrapy.Spider):
    name = 'dedeyne'

    custom_settings = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0',
         'FEEDS': {'dedeyne.csv': {'format': 'csv', 'overwrite': True}}
    }

    allowed_domains = ['apotheekdedeyne.be']
    url = 'https://apotheekdedeyne.be/pharmareserve/?per_page={}'


    def start_requests(self):
        for page in range(1, 2365):
            yield scrapy.Request(url=self.url.format(page))

    def parse(self, response):
        links = response.css('.productbox').xpath('.//a[2]/@href').getall()
        for link in links:
            yield scrapy.Request(
                url=response.urljoin(link),
                callback=self.parse_product_page,
            )

    def parse_product_page(self, response):
        product = response.xpath('//h2[@id="prod_title"]/text()').get()
        price = response.css('.price-new::text').get()
        cnk = response.xpath('//div').re_first(r'CNK\s\d*')

        if response.xpath('//div[@class="model"]'):
            manufacturer = response.xpath('//div[@class="model"]/a/text()').get()
        else:
            manufacturer = 'None'

        yield {
            'product': product,
            'price': price,
            'cnk': cnk,
            'manufacturer': manufacturer
        }


Question: Is there a more efficient way to write this spider? Perhaps parse all product_links asynchronously then parsing_product_page, or any other way? If so, would you kindly demonstrate with a code example ~ Thanks.

\$\endgroup\$
3
  • 1
    \$\begingroup\$ Scraping is slow. Was that "several minutes" to scrape 2300+ pages? If so I'd say it's doubtful you're going to get better than that \$\endgroup\$
    – Reinderien
    Jan 17, 2023 at 2:20
  • \$\begingroup\$ I just thought Scrapy could scrape a lot faster than this spider performed. However, I do appreciate your feedback! \$\endgroup\$
    – Seraph776
    Jan 17, 2023 at 3:07
  • 1
    \$\begingroup\$ Check out the grequests library. GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily. \$\endgroup\$
    – iohans
    Jan 20, 2023 at 15:43

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.