Extracting certain products from a webpage using Scrapy

Question

I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work with Scrapy and parse some data from a web, the csv output looks awkward (if the csv is derived from default command, as in scrapy crawl toscrapesp -o items.csv -t csv). The results found in such CSV file are with a uniform gap between two lines that means there is a line gap between each two rows. I've fixed it using the below script. I didn't use default command to get the CSV output; rather, I've written few lines of code in spider class and got the desired output.

Although It is running smoothly, I'm not sure it is the ideal way of doing such thing. I expect someone to give any suggestion as to how I can improve this script.

"items.py" includes:

import scrapy

class ToscrapeItem(scrapy.Item):
    Name = scrapy.Field()
    Price = scrapy.Field()

Spider contains:

import csv
import scrapy

outfile = open("various_pro.csv", "w", newline='')
writer = csv.writer(outfile)

class ToscrapeSpider(scrapy.Spider):

    name = "toscrapesp"
    start_urls = ["http://books.toscrape.com/"]

    def parse(self, response):
        for link in response.css('.nav-list a::attr(href)').extract():
            yield scrapy.Request(url=response.urljoin(link), callback=self.collect_data)

    def collect_data(self, response):
        global writer                                  
        for item in response.css('.product_pod'):
            product = item.css('h3 a::text').extract_first()
            value = item.css('.price_color::text').extract_first()
            yield {'Name': product, 'Price': value}  
            writer.writerow([product,value])

Please click this link to see what I was having earlier. Upon executing the script, I get CSV output with no line gap or blank rows.

Answer 1

I don't think you should reinvent the wheel and provide your own CSV export. The following works for me as is (note the addition of .strip() calls - though I don't think they are necessary at all):

import scrapy


class ToscrapeSpider(scrapy.Spider):
    name = "toscrapesp"
    start_urls = ["http://books.toscrape.com/"]

    def parse(self, response):
        for link in response.css('.nav-list a::attr(href)').extract():
            yield scrapy.Request(url=response.urljoin(link), callback=self.collect_data)

    def collect_data(self, response):
        for item in response.css('.product_pod'):
            product = item.css('h3 a::text').extract_first().strip()
            value = item.css('.price_color::text').extract_first().strip()
            yield {'Name': product, 'Price': value}  

Running it with scrapy runspider spider.py -o output.csv -t csv produces a CSV file with no blank lines:

Price,Name
£53.74,Tipping the Velvet
£29.69,Forever and Forever: The ...
£55.53,A Flight of Arrows ...
£36.95,The House by the ...
£30.25,Mrs. Houdini
£28.08,The Marriage of Opposites  
...