I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I've tried to handle the list index out of error
issue using a customized function within scrapy. There might be better ways to do so but What I noticed with the one I've tried that when there are several fields to scrape then this customized function play a vital role to deal with those fields whose values are none. I'll be vary glad to have any better ideas to do the same. Thanks in advance.
Here is what I've tried:
import scrapy
from scrapy.crawler import CrawlerProcess
class CraigsspSpider(scrapy.Spider):
name = 'craigssp'
start_urls = ['http://bangalore.craigslist.co.in/search/rea?s=120']
#the below function is able to handle IndexError
get_result = lambda item,path:item.css(path).extract_first() if item.css(path) else ""
def parse(self, response):
for items in response.css(".result-info"):
date = CraigsspSpider.get_result(items,".result-date::text")
name = CraigsspSpider.get_result(items,".hdrlnk::text")
price = CraigsspSpider.get_result(items,".result-price::text")
yield{'Date':date,'Name':name,'Price':price}
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(CraigsspSpider)
c.start()