Scrapy Username Spider

Question

This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper?

import scrapy
from scrapy.crawler import CrawlerProcess
from time import time

def isAscii(s):
    return all(ord(c) < 128 for c in s)

region = input('Region? ')
start = int(input('Start page? '))
end = int(input('End page? '))

startTime = time()

class LolSpider(scrapy.Spider):
    name = 'lolspider'
    start_urls = ['http://lolprofile.net/leaderboards/{}/{}'.format(region, page) for page in range(start, end)]
    def parse(self, response):
        for name in response.xpath('//span[not(@class)]/text()').extract():
            with open('lolusernames.txt', 'a') as f:
                if (' ' not in name) and (name != 'LoL') and isAscii(name):
                    f.write(name+'\n')


process = CrawlerProcess({})

process.crawl(LolSpider)
process.start()

endTime = time()
print('Completed in {:.2f} seconds'.format(endTime-startTime))

My main goal is to be as fast as possible and that's why I used scrapy. If anyone knows of a faster alternative I'd love to hear it.

Answer 1

The main problem is that you are writing/appending to the file inside the spider's parse() method - this is not how this should be done in Scrapy - there is a special place - Item Pipelines. Check this answer to see how to include a custom pipeline when running Scrapy from a script.

Note that in the pipeline you should not be appending to a file, but instead collecting the data in the process_item() method and dumping it when the spider is finished in spider_closed() method. You may also do the writing in chunks to avoid potential memory problems on large output data sizes.

Other things to improve:

• adjust/increase the number of concurrent requests
• turn the logging off
• enable HTTP cache
• Scrapy running on PyPy?