This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper?

import scrapy
from scrapy.crawler import CrawlerProcess
from time import time

def isAscii(s):
    return all(ord(c) < 128 for c in s)

region = input('Region? ')
start = int(input('Start page? '))
end = int(input('End page? '))

startTime = time()

class LolSpider(scrapy.Spider):
    name = 'lolspider'
    start_urls = ['http://lolprofile.net/leaderboards/{}/{}'.format(region, page) for page in range(start, end)]
    def parse(self, response):
        for name in response.xpath('//span[not(@class)]/text()').extract():
            with open('lolusernames.txt', 'a') as f:
                if (' ' not in name) and (name != 'LoL') and isAscii(name):

process = CrawlerProcess({})


endTime = time()
print('Completed in {:.2f} seconds'.format(endTime-startTime))

My main goal is to be as fast as possible and that's why I used scrapy. If anyone knows of a faster alternative I'd love to hear it.


1 Answer 1


The main problem is that you are writing/appending to the file inside the spider's parse() method - this is not how this should be done in Scrapy - there is a special place - Item Pipelines. Check this answer to see how to include a custom pipeline when running Scrapy from a script.

Note that in the pipeline you should not be appending to a file, but instead collecting the data in the process_item() method and dumping it when the spider is finished in spider_closed() method. You may also do the writing in chunks to avoid potential memory problems on large output data sizes.

Other things to improve:

  • \$\begingroup\$ If I did that wouldn't memory leaks happen if I really had a huge amount of data? \$\endgroup\$
    – edsheeran
    Jan 18, 2017 at 21:32
  • \$\begingroup\$ @edsheeran yeah, in this case, you may write to file in chunks. I'd experiment with different variations and see how it affects the overall speed. But, do it in a pipeline. \$\endgroup\$
    – alecxe
    Jan 18, 2017 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.