Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

learn more… | top users | synonyms

3
votes
1answer
336 views

Why is my web scraping script so slow?

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0); This is the whole http://phpfiddle.org/main/code/9qt-78n I think the problem is here: ...
3
votes
2answers
258 views

Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries

The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those. I ...
2
votes
1answer
709 views
6
votes
1answer
2k views

How to optimize a pipline for storing item in mysql?

I am using scrapy framework for data scraping and dumping item in MySQL database. Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...
3
votes
2answers
627 views

Default initialization of items using Scrapy

I am writing a scraper, using Scrapy, in which I need to populate items with default values at first. Could my method be improved? ...
2
votes
2answers
925 views

Data extraction

I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome. ...