Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

learn more… | top users | synonyms

3
votes
1answer
192 views

Why is my web scraping script so slow?

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0); This is the whole http://phpfiddle.org/main/code/9qt-78n I think the problem is here: ...
3
votes
2answers
65 views

Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries

I understand the premise of OOP when shown in basic examples but I cannot get my head around using it in actual programs/applications. I have tried here, but to me it seems to not achieve anything ...
1
vote
1answer
275 views
5
votes
1answer
619 views

How to optimize a pipline for storing item in mysql?

I am using scrapy framework for data scraping and dumping item in MySQL database. Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...
2
votes
2answers
326 views

python / scrapy default initialization of items

i am writing a scraper, using scrapy , in which i need to populate item with default values at first , is there any good way to populate items with default values other then my code logic ...
2
votes
2answers
683 views

python / scrapy code optimization

i wrote this code in python/ scrapy for data extraction it works fine for me but i am newbie in python so any suggestions for code optimization would be welcomed , and thanks in advance. ...