Questions tagged [scrapy]

Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Filter by
Sorted by
Tagged with
2
votes
0answers
25 views

Use Scrapy to find duplicate images across pages

I'm trying to use scrapy to find image URLs that are used more than once on a website across all pages. This is my spider: ...
4
votes
1answer
86 views

Sourcing data format from multiple different structures

Problem I want to read in the data to dictionary person = { 'name': 'John Doe', 'email': 'johndoe@email.com', 'age': 50, 'connected': False } ...
2
votes
2answers
489 views

Creating a csv file using scrapy

I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
5
votes
3answers
4k views

Writing to a csv file in a customized way using scrapy

I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
5
votes
0answers
534 views

Parsing different categories using scrapy from a webpage

I've written a script in python scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach the target page to scrape the ...
2
votes
0answers
122 views

Handling IndexError using lambda function within scrapy

I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
6
votes
1answer
152 views

Locating contact information from a CSV spreadsheet

I'm trying to find a more effective way to access variables within for loops. I felt that my attempt which works is not an effective way to access variables that ...
4
votes
1answer
77 views

Webscraping from yellowbook

I'm fairly good at programming python but of course i want to write cleaner code. This is a simple script that scrapes leads off yellowbook. I want to utilize list comprehensions but im comfortable ...
5
votes
1answer
5k views

Parsing a slow-loading webpage with scrapy in combination with selenium

I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible ...
1
vote
1answer
843 views

Extracting certain products from a webpage using Scrapy

I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
6
votes
3answers
650 views

Scraping a webpage copying with the logic of scrapy

Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
2
votes
1answer
3k views

Scrapy crawler to parse data recursively

I've written a script in python scrapy to parse "name" and "price" of different products from a website. Firstly, it scrapes the links of different categories from the upper sided bar located in the ...
4
votes
0answers
305 views

Crawling thesaurus for a synonym

I have just finished core functionality for Tehsaurux. It crawls three thesauruses and will find a random synonym for you. It works with PHP agnostic and Scrapy for Python. I think the two main ...
5
votes
1answer
120 views

News site scraping

I'm developing scrapy project for scraping news from sites and saving them to Google Sheets. Source repository is here. I want to ask about my class that must be ...
5
votes
1answer
105 views

Check the stock of references list

I did this spider that checks the stock of references list (around 3500 references). Now the spider takes around 37 seconds to scrap 400 references. The CPU is around 5%, the network card (1Gbps) ...
2
votes
1answer
135 views

Scrapy Username Spider

This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper? <...
3
votes
1answer
2k views

Scraping restaurant names and addresses from a website using Scrapy and Selenium

I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
5
votes
1answer
449 views

Scraping names of directors from a website

I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated. The reason I have a ...
17
votes
1answer
12k views

Scraping after login using Scrapy

I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session. It retrieves the title of these pages ...
2
votes
2answers
2k views

Scrapy spider for products on a site

I recently submitted a code sample for a web scraping project and was rejected without feedback as to what they didn't like. The prompt, while I cannot give it here verbatim, basically stated that I ...
3
votes
2answers
3k views

Check duplicate items in Mongodb

I need to query MongoDB and check if an item exists or not: ...
3
votes
1answer
1k views

Why is my web scraping script so slow?

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0); This is the whole http://phpfiddle.org/main/code/9qt-78n I think the problem is here: ...
3
votes
2answers
869 views

Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries

The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those. I ...
2
votes
1answer
2k views
7
votes
1answer
4k views

How to optimize a pipline for storing item in mysql?

I am using scrapy framework for data scraping and dumping item in MySQL database. Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...
3
votes
2answers
3k views

Default initialization of items using Scrapy [closed]

I am writing a scraper, using Scrapy, in which I need to populate items with default values at first. Could my method be improved? ...
2
votes
2answers
1k views

Data extraction

I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome. ...