Questions tagged [scrapy]

Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

Filter by
Sorted by
Tagged with
0 votes
1 answer
48 views

Save data in item wihitn Scrapy and Python

I need to store data in Scrapy Item in parts. To update only the information that I found at the moment. I did it, but the code looks too verbose. Three lines I repeat very often: ...
3 votes
0 answers
61 views

Gather product information from the Apotheek Dedeyne web site

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/ This project is for personal learning & ...
7 votes
1 answer
1k views

Parsing different categories using Scrapy from a webpage

I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach ...
1 vote
1 answer
72 views

Parsing an XML tree of categories

I have parse function which is parsing tree of categories. I've written it in simplest way possible and now struggling with refactoring it. Every nested loop is doing the same stuff but appending ...
5 votes
1 answer
129 views

Check the stock of references list

I did this spider that checks the stock of references list (around 3500 references). Now the spider takes around 37 seconds to scrape 400 references. The CPU is around 5%, the network card (1Gbps) ...
5 votes
2 answers
585 views

Crawling thesaurus for a synonym

I have just finished core functionality for Tehsaurux. It crawls three thesauruses and will find a random synonym for you. It works with PHP agnostic and Scrapy for Python. I think the two main files -...
4 votes
1 answer
948 views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
3 votes
0 answers
40 views

Scraping websites for presence of particular words

Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general ...
6 votes
1 answer
94 views

Website parser for advertisements

I'm new to Python and I have written the spider on the Scrapy framework which parses some advertisement websites and proceed data further. The spider works as I expect but I would like to show you the ...
3 votes
0 answers
332 views

Microservice for scraping images with celery

I made a project, that scrapes images asynchronously and saves them in container. I have access to them through volume. Scrapy finds images on given web page. Any tips will be good. But first I would ...
4 votes
1 answer
102 views

small web scraper to read product highlight from given urls

I have written a Python scraping program, using the Scrapy framework. The following code reads a list of urls that need to crawled and gets product highlights from the list. It also follows the next ...
3 votes
1 answer
146 views

Web Scraping Dynamically Generated Content Python

This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
1 vote
0 answers
154 views

Web scraping dynamic content

Hi I'm fairly new to coding and would appreciate some feedback on the code. This is for a site that has a dynamic login page and infinite scrolling. I wanted to use scrapy not necessarily because it ...
5 votes
3 answers
9k views

Writing to a csv file in a customized way using scrapy

I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
2 votes
0 answers
45 views

Use Scrapy to find duplicate images across pages

I'm trying to use scrapy to find image URLs that are used more than once on a website across all pages. This is my spider: ...
4 votes
1 answer
104 views

Sourcing data format from multiple different structures

Problem I want to read in the data to dictionary person = { 'name': 'John Doe', 'email': '[email protected]', 'age': 50, 'connected': False } ...
2 votes
2 answers
2k views

Creating a csv file using scrapy

I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
2 votes
0 answers
180 views

Handling IndexError using lambda function within scrapy

I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
6 votes
1 answer
172 views

Locating contact information from a CSV spreadsheet

I'm trying to find a more effective way to access variables within for loops. I felt that my attempt which works is not an effective way to access variables that ...
4 votes
1 answer
88 views

Webscraping from yellowbook

I'm fairly good at programming python but of course i want to write cleaner code. This is a simple script that scrapes leads off yellowbook. I want to utilize list comprehensions but im comfortable ...
5 votes
1 answer
8k views

Parsing a slow-loading webpage with scrapy in combination with selenium

I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible ...
1 vote
1 answer
1k views

Extracting certain products from a webpage using Scrapy

I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
3 votes
1 answer
3k views

Scraping restaurant names and addresses from a website using Scrapy and Selenium

I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
6 votes
3 answers
701 views

Scraping a webpage copying with the logic of scrapy

Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
2 votes
1 answer
5k views

Scrapy crawler to parse data recursively

I've written a script in python scrapy to parse "name" and "price" of different products from a website. Firstly, it scrapes the links of different categories from the upper sided bar located in the ...
5 votes
1 answer
141 views

News site scraping

I'm developing scrapy project for scraping news from sites and saving them to Google Sheets. Source repository is here. I want to ask about my class that must be ...
2 votes
1 answer
184 views

Scrapy Username Spider

This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper? <...
5 votes
1 answer
601 views

Scraping names of directors from a website

I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated. The reason I have a ...
17 votes
1 answer
15k views

Scraping after login using Scrapy

I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session. It retrieves the title of these pages ...
3 votes
2 answers
4k views

Check duplicate items in Mongodb

I need to query MongoDB and check if an item exists or not: ...
2 votes
2 answers
2k views

Scrapy spider for products on a site

I recently submitted a code sample for a web scraping project and was rejected without feedback as to what they didn't like. The prompt, while I cannot give it here verbatim, basically stated that I ...
3 votes
2 answers
1k views

Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries

The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those. I ...
2 votes
2 answers
1k views

Data extraction

I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome. ...
3 votes
2 answers
4k views

Default initialization of items using Scrapy [closed]

I am writing a scraper, using Scrapy, in which I need to populate items with default values at first. Could my method be improved? ...
3 votes
1 answer
1k views

Why is my web scraping script so slow?

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0); This is the whole http://phpfiddle.org/main/code/9qt-78n I think the problem is here: ...
2 votes
1 answer
2k views

Scrapy -- Very Procedural -- Better Alternatives?

...
7 votes
1 answer
4k views

How to optimize a pipline for storing item in mysql?

I am using scrapy framework for data scraping and dumping item in MySQL database. Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...