Questions tagged [scrapy]
Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
37
questions
0
votes
1
answer
48
views
Save data in item wihitn Scrapy and Python
I need to store data in Scrapy Item in parts. To update only the information that I found at the moment.
I did it, but the code looks too verbose. Three lines I repeat very often:
...
- 1,600
3
votes
0
answers
61
views
Gather product information from the Apotheek Dedeyne web site
I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/
This project is for personal learning & ...
- 72k
7
votes
1
answer
1k
views
Parsing different categories using Scrapy from a webpage
I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach ...
- 16.7k
1
vote
1
answer
72
views
Parsing an XML tree of categories
I have parse function which is parsing tree of categories. I've written it in simplest way possible and now struggling with refactoring it.
Every nested loop is doing the same stuff but appending ...
- 144k
5
votes
1
answer
129
views
Check the stock of references list
I did this spider that checks the stock of references list (around 3500 references).
Now the spider takes around 37 seconds to scrape 400 references. The CPU is around 5%, the network card (1Gbps) ...
- 59k
5
votes
2
answers
585
views
Crawling thesaurus for a synonym
I have just finished core functionality for Tehsaurux. It crawls three thesauruses and will find a random synonym for you. It works with PHP agnostic and Scrapy for Python.
I think the two main files -...
- 8,544
4
votes
1
answer
948
views
Scraping a dynamic website with Scrapy (or Requests) and Selenium
I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here.
Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
- 417
3
votes
0
answers
40
views
Scraping websites for presence of particular words
Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general ...
6
votes
1
answer
94
views
Website parser for advertisements
I'm new to Python and I have written the spider on the Scrapy framework which parses some advertisement websites and proceed data further. The spider works as I expect but I would like to show you the ...
CommunityBot
- 1
3
votes
0
answers
332
views
Microservice for scraping images with celery
I made a project, that scrapes images asynchronously and saves them in container. I have access to them through volume. Scrapy finds images on given web page.
Any tips will be good. But first I would ...
4
votes
1
answer
102
views
small web scraper to read product highlight from given urls
I have written a Python scraping program, using the Scrapy framework. The following code reads a list of urls that need to crawled and gets product highlights from the list. It also follows the next ...
- 573
3
votes
1
answer
146
views
Web Scraping Dynamically Generated Content Python
This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
- 59k
1
vote
0
answers
154
views
Web scraping dynamic content
Hi I'm fairly new to coding and would appreciate some feedback on the code. This is for a site that has a dynamic login page and infinite scrolling.
I wanted to use scrapy not necessarily because it ...
- 19
5
votes
3
answers
9k
views
Writing to a csv file in a customized way using scrapy
I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
- 2,471
2
votes
0
answers
45
views
Use Scrapy to find duplicate images across pages
I'm trying to use scrapy to find image URLs that are used more than once on a website across all pages.
This is my spider:
...
- 121
4
votes
1
answer
104
views
Sourcing data format from multiple different structures
Problem
I want to read in the data to dictionary
person = {
'name': 'John Doe',
'email': '[email protected]',
'age': 50,
'connected': False
}
...
- 59k
2
votes
2
answers
2k
views
Creating a csv file using scrapy
I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
- 2,471
2
votes
0
answers
180
views
Handling IndexError using lambda function within scrapy
I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
- 2,471
6
votes
1
answer
172
views
Locating contact information from a CSV spreadsheet
I'm trying to find a more effective way to access variables within for loops. I felt that my attempt which works is not an effective way to access variables that ...
- 2,473
4
votes
1
answer
88
views
Webscraping from yellowbook
I'm fairly good at programming python but of course i want to write cleaner code.
This is a simple script that scrapes leads off yellowbook.
I want to utilize list comprehensions but im comfortable ...
- 606
5
votes
1
answer
8k
views
Parsing a slow-loading webpage with scrapy in combination with selenium
I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible ...
- 34.9k
1
vote
1
answer
1k
views
Extracting certain products from a webpage using Scrapy
I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
- 17.2k
3
votes
1
answer
3k
views
Scraping restaurant names and addresses from a website using Scrapy and Selenium
I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
CommunityBot
- 1
6
votes
3
answers
701
views
Scraping a webpage copying with the logic of scrapy
Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
- 21
2
votes
1
answer
5k
views
Scrapy crawler to parse data recursively
I've written a script in python scrapy to parse "name" and "price" of different products from a website. Firstly, it scrapes the links of different categories from the upper sided bar located in the ...
- 17.2k
5
votes
1
answer
141
views
News site scraping
I'm developing scrapy project for scraping news from sites and saving them to Google Sheets. Source repository is here.
I want to ask about my class that must be ...
- 17.2k
2
votes
1
answer
184
views
Scrapy Username Spider
This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper?
<...
CommunityBot
- 1
5
votes
1
answer
601
views
Scraping names of directors from a website
I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated.
The reason I have a ...
- 111k
17
votes
1
answer
15k
views
Scraping after login using Scrapy
I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session.
It retrieves the title of these pages ...
- 42.7k
3
votes
2
answers
4k
views
Check duplicate items in Mongodb
I need to query MongoDB and check if an item exists or not:
...
- 34.9k
2
votes
2
answers
2k
views
Scrapy spider for products on a site
I recently submitted a code sample for a web scraping project and was rejected without feedback as to what they didn't like. The prompt, while I cannot give it here verbatim, basically stated that I ...
CommunityBot
- 1
3
votes
2
answers
1k
views
Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries
The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those.
I ...
- 34.9k
2
votes
2
answers
1k
views
Data extraction
I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome.
...
- 34.9k
3
votes
2
answers
4k
views
Default initialization of items using Scrapy [closed]
I am writing a scraper, using Scrapy, in which I need to populate items with default values at first.
Could my method be improved?
...
- 34.9k
3
votes
1
answer
1k
views
Why is my web scraping script so slow?
My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0);
This is the whole http://phpfiddle.org/main/code/9qt-78n
I think the problem is here:
...
- 33.1k
2
votes
1
answer
2k
views
7
votes
1
answer
4k
views
How to optimize a pipline for storing item in mysql?
I am using scrapy framework for data scraping and dumping item in MySQL database.
Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...
CommunityBot
- 1