Questions tagged [scrapy]
Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
34
questions
3
votes
1answer
193 views
Scraping a dynamic website with Scrapy (or Requests) and Selenium
I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here.
Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
3
votes
0answers
30 views
Scraping websites for presence of particular words
Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general ...
6
votes
1answer
79 views
Website parser for advertisements
I'm new to Python and I have written the spider on the Scrapy framework which parses some advertisement websites and proceed data further. The spider works as I expect but I would like to show you the ...
3
votes
0answers
85 views
Microservice for scraping images with celery
I made a project, that scrapes images asynchronously and saves them in container. I have access to them through volume. Scrapy finds images on given web page.
Any tips will be good. But first I would ...
6
votes
0answers
884 views
Parsing different categories using Scrapy from a webpage
I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach ...
4
votes
1answer
88 views
small web scraper to read product highlight from given urls
I have written a Python scraping program, using the Scrapy framework. The following code reads a list of urls that need to crawled and gets product highlights from the list. It also follows the next ...
3
votes
1answer
115 views
Web Scraping Dynamically Generated Content Python
This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
1
vote
0answers
146 views
Web scraping dynamic content
Hi I'm fairly new to coding and would appreciate some feedback on the code. This is for a site that has a dynamic login page and infinite scrolling.
I wanted to use scrapy not necessarily because it ...
5
votes
3answers
9k views
Writing to a csv file in a customized way using scrapy
I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
2
votes
0answers
36 views
Use Scrapy to find duplicate images across pages
I'm trying to use scrapy to find image URLs that are used more than once on a website across all pages.
This is my spider:
...
4
votes
1answer
92 views
Sourcing data format from multiple different structures
Problem
I want to read in the data to dictionary
person = {
'name': 'John Doe',
'email': 'johndoe@email.com',
'age': 50,
'connected': False
}
...
2
votes
2answers
2k views
Creating a csv file using scrapy
I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
2
votes
0answers
160 views
Handling IndexError using lambda function within scrapy
I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
6
votes
1answer
157 views
Locating contact information from a CSV spreadsheet
I'm trying to find a more effective way to access variables within for loops. I felt that my attempt which works is not an effective way to access variables that ...
4
votes
1answer
85 views
Webscraping from yellowbook
I'm fairly good at programming python but of course i want to write cleaner code.
This is a simple script that scrapes leads off yellowbook.
I want to utilize list comprehensions but im comfortable ...
5
votes
1answer
7k views
Parsing a slow-loading webpage with scrapy in combination with selenium
I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible ...
1
vote
1answer
983 views
Extracting certain products from a webpage using Scrapy
I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
3
votes
1answer
2k views
Scraping restaurant names and addresses from a website using Scrapy and Selenium
I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
6
votes
3answers
683 views
Scraping a webpage copying with the logic of scrapy
Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
2
votes
1answer
4k views
Scrapy crawler to parse data recursively
I've written a script in python scrapy to parse "name" and "price" of different products from a website. Firstly, it scrapes the links of different categories from the upper sided bar located in the ...
4
votes
0answers
466 views
Crawling thesaurus for a synonym
I have just finished core functionality for Tehsaurux. It crawls three thesauruses and will find a random synonym for you. It works with PHP agnostic and Scrapy for Python.
I think the two main files -...
5
votes
1answer
134 views
News site scraping
I'm developing scrapy project for scraping news from sites and saving them to Google Sheets. Source repository is here.
I want to ask about my class that must be ...
5
votes
1answer
109 views
Check the stock of references list
I did this spider that checks the stock of references list (around 3500 references).
Now the spider takes around 37 seconds to scrap 400 references. The CPU is around 5%, the network card (1Gbps) ...
2
votes
1answer
166 views
Scrapy Username Spider
This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper?
<...
5
votes
1answer
516 views
Scraping names of directors from a website
I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated.
The reason I have a ...
17
votes
1answer
14k views
Scraping after login using Scrapy
I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session.
It retrieves the title of these pages ...
3
votes
2answers
4k views
2
votes
2answers
2k views
Scrapy spider for products on a site
I recently submitted a code sample for a web scraping project and was rejected without feedback as to what they didn't like. The prompt, while I cannot give it here verbatim, basically stated that I ...
3
votes
2answers
952 views
Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries
The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those.
I ...
2
votes
2answers
1k views
Data extraction
I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome.
...
3
votes
2answers
3k views
Default initialization of items using Scrapy [closed]
I am writing a scraper, using Scrapy, in which I need to populate items with default values at first.
Could my method be improved?
...
3
votes
1answer
1k views
Why is my web scraping script so slow?
My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0);
This is the whole http://phpfiddle.org/main/code/9qt-78n
I think the problem is here:
...
2
votes
1answer
2k views
7
votes
1answer
4k views
How to optimize a pipline for storing item in mysql?
I am using scrapy framework for data scraping and dumping item in MySQL database.
Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...