Code Review

Questions tagged [scrapy]

Ask Question

Scrapy is a fast open-source high-level screen scraping and web crawling framework written in python, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

34 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
-3
votes
0answers
28 views

How could this web crawler built with pure Python be better written [closed]

I just found a Web Crawler very well made and built using basic python modules and I would like to improve it so the code does a similar thing but with less code but also using best practices and ...
3
votes
0answers
26 views

Scraping websites for presence of particular words

Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general ...
6
votes
1answer
77 views

Website parser for advertisements

I'm new to Python and I have written the spider on the Scrapy framework which parses some advertisement websites and proceed data further. The spider works as I expect but I would like to show you the ...
3
votes
0answers
46 views

Microservice for scraping images with celery

I made a project, that scrapes images asynchronously and saves them in container. I have access to them through volume. Scrapy finds images on given web page. Any tips will be good. But first I would ...
6
votes
0answers
750 views

Parsing different categories using Scrapy from a webpage

I've written a script in Python Scrapy to parse different "model", "country" and "year" of various bikes from a webpage. There are several subcategories to track to reach ...
4
votes
1answer
76 views

small web scraper to read product highlight from given urls

I have written a Python scraping program, using the Scrapy framework. The following code reads a list of urls that need to crawled and gets product highlights from the list. It also follows the next ...
3
votes
1answer
93 views

Web Scraping Dynamically Generated Content Python

This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
1
vote
0answers
131 views

Web scraping dynamic content

Hi I'm fairly new to coding and would appreciate some feedback on the code. This is for a site that has a dynamic login page and infinite scrolling. I wanted to use scrapy not necessarily because it ...
5
votes
3answers
7k views

Writing to a csv file in a customized way using scrapy

I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...
2
votes
0answers
32 views

Use Scrapy to find duplicate images across pages

I'm trying to use scrapy to find image URLs that are used more than once on a website across all pages. This is my spider: ...
4
votes
1answer
92 views

Sourcing data format from multiple different structures

Problem I want to read in the data to dictionary person = { 'name': 'John Doe', 'email': 'johndoe@email.com', 'age': 50, 'connected': False } ...
2
votes
2answers
1k views

Creating a csv file using scrapy

I've created a script using Python in association with Scrapy to parse the movie names and its years spread across multiple pages from a torrent site. My goal here is to write the parsed data in a CSV ...
2
votes
0answers
153 views

Handling IndexError using lambda function within scrapy

I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
6
votes
1answer
155 views

Locating contact information from a CSV spreadsheet

I'm trying to find a more effective way to access variables within for loops. I felt that my attempt which works is not an effective way to access variables that ...
4
votes
1answer
84 views

Webscraping from yellowbook

I'm fairly good at programming python but of course i want to write cleaner code. This is a simple script that scrapes leads off yellowbook. I want to utilize list comprehensions but im comfortable ...
5
votes
1answer
6k views

Parsing a slow-loading webpage with scrapy in combination with selenium

I've written a scraper in Python scrapy in combination with selenium to scrape 1000 company names and their revenue from a website. The site has got lazy-loading method enabled so it is not possible ...
1
vote
1answer
959 views

Extracting certain products from a webpage using Scrapy

I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
3
votes
1answer
2k views

Scraping restaurant names and addresses from a website using Scrapy and Selenium

I have written some Python code that uses Scrapy and Selenium to scrape restaurant names and addresses from a website. I needed to use Selenium because the button to show more restaurants on a page is ...
6
votes
3answers
677 views

Scraping a webpage copying with the logic of scrapy

Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
2
votes
1answer
4k views

Scrapy crawler to parse data recursively

I've written a script in python scrapy to parse "name" and "price" of different products from a website. Firstly, it scrapes the links of different categories from the upper sided bar located in the ...
4
votes
0answers
428 views

Crawling thesaurus for a synonym

I have just finished core functionality for Tehsaurux. It crawls three thesauruses and will find a random synonym for you. It works with PHP agnostic and Scrapy for Python. I think the two main files -...
5
votes
1answer
131 views

News site scraping

I'm developing scrapy project for scraping news from sites and saving them to Google Sheets. Source repository is here. I want to ask about my class that must be ...
5
votes
1answer
109 views

Check the stock of references list

I did this spider that checks the stock of references list (around 3500 references). Now the spider takes around 37 seconds to scrap 400 references. The CPU is around 5%, the network card (1Gbps) ...
2
votes
1answer
148 views

Scrapy Username Spider

This currently starts out with the speed of 2000 pages/min but shortly after starting it becomes very slow with a speed of about 200 pages/min. Why is this happening? How can I improve this scraper? <...
5
votes
1answer
492 views

Scraping names of directors from a website

I am scraping the names of the directors from a website using Python / ScraPy. I am very new to coding (under a year and after work) - any views would be appreciated. The reason I have a ...
17
votes
1answer
14k views

Scraping after login using Scrapy

I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session. It retrieves the title of these pages ...
3
votes
2answers
4k views

Check duplicate items in Mongodb

I need to query MongoDB and check if an item exists or not: ...
2
votes
2answers
2k views

Scrapy spider for products on a site

I recently submitted a code sample for a web scraping project and was rejected without feedback as to what they didn't like. The prompt, while I cannot give it here verbatim, basically stated that I ...
3
votes
2answers
930 views

Using Scrapy/Xpath to scrape ESPN for football (soccer) commentaries

The class takes an input of a game-id from the ESPN soccer website. The code then has to grab the commentary, process that and also grab the player names/ids and create a small dictionary of those. I ...
2
votes
2answers
1k views

Data extraction

I wrote this code in Python/ Scrapy for data extraction. It works fine for me, but I am a newbie in Python, so any suggestions for code optimization would be welcome. ...
3
votes
2answers
3k views

Default initialization of items using Scrapy [closed]

I am writing a scraper, using Scrapy, in which I need to populate items with default values at first. Could my method be improved? ...
3
votes
1answer
1k views

Why is my web scraping script so slow?

My script, which is a web scraping script, is very slow, I even needed to put set_time_limit(0); This is the whole http://phpfiddle.org/main/code/9qt-78n I think the problem is here: ...
7
votes
1answer
4k views

How to optimize a pipline for storing item in mysql?

I am using scrapy framework for data scraping and dumping item in MySQL database. Here is my pipeline that is inserting output to MySQL, but its taking so much time. Any suggestions on how to ...