Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
612
questions
3
votes
1
answer
68
views
Scraping the Divar.ir
I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions:
Am I doing the error handling and logging ok?
Is there a better way to optimize this code? (note ...
-1
votes
2
answers
85
views
Web scraping spider
I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
- 7
3
votes
2
answers
143
views
Enum to deserialize HTML sizes from JSON with serde
I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
- 5,966
1
vote
0
answers
51
views
Web scraper to retrieve articles from HTML embedded in JavaScript
I wrote a web scraper library to parse articles from HTML documents where they are encoded in a JSON object in JavaScript element.
I wrote the example HTML file and ...
- 5,966
3
votes
0
answers
34
views
Automatically extract useful cars from car site
I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well.
I tried to ...
- 31
2
votes
0
answers
45
views
Simplified HTML parsing for LEGO features
The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
- 705
0
votes
1
answer
70
views
Poetry Web Scraping in Python [closed]
I have a script that obtains urls that lead to a specific poem. This code current works and uses multiprocessing pools. I currently am getting restricted or blocked by some way from the website that I ...
- 3
0
votes
1
answer
44
views
HTTP scraper for Python Package
I'm trying to make my first Python package as a learning experience. There's a lot of things that I suspect I am doing poorly, but this post is specifically about my HttpRequest class. I made this ...
- 185
2
votes
1
answer
43
views
URL link scraper and analyser
I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on).
Essentially,...
- 818
4
votes
2
answers
211
views
Test generator I made for practice
Made this generator to practice using imports from other modules and better readability for coding. What could I have done better and what did I do wrong?
File called test_generator.py
...
- 189
1
vote
0
answers
51
views
Search Stack Overflow and GitHub for code in a specified language
This code is designed to scrape Stack Overflow and GitHub, pulling information based on a user-specified programming language and processing the data into a format for AI learning.
It uses a number of ...
- 19
3
votes
1
answer
105
views
A simple web scraper for nature.com news articles
I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text.
I am learning independently, so I would ...
- 31
1
vote
0
answers
756
views
Code scraping oddsportal using beautifulsoup
This code scrapes the website www.oddsportal.com for betting odds and historical games.
code:
...
- 11
2
votes
1
answer
150
views
Beginner Python Web scraping
I'm a newbie in programming, I chose Python. I'm learning on my own.
Currently I'm preparing code for a portfolio on github.
I will be grateful for any code review, especially in the subject of OOP: ...
- 29
3
votes
0
answers
61
views
Gather product information from the Apotheek Dedeyne web site
I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/
This project is for personal learning & ...
- 191
6
votes
1
answer
562
views
Python web scraper for Amazon customer reviews
I'm new to web scraping and tried building a web scraper for Amazon customer reviews. The program works fine as is but I wanted to improve the design and get some feedback. The basic idea was to ...
- 295
0
votes
1
answer
106
views
Selenium/Beautifulsoup web scrape of Grainger
This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
3
votes
2
answers
139
views
Saving Scraped Data to a File
When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file?
open the file first, scrape, and save the data all ...
- 191
5
votes
1
answer
149
views
Web scraper for data sources from Statistics Canada
I've written a parser to scrape data from Canadian Statistics Bureau.
...
- 153
5
votes
1
answer
188
views
Grabbing youtube thumbnails from urls
I made some C# code to grab youtube thumbnails from urls, I originally made this in python took some time converting it to C#. I am VERY new to C# and did this with minimal help.
...
3
votes
1
answer
35
views
request data and print results
On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
- 57
1
vote
0
answers
53
views
Web-scraping bountied questions from Stack Exchange sites
I recently built my first web scraper in python, and decided to use SO and SE as test websites.
Code
...
- 123
2
votes
1
answer
48
views
scraping pollution data tables from government website
i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved:
...
1
vote
1
answer
95
views
Parsing: Сollecting data from a book site
A task:
Сollect data from the site in the following format:
book; user; book_rating; comment_rating; publication_date; comment
For one book at once several pages of ...
- 57
3
votes
1
answer
91
views
A Language Bot: For creating noun declension or verb conjugation tables
I'm a freshman student.
What does my program solve?
This is actually a bot for reddit, and there are language learning subreddits on that website, sometimes when discussing something with people, we ...
- 77
3
votes
1
answer
163
views
Scrape PokeDex and display in tkinter
Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
- 33
3
votes
1
answer
73
views
Get the 64 most popular items from a list of tree species using webscraping
I made this python program to find the 64 most common elements of a list stored in a text file. Since this works using binomial names of trees I decided to use the number of mentions in PubMed Central ...
9
votes
2
answers
2k
views
Python script to scrape and parse the Stanford Encyclopedia of Philosophy
I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback.
There is no function but I didn't think it was worth adding. Also there is no test ...
user260644
1
vote
0
answers
54
views
How to make memory and cpu efficient a code
I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
- 111
1
vote
0
answers
36
views
Iterate tables from table id from href links until no table with specific table id is found
I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html
What I am trying to achieve is to parse the ...
- 11
3
votes
1
answer
140
views
Crawler/scraper for soccer match results
I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
1
vote
1
answer
108
views
Design Pattern: Builder - BeautifulSoup directory navigation and scraping
I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory.
There are a couple navigation ...
- 628
3
votes
1
answer
195
views
web scraping with pandas and apscheduler
Below is a script to scrape data from the National Severe Storms Laboratory Probsevere dataset. Their data is available at a 2 minute interval and is only available for 24 hours before moving to the ...
- 628
1
vote
0
answers
169
views
Web scraper for pokemon prices using Selenium
I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
- 11
1
vote
1
answer
302
views
Selenium web scraper to scrape Apple tweets of the last 3 years
I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
- 11
3
votes
0
answers
557
views
A simple Python script that crawls information about Youtube playlists and your watch history
You will need to follow this guide.
This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
- 2,954
1
vote
1
answer
53
views
Iterating over tables with nested <th> tags with BS4
The following code iterates over a table with nested th headings.
The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
- 417
3
votes
1
answer
89
views
Acupuncture database builder
The following code builds a rudimentary acupuncture database by collecting data from the web.
I would like to hear suggestions about improvements to the database structure, code organization, web-...
- 417
0
votes
0
answers
41
views
Python scraper that gets product information from e-commerce website
I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
- 101
3
votes
0
answers
79
views
Scraping housing dataset row by row from Estate Agent's website
I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame.
...
- 31
4
votes
2
answers
258
views
Scraping campsite availability from a webpage using vba with selenium
I wrote some code to extract the information from a table, but it takes an extremely long time.
The table is in the format of a calendar.
I need the information on an Excel sheet with column 1 as the ...
- 41
8
votes
1
answer
138
views
Scrape a URL for articles to save
Below is an exercise project I was doing on a certain educational site.
It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
- 145
1
vote
1
answer
34
views
Extracting information from HTML with XSLT 3.0 when data is grouped visually as siblings in a td separated by blank lines
I have a work-in-progress where I'm using XSLT 3 to extract information from some preprocessed archaic HTML. I'd like to produce JSON showing the relationships between the various entities for further ...
- 135
1
vote
1
answer
265
views
Web-scrape bonds information from nasdaqomxnordic
I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated!
...
- 13
2
votes
1
answer
333
views
Using XSLT 3.0 to extract information from real-world HTML and produce JSON
For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) ...
- 135
4
votes
1
answer
120
views
Shell script to download Project Euler problems and combine to PDF
This is a script I created that downloads Project Euler webpages and combines them to PDF. The script also downloads animated files.
...
- 1,016
4
votes
2
answers
527
views
Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python
I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths.
The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
- 55
2
votes
0
answers
139
views
Web scraping articles using asyncio
I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages.
I would like to improve ...
- 417
3
votes
1
answer
239
views
Web scraping international trade data
I'm trying to scrape arrival data from this website. My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process?
Here's my script:
...
- 51
2
votes
0
answers
407
views
My first vba-selenium code
I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company.
The asset value is not available other than on ...
- 121