Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
3 votes
1 answer
68 views

Scraping the Divar.ir

I've wrote a code to scrape the Divar, which is an equivalent of Ebay in Iran. I have a few questions: Am I doing the error handling and logging ok? Is there a better way to optimize this code? (note ...
Amirhossein Rezaei's user avatar
-1 votes
2 answers
85 views

Web scraping spider

I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
Max's user avatar
  • 7
3 votes
2 answers
143 views

Enum to deserialize HTML sizes from JSON with serde

I added an enum for my webscraper to deserialize data from a JSON field that represents an HTML image size, which can either be an unsigned int like 1080 or a ...
Richard Neumann's user avatar
1 vote
0 answers
51 views

Web scraper to retrieve articles from HTML embedded in JavaScript

I wrote a web scraper library to parse articles from HTML documents where they are encoded in a JSON object in JavaScript element. I wrote the example HTML file and ...
Richard Neumann's user avatar
3 votes
0 answers
34 views

Automatically extract useful cars from car site

I am using puppeteer to extract data and see when a car that meets my requirements shows up, this is what I did so far. I would like some basic syntax advice, or more advanced tips as well. I tried to ...
Mah Neh's user avatar
  • 31
2 votes
0 answers
45 views

Simplified HTML parsing for LEGO features

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
alvas's user avatar
  • 705
0 votes
1 answer
70 views

Poetry Web Scraping in Python [closed]

I have a script that obtains urls that lead to a specific poem. This code current works and uses multiprocessing pools. I currently am getting restricted or blocked by some way from the website that I ...
watrgoat's user avatar
0 votes
1 answer
44 views

HTTP scraper for Python Package

I'm trying to make my first Python package as a learning experience. There's a lot of things that I suspect I am doing poorly, but this post is specifically about my HttpRequest class. I made this ...
jtb's user avatar
  • 185
2 votes
1 answer
43 views

URL link scraper and analyser

I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on). Essentially,...
Jessica's user avatar
  • 818
4 votes
2 answers
211 views

Test generator I made for practice

Made this generator to practice using imports from other modules and better readability for coding. What could I have done better and what did I do wrong? File called test_generator.py ...
Beginner's user avatar
  • 189
1 vote
0 answers
51 views

Search Stack Overflow and GitHub for code in a specified language

This code is designed to scrape Stack Overflow and GitHub, pulling information based on a user-specified programming language and processing the data into a format for AI learning. It uses a number of ...
Robert3737's user avatar
3 votes
1 answer
105 views

A simple web scraper for nature.com news articles

I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text. I am learning independently, so I would ...
razzleDazzle's user avatar
1 vote
0 answers
756 views

Code scraping oddsportal using beautifulsoup

This code scrapes the website www.oddsportal.com for betting odds and historical games. code: ...
Rander's user avatar
  • 11
2 votes
1 answer
150 views

Beginner Python Web scraping

I'm a newbie in programming, I chose Python. I'm learning on my own. Currently I'm preparing code for a portfolio on github. I will be grateful for any code review, especially in the subject of OOP: ...
Magda's user avatar
  • 29
3 votes
0 answers
61 views

Gather product information from the Apotheek Dedeyne web site

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/ This project is for personal learning & ...
Seraph776's user avatar
  • 191
6 votes
1 answer
562 views

Python web scraper for Amazon customer reviews

I'm new to web scraping and tried building a web scraper for Amazon customer reviews. The program works fine as is but I wanted to improve the design and get some feedback. The basic idea was to ...
nightstand's user avatar
0 votes
1 answer
106 views

Selenium/Beautifulsoup web scrape of Grainger

This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
cahilltyler's user avatar
3 votes
2 answers
139 views

Saving Scraped Data to a File

When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file? open the file first, scrape, and save the data all ...
Seraph776's user avatar
  • 191
5 votes
1 answer
149 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
alphamu's user avatar
  • 153
5 votes
1 answer
188 views

Grabbing youtube thumbnails from urls

I made some C# code to grab youtube thumbnails from urls, I originally made this in python took some time converting it to C#. I am VERY new to C# and did this with minimal help. ...
Joshua Bursill's user avatar
3 votes
1 answer
35 views

request data and print results

On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
Luca's user avatar
  • 57
1 vote
0 answers
53 views

Web-scraping bountied questions from Stack Exchange sites

I recently built my first web scraper in python, and decided to use SO and SE as test websites. Code ...
DialFrost's user avatar
  • 123
2 votes
1 answer
48 views

scraping pollution data tables from government website

i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved: ...
Omar Morales Rivera's user avatar
1 vote
1 answer
95 views

Parsing: Сollecting data from a book site

A task: Сollect data from the site in the following format: book; user; book_rating; comment_rating; publication_date; comment For one book at once several pages of ...
Pom Mop's user avatar
  • 57
3 votes
1 answer
91 views

A Language Bot: For creating noun declension or verb conjugation tables

I'm a freshman student. What does my program solve? This is actually a bot for reddit, and there are language learning subreddits on that website, sometimes when discussing something with people, we ...
user9560064's user avatar
3 votes
1 answer
163 views

Scrape PokeDex and display in tkinter

Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
Salah Zahran's user avatar
3 votes
1 answer
73 views

Get the 64 most popular items from a list of tree species using webscraping

I made this python program to find the 64 most common elements of a list stored in a text file. Since this works using binomial names of trees I decided to use the number of mentions in PubMed Central ...
Omar Morales Rivera's user avatar
9 votes
2 answers
2k views

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user avatar
1 vote
0 answers
54 views

How to make memory and cpu efficient a code

I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
Learner's user avatar
  • 111
1 vote
0 answers
36 views

Iterate tables from table id from href links until no table with specific table id is found

I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html What I am trying to achieve is to parse the ...
Gescof's user avatar
  • 11
3 votes
1 answer
140 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
Olgierd Wiśniewski's user avatar
1 vote
1 answer
108 views

Design Pattern: Builder - BeautifulSoup directory navigation and scraping

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
Jason Leaver's user avatar
3 votes
1 answer
195 views

web scraping with pandas and apscheduler

Below is a script to scrape data from the National Severe Storms Laboratory Probsevere dataset. Their data is available at a 2 minute interval and is only available for 24 hours before moving to the ...
Jason Leaver's user avatar
1 vote
0 answers
169 views

Web scraper for pokemon prices using Selenium

I made this scraper to pull the prices of pokemon cards off tcgcollector, using a csv file from the same site. Because for some reason they don't export price. I'm looking for any kind of noobie ...
Moorgort's user avatar
1 vote
1 answer
302 views

Selenium web scraper to scrape Apple tweets of the last 3 years

I want to use my Web Scraper to get all tweets about Apple up to a date I specify. At the moment I'm scraping the tweets of today, or the last few days. However, my goal is to scrape all tweets from ...
VaKt's user avatar
  • 11
3 votes
0 answers
557 views

A simple Python script that crawls information about Youtube playlists and your watch history

You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
Ξένη Γήινος's user avatar
1 vote
1 answer
53 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
Sati's user avatar
  • 417
3 votes
1 answer
89 views

Acupuncture database builder

The following code builds a rudimentary acupuncture database by collecting data from the web. I would like to hear suggestions about improvements to the database structure, code organization, web-...
Sati's user avatar
  • 417
0 votes
0 answers
41 views

Python scraper that gets product information from e-commerce website

I have built a scraper that scrapes data from e-commerce website and saves it to json using my own module. Although, I made the scraper multi-threaded, it starts slow, then speeds up and uses a lot of ...
dovexz12323's user avatar
3 votes
0 answers
79 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
berkersenol's user avatar
4 votes
2 answers
258 views

Scraping campsite availability from a webpage using vba with selenium

I wrote some code to extract the information from a table, but it takes an extremely long time. The table is in the format of a calendar. I need the information on an Excel sheet with column 1 as the ...
Nelj's user avatar
  • 41
8 votes
1 answer
138 views

Scrape a URL for articles to save

Below is an exercise project I was doing on a certain educational site. It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
Gavin Greenhorn's user avatar
1 vote
1 answer
34 views

Extracting information from HTML with XSLT 3.0 when data is grouped visually as siblings in a td separated by blank lines

I have a work-in-progress where I'm using XSLT 3 to extract information from some preprocessed archaic HTML. I'd like to produce JSON showing the relationships between the various entities for further ...
Forensic_07's user avatar
1 vote
1 answer
265 views

Web-scrape bonds information from nasdaqomxnordic

I'm a newbie getting into web scrapers. I've made something that works, it takes 3.2 hours to complete job and randomly have 10 lines blank each time I run this job. Help is much appreciated! ...
yogi06's user avatar
  • 13
2 votes
1 answer
333 views

Using XSLT 3.0 to extract information from real-world HTML and produce JSON

For work, I extract information from HTML and XML sources to save in databases. The objective is to generate JSON representing the source document's information and its relationships in order to 1) ...
Forensic_07's user avatar
4 votes
1 answer
120 views

Shell script to download Project Euler problems and combine to PDF

This is a script I created that downloads Project Euler webpages and combines them to PDF. The script also downloads animated files. ...
qwr's user avatar
  • 1,016
4 votes
2 answers
527 views

Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python

I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths. The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
Nini's user avatar
  • 55
2 votes
0 answers
139 views

Web scraping articles using asyncio

I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
AlexDotis's user avatar
  • 417
3 votes
1 answer
239 views

Web scraping international trade data

I'm trying to scrape arrival data from this website. My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
lockey's user avatar
  • 51
2 votes
0 answers
407 views

My first vba-selenium code

I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company. The asset value is not available other than on ...
Ron Rosenfeld's user avatar

1
2 3 4 5
13