Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

80 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
0 answers
2k views

Scraping OddsPortal with requests only

This is a scraper written to do most of what had been attempted by another user in this question: How can I optimise this webscraping code I did the rewrite because I felt bad that the new user didn't ...
Reinderien's user avatar
  • 62.1k
5 votes
0 answers
97 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
Iduoad's user avatar
  • 151
5 votes
0 answers
453 views

Scraper for parsing, downloading, and setting images beside each link in an Excel file

I've written a script in VBA which is able to parse image links from a website, download and store them in a local folder and finally set those images beside each link in an Excel file. As the script ...
SIM's user avatar
  • 2,501
5 votes
0 answers
279 views

Perl crypto-scraper

I created a crypto-scraper script to scrape crypto currencies from Coingecko. I have used the library LWP::UserAgent and now I want to know if you could give me ...
Patrick85's user avatar
  • 491
4 votes
0 answers
83 views

Scraping and filling out a time-off request

In my ongoing quest to automate everything I have to do at my job, I've decided to learn how to do webscraping with VBA. I wanted to start with something easy, so the following simply fills out a ...
Hayden Moss's user avatar
4 votes
0 answers
181 views

Return different types from web scraper method

Application Logic The software take odds data from an internet site and store them inside my own database, this application is basically a scraper. There are different types of odds, eg: FullTime ...
sfarzoso's user avatar
  • 137
4 votes
0 answers
67 views

Class based Kjijii Webscraper

Using PowerShell I have made a class based Kijiji webscraper that records progress in a MariaDB table (for use in another project). I had made this as a module with cmdlets months ago but I wanted to ...
Matt's user avatar
  • 2,558
4 votes
0 answers
671 views

Scraping website and print them via routers

I've been working on a personal project for a while which simply extracts some information from a website and save them as JSON files. Then I've decided to move those scraping functionalities to the ...
kondanta's user avatar
4 votes
0 answers
114 views

List all links in a website

I wrote this code as part of a project I'm working on. It's supposed to get all links from a website and then perform black-box tests on them. How can I improve this code to be faster and more ...
Enass Hussien's user avatar
4 votes
0 answers
819 views

Speeding web-scraping up Python 3

I want to get information from two websites and display it on 'real-time' in the console. To get the information from the website I am using BeautifulSoup 4. I have read, that the bottleneck of ...
OcK's user avatar
  • 41
4 votes
0 answers
218 views

Web scraping with Nokogiri

At work we have a need to know what printers are getting dangerously low on their toner, and paper consumption, etc.. So I've created a program that pulls the printer information off the websites the ...
user97942's user avatar
4 votes
0 answers
2k views

Web scraping from the Google Play store

I am using this R function to web scrape data from the google play store. Is there a way to increase its efficiency using R? This code takes about 4 seconds for 14 urls with my machine/internet ...
CFM's user avatar
  • 41
4 votes
0 answers
449 views

Crawling and parsing meteorological data from the web into R

I am interested in collecting directly into R data published by the Mexican Met-office. The data pieces are spread through several URLs, but one can start here. There I can get the names and ...
MiEquiZ's user avatar
  • 41
3 votes
0 answers
61 views

Gather product information from the Apotheek Dedeyne web site

I'm learning Python Scrapy, and I'm scraping product information from the following website: https://apotheekdedeyne.be/ This project is for personal learning & ...
Seraph776's user avatar
  • 191
3 votes
0 answers
774 views

A simple Python script that crawls information about Youtube playlists and your watch history

You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
Ξένη Γήινος's user avatar
3 votes
0 answers
86 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
berkersenol's user avatar
3 votes
0 answers
184 views

requests vs selenium vs scrapy

This is a follow-up of my question over here. I have been working on my web-scraping techniques by trying out different approaches and rewriting code for a handful of online databases. I've recently ...
Sati's user avatar
  • 417
3 votes
0 answers
383 views

Speed Up API Requests & Overall Python Code

I'm not asking for help solving a problem but rather asking for help for possible ways to improve the speed of my program. Essentially what this does is: Tracks market data by pulling the data from ...
Steel Hard's user avatar
3 votes
0 answers
77 views

Web Scraping data from website using Selenium in python

ApplicationDetail.txt: URL: https://www.abc.com applicationNo : 123456 Application Class: ...
Mr.A I's user avatar
  • 31
3 votes
0 answers
60 views

Performance and abstraction questions re: scraper for carfolio.com

I'm a new Rust programmer, primarily coming from Ruby experience. I started working on a small project to scrape vehicle specifications from https://carfolio.com I chose to write it in Rust and ...
hwrd's user avatar
  • 151
3 votes
0 answers
74 views

Scraping reddit using Python

My objective is to find out on what other subreddit users from r/(subreddit) are posting on; you can see my code below. It works pretty well, but I am curious to know if I could improve it by: First, ...
User00257's user avatar
3 votes
0 answers
49 views

Scraping local news sites

This is my first Python web scraper (and overall my first Python project). I am also relatively new to OOP but do understand its core fundamentals. The script below scrapes two local news sites for ...
Giuseppe's user avatar
3 votes
0 answers
341 views

Python web scraper

This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused. However, ...
Joel's user avatar
  • 131
3 votes
0 answers
660 views

Scraping StackOverflow's questions with Node.js

I'm new to node and web scraping in general but with some research I've written this working code. My goal is to get all questions from X Stack Overflow's pages sorted by most rated and save that ...
Régis B.'s user avatar
3 votes
0 answers
182 views

Web scrape data for a list of stocks

I wrote a script that will web scrape data for a list of stocks. The scraper has to get the data from 2 separate pages so each stock symbol must scrape 2 different pages. If I run the process on a ...
jucas85's user avatar
  • 31
3 votes
0 answers
5k views

Fetching data from a website using "POST" request

I've written some code in vba for the purpose of making twofold "POST" requests to get to the destination page and harvest name and address from there. There are two types of structures within which ...
SIM's user avatar
  • 2,501
3 votes
0 answers
618 views

Scraper Class with Regex

I already posted a not (yet) complete version of this class, but that question got closed, because it contained not passing tests. Here is the full version, all 5 tests pass. Originally there was ...
user3568719's user avatar
3 votes
0 answers
184 views

Scrape a stock value and add it to a csv file along with other data. [Python 3]

Any and all suggestions are welcome, mainly looking for suggestions on clean up, and making it more readable as I feel that it is inefficient and not very clean. Some specific things: A better way ...
Ari Madian's user avatar
3 votes
0 answers
1k views

Scraping table data from websites

I've written a script in vba to parse table from any website if the html elements are not encrypted in javascript. There is a table index in my script in which inputting the right number it is ...
SIM's user avatar
  • 2,501
3 votes
0 answers
46 views

Regularly watch recent posts of a blog for specific words with HTML scraping

Task I want to watch the "Recent Posts" section of a blog for changes/new posts but only for specific posts containing a pre-defined word. Afterwards a list should be outputted in the console with ...
sceiler's user avatar
  • 305
3 votes
0 answers
55 views

VIM colors downloader in Python, using multiprocessing

I recently posted this script: VIM colors downloader in Python But since I'm not allowed to update the code there, I wanted to get an idea on this version, that uses multiprocessing: ...
Kernel.Panic's user avatar
3 votes
0 answers
239 views

Checking paginated website for new entries

I'm interested in determining the best way to check a paginated website for new entries. I want to be able to scrape pages 1, 2, 3, ... as necessary to get all updates. However the scraping is fairly ...
Six's user avatar
  • 250
3 votes
0 answers
670 views

Scraping links from the first page of Google using Kivy

I'm making a scraper/web crawler using Kivy when I run the code it works but I'm not sure if what I'm doing is Pythonic because all the language I can find is about using the Kivy library. I'm unsure ...
Eric MacLeod's user avatar
3 votes
0 answers
137 views

Using Nokogiri to scrape Oscars winners from Wikipedia

I am scraping a Wikipedia page, getting info from that page and instantiating a new object with the information collected: ...
Cyzanfar's user avatar
  • 223
3 votes
0 answers
570 views

Rails app that scrapes forum using Nokogiri gem

I've built a website that scrapes a guitar forum's pages and populates Rails model. I'm using rake task along with heroku scheduler to run background scrapes every hour. On the homepage, the forum ads ...
Jamie B's user avatar
  • 315
2 votes
0 answers
57 views

Simplified HTML parsing for LEGO features

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
alvas's user avatar
  • 705
2 votes
0 answers
159 views

Web scraping articles using asyncio

I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
AlexDotis's user avatar
  • 417
2 votes
0 answers
598 views

My first vba-selenium code

I have been obtaining financial information on an annuity I have which is invested in multiple funds which are proprietary to the insurance company. The asset value is not available other than on ...
Ron Rosenfeld's user avatar
2 votes
0 answers
408 views

How to speed up the VBA scraping process

I want to scrape 25000 addresses using Column A cell values as search strings. I'm using the following code, it is working fine but it's too slow to extract 25000 results. Is there any way to speed up ...
Jamshaid's user avatar
2 votes
0 answers
2k views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
WhiteWood's user avatar
2 votes
0 answers
589 views

Discord Bot Python. Selenium screenshots

I wrote my own Discord Bot which is taking screenshot of a specific website. This scrip is very simple and thought it should work fast, but it isn't. I read a lot, and I think I'm not able to improve ...
Lukas's user avatar
  • 21
2 votes
0 answers
87 views

C# Scraper - EventHandler

It might help to check my previous question out, its part of the same application. The only minor changes are with the entity class names. They now each have their own entities, and I use interfaces ...
Josh Hallow's user avatar
  • 1,345
2 votes
0 answers
707 views

Forum web scraper in Lua

I've written a short web scraper in Lua (run in LuaJIT for speed) to download every post from a specific forum (not general purpose... yet). It takes about 12 hours to run and download 500,000 posts ...
Allister's user avatar
  • 171
2 votes
0 answers
52 views

Scraping Forum Tables w/ Links Using Beautiful Soup

This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
user53526356's user avatar
2 votes
0 answers
989 views

Downloading multiple urls with aiohttp Python 3

I am trying to use aiohttp library in python to download information from url. I have about 300 000 urls. They are saved in file "my_file.txt". When I get web page, I extract pairs of a question and ...
Danil Kononyhin's user avatar
2 votes
0 answers
407 views

Wrapping a web scraper in a RESTful API

The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
25Postcards's user avatar
2 votes
0 answers
1k views

Grabbing information traversing multiple pages

I've written a script in python in combination with selenium to parse different information from a webpage and store the collected data in a csv file. The data are rightly coming through. The email ...
SIM's user avatar
  • 2,501
2 votes
0 answers
1k views

Recursively scrape links from web pages and check them

I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles. The overall idea ...
ron_g's user avatar
  • 121
2 votes
0 answers
181 views

Handling IndexError using lambda function within scrapy

I've written a script using python's scrapy library to parse some fields from craigslist. The spider I've created here is way normal than what usually gets considered ideal to be reviewed. However, I'...
SIM's user avatar
  • 2,501
2 votes
0 answers
240 views

Multi Site Concurrent web crawler using gRPC

This is the server portion of a gRPC implementation of a multi site concurrent web crawler. ...
Angelo's user avatar
  • 31