Questions tagged [web-scraping]
Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.
529
questions
4
votes
1answer
99 views
Fetching API with requests.Session
I wrote a script that parses API on schedule (Tuesday-Saturday), downloading everything for the previous day.
...
4
votes
2answers
310 views
web scraper for emails and links
I created a class to scrape URLS, parse and validate emails and get internal links.
How can I achieve the SOLID principles in this class written in Javascript to make a web scraper?
...
2
votes
1answer
120 views
Excel VBA code that displays a 14 day weather forecast
I am trying to see if anyone can help me in making this Excel VBA code a little faster in rendering the data from the weather site I am using. Maybe instead of getting all 14 days it includes in the ...
2
votes
0answers
61 views
C# Scraper - EventHandler
It might help to check my previous question out, its part of the same application. The only minor changes are with the entity class names. They now each have their own entities, and I use interfaces ...
8
votes
1answer
153 views
C# Profile Scraper
I'm a 16-year-old so I never studied programming, although I do try to adopt the best practices I can when given the chance.
I would like to clean up the code posted below by making it comply with ...
6
votes
0answers
52 views
Booking an East London Tennis Court
Description
I'm not sure if it Covid-19 but lately it is impossible to book a tennis court in my area on time. It's always full or maybe just don't check enough :)
To beat the queue and get notified ...
6
votes
1answer
119 views
Concurrent Web Crawler
Description
As an exercise of learning go concurrency patterns, I decided too build a concurrent web crawler.
I made use of the argparse module I put up for review a while back.
I'm looking for ...
3
votes
1answer
41 views
Python Web-Scraping with BeautifulSoup
I'm relatively new to python and for an assignment I had to write a program that fetches a webpage with BeautifulSoup, extract all Paragraphs from it, and extract all words ending with "ing",...
3
votes
0answers
54 views
Taking time off of taking time off
In my ongoing quest to automate everything I have to do at my job, I've decided to learn how to do webscraping with VBA. I wanted to start with something easy, so the following simply fills out a ...
3
votes
0answers
31 views
Performance and abstraction questions re: scraper for carfolio.com
I'm a new Rust programmer, primarily coming from Ruby experience.
I started working on a small project to scrape vehicle specifications from https://carfolio.com
I chose to write it in Rust and ...
3
votes
1answer
46 views
Scraping weather conditions for mountain peeks
I have built a website in Django in which I tried for the first time to create a complex app. It's an app that scrapes weather conditions on a peak in mountains, which are 17. I wanted to show a ...
5
votes
1answer
148 views
Game scraper for Steam
I made a scraper for Steam that get different info about a Steam game, such as the price, the specs and the supported platforms. The reason I made this was because I have a super slow laptop, so ...
3
votes
0answers
64 views
Scraping reddit using Python
My objective is to find out on what other subreddit users from r/(subreddit) are posting on; you can see my code below. It works pretty well, but I am curious to know if I could improve it by:
First, ...
2
votes
1answer
38 views
d20pfsrd Discord Bot Scraper Abstraction and Deployment
I wrote a discord bot that listens for links to the d20pfsrd rulebook in discord chat, checks if they are feats or magic, and then scrapes the page, formats it, and spits it back into chat as ...
7
votes
1answer
148 views
A library for interacting with Pinnacle Sports Bets API
My code provides the following functionality for interacting with Pinnacle Bets API:
retrieving betting history
retrieving fixtures (future events)
retrieving odds for the given leagues (competitions)...
4
votes
1answer
75 views
small web scraper to read product highlight from given urls
I have written a Python scraping program, using the Scrapy framework. The following code reads a list of urls that need to crawled and gets product highlights from the list. It also follows the next ...
6
votes
1answer
76 views
Tool for converting OEIS sequences into Desmos lists
https://github.com/CascadeIllusion/DesmOEIS
Recently began work on a Python project called DesmOEIS to build up my portfolio. It's a simple console program that looks up integer sequences from the ...
2
votes
1answer
77 views
Do I need to use inheritance and classes for my OOP webscraper? [closed]
I am currently writing python code that scrapes information from the web.
I have to scrape several sites, but there a two types of procedures:
Directly scrape from the website
Download pdf and scrape ...
6
votes
2answers
72 views
Web scraper for driver's license test times
I have created a small selenium script that checks for available times to write a test for a drivers license. The program runs every minute and takes approx 50 seconds to run. I have noticed that it's ...
3
votes
2answers
75 views
6
votes
1answer
234 views
3
votes
1answer
115 views
Get future events from Pinnacle API
My program makes API requests to Pinnacle Sports to retrieve future events and saves them to the database.
Questions:
Do I manage the database resources correctly? It takes about 3 seconds to check ...
4
votes
1answer
64 views
Webscraping tennis data 1.1
I incorporated the substantial changes suggested in my previous question that involved building a web-scraper for gathering tennis data.
The improved code is shown below:
...
5
votes
1answer
72 views
FanFiction Downloader in Python, with GUI
I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I ...
8
votes
1answer
131 views
Webscraping tennis data
So as a starter Java project I decided to web scrape some data (specifically all historically No. 1 ranked players for weeks starting from 1973) from the ATP website, and do something with it (IPR). I'...
3
votes
1answer
28 views
Scrape every Java themed codingbat problem set and your personal solutions
I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized?
...
1
vote
1answer
101 views
Unofficial API for 1001tracklists.com
I wanted to build a scaping-API for the website 1001tracklists.com by writing classes for Tracklist and Track objects and some helpful functions:
Here's the full code (GitHub):
...
3
votes
2answers
83 views
Web Scraping Newspapers
Wrote a python script to web scrape multiple newspapers and arrange them in their respective directories. I have completed the course Using Python to access web data on coursera and I tried to ...
6
votes
1answer
72 views
Web scraper for restaurant ratings
I'm new to using type hinting in Python. I've used it for a small scraper I had to build (see code below), everything works fine and mypy gives no errors. However I'm sure there are better ways to ...
3
votes
1answer
71 views
Web Scraping Dynamically Generated Content Python
This is a small web scraping project I made in 2 hours that targets the website remote.co . I am looking forward for improvements in my code. I know about the inconsistency with the WebDriverWait and ...
2
votes
1answer
39 views
Refactoring string incrementing and duplicating function
I wrote this script and got what I wanted from it. However I'm looking for input on ways I can improve the function.
My goal was to scrape a bunch of image links from a website I'd like to rebuild. ...
5
votes
2answers
256 views
How can I make this better? Python Web scraping
I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any.
What do I want to achieve?
I want to scrape ...
2
votes
1answer
60 views
Text notifications for Facebook Marketplace posting
Application Summary
I received a call from a client asking for a "simple app" that notified him via text message whenever a "Jeep Wrangler" is posted to Facebook Marketplace. It ...
4
votes
1answer
98 views
Webscraping With Selenium - a Course Downloader and Sorter
I've been working on a program that automates the painful task of downloading each file from a website for courses one by one and sorting them using python. This is my first python webscraping project ...
3
votes
0answers
40 views
Scraping local news sites
This is my first Python web scraper (and overall my first Python project). I am also relatively new to OOP but do understand its core fundamentals. The script below scrapes two local news sites for ...
3
votes
1answer
61 views
Gold Price Tracker Web Scraper using Python
Hi I've written a web scraper for tracking gold price. Can someone please review my code and suggest improvements. It gets the price of gold from a specific url then stores the price against date in a ...
2
votes
1answer
299 views
Instagram Scraping Using Selenium - Download Posts - Photos - Videos
Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium ...
1
vote
0answers
33 views
Webscraping code to import logs from a website which is about to die
We all know that the famous twitch log site OverRustleLogs is getting shut down. So I decided to do some web scraping to download my favourite streamer's logs using BeautifulSoup. How can make this ...
1
vote
0answers
39 views
2
votes
1answer
77 views
A Text-Based Browser using Requests, Beautiful Soup, Sys, OS, and colorama
I have worked on this for quite a long while, and this is my 3rd attempt after completely demolishing 3 other 200 liners, and i really need ideas on how to make my code add more functionality or ...
5
votes
1answer
704 views
Scraping Instagram - Download posts, photos - videos
Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium ...
1
vote
1answer
59 views
Code structure of website API
I manage the Python API for the Manganelo (and Mangakalot) sites and I am not fully convinced that the code structure I currently use is the best. (https://github.com/nixonjoshua98/manganelo)
The API ...
5
votes
2answers
474 views
OOP Web-scraper w/ Python and BeautifulSoup
This is my first major web scraping program in python. My code works nonetheless, I'm just not sure if it's the best OOP design. My code is below:
...
2
votes
1answer
96 views
Download pictures (or videos) from Instagram using Selenium
Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium ...
2
votes
0answers
101 views
Forum web scraper in Lua
I've written a short web scraper in Lua (run in LuaJIT for speed) to download every post from a specific forum (not general purpose... yet). It takes about 12 hours to run and download 500,000 posts ...
3
votes
1answer
50 views
Command line dictionary tool, using webscraping
This is my first proper Go program, after completing Go by Example and https://tour.golang.org. I have a background in Python.
This program scrapes definitions from Wordnik, then prints them nicely ...
2
votes
1answer
192 views
Instagram Scraping Posts Using Selenium
Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder.
How it works:
Log in in instragram using selenium and navigate to ...
3
votes
1answer
36 views
Dynamic web scraper in Python
The other week I faced the struggle of scraping dynamically generated content. So I used the selenium library with combination of requests & bs4, the thing is I am unsure of the quality of the ...
1
vote
3answers
60 views
Scraping the Bundesliga table and saving it to CSV
Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance!
...
4
votes
1answer
72 views
Web Scraping with Python
My target website is Indeed. I tried to implement the scraper such that it is convenient for the end user. I am introducing my code in the readme file on my github repo.
I am somewhat a beginner in ...
