Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
13 votes
3 answers
2k views

AniPop - The anime downloader

Note: The topics of performance and Selenium/BS4 have not yet been addressed, so this question can still receive a better answer! Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
T145's user avatar
  • 3,099
8 votes
1 answer
224 views

Webscraping tennis data

So as a starter Java project I decided to web scrape some data (specifically all historically No. 1 ranked players for weeks starting from 1973) from the ATP website, and do something with it (IPR). I'...
cloudy_eclispse's user avatar
5 votes
1 answer
708 views

Instagram scraper Posts (Videos and Photos)

I wrote this code which has the ability to download images and videos from a specific Instagram profile. Using multiprocessing and threading I managed to speed up the extraction of data. My goal is ...
AlexDotis's user avatar
  • 417
3 votes
1 answer
4k views

Scraping Instagram with selenium, extract URLs, download posts

I made a very simple Instagram Bot that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder. How it works: Creating directory for saving ...
AlexDotis's user avatar
  • 417
6 votes
1 answer
3k views

Scraping Instagram - Download posts, photos - videos

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
AlexDotis's user avatar
  • 417
6 votes
1 answer
561 views

Web scraper that extracts urls from Amazon and eBay

Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case of previously saved files, the files will be read and no ...
watch-this's user avatar
5 votes
0 answers
2k views

Scraping OddsPortal with requests only

This is a scraper written to do most of what had been attempted by another user in this question: How can I optimise this webscraping code I did the rewrite because I felt bad that the new user didn't ...
Reinderien's user avatar
  • 62.1k
4 votes
1 answer
75 views

Webscraping tennis data 1.1

I incorporated the substantial changes suggested in my previous question that involved building a web-scraper for gathering tennis data. The improved code is shown below: ...
cloudy_eclispse's user avatar
3 votes
1 answer
778 views

Web scraping using selenium, multiprocessing, InstagramBot

An Instagram Bot which downloads the posts from profile I have to mention my previous posts: Instagram scraper Posts (Videos and Photos) Scraping Instagram with selenium, extract URLs, download ...
AlexDotis's user avatar
  • 417
2 votes
1 answer
1k views

Instagram Scraping Posts Using Selenium

Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium and navigate to ...
AlexDotis's user avatar
  • 417
2 votes
1 answer
725 views

Instagram Scraping Using Selenium

Python script that can download images and videos of the user, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium and navigate to ...
AlexDotis's user avatar
  • 417
2 votes
1 answer
551 views

Download pictures (or videos) from Instagram using Selenium

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
AlexDotis's user avatar
  • 417
1 vote
1 answer
983 views

Instagram Bot, selenium, web scraping

I made some changes in my code from the previous post. The changes that I made: I put all the functions to the class All the global arrays I moved them to class too Created ...
AlexDotis's user avatar
  • 417
44 votes
3 answers
2k views

We'll be counting stars

Lately, I've been, I've been losing sleep Dreaming about the things that we could be But baby, I've been, I've been praying hard, Said, no more counting dollars We'll be counting stars, yeah we'll be ...
Simon Forsberg's user avatar
20 votes
4 answers
10k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...
austingae's user avatar
  • 1,080
14 votes
2 answers
842 views

VIM colors downloader in Python

Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors. So I ...
Kernel.Panic's user avatar
12 votes
3 answers
1k views

Minimal webcrawler - bad structure and error handling?

I did this code over one day as a part of a job application, where they wanted me to make a minimal webcrawler in any language. The purpose was to crawl a site, find all of the URLs on that page, and ...
bjornasm's user avatar
  • 417
10 votes
1 answer
883 views

Let's read a random Goodreads book in an optimal way

I have made the following program to gather data on random books from Goodreads, via their random books feature. ...
esote's user avatar
  • 3,770
10 votes
1 answer
706 views

River Flood Warning system in Python

This code represents my first real Python 3 program. It retrieves flood data from the NWS weather center for the river near my home and posts a warning to my Facebook page whenever certain flood ...
Aaron Nelson's user avatar
10 votes
2 answers
418 views

Financial Data From Webqueries in Excel

I'm new (to CR and to programming in general). I wrote my first VBA on Monday. This is my first working project. It Takes a bunch of financial data from a company called Financial Analytics and a ...
BlackHatGuy's user avatar
9 votes
1 answer
3k views

A library for interacting with Pinnacle Sports Bets API

My code provides the following functionality for interacting with Pinnacle Bets API: retrieving betting history retrieving fixtures (future events) retrieving odds for the given leagues (competitions)...
Konstantin Kostanzhoglo's user avatar
9 votes
1 answer
222 views

Chart showing the Percentage of Answered CR questions

Every day Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor (so we ...
Sᴀᴍ Onᴇᴌᴀ's user avatar
8 votes
1 answer
169 views

C# Profile Scraper

I'm a 16-year-old so I never studied programming, although I do try to adopt the best practices I can when given the chance. I would like to clean up the code posted below by making it comply with ...
Josh Hallow's user avatar
  • 1,345
8 votes
2 answers
2k views

Parsing Wikipedia table with Python

I am new to Python and recently started exploring web crawling. The code below parses the S&P 500 List Wikipedia page and writes the data of a specific table into a database. While this script is ...
DatenBergwerker's user avatar
8 votes
2 answers
10k views

Fast(er) web scraping with VBA

I have a code that fetches rates from a website called X-Rates, and outputs to excel the monthly averages of a chosen country. The code runs quite fast, but I still think I could improve the code a ...
svacx's user avatar
  • 433
8 votes
2 answers
4k views

Finding words that rhyme

Preface I was trying to review this question on the same topic, but in the end many points I wanted to make were excellently explained by @ferada so I felt that posting my code and explaining the ...
Caridorc's user avatar
  • 27.5k
8 votes
2 answers
2k views

Image downloader for a website v2

This code takes a website and downloads all .jpg images in the webpage. It supports only websites that have the img element and src contains a .jpg link. The previous version can be found here ...
Salah Eddine's user avatar
6 votes
3 answers
10k views

Email a notification when detecting changes on a website

The text of a website is checked in a given time period. If there are any changes a mail is sent. There is a option to show/mail the new parts in the website. What could be improved? ...
questionanswer's user avatar
6 votes
3 answers
707 views

Scraping a webpage copying with the logic of scrapy

Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be ...
SIM's user avatar
  • 2,501
6 votes
2 answers
2k views

Retrieving stock prices

It takes around 5-8 seconds for me to retrieve a previously-closed stock price and a dividend rate from US Yahoo! Finance. If I wanted to retrieve 10+ stock prices, it would take me more than a minute ...
pexpex223's user avatar
  • 163
5 votes
1 answer
610 views

Ultra fast Amazon scraper multi-threaded

This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
watch-this's user avatar
5 votes
1 answer
219 views

Richelieu - product scraper

I wanted to see how would I deal with a large amount of data being scraped and written into a CSV file, so I decided to get the info out of a random website. First off, I found a way to search for ...
Cajuu''s user avatar
  • 331
5 votes
1 answer
2k views

Web scraping VBA - Internet Explorer

The code below extracts data from one web page - I emulate search, select all results from the list and when the list appears (42000 items) I loop through these items. I get an id value from their ...
Ryszard Jędraszyk's user avatar
4 votes
2 answers
10k views

Scraping HTML using Beautiful Soup

I have written a script using Beautiful Soup to scrape some HTML and do some stuff and produce HTML back. However, I am not convinced with my code and I am looking for some improvements. Structure of ...
avi's user avatar
  • 973
4 votes
1 answer
2k views

Web scraper for Football (Soccer) data with BeautifulSoup and Requests

I wrote a web scraper to get football scores from here. I'm getting the data for all seasons for the three major German leagues. It all works at the moment, but I'm sure it's possible to make it a lot ...
iuvbio's user avatar
  • 442
4 votes
2 answers
335 views

PHP script that writes a JSON file with iextrading API data

iextrading has two APIs (v1, v2) which provide financial market data. Following script records a JSON file with their equity information (using a CRON job). Would you be kind and review it for any ...
Emma's user avatar
  • 3,527
4 votes
1 answer
899 views

Cleaner way of appending data to List in BeautifulSoup

So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scraper to collect data such as <...
Minial's user avatar
  • 229
4 votes
1 answer
411 views

Beginner web scraper for Nagios

I am attempting to learn Python. It was suggested to me to try a web scraper, so I thought to get myself to look at multiple Nagios instances. I have not programmed in Python before, but learned from ...
Canadian Luke's user avatar
4 votes
0 answers
125 views

The anime downloader [duplicate]

NOTE: Here's the latest version of this program, since this question idled out. This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
T145's user avatar
  • 3,099
4 votes
1 answer
1k views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
Sati's user avatar
  • 417
4 votes
1 answer
621 views

Fetching specific foreign exchange rates from fixer using curl and jsconcpp in C++

I am trying to create my own algorithmic trading system using C++. I have searched the web for a nice tutorial for such systems and I didn't find any. Then I started to learn about ...
Eka's user avatar
  • 349
3 votes
0 answers
183 views

requests vs selenium vs scrapy

This is a follow-up of my question over here. I have been working on my web-scraping techniques by trying out different approaches and rewriting code for a handful of online databases. I've recently ...
Sati's user avatar
  • 417
3 votes
1 answer
95 views

Collect Pokémon from a URL and store them in a dataframe

...
Ashok's user avatar
  • 31
3 votes
1 answer
598 views

Web crawlers for three image sites

I'm very new to python and only vaguely remember OOP from doing some Java a few years ago so I don't know what the best way to do this is. I've build a bunch of classes that represent a crawler that ...
davegri's user avatar
  • 153
3 votes
2 answers
2k views

Instagram Scraping Using Selenium - Download Posts - Photos - Videos

Python script that can downloads public and private profiles images and videos, like Gallery with photos or videos. It saves the data in the folder. How it works: Log in in instragram using selenium ...
AlexDotis's user avatar
  • 417
3 votes
1 answer
138 views

Reaching the philosophy wiki page

I've written a class that will start from a random Wikipedia page, then choose the first link in the main body, and then navigate following the links until it finds the Philosophy page. When I run the ...
loremIpsum1771's user avatar
3 votes
1 answer
153 views

Reaching the philosophy wiki page - Follow Up

This is a follow up to my original post: I've written a class that will start from a random Wikipedia page, then choose the first link in the main body, and then navigate following the links until ...
loremIpsum1771's user avatar
3 votes
3 answers
2k views

Image downloader for a website

This code takes a website and downloads all .jpg images in the webpage. It supports only websites that have the <img> element and ...
Salah Eddine's user avatar
3 votes
1 answer
500 views

Scraping current day Counter-Strike match results from a website

As a fan of competitive Counter-Strike, I like to keep up with who is currently winning and who is losing. There is a website that provides me with just that. I thought it would be cool if I could ...
Luke's user avatar
  • 1,100
3 votes
2 answers
5k views

VBA - XMLHTTP web scraping

I navigate with IE, do various things, then select all results option from a list and fire on click event. Once all results have been listed, I loop through their URLs, using the following code to ...
Ryszard Jędraszyk's user avatar