Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [web-scraping]

Web scraping is the use of a program to simulate human interaction with a web server or to extract specific information from a web page.

Filter by
Sorted by
Tagged with
44 votes
3 answers
2k views

We'll be counting stars

Lately, I've been, I've been losing sleep Dreaming about the things that we could be But baby, I've been, I've been praying hard, Said, no more counting dollars We'll be counting stars, yeah we'll be ...
Simon Forsberg's user avatar
33 votes
7 answers
9k views

Tiny image scraper for xkcd.com

This came out on the spur of the moment, as a quick and dirty tool to have the job done. This is a simple image scraper for immensely popular and legendary comic website inspired by a Python Easter ...
baduker's user avatar
  • 1,348
21 votes
5 answers
2k views

An OEIS Lookup tool in Python

I'm from PPCG so I was making an esolang and I decided to write it in Python. Eventually it went from an esolang to an OEIS (Online Encyclopedia of Integer Sequences) lookup tool. I'm very new to ...
Downgoat's user avatar
  • 902
21 votes
2 answers
10k views

Spliterator implementation

I'm trying to post a little tutorial on the new Spliterator class. There are many tutorials these days on using stream starting from a standard Java collection, but ...
trapo's user avatar
  • 313
20 votes
4 answers
10k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...
austingae's user avatar
  • 1,080
17 votes
1 answer
15k views

Scraping after login using Scrapy

I just finished a scraper in python using scrapy. The scraper logs in to a certain page and then scrapes a list of other pages using the authenticated session. It retrieves the title of these pages ...
OutOfTheBox's user avatar
16 votes
1 answer
1k views

Simple xkcd comic downloader

I'd really appreciate some harsh/constructive criticism of what I would consider as my first program in Haskell. The program should download all of the xkcd comics into a folder in the current ...
adrian's user avatar
  • 371
15 votes
2 answers
4k views

Web crawler in Rust

I am learning Rust. I have written a web crawler that would scrape all the pages from my own blog (which is running on Ghost) and would generate a static version of it. Because of this, I'm not ...
rolisz's user avatar
  • 253
15 votes
2 answers
37k views

Scrape an HTML table with python

I think I'm on the right track, but ANY suggestions or critques are welcome. The program just scrapes an HTML table and prints it to stdout. ...
Creek's user avatar
  • 305
15 votes
1 answer
60k views

Getting data correctly from <span> tag with beautifulsoup and regex

I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned: ...
avi's user avatar
  • 973
14 votes
2 answers
842 views

VIM colors downloader in Python

Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors. So I ...
Kernel.Panic's user avatar
14 votes
2 answers
290 views

OLog Userscript - Logging messages, planets and researches

For the online text-based browser game OGame I am working on an application with as aim to assist the users where possible, for this I have a server-side part and a client-side part, the respective ...
skiwi's user avatar
  • 10.6k
14 votes
2 answers
754 views

Nokogiri crawler

The following code works but is a mess. But being totally new to Ruby I have had big problems trying to refactor it into something resembling clean OOP code. Could you help with this and explain what ...
Dave Gordon's user avatar
14 votes
2 answers
861 views

Getting to Wikipedia's "Philosophy" article using Python

On Wikipedia, if you click the first non-italicised internal link in the main text of an article that's not within parentheses, and then repeat the process, you usually end up on the "Philosophy" ...
Sumit's user avatar
  • 241
13 votes
3 answers
2k views

AniPop - The anime downloader

Note: The topics of performance and Selenium/BS4 have not yet been addressed, so this question can still receive a better answer! Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
T145's user avatar
  • 3,099
13 votes
1 answer
2k views

Soup of the day: best served during election season

Community moderator elections on the Stack Exchange network are really exciting. Alas, on the page of the primaries, I find it mildly annoying that candidates are randomly reordered on every page load,...
janos's user avatar
  • 111k
13 votes
1 answer
363 views

Regex-guided crawler that downloads regex-matching images up to a crawling level

This is one simple crawler that downloads images from websites, the website's URL to be crawled to must match the regex, as well as any image-to-download's URL. (Also, I know, I made my own thread ...
wallabra's user avatar
  • 789
12 votes
3 answers
1k views

Minimal webcrawler - bad structure and error handling?

I did this code over one day as a part of a job application, where they wanted me to make a minimal webcrawler in any language. The purpose was to crawl a site, find all of the URLs on that page, and ...
bjornasm's user avatar
  • 417
12 votes
1 answer
20k views

Scraping the full content from a lazy-loading webpage

I've written a script in python in combination with selenium which is able to scrape 1000 links from a webpage in which lazy-loading method is applied for that reason it displays it's content 20 at a ...
SIM's user avatar
  • 2,501
11 votes
3 answers
703 views

Asynchronously scrape fabric information

This is my first asyncio/aiohttp web scraper. I am trying to wrap my head around Python's asyncio/aiohttp libs these days and I am not sure if I fully understand it or not yet. So I'd like have some ...
dougj's user avatar
  • 111
11 votes
2 answers
1k views

Comic Image Web Scraper

I've written my first web scraper, which (surprisingly) does the job. I'm scraping a popular comic website for the images (there are over 950 of them) but the problem is the scraper is way too slow. ...
baduker's user avatar
  • 1,348
11 votes
3 answers
2k views

Web-scraping a list of lawyers

I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
SIM's user avatar
  • 2,501
11 votes
1 answer
944 views

Is this the Clojure way to web-scrape a book cover image?

Is there a way to write this better or more Clojure way? Especially the last part with with-open and the let. Should I put the <...
f00860's user avatar
  • 213
11 votes
1 answer
1k views

Scraping my CS teacher's website, then emailing me when the site is updated

I've been working on creating an individual final project for my python CS class that checks my teacher's website on a daily basis and determines if he's changed any of the web pages on his website ...
cody.codes's user avatar
  • 1,935
10 votes
2 answers
1k views

DeliciousSoda: A simple parser for robots.txt files

This is my first big project that I've decided to release publically. It's available on my Github, and through pip3: ...
Linny's user avatar
  • 10.1k
10 votes
1 answer
883 views

Let's read a random Goodreads book in an optimal way

I have made the following program to gather data on random books from Goodreads, via their random books feature. ...
esote's user avatar
  • 3,770
10 votes
3 answers
4k views

Scraping the date of most recent post from various social media services

Task I have a large spreadsheet where each line should include: The URL of a social media account A field indicating whether the account is "active" A name and UID number for each account I have to ...
A_S00's user avatar
  • 203
10 votes
2 answers
418 views

Financial Data From Webqueries in Excel

I'm new (to CR and to programming in general). I wrote my first VBA on Monday. This is my first working project. It Takes a bunch of financial data from a company called Financial Analytics and a ...
BlackHatGuy's user avatar
10 votes
1 answer
7k views

Simple yet Hackable WhatsApp API [UNOFFICIAL]

There is no official WhatsApp API. Here is a simple python class which satisfies the need. ...
VISWESWARAN NAGASIVAM's user avatar
10 votes
1 answer
706 views

River Flood Warning system in Python

This code represents my first real Python 3 program. It retrieves flood data from the NWS weather center for the river near my home and posts a warning to my Facebook page whenever certain flood ...
Aaron Nelson's user avatar
10 votes
1 answer
364 views

Strip Dilbert Comics

A friend of mine is a huge fan of a well-known comic website and he recently said that it'd be great to have a tool that downloads comic strips for a given date range, instead of clicking through the ...
baduker's user avatar
  • 1,348
10 votes
1 answer
6k views

The YouTube crawler

I have coded a program to scrape YouTube data (for educational purposes). When the link of the channel is entered it scrapes the channel name, description of the channel, the videos posted by the ...
VISWESWARAN NAGASIVAM's user avatar
10 votes
1 answer
6k views

Node.js parallel file download, the ES6 way

I wrote a script that downloads all PDFs found on the web page of a particular government agency. I would have chosen bash for such a task, but I want the script to ...
Nicolas Raoul's user avatar
9 votes
2 answers
2k views

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user avatar
9 votes
2 answers
3k views

Simple Python username scraper

I started learning Python recently and I really like it, so I decided to share one of my first projects mainly in hopes of someone telling me what I can do to make it run faster (threading/...
edsheeran's user avatar
  • 125
9 votes
3 answers
2k views

Web Scraping with VBA

I wrote this to scrape album review data from AOTY into a spreadsheet. Check it out and let me know what I could've done better. ...
ForrestA's user avatar
  • 313
9 votes
2 answers
643 views

Web crawler that filters out non diseases

It is very messy and I lack the experience to make it eloquent so I wanted some help with it. The process time is also very slow. Currently it goes into the first page and goes through the links in ...
Kunal Sondhi's user avatar
9 votes
1 answer
222 views

Chart showing the Percentage of Answered CR questions

Every day Duga reports the number of unanswered questions and percentage of answered questions (after she calls the API endpoint /info) in The 2nd Monitor (so we ...
Sᴀᴍ Onᴇᴌᴀ's user avatar
9 votes
1 answer
1k views

A web scraper that looks for pre-defined words in news articles

I'm still fairly new to Python and web-scraping, but a colleague asked asked me if I could build a web-scraper that could be used by a think tank he's a member of to find news articles matching a pre-...
DanielH's user avatar
  • 177
9 votes
1 answer
6k views

Scraping Instagram for Hashtag data

Over the past few months, I've been actively using python and I have made a few scripts to scrape #hashtag data from Instagram. It all started with some basic script I had made early 2017 and I have ...
ThomasSt's user avatar
  • 193
9 votes
1 answer
3k views

Scan a webpage to find the start time and date for an event

I am working on a simple web crawler that returns the start time and date for an event listed on a webpage. The webpage can be in two different formats and there are multiple other dates listed on the ...
BruceM's user avatar
  • 93
9 votes
1 answer
3k views

A library for interacting with Pinnacle Sports Bets API

My code provides the following functionality for interacting with Pinnacle Bets API: retrieving betting history retrieving fixtures (future events) retrieving odds for the given leagues (competitions)...
Konstantin Kostanzhoglo's user avatar
8 votes
3 answers
654 views

Function in Python to extract web Data

I developed this feature that I think can be improved quite a bit. The result is the desired one, but it took me many lines of code. Any idea to optimize it? ...
Raymont's user avatar
  • 215
8 votes
2 answers
2k views

Image downloader for a website v2

This code takes a website and downloads all .jpg images in the webpage. It supports only websites that have the img element and src contains a .jpg link. The previous version can be found here ...
Salah Eddine's user avatar
8 votes
1 answer
6k views

Refactoring a Crawler

I've recently ported an old project and made it object-oriented. However, I've noticed that rubocop points out the following status: ...
user avatar
8 votes
1 answer
2k views

Scraping new product data from an online store

I get a lot done with Python scripts but always feel like I'm working with a messy desk when I'm using it. I assume this is because I am not coding in Python correctly. Coming from a C# and Ruby ...
Ecnalyr's user avatar
  • 413
8 votes
2 answers
2k views

Parsing Wikipedia table with Python

I am new to Python and recently started exploring web crawling. The code below parses the S&P 500 List Wikipedia page and writes the data of a specific table into a database. While this script is ...
DatenBergwerker's user avatar
8 votes
2 answers
6k views

Pure Python script that saves an HTML page with all images

Here is a pure Python script that saves an HTML page without CSS but with all images on it and replaces all hrefs with a path of an image on the hard drive. I know that there are great libraries like ...
micgeronimo's user avatar
8 votes
2 answers
600 views

Microservice that fetches data from REST repository endpoints on Github

I have code in Python 3 where I fetch data from the GitHub API, and I want some advice on the architecture of this code. ...
E.Mohammed's user avatar
8 votes
2 answers
4k views

Finding words that rhyme

Preface I was trying to review this question on the same topic, but in the end many points I wanted to make were excellently explained by @ferada so I felt that posting my code and explaining the ...
Caridorc's user avatar
  • 27.5k

1
2 3 4 5
13