Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
41
votes
3answers
2k views

We'll be counting stars

Lately, I've been, I've been losing sleep Dreaming about the things that we could be But baby, I've been, I've been praying hard, Said, no more counting dollars We'll be counting stars, yeah we'll be ...
22
votes
5answers
1k views

An OEIS Lookup tool in Python

I'm from PPCG so I was making an esolang and I decided to write it in Python. Eventually it went from an esolang to an OEIS (Online Encyclopedia of Integer Sequences) lookup tool. I'm very new to ...
21
votes
3answers
4k views

Financial website scraper

I wrote the code that would scrap Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to python programming and therefore believe my code is ...
19
votes
4answers
5k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...
16
votes
6answers
1k views

Storytelling program: Gathers stories from certain popular story websites/formats and reads them aloud

Audiobooks make my life easier when I have idle ears and a mind free to wander, but something better for my eyes to do. This program is one of my more recent efforts to allow most of the stories ...
14
votes
3answers
748 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...
14
votes
2answers
795 views

VIM colors downloader in Python

Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors. So I ...
13
votes
1answer
2k views

Soup of the day: best served during election season

Community moderator elections on the Stack Exchange network are really exciting. Alas, on the page of the primaries, I find it mildly annoying that candidates are randomly reordered on every page load,...
13
votes
1answer
41k views

Getting data correctly from <span> tag with beautifulsoup and regex

I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned: ...
11
votes
2answers
374 views

Comic Image Web Scraper

I've written my first web scraper, which (surprisingly) does the job. I'm scraping a popular comic website for the images (there are over 950 of them) but the problem is the scraper is way too slow. ...
11
votes
3answers
1k views

Web-scraping a list of lawyers

I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
11
votes
1answer
966 views

Scraping my CS teacher's website, then emailing me when the site is updated

I've been working on creating an individual final project for my python CS class that checks my teacher's website on a daily basis and determines if he's changed any of the web pages on his website ...
10
votes
3answers
2k views

Scraping the date of most recent post from various social media services

Task I have a large spreadsheet where each line should include: The URL of a social media account A field indicating whether the account is "active" A name and UID number for each account I have to ...
10
votes
2answers
317 views

Console based Weather forecasting app

I'm new to programming. I wrote this following weather app in python. This app can provide simple weather forecast for three days. If the user provides the location in the correct format then it ...
10
votes
1answer
7k views

Adding a new class to HTML tag and writing it back with Beautiful Soup

I am working on an HTML document to which I need to add certain classes to some elements. In the following code, I am adding class img-responsive. ...
10
votes
1answer
468 views

River Flood Warning system in Python

This code represents my first real Python 3 program. It retrieves flood data from the NWS weather center for the river near my home and posts a warning to my Facebook page whenever certain flood ...
9
votes
2answers
2k views

Simple Python username scraper

I started learning Python recently and I really like it, so I decided to share one of my first projects mainly in hopes of someone telling me what I can do to make it run faster (threading/...
9
votes
2answers
634 views

Web crawler that filters out non diseases

It is very messy and I lack the experience to make it eloquent so I wanted some help with it. The process time is also very slow. Currently it goes into the first page and goes through the links in ...
8
votes
3answers
22k views

Downloading images from URLs

I'm just beginning with Python and programming, so been trying to get as much experience reading code as possible. The script mentioned below do grab images from URLs and put them in into a folder. ...
8
votes
1answer
165 views

A web scraper that looks for pre-defined words in news articles

I'm still fairly new to Python and web-scraping, but a colleague asked asked me if I could build a web-scraper that could be used by a think tank he's a member of to find news articles matching a pre-...
8
votes
2answers
102 views

Opening all the links

I am looking to improve my code structure. In this code it is opening all the links with ?pid= in the url. However, the structure of the code can be improved. <...
8
votes
2answers
138 views

Scraping lunar/solar eclipse data with BeautifulSoup

I created a script to scrape this website and gather the eclipse data (date, type, and location): ...
7
votes
2answers
467 views

Items derived from one function are getting printed within another for loop in another function

I have written some code in python to scrape some items from a webpage. There is a link (titling see more) attach to each container. If I click on that link I can reach a certain page where all the ...
7
votes
5answers
1k views

Finding the occurrences of all words in movie scripts

I was wondering if someone could tell me things I could improve in this code. This is one of my first Python projects. This program gets the script of a movie (in this case Interstellar) and then ...
7
votes
2answers
3k views

Using python and beautifulsoup to iterate through a list of websites to find a particular string

I'm attempting to find companies who mention a particular service in on their homepage. To do this, I am iterating through a csv file with two columns - ID and URL. I'm using BeautifulSoup to get the ...
7
votes
3answers
569 views

RateBeer.com scraper

This was largely an exercise in making my code more Pythonic, especially in catching errors and doing things the right way. I opted to make the PageNotFound ...
7
votes
1answer
224 views

Simple image scraping

I wrote this code over the last few days and I learned a lot, and it works as I expect. However I suspect it's woefully inefficient: ...
7
votes
2answers
708 views

Web Scraper in Python

So, this is my first web scraper (or part of it at least) and looking for things that I may have done wrong, or things that could be improved so I can learn from my mistakes. I made a few short ...
7
votes
2answers
4k views

YouTube Search Result Scraper

This is a program I wrote in Python using the BeautifulSoup library. The program scrapes YouTube search results for a given query and extracts data from the channels returned in the search results. ...
7
votes
2answers
2k views

Optimizing the speed of a web scraper

I have just written this code to scrape some data from a website. In its current state it works fine, however, going by my tests on the script, I discovered that with the amount of data I am ...
7
votes
2answers
12k views

Parsing locally stored HTML files

I am working with this code to parse through HTML files stored on my computer and extract HTML text by defining a certain tag that should be found: ...
7
votes
1answer
195 views

subreddit image bot

I wrote a bot in python 3 using PRAW, requests, and BeautifulSoup that downloads a specified amount of image posts from a specified subreddit. One limitation is that it can only parse the direct ...
7
votes
3answers
524 views

Scraping Clinical Studies database (via ClinicalTrials.Gov)

I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID. ...
6
votes
1answer
800 views

Reddit bot to check reposts

I am a moderator of /r/sweepstakes on Reddit, which lets users post their referral links to contests/sweepstakes. One main rule is that a user is not allowed to post their link to a contest, if ...
6
votes
2answers
57 views

Scraping columns from SEDE results

I use the following script to download the result of a SEDE query and scrape a specific column from it using BeautifulSoup: ...
6
votes
2answers
171 views

Covering array scraper and sorter

A covering array is an \$N \times k\$ array in which each element is a from a set of \$v\$ symbols, and for every \$t\$ columns, every possible set of \$v^t\$ choices of the symbols appears at least ...
6
votes
1answer
94 views

Scraping the names of a whole category with a three-liner code

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for ...
6
votes
1answer
171 views

Improving the performance of a webscraper

I have here a modified version of a web scraping code I wrote some weeks back. With some help from this forum, this modified version is faster (at 4secs per iteration) than the earlier version. ...
6
votes
1answer
124 views

Scraping SEDE query results with caching

I use this script to scrape the results of a SEDE page and return as a BeautifulSoup object. A small twist is that if I don't use a SEDE query manually in the browser for a few days, then non-...
6
votes
1answer
63 views

Print the list of winter bash 2014 hats as a list of checkboxes in GFM format

In Winter Bash 2014, since there is no easy way to see the hats I'm missing per site, I decided to use Gists for that. A perhaps not so well-known feature of GitHub Flavered Markdown (GFM) format ...
6
votes
1answer
791 views

Scraping efficiently with mechanize and bs4

I have written some code that scrapes data on asteroids, but the problem is that is super slow! I understand that it has a lot to scrape, but as of now it has been running for 5 days and is bot even a ...
6
votes
1answer
79 views

Scrape points from CTF sites

I am relatively new to classes. This one uses one. I am pretty sure this is not the correct way to do this. But I also do not know the correct way. You can use the ...
6
votes
1answer
62 views

Helper functions to extract SEDE query results into more user-friendly format

This Python module contains helper functions to download the result page of SEDE queries and extract columns from it, most prominently: fetch_sede_soup(label, url):...
5
votes
2answers
406 views

Webscrape Podcast site for mp3s to (legally) add to collection

History I am new to Python and this is my first attempt. Utilizing a legal login (omitted from the code below) to the Bob and Tom VIP website, I have created this script to access their Full Show mp3....
5
votes
2answers
1k views

Python webscraping for lists of company names and details using BeautifulSoup

I just started learning Python and made a script that scrapes a corporate directory https://www.sccci.org.sg/en/directory/corporate/members So far it gets all the company names and their details ...
5
votes
1answer
165 views

Archives xkcd comics

This is a work in progress, as you may read I plan to revise to utilise JSON data rather than directly scraping. I am considering refactoring the try//...
5
votes
1answer
3k views

Walmart scraping (crawling) code

I am currently trying to make a Walmart Scraper, where I have to scrape the title, link, image URL, price, category and subcategory of the product. Now my approach is that I will collect seed pages ...
5
votes
1answer
705 views

Parsing item page for an online shop

In this function I'm parsing the page of the item in the online shop. Some items lack the picture, some lack price etc, so there are few if-else checks. My ...
5
votes
1answer
641 views

Web crawler that charts stock ticker data using matplotlib

I've built a web crawler using the BeautifulSoup library that pulls stock ticker data from CSV files on Yahoo finance, and charts the data using ...
5
votes
2answers
84 views

Scraping lunar/solar eclipse data with BeautifulSoup Followup

I decided to refactor my code based on this review. Eclipse: ...