Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
7
votes
2answers
12k views

Parsing locally stored HTML files

I am working with this code to parse through HTML files stored on my computer and extract HTML text by defining a certain tag that should be found: ...
22
votes
5answers
1k views

An OEIS Lookup tool in Python

I'm from PPCG so I was making an esolang and I decided to write it in Python. Eventually it went from an esolang to an OEIS (Online Encyclopedia of Integer Sequences) lookup tool. I'm very new to ...
2
votes
1answer
121 views

Python webpage parser

I have a working code, but it has a lot of flaws. The main project is to get a value from a website. I need to complete this action million of times, so i download the webpage returned from the class ...
14
votes
3answers
749 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...
5
votes
1answer
890 views

Inserting products scraped from the web into a database

I am trying to speed up the runtime of my program as the API can make calls. However after adding the inserts to a database 3 mins turns in 7 mins after 180 API calls, which can be one a second. How ...
4
votes
1answer
179 views

HTML Scraper for Plex downloads page

I have written a scraper in Python 3 using Beautiful Soup 4 to retrieve the latest version of Plex Media Server from https://plex.tv, and I'd like some feedback on how to improve it. The HTML the ...
6
votes
1answer
62 views

Helper functions to extract SEDE query results into more user-friendly format

This Python module contains helper functions to download the result page of SEDE queries and extract columns from it, most prominently: fetch_sede_soup(label, url):...
5
votes
1answer
654 views

Web crawler that charts stock ticker data using matplotlib

I've built a web crawler using the BeautifulSoup library that pulls stock ticker data from CSV files on Yahoo finance, and charts the data using ...
6
votes
1answer
129 views

Scraping SEDE query results with caching

I use this script to scrape the results of a SEDE page and return as a BeautifulSoup object. A small twist is that if I don't use a SEDE query manually in the browser for a few days, then non-...
6
votes
2answers
58 views

Scraping columns from SEDE results

I use the following script to download the result of a SEDE query and scrape a specific column from it using BeautifulSoup: ...
7
votes
2answers
736 views

Web Scraper in Python

So, this is my first web scraper (or part of it at least) and looking for things that I may have done wrong, or things that could be improved so I can learn from my mistakes. I made a few short ...
-2
votes
1answer
96 views

Returning a tuple utilizing this script [closed]

The script below is taken from this site. It doesn't currently work, but I have made it work at my own computer (not currently accessible) by changing what BeautifulSoup looks for. The script is ...
1
vote
2answers
827 views

Remove (not extract) timestamp from HTML string

I've got strings coming from irregularly and ugly formatted HTML sites, that contain a timestamp. I am interested in removing the timestamp entirely and get all the rest. ...
8
votes
2answers
102 views

Opening all the links

I am looking to improve my code structure. In this code it is opening all the links with ?pid= in the url. However, the structure of the code can be improved. <...
4
votes
2answers
1k views

Extracting and normalizing URLs in an HTML document

I have written code to get all urls on a webpage & put them in a set, and would like tips on simple changes I can make to increase its performance. ...
5
votes
1answer
137 views

Scraping a table from Texas Dept. of Criminal Justice website

The script scrapes a table from the website mentioned, looks at the last 2 columns, takes that information, and then sorts it (and then returns the largest county, and the set of races and their ...
7
votes
1answer
823 views

Reddit bot to check reposts

I am a moderator of /r/sweepstakes on Reddit, which lets users post their referral links to contests/sweepstakes. One main rule is that a user is not allowed to post their link to a contest, if ...
13
votes
1answer
2k views

Soup of the day: best served during election season

Community moderator elections on the Stack Exchange network are really exciting. Alas, on the page of the primaries, I find it mildly annoying that candidates are randomly reordered on every page load,...
9
votes
2answers
635 views

Web crawler that filters out non diseases

It is very messy and I lack the experience to make it eloquent so I wanted some help with it. The process time is also very slow. Currently it goes into the first page and goes through the links in ...
16
votes
6answers
1k views

Storytelling program: Gathers stories from certain popular story websites/formats and reads them aloud

Audiobooks make my life easier when I have idle ears and a mind free to wander, but something better for my eyes to do. This program is one of my more recent efforts to allow most of the stories ...
7
votes
2answers
4k views

YouTube Search Result Scraper

This is a program I wrote in Python using the BeautifulSoup library. The program scrapes YouTube search results for a given query and extracts data from the channels returned in the search results. ...
6
votes
2answers
173 views

Covering array scraper and sorter

A covering array is an \$N \times k\$ array in which each element is a from a set of \$v\$ symbols, and for every \$t\$ columns, every possible set of \$v^t\$ choices of the symbols appears at least ...
2
votes
0answers
258 views

BeautifulSoup web spider for driver links

The following spider will grab some driver links, OS version, and the name. All the info is in a table class, but some pages might be a little different in the location and Number of cells in each row....
3
votes
1answer
457 views

Web-scraping Reddit Bot

I have been working on a web-scraping Reddit bot in Python 2.7 with the premise of going to /r/eve (a game sub-reddit) finding posts that contain a link to a website hosting killmail information (...
1
vote
3answers
102 views

Returning a list [closed]

The first two lines of The Zen of Python: Beautiful is better than ugly. Explicit is better than implicit. I'm writing a function: ...
2
votes
1answer
3k views

Using BeautifulSoup to scrape various tables and combine in a .csv file

A page contains a table of links, each link contains a table relevant to the link (a subject). Create a list of these links to pass to the function called ...
2
votes
2answers
94 views

Extracts marks of all students in class from website

This code extracts marks of all students in class and stores the result in a file results.txt, using BeautifulSoup. I'm looking for code review and suggestions. ...
2
votes
1answer
5k views

Check existence of HTML tags in a row

Using Python's Beautiful Soup, I analyze a number of html files (internally: soup). They are all a little different and my target comes in four different tags - or ...
11
votes
1answer
992 views

Scraping my CS teacher's website, then emailing me when the site is updated

I've been working on creating an individual final project for my python CS class that checks my teacher's website on a daily basis and determines if he's changed any of the web pages on his website ...
2
votes
1answer
234 views

Scraping through product pages

I'm working through a scraping function where pages of results lead to product pages. I've added a default maximum number of results pages, and pages per set of results, to prevent a simple mistake ...
2
votes
1answer
166 views

Little command line video grabber

I made a little Python3 script to grab video clips from a website for viewing off-line later and would like your critique, advice, tips & tricks and/or learning resources to create better code. (...
6
votes
1answer
64 views

Print the list of winter bash 2014 hats as a list of checkboxes in GFM format

In Winter Bash 2014, since there is no easy way to see the hats I'm missing per site, I decided to use Gists for that. A perhaps not so well-known feature of GitHub Flavered Markdown (GFM) format ...
7
votes
5answers
1k views

Finding the occurrences of all words in movie scripts

I was wondering if someone could tell me things I could improve in this code. This is one of my first Python projects. This program gets the script of a movie (in this case Interstellar) and then ...
6
votes
1answer
792 views

Scraping efficiently with mechanize and bs4

I have written some code that scrapes data on asteroids, but the problem is that is super slow! I understand that it has a lot to scrape, but as of now it has been running for 5 days and is bot even a ...
7
votes
3answers
593 views

RateBeer.com scraper

This was largely an exercise in making my code more Pythonic, especially in catching errors and doing things the right way. I opted to make the PageNotFound ...
1
vote
0answers
164 views

Extracting country data from HTML tables

I am trying to optimize/improve my coding skills in python The rationale behind this code is to extract some data from files and store them in a dictionary. I tried to comments lines to improve ...
4
votes
1answer
2k views

Web scraper running extremely slow

I am making my first web scraper in Python. It works great but it runs extremely slow. The website loads in about 10ms but it only does like 1 every couple of seconds. There are about 4-6 million ...
3
votes
1answer
10k views

Getting rid of certain HTML tags

This code simply returns a small section of HTML code and then gets rid of all tags except for break tags. It seems inefficient because you cannot search and replace with a beautiful soup object as ...
2
votes
1answer
449 views

Optimizing Beautiful Soup elements

I have this function which parses Beautiful Soup elements: ...
4
votes
1answer
247 views

Script for saving top wallpapers from wallbase.cc into a directory

I am newish to Python and I am looking for some extra eyes on a script I wrote. I wrote this script to learn more about web-scraping and using requests and Beautiful Soup. I use it to connect to the ...
5
votes
1answer
3k views

Walmart scraping (crawling) code

I am currently trying to make a Walmart Scraper, where I have to scrape the title, link, image URL, price, category and subcategory of the product. Now my approach is that I will collect seed pages ...
5
votes
1answer
716 views

Parsing item page for an online shop

In this function I'm parsing the page of the item in the online shop. Some items lack the picture, some lack price etc, so there are few if-else checks. My ...
2
votes
2answers
2k views

Number of Google search results over a period of time, saved to database

I am writing a Python script that scrapes data from Google search results and stores it in a database. I couldn't find any Google API for this, so I am just sending a HTTP GET request on Google's main ...
41
votes
3answers
2k views

We'll be counting stars

Lately, I've been, I've been losing sleep Dreaming about the things that we could be But baby, I've been, I've been praying hard, Said, no more counting dollars We'll be counting stars, yeah we'll be ...
13
votes
1answer
43k views

Getting data correctly from <span> tag with beautifulsoup and regex

I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned: ...
3
votes
1answer
10k views

eBay scraper with BeautifulSoup

A friend of mine needs a scraper that reads product titles, prices and pictures from eBay and saves them to an Excel-ready .csv file. The program reads product pages into a list and then loops through ...
10
votes
1answer
7k views

Adding a new class to HTML tag and writing it back with Beautiful Soup

I am working on an HTML document to which I need to add certain classes to some elements. In the following code, I am adding class img-responsive. ...
4
votes
2answers
8k views

Scraping HTML using Beautiful Soup

I have written a script using Beautiful Soup to scrape some HTML and do some stuff and produce HTML back. However, I am not convinced with my code and I am looking for some improvements. Structure of ...
4
votes
1answer
181 views

Opening TV episodes in a web browser

I put this together and it seems to work. 1channel.py -breakingbad 1 The Output (1-10): (1) - Currently 3.80/5 (2) - Currently 3.50/5 (1 opens the link in ...
3
votes
2answers
6k views

Beautifulsoup scraper for sport events

I've written a simple scraper that parses HTML using BeautifulSoup and collects the data (schedule of sports events), then clubs them together in a list of dicts. The code works just fine, but the ...