Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
-4
votes
1answer
351 views

Python - Performance of this code very slow [closed]

This is an excerpt of my html code ...
2
votes
1answer
99 views

Extracting top subreddits from an HTML table

I'm scraping a table and am trying to get nested td tags down the tbody tree but the code seems kind of verbose. Is there a more Pythonic way to do this? ...
4
votes
1answer
187 views

Py3 ebay-scraper - Project I need cheap GPU

I need cheap GPU's and decided to make a little ebay-scraper to make sure I can buy it at cheap prices. It works and I can search for (any) product to scrape the Title, Price and Shipping cost of that ...
1
vote
1answer
85 views

Console based English dictionary app

I'm new to programming. I wrote this following console based app in Python. It's an English dictionary app. If the user inputs the desired word that he/she wishes to search for, then this app should ...
5
votes
2answers
1k views

Python webscraping for lists of company names and details using BeautifulSoup

I just started learning Python and made a script that scrapes a corporate directory https://www.sccci.org.sg/en/directory/corporate/members So far it gets all the company names and their details ...
3
votes
3answers
449 views

Web-crawler for iTunes

I've written a script in Python to scrape app name, developer and price from ITunes site. In its first page there are 10 different recommended app links which my scraper is able to track down and ...
3
votes
2answers
12k views

Find all attributes for each HTML tag

This code outputs each tag followed by its attributes in the given HTML, both in lexicographic order. It must be possible to keep the spirit of the code without these many nested cycles and ...
10
votes
2answers
317 views

Console based Weather forecasting app

I'm new to programming. I wrote this following weather app in python. This app can provide simple weather forecast for three days. If the user provides the location in the correct format then it ...
3
votes
1answer
2k views

A tool to extract all links from a web-page in Python

I'm starting writing in Python and is tasked with writing a simple tool that opens a web-page, collects all the URLs (both complete like http://mail.com and ...
2
votes
1answer
220 views

URL List generator

The below code will read a downloaded HTML file. Uses BeautifulSoup and lxml to parse the document and extract all the URLs ...
2
votes
1answer
44 views

Finding the last successful route by ID using Python

I have a script that iterates through a series of URLs that have the same pattern to parse them via BeautifulSoup. The URL structure is an aspx page that ends its route with a sequential id, very ...
6
votes
1answer
94 views

Scraping the names of a whole category with a three-liner code

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for ...
4
votes
1answer
125 views

Python 3 & BeautifulSoup4 - Food Hygiene Scraper to CSV file

I am very new to python and I am just learning things from tutorials online and code snippets. My code is below, it works in that it scrapes the data I want but I think I could make the code much ...
2
votes
1answer
668 views

First Python Script - Web Scraper with results saved to MySQL DB

I am working on getting more experience with Python so I made a basic Web Scraper, that saves it to MySQL. The code is currently working, but looking at it is very frustrating because it seems ...
4
votes
3answers
579 views

Simple RSS to HTML converter

Is this code pythonic, easy to read, stable, or missing something? ...
3
votes
2answers
757 views

Performance and Readability Improvements for HTML Parser with BeautifulSoup

This function takes as an argument a json file (could contain anything in json format, since I scrap hundreds of random pages) and returns a list of dictionaries where a URL is mapped to its ...
-4
votes
1answer
1k views

Removal of Duplicate Urls in list [closed]

Here is a piece of code for acquiring URLS ...
8
votes
3answers
22k views

Downloading images from URLs

I'm just beginning with Python and programming, so been trying to get as much experience reading code as possible. The script mentioned below do grab images from URLs and put them in into a folder. ...
4
votes
1answer
2k views

Scraping all closed captions (subtitles) of a YouTube's creator's video library

A script that scrapes all YouTube's videos closed captions(subtitles) of a single creator's video library. It scrapes either auto-generated or manually written subtitles. The script accepts a link to ...
2
votes
0answers
49 views

River Flood Warning System v2.1 - Those Pesky NoneTypes

Below is my River Flood Warning System version 2 build 1. After following the help and advice given for version 1 the whole program is looking and behaving much better. The original code would only ...
7
votes
1answer
195 views

subreddit image bot

I wrote a bot in python 3 using PRAW, requests, and BeautifulSoup that downloads a specified amount of image posts from a specified subreddit. One limitation is that it can only parse the direct ...
0
votes
1answer
32 views

Extracting book information from a web page, with possible missing attributes

I'm just starting to code and I found a solution to first check if a div class exist before to try to get_text (and avoid an error if it doesn't). But the code ...
10
votes
1answer
468 views

River Flood Warning system in Python

This code represents my first real Python 3 program. It retrieves flood data from the NWS weather center for the river near my home and posts a warning to my Facebook page whenever certain flood ...
4
votes
1answer
4k views

Web scraping many URLs and writing to Excel file

I am using requests and BeautifulSoup to scrape 20000 URLs, each web page containing a table of information. Essentially each ...
4
votes
0answers
735 views

Speeding web-scraping up Python 3

I want to get information from two websites and display it on 'real-time' in the console. To get the information from the website I am using BeautifulSoup 4. I have read, that the bottleneck of ...
7
votes
2answers
3k views

Using python and beautifulsoup to iterate through a list of websites to find a particular string

I'm attempting to find companies who mention a particular service in on their homepage. To do this, I am iterating through a csv file with two columns - ID and URL. I'm using BeautifulSoup to get the ...
3
votes
1answer
212 views

March Madness Simulator

It's a march madness simulator for the final 4!!! Please be as critical as you can :) ...
6
votes
1answer
171 views

Improving the performance of a webscraper

I have here a modified version of a web scraping code I wrote some weeks back. With some help from this forum, this modified version is faster (at 4secs per iteration) than the earlier version. ...
7
votes
2answers
2k views

Optimizing the speed of a web scraper

I have just written this code to scrape some data from a website. In its current state it works fine, however, going by my tests on the script, I discovered that with the amount of data I am ...
3
votes
1answer
4k views

Convert BeautifulSoup4 HTML Table to a list of lists, iterating over each Tag elements

I am trying to convert a BeautifulSoup4 HTML Table to a list of lists, iterating over each Tag elements and handling them accordingly. I have an implementation of this that works at a surface level ...
4
votes
1answer
123 views

Python automate job ads search

I recently started to work with python and came to idea to automate process of looking for new jobs. There is nice website and I managed to get content, because they use angular to render webpage and ...
5
votes
1answer
70 views

Plotting some displays from a weather URL

I've got some code that plots some displays from a weather URL and locations in a CSV file. I'd like to see if anyone can make the code more efficient, the code runs fine without any errors and I'm ...
5
votes
1answer
604 views

Webscraping calendar events using Python 3, with or without BeautifulSoup

I'm trying to find out why my web-scraping code with BeautifulSoup (BS) is slower than my code without BS. I would think that BS code would be faster than the other code - so, maybe I'm doing ...
0
votes
3answers
424 views

Image hosting site image downloader using requests and BeautifulSoup

I went about this the way I have because Selenium is slow and inconvenient (opens browser) and I was unable to find the href attribute in the link element using ...
4
votes
2answers
897 views

Simple Python job vacancies downloader

I have created a BeautifulSoup vacancies parser, which works, but I do not like how it looks. Therefore, I'd be very happy if somebody could give me any improvements. ...
3
votes
1answer
877 views

Sample scraping Project Gutenberg using Beautiful Soup and requests

I am trying to learn web scraping in Python using Beautiful Soup and requests. My program goes to the book page on Project Gutenberg with the given book number (Example). It then finds the link for ...
2
votes
1answer
168 views

Downloading and saving news articles

To be honest, I am pretty new to coding. I want to analyse the articles my local newspaper publishes on ther website. To do that, I have a list of URLs and from that, I want to download and save all ...
6
votes
1answer
79 views

Scrape points from CTF sites

I am relatively new to classes. This one uses one. I am pretty sure this is not the correct way to do this. But I also do not know the correct way. You can use the ...
3
votes
2answers
172 views

Extract information from product pages

I'm new to Python, and just started to learn the principles of Object Oriented Programming, this does not judge strictly. This code is fully working but confused some elements: I don't understand ...
14
votes
2answers
794 views

VIM colors downloader in Python

Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors. So I ...
9
votes
2answers
2k views

Simple Python username scraper

I started learning Python recently and I really like it, so I decided to share one of my first projects mainly in hopes of someone telling me what I can do to make it run faster (threading/...
2
votes
2answers
2k views

A simple daily horoscope

I wrote a simple daily horoscope in Python using Beautiful Soup as scraper. It works fine but the code can use some sort of a... direction if you want to put it that way. I have used underscores in my ...
3
votes
1answer
88 views

Parses XML to return the bus times at specified bus stop

The code uses the XML-based OC Transpo data feed to create a list of the bus name, where it's headed, and the times. Keep in mind that I am a beginner at python so any advice at all is appreciated. <...
7
votes
1answer
224 views

Simple image scraping

I wrote this code over the last few days and I learned a lot, and it works as I expect. However I suspect it's woefully inefficient: ...
4
votes
2answers
73 views

I'll visit the 18th

I wrote this program, which purpose is to visit the 18th link on the list of links and then on the new page visit the 18th link again. This program works as intended, but it's a little repetitive and ...
10
votes
3answers
2k views

Scraping the date of most recent post from various social media services

Task I have a large spreadsheet where each line should include: The URL of a social media account A field indicating whether the account is "active" A name and UID number for each account I have to ...
7
votes
2answers
12k views

Parsing locally stored HTML files

I am working with this code to parse through HTML files stored on my computer and extract HTML text by defining a certain tag that should be found: ...
22
votes
5answers
1k views

An OEIS Lookup tool in Python

I'm from PPCG so I was making an esolang and I decided to write it in Python. Eventually it went from an esolang to an OEIS (Online Encyclopedia of Integer Sequences) lookup tool. I'm very new to ...
2
votes
1answer
121 views

Python webpage parser

I have a working code, but it has a lot of flaws. The main project is to get a value from a website. I need to complete this action million of times, so i download the webpage returned from the class ...
14
votes
3answers
748 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...