Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
2
votes
1answer
76 views

Data scraper in Python 3

This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
1answer
58 views

Crawling pages to obtain .htm and .txt files from sec.gov

I am pretty much a python newbie and was looking at this question on StackOverflow. Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
2
votes
0answers
235 views

Wrapping a web scraper in a RESTful API

The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
2
votes
1answer
139 views

Sourcing content from a webpage in an organized manner

I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
0answers
229 views

Scraping symfonycasts.com using BeautifulSoup

I'm learning python 3.x and I'm still feeling like a newbie. The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all ...
3
votes
0answers
271 views

Python web scraper

This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused. However, ...
3
votes
2answers
8k views

Python - Iterating over contents of BeautifulSoup element very slow

I have a large HTML document where everything is inside a main div, structured like this: ...
5
votes
1answer
4k views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
1answer
4k views

Web-scraping through a rotating proxy script

I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
5
votes
2answers
89 views

Scraping lunar/solar eclipse data with BeautifulSoup Followup

I decided to refactor my code based on this review. Eclipse: ...
11
votes
3answers
1k views

Web-scraping a list of lawyers

I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
3
votes
1answer
632 views

Web scraping news articles with Beautiful Soup

It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a ...
7
votes
2answers
166 views

Scraping lunar/solar eclipse data with BeautifulSoup

I created a script to scrape this website and gather the eclipse data (date, type, and location): ...
1
vote
0answers
157 views

Extracting data from a used car sales site

I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
6
votes
3answers
929 views

Scraping Clinical Studies database (via ClinicalTrials.Gov)

I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID. ...
2
votes
0answers
721 views

Recursively scrape links from web pages and check them

I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles. The overall idea ...
0
votes
1answer
156 views

Program to automate clicking on the new button on a chat website until the username matches certain conditions

This is a python program that automates skipping through users on a chat website called talkwithstrangers using the Selenium Webdriver and ...
4
votes
0answers
84 views

List all links in a website

I wrote this code as part of a project I'm working on. It's supposed to get all links from a website and then perform black-box tests on them. How can I improve this code to be faster and more ...
11
votes
2answers
589 views

Comic Image Web Scraper

I've written my first web scraper, which (surprisingly) does the job. I'm scraping a popular comic website for the images (there are over 950 of them) but the problem is the scraper is way too slow. ...
22
votes
3answers
4k views

Financial website scraper

I wrote the code that would scrap Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to python programming and therefore believe my code is ...
9
votes
1answer
567 views

A web scraper that looks for pre-defined words in news articles

I'm still fairly new to Python and web-scraping, but a colleague asked asked me if I could build a web-scraper that could be used by a think tank he's a member of to find news articles matching a pre-...
2
votes
0answers
972 views

Scrape webpage with Beautifulsoup and export relevant data to csv

I am looking for feedback on my code, how I can simplify some operations, use more efficient methods or other best practices. This is my first web scraping project. The goal here is to scrape a ...
5
votes
1answer
2k views

Recursive Web Scraping with Python Beautiful Soup

I wrote a short program which should allow a user to specify a starting page in Discogs Wiki Style Guide, scrape the other styles listed on the page, and then output a graph (represented here as a ...
4
votes
1answer
173 views

Scraping data from Bureau of Labor Statistics website

I'm currently scraping this website Bureau of Labor Statistics and I just want some feedback on how my approach was. I'm mainly focusing on the occupation groups table to the left of the website and ...
2
votes
1answer
2k views

Parsing HTML table into Pandas DataFrame

There is a text (link clickable) file with HTML table. The table is a bank statement. I'd like to parse it into pandas DataFrame. Is there a way to do it more gracefully? I've started to learn Python ...
7
votes
2answers
474 views

Items derived from one function are getting printed within another for loop in another function

I have written some code in python to scrape some items from a webpage. There is a link (titling see more) attach to each container. If I click on that link I can reach a certain page where all the ...
4
votes
2answers
117 views

Scrape a jobs board and keep persistent record

I'm trying to improve my pretty basic coding skills, and I've written this program to scrape a jobs board (I'd ultimately add several) and put the data in a db file with SQLITE3. The code works, but I ...
5
votes
2answers
516 views

Webscrape Podcast site for mp3s to (legally) add to collection

History I am new to Python and this is my first attempt. Utilizing a legal login (omitted from the code below) to the Bob and Tom VIP website, I have created this script to access their Full Show mp3....
3
votes
1answer
45 views

Exception handling of HTML parsing

I am currently working on a program that will scrape data from a psych database of various article metadata. Since not every piece is formatted perfectly, I have accepted that some of the info may not ...
0
votes
2answers
52 views

Web scraper for a webpage article

I am a beginner in Python and have just coded a simple web scraper for a webpage article, output to a text file, using BeautifulSoup and ...
2
votes
0answers
148 views

Football Web Scraper Part 2

A revised version of the code in this question. Things I have done so far: Adjusted some formatting things like constant & variable naming and indentation Wrapped most of the functions into a ...
4
votes
1answer
2k views

Web scraper for Football (Soccer) data with BeautifulSoup and Requests

I wrote a web scraper to get football scores from here. I'm getting the data for all seasons for the three major German leagues. It all works at the moment, but I'm sure it's possible to make it a lot ...
3
votes
1answer
82 views

Beating the queue

Inspired by this question on CodeReview, and because I'm never able to beat the queue. I'm just too slow I guess.. :( I made a little python utility that reads the queue's information every 30 ...
6
votes
1answer
193 views

Archives xkcd comics

This is a work in progress, as you may read I plan to revise to utilise JSON data rather than directly scraping. I am considering refactoring the try//...
-4
votes
1answer
417 views

Python - Performance of this code very slow [closed]

This is an excerpt of my html code ...
2
votes
1answer
122 views

Extracting top subreddits from an HTML table

I'm scraping a table and am trying to get nested td tags down the tbody tree but the code seems kind of verbose. Is there a more Pythonic way to do this? ...
4
votes
1answer
216 views

Py3 ebay-scraper - Project I need cheap GPU

I need cheap GPU's and decided to make a little ebay-scraper to make sure I can buy it at cheap prices. It works and I can search for (any) product to scrape the Title, Price and Shipping cost of that ...
1
vote
1answer
87 views

Console based English dictionary app

I'm new to programming. I wrote this following console based app in Python. It's an English dictionary app. If the user inputs the desired word that he/she wishes to search for, then this app should ...
5
votes
2answers
2k views

Python webscraping for lists of company names and details using BeautifulSoup

I just started learning Python and made a script that scrapes a corporate directory https://www.sccci.org.sg/en/directory/corporate/members So far it gets all the company names and their details ...
3
votes
3answers
564 views

Web-crawler for iTunes

I've written a script in Python to scrape app name, developer and price from ITunes site. In its first page there are 10 different recommended app links which my scraper is able to track down and ...
4
votes
2answers
15k views

Find all attributes for each HTML tag

This code outputs each tag followed by its attributes in the given HTML, both in lexicographic order. It must be possible to keep the spirit of the code without these many nested cycles and ...
10
votes
2answers
346 views

Console based Weather forecasting app

I'm new to programming. I wrote this following weather app in python. This app can provide simple weather forecast for three days. If the user provides the location in the correct format then it ...
4
votes
1answer
3k views

A tool to extract all links from a web-page in Python

I'm starting writing in Python and is tasked with writing a simple tool that opens a web-page, collects all the URLs (both complete like http://mail.com and ...
2
votes
1answer
245 views

URL List generator

The below code will read a downloaded HTML file. Uses BeautifulSoup and lxml to parse the document and extract all the URLs ...
2
votes
1answer
45 views

Finding the last successful route by ID using Python

I have a script that iterates through a series of URLs that have the same pattern to parse them via BeautifulSoup. The URL structure is an aspx page that ends its route with a sequential id, very ...
6
votes
1answer
102 views

Scraping the names of a whole category with a three-liner code

I've written a script in python using BeautifulSoup to parse the name of different coffee shops spreading across 51 pages in yellowpage website. I'm thrilled to see that it works perfectly. Except for ...
4
votes
1answer
137 views

Python 3 & BeautifulSoup4 - Food Hygiene Scraper to CSV file

I am very new to python and I am just learning things from tutorials online and code snippets. My code is below, it works in that it scrapes the data I want but I think I could make the code much ...
2
votes
1answer
848 views

First Python Script - Web Scraper with results saved to MySQL DB

I am working on getting more experience with Python so I made a basic Web Scraper, that saves it to MySQL. The code is currently working, but looking at it is very frustrating because it seems ...
4
votes
3answers
756 views

Simple RSS to HTML converter

Is this code pythonic, easy to read, stable, or missing something? ...
3
votes
2answers
999 views

Performance and Readability Improvements for HTML Parser with BeautifulSoup

This function takes as an argument a json file (could contain anything in json format, since I scrap hundreds of random pages) and returns a list of dictionaries where a URL is mapped to its ...