Questions tagged [beautifulsoup]
Beautiful Soup is a Python library for pulling data out of HTML and XML files.
146
questions
2
votes
1answer
30 views
Beautifulsoup and lxml (xpath) too slow with respect to regex when parsing HTML
I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.
The problem is that any alternatives are really too slow....
3
votes
1answer
47 views
the fox is black - a simple image scraper
I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
41 views
Scraping a hiring website using python's requests and BeautifulSoup
I'm designing a scraping application using python, requests and BeautifulSoup4.
I decided to divide the logic into two classes:
Spider : gets the base url ...
2
votes
0answers
23 views
Scraping Forum Tables w/ Links Using Beautiful Soup
This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
5
votes
0answers
53 views
Get a movie directory, rename it and save imdb info as a webpage
I have my movie folders formatted:
~: no sub included
{}: already watched
nothing: sub included, ready to watch
the program asks for the movie directory; extracts the name and searches IMDb for the ...
3
votes
1answer
337 views
Python script to download adult comics from 8muses
I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the ...
4
votes
1answer
48 views
Eliminate unnecessary looping while parsing cable modem status table
I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and ...
3
votes
1answer
66 views
Web scraping job postings from Facebook
I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
56 views
Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)
The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
201 views
NFL data scraper
I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
3
votes
1answer
63 views
Pulls href tags using BeautifulSoup with Python
As if this isn't extremely obvious, I am a new coder.
My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
68 views
Google Analytic Webcrawler for every country code
At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
1
vote
1answer
60 views
Webcrawler for a car forum
I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and ...
4
votes
1answer
104 views
Cleaner way of appending data to List in BeautifulSoup
So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
19
votes
4answers
5k views
Web scraping the titles and descriptions of trending YouTube videos
This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?
...
1
vote
0answers
91 views
Twitter scraper using tweepy
I wrote a Twitter scraper using the tweepy so I can scrape user information and tweets. Given that the free API doesn't let me get the number of messages per tweet, I had to rely on BeautifulSoup to ...
2
votes
1answer
69 views
Data scraper in Python 3
This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
1answer
53 views
Crawling pages to obtain .htm and .txt files from sec.gov
I am pretty much a python newbie and was looking at this question on StackOverflow.
Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
2
votes
0answers
162 views
Wrapping a web scraper in a RESTful API
The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
1
vote
1answer
133 views
Sourcing content from a webpage in an organized manner
I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
0answers
149 views
Scraping symfonycasts.com using BeautifulSoup
I'm learning python 3.x and I'm still feeling like a newbie.
The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all ...
3
votes
0answers
136 views
Python web scraper
This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused.
However, ...
3
votes
2answers
3k views
Python - Iterating over contents of BeautifulSoup element very slow
I have a large HTML document where everything is inside a main div, structured like this:
...
5
votes
1answer
2k views
Python scraping tables
I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
1answer
2k views
Web-scraping through a rotating proxy script
I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
5
votes
2answers
84 views
Scraping lunar/solar eclipse data with BeautifulSoup Followup
I decided to refactor my code based on this review.
Eclipse:
...
11
votes
3answers
1k views
Web-scraping a list of lawyers
I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
2
votes
1answer
472 views
Web scraping news articles with Beautiful Soup
It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a ...
8
votes
2answers
138 views
Scraping lunar/solar eclipse data with BeautifulSoup
I created a script to scrape this website and gather the eclipse data (date, type, and location):
...
1
vote
0answers
91 views
Extracting data from a used car sales site
I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
7
votes
3answers
529 views
Scraping Clinical Studies database (via ClinicalTrials.Gov)
I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID.
...
2
votes
0answers
516 views
Recursively scrape links from web pages and check them
I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles.
The overall idea ...
0
votes
1answer
139 views
Program to automate clicking on the new button on a chat website until the username matches certain conditions
This is a python program that automates skipping through users on a chat website called
talkwithstrangers using the Selenium Webdriver and ...
4
votes
0answers
77 views
List all links in a website
I wrote this code as part of a project I'm working on. It's supposed to get all links from a website and then perform black-box tests on them. How can I improve this code to be faster and more ...
11
votes
2answers
375 views
Comic Image Web Scraper
I've written my first web scraper, which (surprisingly) does the job. I'm scraping a popular comic website for the images (there are over 950 of them) but the problem is the scraper is way too slow.
...
21
votes
3answers
4k views
Financial website scraper
I wrote the code that would scrap Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to python programming and therefore believe my code is ...
8
votes
1answer
171 views
A web scraper that looks for pre-defined words in news articles
I'm still fairly new to Python and web-scraping, but a colleague asked asked me if I could build a web-scraper that could be used by a think tank he's a member of to find news articles matching a pre-...
2
votes
0answers
852 views
Scrape webpage with Beautifulsoup and export relevant data to csv
I am looking for feedback on my code, how I can simplify some operations, use more efficient methods or other best practices. This is my first web scraping project.
The goal here is to scrape a ...
4
votes
1answer
1k views
Recursive Web Scraping with Python Beautiful Soup
I wrote a short program which should allow a user to specify a starting page in Discogs Wiki Style Guide, scrape the other styles listed on the page, and then output a graph (represented here as a ...
4
votes
1answer
120 views
Scraping data from Bureau of Labor Statistics website
I'm currently scraping this website Bureau of Labor Statistics and I just want some feedback on how my approach was. I'm mainly focusing on the occupation groups table to the left of the website and ...
2
votes
1answer
2k views
Parsing HTML table into Pandas DataFrame
There is a text (link clickable) file with HTML table. The table is a bank statement. I'd like to parse it into pandas DataFrame. Is there a way to do it more gracefully? I've started to learn Python ...
7
votes
2answers
467 views
Items derived from one function are getting printed within another for loop in another function
I have written some code in python to scrape some items from a webpage. There is a link (titling see more) attach to each container. If I click on that link I can reach a certain page where all the ...
4
votes
2answers
102 views
Scrape a jobs board and keep persistent record
I'm trying to improve my pretty basic coding skills, and I've written this program to scrape a jobs board (I'd ultimately add several) and put the data in a db file with SQLITE3. The code works, but I ...
5
votes
2answers
407 views
Webscrape Podcast site for mp3s to (legally) add to collection
History
I am new to Python and this is my first attempt. Utilizing a legal login (omitted from the code below) to the Bob and Tom VIP website, I have created this script to access their Full Show mp3....
3
votes
1answer
41 views
Exception handling of HTML parsing
I am currently working on a program that will scrape data from a psych database of various article metadata. Since not every piece is formatted perfectly, I have accepted that some of the info may not ...
0
votes
2answers
42 views
Web scraper for a webpage article
I am a beginner in Python and have just coded a simple web scraper for a webpage article, output to a text file, using BeautifulSoup and ...
2
votes
0answers
101 views
Football Web Scraper Part 2
A revised version of the code in this question.
Things I have done so far:
Adjusted some formatting things like constant & variable naming and indentation
Wrapped most of the functions into a ...
3
votes
1answer
1k views
Web scraper for Football (Soccer) data with BeautifulSoup and Requests
I wrote a web scraper to get football scores from here. I'm getting the data for all seasons for the three major German leagues. It all works at the moment, but I'm sure it's possible to make it a lot ...
3
votes
1answer
81 views
Beating the queue
Inspired by this question on CodeReview, and because I'm never able to beat the queue.
I'm just too slow I guess.. :(
I made a little python utility that reads the queue's information every 30 ...
5
votes
1answer
165 views
Archives xkcd comics
This is a work in progress, as you may read I plan to revise to utilise JSON data rather than directly scraping.
I am considering refactoring the try//...
