Questions tagged [beautifulsoup]
Beautiful Soup is a Python library for pulling data out of HTML and XML files.
4
votes
1answer
39 views
Eliminate unnecessary looping while parsing cable modem status table
I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and ...
3
votes
1answer
49 views
Web scraping job postings from Facebook
I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
41 views
Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)
The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
81 views
NFL data scraper
I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
3
votes
1answer
33 views
Pulls href tags using BeautifulSoup with Python
As if this isn't extremely obvious, I am a new coder.
My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
65 views
Google Analytic Webcrawler for every country code
At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
1
vote
1answer
59 views
Webcrawler for a car forum
I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and ...
4
votes
1answer
77 views
Cleaner way of appending data to List in BeautifulSoup
So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
19
votes
4answers
3k views
Web scraping the titles and descriptions of trending YouTube videos
This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?
...
1
vote
0answers
59 views
Twitter scraper using tweepy
I wrote a Twitter scraper using the tweepy so I can scrape user information and tweets. Given that the free API doesn't let me get the number of messages per tweet, I had to rely on BeautifulSoup to ...
2
votes
1answer
67 views
Data scraper in Python 3
This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
1answer
52 views
Crawling pages to obtain .htm and .txt files from sec.gov
I am pretty much a python newbie and was looking at this question on StackOverflow.
Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
2
votes
0answers
127 views
Wrapping a web scraper in a RESTful API
The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
1
vote
1answer
133 views
Sourcing content from a webpage in an organized manner
I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
0answers
122 views
Scraping symfonycasts.com using BeautifulSoup
I'm learning python 3.x and I'm still feeling like a newbie.
The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all ...
3
votes
0answers
99 views
Python web scraper
This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused.
However, ...
3
votes
2answers
2k views
Python - Iterating over contents of BeautifulSoup element very slow
I have a large HTML document where everything is inside a main div, structured like this:
...
5
votes
1answer
1k views
Python scraping tables
I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
1answer
2k views
Web-scraping through a rotating proxy script
I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
5
votes
2answers
82 views
Scraping lunar/solar eclipse data with BeautifulSoup Followup
I decided to refactor my code based on this review.
Eclipse:
...
11
votes
3answers
1k views
Web-scraping a list of lawyers
I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
2
votes
1answer
421 views
Web scraping news articles with Beautiful Soup
It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a ...
8
votes
2answers
130 views
Scraping lunar/solar eclipse data with BeautifulSoup
I created a script to scrape this website and gather the eclipse data (date, type, and location):
...
1
vote
0answers
79 views
Extracting data from a used car sales site
I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
7
votes
3answers
370 views
Scraping Clinical Studies database (via ClinicalTrials.Gov)
I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID.
...
2
votes
0answers
457 views
Recursively scrape links from web pages and check them
I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles.
The overall idea ...
0
votes
1answer
104 views
Program to automate clicking on the new button on a chat website until the username matches certain conditions
This is a python program that automates skipping through users on a chat website called
talkwithstrangers using the Selenium Webdriver and ...
4
votes
0answers
72 views
List all links in a website
I wrote this code as part of a project I'm working on. It's supposed to get all links from a website and then perform black-box tests on them. How can I improve this code to be faster and more ...
11
votes
2answers
334 views
Comic Image Web Scraper
I've written my first web scraper, which (surprisingly) does the job. I'm scraping a popular comic website for the images (there are over 950 of them) but the problem is the scraper is way too slow.
...
21
votes
3answers
4k views
Financial website scraper
I wrote the code that would scrap Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to python programming and therefore believe my code is ...
4
votes
1answer
96 views
A web scraper that looks for pre-defined words in news articles
I'm still fairly new to Python and web-scraping, but a colleague asked asked me if I could build a web-scraper that could be used by a think tank he's a member of to find news articles matching a pre-...
2
votes
0answers
783 views
Scrape webpage with Beautifulsoup and export relevant data to csv
I am looking for feedback on my code, how I can simplify some operations, use more efficient methods or other best practices. This is my first web scraping project.
The goal here is to scrape a ...
4
votes
1answer
981 views
Recursive Web Scraping with Python Beautiful Soup
I wrote a short program which should allow a user to specify a starting page in Discogs Wiki Style Guide, scrape the other styles listed on the page, and then output a graph (represented here as a ...
4
votes
1answer
107 views
Scraping data from Bureau of Labor Statistics website
I'm currently scraping this website Bureau of Labor Statistics and I just want some feedback on how my approach was. I'm mainly focusing on the occupation groups table to the left of the website and ...
2
votes
1answer
2k views
Parsing HTML table into Pandas DataFrame
There is a text (link clickable) file with HTML table. The table is a bank statement. I'd like to parse it into pandas DataFrame. Is there a way to do it more gracefully? I've started to learn Python ...
7
votes
2answers
460 views
Items derived from one function are getting printed within another for loop in another function
I have written some code in python to scrape some items from a webpage. There is a link (titling see more) attach to each container. If I click on that link I can reach a certain page where all the ...
4
votes
2answers
96 views
Scrape a jobs board and keep persistent record
I'm trying to improve my pretty basic coding skills, and I've written this program to scrape a jobs board (I'd ultimately add several) and put the data in a db file with SQLITE3. The code works, but I ...
5
votes
2answers
377 views
Webscrape Podcast site for mp3s to (legally) add to collection
History
I am new to Python and this is my first attempt. Utilizing a legal login (omitted from the code below) to the Bob and Tom VIP website, I have created this script to access their Full Show mp3....
3
votes
1answer
38 views
Exception handling of HTML parsing
I am currently working on a program that will scrape data from a psych database of various article metadata. Since not every piece is formatted perfectly, I have accepted that some of the info may not ...
0
votes
2answers
41 views
Web scraper for a webpage article
I am a beginner in Python and have just coded a simple web scraper for a webpage article, output to a text file, using BeautifulSoup and ...
2
votes
0answers
92 views
Football Web Scraper Part 2
A revised version of the code in this question.
Things I have done so far:
Adjusted some formatting things like constant & variable naming and indentation
Wrapped most of the functions into a ...
3
votes
1answer
974 views
Web scraper for Football (Soccer) data with BeautifulSoup and Requests
I wrote a web scraper to get football scores from here. I'm getting the data for all seasons for the three major German leagues. It all works at the moment, but I'm sure it's possible to make it a lot ...
3
votes
1answer
77 views
Beating the queue
Inspired by this question on CodeReview, and because I'm never able to beat the queue.
I'm just too slow I guess.. :(
I made a little python utility that reads the queue's information every 30 ...
5
votes
1answer
148 views
Archives xkcd comics
This is a work in progress, as you may read I plan to revise to utilise JSON data rather than directly scraping.
I am considering refactoring the try//...
-4
votes
1answer
317 views
2
votes
1answer
90 views
Extracting top subreddits from an HTML table
I'm scraping a table and am trying to get nested td tags down the tbody tree but the code seems kind of verbose. Is there a more Pythonic way to do this?
...
4
votes
1answer
177 views
Py3 ebay-scraper - Project I need cheap GPU
I need cheap GPU's and decided to make a little ebay-scraper to make sure I can buy it at cheap prices. It works and I can search for (any) product to scrape the Title, Price and Shipping cost of that ...
1
vote
1answer
81 views
Console based English dictionary app
I'm new to programming. I wrote this following console based app in Python. It's an English dictionary app. If the user inputs the desired word that he/she wishes to search for, then this app should ...
5
votes
2answers
1k views
Python webscraping for lists of company names and details using BeautifulSoup
I just started learning Python and made a script that scrapes a corporate directory
https://www.sccci.org.sg/en/directory/corporate/members
So far it gets all the company names and their details ...
3
votes
3answers
375 views
Web-crawler for iTunes
I've written a script in Python to scrape app name, developer and price from ITunes site.
In its first page there are 10 different recommended app links which my scraper is able to track down and ...
