Questions tagged [beautifulsoup]
Beautiful Soup is a Python library for pulling data out of HTML and XML files.
164
questions
10
votes
1answer
99 views
Wordcloud from all answers of a user here on CR
Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
1
vote
0answers
23 views
Trouble returning web scraping output as python dictionary [closed]
So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
1
vote
0answers
38 views
bs4 HTML cleaner function with repetitive nested if clauses,
[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
4
votes
1answer
101 views
Scraping next page using BeautifulSoup
I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
2
votes
2answers
77 views
CeWL written in Python
This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers.
My main concern is performance, as it takes a couple seconds to run. I ...
4
votes
1answer
91 views
XML parser + bs4 combo
I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry.
Now, each job entry page is visited to grab info not present in the xml feed (...
2
votes
1answer
67 views
Scrape company data from Google search
I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.
What it does: read a pandas column, search values in Google and take the first result ...
3
votes
1answer
70 views
Scraping IMDB and exporting to JSON or/and CSV
This code is scraping the data from IMDB and displays it in JSON or CSV format.
I plan to extend it beyond the top rated movies. How can I improve the quality of my code? I am using Python 3.7.3.
<...
5
votes
1answer
214 views
Ultra fast Amazon scraper multi-threaded
This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay
A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
13
votes
3answers
2k views
AniPop - The anime downloader
Note: The topics of performance and Selenium/BS4 have not yet been addressed,
so this question can still receive a better answer!
Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
3
votes
1answer
72 views
Webscraping cryptocurrency
In this class I made a webscraper for a cryptocurrency website to get the name and the price of each currency. There are maybe API's that I can also use, but I thought it is better to gain some ...
6
votes
1answer
164 views
Web scraper that extracts urls from Amazon and eBay
Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case
of previously saved files, the files will be read and no ...
4
votes
0answers
122 views
The anime downloader [duplicate]
NOTE: Here's the latest version of this program, since this question idled out.
This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
3
votes
2answers
93 views
Get a notification when questions get posted
I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question.
With that being said, I can't always keep a tab and an eye to refresh the ...
5
votes
3answers
133 views
Effective fetching data algorithm
I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses.
I'm using thread, and I know threading isn't so fast because of GIL.
My problem is that ...
29
votes
5answers
3k views
Python web-scraper to download table of transistor counts from Wikipedia
I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it?
...
4
votes
1answer
99 views
Footwear scraper
I've done this parser to scrape all footwear data, but I don't know if it is good to use OOP in this case. Can you please check this out and give me the strongest feedback? I'm working on improving my ...
3
votes
2answers
201 views
Web-scraping investment fund data into Pandas
Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
2
votes
1answer
151 views
Beautifulsoup and lxml (xpath) too slow with respect to regex when parsing HTML
I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.
The problem is that any alternatives are really too slow....
3
votes
1answer
75 views
the fox is black - a simple image scraper
I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
52 views
Scraping a hiring website using python's requests and BeautifulSoup
I'm designing a scraping application using python, requests and BeautifulSoup4.
I decided to divide the logic into two classes:
Spider : gets the base url ...
2
votes
0answers
37 views
Scraping Forum Tables w/ Links Using Beautiful Soup
This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
6
votes
0answers
85 views
Get a movie directory, rename it and save imdb info as a webpage
I have my movie folders formatted:
~: no sub included
{}: already watched
nothing: sub included, ready to watch
the program asks for the movie directory; extracts the name and searches IMDb for the ...
3
votes
1answer
910 views
Python script to download adult comics from 8muses
I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the ...
4
votes
1answer
56 views
Eliminate unnecessary looping while parsing cable modem status table
I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and ...
3
votes
1answer
91 views
Web scraping job postings from Facebook
I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
83 views
Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)
The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
661 views
NFL data scraper
I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
3
votes
1answer
466 views
Pulls href tags using BeautifulSoup with Python
As if this isn't extremely obvious, I am a new coder.
My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
71 views
Google Analytic Webcrawler for every country code
At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
1
vote
1answer
61 views
Webcrawler for a car forum
I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and ...
4
votes
1answer
200 views
Cleaner way of appending data to List in BeautifulSoup
So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
19
votes
4answers
6k views
Web scraping the titles and descriptions of trending YouTube videos
This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?
...
1
vote
0answers
158 views
Twitter scraper using tweepy
I wrote a Twitter scraper using the tweepy so I can scrape user information and tweets. Given that the free API doesn't let me get the number of messages per tweet, I had to rely on BeautifulSoup to ...
2
votes
1answer
73 views
Data scraper in Python 3
This was my first attempt at scraping approximately 1800 open datasets from the web. On the first go-around it takes approximately 1 hour to complete. Subsequent iterations only take 4 seconds. I'm ...
2
votes
1answer
57 views
Crawling pages to obtain .htm and .txt files from sec.gov
I am pretty much a python newbie and was looking at this question on StackOverflow.
Whereas the OP was interested in downloading the .htm |.txt files, I was simply ...
2
votes
0answers
214 views
Wrapping a web scraper in a RESTful API
The problem I am looking to solve is wrapping a web scraper in a RESTful API such that it can be called programmatically from another application, frontend or microservice. The overall goal is that ...
2
votes
1answer
137 views
Sourcing content from a webpage in an organized manner
I've written a script in python to grab different title and address from different pages of a website. Firstly the script will ...
1
vote
0answers
203 views
Scraping symfonycasts.com using BeautifulSoup
I'm learning python 3.x and I'm still feeling like a newbie.
The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all ...
3
votes
0answers
245 views
Python web scraper
This is my first attempt at a sizeable amount of code in Python and I've made an attempt to standardize a script into a library so that it can be reused.
However, ...
3
votes
2answers
5k views
Python - Iterating over contents of BeautifulSoup element very slow
I have a large HTML document where everything is inside a main div, structured like this:
...
5
votes
1answer
3k views
Python scraping tables
I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
2
votes
1answer
4k views
Web-scraping through a rotating proxy script
I've created a script in python which is able to parse proxies (supposed to support "https") from a website. Then the script will use those proxies randomly to parse the title of different coffe shops ...
5
votes
2answers
88 views
Scraping lunar/solar eclipse data with BeautifulSoup Followup
I decided to refactor my code based on this review.
Eclipse:
...
11
votes
3answers
1k views
Web-scraping a list of lawyers
I've written a program to get the names along with the titles of some practitioners out of a webpage. The content stored within disorganized HTML elements (at least it seemed to me) and as a result ...
3
votes
1answer
560 views
Web scraping news articles with Beautiful Soup
It's been nearly a year I've been working with C++ and now I am diving in Python and its libraries and frameworks. I am currently creating a website for my portfolio that gathers news articles from a ...
8
votes
2answers
158 views
Scraping lunar/solar eclipse data with BeautifulSoup
I created a script to scrape this website and gather the eclipse data (date, type, and location):
...
1
vote
0answers
132 views
Extracting data from a used car sales site
I am developing code for extracting data from a used car sales site. There are 4 sites in total. In 3 of them I use requests and beautifulsoup. The time taken to extract data from these sites was ...
7
votes
3answers
752 views
Scraping Clinical Studies database (via ClinicalTrials.Gov)
I have written the following script to scrape data from the U.S. National Library of Medicine website ClinicalTrials.Gov based on an NCTID.
...
2
votes
0answers
640 views
Recursively scrape links from web pages and check them
I'm new to programming and especially new to object oriented programming. I have built a web scraper using functional programming and am trying to build another using OOP principles.
The overall idea ...
