Questions tagged [beautifulsoup]
Beautiful Soup is a Python library for pulling data out of HTML and XML files.
197
questions
0
votes
0
answers
40
views
1
vote
1
answer
34
views
Iterating over tables with nested <th> tags with BS4
The following code iterates over a table with nested th headings.
The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
- 321
0
votes
0
answers
31
views
Web Scraping - Python & BeautifulSoup
I'm new to Python, and well, I had to develop web scraping code & generate data CSV file.
Here is what I have written so far:
...
3
votes
0
answers
44
views
Scrape classifications from Wikipedia with bs4
I wrote my first bs4 script that scrapes animal classifications from Wikipedia. It works as long as the classifications are stored within these tags ...
- 131
2
votes
0
answers
81
views
Web scraping articles using asyncio
I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages.
I would like to improve ...
- 387
2
votes
2
answers
62
views
Web scraping playlist from website and writing to CSV
I have very recently learned some Python and found out about web scraping.
This is my first bash at writing some code to get a playlist from a site I had previously been copying and pasting. I would ...
3
votes
1
answer
437
views
Scraping a dynamic website with Scrapy (or Requests) and Selenium
I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here.
Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
- 321
2
votes
0
answers
830
views
Python script to scrape google maps without API
I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance.
It uses selenium ...
- 21
-1
votes
1
answer
229
views
Scraping odds portal with beautiful soup
This code scrapes www.oddsportal.com for all the URLs provided in the code and appends it to a dataframe.
I am not very well versed with iterative logic hence I am finding it difficult to improvise on ...
- 169
3
votes
1
answer
51
views
Refactor Web Scraper
I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
0
votes
1
answer
38
views
Content aggregator using bs4 and requests
Standard code review, tell me what's good, what's bad, and how to improve. Critical suggestions welcomed. This is a content aggregator using bs4 and requests. I did not use any tutorials or help.
<...
- 137
2
votes
2
answers
71
views
Scrape and parse dictionary website and get meaning of each word in a text file
In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website.
The function eng_meaning accepts ...
- 2,439
2
votes
1
answer
58
views
Scraping webpage elements using Python
I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
- 503
2
votes
1
answer
74
views
Beautifulsoup scrape posts that has the word python in it with no python tag
This is my first time web scraping, and here is the code I whipped up:
...
- 964
5
votes
1
answer
47
views
Scraper for basketball rosters
I am very new to Python and I am trying to learn through personnal projects and today I needed to collect a lot of baksetball player names and decided this was a good time to learn and practice.
The ...
- 51
3
votes
1
answer
99
views
Python Web-Scraping with BeautifulSoup
I'm relatively new to python and for an assignment I had to write a program that fetches a webpage with BeautifulSoup, extract all Paragraphs from it, and extract all words ending with "ing",...
- 179
4
votes
1
answer
60
views
MultiLingual Online Translator
So i made a program, which allows you to translate from one language to another using the command prompt.
I would like advice about the design, optimizations, and the tips and tricks to avoid ...
- 1,180
3
votes
1
answer
49
views
Class or not in Page Scraper (Python BeautifulSoup)
I am trying to figure out whether the below is better than just having a raw script or just a script with functions.
...
- 133
3
votes
2
answers
92
views
6
votes
1
answer
375
views
5
votes
1
answer
230
views
FanFiction Downloader in Python, with GUI
I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I ...
- 289
3
votes
1
answer
36
views
Scrape every Java themed codingbat problem set and your personal solutions
I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized?
...
1
vote
1
answer
449
views
Unofficial API for 1001tracklists.com
I wanted to build a scaping-API for the website 1001tracklists.com by writing classes for Tracklist and Track objects and some helpful functions:
Here's the full code (GitHub):
...
- 11
2
votes
1
answer
765
views
Better way to extract html table to dictionary using beautifulsoup in python
I am scraping HTML pages. Part of the page has a table which has acts and sections of those acts mentioned in table format. For some other project I need to convert them to Dictionary. The key values ...
- 231
5
votes
2
answers
282
views
Python Web scraping
I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any.
What do I want to achieve?
I want to scrape ...
3
votes
2
answers
595
views
First Python Web Scraping Project with BeautifulSoup
This is my first Web Scraping project in which I am retrieving the current stock information from here. This program works as expected, but I would certainly think someone with more experience with ...
- 307
1
vote
0
answers
37
views
Webscraping code to import logs from a website which is about to die
We all know that the famous twitch log site OverRustleLogs is getting shut down. So I decided to do some web scraping to download my favourite streamer's logs using BeautifulSoup. How can make this ...
- 11
2
votes
1
answer
152
views
A Text-Based Browser using Requests, Beautiful Soup, Sys, OS, and colorama
I have worked on this for quite a long while, and this is my 3rd attempt after completely demolishing 3 other 200 liners, and i really need ideas on how to make my code add more functionality or ...
- 1,180
5
votes
2
answers
880
views
OOP Web-scraper w/ Python and BeautifulSoup
This is my first major web scraping program in python. My code works nonetheless, I'm just not sure if it's the best OOP design. My code is below:
...
- 153
3
votes
1
answer
69
views
Dynamic web scraper in Python
The other week I faced the struggle of scraping dynamically generated content. So I used the selenium library with combination of requests & bs4, the thing is I am unsure of the quality of the ...
- 105
1
vote
3
answers
98
views
Scraping the Bundesliga table and saving it to CSV
Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance!
...
- 21
4
votes
1
answer
778
views
Scraping Data From Multiple URLs into Single Dataframe
I have a Class that:
goes to a url
grabs a link from that page, and a date (filing_date)
navigates to the link, and
writes the table from that page to a ...
- 207
6
votes
1
answer
114
views
Reach certain Wiki page through random links
I just read about Wiki game on Wikipedia and wanted to implement it as a fun project for myself using python.
Task:To reach Mathematics Wikipedia page using wikipedia hyperlinks randomly.
Using ...
- 145
10
votes
1
answer
137
views
Wordcloud from all answers of a user here on CR
Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
- 40.5k
1
vote
0
answers
26
views
Trouble returning web scraping output as python dictionary [closed]
So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
- 11
1
vote
0
answers
44
views
bs4 HTML cleaner function with repetitive nested if clauses,
[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
- 111
4
votes
1
answer
679
views
Scraping next page using BeautifulSoup
I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
- 320
2
votes
2
answers
239
views
CeWL written in Python
This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers.
My main concern is performance, as it takes a couple seconds to run. I ...
- 9,476
4
votes
1
answer
152
views
XML parser + bs4 combo
I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry.
Now, each job entry page is visited to grab info not present in the xml feed (...
- 195
2
votes
1
answer
92
views
Scrape company data from Google search
I have written this code for crawling and scraping search results from Google. But it's super slow when I run it.
What it does: read a pandas column, search values in Google and take the first result ...
- 123
3
votes
1
answer
465
views
Scraping IMDB and exporting to JSON or/and CSV
This code is scraping the data from IMDB and displays it in JSON or CSV format.
I plan to extend it beyond the top rated movies. How can I improve the quality of my code? I am using Python 3.7.3.
<...
- 43
5
votes
1
answer
475
views
Ultra fast Amazon scraper multi-threaded
This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay
A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
- 1,887
13
votes
3
answers
2k
views
AniPop - The anime downloader
Note: The topics of performance and Selenium/BS4 have not yet been addressed,
so this question can still receive a better answer!
Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
- 2,897
3
votes
1
answer
78
views
Webscraping cryptocurrency
In this class I made a webscraper for a cryptocurrency website to get the name and the price of each currency. There are maybe API's that I can also use, but I thought it is better to gain some ...
- 133
6
votes
1
answer
455
views
Web scraper that extracts urls from Amazon and eBay
Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case
of previously saved files, the files will be read and no ...
- 1,887
4
votes
0
answers
124
views
The anime downloader [duplicate]
NOTE: Here's the latest version of this program, since this question idled out.
This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
- 2,897
3
votes
2
answers
150
views
Get a notification when questions get posted
I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question.
With that being said, I can't always keep a tab and an eye to refresh the ...
- 193
5
votes
3
answers
153
views
Effective fetching data algorithm
I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses.
I'm using thread, and I know threading isn't so fast because of GIL.
My problem is that ...
- 51
30
votes
6
answers
4k
views
Python web-scraper to download table of transistor counts from Wikipedia
I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it?
...
- 301
4
votes
1
answer
143
views
Footwear scraper
I've done this parser to scrape all footwear data, but I don't know if it is good to use OOP in this case. Can you please check this out and give me the strongest feedback? I'm working on improving my ...
- 55