Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
1
vote
0answers
42 views

Integrating web scraping code into loop [closed]

Background: I have systematically written code segments that scape a recipe website for certain recipe stats of information including: Ingredients list Method / Instructions Prep time Cook time Yield ...
3
votes
1answer
43 views

Class or not in Page Scraper (Python BeautifulSoup)

I am trying to figure out whether the below is better than just having a raw script or just a script with functions. ...
5
votes
1answer
62 views

FanFiction Downloader in Python, with GUI

I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I ...
3
votes
1answer
23 views

Scrape every Java themed codingbat problem set and your personal solutions

I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized? ...
1
vote
1answer
61 views

Unofficial API for 1001tracklists.com

I wanted to build a scaping-API for the website 1001tracklists.com by writing classes for Tracklist and Track objects and some helpful functions: Here's the full code (GitHub): ...
2
votes
1answer
37 views

Better way to extract html table to dictionary using beautifulsoup in python

I am scrapping html pages. Part of the page has a table which has acts and sections of those acts mentioned in table format. For some other project I need to convert them to Dictionary. The key values ...
5
votes
2answers
254 views

How can I make this better? Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
3
votes
2answers
70 views

First Python Web Scraping Project with BeautifulSoup

This is my first Web Scraping project in which I am retrieving the current stock information from here. This program works as expected, but I would certainly think someone with more experience with ...
1
vote
0answers
29 views

Webscraping code to import logs from a website which is about to die

We all know that the famous twitch log site OverRustleLogs is getting shut down. So I decided to do some web scraping to download my favourite streamer's logs using BeautifulSoup. How can make this ...
2
votes
1answer
61 views

A Text-Based Browser using Requests, Beautiful Soup, Sys, OS, and colorama

I have worked on this for quite a long while, and this is my 3rd attempt after completely demolishing 3 other 200 liners, and i really need ideas on how to make my code add more functionality or ...
5
votes
2answers
437 views

OOP Web-scraper w/ Python and BeautifulSoup

This is my first major web scraping program in python. My code works nonetheless, I'm just not sure if it's the best OOP design. My code is below: ...
3
votes
1answer
35 views

Dynamic web scraper in Python

The other week I faced the struggle of scraping dynamically generated content. So I used the selenium library with combination of requests & bs4, the thing is I am unsure of the quality of the ...
1
vote
3answers
57 views

Scraping the Bundesliga table and saving it to CSV

Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance! ...
4
votes
1answer
98 views

Scraping Data From Multiple URLs into Single Dataframe

I have a Class that: goes to a url grabs a link from that page, and a date (filing_date) navigates to the link, and writes the table from that page to a ...
6
votes
1answer
65 views

Reach certain Wiki page through random links

I just read about Wiki game on Wikipedia and wanted to implement it as a fun project for myself using python. Task:To reach Mathematics Wikipedia page using wikipedia hyperlinks randomly. Using ...
10
votes
1answer
117 views

Wordcloud from all answers of a user here on CR

Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
1
vote
0answers
24 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
1
vote
0answers
41 views

bs4 HTML cleaner function with repetitive nested if clauses,

[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
4
votes
1answer
133 views

Scraping next page using BeautifulSoup

I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
2
votes
2answers
114 views

CeWL written in Python

This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers. My main concern is performance, as it takes a couple seconds to run. I ...
4
votes
1answer
113 views

XML parser + bs4 combo

I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry. Now, each job entry page is visited to grab info not present in the xml feed (...
2
votes
1answer
71 views

Scrape company data from Google search

I have written this code for crawling and scraping search results from Google. But it's super slow when I run it. What it does: read a pandas column, search values in Google and take the first result ...
3
votes
1answer
131 views

Scraping IMDB and exporting to JSON or/and CSV

This code is scraping the data from IMDB and displays it in JSON or CSV format. I plan to extend it beyond the top rated movies. How can I improve the quality of my code? I am using Python 3.7.3. <...
5
votes
1answer
250 views

Ultra fast Amazon scraper multi-threaded

This is a follow up to the code here: Web scraper that extracts urls from Amazon and eBay A multi-threaded modification to the previous version that is Amazon focused and most of the necessary ...
13
votes
3answers
2k views

AniPop - The anime downloader

Note: The topics of performance and Selenium/BS4 have not yet been addressed, so this question can still receive a better answer! Chat Room: https://chat.stackexchange.com/rooms/100275/anipop-...
3
votes
1answer
75 views

Webscraping cryptocurrency

In this class I made a webscraper for a cryptocurrency website to get the name and the price of each currency. There are maybe API's that I can also use, but I thought it is better to gain some ...
6
votes
1answer
196 views

Web scraper that extracts urls from Amazon and eBay

Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case of previously saved files, the files will be read and no ...
4
votes
0answers
124 views

The anime downloader [duplicate]

NOTE: Here's the latest version of this program, since this question idled out. This is a recreational script made to update my home server w/ the latest anime from HorribleSubs. I'd like to know if ...
3
votes
2answers
128 views

Get a notification when questions get posted

I like to help people on StackOverflow, matter of fact nowadays it depends on how quick you are when answering question. With that being said, I can't always keep a tab and an eye to refresh the ...
5
votes
3answers
138 views

Effective fetching data algorithm

I'm fetching some data by using bs4, and want to know why access speed gets slower as my program progresses. I'm using thread, and I know threading isn't so fast because of GIL. My problem is that ...
29
votes
5answers
3k views

Python web-scraper to download table of transistor counts from Wikipedia

I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it? ...
4
votes
1answer
105 views

Footwear scraper

I've done this parser to scrape all footwear data, but I don't know if it is good to use OOP in this case. Can you please check this out and give me the strongest feedback? I'm working on improving my ...
3
votes
2answers
326 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
2
votes
1answer
307 views

Beautifulsoup and lxml (xpath) too slow with respect to regex when parsing HTML

I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML. The problem is that any alternatives are really too slow....
3
votes
1answer
78 views

the fox is black - a simple image scraper

I wrote a simple image scraper for the Desktop Wallpaper project by The Fox is Black. The site is a fantastic (yet, apparently dead) blog full of fabulous images. However, there's no way to download ...
4
votes
0answers
55 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
2
votes
0answers
39 views

Scraping Forum Tables w/ Links Using Beautiful Soup

This code scrapes post links (among other information) from a table on a forum. While the current code works, I would like to know if there is a better/simpler way of writing it (maybe not as many for-...
7
votes
0answers
105 views

Get a movie directory, rename it and save imdb info as a webpage

I have my movie folders formatted: ~: no sub included {}: already watched nothing: sub included, ready to watch the program asks for the movie directory; extracts the name and searches IMDb for the ...
3
votes
1answer
1k views

Python script to download adult comics from 8muses

I made a simple python script using the BeautifulSoup and Selenium to automatically download adult comics from 8muses. I've used selenium for the reason that the website uses javascript to load the ...
4
votes
1answer
61 views

Eliminate unnecessary looping while parsing cable modem status table

I am writing a Python script that will be used to generate line protocol for inserting metrics into an InfluxDB. I'm parsing the status page of my cable modem. That part works. I receive the data and ...
3
votes
1answer
122 views

Web scraping job postings from Facebook

I built a web scraper to pull a list of jobs on Facebook and other websites but want to break up the code into functions that I can reuse for other websites. This structure is working but I think it ...
3
votes
1answer
118 views

Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)

The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
4
votes
1answer
800 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
3
votes
1answer
719 views

Pulls href tags using BeautifulSoup with Python

As if this isn't extremely obvious, I am a new coder. My code works, but is quite far from ideal. I'm also not sure if there are unnecessary bits, since I've been trying to get this to work for a ...
4
votes
1answer
74 views

Google Analytic Webcrawler for every country code

At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
1
vote
1answer
61 views

Webcrawler for a car forum

I've been experimenting more with webcrawling and hence have started to get a better understanding compared to my previous questions. Right now, my code scraps from a car forum on each page and ...
4
votes
1answer
382 views

Cleaner way of appending data to List in BeautifulSoup

So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scrapper to collect data such as <...
19
votes
4answers
7k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...