Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
-1 votes
2 answers
85 views

Web scraping spider

I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
Max's user avatar
  • 7
2 votes
0 answers
45 views

Simplified HTML parsing for LEGO features

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
alvas's user avatar
  • 705
2 votes
1 answer
43 views

URL link scraper and analyser

I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on). Essentially,...
Jessica's user avatar
  • 818
1 vote
0 answers
755 views

Code scraping oddsportal using beautifulsoup

This code scrapes the website www.oddsportal.com for betting odds and historical games. code: ...
Rander's user avatar
  • 11
3 votes
0 answers
563 views

Python BeautifulSoup - preparing HTML rows and td tags for Pandas

I'm using BeautifulSoup to parse a bunch of combined tables' rows, row by row, column by column to prepare it for import into Pandas. I can't use to_html() because ...
Meghan M.'s user avatar
0 votes
1 answer
106 views

Selenium/Beautifulsoup web scrape of Grainger

This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
cahilltyler's user avatar
5 votes
1 answer
149 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
alphamu's user avatar
  • 153
2 votes
1 answer
48 views

scraping pollution data tables from government website

i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved: ...
Omar Morales Rivera's user avatar
2 votes
1 answer
91 views

Monitoring webpages to be notified

I have been writing a monitoring script where I check for whenever there is changes on a webpage. When a change has happend I want to be notified by printing out that there is a difference. I have ...
PythonNewbie's user avatar
3 votes
1 answer
73 views

Get the 64 most popular items from a list of tree species using webscraping

I made this python program to find the 64 most common elements of a list stored in a text file. Since this works using binomial names of trees I decided to use the number of mentions in PubMed Central ...
Omar Morales Rivera's user avatar
2 votes
1 answer
95 views

Compare between two dictionaries

Hello beautiful people! I have currently worked on a small script that I would of course continue to work with whenever I get a good feedback from the best code reviewer in here <3 - I have worked ...
PythonNewbie's user avatar
1 vote
0 answers
36 views

Iterate tables from table id from href links until no table with specific table id is found

I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html What I am trying to achieve is to parse the ...
Gescof's user avatar
  • 11
3 votes
1 answer
140 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
Olgierd Wiśniewski's user avatar
1 vote
1 answer
108 views

Design Pattern: Builder - BeautifulSoup directory navigation and scraping

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
Jason Leaver's user avatar
1 vote
1 answer
53 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
Sati's user avatar
  • 417
2 votes
0 answers
139 views

Web scraping articles using asyncio

I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
AlexDotis's user avatar
  • 417
2 votes
2 answers
86 views

Web scraping playlist from website and writing to CSV

I have very recently learned some Python and found out about web scraping. This is my first bash at writing some code to get a playlist from a site I had previously been copying and pasting. I would ...
Noob_warning2's user avatar
4 votes
1 answer
899 views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
Sati's user avatar
  • 417
2 votes
0 answers
2k views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
WhiteWood's user avatar
0 votes
1 answer
725 views

Scraping odds portal with beautiful soup

This code scrapes www.oddsportal.com for all the URLs provided in the code and appends it to a dataframe. I am not very well versed with iterative logic hence I am finding it difficult to improvise on ...
PyNoob's user avatar
  • 179
3 votes
1 answer
70 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
Tauqeer Abbas's user avatar
0 votes
1 answer
49 views

Content aggregator using bs4 and requests

Standard code review, tell me what's good, what's bad, and how to improve. Critical suggestions welcomed. This is a content aggregator using bs4 and requests. I did not use any tutorials or help. <...
cold10's user avatar
  • 459
2 votes
2 answers
248 views

Scrape and parse dictionary website and get meaning of each word in a text file

In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website. The function eng_meaning accepts ...
coder's user avatar
  • 2,449
2 votes
1 answer
62 views

Scraping webpage elements using Python

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
PythonNewbie's user avatar
2 votes
1 answer
94 views

Beautifulsoup scrape posts that has the word python in it with no python tag

This is my first time web scraping, and here is the code I whipped up: ...
Chocolate's user avatar
  • 948
5 votes
1 answer
51 views

Scraper for basketball rosters

I am very new to Python and I am trying to learn through personnal projects and today I needed to collect a lot of baksetball player names and decided this was a good time to learn and practice. The ...
nouse's user avatar
  • 51
3 votes
1 answer
222 views

Python Web-Scraping with BeautifulSoup

I'm relatively new to python and for an assignment I had to write a program that fetches a webpage with BeautifulSoup, extract all Paragraphs from it, and extract all words ending with "ing",...
elauser's user avatar
  • 179
4 votes
1 answer
113 views

MultiLingual Online Translator

So i made a program, which allows you to translate from one language to another using the command prompt. I would like advice about the design, optimizations, and the tips and tricks to avoid ...
Anonymous's user avatar
  • 1,204
3 votes
1 answer
70 views

Class or not in Page Scraper (Python BeautifulSoup)

I am trying to figure out whether the below is better than just having a raw script or just a script with functions. ...
Datacrawler's user avatar
3 votes
2 answers
109 views

Real time metal prices and conversion fro USD to MXN using BeautifulSoup updated version

...
Ethan Hetrick's user avatar
7 votes
1 answer
637 views

Webscraping application that collects metal prices and converts them from USD to MXN real-time using BeautifulSoup

...
Ethan Hetrick's user avatar
5 votes
1 answer
439 views

FanFiction Downloader in Python, with GUI

I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I ...
Unsigned_Arduino's user avatar
3 votes
1 answer
47 views

Scrape every Java themed codingbat problem set and your personal solutions

I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized? ...
Tres The Goat's user avatar
1 vote
1 answer
760 views

Unofficial API for 1001tracklists.com

I wanted to build a scaping-API for the website 1001tracklists.com by writing classes for Tracklist and Track objects and some helpful functions: Here's the full code (GitHub): ...
leanderto's user avatar
2 votes
1 answer
2k views

Better way to extract html table to dictionary using beautifulsoup in python

I am scraping HTML pages. Part of the page has a table which has acts and sections of those acts mentioned in table format. For some other project I need to convert them to Dictionary. The key values ...
sangharsh's user avatar
  • 241
5 votes
2 answers
295 views

Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
Faheem Akhtar's user avatar
3 votes
2 answers
688 views

First Python Web Scraping Project with BeautifulSoup

This is my first Web Scraping project in which I am retrieving the current stock information from here. This program works as expected, but I would certainly think someone with more experience with ...
unkn0wn.dev's user avatar
1 vote
0 answers
40 views

Webscraping code to import logs from a website which is about to die

We all know that the famous twitch log site OverRustleLogs is getting shut down. So I decided to do some web scraping to download my favourite streamer's logs using BeautifulSoup. How can make this ...
th3_ h0pZe's user avatar
2 votes
1 answer
231 views

A Text-Based Browser using Requests, Beautiful Soup, Sys, OS, and colorama

I have worked on this for quite a long while, and this is my 3rd attempt after completely demolishing 3 other 200 liners, and i really need ideas on how to make my code add more functionality or ...
Anonymous's user avatar
  • 1,204
5 votes
2 answers
1k views

OOP Web-scraper w/ Python and BeautifulSoup

This is my first major web scraping program in python. My code works nonetheless, I'm just not sure if it's the best OOP design. My code is below: ...
Practical1's user avatar
3 votes
1 answer
134 views

Dynamic web scraper in Python

The other week I faced the struggle of scraping dynamically generated content. So I used the selenium library with combination of requests & bs4, the thing is I am unsure of the quality of the ...
Viktor Stefanov's user avatar
1 vote
3 answers
136 views

Scraping the Bundesliga table and saving it to CSV

Wrote my first mini-script to get data from the german football league table and save the data to a csv file. Do you like the approach or could I be more efficient? Thanks in advance! ...
Martin.3112's user avatar
4 votes
1 answer
1k views

Scraping Data From Multiple URLs into Single Dataframe

I have a Class that: goes to a url grabs a link from that page, and a date (filing_date) navigates to the link, and writes the table from that page to a ...
user53526356's user avatar
5 votes
1 answer
173 views

Reach certain Wiki page through random links

I just read about Wiki game on Wikipedia and wanted to implement it as a fun project for myself using python. Task:To reach Mathematics Wikipedia page using wikipedia hyperlinks randomly. Using ...
Akhil Sharma's user avatar
10 votes
1 answer
158 views

Wordcloud from all answers of a user here on CR

Since I haven't really used Python's new async features yet, I took some older code of mine, which took all of my answers here on Code Review and generated a word cloud from them, and updated it to ...
Graipher's user avatar
  • 40.8k
1 vote
0 answers
27 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
Zack's user avatar
  • 11
1 vote
0 answers
54 views

bs4 HTML cleaner function with repetitive nested if clauses,

[I made a little web-scraper that downloads the source html files as well. Now for the sake of saving storage space i wrote a small function to delete quite a bit of stuff in the html file (specific ...
Nick's user avatar
  • 111
4 votes
1 answer
926 views

Scraping next page using BeautifulSoup

I have created a script for article scraping - it finds title, subtitle, href-link, and the time of publication. Once retrieved, information is converted to a pandas dataframe, and the link for the ...
help-ukraine-now's user avatar
2 votes
2 answers
332 views

CeWL written in Python

This is my implementation of CeWL, a ruby application that returns a list of words which can be used for password crackers. My main concern is performance, as it takes a couple seconds to run. I ...
Linny's user avatar
  • 9,948
4 votes
1 answer
174 views

XML parser + bs4 combo

I just wrote a short function to parse SO jobs XML feed and return dicts containing information about each job entry. Now, each job entry page is visited to grab info not present in the xml feed (...
anddt's user avatar
  • 195

1
2 3 4 5