Skip to main content

Questions tagged [beautifulsoup]

Beautiful Soup is a Python library for pulling data out of HTML and XML files.

Filter by
Sorted by
Tagged with
5 votes
3 answers
819 views

code format and steps web scraping using beautiful soup

I've done simple web scraping and want to make sure all my steps are correct? Is it considered clean code? Is there a better way to use the multi-page scraping feature? ...
0 votes
2 answers
144 views

Web scraping spider

I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
2 votes
0 answers
58 views

Simplified HTML parsing for LEGO features

The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (...
4 votes
1 answer
120 views

MultiLingual Online Translator

So i made a program, which allows you to translate from one language to another using the command prompt. I would like advice about the design, optimizations, and the tips and tricks to avoid ...
2 votes
1 answer
57 views

URL link scraper and analyser

I recently wrote a testing tool (called plink) for retrieving all the links from a website (and then retreiving links from the linked pages, and so on). Essentially,...
1 vote
0 answers
1k views

Code scraping oddsportal using beautifulsoup

This code scrapes the website www.oddsportal.com for betting odds and historical games. code: ...
3 votes
0 answers
916 views

Python BeautifulSoup - preparing HTML rows and td tags for Pandas

I'm using BeautifulSoup to parse a bunch of combined tables' rows, row by row, column by column to prepare it for import into Pandas. I can't use to_html() because ...
0 votes
1 answer
195 views

Selenium/Beautifulsoup web scrape of Grainger

This is my first time writing a web scrape using Selenium and Beautifulsoup. The website I'm scraping is https://www.grainger.com/ and I have it pulling a specific set of SKUs stored in an Excel file. ...
5 votes
1 answer
165 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
2 votes
1 answer
54 views

scraping pollution data tables from government website

i made this python program to scrape and save the daily maximum pollution values in mexico city, the data will be used in machine learning but i wonder what could be improved: ...
7 votes
1 answer
172 views

Scrape points from CTF sites

I am relatively new to classes. This one uses one. I am pretty sure this is not the correct way to do this. But I also do not know the correct way. You can use the ...
2 votes
1 answer
97 views

Monitoring webpages to be notified

I have been writing a monitoring script where I check for whenever there is changes on a webpage. When a change has happend I want to be notified by printing out that there is a difference. I have ...
3 votes
1 answer
77 views

Get the 64 most popular items from a list of tree species using webscraping

I made this python program to find the 64 most common elements of a list stored in a text file. Since this works using binomial names of trees I decided to use the number of mentions in PubMed Central ...
2 votes
1 answer
99 views

Compare between two dictionaries

Hello beautiful people! I have currently worked on a small script that I would of course continue to work with whenever I get a good feedback from the best code reviewer in here <3 - I have worked ...
1 vote
0 answers
44 views

Iterate tables from table id from href links until no table with specific table id is found

I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html What I am trying to achieve is to parse the ...
3 votes
1 answer
222 views

Data retrieval from Dynamic HTML page with time-out (Web scraping w. Python)

The HTML page shows list of a friend network of a person (each Name has anchor <a> tag w. link to list of friend network). Since the page has a timer, I've ...
3 votes
2 answers
1k views

Performance and Readability Improvements for HTML Parser with BeautifulSoup

This function takes as an argument a JSON file (could contain anything in JSON format, since I scrape hundreds of random pages) and returns a list of dictionaries where a URL is mapped to its ...
23 votes
3 answers
4k views

Financial website scraper

I wrote the code that would scrape Finviz for tickers and href that attach to the ticker. I tested with multi-page as well as one page. I'm new to Python ...
3 votes
1 answer
184 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
1 vote
1 answer
129 views

Design Pattern: Builder - BeautifulSoup directory navigation and scraping

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
1 vote
1 answer
62 views

Iterating over tables with nested <th> tags with BS4

The following code iterates over a table with nested th headings. The decision has been made to just go with the main heading and ignore sub-headings, but nested ...
6 votes
1 answer
561 views

Web scraper that extracts urls from Amazon and eBay

Description: This is a simple script for scraping Amazon and eBay category, sub-category and product URLs and saving contents to files. In case of previously saved files, the files will be read and no ...
2 votes
0 answers
2k views

Python script to scrape google maps without API

I wrote a python script to scrape google maps for my app. I really want my code to be readable and have tried to follow PEP-8 wherever I could, so I have come to you all for guidance. It uses selenium ...
2 votes
1 answer
95 views

Beautifulsoup scrape posts that has the word python in it with no python tag

This is my first time web scraping, and here is the code I whipped up: ...
1 vote
0 answers
28 views

Trouble returning web scraping output as python dictionary [closed]

So I am attempting to scrape a website of its staff roster (https://www.milb.com/greenville/ballpark/frontoffice) and I want the end product to be a dictionary in the format of {staff: position}. I am ...
5 votes
2 answers
303 views

Python Web scraping

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any. What do I want to achieve? I want to scrape ...
2 votes
1 answer
64 views

Scraping webpage elements using Python

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there ...
5 votes
1 answer
5k views

Python scraping tables

I started practicing in web-scraping few days ago. I made this code to extract data from a wikipedia page. There are several tables that classify mountains based on their height. However there is a ...
3 votes
1 answer
77 views

Refactor Web Scraper

I wrote a simple Zoopla real estate scraper for just practicing what I learned so far in Python, Requests, BeautifulSoup and overall web scraping fundamentals. By looking at my code I feel like there ...
5 votes
0 answers
98 views

Scraping a hiring website using python's requests and BeautifulSoup

I'm designing a scraping application using python, requests and BeautifulSoup4. I decided to divide the logic into two classes: Spider : gets the base url ...
7 votes
1 answer
241 views

Get a movie directory, rename it and save imdb info as a webpage

I have my movie folders formatted: ~: no sub included {}: already watched nothing: sub included, ready to watch the program asks for the movie directory; extracts the name and searches IMDb for the ...
4 votes
1 answer
901 views

Cleaner way of appending data to List in BeautifulSoup

So I've been experimenting various way to get data from different variety of website; as such, between the use of JSON or BeautifulSoup. Currently, I have written a scraper to collect data such as <...
2 votes
1 answer
2k views

Better way to extract html table to dictionary using beautifulsoup in python

I am scraping HTML pages. Part of the page has a table which has acts and sections of those acts mentioned in table format. For some other project I need to convert them to Dictionary. The key values ...
5 votes
1 answer
3k views

Walmart scraping (crawling) code

I am currently trying to make a Walmart Scraper, where I have to scrape the title, link, image URL, price, category and subcategory of the product. Now my approach is that I will collect seed pages ...
2 votes
0 answers
159 views

Web scraping articles using asyncio

I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
2 votes
2 answers
101 views

Web scraping playlist from website and writing to CSV

I have very recently learned some Python and found out about web scraping. This is my first bash at writing some code to get a playlist from a site I had previously been copying and pasting. I would ...
4 votes
1 answer
1k views

Scraping a dynamic website with Scrapy (or Requests) and Selenium

I am trying to use Scrapy for one of the sites I've scraped before using Selenium over here. Because the search field for this site is dynamically generated and requires the user to hover the cursor ...
0 votes
1 answer
920 views

Scraping odds portal with beautiful soup

This code scrapes www.oddsportal.com for all the URLs provided in the code and appends it to a dataframe. I am not very well versed with iterative logic hence I am finding it difficult to improvise on ...
0 votes
1 answer
50 views

Content aggregator using bs4 and requests

Standard code review, tell me what's good, what's bad, and how to improve. Critical suggestions welcomed. This is a content aggregator using bs4 and requests. I did not use any tutorials or help. <...
2 votes
2 answers
308 views

Scrape and parse dictionary website and get meaning of each word in a text file

In this program, I have done automation using Selenium, to find english meaning of each sanskrit word from a dictionary website. The function eng_meaning accepts ...
15 votes
1 answer
60k views

Getting data correctly from <span> tag with beautifulsoup and regex

I am scraping an online shop page, trying to get the price mentioned in that page. In the following block the price is mentioned: ...
31 votes
6 answers
4k views

Python web-scraper to download table of transistor counts from Wikipedia

I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it? ...
5 votes
1 answer
52 views

Scraper for basketball rosters

I am very new to Python and I am trying to learn through personnal projects and today I needed to collect a lot of baksetball player names and decided this was a good time to learn and practice. The ...
3 votes
1 answer
288 views

Python Web-Scraping with BeautifulSoup

I'm relatively new to python and for an assignment I had to write a program that fetches a webpage with BeautifulSoup, extract all Paragraphs from it, and extract all words ending with "ing",...
4 votes
2 answers
729 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
3 votes
1 answer
76 views

Class or not in Page Scraper (Python BeautifulSoup)

I am trying to figure out whether the below is better than just having a raw script or just a script with functions. ...
3 votes
2 answers
117 views

Real time metal prices and conversion fro USD to MXN using BeautifulSoup updated version

...
7 votes
1 answer
759 views

Webscraping application that collects metal prices and converts them from USD to MXN real-time using BeautifulSoup

...
5 votes
1 answer
510 views

FanFiction Downloader in Python, with GUI

I recently have been ̶w̶a̶s̶t̶i̶n̶g̶ spending my time on making a web scraper specifically designed for FanFictions from fanfiction.net (Okay, I'll admit, I like reading them in my spare time that I ...
3 votes
1 answer
51 views

Scrape every Java themed codingbat problem set and your personal solutions

I made the switch from JavaScript to Python lately. This is my first Python project and my first web-scraper. Do I have good practices? And could this be optimized? ...

1
2 3 4 5