60
votes
Python web-scraper to download table of transistor counts from Wikipedia
Is there a way to simplify this code?
Yes. Don't scrape Wikipedia. Your first thought before "should I need to scrape this thing?" should be "Is there an API that can give me the data I want?" In ...
- 65.4k
51
votes
Python web-scraper to download table of transistor counts from Wikipedia
Let me tell you about IMPORTHTML()...
So here's all the code you need in Google Sheets:
...
- 3,960
41
votes
Web scraping the titles and descriptions of trending YouTube videos
Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a ...
- 144k
20
votes
Accepted
Financial website scraper
Here a few suggestions (in no particular order) for improvements on your code:
Make variables have locality
The line:
items = 1
is at the top of the method, ...
- 4,214
16
votes
Financial website scraper
In addition to what @StephenRauch wrote in his excellent answer, let me add one more comment:
Your class is completely useless. Not its functionality, which is fine, but it being a ...
- 41.1k
16
votes
Python web-scraper to download table of transistor counts from Wikipedia
Make sure to follow naming conventions. You name two variables inappropriately:
...
- 16.3k
15
votes
Web scraping the titles and descriptions of trending YouTube videos
Context manager
You open a file at the beginning of the program and close it explicitly at the end.
Python provides a nice way to allocate and release resources (such as files) easily: they are ...
- 29.3k
12
votes
Accepted
Comic Image Web Scraper
This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed.
General
You use two spaces for ...
- 4,500
11
votes
Accepted
Simple Python job vacancies downloader
A couple of suggestions:
Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it ...
- 9,176
11
votes
Accepted
Items derived from one function are getting printed within another for loop in another function
First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions):
initialize session as ...
- 17.3k
10
votes
Web scraping the titles and descriptions of trending YouTube videos
I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly ...
- 17.3k
9
votes
Accepted
A web scraper that looks for pre-defined words in news articles
At first glance, I'd say that this code isn't bad. I am particularly impressed by this line:
regex = r"\b"+ re.escape(word) + r"\b"
because you took the care to ...
- 144k
9
votes
Accepted
AniPop - The anime downloader
I have no familiarity with any of the libraries used here, so I can't comment on their usage.
What I will mention though is the giant chunk of ...
- 16.3k
9
votes
Accepted
Webscraping application that collects metal prices and converts them from USD to MXN real-time using BeautifulSoup
Check for failure
metals = requests.get('https://www.lme.com/').text
should be
...
- 65.4k
8
votes
Accepted
Downloading images from URLs
Notes about your code
You can also simplify the way you look for the links to follow:
...
- 17.3k
8
votes
Items derived from one function are getting printed within another for loop in another function
I would use requests.Session, which allows re-using the connection. This speeds up successive requests (as you are doing here).
I would factor out the getting and ...
- 41.1k
8
votes
Financial website scraper
You should also split the code in functional parts, and not mix the IO (print(...)) with generating the result
base_url
First you generate the base_url
...
- 9,130
7
votes
Accepted
River Flood Warning system in Python
Improving Data-mining
Using a list of dictionaries
What if you, instead of keeping and supporting multiple data structures, would construct a list of dictionaries where each dictionary correspond to ...
- 17.3k
7
votes
Console based Weather forecasting app
One of the big issue with this program is the variable names. Variable names like a, b, c, <...
- 3,095
7
votes
Accepted
Archives xkcd comics
First of all, you have multiple PEP8 code style violations, like using camel case styled variable naming instead of a "lower case with underscores"; organizing, grouping imports, comments not ...
- 17.3k
7
votes
Web-scraping a list of lawyers
Don't use a bare except statement if it's not absolutely necessary. Catch the errors you anticipate and explicitly want to handle.
Also, there's no reason for the ...
- 2,030
7
votes
Accepted
Web-scraping a list of lawyers
As @scnerd already discussed avoiding the use of a bare try/except clause, I'll focus on removing it entirely.
After inspecting website source, you will notice ...
- 1,100
7
votes
Python web-scraper to download table of transistor counts from Wikipedia
Using the Wikipedia API
As mentioned in another answer, Wikipedia provides an HTTP API for fetching article content, and you can use it to get the content in much cleaner formats than HTML. Often ...
- 404
7
votes
Accepted
OOP Web-scraper w/ Python and BeautifulSoup
Bugs
Lines 32 and 42:
for titles, info in zip(self, titles, self.comicinfo):
for dol_amount in comic_prices:
both have unresolved variable ...
- 65.4k
7
votes
code format and steps web scraping using beautiful soup
I agree here is little to review. Two tips though: use requests.Session() and always check the status code. If it's not 200, the request has not succeeded. You should not proceed with parsing.
In the ...
- 7,578
6
votes
Simple Python job vacancies downloader
Building on @Dex'ter answer, I think that using a dictionary to store your results call for the wrong iteration patterns. I would store them in a list and grab them directly with a ...
- 29.1k
6
votes
Webscrape Podcast site for mp3s to (legally) add to collection
There is a space for multiple improvements, here are some of the high-level ideas:
the code itself is long and "complex" and should definitely be "cut" into pieces logically - apply "Extract Method" ...
- 17.3k
6
votes
Webscrape Podcast site for mp3s to (legally) add to collection
Comments inline.
usr = 'real usr removed'
pwd = 'real pwd removed'
Get these from a configuration file. Never store credentials in your code. Check out the ...
- 8,462
6
votes
Comic Image Web Scraper
I'm adding a few points to what @Coal_ wrote in his excellent answer.
You can use lxml instead of html.parser as it is faster (...
- 165
6
votes
Accepted
Python - Iterating over contents of BeautifulSoup element very slow
It seems like you have the following:
...
- 2,904
Only top scored, non community-wiki answers of a minimum length are eligible