60 votes

Python web-scraper to download table of transistor counts from Wikipedia

Is there a way to simplify this code? Yes. Don't scrape Wikipedia. Your first thought before "should I need to scrape this thing?" should be "Is there an API that can give me the data I want?" In ...
user avatar
51 votes

Python web-scraper to download table of transistor counts from Wikipedia

Let me tell you about IMPORTHTML()... So here's all the code you need in Google Sheets: ...
user avatar
  • 3,958
41 votes

Web scraping the titles and descriptions of trending YouTube videos

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a ...
user avatar
27 votes
Accepted

An OEIS Lookup tool in Python

import*: Please avoid importing all modules at all costs. Import each module you use separately Regex ಠ_ಠ Regex is evil, the ...
user avatar
  • 11.8k
21 votes

An OEIS Lookup tool in Python

You do not need HTML parsing at all. OEIS has a nice JSON output format. https://oeis.org/search?fmt=json&q=id:A000045 So the core functionality of your program can be written as something like ...
user avatar
  • 311
20 votes
Accepted

Financial website scraper

Here a few suggestions (in no particular order) for improvements on your code: Make variables have locality The line: items = 1 is at the top of the method, ...
user avatar
16 votes

Financial website scraper

In addition to what @StephenRauch wrote in his excellent answer, let me add one more comment: Your class is completely useless. Not its functionality, which is fine, but it being a ...
user avatar
  • 40.6k
16 votes

Python web-scraper to download table of transistor counts from Wikipedia

Make sure to follow naming conventions. You name two variables inappropriately: ...
user avatar
15 votes

Web scraping the titles and descriptions of trending YouTube videos

Context manager You open a file at the beginning of the program and close it explicitly at the end. Python provides a nice way to allocate and release resources (such as files) easily: they are ...
user avatar
  • 28.7k
12 votes
Accepted

Analyze frequency and content of political fundraising E-mails

Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... ...
user avatar
12 votes
Accepted

Comic Image Web Scraper

This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed. General You use two spaces for ...
user avatar
  • 4,492
11 votes

Simple Python username scraper

If you're after speed, I'd suggest scrapy. I was looking for an excuse to try it out and saw your question. When I ran your code on the first 10 pages of the NA ...
user avatar
  • 211
11 votes
Accepted

Simple Python job vacancies downloader

A couple of suggestions: Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it ...
user avatar
11 votes
Accepted

Items derived from one function are getting printed within another for loop in another function

First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions): initialize session as ...
user avatar
  • 17.1k
10 votes
Accepted

VIM colors downloader in Python

Very nice code in general! Here are a few points, though: Think about using os.makedirs instead of os.mkdir. The former allows ...
user avatar
  • 40.6k
10 votes

Web scraping the titles and descriptions of trending YouTube videos

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly ...
user avatar
  • 17.1k
9 votes
Accepted

Storytelling program: Gathers stories from certain popular story websites/formats and reads them aloud

I'm just going to review these eight lines of code. You'll see that there's plenty here for one answer. ...
user avatar
  • 49.2k
9 votes
Accepted

A web scraper that looks for pre-defined words in news articles

At first glance, I'd say that this code isn't bad. I am particularly impressed by this line: regex = r"\b"+ re.escape(word) + r"\b" because you took the care to ...
user avatar
9 votes
Accepted

AniPop - The anime downloader

I have no familiarity with any of the libraries used here, so I can't comment on their usage. What I will mention though is the giant chunk of ...
user avatar
9 votes
Accepted

Webscraping application that collects metal prices and converts them from USD to MXN real-time using BeautifulSoup

Check for failure metals = requests.get('https://www.lme.com/').text should be ...
user avatar
8 votes

Finding the occurrences of all words in movie scripts

A few things I can think of are: parse_text can be marked as a private method. def _parse_text(self):. I understand this won't ...
user avatar
  • 472
8 votes

VIM colors downloader in Python

Nice script! A few points on top of what @Graipher already pointed out. Passing url parameters to requests This is a hacky way to pass query parameters: ...
user avatar
  • 107k
8 votes
Accepted

Downloading images from URLs

Notes about your code You can also simplify the way you look for the links to follow: ...
user avatar
  • 17.1k
8 votes

Items derived from one function are getting printed within another for loop in another function

I would use requests.Session, which allows re-using the connection. This speeds up successive requests (as you are doing here). I would factor out the getting and ...
user avatar
  • 40.6k
8 votes

Financial website scraper

You should also split the code in functional parts, and not mix the IO (print(...)) with generating the result base_url First you generate the base_url ...
user avatar
7 votes
Accepted

Soup of the day: best served during election season

Since your list SITES_INFO_HELPER remains constant throughout the entire program, that list should be a tuple instead Tuples are faster than lists and are write-...
user avatar
  • 13.2k
7 votes

Reddit bot to check reposts

Use modern versions of things The most obvious thing here is use Python 3. This will help massively with your Unicode problems, because Python 3 maintains a stricter separation between things that ...
user avatar
  • 1,178
7 votes
Accepted

River Flood Warning system in Python

Improving Data-mining Using a list of dictionaries What if you, instead of keeping and supporting multiple data structures, would construct a list of dictionaries where each dictionary correspond to ...
user avatar
  • 17.1k
7 votes

Console based Weather forecasting app

One of the big issue with this program is the variable names. Variable names like a, b, c, <...
user avatar
7 votes
Accepted

Archives xkcd comics

First of all, you have multiple PEP8 code style violations, like using camel case styled variable naming instead of a "lower case with underscores"; organizing, grouping imports, comments not ...
user avatar
  • 17.1k

Only top scored, non community-wiki answers of a minimum length are eligible