90 votes
Accepted

Tiny image scraper for xkcd.com

Stop hammering XKCD server You basically wrote a denial-of-service attack, since you try to connect as fast and as often as possible to the server. By sharing this code on CodeReview, it becomes a ...
user avatar
  • 3,948
40 votes

Web scraping the titles and descriptions of trending YouTube videos

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a ...
user avatar
28 votes
Accepted

We'll be counting stars

That's a beautiful excuse for writing code, and the final product is quite nice as well. ★ First of all, congrats to your Java-ridden mind for not forcing classes into Python where they aren't needed....
user avatar
  • 12.4k
27 votes
Accepted

An OEIS Lookup tool in Python

import*: Please avoid importing all modules at all costs. Import each module you use separately Regex ಠ_ಠ Regex is evil, the ...
user avatar
  • 11.8k
23 votes
Accepted

Web crawler in Rust

Cargo Fmt There's a very common tool accessible through cargo which can format all of the code in your project according to Rust's official style guide. Many major ...
user avatar
22 votes
Accepted

Spliterator implementation

I see a few, actually, a number of problems in this code. I am afraid this will be something of a scathing review, in part because you mention this is intended to be for a tutorial.... I see a number ...
user avatar
  • 96.7k
21 votes

An OEIS Lookup tool in Python

You do not need HTML parsing at all. OEIS has a nice JSON output format. https://oeis.org/search?fmt=json&q=id:A000045 So the core functionality of your program can be written as something like ...
user avatar
  • 311
20 votes
Accepted

Let's read a random Goodreads book in an optimal way

After I've gone through your code, I decided to rewrite it because using regex to parse html isn't a good idea at all. When you're parsing html, is recommended to use ...
user avatar
18 votes
Accepted

Minimal webcrawler - bad structure and error handling?

import urllib import re import os # The parameter is the url the crawler is gonna crawl. That's a pretty useless comment. The parameter is named ...
user avatar
16 votes
Accepted

Refactoring a Crawler

Assignment Branch Condition size for fetch_images is too high. [22.41/15] What that warning is saying is that, according to rubocop's standard, you have too many ...
user avatar
  • 2,722
15 votes

Web scraping the titles and descriptions of trending YouTube videos

Context manager You open a file at the beginning of the program and close it explicitly at the end. Python provides a nice way to allocate and release resources (such as files) easily: they are ...
user avatar
  • 28.7k
15 votes

Tiny image scraper for xkcd.com

Globals As it is, there would be an advantage to LOGO being a local instead of a global - it's only used by show_logo, and ...
user avatar
  • 47.7k
15 votes
Accepted

Function in Python to extract web Data

Use Requests for HTTP and let it deal with the encoding Do not bake the query into the URL string Use BeautifulSoup for HTML parsing Use typehints Nice and simple: ...
user avatar
  • 47.7k
13 votes
Accepted

Image downloader for a website v2

There's a lot to improve about your code, and I'll start from the top. I'll also rewrite some of the logic you had there because IMO it's a bit clumsy and hard to understand. Imports Don't use ...
user avatar
13 votes
Accepted

DeliciousSoda: A simple parser for robots.txt files

I tried using __ once and found it to produce more problems than it's worth. Take: ...
user avatar
  • 41.6k
12 votes
Accepted

Comic Image Web Scraper

This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed. General You use two spaces for ...
user avatar
  • 4,492
11 votes
Accepted

IP and router connections

Unused variables x, proc_c, start and JustDOit are set ...
user avatar
11 votes

Simple Python username scraper

If you're after speed, I'd suggest scrapy. I was looking for an excuse to try it out and saw your question. When I ran your code on the first 10 pages of the NA ...
user avatar
  • 211
11 votes
Accepted

Simple Python job vacancies downloader

A couple of suggestions: Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it ...
user avatar
11 votes
Accepted

Parsing Wikipedia table with Python

The right tool As you've said, you are not using the right tool for this task : you can't parse HTML with regexps. A better approach would be to use an already existing parser like BeautifulSoup. A ...
user avatar
  • 28.7k
11 votes
Accepted

Items derived from one function are getting printed within another for loop in another function

First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions): initialize session as ...
user avatar
  • 17k
11 votes

Tiny image scraper for xkcd.com

...
user avatar
  • 405
11 votes

Tiny image scraper for xkcd.com

As mentioned in a comment by Thomas, another option is to use XKCD's JSON interface rather than scraping HTML: ...
user avatar
  • 119
10 votes
Accepted

Scraping after login using Scrapy

I'll start off by saying this is actually really good. There are a few minor PEP8 errors: In after_login you call Request. the ...
user avatar
  • 41.6k
10 votes
Accepted

VIM colors downloader in Python

Very nice code in general! Here are a few points, though: Think about using os.makedirs instead of os.mkdir. The former allows ...
user avatar
  • 40.5k
10 votes

Image downloader for a website v2

Do not use capitalization for function or variable names. This violates idiomatic python naming conventions (PEP8 link). Try out pylint to catch these kinds of issues Prefer ...
user avatar
10 votes

Webscraper for www.coinmarketcap.com

Welcome to Code Review SE, nice first question! Here are some of my comments, not pertaining to performance: There's no reason to import modules in the body of ...
user avatar
  • 4,492
10 votes

Web scraping the titles and descriptions of trending YouTube videos

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly ...
user avatar
  • 17k
10 votes

Asynchronously scrape fabric information

While you can put request parameters in the URI directly, it's usually neater to pass those using the params argument of ...
user avatar
  • 2,499
9 votes
Accepted

Scrape an HTML table with python

In general, your code is very well written. PEP8 compliance is very good, names make sense, you have docstrings, you use if __name__ == '__main__:. Here's a few ...
user avatar
  • 5,160

Only top scored, non community-wiki answers of a minimum length are eligible