Skip to main content
88 votes
Accepted

Tiny image scraper for xkcd.com

Stop hammering XKCD server You basically wrote a denial-of-service attack, since you try to connect as fast and as often as possible to the server. By sharing this code on CodeReview, it becomes a ...
Eric Duminil's user avatar
  • 3,950
41 votes

Web scraping the titles and descriptions of trending YouTube videos

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a ...
200_success's user avatar
23 votes
Accepted

Web crawler in Rust

Cargo Fmt There's a very common tool accessible through cargo which can format all of the code in your project according to Rust's official style guide. Many major ...
Cedric Hutchings's user avatar
20 votes
Accepted

Let's read a random Goodreads book in an optimal way

After I've gone through your code, I decided to rewrite it because using regex to parse html isn't a good idea at all. When you're parsing html, is recommended to use ...
Grajdeanu Alex's user avatar
15 votes

Web scraping the titles and descriptions of trending YouTube videos

Context manager You open a file at the beginning of the program and close it explicitly at the end. Python provides a nice way to allocate and release resources (such as files) easily: they are ...
SylvainD's user avatar
  • 29.2k
15 votes

Tiny image scraper for xkcd.com

Globals As it is, there would be an advantage to LOGO being a local instead of a global - it's only used by show_logo, and ...
Reinderien's user avatar
  • 62.4k
15 votes
Accepted

Function in Python to extract web Data

Use Requests for HTTP and let it deal with the encoding Do not bake the query into the URL string Use BeautifulSoup for HTML parsing Use typehints Nice and simple: ...
Reinderien's user avatar
  • 62.4k
13 votes
Accepted

Image downloader for a website v2

There's a lot to improve about your code, and I'll start from the top. I'll also rewrite some of the logic you had there because IMO it's a bit clumsy and hard to understand. Imports Don't use ...
Grajdeanu Alex's user avatar
13 votes
Accepted

DeliciousSoda: A simple parser for robots.txt files

I tried using __ once and found it to produce more problems than it's worth. Take: ...
Peilonrayz's user avatar
  • 43.4k
12 votes
Accepted

Comic Image Web Scraper

This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed. General You use two spaces for ...
Daniel's user avatar
  • 4,490
12 votes
Accepted

Python script to scrape and parse the Stanford Encyclopedia of Philosophy

Add a hashbang at the top of your script since it's expected to be executable. There is no function but I didn't think it was worth adding Functions are worth adding: the visual and performance ...
Reinderien's user avatar
  • 62.4k
11 votes

Simple Python username scraper

If you're after speed, I'd suggest scrapy. I was looking for an excuse to try it out and saw your question. When I ran your code on the first 10 pages of the NA ...
smarx's user avatar
  • 211
11 votes
Accepted

Simple Python job vacancies downloader

A couple of suggestions: Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it ...
Grajdeanu Alex's user avatar
11 votes
Accepted

Parsing Wikipedia table with Python

The right tool As you've said, you are not using the right tool for this task : you can't parse HTML with regexps. A better approach would be to use an already existing parser like BeautifulSoup. A ...
SylvainD's user avatar
  • 29.2k
11 votes
Accepted

Items derived from one function are getting printed within another for loop in another function

First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions): initialize session as ...
alecxe's user avatar
  • 17.3k
11 votes

Tiny image scraper for xkcd.com

...
kutschkem's user avatar
  • 415
11 votes

Tiny image scraper for xkcd.com

As mentioned in a comment by Thomas, another option is to use XKCD's JSON interface rather than scraping HTML: ...
suvonnie's user avatar
  • 119
10 votes
Accepted

Scraping after login using Scrapy

I'll start off by saying this is actually really good. There are a few minor PEP8 errors: In after_login you call Request. the ...
Peilonrayz's user avatar
  • 43.4k
10 votes
Accepted

VIM colors downloader in Python

Very nice code in general! Here are a few points, though: Think about using os.makedirs instead of os.mkdir. The former allows ...
Graipher's user avatar
  • 41k
10 votes

Image downloader for a website v2

Do not use capitalization for function or variable names. This violates idiomatic python naming conventions (PEP8 link). Try out pylint to catch these kinds of issues Prefer ...
Bryce Guinta's user avatar
10 votes

Webscraper for www.coinmarketcap.com

Welcome to Code Review SE, nice first question! Here are some of my comments, not pertaining to performance: There's no reason to import modules in the body of ...
Daniel's user avatar
  • 4,490
10 votes

Web scraping the titles and descriptions of trending YouTube videos

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly ...
alecxe's user avatar
  • 17.3k
10 votes

Asynchronously scrape fabric information

While you can put request parameters in the URI directly, it's usually neater to pass those using the params argument of ...
Sara J's user avatar
  • 3,471
9 votes
Accepted

A web scraper that looks for pre-defined words in news articles

At first glance, I'd say that this code isn't bad. I am particularly impressed by this line: regex = r"\b"+ re.escape(word) + r"\b" because you took the care to ...
200_success's user avatar
9 votes
Accepted

Chart showing the Percentage of Answered CR questions

When I ran your script and started digging through the dom, I noticed that Apr 21 12:00 AM was missing -- I wonder why that was. Anyhow, I wanted to encourage ...
mickmackusa's user avatar
  • 8,674
9 votes
Accepted

AniPop - The anime downloader

I have no familiarity with any of the libraries used here, so I can't comment on their usage. What I will mention though is the giant chunk of ...
Carcigenicate's user avatar
9 votes
Accepted

Webscraping application that collects metal prices and converts them from USD to MXN real-time using BeautifulSoup

Check for failure metals = requests.get('https://www.lme.com/').text should be ...
Reinderien's user avatar
  • 62.4k
8 votes

VIM colors downloader in Python

Nice script! A few points on top of what @Graipher already pointed out. Passing url parameters to requests This is a hacky way to pass query parameters: ...
janos's user avatar
  • 111k
8 votes
Accepted

E-mail crawler for yellowpages

Knowing that you had a ton of scraper questions, this doesn't look nice at all. It looks like you were in a hurry and didn't much care about how the code looks. Styling Use 4 spaces per indentation ...
Grajdeanu Alex's user avatar
8 votes
Accepted

Simple yet Hackable WhatsApp API [UNOFFICIAL]

Here is a collection of things to improve. Code Style Notes Overall, there is a number of PEP8 Code Style violations, like lack of spaces around operators, blank lines, spaces before the ...
alecxe's user avatar
  • 17.3k

Only top scored, non community-wiki answers of a minimum length are eligible