90
votes
Accepted
Tiny image scraper for xkcd.com
Stop hammering XKCD server
You basically wrote a denial-of-service attack, since you try to connect as fast and as often as possible to the server.
By sharing this code on CodeReview, it becomes a ...
- 3,948
40
votes
Web scraping the titles and descriptions of trending YouTube videos
Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a ...
- 142k
28
votes
Accepted
We'll be counting stars
That's a beautiful excuse for writing code, and the final product is quite nice as well.
★ First of all, congrats to your Java-ridden mind for not forcing classes into Python where they aren't needed....
- 12.4k
27
votes
Accepted
An OEIS Lookup tool in Python
import*:
Please avoid importing all modules at all costs. Import each module you use separately
Regex ಠ_ಠ
Regex is evil, the ...
- 11.8k
23
votes
Accepted
Web crawler in Rust
Cargo Fmt
There's a very common tool accessible through cargo which can format all of the code in your project according to Rust's official style guide. Many major ...
- 406
22
votes
Accepted
Spliterator implementation
I see a few, actually, a number of problems in this code. I am afraid this will be something of a scathing review, in part because you mention this is intended to be for a tutorial....
I see a number ...
- 96.7k
21
votes
An OEIS Lookup tool in Python
You do not need HTML parsing at all. OEIS has a nice JSON output format.
https://oeis.org/search?fmt=json&q=id:A000045
So the core functionality of your program can be written as something like
...
- 311
20
votes
Accepted
Let's read a random Goodreads book in an optimal way
After I've gone through your code, I decided to rewrite it because using regex to parse html isn't a good idea at all.
When you're parsing html, is recommended to use ...
- 8,917
18
votes
Accepted
Minimal webcrawler - bad structure and error handling?
import urllib
import re
import os
# The parameter is the url the crawler is gonna crawl.
That's a pretty useless comment. The parameter is named ...
- 28.6k
16
votes
Accepted
Refactoring a Crawler
Assignment Branch Condition size for fetch_images is too high. [22.41/15]
What that warning is saying is that, according to rubocop's standard, you have too many ...
- 2,722
15
votes
Web scraping the titles and descriptions of trending YouTube videos
Context manager
You open a file at the beginning of the program and close it explicitly at the end.
Python provides a nice way to allocate and release resources (such as files) easily: they are ...
- 28.7k
15
votes
Tiny image scraper for xkcd.com
Globals
As it is, there would be an advantage to LOGO being a local instead of a global - it's only used by show_logo, and ...
- 47.7k
15
votes
Accepted
Function in Python to extract web Data
Use Requests for HTTP and let it deal with the encoding
Do not bake the query into the URL string
Use BeautifulSoup for HTML parsing
Use typehints
Nice and simple:
...
- 47.7k
13
votes
Accepted
Image downloader for a website v2
There's a lot to improve about your code, and I'll start from the top. I'll also rewrite some of the logic you had there because IMO it's a bit clumsy and hard to understand.
Imports
Don't use ...
- 8,917
13
votes
Accepted
DeliciousSoda: A simple parser for robots.txt files
I tried using __ once and found it to produce more problems than it's worth. Take:
...
- 41.6k
12
votes
Accepted
Comic Image Web Scraper
This seems like a cool project, and a nice first introduction to web scraping! I'll cover some general advice first, and then address your main concern: speed.
General
You use two spaces for ...
- 4,492
11
votes
Accepted
11
votes
Simple Python username scraper
If you're after speed, I'd suggest scrapy. I was looking for an excuse to try it out and saw your question. When I ran your code on the first 10 pages of the NA ...
- 211
11
votes
Accepted
Simple Python job vacancies downloader
A couple of suggestions:
Use requests instead of urllib. This is mostly a matter of preference but I think you'll find it ...
- 8,917
11
votes
Accepted
Parsing Wikipedia table with Python
The right tool
As you've said, you are not using the right tool for this task : you can't parse HTML with regexps.
A better approach would be to use an already existing parser like BeautifulSoup.
A ...
- 28.7k
11
votes
Accepted
Items derived from one function are getting printed within another for loop in another function
First of all, usual things (I feel like I'm suggesting these things in most of the web-scraping related discussions):
initialize session as ...
- 17k
11
votes
11
votes
Tiny image scraper for xkcd.com
As mentioned in a comment by Thomas, another option is to use XKCD's JSON interface rather than scraping HTML:
...
- 119
10
votes
Accepted
Scraping after login using Scrapy
I'll start off by saying this is actually really good.
There are a few minor PEP8 errors:
In after_login you call Request. the ...
- 41.6k
10
votes
Accepted
VIM colors downloader in Python
Very nice code in general! Here are a few points, though:
Think about using os.makedirs instead of os.mkdir. The former allows ...
- 40.5k
10
votes
Image downloader for a website v2
Do not use capitalization for function or variable names. This violates idiomatic python naming conventions (PEP8 link). Try out pylint to catch these kinds of issues
Prefer ...
- 341
10
votes
Webscraper for www.coinmarketcap.com
Welcome to Code Review SE, nice first question! Here are some of my comments, not pertaining to performance:
There's no reason to import modules in the body of ...
- 4,492
10
votes
Web scraping the titles and descriptions of trending YouTube videos
I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly ...
- 17k
10
votes
Asynchronously scrape fabric information
While you can put request parameters in the URI directly, it's usually neater to pass those using the params argument of ...
- 2,499
9
votes
Accepted
Scrape an HTML table with python
In general, your code is very well written. PEP8 compliance is very good, names make sense, you have docstrings, you use if __name__ == '__main__:.
Here's a few ...
- 5,160
Only top scored, non community-wiki answers of a minimum length are eligible