I am trying to learn web scraping in Python using Beautiful Soup and requests. My program goes to the book page on Project Gutenberg with the given book number (Example). It then finds the link for the given format (text in this case) and then writes the contents of the book to a file on the hard disk.

How and what can I improve in this code?

import requests
from bs4 import BeautifulSoup
def go_gutenberg(file_format,book_no):
       url = "https://www.gutenberg.org/ebooks/"
       r =requests.get(url+str(book_no))
       r_html  = r.text
       soup = BeautifulSoup(r_html,"html.parser")
       for file in soup.find_all('a',class_="link"):
         if file_format in file.text:
           g = requests.get("https:"+get_book)

           with open("C:\\Users\\syed\\Documents\\Gutenberg\\Book"+str(book_no)+".txt",'wb') as open_file:
               for chunk in g.iter_content(10000):

def main():

if __name__=="__main__":main()

And this is sample HTML of the book links:

<a href="//www.gutenberg.org/files/53092/53092-0.txt"   
type="text/plain;charset=utf-8" charset="utf-8" class="link" 
title="Download">Plain Text UTF-8</a>
  • 3
    \$\begingroup\$ Just a heads-up, they have specific rules about automating these downloads. Read more about automated downloading of the ebooks at: gutenberg.org/wiki/Gutenberg:Terms_of_Use Which tells us, after you hit 100 ebooks, you will be automatically blocked. Not sure how many you want to scrape, but might be a important detail to keep in mind. \$\endgroup\$ Commented Sep 20, 2016 at 10:18
  • \$\begingroup\$ This is just for learning , I wont scrape that many . Thanks for the info , by the way. \$\endgroup\$
    – Syed Saad
    Commented Sep 20, 2016 at 10:27

1 Answer 1


Easy stuff

There are a lot of stylistic issues (indentation, spacing between operators, etc.) that violate PEP 8, you can use tools like pylint to find what they are.

Moving on

 r_html = r.text

Do you really need this? You don't save any characters (not that it is always about saving characters) and you use r_html once. I would just get rid of the line.

I would also say the same about:

url = "https://www.gutenberg.org/ebooks/"

Just get rid of it. Also, in this instance in particular I would get rid of it, because when I read it, I believe there might be a way to abstract it. "Maybe let the user change the url, somehow?") There isn't a particularly good way to "abstract" this url out in particular.

Furthermore, it is considered good practice to use format instead of the addition of strings so:

"https://www.gutenberg.org/ebooks/" + str(book_no)


"https://www.gutenberg.org/ebooks/%d" % book_no

This on the other hand:


Seems like it can be abstracted, what if the user wants to change the location? Maybe by default use this location, but allow the user to change it. Create a default argument to "C:\\Users\\syed\\Documents\\Gutenberg\\Book" but give the user the option to specify it.

Same with:

for chunk in g.iter_content(10000):

What is 10000? Maybe make a default argument for this with the value of 10000 and the name of (possibly) chunk_size?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.