I am trying to learn web scraping in Python using Beautiful Soup and requests. My program goes to the book page on Project Gutenberg with the given book number (Example). It then finds the link for the given format (text in this case) and then writes the contents of the book to a file on the hard disk.
How and what can I improve in this code?
import requests
from bs4 import BeautifulSoup
def go_gutenberg(file_format,book_no):
url = "https://www.gutenberg.org/ebooks/"
r =requests.get(url+str(book_no))
r_html = r.text
soup = BeautifulSoup(r_html,"html.parser")
for file in soup.find_all('a',class_="link"):
if file_format in file.text:
get_book=file.get('href')
g = requests.get("https:"+get_book)
with open("C:\\Users\\syed\\Documents\\Gutenberg\\Book"+str(book_no)+".txt",'wb') as open_file:
for chunk in g.iter_content(10000):
open_file.write(chunk)
def main():
go_gutenberg("Text",1000)
if __name__=="__main__":main()
And this is sample HTML of the book links:
<a href="//www.gutenberg.org/files/53092/53092-0.txt" type="text/plain;charset=utf-8" charset="utf-8" class="link" title="Download">Plain Text UTF-8</a>