3
\$\begingroup\$

Wrote a python script to web scrape multiple newspapers and arrange them in their respective directories. I have completed the course Using Python to access web data on coursera and I tried to implement what I learned by a mini project. I am sure there would be multiple improvements to this script and I would like to learn and implement them to better.

import urllib.request, urllib.error, urllib.parse
from bs4 import BeautifulSoup
import ssl
import requests
import regex as re
import os
from datetime import date, timedelta

today = date.today()

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

def is_downloadable(url):
    """
    Does the url contain a downloadable resource
    """
    h = requests.head(url, allow_redirects=True)
    header = h.headers
    content_type = header.get('content-type')
    if 'text' in content_type.lower():
        return False
    if 'html' in content_type.lower():
        return False
    return True

# dictionary for newspaper names and their links
newspaper = dict({'Economic_times':'https://dailyepaper.in/economic-times-epaper-pdf-download-2020/', 'Times_of_India':'https://dailyepaper.in/times-of-india-epaper-pdf-download-2020/',
    'Financial_Express':'https://dailyepaper.in/financial-express-epaper-pdf-download-2020/', 'Deccan_Chronicle':'https://dailyepaper.in/deccan-chronicle-epaper-pdf-download-2020/',
    'The_Telegraph':'https://dailyepaper.in/the-telegraph-epaper-pdf-download-2020/', 'The_Pioneer':'https://dailyepaper.in/the-pioneer-epaper-pdf-download-2020/',
    'Business_Line':'https://dailyepaper.in/business-line-epaper-pdf-download-2020/', 'Indian_Express':'https://dailyepaper.in/indian-express-epaper-pdf-download-2020/',
    'Hindustan_Times':'https://dailyepaper.in/hindustan-times-epaper-pdf-free-download-2020/', 'The_Hindu':'https://dailyepaper.in/the-hindu-pdf-newspaper-free-download/',
    'Dainik_Jagran':'https://dailyepaper.in/dainik-jagran-newspaper-pdf/', 'Dainik_Bhaskar':'https://dailyepaper.in/dainik-bhaskar-epaper-pdf-download-2020/',
    'Amar_Ujala':'https://dailyepaper.in/amar-ujala-epaper-pdf-download-2020/'})

#dictionary to give serial numbers to each newspaper
#I think something better could be done instead of this dictionary
serial_num = dict({1:'Economic_times', 2:'Times_of_India', 3:'Financial_Express', 4:'Deccan_Chronicle', 5:'The_Telegraph', 6:'The_Pioneer', 7:'Business_Line', 
    8:'Indian_Express', 9:'Hindustan_Times', 10:'The_Hindu', 11:'Dainik_Jagran', 12:'Dainik_Bhaskar', 13:'Amar_Ujala'})

print("The following Newspapers are available for download. Select any of them by giving number inputs - ")
print("1. Economic Times")
print("2. Times of India")
print("3. Financial Express")
print("4. Deccan Chronicle")
print("5. The Telegraph")
print("6. The Pioneer")
print("7. Business Line")
print("8. Indian Express")
print("9. Hindustan Times")
print("10. The Hindu")
print("11. Dainik Jagran")
print("12. Dainik Bhaskar")
print("13. Amar Ujala")

#taking serial numbers for multiple nespapers and storing them in a list
serial_index = input('Enter the number for newspapers - ')
serial_index = serial_index.split()
indices = [int(x) for x in serial_index]

for ser_ind in indices:
    url = newspaper[serial_num[ser_ind]]

    req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    html = urllib.request.urlopen(req).read()
    soup = BeautifulSoup(html, 'html.parser')
    tags = soup('a')
    list_paper = list()

    directory = serial_num[ser_ind]
    parent_dir = os.getcwd()
    path = os.path.join(parent_dir, directory)
    #make a new directory for given newspaper, if that exists then do nothing
    try:
        os.mkdir(path)
    except OSError as error:
        pass
    os.chdir(path) #enter the directory for newspaper

    #storing links for given newspaper in a list
    for i in range(len(tags)):
        links = tags[i].get('href',None)
        x = re.search("^https://vk.com/", links)
        if x:
            list_paper.append(links)

    print('For how many days you need the '+ serial_num[ser_ind]+' paper?')
    print('i.e. if only todays paper press 1, if want whole weeks paper press 7')
    print('Size of each paper is 5-12MB')
    for_how_many_days = int(input('Enter your number - '))

    for i in range(for_how_many_days):
        url = list_paper[i]

        req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
        html = urllib.request.urlopen(req).read()
        soup = BeautifulSoup(html, 'html.parser')
        tags = soup('iframe')
        link = tags[0].get('src',None)      

        date_that_day = today - timedelta(days=i) #getting the date

        if is_downloadable(link):
            print('Downloading '+serial_num[ser_ind]+'...')
            r = requests.get(link, allow_redirects=True)
            with open(serial_num[ser_ind]+"_"+str(date_that_day)+".pdf",'wb') as f:
                f.write(r.content)
            print('Done :)')
        else:
            print(serial_num[ser_ind] + ' paper not available for '+ str(date_that_day))
    os.chdir('../') #after downloading all the newspapers go back to parent directory

```
\$\endgroup\$

2 Answers 2

Reset to default
4
\$\begingroup\$

Usage of requests

Strongly consider replacing your use of bare urllib with requests. It's much more usable. Among other things, it should prevent you from having to worry about an SSL context.

Type hints

def is_downloadable(url):

can be

def is_downloadable(url: str) -> bool:

And so on for your other functions.

Boolean expressions

content_type = header.get('content-type')
if 'text' in content_type.lower():
    return False
if 'html' in content_type.lower():
    return False
return True

can be

content_type = header.get('content-type', '').lower()
return not (
    'text' in content_type or
    'html' in content_type
)

Also note that if a content type is not provided, this function will crash unless you change the default of the get to ''.

Dictionary literals

This:

newspaper = dict({ ...

does not need a call to dict; simply use the braces and they will make a dictionary literal.

URL database

Note what is common in all of your newspaper links and factor it out. In other words, all URLs match the pattern

https://dailyepaper.in/...

so you do not need to repeat the protocol and host in those links; save that to a different constant.

Newspaper objects

dictionary to give serial numbers to each newspaper

I think something better could be done instead of this dictionary

Indeed. Rather than keeping separate dictionaries, consider making a class Newspaper with attributes name: str, link: str and serial: int.

Then after The following Newspapers are available for download, do not hard-code that list; instead loop through your sequence of newspapers and output their serial number and name.

List literals

list_paper = list()

can be

papers = []

Get default

Here:

links = tags[i].get('href',None)

None is the implicit default, so you can omit it. However, it doesn't make sense for you to allow None, because you immediately require a non-null string:

x = re.search("^https://vk.com/", links)

so instead you probably want '' as a default.

String interpolation

'For how many days you need the '+ serial_num[ser_ind]+' paper?'

can be

f'For how many days do you need the {serial_num[ser_ind]} paper?'

Raw transfer

        r = requests.get(link, allow_redirects=True)
        with open(serial_num[ser_ind]+"_"+str(date_that_day)+".pdf",'wb') as f:
            f.write(r.content)

requires that the entire response be loaded into memory before being written out to a file. In the (unlikely) case that the file is bigger than your memory, the program will probably crash. Instead, consider using requests, passing stream=True to your get, and passing response.raw to shutil.copyfileobj. This will stream the response directly to the disk with a much smaller buffer.

\$\endgroup\$
1
\$\begingroup\$

Just one contribution from me: you can get rid of redundant declarations and make your code lighter. The newspapers should be defined just once and then reused. You are almost there. Build a list of dictionaries (or use a database).

# dictionary for newspaper names and their links
newspapers = (
    {"name": 'Economic_times', 'url': 'https://dailyepaper.in/economic-times-epaper-pdf-download-2020/'},
    {"name": 'Times_of_India', 'url': 'https://dailyepaper.in/times-of-india-epaper-pdf-download-2020/'},
    {"name": 'Financial_Express', 'url': 'https://dailyepaper.in/financial-express-epaper-pdf-download-2020/'},
    {"name": 'Deccan_Chronicle', 'url': 'https://dailyepaper.in/deccan-chronicle-epaper-pdf-download-2020/'},
    {"name": 'The_Telegraph', 'url': 'https://dailyepaper.in/the-telegraph-epaper-pdf-download-2020/'},
    {"name": 'The_Pioneer', 'url': 'https://dailyepaper.in/the-pioneer-epaper-pdf-download-2020/'},
    {"name": 'Business_Line', 'url': 'https://dailyepaper.in/business-line-epaper-pdf-download-2020/'},
    {"name": 'Indian_Express', 'url': 'https://dailyepaper.in/indian-express-epaper-pdf-download-2020/'},
    {"name": 'Hindustan_Times', 'url': 'https://dailyepaper.in/hindustan-times-epaper-pdf-free-download-2020/'},
    {"name": 'The_Hindu', 'url': 'https://dailyepaper.in/the-hindu-pdf-newspaper-free-download/'},
    {"name": 'Dainik_Jagran', 'url': 'https://dailyepaper.in/dainik-jagran-newspaper-pdf/'},
    {"name": 'Dainik_Bhaskar', 'url': 'https://dailyepaper.in/dainik-bhaskar-epaper-pdf-download-2020/'},
    {"name": 'Amar_Ujala', 'url': 'https://dailyepaper.in/amar-ujala-epaper-pdf-download-2020/'}
)
print("The following Newspapers are available for download. Select any of them by giving number inputs - ")
for counter, newspaper in enumerate(newspapers, start=1):
    print(f'{counter}. {newspaper["name"]}')

selected_numbers = input('Enter the number for newspapers - ')

print("You selected the following Newspapers:")
for index in selected_numbers.split():
    newspaper_number = int(index)
    newspaper_detail = newspapers[newspaper_number-1]
    print(f"Number: {newspaper_number}")
    print(f"Name: {newspaper_detail['name']}")
    print(f"URL: {newspaper_detail['url']}")

Output:

The following Newspapers are available for download. Select any of them by giving number inputs - 
1. Economic_times
2. Times_of_India
3. Financial_Express
4. Deccan_Chronicle
5. The_Telegraph
6. The_Pioneer
7. Business_Line
8. Indian_Express
9. Hindustan_Times
10. The_Hindu
11. Dainik_Jagran
12. Dainik_Bhaskar
13. Amar_Ujala
Enter the number for newspapers - 1 12 13
You selected the following Newspapers:
Number: 1
Name: Economic_times
URL: https://dailyepaper.in/economic-times-epaper-pdf-download-2020/
Number: 12
Name: Dainik_Bhaskar
URL: https://dailyepaper.in/dainik-bhaskar-epaper-pdf-download-2020/
Number: 13
Name: Amar_Ujala
URL: https://dailyepaper.in/amar-ujala-epaper-pdf-download-2020/

Warning: the code does not check that the input contains valid numbers (use a regex for that), and that all numbers are within the list.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.