Script to download Annual Reports (~70,000 in total) from a Website

Question

It's only 38 lines of code and I haven't worked with web scraping that much before.

• How heavy would my code be on their server?
• It's for Deep Learning purposes and I haven't ran it yet but would it result in my IP getting banned a quarter way through once I start downloading the 70k pdfs?

Also, I'm not sure how efficient this is (right now I don't think I'm even checking if a file exists and I'm overwriting it each time I run my program, so if my code is interrupted halfway through, I'll have to run it again and it'll start downloading everything again from scratch. I'll have to fix that).

But anyways, here is the code:

# Scrapes all pdfs off from www.annualreports.com
# Haven't tested yet but should be somewhere around 70,963 pdfs since my empty search returned 5,479/5,674 stated companies

import requests
from urllib.parse import urljoin
import pandas as pd
from bs4 import BeautifulSoup as bs
import os, re
import pickle

def extract_table():
    r = requests.get('http://www.annualreports.com/Companies?search=')
    soup = bs(r.content, 'lxml')
    df = pd.DataFrame([(i.text, 'http://www.annualreports.com' + i['href']) for i in soup.select('tbody td:nth-of-type(1) a')], columns = ['Company','Link'])
    df.to_pickle('links.pkl') # saves into a dataframe the name of the company plus the href link it points to

def scrap_pdfs():
    df = pd.read_pickle('links.pkl')
    a = 0 # for naming the filenames numerically

    for x in range(df.Link.count()):

        url = df['Link'][x] # reads the "Link" column from the dataframe

        folder_location = r'/home/duke/Annual_Reports/Data' # SPECIFY FULL DIRECTORY

        response = requests.get(url)
        soup= bs(response.text, "html.parser")

        for link in soup.select("a[href$='.pdf']"): #goes through all the .pdfs in all of the href links

            #filename = os.path.join(folder_location, ['href'].split('/')[-1]) # Names the pdf files using the last portion of each link

            filename = os.path.join(folder_location, str(a))
            a+=1

            with open(filename, 'wb') as f:
                f.write(requests.get(urljoin(url,link['href'])).content)

Answer 1

Yes, right now this would be inefficient, I'd definitely say that using a predictable filename would be the first key change here, if the link target or description isn't usable as part of the filename, consider hashing it. Simply enumerating all links is, well, suboptimal: What if, even if previous files are kept around and not overwritten, between runs the contents of the website changes (seems likely, doesn't it?) and suddenly the order of the files being download doesn't match what's on disk? Definitely do something about that.

Secondly, the code's separating the creation of that pickle file because it's supposed to be run multiple times? A bit odd, but as an impromptu database, why not. I usually consider pickle files somewhat ephemeral since the format is (was?) specific to Python.

And then simply check if the file exists and skip it (maybe also check if it's non-empty; going further before downloading the full data the HTTP header for the size of the returned content could also be compared, again, before actually downloading the full files).

The range over the data frame I'd have expected to be simpler, but right now I can't find if there's an easier way (not using the index, but simply looping over the values themselves that is).

f.write(requests.get(...).content) - that's gonna buffer the full file in memory I think. Better don't do that, instead either requests might have some facility to write directly to files (or a file-like object), or alternatively the download content would have to be read piece-by-piece into a smallish buffer, alternating it with writing that buffer to the output file.

Finally the formatting could be more consistent (whitespace between expressions mostly). Take a look at PEP8 and perhaps an autoformatter to do that automatically. Also leftover comments should be removed.

Right so overall it's effective, but for production code there's lots of things that could be done, especially making it a more fully fleshed out script (read: parse command line arguments, have default parameters for e.g. the output directory; etc.). The parsing is fine and the selectors again are succinct and easy to read.