I am pretty much a python newbie and was looking at this question on StackOverflow.
Whereas the OP was interested in downloading the
.htm |.txt files, I was simply interested in using
Beautiful Soup and
requests to gather all the links, to those files, into one structure. Ideally, it would have been a single list but I have a list of lists.
The process is:
Start with the landing page and grab all the links in the leftmost column which has the header
CIK column sample with single link selected on right
Navigate to each of those links and grab the document button links in the
Format column sample with single link selected on right
Navigate to each of those document button links and grab the document links in the
Document Format Filessection, if the file mask is
.htm | .txt.
Document Format Files sample with single link selected on right
- Print the list of links (which is a list of lists)
I use CSS selectors throughout to target the links.
What particularly concerns me, given my limited Python experience, is the fact I have a list of lists rather than a single list, whether my re-use of variable names is confusing and potentially bug prone, and I guess whether what I am doing is particularly pythonic and efficient.
In languages I am more familiar with, I might attempt to create a class and provide the class with methods; which in effect are my current functions.
The version of the jupyter notebook server is: 5.5.0
The server is running on this version of Python:
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
from bs4 import BeautifulSoup import requests from urllib.parse import urljoin base = 'https://www.sec.gov' start_urls = ["https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany"] def makeSoup(url): res = requests.get(url) soup = BeautifulSoup(res.content, "lxml") return soup def getLinks(pattern): links = soup.select(pattern) return links def getFullLink(links): outputLinks =  for link in links: outputLinks.append(urljoin(base, link.get("href"))) return outputLinks soup = makeSoup(start_urls) links = getLinks("#seriesDiv [href*='&CIK']") firstLinks = getFullLink(links) penultimateLinks =  for link in firstLinks: soup = makeSoup(link) links = getLinks('[id=documentsbutton]') nextLinks = getFullLink(links) penultimateLinks.append(nextLinks) finalLinks =  for link in penultimateLinks: for nextLink in link: soup = makeSoup(nextLink) links = getLinks("[summary='Document Format Files'] td:nth-of-type(3) [href$='.htm'],td:nth-of-type(3) [href$='.txt']") nextLinks = getFullLink(links) finalLinks.append(nextLinks) print(finalLinks)