Return to Revisions

Code Style

There are multiple PEP8 code style violations, some of them are:

• naming - use lower_case_with_underscores naming style
• spaces around operators
• blank lines between imports and functions

You should also improve your variable naming - for example, d, b, i are not meaningful - think of more descriptive names - remember: Code is much more often read than written.

HTML-parsing and Web-scraping

• instantiate requests.Session() and reuse to make requests - this would give you a performance boost "for free":

if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

• you can replace .find_all() calls with more explicit and robust select() calls and use CSS selectors. For instance, soup.find_all("h1", "branded-page-header-title") would become soup.select("h1.branded-page-header-title")

• it is also a good idea to specify a parse that BeautifulSoup uses under-the-hood explicitly:

   soup = BeautifulSoup(r.text, "html.parser")  # built-in, no extra dependencies
   # soup = BeautifulSoup(r.text, "lxml")  # the fastest
   # soup = BeautifulSoup(r.text, "html5lib")  # the most lenient
  

Other

• use with context manager when dealing with file-like objects
• either remove the unused csv import or use it to write the results into CSV files
• convert comments before the functions into proper docstrings