Return to Revisions

Code Style

There are multiple PEP8 code style violations, some of them are:

• naming - use lower_case_with_underscores naming style
• spaces around operators
• blank lines between imports and functions

You should also improve your variable naming - for example, d, b, i are not meaningful - think of more descriptive names - remember: Code is much more often read than written.

HTML-parsing and Web-scraping

• instantiate requests.Session() and reuse to make requests - this would give you a performance boost "for free":

if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

• you can replace .find_all() calls with more explicit and robust select() calls and use CSS selectors. For instance, soup.find_all("h1", "branded-page-header-title") would become soup.select("h1.branded-page-header-title")

Other

• use with context manager when dealing with file-like objects
• either remove the unused csv import or use it to write the results into CSV files
• convert comments before the functions into proper docstrings