I'd reallydefinitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:
- you could get some speed and memory improvements if you would use a
SoupStrainer
to allowBeautifulSoup
parse out only the desired elements from the HTML:
The
SoupStrainer
class allows you to choose which parts of an incoming document are parsed.
from bs4 import BeautifulSoup, SoupStrainer
trending_containers = SoupStrainer(class_="yt-lockup-content")
soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
instead of
.find_all()
and.find()
you could have used more concise CSS selectors. You would have:soup.select('.yt-lockup-content')
instead of:
soup.find_all('div', class_= "yt-lockup-content")
and:
content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')
instead of:
content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
- note how I've omitted
div
tag names above - I think they are irrelevant as the class values actually define the type of an element in this case - organize imports as per PEP8