Skip to main content
added 4 characters in body
Source Link
alecxe
  • 17.3k
  • 8
  • 51
  • 93

I'd reallydefinitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

  • you could get some speed and memory improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

    from bs4 import BeautifulSoup, SoupStrainer

    trending_containers = SoupStrainer(class_="yt-lockup-content")
    soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
  • instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

     soup.select('.yt-lockup-content')
    

instead of:

    soup.find_all('div', class_= "yt-lockup-content")

and:

    content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')

instead of:

    content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
  • note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case
  • organize imports as per PEP8

I'd really look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

  • you could get some speed improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

    from bs4 import BeautifulSoup, SoupStrainer

    trending_containers = SoupStrainer(class_="yt-lockup-content")
    soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
  • instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

     soup.select('.yt-lockup-content')
    

instead of:

    soup.find_all('div', class_= "yt-lockup-content")

and:

    content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')

instead of:

    content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
  • note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case
  • organize imports as per PEP8

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

  • you could get some speed and memory improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

    from bs4 import BeautifulSoup, SoupStrainer

    trending_containers = SoupStrainer(class_="yt-lockup-content")
    soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
  • instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

     soup.select('.yt-lockup-content')
    

instead of:

    soup.find_all('div', class_= "yt-lockup-content")

and:

    content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')

instead of:

    content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
  • note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case
  • organize imports as per PEP8
Source Link
alecxe
  • 17.3k
  • 8
  • 51
  • 93

I'd really look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

  • you could get some speed improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

    from bs4 import BeautifulSoup, SoupStrainer

    trending_containers = SoupStrainer(class_="yt-lockup-content")
    soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
  • instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

     soup.select('.yt-lockup-content')
    

instead of:

    soup.find_all('div', class_= "yt-lockup-content")

and:

    content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')

instead of:

    content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
  • note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case
  • organize imports as per PEP8