Web scraping the titles and descriptions of trending YouTube videos

Question

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?

from bs4 import BeautifulSoup
import requests
import csv

source = requests.get("https://www.youtube.com/feed/trending").text
soup = BeautifulSoup(source, 'lxml')

csv_file = open('YouTube Trending Titles on 12-30-18.csv','w')
csv_writer = csv.writer(csv_file)
csv_writer.writerow(['Title', 'Description'])

for content in soup.find_all('div', class_= "yt-lockup-content"):
    try:
        title = content.h3.a.text
        print(title)

        description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
        print(description)

    except Exception as e:
        description = None

    print('\n')
    csv_writer.writerow([title, description])

csv_file.close()

Answer 1

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a GET request to https://www.googleapis.com/youtube/v3/videos?key=…&part=snippet&chart=mostpopular, you will get the same information in a documented JSON format.

Using the Python client, the code looks like:

import csv
import googleapiclient.discovery

def most_popular(yt, **kwargs):
    popular = yt.videos().list(chart='mostPopular', part='snippet', **kwargs).execute()
    for video in popular['items']:
        yield video['snippet']

yt = googleapiclient.discovery.build('youtube', 'v3', developerKey=…)
with open('YouTube Trending Titles on 12-30-18.csv', 'w') as f:
    csv_writer = csv.writer(f)
    csv_writer.writerow(['Title', 'Description'])
    csv_writer.writerows(
        [snip['title'], snip['description']]
        for snip in most_popular(yt, maxResults=20, regionCode=…)
    )

I've also restructured the code so that all of the CSV-writing code appears together, an inside a with open(…) as f: … block.

Answer 2

Context manager

You open a file at the beginning of the program and close it explicitly at the end.

Python provides a nice way to allocate and release resources (such as files) easily: they are called Context managers. They give you the guarantee that the cleanup is performed at the end even in case of exception.

In your case, you could write:

with open('YouTube Trending Titles on 12-30-18.csv','w') as file:
    ....

Exception

All exceptions are caught by except Exception as e. It may look like a good idea at first but this can lead to various issues:

• it's hard to know what types of error are actually expected here
• most errors are better not caught (except for special situations). For instance, if you write a typo, you'll end up with an ignored NameError or AttributeError and debugging will be more painful than it should be.

Also, from the content of the except block, it looks like you are only expecting the logic about description to fail. If so, it would be clearer to put in the try (...) except the smallest amount of code.

For instance:

title = content.h3.a.text
try:
    description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
except Exception as e:
    description = None
print(title)
print(description)
print('\n')

Proper solution

Google usually offers API to retrieve things such like trending videos. I haven't found it but I'll let you try to find something that works properly. Google is your friend...

Answer 3

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

• you could get some speed and memory improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

  The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

  from bs4 import BeautifulSoup, SoupStrainer
  
  trending_containers = SoupStrainer(class_="yt-lockup-content")
  soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
  

• instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

  soup.select('.yt-lockup-content')
  

  instead of:

  soup.find_all('div', class_= "yt-lockup-content")
  

  and:

  content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')
  

  instead of:

  content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
  

• note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case

• organize imports as per PEP8

Answer 4

I would separate the "input" (finding titles and descriptions) from the output (writing to screen or file). One good way to do that is to use a generator:

from bs4 import BeautifulSoup
import requests
import csv

def soup():
    source = requests.get("https://www.youtube.com/feed/trending").text
    soup = BeautifulSoup(source, 'lxml')

def find_videos(soup):
    for content in soup.find_all('div', class_= "yt-lockup-content"):
        try:
            title = content.h3.a.text
            description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
        except Exception as e:
            description = None
        yield (title, description)

with open('YouTube Trending Titles on 12-30-18.csv', 'w') as csv_file:

    csv_writer = csv.writer(csv_file)
    csv_writer.writerow(['Title', 'Description'])

    for (title, description) in find_videos(soup()):
        csv_writer.writerow([title, description])

Disclaimar: I haven't tested this code.