19
\$\begingroup\$

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?

from bs4 import BeautifulSoup
import requests
import csv

source = requests.get("https://www.youtube.com/feed/trending").text
soup = BeautifulSoup(source, 'lxml')

csv_file = open('YouTube Trending Titles on 12-30-18.csv','w')
csv_writer = csv.writer(csv_file)
csv_writer.writerow(['Title', 'Description'])

for content in soup.find_all('div', class_= "yt-lockup-content"):
    try:
        title = content.h3.a.text
        print(title)

        description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
        print(description)

    except Exception as e:
        description = None

    print('\n')
    csv_writer.writerow([title, description])

csv_file.close()
\$\endgroup\$
40
\$\begingroup\$

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a GET request to https://www.googleapis.com/youtube/v3/videos?key=…&part=snippet&chart=mostpopular, you will get the same information in a documented JSON format.

Using the Python client, the code looks like:

import csv
import googleapiclient.discovery

def most_popular(yt, **kwargs):
    popular = yt.videos().list(chart='mostPopular', part='snippet', **kwargs).execute()
    for video in popular['items']:
        yield video['snippet']

yt = googleapiclient.discovery.build('youtube', 'v3', developerKey=…)
with open('YouTube Trending Titles on 12-30-18.csv', 'w') as f:
    csv_writer = csv.writer(f)
    csv_writer.writerow(['Title', 'Description'])
    csv_writer.writerows(
        [snip['title'], snip['description']]
        for snip in most_popular(yt, maxResults=20, regionCode=…)
    )

I've also restructured the code so that all of the CSV-writing code appears together, an inside a with open(…) as f: … block.

\$\endgroup\$
  • \$\begingroup\$ Great suggestion! I searched but didn't find that API. Are mostpopular and trending synonyms ? \$\endgroup\$ – SylvainD Dec 30 '18 at 21:56
  • 5
    \$\begingroup\$ @Josay When I include the regionCode in the API call ('CA' for me), then the first 20 results are identical to the list on the "Trending" page, which indicates that they are indeed different names for the same thing. \$\endgroup\$ – 200_success Dec 30 '18 at 22:06
  • \$\begingroup\$ Oh, I see; I didn't know the purpose of API, but I will look more into it. Thanks for the info. \$\endgroup\$ – austingae Dec 30 '18 at 23:55
  • 3
    \$\begingroup\$ Is the developerkey a thing required to be taken from Google ? If yes, then the scraping doesn't require it and scraping may be better from that point of view. why sign up for something that you don't need to ? \$\endgroup\$ – Whirl Mind Dec 31 '18 at 15:32
  • 3
    \$\begingroup\$ @WhirlMind Because the HTML structure of the Trending page could change at any time, and is essentially an undocumented API. The YouTube Data API is guaranteed to be stable, and any changes to it will be preceded by a suitable transition period. Obtaining a developer key is a trivial process, if you already have a Google account. \$\endgroup\$ – 200_success Dec 31 '18 at 16:09
15
\$\begingroup\$

Context manager

You open a file at the beginning of the program and close it explicitly at the end.

Python provides a nice way to allocate and release resources (such as files) easily: they are called Context managers. They give you the guarantee that the cleanup is performed at the end even in case of exception.

In your case, you could write:

with open('YouTube Trending Titles on 12-30-18.csv','w') as file:
    ....

Exception

All exceptions are caught by except Exception as e. It may look like a good idea at first but this can lead to various issues:

  • it's hard to know what types of error are actually expected here
  • most errors are better not caught (except for special situations). For instance, if you write a typo, you'll end up with an ignored NameError or AttributeError and debugging will be more painful than it should be.

Also, from the content of the except block, it looks like you are only expecting the logic about description to fail. If so, it would be clearer to put in the try (...) except the smallest amount of code.

For instance:

title = content.h3.a.text
try:
    description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
except Exception as e:
    description = None
print(title)
print(description)
print('\n')

Proper solution

Google usually offers API to retrieve things such like trending videos. I haven't found it but I'll let you try to find something that works properly. Google is your friend...

\$\endgroup\$
10
\$\begingroup\$

I'd definitely look into using an API directly as @200_success suggested to avoid any web-scraping or HTML parsing, but here are some additional suggestions to improve your current code focused mostly around HTML parsing:

  • you could get some speed and memory improvements if you would use a SoupStrainer to allow BeautifulSoup parse out only the desired elements from the HTML:

    The SoupStrainer class allows you to choose which parts of an incoming document are parsed.

    from bs4 import BeautifulSoup, SoupStrainer
    
    trending_containers = SoupStrainer(class_="yt-lockup-content")
    soup = BeautifulSoup(source, 'lxml', parse_only=trending_containers)
    
  • instead of .find_all() and .find() you could have used more concise CSS selectors. You would have:

    soup.select('.yt-lockup-content')
    

    instead of:

    soup.find_all('div', class_= "yt-lockup-content")
    

    and:

    content.select_one('.yt-lockup-description.yt-ui-ellipsis.yt-ui-ellipsis-2')
    

    instead of:

    content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2")
    
  • note how I've omitted div tag names above - I think they are irrelevant as the class values actually define the type of an element in this case

  • organize imports as per PEP8
\$\endgroup\$
5
\$\begingroup\$

I would separate the "input" (finding titles and descriptions) from the output (writing to screen or file). One good way to do that is to use a generator:

from bs4 import BeautifulSoup
import requests
import csv

def soup():
    source = requests.get("https://www.youtube.com/feed/trending").text
    soup = BeautifulSoup(source, 'lxml')

def find_videos(soup):
    for content in soup.find_all('div', class_= "yt-lockup-content"):
        try:
            title = content.h3.a.text
            description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text
        except Exception as e:
            description = None
        yield (title, description)

with open('YouTube Trending Titles on 12-30-18.csv', 'w') as csv_file:

    csv_writer = csv.writer(csv_file)
    csv_writer.writerow(['Title', 'Description'])

    for (title, description) in find_videos(soup()):
        csv_writer.writerow([title, description])

Disclaimar: I haven't tested this code.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.