1
\$\begingroup\$

I'm learning python 3.x and I'm still feeling like a newbie.

The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all the videos in that course.

General programming-related tips and better ways to use beautifulsoup would be great.

import requests
from bs4 import BeautifulSoup

agents = {
    'mozilla':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0'
}

download_suffix = '/download/video'

base_url = 'https://symfonycasts.com'
course_url = '/screencast/rest-ep2/serializer'

def get_page_dom():
    url = base_url + course_url
    headers = {'user-agent': agents['mozilla']}
    r = requests.get(url=url,headers=headers)
    return r.text

def get_links(html):
    chapter_list = html.find('div',class_='chapter-list')
    chapter_list_item = chapter_list.find_all('li')
    for item in chapter_list_item:
        yield item.div.div.a['href']

def generate_download_link(dl):
    for i in dl:
        yield base_url + i + download_suffix

def save_to_file(links):
    links = generate_download_link(links)

    with open('links.txt', 'w') as f:
        for i in links:
            f.write(i + '\n')

def Main():
    html = BeautifulSoup(get_page_dom(), 'html.parser')
    download_links = get_links(html)

    save_to_file(download_links)

if __name__ == '__main__':
    Main()
\$\endgroup\$
  • \$\begingroup\$ From my experience, scraping web pages is almost always sketchy. Websites are prone to change and inconcistencies, so it's hard to lay down guidelines specific to scraping. \$\endgroup\$ – TwiN Oct 1 '18 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.