I'm learning python 3.x and I'm still feeling like a newbie.

The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all the videos in that course.

General programming-related tips and better ways to use beautifulsoup would be great.

import requests
from bs4 import BeautifulSoup

agents = {
    'mozilla':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0'

download_suffix = '/download/video'

base_url = 'https://symfonycasts.com'
course_url = '/screencast/rest-ep2/serializer'

def get_page_dom():
    url = base_url + course_url
    headers = {'user-agent': agents['mozilla']}
    r = requests.get(url=url,headers=headers)
    return r.text

def get_links(html):
    chapter_list = html.find('div',class_='chapter-list')
    chapter_list_item = chapter_list.find_all('li')
    for item in chapter_list_item:
        yield item.div.div.a['href']

def generate_download_link(dl):
    for i in dl:
        yield base_url + i + download_suffix

def save_to_file(links):
    links = generate_download_link(links)

    with open('links.txt', 'w') as f:
        for i in links:
            f.write(i + '\n')

def Main():
    html = BeautifulSoup(get_page_dom(), 'html.parser')
    download_links = get_links(html)


if __name__ == '__main__':
  • \$\begingroup\$ From my experience, scraping web pages is almost always sketchy. Websites are prone to change and inconcistencies, so it's hard to lay down guidelines specific to scraping. \$\endgroup\$
    – TwiN
    Oct 1, 2018 at 18:16
  • \$\begingroup\$ As a matter of facts I have been updating the code lately, you can find it here and give some comments if you feel it: github.com/n3s7or/symfonycasts-scraper \$\endgroup\$
    – Nestor
    May 22, 2020 at 3:10
  • \$\begingroup\$ @TwiN it's been long time since I wrote this, thanks for your reply! What I was looking for wasn't guidelines for web scrapping but comments, hints, etc about the code. Thanks anyway \$\endgroup\$
    – Nestor
    May 22, 2020 at 3:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.