1
\$\begingroup\$

I'm learning python 3.x and I'm still feeling like a newbie.

The program below scrape a given course in https://symfonycasts.com/ and creates a file containing all the dynamics download links to all the videos in that course.

General programming-related tips and better ways to use beautifulsoup would be great.

import requests
from bs4 import BeautifulSoup

agents = {
    'mozilla':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0'
}

download_suffix = '/download/video'

base_url = 'https://symfonycasts.com'
course_url = '/screencast/rest-ep2/serializer'

def get_page_dom():
    url = base_url + course_url
    headers = {'user-agent': agents['mozilla']}
    r = requests.get(url=url,headers=headers)
    return r.text

def get_links(html):
    chapter_list = html.find('div',class_='chapter-list')
    chapter_list_item = chapter_list.find_all('li')
    for item in chapter_list_item:
        yield item.div.div.a['href']

def generate_download_link(dl):
    for i in dl:
        yield base_url + i + download_suffix

def save_to_file(links):
    links = generate_download_link(links)

    with open('links.txt', 'w') as f:
        for i in links:
            f.write(i + '\n')

def Main():
    html = BeautifulSoup(get_page_dom(), 'html.parser')
    download_links = get_links(html)

    save_to_file(download_links)

if __name__ == '__main__':
    Main()
Improve this question
\$\endgroup\$
3
  • \$\begingroup\$ From my experience, scraping web pages is almost always sketchy. Websites are prone to change and inconcistencies, so it's hard to lay down guidelines specific to scraping. \$\endgroup\$
    – TwiN
    Oct 1, 2018 at 18:16
  • \$\begingroup\$ As a matter of facts I have been updating the code lately, you can find it here and give some comments if you feel it: github.com/n3s7or/symfonycasts-scraper \$\endgroup\$
    – Nestor
    May 22, 2020 at 3:10
  • \$\begingroup\$ @TwiN it's been long time since I wrote this, thanks for your reply! What I was looking for wasn't guidelines for web scrapping but comments, hints, etc about the code. Thanks anyway \$\endgroup\$
    – Nestor
    May 22, 2020 at 3:12

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.