Uploading songs from website to database and then to Spotify

Question

My Python program parses songs from a website, corrects song titles and artists with the Last.fm API, searches for the spotify uri using the Spotify API, stores all the information in a SQLite database and then uploads it into a Spotify playlist with the Spotify API.

I would like to make the program object oriented and need some advice on how to do that. Some general python advice would also be useful.

I have a separate config.py file with all the needed API variables.

scraper.py

# -*- coding: utf-8 -*-
# import config file
import config

# import libraries
from bs4 import BeautifulSoup
import datetime
import urllib.request as urllib
import sys
import time
import re
import sqlite3

# webdriver libraries
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# spotipy library
import spotipy
import spotipy.util as util

# import pylast
import pylast 

# song class holds information about each song
class Song:
    artist = None
    song = None
    spotify_uri = None

    def __init__(self, artist, song, spotify_uri):
        self.artist = artist
        self.song = song
        self.spotify_uri = spotify_uri

    def printSong(self):
        print(self.artist, '-', self.song, ', Uri:', self.spotify_uri)

##------------------------------------------------------------------------------
## Get Date of latest sunday
## 
## @return formatted date of last sunday as yyyymmdd
#
def getSundayDate():
    today = datetime.date.today()
    sun_offset = (today.weekday() - 6) % 7
    sunday_of_week = today - datetime.timedelta(days=sun_offset)
    sunday_date = sunday_of_week.strftime('%Y%m%d')
    return sunday_date

##------------------------------------------------------------------------------
## URL Pattern
##
## https://fm4.orf.at/player/20190120/SSU
## URL pattern:
## /yyyymmdd/SSU
## /20190120/SSU
## SSU is just Sunny Side Up the show from 10am till 1pm
## URL pattern changes ever day, we need to change it every week, 
## to only get sundays
## 
## @return concatenated URL of website
def getURLPattern():
    return 'https://fm4.orf.at/player/' + getSundayDate() + '/SSU'

##------------------------------------------------------------------------------
## Get html source from page specified by page_url
## 
## @return html source as beautiful soup object
#
def getHtmlFromPage():
    page_URL = getURLPattern()

    options = Options()
    options.headless = True
    profile = webdriver.FirefoxProfile()
    profile.set_preference("media.volume_scale", "0.0")
    driver = webdriver.Firefox(options=options, firefox_profile=profile)

    driver.get(page_URL)

    wait = WebDriverWait(driver, 3)
    wait.until(EC.presence_of_element_located((By.CLASS_NAME, 
        'broadcast-items-list')))
    time.sleep(1)

    soup = BeautifulSoup(driver.page_source, "html.parser")
    driver.quit()
    return soup

##------------------------------------------------------------------------------
## remove bad characters from list 
## 
## @param list, list with elements to check
#
def sanitize(strList):
    regex_remove = r'([^A-z\s\däöüÄÖÜß-][\\\^]?)'
    regex_ft = r'(ft\.?([^\n]\s?\w*)+)'

    # check for bad characters
    for i in range(len(strList)):
        strList[i] = re.sub(regex_remove, "", str(strList[i]))
        strList[i] = re.sub(regex_ft, "", strList[i])

##------------------------------------------------------------------------------
## print music 
## 
## @param lists to print
#
def printMusic(interpreter_list, title_list):
    for element in range(len(interpreter_list)):
        print(interpreter_list[element] + " : " + title_list[element])

##------------------------------------------------------------------------------
## parse html
## 
## @param lists to write results to
#
def parseHtml(interpreter_list, title_list):
    soup = getHtmlFromPage()

    # find all interpreter in playlist
    interpreter = soup.find_all("div", {"class": "interpreter"})

    # find all titles in playlist
    title = soup.find_all("div", {"class": "title"})

    # Check for errors
    if (len(interpreter) != len(title)):
        raise Exception("The amount of interpreters don't correspond" +
                "to the amount of titles.")
    if (len(interpreter) == 0):
        raise Exception("No FM4 music playlist found in given url") 

    for element in range(len(interpreter)):
        interpreter_list.append(interpreter[element].text)
        title_list.append(title[element].text)

##------------------------------------------------------------------------------
## create Token with given credentials
## 
## @return authentication token
#
def getToken():
    # authetication token
    token = util.prompt_for_user_token(config.USERNAME, config.SCOPE, config.CLIENT_ID, 
            config.CLIENT_SECRET, config.REDIRECT_URI)

    if token:
        return token
    else:
        raise Exception("Could not get authentication token from spotify!")

##------------------------------------------------------------------------------
## search track and get spotify uri
## 
## @param token, authentication token
## @param interpreter && title, strings containing track info
## @return uri string
#
def getUri(spotify_Obj, interpreter, title):
    result = spotify_Obj.search(q=interpreter + ' ' + title)

    if (result != None):
        if (len(result['tracks']['items']) != 0):
            track_id = result['tracks']['items'][0]['uri']
            uri = str(track_id)
            return uri

##------------------------------------------------------------------------------
## correct artist name and track title with lastFm api
## 
## @param1 artist_name, name of artist to correct
## @param2 title_name, title name to correct
## @return track_corrected, corrected Track object
#
def getTrackInfo(artist_name, track_name):
    # network authentication
    last_Fm = getLastFmNetworkAuth()

    # declare artist_name as artist object
    artist = last_Fm.get_artist(artist_name)
    # correct artist name
    artist_corrected_name = artist.get_correction()

    track = last_Fm.get_track(artist_corrected_name, track_name)
    track_corrected_name = track.get_correction()

    trackInfo = pylast.Track(artist_corrected_name, track_corrected_name, 
            last_Fm)
    return trackInfo

##------------------------------------------------------------------------------
## get last fm network authentication
## 
## @return network authentication token
#
def getLastFmNetworkAuth():
    network = pylast.LastFMNetwork(config.LASTFM_API_KEY, config.LASTFM_API_SECRET)
    return network

##------------------------------------------------------------------------------
## parse music items from website, put them into a list, sanitize lists, 
## correct artist names and song titles with last.fm API and save list in a 
## sqlite database for further usage
## 
## @return network authentication token
#
def parseTracksIntoSongClassList(song_list):
    # lists containing the Interpreter and title
    interpreter_list = []
    title_list = []

    # fill lists with results
    parseHtml(interpreter_list, title_list)

    print(datetime.datetime.now(), "Done parsing html")

    # remove bad characters from lists
    sanitize(interpreter_list)
    sanitize(title_list)

    # get Token and create spotify object
    sp = spotipy.Spotify(getToken())

    # correct artist and title names
    for element in range(len(interpreter_list)):
        track_info = getTrackInfo(interpreter_list[element], 
                title_list[element])

        title = str(track_info.get_name())
        artist = str(track_info.get_artist())

        if (title != artist):
            if (title is not None):
                title_list[element] = title
            if (artist is not None):
                interpreter_list[element] = artist
        else:
            title_list[element] = title_list[element]
            interpreter_list[element] = interpreter_list[element]

        # get spotify uri for song
        spotify_uri = getUri(sp, interpreter_list[element], title_list[element]) 

        if (spotify_uri != None and len(spotify_uri) != 0):
            track_uri = str(spotify_uri)
            song_list.append(Song(interpreter_list[element], 
                title_list[element], track_uri))

    print(datetime.datetime.now(), "Done parsing songs")

##------------------------------------------------------------------------------
## insert new songs to database, checks for duplicates and ignores them
## 
## @param song_list, list containing songs which need to be inserted 
## into database
#
def updateDatabase(song_list):
    conn = sqlite3.connect('SongDatabase.db')
    c = conn.cursor()

    # date to insert into table
    today = datetime.date.today()                                                                             
    today.strftime('%Y-%m-%d')

    c.execute('''CREATE TABLE IF NOT EXISTS songs 
            (SongID INTEGER PRIMARY KEY, artist_name TEXT, song_name TEXT, 
            spotify_uri TEXT, UploadDate TIMESTAMP, Uploaded INTEGER, 
            UNIQUE(artist_name, song_name, spotify_uri) ON CONFLICT IGNORE)''')

    for item in range(len(song_list)):
        c.execute('''INSERT INTO songs 
        (artist_name, song_name, spotify_uri, UploadDate, Uploaded) 
        VALUES (?,?,?,?,?)''', (song_list[item].artist, song_list[item].song, 
            song_list[item].spotify_uri, today, 0))

    conn.commit()
    c.close()
    print(datetime.datetime.now(), "Done updating Database")

##------------------------------------------------------------------------------
## copy Uris from song_list into new list
##
## @param song_list, list containing songs which get copied into new list
## @return track_list, list containing all song uris
#
def getUrisList(song_list):
    uri_list = []

    for song in range(len(song_list)):
        uri_list.append(song_list[song].spotify_uri)
    print(uri_list)
    return uri_list

##------------------------------------------------------------------------------
## Main part of the program
## get html and parse important parts into file
#
if __name__ == '__main__':
    # list to fill with corrected songs 
    song_list = []

    # parse songs into song_list
    parseTracksIntoSongClassList(song_list)

    # insert song_list into database
    updateDatabase(song_list)

dataManager.py

# -*- coding: utf-8 -*-
# import config file
import config

import sqlite3
import pandas as pd

# spotipy library
import spotipy
import spotipy.util as util

##------------------------------------------------------------------------------
## create Token with given credentials
## 
## @return authentication token
#
def getToken():
    # authetication token
    token = util.prompt_for_user_token(config.USERNAME, config.SCOPE, config.CLIENT_ID, 
            config.CLIENT_SECRET, config.REDIRECT_URI)
    return token

##------------------------------------------------------------------------------
## insert new songs to database, checks for duplicates and ignores them
## 
## @param song_list, list containing songs to be inserted into database
#
def uploadSongsToSpotify():
    # declare db name
    database_name = 'SongDatabase.db'

    # spotify auth token
    sp = spotipy.Spotify(getToken())
    if sp:
        # spotify username
        username = config.USERNAME
        # spotify ide of playlist
        playlist_id = config.PLAYLIST_ID

        conn = sqlite3.connect(database_name)
        c = conn.cursor()

        c.execute("""SELECT spotify_uri FROM songs WHERE (Uploaded = 0)""")

        # save query results in tuple
        data = c.fetchall()

        # save uris in list, for spotipy
        uri_list = []
        for item in range(len(data)):
            uri_list.append(str(data[item][0]))

        print(uri_list)

        # upload uri_list to spotify
        # check for empty list
        if (len(uri_list) != 0):
            sp.user_playlist_add_tracks(username, playlist_id, uri_list)

            # set Uploaded values in database to 1
            c.execute("""UPDATE songs SET Uploaded = ? WHERE Uploaded = ?""", (1, 0))
            conn.commit()

        else:
            raise Exception("There aren't any new songs in database, songs were already uploaded")

        c.close()

    else:
        raise Exception("Could not get token from spotify API")


if __name__ == '__main__':
    uploadSongsToSpotify()

Answer 1

Some simple suggestions after a first look at scraper.py:

1. class Song defines a method called printSong. Don't do this. Instead, use the dundermethod __str__ (and maybe __repr__) to define a mechanism to "stringify" the song, and then let the normal print function handle it:

   print(str(song)) # or ... print(repr(song))

2. Your getSundayDate computes the date of the appropriate Sunday, then returns it as a string. Instead, return the date object. Let the caller handle formatting the string, since the caller is getUrlPattern which does nothing but format strings...

3. Throughout your code you have these giant banner comments introducing your functions. Get rid of them, and put the descriptive text inside a docblock comment. This is why docblocks exist in Python:

   No!

   ##------------------------------------------------------------------------------
   ## remove bad characters from list 
   ## 
   ## @param list, list with elements to check
   #
   def sanitize(strList):
   

   Yes.

   def sanitize(strList):
       """Remove bad characters from list.
   
       @param strList,    list with elements to check.
       """
   

4. Don't raise Exception objects. Class Exception is the base class of the standard error types. If you have to install a block to catch what you're raising, you are going to have to do except Exception: or maybe just except:, and that's no good. Either create your own exception class, like class SongException(Exception): ;pass or use the standard types (IndexError, ValueError, and TypeError for the most part).

5. In parseHtml you do this:

   for element in range(len(interpreter)):
       interpreter_list.append(interpreter[element].text)
       title_list.append(title[element].text)
   

   Written like a true Java programmer! But this isn't Java. So watch this video first: Loop Like a Native by Ned Batchelder. There are a couple of ways to rewrite this loop. You could zip the two source lists together, unpack them into a tuple, and operate on them:

   for interp, elt in zip(interpreter, element):
       interpreter_list.append(interp.text)
       element_list.append(elt.text)
   

   Or you could use a comprehension to iterate over each list separately to generate the text values, and use the list.extend method to implicitly .append each element of a sequence:

   interpreter_list.extend((elt.text for elt in interpreter))
   element_list.extend((elt.text for elt in element))
   

   Have some of this Python-flavored Cool-aid! It's quite delicious... ;-)

6. You define getToken in both source files. I'm not sure what that's about...

Looking at your dataManager.py file, it's quite short. I'd suggest that you just roll both files into a single source file.

Your post title asks how you can make your code more object-oriented. I don't think you need to do that, and I don't think you should try. You are writing a program that is very procedural: do this, then do that, next do the other, and finally store things here. That's not a good match for OO code, especially since the elements in question are all different. I encourage you to focus on using simple functions to ensure that you have good separation of concerns and encapsulation. I would also suggest visiting the documentation for Python's "magic methods" (aka dundermethods), and sitting through the Batchelder video I linked. There's a huge amount of Python mastery in that one 30-minute presentation.