2
\$\begingroup\$

I have a lot of files. I care about those files. I also care about good programming practices and learning proper python. Given the described context, it seemed appropriate to implement my own automated backup utility (as an exercise) What I bring here is my attempt at the task and any advice is welcome.

Description:

This script is scheduled to run every day at 04:00 AM. It'll zip my whole docs folder (lots of stuff in here) and upload it to my google drive. It does some checking for available space, it also ensures the total space used by backups doesn't go over a given limit (wouldn't be nice to have backups taking up all my drive space, so it deletes the oldest files to keep it within the desired range). The script follows the pattern "Data_DD-MM-YY" as naming convention for the zip files, where DD-MM-YY is the date of the respective backup. For example, the backup files of the last three days would be "Data_28-06-2019", "Data_27-06-2019" and "Data_26-06-2019". I also use the shelve module to keep a simple execution log which can be listed by passing the argument list to the script.

#! python3
# -*- coding: utf-8 -*-

"""bur.py - Backup utilitity

    Compress all relevant data in a given drive into a single 
    .zip file and upload it to a backup folder on google drive, 
    keeping log of every operation on local machine.

    Usage: python bur.py - Start backup routine
           python bur.py list - Show backup logs

    Obs. Script is supposed to run once a day at a given scheduled time

    by WesternSage 20/06/2016
"""

import zipfile, shelve, os, sys
from datetime import date
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive

# CONTROL PARAMS
# =========================================================
CREDENTIALS      = 'mycreds.txt'
BACKUP_TARGET    = 'D:\\Data'   # Root folder of backup
MAX_BACKUP_SPACE = 5368709120   # 5GB  - limit of drive space used by backups
MAX_DRIVE_SPACE  = 16106127360  # 15GB - total google drive space
# =========================================================

def get_drive_used_space():
    """
        Description:

        Calculates two things:
            a - Total space used in google drive folder by all files
            b - Space used in google drive folder by backup files only

        Results are returned as a dictionary under the keys:
            'total_used_space' for item a
            'backup_used_space for item b

        Return: Dictionary - Result of queries 'a' and 'b' as strings
    """

    results = dict()
    results['total_used_space'] = 0
    results['backup_used_space'] = 0

    file_list = drive.ListFile({'q': "'root' in parents"}).GetList()

    for cloud_file in file_list:
        file_size = int(cloud_file.get('fileSize', '0'))
        results['total_used_space'] += file_size
        if cloud_file.get('title', 'unknown').startswith('Data_'):
            results['backup_used_space'] += file_size

    return results

def auth_google_drive(credentials_file):
    """ 
        Input:
            - credentials_file: String

        Authenticate google drive session using given 
        credentials file. There are three possibilities here:
            1 - credentials file doesn't exist (user will authenticate on browser)
            2 - token expired (token will be refreshed)
            3 - credials file is valid (normal authentication)

        Return: Authenticated GooogleDrive object
    """

    gauth = GoogleAuth()
    gauth.LoadCredentialsFile(credentials_file)
    if gauth.credentials is None:
        gauth.LocalWebserverAuth()
    elif gauth.access_token_expired:
        gauth.Refresh()
    else:
        gauth.Authorize()
    gauth.SaveCredentialsFile(credentials_file)

    return GoogleDrive(gauth)

def zip_dir(target_dir):
    """ 
        Input:
        - target_dir: String

        Description:

        Create a zip file with name following the pattern 
        'Data_DD/MM/YYYY' (where DD/MM/YYYY is the current 
        date at the time of function execution) and recursively 
        walk given target directory, adding all files (excluding 
        hidden dirs) to created zip 

        Return: String - Name of the created zip file"""

    zip_filename = os.path.basename(target_dir) + '_' + str(date.today().strftime("%d-%m-%Y")) + '.zip'
    backup_zip = zipfile.ZipFile(zip_filename, 'w')
    for foldername, subfolders, filenames in os.walk(target_dir):
        # Skip hidden stuff
        filenames = [f for f in filenames if not f[0] == '.']
        subfolders[:] = [d for d in subfolders if not d[0] == '.']
        backup_zip.write(foldername)
        for filename in filenames:
            new_base = os.path.basename(target_dir) + '_'
            # don't backup the backup ZIP files
            if filename.startswith(new_base) and filename.endswith('.zip'):
                continue
            backup_zip.write(os.path.join(foldername, filename))
    backup_zip.close()
    return zip_filename

def upload_file(filename):
    """ 
        Input:
            - filename: String

        Description:
            Uploads given file to backup folder on google drive      
     """
    with open(filename, "r") as file:
        file_drive = drive.CreateFile({'title':os.path.basename(file.name) })  
        file_drive.SetContentFile(file.name) 
        file_drive.Upload()

# ====================================================================================

backup_shelf = shelve.open('bkp')

# checks if utility was called to list backup logs
if len(sys.argv) == 2 and sys.argv[1] == 'list':
    for value in backup_shelf.values():
        print(value)
    sys.exit("End of log.") # done listing, just quit

# No 'list' param, so start back up routine

drive       = auth_google_drive(CREDENTIALS)
zip_name    = zip_dir(BACKUP_TARGET)
zip_size    = os.path.getsize(zip_name)

while True:
    drive_space = get_drive_used_space()

    if drive_space['backup_used_space'] + zip_size < MAX_BACKUP_SPACE:
        break

    file_list   = drive.ListFile({'q': "title contains 'Data'"}).GetList()
    oldest_file = file_list[0]

    for cloud_file in file_list:
        # we can use the titles to compare creation date
        # of files since the script follows 'Data_DD-MM-YYYY' 
        # pattern for naming zip files, where DD-MM-YY is the 
        # date when the backup was made
        if (oldest_file['title'] > cloud_file['title']):
            oldest_file = cloud_file

    oldest_file.Delete()

if drive_space['total_used_space'] + zip_size < MAX_DRIVE_SPACE:
    upload_file(zip_name)
    uploaded = True
else:
    uploaded = False

today = str(date.today().strftime("%d-%m-%Y"))
backup_shelf[today] = 'Backup log (' + today + ') - Zip pack size (bytes): ' + str(zip_size) + ' Total drive space in use (bytes): ' \
    + str(drive_space['backup_used_space']) + ' Drive space used by backup files (bytes): ' + str(drive_space['backup_used_space'])

if not uploaded:
    backup_shelf[today] += ' File has not been uploaded!!!'

backup_shelf.close()

Some questions that occurred to me when codding:

  • What would be the advised way of commenting/documenting my code and describing my functions?
  • Some of the functions are very simple and only exist in an attempt to make the code more readable, I wonder if I'm doing it right.
  • Error handling: I don't have any. How could I go about implementing some basic error handling for this script?
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.