This is a script that I've written for personal use, and to educate myself. The script is a bit comment heavy to help future me remember all the details & design choices made at the time of writing.
The use case is simply to synchronize/copy files to an AWS S3 bucket using AWS CLI. There's a separate CDK part of the project, which sets up the AWS infrastructure, but that isn't really relevant here. There are some configuration items in a properties file, which are read, and then the script checks whether everything is in place on the AWS end, and if so, it reads through a config folder structure which folders to backup, and how (include & exclude patterns in respective files).
Going with Bash instead of a basic shell script was a deliberate choice, since this wouldn't be run on any production server, and extreme portability wasn't the main point here.
Folder structure of the overall project is:
- aws-infra -- [various things here, that are out of the scope of the question] - config -- backup --- Documents ---- includes.txt --- Pictures ---- includes.txt (example at the end) ---- excludes.txt (example at the end) --- [more files/folders following the same structure] -- configuration.properties - scripts -- sync.sh
Theoretically I could've just run
aws s3 sync on the base path, but since it's a recursive command, and there are a lot (about 500k) of unnecessary files, it would take a lot of time to go through each of them separately.
Properties file (SSM parameter name censored):
# AWS profile to be used aws_profile=personal # Bucket to sync files to bucket_parameter_name=[my_ssm_parameter_name] # Config folder where backup folders & files are found. backup_folder=backup/ # Names of the files defining the include & exclude patterns for each folder. include_file_name=includes.txt exclude_file_name=excludes.txt
Example include file:
Example exclude file to pair with above:
The script works, but I'm interested on how to improve it. For instance, the error handling feels a bit clumsy.