I'm doing a little more grad school, and for a group project I wanted us to be able to quickly share and update our project, so I coded up a Python script to handle that. As usual, I'm proud of the work, but I'm here for you to tear it to shreds again.
First, the imports, some globals, and the main:
import time
from subprocess import run
from pathlib import Path
from datetime import datetime
from shlex import split
WD = Path.home() / 'project_name'
SERVEDIR = Path('/var/www/main/project_name')
def main():
while True:
just_built = False
try:
if git_pull():
print('pulled at', datetime.now())
build()
print('built at', datetime.now())
move()
list_index()
just_built = True
except Exception as error:
print(repr(error))
print('polled at ', datetime.now())
if not just_built:
time.sleep(5 * 60)
The main function outlines the work of the script. The just_built
variable ensures that if we just built the project (probably more than 5 minutes to do) we don't sleep for another 5 minutes, we first do another git_pull()
. The try
wasn't really used, but it would keep the script running if there was an problem encountered. The rest is straightforward.
The WD
is the working directory where the git repo resides. To make this work I did have to chown the serving subdirectory to my user from root.
As a matter of style, I prefer to put my main function at the top - it's where the outline or table of contents should go, right? It calls the following functions in the rest of the script:
def git_pull():
proc = run(split("git pull --verbose"), cwd=WD, capture_output=True)
print(proc.stdout)
return b"Already up to date." not in proc.stdout
def build():
run(split('nix-shell --pure --command "make all"'), cwd=WD)
def move():
timestamp = datetime.now().isoformat(timespec="minutes", sep=" ")
new_name = f'project{timestamp}.'
for ext in ('pdf', 'html'):
new = SERVEDIR / (new_name + ext)
(WD / f'project.{ext}').rename(new)
symlink = SERVEDIR / f'project.{ext}'
symlink.unlink(missing_ok=True)
symlink.symlink_to(new)
def list_index():
files = sorted(SERVEDIR.iterdir())
files = [f'<a href="{f.name}">{f.name}</a>'
for f in files if 'project' in f.name]
index = SERVEDIR / 'index.html'
index.write_text('\n<br>\n'.join(files))
if __name__ == '__main__':
main()
To sum up, I poll every 5 minutes with git pull
and if we don't pull down anything, we don't build. I do this under the presumption that git has the best API to check to see if there's anything to do. Yes I could have used github webhooks instead of polling, but I'm not set up to accept POSTs yet (and not sure I want to expose that functionality yet...) and besides, github didn't complain.
To build, it calls 'nix-shell --pure --command "make all"'
. To sum up, Nix ensures the requirements (via shell.nix
, at the bottom) and then make runs the all
in my makefile:
.PHONY : all
all:
Rscript -e 'rmarkdown::render("project.Rmd", "all")'
In spite of calling rmarkdown::render
one time, it seems to re-run all the R code twice.
The upsides are all I had to do to kick off a build was
git commit -ac "descriptive comment" && git push
(and then pull, reconcile, merge any changes, and re-push, if it's necessary.)
Other features:
- retain every build (quick output comparisons, see image below), listed in
index.html
- canonical link points to the latest build (quick collaboration)
- lots of builds every day (fast iteration)
- updates merge early and often (continuously integrate)
- simple, easy to maintain Python
One downside of this approach versus alternatives is that I had to have a user shell open and running it. I could have detached via tmux, but... I didn't. I'm locked down at home anyways, so no big deal.
Other downsides:
- no unit tests or types checked with mypy
- no style checking
- not represented by any kind of reusable object model, just functions written in a very side-effect-y way - like a script.
Alternatives
I could have written this as a shell script, but I'm not an expert on shell substitution rules (yet). That might be a good response - how to do this with a shell script. I doubt we'd get noticeable improvements in performance or stability with a shell script, though.
I could have used a cron job to run this every 5 minutes (without the while loop) but that seemed like unnecessary configuration fiddling, with the problem of which user to run under as well (a user with minimal perms, naturally).
I could have also used Jenkins (which I will eventually get set up with regardless) but I didn't have the time to set it up, and until I do some version of this script will work fine.
shell.nix
Here's my shell.nix file, which ensures my requirements are in-place in the environment (i.e. in my PATH file) before building the project. I'm using NixOS on this server, so Nix is a natural choice for this purpose:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
mkShell {
buildInputs = [
texlive.combined.scheme-full
entr
ncurses # for tput
tree
R
pandoc
rPackages.choroplethr
rPackages.rmarkdown
rPackages.nycflights13
rPackages.viridis
rPackages.tidyverse
rPackages.ALSM
rPackages.nortest
rPackages.alr4
rPackages.lmtest
rPackages.EnvStats
rPackages.GGally
];
shellHook = ''
source ~/.bashrc || source /etc/bashrc
'';
}
This all built an Rmarkdown file that's also the work of others, so we can't show that here.
The question is, how do I improve my code?