4
\$\begingroup\$

My deployment and configuration process entails multiple processes trying to invoke dpkg on my VM at the same time. While dpkg has a locking mechanism, it causes anyone not holding the lock who is trying to acquire it to fail, as opposed to blocking.

I want blocking behavior, so I renamed dpkg to dpkg-exec and am using the following shell script as dpkg (on Ubuntu 22.04). One tricky bit is making the lock re-entrant for spawned invocations of dpkg -- we only want locking against independent processes calling dpkg.

What I have below seems to work. I know there is at least one race condition, involving writing the PID of the acquiring process into the lockfile after acquiring it (somebody might be checking their process tree against the contents of this file between these two steps). I would appreciate any suggestions or improvements.

#!/bin/bash

# Path to the original dpkg executable
DPKG_EXEC="/usr/bin/dpkg-exec"

# Lock file path
LOCKFILE="/var/lib/dpkg/lock-custom"

# Acquire an exclusive lock using a file descriptor
touch $LOCKFILE
exec {LOCK_FD}<>$LOCKFILE

# Function to check if a PID is an ancestor of the current process
is_ancestor() {
    target_pid=$1
    current_pid=$2
    while [[ $current_pid -ne 1 ]]; do
        if [[ $current_pid -eq $target_pid ]]; then
            return 0
        fi
        current_pid=$(awk '{ print $4 }' "/proc/$current_pid/stat")
    done
    return 1
}

# Function to clean up lock on exit
cleanup() {
    # Check if the current process holds the lock
    if [[ "$(cat $LOCKFILE)" == "$$" ]]; then
    echo "*** Process $$ releasing dpkg lock" >&2
        rm -f $LOCKFILE
        flock -u $LOCK_FD
    fi
}


# Set up a trap to call cleanup on script exit
trap cleanup EXIT

# Create the lock file if it doesn't exist
touch $LOCKFILE

# Check if the lock is already held by an ancestor of this process
lock_pid=$(cat $LOCKFILE)
if ! is_ancestor "$lock_pid" $$; then
# Acquire an exclusive lock
echo "*** Process $$ trying to acquire dpkg lock" >&2
    flock -x $LOCK_FD
    echo "$$" > $LOCKFILE
echo "*** Process $$ acquired dpkg lock" >&2
fi

# Execute the original dpkg command with all arguments passed to the script
"$DPKG_EXEC" "$@"
```
\$\endgroup\$

1 Answer 1

3
\$\begingroup\$

tl;dr: I don't like the Theory of Operation. And I agree with you that it races.


existing design

There is a single filename, dpkg, which two classes of callers may invoke:

  1. ordinary (RUN lines in your Dockerfile)
  2. re-entrant (a dpkg child calling itself for a subtask)

There are two touches and a flock -x which race to create a file which is either zero length or which may contain a PID.

exec {LOCK_FD}<>$LOCKFILE

If I needed to concisely explain to someone on the phone what that does I would have trouble with it, so I am reluctant to sign up for maintaining this script. I saw no comment describing its behavior.

argv

I haven't investigated. I guess we're pretty confident that re-entrant calls use hardcoded dpkg (or /usr/bin/dpkg), ignoring argv[0]?

Patching the installer source code would let you honor argv[0], obviating the need to worry about lock reentrancy. The inconvenience of doing that would be comparable to what's involved for the current scheme to switch files around.

shellcheck

touch $LOCKFILE

nit: Please routinely quote such variable interpolations. Yes, I know, in this case there's no whitespace so it doesn't matter that it doesn't lint clean.

is_ancestor

This seems nice and I imagine it works correctly, modulo certain assumptions about how PIDs come and go on a running system. It is certainly non-portable (for example to a MacOS development system).

cleanup

We rm while in the critical section. I would prefer to see no -f force flag, since it is an invariant of this script that we shall have exclusive access when we attempt the unlink. Better to see a diagnostic on stderr if some surprise is lurking in there.

I am accustomed to letting flock -u be responsible for all lock cleanup.

nit: The idiosyncratic indents make this code harder to read than necessary.

exec

"$DPKG_EXEC" "$@"

Surely we could get away with exec "$DPKG_EXEC" "$@" there? I mean, we don't need the current shell nor any of its resources (like file descriptors) any more, right?


proposed design

Don't disturb the dpkg that Debian installed.

Two classes of callers will invoke the appropriate name, according to their needs.

  1. ordinary: dpkg.sh
  2. re-entrant: /usr/bin/dpkg

If it's inconvenient to list "dpkg.sh" on many lines of your Dockerfiles, we can fix it up with an alias, prepending a small bin directory to $PATH, a symlink, or similar tricks.

So "ordinary" callers agree to "play nice" and come in through the front door.

Now it's simple. We can use flock in exactly the way its man page suggests. This reduces support costs and increases reliability.

Caller asks for some installs to happen, flock exclusively locks, time passes, the installs are done, flock releases. Simple.

A racing caller asks for other installs, gets hung up on the lock, patiently waits to acquire it, and then carries on as above.

Re-entrant calls to dpkg are honored immediately with no locking, exactly as they're handled on every other production Debian system.


alternate design

A daemon is always doing tail -f on an install request file, or named pipe.

We replace /usr/bin/dpkg with a simple script which >> appends PID plus the command (dpkg-exec -i baz-dev) to that file. Optionally it verifies the daemon is running, starting a new background daemon if need be. The script uses tail -f to poll a done.txt status file, and exits when it sees its request satisfied.

Daemon runs each install request in sequence, with no racing overlaps. As it finishes each one, it appends requestor's PID to done.txt, perhaps annotated with things like exit status and timestamp.

Or more simply, script does “sleep 3600” (sleep forever). Daemon sends ALRM kill signal to wake it up. Script either does a simple “exit 0” immediately, or it takes a moment to “tail” the status file looking for the appropriate exit code.

\$\endgroup\$
1
  • \$\begingroup\$ Helpful review. Thank you! I should have said more about the use case. I'm running into the dpkg collisions when deploying VMs using Terraform. I use cloudinit to do some post-provision configuration (which calls dpkg), but the resource provider I'm using also does some configuration in parallel (which also calls dpkg). Consolidating the two are, unfortunately, not possible. The alternative to something like the above would be to pull the cloudinit configuration out of Terraform, which is a better pattern anyway IMO. So I may end up doing that. Thanks again for the feedback. \$\endgroup\$
    – billp
    Commented Jan 22 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.