Multi stage Dockerfile for eccode compilation

Question

I wrote a Dockerfile to install ecCodes for use in a python environment using cfgrib and xarray.

The tarball can be found on the ecCode releases page

Dockerfile

# syntax=docker/dockerfile:1
# ECCODE COMPILE
FROM python:3.10.4 as compiler
# /path/to/where/you/install/eccodes
ENV ECCODES_DIR=/usr/src/eccodes
# the zip file for eccodes
COPY ./eccodes-2.24.2-Source.tar.gz ./eccodes-2.24.2-Source.tar.gz

RUN apt-get update -y
# compiler tools
RUN apt-get install -y --no-install-recommends \
    cmake \
    gfortran \
    build-essential

# unzip 
RUN tar -xzf eccodes-2.24.2-Source.tar.gz 
# prepare the build folders
RUN mkdir $ECCODES_DIR build && cd build
# 
RUN cmake ../eccodes-2.24.2-Source -DCMAKE_INSTALL_PREFIX=$ECCODES_DIR -DENABLE_JPG=ON

RUN make && ctest && make install


# PYTHON BUILD
FROM python:3.10.4 as builder

ENV VIRTUAL_ENV=/opt/venv

COPY ./requirements.txt ./requirements.txt 
# create the virtual env
RUN python3 -m venv $VIRTUAL_ENV 
# upgrade pip
RUN python3 -m pip install --upgrade pip
# install the requirements
RUN pip install -r requirements.txt


# FINAL
FROM python:3.10.4

COPY --from=compiler $ECCODES_DIR $ECCODES_DIR

ENV ECCODES_DIR=$ECCODES_DIR

COPY --from=builder $VIRTUAL_ENV $VIRTUAL_ENV

ENV PATH="$VIRTUAL_ENV/bin:$PATH"

RUN python -m cfgrib selfcheck

requirements.txt

attrs==21.4.0
cffi==1.15.0
cfgrib==0.9.10.1
click==8.1.3
eccodes==1.4.2
findlibs==0.0.2
numpy==1.22.4
packaging==21.3
pandas==1.4.2
pycparser==2.21
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
six==1.16.0
xarray==2022.3.0

Answer 1

• Instead of a COPY + RUN tar, you can get Docker to do the extraction for you by using ADD, since you have the file downloaded externally anyway:

  If <src> is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed. When a directory is copied or unpacked, it has the same behavior as tar -x, the result is the union of:

  1. Whatever existed at the destination path and
  2. The contents of the source tree, with conflicts resolved in favor of "2." on a file-by-file basis.

  Note

  Whether a file is identified as a recognized compression format or not is done solely based on the contents of the file, not the name of the file. For example, if an empty file happens to end with .tar.gz this will not be recognized as a compressed file and will not generate any kind of decompression error message, rather the file will simply be copied to the destination.

• apt-get install -y will not suppress debconf questions (e.g., selecting the local timezone when install the timezone database package). However, since Docker doesn't provide an terminal on which debconf questions can be presented, they'll be automatically answered anyway, but this will lead to more noise in the build log. You should set the environment variable DEBIAN_FRONTEND=noninteractive to suppress debconf questions.

  The Docker documentation on ENV has an incorrect example of using DEBIAN_FRONTEND:

  If an environment variable is only needed during build, and not in the final image, consider setting a value for a single command instead:

  RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y ...
  

  In this command, DEBIAN_FRONTEND will only affect apt-get update, which won't be asking debconf questions anyway. It should be either:

  RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y ...
  

  Or:

  RUN export DEBIAN_FRONTEND=noninteractive; apt-get update && apt-get install -y ...
  

• RUN commands are run in independent shells and a cd in one wouldn't affect the working directory in another:

  Note that each instruction is run independently, and causes a new image to be created - so RUN cd /tmp will not have any effect on the next instructions.

  You should use WORKDIR instead, and since it can create the directory if it doesn't exist, skip the mkdir for that directory:

  RUN mkdir "$ECCODES_DIR"
  WORKDIR build
  RUN cmake ../eccodes-2.24.2-Source -DCMAKE_INSTALL_PREFIX="$ECCODES_DIR" -DENABLE_JPG=ON
  

  (Also, quote your variables.)

• Try to order layers in a way that matches their dependencies. In this snippet, only the last RUN depends on the COPY, but the preceding RUN statements will be re-executed each time requirements.txt changes, even if the cached results are perfectly usable:

  COPY ./requirements.txt ./requirements.txt 
  # create the virtual env
  RUN python3 -m venv $VIRTUAL_ENV 
  # upgrade pip
  RUN python3 -m pip install --upgrade pip
  # install the requirements
  RUN pip install -r requirements.txt
  

  I'd suggest:

  # create the virtual env
  RUN python3 -m venv "$VIRTUAL_ENV"
  # upgrade pip
  RUN python3 -m pip install --upgrade pip
  COPY ./requirements.txt ./requirements.txt 
  # install the requirements
  RUN pip install -r requirements.txt
  

  (Or keep the COPY before upgrading pip. In either case, the virtualenv creation doesn't need to be re-run each time requirements.txt changes.)

Answer 2

Minor: tar -xzf eccodes-2.24.2-Source.tar.gz can simply use tar xf as the z is implied by the extension; from the manual:

The only case when you have to specify a decompression option while reading the archive is when reading from a pipe or from a tape drive that does not support random access.