2
\$\begingroup\$

I wrote a Dockerfile to install ecCodes for use in a python environment using cfgrib and xarray.

The tarball can be found on the ecCode releases page

Dockerfile

# syntax=docker/dockerfile:1
# ECCODE COMPILE
FROM python:3.10.4 as compiler
# /path/to/where/you/install/eccodes
ENV ECCODES_DIR=/usr/src/eccodes
# the zip file for eccodes
COPY ./eccodes-2.24.2-Source.tar.gz ./eccodes-2.24.2-Source.tar.gz

RUN apt-get update -y
# compiler tools
RUN apt-get install -y --no-install-recommends \
    cmake \
    gfortran \
    build-essential

# unzip 
RUN tar -xzf eccodes-2.24.2-Source.tar.gz 
# prepare the build folders
RUN mkdir $ECCODES_DIR build && cd build
# 
RUN cmake ../eccodes-2.24.2-Source -DCMAKE_INSTALL_PREFIX=$ECCODES_DIR -DENABLE_JPG=ON

RUN make && ctest && make install


# PYTHON BUILD
FROM python:3.10.4 as builder

ENV VIRTUAL_ENV=/opt/venv

COPY ./requirements.txt ./requirements.txt 
# create the virtual env
RUN python3 -m venv $VIRTUAL_ENV 
# upgrade pip
RUN python3 -m pip install --upgrade pip
# install the requirements
RUN pip install -r requirements.txt


# FINAL
FROM python:3.10.4

COPY --from=compiler $ECCODES_DIR $ECCODES_DIR

ENV ECCODES_DIR=$ECCODES_DIR

COPY --from=builder $VIRTUAL_ENV $VIRTUAL_ENV

ENV PATH="$VIRTUAL_ENV/bin:$PATH"

RUN python -m cfgrib selfcheck

requirements.txt

attrs==21.4.0
cffi==1.15.0
cfgrib==0.9.10.1
click==8.1.3
eccodes==1.4.2
findlibs==0.0.2
numpy==1.22.4
packaging==21.3
pandas==1.4.2
pycparser==2.21
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
six==1.16.0
xarray==2022.3.0
\$\endgroup\$

2 Answers 2

3
\$\begingroup\$
  • Instead of a COPY + RUN tar, you can get Docker to do the extraction for you by using ADD, since you have the file downloaded externally anyway:

    If <src> is a local tar archive in a recognized compression format (identity, gzip, bzip2 or xz) then it is unpacked as a directory. Resources from remote URLs are not decompressed. When a directory is copied or unpacked, it has the same behavior as tar -x, the result is the union of:

    1. Whatever existed at the destination path and
    2. The contents of the source tree, with conflicts resolved in favor of "2." on a file-by-file basis.

    Note

    Whether a file is identified as a recognized compression format or not is done solely based on the contents of the file, not the name of the file. For example, if an empty file happens to end with .tar.gz this will not be recognized as a compressed file and will not generate any kind of decompression error message, rather the file will simply be copied to the destination.

  • apt-get install -y will not suppress debconf questions (e.g., selecting the local timezone when install the timezone database package). However, since Docker doesn't provide an terminal on which debconf questions can be presented, they'll be automatically answered anyway, but this will lead to more noise in the build log. You should set the environment variable DEBIAN_FRONTEND=noninteractive to suppress debconf questions.

    The Docker documentation on ENV has an incorrect example of using DEBIAN_FRONTEND:

    If an environment variable is only needed during build, and not in the final image, consider setting a value for a single command instead:

    RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y ...
    

    In this command, DEBIAN_FRONTEND will only affect apt-get update, which won't be asking debconf questions anyway. It should be either:

    RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y ...
    

    Or:

    RUN export DEBIAN_FRONTEND=noninteractive; apt-get update && apt-get install -y ...
    
  • RUN commands are run in independent shells and a cd in one wouldn't affect the working directory in another:

    Note that each instruction is run independently, and causes a new image to be created - so RUN cd /tmp will not have any effect on the next instructions.

    You should use WORKDIR instead, and since it can create the directory if it doesn't exist, skip the mkdir for that directory:

    RUN mkdir "$ECCODES_DIR"
    WORKDIR build
    RUN cmake ../eccodes-2.24.2-Source -DCMAKE_INSTALL_PREFIX="$ECCODES_DIR" -DENABLE_JPG=ON
    

    (Also, quote your variables.)

  • Try to order layers in a way that matches their dependencies. In this snippet, only the last RUN depends on the COPY, but the preceding RUN statements will be re-executed each time requirements.txt changes, even if the cached results are perfectly usable:

    COPY ./requirements.txt ./requirements.txt 
    # create the virtual env
    RUN python3 -m venv $VIRTUAL_ENV 
    # upgrade pip
    RUN python3 -m pip install --upgrade pip
    # install the requirements
    RUN pip install -r requirements.txt
    

    I'd suggest:

    # create the virtual env
    RUN python3 -m venv "$VIRTUAL_ENV"
    # upgrade pip
    RUN python3 -m pip install --upgrade pip
    COPY ./requirements.txt ./requirements.txt 
    # install the requirements
    RUN pip install -r requirements.txt
    

    (Or keep the COPY before upgrading pip. In either case, the virtualenv creation doesn't need to be re-run each time requirements.txt changes.)

\$\endgroup\$
2
  • \$\begingroup\$ I really appreciate your input. (Also, quote your variables.) this is the thing I think gives me the most trouble. I don't understand when $MY_VAR or "$MY_VAR" or ${MY_VAR} and "${MY_VAR}" is more appropriate. Could you point me to a resource where I can better inform myself. My linter flagged your WORKDIR build with WORKDIR paths should be absolutedockerfile-utils(49) \$\endgroup\$ Jun 20 at 19:37
  • \$\begingroup\$ @JasonLeaver see unix.stackexchange.com/a/4910/70524 and unix.stackexchange.com/a/171347/70524 about the quoting. As for your linter, you should exercise your judgement when it comes to linter warnings. In this case it makes no difference where, relatively, the workdir is, since what you care about in the end is $ECCODES_DIR, and that's an absolute path. But if you want to silence it, maybe add a WORKDIR /eccodes before copying the tar file and then use WORKDIR /eccodes/build instead of plain WORKDIR build. This way all relevant paths are absolute and set by you. \$\endgroup\$
    – muru
    Jun 21 at 3:19
3
\$\begingroup\$

Minor: tar -xzf eccodes-2.24.2-Source.tar.gz can simply use tar xf as the z is implied by the extension; from the manual:

The only case when you have to specify a decompression option while reading the archive is when reading from a pipe or from a tape drive that does not support random access.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.