Immerda Techblog

Container images can be layered on each other, so that you do not need to always rebuild different layers from scratch.
This helps you to a) standardize your container images and b) to not to repeat yourself all over the place.
As an example, we have a couple of ruby applications, and they all require ruby, bundler and some common dependencies, since every ruby app eventually bundles nokogiri and thus requires a bunch of libxml and libxslt libraries. Simplified this means we have the following chain:

base -> ruby -> app

At immerda we build our own base images from scratch so we can fully control what is in them. E.g. we add our repository, our CA certificate and so on. Then we have a common ruby image, that can be used to develop and test ruby applications. And in the end we might package the application as an image, like our WKD service.

We are updating our systems on regular and unattended basis to keep the systems itself up to date with the latest security releases and bug fixes. Now when it comes to container images, we are also pulling the images of running containers on a regular basis and restarting the containers / pods when there was update to the image tag the container is referring to. This way we do not only keep the applications packaged in containers up to date, but are also making sure, that we get security fixes for lower level libraries, like e.g. openssl (remember heartbleed?) in a timely manner.

Since container images are intended to be immutable (and we actually run nearly all of our containers, with –readonly=true) you want to and must rebuild to keep them updated.

Now when it comes to pull and use images from random projects or registries (looking at you docker-hub), we try to vet images at the time we start using them that they are also continuously built. As an example the official library images, like the official postgres images are good examples for that.
For our own images, we are doing that through gitlab-ci.

However, since images are layered on each other, it means, that you need to rebuild your images in the right order, so that your ruby image builds on top of your latest base image and your app image uses the latest ruby updates.

Gitlab-ci has limited capabilities (especially in the free core version) when it comes to modelling pipelines across projects in the right order. However, with building blocks as webhook events for pipeline runs, we can orchestrate that pretty well.

This orchestration is done using a project form our friends at Autistici/Inventati called gitlab-deps. This tool keeps track of the dependency chain of your images and triggers the pipelines to build these images in the right order.

Central to gitlab-deps is a small webhook receiver, that receives events from our gitlab-ci pipelines and based on the dependency list of your images, it triggers the next pipelines. We are running the webhook receiver in a container and it scans the immerda group in gitlab for projects, either using a Containerfile/Dockerfile FROM statement or a list of depending projects in a file called .gitlab-deps. The latter is much more accurate and likely the better idea to describe the dependencies of your project. As an example see the gitlab-deps container project itself.

gitlab-deps itself requires an API token for gitlab, that is able to trigger pipeline runs, as well as adding webhooks to the project. For now this seems to be a token from a user with the Maintainer role.

We then also trigger a systemd-timer, that runs a script in the container to update all the dependencies as well as add webhooks to all the projects we know of.

For our base images, we defined a scheduled pipeline run to kick off building the initial lowest layer of our images, which – if successful – will trigger the next layers and then the next layer and so on.

If one of the pipelines fails gitlab-deps won’t trigger anything further down the chain. Gitlab will notify you (e.g. via email) about the failed pipeline run, you can investigate and if you are able to re-trigger a successful pipeline run, the chain continuous to be built.

While we are using this now for our image build chain, gitlab-deps can also be used to rebuild/update projects that are depending on a library and so on. While moving more of our work into gitlab and hooking it into CI, we might also start using these dependency triggers for other kinds of pipeline chains.

We know GitLab CI with docker runners for quiet a while now, but what’s about GitLab CI with podman? Podman is the next generation container tool under Linux, it can start docker containers within the user space, no root privileges are required. With RHEL 8 there is no docker runtime available at the moment, but Red Hat supports podman. But how can we integrate that with GitLab CI? The GitLab CI runner has some native support (called executor) for docker, shell, …, but there is no native support for podman. There are two possibilities, using the shell runner or using the custom runner. With the shell runner, you have to ensure that every project starts podman, and only podman. So let’s try the custom runner.

GitLab CI runner with custom executor

Let’s start build a GitLab CI custom executor with podman on a RHEL/CentOS 7 or 8 with a really basic container. First, install the gitlab-ci-runner Go binary and create a user with a home directory under which the gitlab-ci-runner should run later.
For this example we assume there is a unix user called gitlab-runner with the home directory /home/gitlab-runner. This user is able to run podman. Let’s try that:

sudo -u gitlab-runner podman run -it --rm \
    registry.code.immerda.ch/immerda/container-images/base/fedora:30 \
    bash

Next, let’s make a systemd service for the GitLab runner (/etc/systemd/system/gitlab-runner.service):

[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/local/bin/gitlab-runner

[Service]
User=gitlab-runner
Group=gitlab-runner
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/local/bin/gitlab-runner run --working-directory /home/gitlab-runner
Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target

Now, let’s register a runner to a GitLab instance.

sudo -u gitlab-runner gitlab-runner register \
    --url https://code.immerda.ch/ \
    --registration-token $GITLAB_REGISTRATION_TOKEN \
    --name "Podman fedora runner" \
    --executor custom \
    --builds-dir /home/user \
    --cache-dir /home/user/cache \
    --custom-prepare-exec "/home/gitlab-runner/fedora/prepare.sh" \
    --custom-run-exec "/home/gitlab-runner/fedora/run.sh" \
    --custom-cleanup-exec "/home/gitlab-runner/fedora/cleanup.sh"

• –builds-dir: The build directory within the container.
• –cache-dir: The cache directory within the container.
• –custom-prepare-exec: Prepare the container before each job.
• –custom-run-exec: Pass the .gitlab-ci.yml script items to the container.
• –custom-cleanup-exec: Cleanup all left-overs after each job.

There are three scripts referenced at this point. Those script will be executed for each job (a CI/CD pipeline can contain multiple jobs, e.g. build, test, deploy). The whole magic will happen within those scripts. The output of those scripts is always shown in the GitLab job, so for debugging reasons it’s possible to do a set -x.

Scripts

Every job will start all the referenced scripts. First have a look on some variables we need during all scripts. Let’s create a file /home/gitlab-runner/fedora/base.sh

CONTAINER_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-$CUSTOM_ENV_CI_JOB_ID"
IMAGE="registry.code.immerda.ch/immerda/container-images/base/fedora:30"
CACHE_DIR="$(dirname "${BASH_SOURCE[0]}")/../_cache/runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-pipeline-$CUSTOM_ENV_CI_PIPELINE_ID"

• CONTAINER_ID: Name of the container.
• IMAGE: Image to use for the container.
• CACHE_DIR: The cache directory on the host system.

Prepare script

The prepare executable (/home/gitlab-runner/fedora/prepare.sh) will

• pull the image from the registry
• start a container
• install the dependencies (curl, git, gitlab-runner)

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

start_container() {
    if podman inspect "$CONTAINER_ID" >/dev/null 2>&1; then
        echo 'Found old container, deleting'
        podman kill "$CONTAINER_ID"
        podman rm "$CONTAINER_ID"
    fi

    # Container image is harcoded at the moment, since Custom executor
    # does not provide the value of `image`. See
    # https://gitlab.com/gitlab-org/gitlab-runner/issues/4357 for
    # details.
    mkdir -p "$CACHE_DIR"
    podman pull "$IMAGE"
    podman run \
        --detach \
        --interactive \
        --tty \
        --name "$CONTAINER_ID" \
        --volume "$CACHE_DIR":"/home/user/cache" \
        "$IMAGE"
}

install_dependencies() {
    podman exec -u 0 "$CONTAINER_ID" sh -c "dnf install -y git curl"

    # Install gitlab-runner binary since we need for cache/artifacts.
    podman exec -u 0 "$CONTAINER_ID" sh -c "curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
    podman exec -u 0 "$CONTAINER_ID" sh -c "chmod +x /usr/local/bin/gitlab-runner"
}

echo "Running in $CONTAINER_ID"

start_container
install_dependencies

Run script

The run executable (/home/gitlab-runner/fedora/run.sh) will run the commands defined in the .gitlab-ci.yml within the container

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

podman exec "$CONTAINER_ID" /bin/bash < "$1"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit $BUILD_FAILURE_EXIT_CODE
fi

Cleanup script

And finally the cleanup executable (/home/gitlab-runner/fedora/cleanup.sh) will cleanup after every job.

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

echo "Deleting container $CONTAINER_ID"

podman kill "$CONTAINER_ID"
podman rm "$CONTAINER_ID"
exit 0

The script above doesn’t cleanup the cache. The reason is, that perhaps we need the cache during the next job or during the next pipeline. So an additional cleanup on the host system is needed for purging the cache after a while.

Last but not least

This is just a quick howto. If you wanna implement that, there are a lot of improvements. This should just explain how the custom executor can be used and how to use podman for GitLab CI runner. At the moment there is no support for the tag image (see #4357).