GitLab CI with podman

We know GitLab CI with docker runners for quiet a while now, but what’s about GitLab CI with podman? Podman is the next generation container tool under Linux, it can start docker containers within the user space, no root privileges are required. With RHEL 8 there is no docker runtime available at the moment, but Red Hat supports podman. But how can we integrate that with GitLab CI? The GitLab CI runner has some native support (called executor) for docker, shell, …, but there is no native support for podman. There are two possibilities, using the shell runner or using the custom runner. With the shell runner, you have to ensure that every project starts podman, and only podman. So let’s try the custom runner.

GitLab CI runner with custom executor

Let’s start build a GitLab CI custom executor with podman on a RHEL/CentOS 7 or 8 with a really basic container. First, install the gitlab-ci-runner Go binary and create a user with a home directory under which the gitlab-ci-runner should run later.
For this example we assume there is a unix user called gitlab-runner with the home directory /home/gitlab-runner. This user is able to run podman. Let’s try that:

sudo -u gitlab-runner podman run -it --rm \
    registry.code.immerda.ch/immerda/container-images/base/fedora:30 \
    bash

Next, let’s make a systemd service for the GitLab runner (/etc/systemd/system/gitlab-runner.service):

[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/local/bin/gitlab-runner

[Service]
User=gitlab-runner
Group=gitlab-runner
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/local/bin/gitlab-runner run --working-directory /home/gitlab-runner
Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target

Now, let’s register a runner to a GitLab instance.

sudo -u gitlab-runner gitlab-runner register \
    --url https://code.immerda.ch/ \
    --registration-token $GITLAB_REGISTRATION_TOKEN \
    --name "Podman fedora runner" \
    --executor custom \
    --builds-dir /home/user \
    --cache-dir /home/user/cache \
    --custom-prepare-exec "/home/gitlab-runner/fedora/prepare.sh" \
    --custom-run-exec "/home/gitlab-runner/fedora/run.sh" \
    --custom-cleanup-exec "/home/gitlab-runner/fedora/cleanup.sh"
  • –builds-dir: The build directory within the container.
  • –cache-dir: The cache directory within the container.
  • –custom-prepare-exec: Prepare the container before each job.
  • –custom-run-exec: Pass the .gitlab-ci.yml script items to the container.
  • –custom-cleanup-exec: Cleanup all left-overs after each job.

There are three scripts referenced at this point. Those script will be executed for each job (a CI/CD pipeline can contain multiple jobs, e.g. build, test, deploy). The whole magic will happen within those scripts. The output of those scripts is always shown in the GitLab job, so for debugging reasons it’s possible to do a set -x.

Scripts

Every job will start all the referenced scripts. First have a look on some variables we need during all scripts. Let’s create a file /home/gitlab-runner/fedora/base.sh

CONTAINER_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-$CUSTOM_ENV_CI_JOB_ID"
IMAGE="registry.code.immerda.ch/immerda/container-images/base/fedora:30"
CACHE_DIR="$(dirname "${BASH_SOURCE[0]}")/../_cache/runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-pipeline-$CUSTOM_ENV_CI_PIPELINE_ID"
  • CONTAINER_ID: Name of the container.
  • IMAGE: Image to use for the container.
  • CACHE_DIR: The cache directory on the host system.

Prepare script

The prepare executable (/home/gitlab-runner/fedora/prepare.sh) will

  • pull the image from the registry
  • start a container
  • install the dependencies (curl, git, gitlab-runner)
#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

start_container() {
    if podman inspect "$CONTAINER_ID" >/dev/null 2>&1; then
        echo 'Found old container, deleting'
        podman kill "$CONTAINER_ID"
        podman rm "$CONTAINER_ID"
    fi

    # Container image is harcoded at the moment, since Custom executor
    # does not provide the value of `image`. See
    # https://gitlab.com/gitlab-org/gitlab-runner/issues/4357 for
    # details.
    mkdir -p "$CACHE_DIR"
    podman pull "$IMAGE"
    podman run \
        --detach \
        --interactive \
        --tty \
        --name "$CONTAINER_ID" \
        --volume "$CACHE_DIR":"/home/user/cache" \
        "$IMAGE"
}

install_dependencies() {
    podman exec -u 0 "$CONTAINER_ID" sh -c "dnf install -y git curl"

    # Install gitlab-runner binary since we need for cache/artifacts.
    podman exec -u 0 "$CONTAINER_ID" sh -c "curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
    podman exec -u 0 "$CONTAINER_ID" sh -c "chmod +x /usr/local/bin/gitlab-runner"
}

echo "Running in $CONTAINER_ID"

start_container
install_dependencies

Run script

The run executable (/home/gitlab-runner/fedora/run.sh) will run the commands defined in the .gitlab-ci.yml within the container

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

podman exec "$CONTAINER_ID" /bin/bash < "$1"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit $BUILD_FAILURE_EXIT_CODE
fi

Cleanup script

And finally the cleanup executable (/home/gitlab-runner/fedora/cleanup.sh) will cleanup after every job.

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

echo "Deleting container $CONTAINER_ID"

podman kill "$CONTAINER_ID"
podman rm "$CONTAINER_ID"
exit 0

The script above doesn’t cleanup the cache. The reason is, that perhaps we need the cache during the next job or during the next pipeline. So an additional cleanup on the host system is needed for purging the cache after a while.

Last but not least

This is just a quick howto. If you wanna implement that, there are a lot of improvements. This should just explain how the custom executor can be used and how to use podman for GitLab CI runner. At the moment there is no support for the tag image (see #4357).

The state of Forward Secrecy in OpenSSL

It could be possible that your SSL services are not providing
forward secrecy and you haven’t noticed yet!

Many SSL ciphers provide forward secrecy by using ephedermal Diffie-Hellman (EDH) keys. This means that for every SSL session a temporary encryption key is negotiated and the normal key is only used for verifying authenticity. As the OpenSSL documentation states:

“By generating a temporary DH key inside the server application that is lost when the application is left, it becomes impossible for an attacker to decrypt past sessions, even if he gets hold of the normal (certified) key, as this key was only used for signing.”

Although ciphers using EDH will most probably be available in your setup, often they are disabled because the application fails to provide DH params to OpenSSL. Since it is costly to generate those parameters – which are needed to negotiate a DH key exchange – OpenSSL suggests to create them when an application is installed.

Many application will not do this, but rather let the user generate and include the parameters in the configuration manually. Since (i) most administrators are not aware of this problem, (ii) those applications do not yield any warnings if the parameters are missing and (iii) OpenSSL silently disables ciphers with unsatisfied requirements, forward secrecy is not available in many SSL connections.

Update: Also see Bernats blog for a nice roundup on the cryptographic background of perfect forward secrecy and the new, faster elliptic curve implementations.

Verify your Setup

Try to open an SSL session to your service (https, imap, smtp, jabber, irc, …) with

openssl s_client -port <port> -host <yourdomain.tld>

this will show you the details of the SSL session and you can verify that the used cipher includes EDH:

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

or not:

New, TLSv1/SSLv3, Cipher is AES256-SHA

Fix your Setup

Applications which we found to work with EDH ciphers are Apache and Dovecot.

Update: Applications which we found to not support EDH out of the box are: squid, exim, courier

In most applications you can configure a dhparams variable somewhere. The dhparams can be generated with the following command:

openssl dhparam -out dhparams.pem 2048

We already fixed the problem in the following services:

Squid (reverse proxy)

In /etc/squid/include.d/https_port add dhparams=/path/dhparams.pem to every line

Exim

In /etc/exim.conf add the line tls_dhparam = /path/dhparams.pem

Fix the general Problem

This problem has two main reasons:

  1. Applications do not check whether the requirements of the user selected ciphers are satisfied. The requirements are listed in the OpenSSL doku. Or they could just always generate dhparams when they are installed, since EDH ciphers should be preferred anyway.
  2. The OpenSSL API does not provide any means to verify the state of the configuration. There is no function to check if cipher requirements are met and the SSL_CTX setup is consistent. As long as at least a single cipher (even the least secure) in the acceptable ciphers list can be initialized OpenSSL will not complain to the application.

If you find any application which exhibits this problem, please file a bug report and convince the maintainers to at least generate a warning to the user and state the consequences in the documentation.

If you are a developer of an application which uses OpenSSL please consider shipping install scripts that generate dhparams or generate them on the fly if they are missing. Please do not just let OpenSSL silently disable a key feature of SSL.