Immerda Techblog

We know GitLab CI with docker runners for quiet a while now, but what’s about GitLab CI with podman? Podman is the next generation container tool under Linux, it can start docker containers within the user space, no root privileges are required. With RHEL 8 there is no docker runtime available at the moment, but Red Hat supports podman. But how can we integrate that with GitLab CI? The GitLab CI runner has some native support (called executor) for docker, shell, …, but there is no native support for podman. There are two possibilities, using the shell runner or using the custom runner. With the shell runner, you have to ensure that every project starts podman, and only podman. So let’s try the custom runner.

GitLab CI runner with custom executor

Let’s start build a GitLab CI custom executor with podman on a RHEL/CentOS 7 or 8 with a really basic container. First, install the gitlab-ci-runner Go binary and create a user with a home directory under which the gitlab-ci-runner should run later.
For this example we assume there is a unix user called gitlab-runner with the home directory /home/gitlab-runner. This user is able to run podman. Let’s try that:

sudo -u gitlab-runner podman run -it --rm \
    registry.code.immerda.ch/immerda/container-images/base/fedora:30 \
    bash

Next, let’s make a systemd service for the GitLab runner (/etc/systemd/system/gitlab-runner.service):

[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/local/bin/gitlab-runner

[Service]
User=gitlab-runner
Group=gitlab-runner
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/local/bin/gitlab-runner run --working-directory /home/gitlab-runner
Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target

Now, let’s register a runner to a GitLab instance.

sudo -u gitlab-runner gitlab-runner register \
    --url https://code.immerda.ch/ \
    --registration-token $GITLAB_REGISTRATION_TOKEN \
    --name "Podman fedora runner" \
    --executor custom \
    --builds-dir /home/user \
    --cache-dir /home/user/cache \
    --custom-prepare-exec "/home/gitlab-runner/fedora/prepare.sh" \
    --custom-run-exec "/home/gitlab-runner/fedora/run.sh" \
    --custom-cleanup-exec "/home/gitlab-runner/fedora/cleanup.sh"

• –builds-dir: The build directory within the container.
• –cache-dir: The cache directory within the container.
• –custom-prepare-exec: Prepare the container before each job.
• –custom-run-exec: Pass the .gitlab-ci.yml script items to the container.
• –custom-cleanup-exec: Cleanup all left-overs after each job.

There are three scripts referenced at this point. Those script will be executed for each job (a CI/CD pipeline can contain multiple jobs, e.g. build, test, deploy). The whole magic will happen within those scripts. The output of those scripts is always shown in the GitLab job, so for debugging reasons it’s possible to do a set -x.

Scripts

Every job will start all the referenced scripts. First have a look on some variables we need during all scripts. Let’s create a file /home/gitlab-runner/fedora/base.sh

CONTAINER_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-$CUSTOM_ENV_CI_JOB_ID"
IMAGE="registry.code.immerda.ch/immerda/container-images/base/fedora:30"
CACHE_DIR="$(dirname "${BASH_SOURCE[0]}")/../_cache/runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-pipeline-$CUSTOM_ENV_CI_PIPELINE_ID"

• CONTAINER_ID: Name of the container.
• IMAGE: Image to use for the container.
• CACHE_DIR: The cache directory on the host system.

Prepare script

The prepare executable (/home/gitlab-runner/fedora/prepare.sh) will

• pull the image from the registry
• start a container
• install the dependencies (curl, git, gitlab-runner)

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

start_container() {
    if podman inspect "$CONTAINER_ID" >/dev/null 2>&1; then
        echo 'Found old container, deleting'
        podman kill "$CONTAINER_ID"
        podman rm "$CONTAINER_ID"
    fi

    # Container image is harcoded at the moment, since Custom executor
    # does not provide the value of `image`. See
    # https://gitlab.com/gitlab-org/gitlab-runner/issues/4357 for
    # details.
    mkdir -p "$CACHE_DIR"
    podman pull "$IMAGE"
    podman run \
        --detach \
        --interactive \
        --tty \
        --name "$CONTAINER_ID" \
        --volume "$CACHE_DIR":"/home/user/cache" \
        "$IMAGE"
}

install_dependencies() {
    podman exec -u 0 "$CONTAINER_ID" sh -c "dnf install -y git curl"

    # Install gitlab-runner binary since we need for cache/artifacts.
    podman exec -u 0 "$CONTAINER_ID" sh -c "curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
    podman exec -u 0 "$CONTAINER_ID" sh -c "chmod +x /usr/local/bin/gitlab-runner"
}

echo "Running in $CONTAINER_ID"

start_container
install_dependencies

Run script

The run executable (/home/gitlab-runner/fedora/run.sh) will run the commands defined in the .gitlab-ci.yml within the container

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

podman exec "$CONTAINER_ID" /bin/bash < "$1"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit $BUILD_FAILURE_EXIT_CODE
fi

Cleanup script

And finally the cleanup executable (/home/gitlab-runner/fedora/cleanup.sh) will cleanup after every job.

#!/usr/bin/env bash

currentDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
source ${currentDir}/base.sh

echo "Deleting container $CONTAINER_ID"

podman kill "$CONTAINER_ID"
podman rm "$CONTAINER_ID"
exit 0

The script above doesn’t cleanup the cache. The reason is, that perhaps we need the cache during the next job or during the next pipeline. So an additional cleanup on the host system is needed for purging the cache after a while.

Last but not least

This is just a quick howto. If you wanna implement that, there are a lot of improvements. This should just explain how the custom executor can be used and how to use podman for GitLab CI runner. At the moment there is no support for the tag image (see #4357).