Using Containers¶

Singularity and Docker (via Enroot) are supported on both HoreKa and bwUniCluster2. Both solutions can be used ,,just as normal'', both on [hk,uc2]-ci-controller.scc.kit.edu as well as on the HPC nodes when started from within a batch job.

The Enroot setup currently supports root-less execution of Docker images with native access to NVIDIA GPUs, but it does not support MPI jobs yet.

Using Docker images with the Enroot Custom executor¶

GitLab Runner offers the possibility to use so-called ,,Custom Executors'' for environments that are not natively supported. Since we cannot offer native Docker support on the HPC and FTP systems due to security and stability issues, we provide a custom executor that makes it possible to run Docker Images without root privileges using Enroot.

The Custom Executor consists of a set of scripts and an associated GitLab-Runner configuration template. The scripts can be found under /usr/share/gitlab-runner/custom-executor-enroot/ on the HPC and FTP systems and can be used without any modification.

In each stage, GitLab-Runner executes the following scripts in the given order:

config.sh
prepare.sh
run.sh
cleanup.sh

The prepare.sh script imports the configured Docker image and creates an Enroot container based on it. A separate container is created for each stage, so that parallel stages do not interfere with each other. The run.sh script executes the CI jobs configured in your .gitlab-ci.yml file.

The actual build steps are performed within the enroot container. All tasks concerning the handling of the GIT repositories, the artifacts and the cache are executed outside the container. As a result, the container does not need to contain a git and gitlab-runner binary. By mapping the build directory into the container, the corresponding data is available in the container. In the cleanup.sh script, the previously created container is deleted, self-created containers remain unaffected.

In the configuration template config.toml in the directory /usr/share/gitlab-runner/custom-executor-enroot/ the following areas to be edited are marked by angle brackets:

<REGISTRATION_URL> Register the runner with this URL (e.g. https://git.scc.kit.edu/). You can find the required information in the GitLab Web interface in the Your Project → Settings menu → CI / CD sub menu by selecting Runners → Specific runners sub section → Set up a specific runner manually.
<TOKEN> Register the runner with this registration token. You can find the required information in the GitLab Web interface in the Your Project → Settings menu → CI / CD sub menu by selecting Runners → Specific runners sub section → Set up a specific runner manually.
<BUILDS_DIR> The base directory where the working directory of the job will be created (e.g. /AbsolutPath/To/Home/gitlab-runner/builds/)
<CACHE_DIR> The base directory where local cache will be stored (e.g. /AbsolutPath/To/Home/gitlab-runner/cache/)

Setup steps¶

Copy the GitLab-Runner configuration template to the usual position in your home directory:

$ install --backup --verbose \
      "/usr/share/gitlab-runner/custom-executor-enroot/config.toml" \
      "${HOME}/.gitlab-runner/config.toml"

Use the editor of your choice to edit the GitLab-Runner configuration file. The placeholders in the angle brackets have to be replaced by the actual configuration options:
```
$ vim "${HOME}/.gitlab-runner/config.toml"
```
Start or restart the gitlab-runner as specified in the instructions for the CI Level you are using. If your runner is located on a dedicated worker node managed by the CI admin team, the admins will set up the Enroot Custom Executor.
Once the Enroot Custom Executor is set up, it will pull and execute Docker images specified in .gitlab-ci.yml like the Docker executor does.. For example, the following block will pull the Docker image called bash from the Standard Registry, set it up and execute the command cat /etc/issue inside the container:
```
default:
  image: bash

Show prelogin message and identification file:
  stage: build
  script:
    - cat /etc/issue
```

Last update: September 28, 2023