CI Level 2¶
Each HPC system has a dedicated login node on which it is allowed to run CI workers, and some obstacles for doing so have been removed. For HoreKa the DNS entry hk-ci-controller.scc.kit.edu points to this node, for bwUniCluster 2.0 (UC2) the DNS entry is uc2-ci-controller.scc.kit.edu.
CI Worker Management¶
Currently gitlab-runner is pre-installed and supported. After logging into [hk,uc2]-ci-controller.scc.kit.edu you can register CI jobs with gitlab-runner as usual:
$ gitlab-runner register
$ gitlab-runner list
Two executors are available: ,,shell'' and our Custom Enroot Executor for Docker. There are no restrictions as to which GitLab instances can be registered. Private instances might be unreachable due to network/firewall limitations, though.
The install, start, stop, status and restart actions of gitlab-runner do not work. Please use the following commands instead:
$ systemctl --user enable gitlab-runner
$ systemctl --user start gitlab-runner
$ systemctl --user status gitlab-runner
$ systemctl --user restart gitlab-runner
gitlab-runner needs ${HOME}/.gitlab-runner/config.toml !
It is not possible to start gitlab-runner if the configuration file ${HOME}/.gitlab-runner/config.toml does not exist, i.e. gitlab-runner register has not been executed successfully before!
Use the following command to see the log output of your gitlab-runner:
$ journalctl --user --unit gitlab-runner
Your gitlab-runner will keep running after you have logged out. It will be restarted automatically after system reboots and in case it crashes.
The shell executor of your gitlab-runner will run all commands on the login node. This is okay for simple commands and small to medium-size build jobs. Please move all elaborate and time-consuming tasks into a HPC job.
Running HPC jobs from a gitlab-runner script¶
salloc can be used to run a job on the HPC system, block until it has been executed and collect all output. Example entry for gitlab-ci.yml:
myproject:
stage: test
script:
- salloc -p develop -t 00:20:00 -N1 -n10 test-on-HoreKa.sh
GitLab Pipeline timeout settings
You might have to adjust the Timeout value in the GitLab settings of your project (Settings -> CI/CD -> General pipelines) to match the maximum time your CI jobs need to execute, especially if you are submitting batch jobs from within your CI scripts. The GitLab instance wile mark your job as failed if the timeout is exceeded.
Triggering builds¶
These types of jobs can be set up in GitLab itself. Also it is allowed to setup Cron jobs on [hk,uc2]-ci-controller.scc.kit.edu that trigger builds via the GitLab HTTP API.
Reservations¶
Reservations can be set up on request, e.g. for regular nightly builds with timing constraints. Please contact the HPC Operations team by sending an e-mail to hpc-ci-hotline@lists.kit.edu and include a short explanation of your use case.