Skip to content

Containers

Containers on HPC systems

To date, only few container runtime environments integrate well with HPC environments due to security concerns and differing assumptions in some areas.

For example native Docker environments require elevated privileges, which is not an option on shared HPC resources. Docker's "rootless mode" is also currently not supported on our HPC systems because it does not support necessary features such as cgroups resource controls, security profiles, overlay networks, furthermore GPU passthrough is difficult. Necessary subuid (newuidmap) and subgid (newgidmap) settings may impose security issues.

On HoreKa Enroot and Singularity are supported.

Further rootless container runtime environments (Podman, …) might be supported in the future, depending on how support for e.g. network interconnects, security features and HPC file systems develops.

docker logo Enroot enables you to run Docker containers on HPC systems. It is developed by NVIDIA. It is the recommended tool to use containers on HoreKa and integrates well with GPU usage.

Enroot is available to all users by default.

Usage

Excellent documentation is provided on NVIDIA's github page. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.

Using Docker containers with Enroot requires three steps:

  1. Importing an image
  2. Creating a container
  3. Starting a container

Optionally containers can also be exported and transferred.

Importing a container image

  • enroot import docker://alpine
    This pulls the latest alpine image from dockerhub (default registry). You will obtain the file alpine.sqsh.

  • enroot import docker://nvcr.io#nvidia/pytorch:21.04-py3
    This pulls the latest pytorch image from NVIDIA's NGC registry. You will obtain the file nvidia+pytorch+21.04-py3.sqsh.

  • enroot import docker://registry.scc.kit.edu#myProject/myImage:latest
    This pulls your latest Image from the KIT registry. You obtain the file myImage.sqsh.

Creating a container

  • enroot create --name nvidia+pytorch+21.04-py3 nvidia+pytorch+21.04-py3.sqsh
    Create a container named nvidia+pytorch+21.04-py3 by unpacking the .sqsh-file.

"Creating" a container means that the squashed container image is unpacked inside $ENROOT_DATA_PATH/. By default this variable points to $HOME/.local/share/enroot/.

Starting a container

  • enroot start --rw nvidia+pytorch+21.04-py3 bash
    Start the container nvidia+pytorch+21.04-py3 in read-write mode (--rw) and run bash inside the container.

  • enroot start --root --rw nvidia+pytorch+21.04-py3 bash
    Start container in --rw-mode and get root access (--root) inside the container.
    You can now install software with root privileges, depending on the containerized Linux distribution e.g. with apt-get install …, apk add …, yum install …, pacman -S …

  • enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 bash
    Start container and mount (-m) a local directory to /work inside the container.

  • enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab
    Start container, mount a directory and start the application jupyter lab.

Exporting and transfering containers

If you intend to use Docker images which you built e.g. on your local desktop, and transfer them somewhere else, there are several possibilities to do so:

  1. Build an image via docker build and a Dockerfile, import this image from the Docker daemon via
    enroot import --output myImage.sqsh dockerd://myImage.
    Copy the .sqsh-file to HoreKa and import it with enroot import.

  2. Export an existing enroot container via
    enroot export --output myImage.sqsh myImage.
    Copy the .sqsh-file to HoreKa and import it with enroot import.

  3. Create a self extracting bundle from a container image via
    enroot bundle --output myImage.run myImage.sqsh.
    Copy the .run-file to HoreKa. You can run the self extracting image via ./myImage.run even if enroot is not installed!

Container management

You can list all containers on the system and additional information (--fancy parameter) with the enroot list command.

The unpacked images can be removed with the enroot remove command.

ENROOT FAQ

How can I run JupyterLab in a container and connect to it?
  • Start an interactive session with or without GPUs. Notice the compute node ID the session is running on, and start a container with a running JupyterLab, e.g.:
    enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab --no-browser --ip 0.0.0.0
  • Open a terminal on your desktop and create a SSH-tunnel to the running JupyterLab instance on the compute node. Insert the node ID, where the interactive session is running on:
    ssh -L8888:<computeNodeID>:8888 <yourAccount>@hk.scc.kit.edu
  • Open a web browser and open the URL localhost:8888
  • Enter the token, which is visible in the output of the first terminal. The output should look like this:
    Or copy and paste this URL:
            http://hostname:8888/?token=fdaa7bf344b9ef3c0623c6e…4ce56dd845
    
    Copy the string behind the token= and paste it into the input field in the browser.
Are GPUs accessible from within a running container?

Yes.
Unlike Docker, Enroot does not need further command line options to enable GPU passthrough like --runtime=nvidia or --privileged.

Is there something like enroot-compose?

AFAIK no. Enroot is mainly intended for HPC workloads, not for operating multi-container applications. However, starting and running these applications separately is possible.

Can I use workspaces to store containers?

Yes.
You can define the location of configuration files and storage with environment variables. The ENROOT_DATA_PATH variable should be set accordingly. Please refer to NVIDIA's documentation on runtime configuration.

Additional resources

Source code: https://github.com/NVIDIA/enroot
Documentation: https://github.com/NVIDIA/enroot/blob/master/doc
Additional information: FOSDEM 2020 talk + slides

Singularity

singularity logo Membership in a special group is required to be able to use Singularity . Please refer to the Support channels to request access.

Usage

Excellent documentation is provided on the Documentation&Examples page provided by Sylabs, the company behind Singularity. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.

Using Singularity usually involves two steps:

  • Building a container image using singularity build

  • Running a container image using singularity run or singularity exec

Building an image

  • singularity build ubuntu.sif library://ubuntu
    This pulls the latest Ubuntu image from Singularity's Container Library and locally creates a container image file called ubuntu.sif.

  • singularity build alpine.sif docker://alpine
    This pulls the latest alpine image from Dockerhub and locally creates a container image file called alpine.sif.

  • singularity build pytorch-21.04-p3.sif docker://nvcr.io#nvidia/pytorch:21.04-py3
    This pulls the latest pytorch image from NVIDIA's NGC registry and locally creates a container image file called pytorch-21.04-p3.sif.

Running an image

  • singularity shell ubuntu.sif Start a shell in the Ubuntu container.

  • singularity run alpine.sif
    Start the container alpine.sif and run the default runscript provided by the image.

  • singularity exec alpine.sif /bin/ls
    Start the container alpine.sif and run the /bin/ls command.

Container management

You can use the singularity search command to search for images on Singularity's Container Library.


Last update: June 10, 2021