Maintaining Open Source Singularity Since 2017

SCS Login

Introducing CDI Support to SingularityCE 4.0

By Staff

Oct 18, 2023 | Blog

With the ever increasing adoption of AI techniques in scientific research, as well as growing use of accelerators for traditional numerical workloads, easy access to GPUs and other devices in HPC environments is critical.
The 4.0 release of the SingularityCE container runtime introduces Container Device Interface (CDI) support. CDI is a new standard for making devices such as GPUs available in containers. It offers an alternative to previous approaches that were vendor specific, and unevenly supported across different container runtimes. Users of NVIDIA GPUs, and other devices with CDI configurations, will benefit from a consistent way of using them in containers that spans the cloud native and HPC fields.

What is CDI?

The Container Device Interface (CDI) is a CNCF-sponsored project under active development. It aims to define a standard for injecting devices into containerised environments and introduces the concept of a Device as a named resource that can be requested by a client. A Device is uniquely identified by its fully-qualified name which has the form:
vendor.com/class=name
and references the device vendor, the device class , and the device name. The combination of vendor and class is referred to as the device kind and names must be unique per kind. In the case of NVIDIA GPUs, nvidia.com is typically used as the vendor string and gpu as the class, meaning that a GPU (referenced by UUID) would have a fully-qualified name such as:
nvidia.com/gpu=GPU-79a2ba02-a537-ccbf-2965-8e9d90c0bd54
A CDI specification is a file on disk that defines one or more such Devices and defines the entities – device nodes, mounts, environment variables, and container lifecycle hooks – that are required to make these accessible in containerized environments. When a CDI Device is requested, these entities are mapped to modifications of the OCI Runtime Specification for a container, allowing a low-level runtime such as runc or crun to make these available.
Some of the advantages of CDI over ad-hoc modification of containers to allow for device injection include:
  • Container engines remain vendor-agnostic – a CDI enabled container engine will support CDI devices for any vendor as long as a valid CDI specification has been generated. The CDI project includes an API to perform specification validation and OCI Runtime Specification modification making it relatively straightforward for new engines to enable CDI. From a vendor perspective this also means that little to no work is required to allow customers to access devices in new CDI-enabled container engines.
  • Modifications are easier to reason about – the entities defined in a CDI specification map to common command line options or kubernetes pod spec values making their resultant modifications clearer. The one exception is the hooks, which remain somewhat opaque in terms of actual modifications. Here the use of hooks with command lines that call out specific functionality such as updating the ldcache or creating a symlink are used to provide some insight.
  • CDI specification generation is separate from the container lifecycle – a CDI-enabled container engine only needs to access the CDI specification once when creating a container. As long as the specification is valid at the point of creation, it doesn’t matter when it was generated. This allows for the CDI specification to be adjusted to a specific use case or platform. For example, in the case of an installation where devices remain static, a CDI specification can be generated once and reused for multiple containers. In cases where dynamic resources are required, the specification for a particular container can be generated “just-in-time” for it to be created.

OCI Support In SingularityCE

SingularityCE, a popular open-source container runtime for HPC workloads, led by Sylabs, has long supported running GPU enabled Docker/OCI containers on HPC systems that contain NVIDIA hardware. A large number of containers distributed through NVIDIA’s NGC Catalog directly support execution with Singularity.
Up until SingularityCE 4.0, OCI containers such as those available in NGC have been converted on the fly to Singularity’s own container format for execution. The converted containers are stored in individual SIF (Singularity Image Format) files. Running containers out of a single large file has significant advantages in many HPC environments.
While most OCI containers work well when converted by Singularity, some functionality isn’t supported. With SingularityCE 4.0, a new OCI-mode has been introduced that avoids converting the OCI images. Instead, their layers and config are encapsulated in a single SIF file and executed with a true OCI low-level runtime to improve compatibility.

GPU Support In SingularityCE

To date, SingularityCE has provided support for using NVIDIA GPUs in containers in two ways:
  • A naive approach, that binds in specific NVIDIA libraries and binaries from the host into the container. This doesn’t require external software, and generally works on all systems. The same basic code can also be used for other vendor’s devices. However, it lacks support for more complex configurations that NVIDIA’s own tooling provides.
  • A method that leverages nvidia-container-cli, from the NVIDIA Container Toolkit, to perform container setup. This allows more complex GPU configurations in the container, and support for additional environments such as GPU-enabled WSL2 on Windows. However, the approach is NVIDIA specific and means that Singularity depends on external software that may change outside of the constraints of a public standard, and that must be provided and maintained by system administrators.
The introduction of OCI-mode to SingularityCE 4.0 allows CDI driven GPU setup to be incorporated. This offers the benefits of:
  • Complete implementation of CDI GPU configurations that NVIDIA’s container toolkit can specify.
  • Compatibility and consistency with other container runtimes that support CDI.
  • Vendor neutrality – any vendor’s device for which a CDI configuration can be provided can be used in the same way.

Demo – PyTorch

We’ll perform some AI model training on AWS EC2 using the NVIDIA GPU-Optimized AMI. This image runs Ubuntu 20.04 and suggests a p3.2xlarge instance, providing a 16GiB P100 GPU by default. A pre-release version of SingularityCE 4 was installed from a .deb package. Both .deb and .rpm packages are available on the GitHub releases page.
Because making full use of rootless container runtimes (including SingularityCE’s OCI mode) requires cgroups v2, but Ubuntu 20.04 boots with cgroups v1, we’ve followed the instructions to enable cgroups v2 at the rootless containers getting started guide. If you use an AMI based on Ubuntu 22.04 you won’t need to do this, because 22.04 uses cgroups v2 by default.
The NVIDIA Container Toolkit is pre-installed in the GPU-Optimized AMI, but we need to generate the CDI configuration for the system, and ensure it’s accessible to normal users. Note that we must make sure that the nvidia_uvm kernel module is loaded first, so that CDI config generation finds the nvidia-uvm device.
$ nvidia-modprobe -u

$ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
…
INFO[0000] Generated CDI spec with version 0.5.0


$ sudo chmod 755 /etc/cdi
We now have a CDI configuration in place that gives access to the GPU as a CDI device named nvidia.com/gpu=0. Let’s check that it’s visible inside a container run with the --device option:
$ singularity exec --oci --device nvidia.com/gpu=0 \
    docker://ubuntu nvidia-smi --list-gpus
INFO:	Using cached OCI-SIF image
GPU 0: Tesla V100-SXM2-16GB (UUID: GPU-06505224-0c9b-8a89-1ee9-a4bd7d4e15c0)
Success! This shows that the CDI configuration is correctly exposing the device in the container, and making the essential GPU libraries and binaries available.
In the home directory of the AMI there’s a script (mnist_example.sh) that runs an MNIST training demo using Docker and a GPU-optimized PyTorch container from NVIDIA NGC. Let’s run the same training demo using SingularityCE, in OCI mode and using CDI.
First, we’ll pull the container from NGC to a local SIF file. Using the –oci flag here tells SingularityCE to fetch the container and store it as an OCI image inside the single SIF file. The container’s layers are squashed in the process, but it’s still an OCI image that we’ll run in the new OCI mode which provides better compatibility than Singularity’s traditional container conversion.
$ singularity pull --oci docker://nvcr.io/nvidia/pytorch:22.05-py3
…
INFO:	Converting OCI image to OCI-SIF format
INFO:	Squashing image to single layer
INFO:	Writing OCI-SIF image
INFO:	Cleaning up.
Once the SIF file has been generated, we’ll run the pytorch example. It’s located under /opt in the container, so we’ll use the --cwd flag to run it as python main.py from the correct place:
$ export EXAMPLE_DIR=/opt/pytorch/examples/upstream/mnist

$ singularity exec --oci --device nvidia.com/gpu=0 --cwd $EXAMPLE_DIR \
    pytorch_22.05-py3.sif python main.py
…
Train Epoch: 1 [0/60000 (0%)]    Loss: 2.305988
Train Epoch: 1 [640/60000 (1%)]    Loss: 1.823968
Train Epoch: 1 [1280/60000 (2%)]    Loss: 1.573821
Train Epoch: 1 [1920/60000 (3%)]    Loss: 1.145820
Train Epoch: 1 [2560/60000 (4%)]    Loss: 1.572196
Train Epoch: 1 [3200/60000 (5%)]    Loss: 1.592174
…
Training continues, with the python code for this simple example reaching approx. 12% GPU utilization, and using 1250MiB of GPU memory. Clearly the P100 is excessive for this dataset, but we can see how SingularityCE’s OCI mode, and CDI support, makes it easy to run containerized GPU applications in an OCI compatible manner.

Development / Collaboration

The HPC Containers Advisory Meeting (README) is a forum for technical discussions across the wide containers and orchestration community, seeking to identify holes and opportunities and to drive technical solutions forward. CDI discussions started there back in September of 2020, with Renaud Gaubert. Evan Lezar significantly drove forward that work and presented it again in more detail there and to the Singularity Community in May of 2023.

Resources

Join Our Mailing List

Recent Posts

Related Posts