Maintaining Open Source Singularity Since 2017

SCS Login

Singularity Enterprise 2.3: Streamlining OCI Workflows

By Adam Hughes

Jul 12, 2023 | Blog

Singularity Enterprise 2.3 release incorporates consolidated OCI registry, helping to streamline container tool and workflow requirements

At Sylabs, we are passionate about streamlining technologies for containerized workflows. Our passion is driven by the community of engineers, scientists, and researchers that make up disciplines within High Performance Computing. In general, the feedback we hear from these teams is that they need help making computing technologies more accessible by simplifying the tooling needed to solve humanity’s greatest challenges.
That’s why we are thrilled to announce the Singularity Enterprise 2.3 release! This exciting update incorporates an Open Container Initiative (OCI) registry, fostering interoperability between major container platforms like Docker and Podman. With this integration, we’re making it easier than ever to leverage the power of containers to simplify complex tasks. Join us on this journey of simplified computing and unlock new possibilities with Singularity Enterprise 2.3.

This blog post:

  • Revisits the history of containers and why increased compatibility is invaluable to end users.
  • Highlights enhancements in the Singularity Enterprise 2.3 release along with their benefits.
  • Looks forward at how convergence is poised to help with various industry challenges along with what’s next in the evolution of Singularity.

A brief history of container evolution

Tracing the history of computing containers back to the early 2000s, technologies such as FreeBSD Jails and Solaris Zones emerged with the aim of providing lightweight virtualization at the operating system level. However, it wasn’t until 2008, with the introduction of Docker by Solomon Hykes, that containers gained widespread attention. Docker revolutionized application packaging, equipping developers with a tool to create lightweight, portable, and isolated environments that could run consistently across different machines. This breakthrough triggered a containerization wave, leading to the emergence of other container technologies like Singularity, Kubernetes, and various container orchestration platforms.
Today, containers are a fundamental building block of modern application development and deployment. They bring to the table the agility, scalability, and reproducibility needed to navigate and thrive in the ever-shifting computing landscape. However, like everything in computing, they need to continually evolve to keep pace with rapidly changing application and infrastructure needs. One significant challenge that persists in the world of containers and contributes to workflow complexity and management overhead is the lack of compatibility between different formats. This problem is often particularly pronounced in high performance computing (HPC) environments where various user groups depend on different container tools that cater to their specific needs. And the surge of use cases like artificial intelligence (AI), machine learning (ML), and quantum computing is only adding to the challenges.

Embracing OCI to increase compatibility and flexibility

The adoption of OCI in open source Singularity started in 2016 when the community recognized its potential for removing barriers to container adoption and workflows in HPC environments. Now, with the inclusion of OCI compatibility in Singularity Enterprise, Sylabs is providing a way to seamlessly bring together workflows for every kind of HPC workload, including AI, ML, and quantum.
In the dynamic world of container technologies, container registries have emerged as a vital hub for collaboration. In Singularity Enterprise, the Library (registry) plays a key role in bridging the divide between Docker and its OCI containers, with Singularity and its SIF based containers. The inclusion of an OCI registry in Singularity Enterprise, which includes our enterprise-grade SingularityPRO runtime, takes the value of compatibility and adaptability in containers to another level, providing new features that better support diverse user needs across a wide range of use cases.

A closer look at OCI-associated enhancements

OCI support in Singularity Enterprise provides researchers, scientists, and developer organizations access to an expansive range of container images and key management technologies. The fusion of Singularity Image Format (SIF) and OCI workflows enables the Singularity Enterprise Library to function as a universal language, fostering interoperability between container platforms and creating new opportunities for innovation and collaboration. Key enhancements and features in this release include:
  • Ability to store artifacts from a diverse range of sources alongside SIF images in the Cloud Library.
  • A consolidated registry for improved management and streamlined workflows involving Singularity SIF and Docker OCI containers.
  • The ability to consume shared OCI Images , giving users the option to leverage other people’s images without having to pull and convert them, saving a step in the workflow.
Let’s take a closer look at each enhancement.

Diverse artifact storage

In Singularity Enterprise 2.3, users are empowered to store a diverse range of artifacts, not just limited to SIF images, in the Container library. This includes Docker or Podman images, Helm charts, operating system packages such as RPMs for systems like RedHat Linux, Debian package files for systems like Ubuntu, source code, binary programs, Microsoft Office documents, PDFs, and more.

Improved management and security

In common HPC environments, Singularity users often base their SIF images on existing Docker container images. With Singularity Enterprise 2.3, users can now manage Docker and SIF workflows in a single tool and store all artifacts in a central registry. This simplification not only streamlines workflows but also mitigates concerns over image-pull-limits in Docker for both registered and unregistered users. It also eliminates the need for duplicate server resources and reduces dependency on a specialist or service provider for maintaining a separate Docker registry.
This unified workflow approach enhances efficiency and security. For instance, users no longer need to log into external registries, saving time and reducing the steps in the process. If the enterprise instance is within the same network, there is no need to transport large files over the internet, eliminating security concerns related to pulling images from external sources.

Consuming OCI images from definition files

From a CI/CD workflow perspective, OCI integration in Singularity Enterprise 2.3 offers both time savings and optimized storage. Users can leverage the ‘scs-build’ binary, which is designed to trigger remote builds in Enterprise or in Singularity Container Services (SCS). Instead of pulling Docker images from Docker Hub, the ‘scs-build command enables users to reference remote images directly from OCI registries, including Docker Hub and the Singularity Enterprise registry. They can also store all images in the Singularity Enterprise Library, eliminating the need to push them back to Docker Hub. The use of ‘scs-build’ to pull images from local registries can result in significant time savings, particularly for larger files.

Key benefits of OCI adoption

Overall, relying on a container registry that supports both OCI- and SIF-based containers unlocks several advantages, including:
  • Enhanced compatibility and flexibility: By supporting both OCI and SIF, we ensure compatibility with a wide range of container images. OCI is a widely adopted standard for container images and runtimes, while SIF is primarily used in the HPC community. By supporting both formats, Sylabs accommodates different use cases to cater to diverse user requirements while merging industry standards and scheduler batch computing workflows.
  • Expanded user base: The ability to handle OCI and SIF broadens your potential user base. Accommodating both formats accommodates a wider array of users, including researchers, data scientists, and engineers across diverse disciplines.
  • Future-readiness: The container ecosystem is continuously evolving, his level of adaptability keeps you ahead of the curve ready and ready to meet changing user needs more easily.
Beyond direct benefits to the enterprise, the adoption of OCI can arguably help address larger challenges within the industry, particularly in promoting FAIR (Findable, Accessible, Interoperable, and Reusable)-compliant computation and data management practices in data science, AI, and HPC. By facilitating the seamless sharing, discovery, and utilization of diverse computational tools and data, OCI adoption streamlines workflows within the enterprise and reinforces the broader commitment to FAIR principles. This enhancement is particularly beneficial for researchers and developers who are increasingly reliant on accessible and interoperable data to drive innovation and progress.

More updates and innovations to come

Now that Singularity Enterprise 2.3 is available, keep an eye out for a series of technical blog posts that will cover everything from hybrid Docker and Singularity workflows to using common tools in the integrated Harbor registry and more.
Moving forward, Sylabs will continue to improve compatibility between Singularity and other key tools, including Docker. Now that Singularity can support OCI images, a key milestone on our roadmap is to enable users to run original Docker images using Singularity.

Join Our Mailing List

Recent Posts

Related Posts