Singularity Enterprise 2.3 release incorporates consolidated OCI registry, helping to streamline container tool and workflow requirements
This blog post:
- Revisits the history of containers and why increased compatibility is invaluable to end users.
- Highlights enhancements in the Singularity Enterprise 2.3 release along with their benefits.
- Looks forward at how convergence is poised to help with various industry challenges along with what’s next in the evolution of Singularity.
A brief history of container evolutionTracing the history of computing containers back to the early 2000s, technologies such as FreeBSD Jails and Solaris Zones emerged with the aim of providing lightweight virtualization at the operating system level. However, it wasn’t until 2008, with the introduction of Docker by Solomon Hykes, that containers gained widespread attention. Docker revolutionized application packaging, equipping developers with a tool to create lightweight, portable, and isolated environments that could run consistently across different machines. This breakthrough triggered a containerization wave, leading to the emergence of other container technologies like Singularity, Kubernetes, and various container orchestration platforms.
Embracing OCI to increase compatibility and flexibilityThe adoption of OCI in open source Singularity started in 2016 when the community recognized its potential for removing barriers to container adoption and workflows in HPC environments. Now, with the inclusion of OCI compatibility in Singularity Enterprise, Sylabs is providing a way to seamlessly bring together workflows for every kind of HPC workload, including AI, ML, and quantum.
A closer look at OCI-associated enhancementsOCI support in Singularity Enterprise provides researchers, scientists, and developer organizations access to an expansive range of container images and key management technologies. The fusion of Singularity Image Format (SIF) and OCI workflows enables the Singularity Enterprise Library to function as a universal language, fostering interoperability between container platforms and creating new opportunities for innovation and collaboration. Key enhancements and features in this release include:
- Ability to store artifacts from a diverse range of sources alongside SIF images in the Cloud Library.
- A consolidated registry for improved management and streamlined workflows involving Singularity SIF and Docker OCI containers.
- The ability to consume shared OCI Images , giving users the option to leverage other people’s images without having to pull and convert them, saving a step in the workflow.
Diverse artifact storageIn Singularity Enterprise 2.3, users are empowered to store a diverse range of artifacts, not just limited to SIF images, in the Container library. This includes Docker or Podman images, Helm charts, operating system packages such as RPMs for systems like RedHat Linux, Debian package files for systems like Ubuntu, source code, binary programs, Microsoft Office documents, PDFs, and more.
Improved management and securityIn common HPC environments, Singularity users often base their SIF images on existing Docker container images. With Singularity Enterprise 2.3, users can now manage Docker and SIF workflows in a single tool and store all artifacts in a central registry. This simplification not only streamlines workflows but also mitigates concerns over image-pull-limits in Docker for both registered and unregistered users. It also eliminates the need for duplicate server resources and reduces dependency on a specialist or service provider for maintaining a separate Docker registry.
Consuming OCI images from definition filesFrom a CI/CD workflow perspective, OCI integration in Singularity Enterprise 2.3 offers both time savings and optimized storage. Users can leverage the ‘scs-build’ binary, which is designed to trigger remote builds in Enterprise or in Singularity Container Services (SCS). Instead of pulling Docker images from Docker Hub, the ‘scs-build command enables users to reference remote images directly from OCI registries, including Docker Hub and the Singularity Enterprise registry. They can also store all images in the Singularity Enterprise Library, eliminating the need to push them back to Docker Hub. The use of ‘scs-build’ to pull images from local registries can result in significant time savings, particularly for larger files.
Key benefits of OCI adoptionOverall, relying on a container registry that supports both OCI- and SIF-based containers unlocks several advantages, including:
- Enhanced compatibility and flexibility: By supporting both OCI and SIF, we ensure compatibility with a wide range of container images. OCI is a widely adopted standard for container images and runtimes, while SIF is primarily used in the HPC community. By supporting both formats, Sylabs accommodates different use cases to cater to diverse user requirements while merging industry standards and scheduler batch computing workflows.
- Expanded user base: The ability to handle OCI and SIF broadens your potential user base. Accommodating both formats accommodates a wider array of users, including researchers, data scientists, and engineers across diverse disciplines.
- Future-readiness: The container ecosystem is continuously evolving, his level of adaptability keeps you ahead of the curve ready and ready to meet changing user needs more easily.
More updates and innovations to comeNow that Singularity Enterprise 2.3 is available, keep an eye out for a series of technical blog posts that will cover everything from hybrid Docker and Singularity workflows to using common tools in the integrated Harbor registry and more.
Join Our Mailing List
Introduction The Acorn RISC Machine, or ARM as it is now known, and later processor architecture has become a cornerstone in the landscape of Information Technology (IT) and modern computing. Initially designed for Acorn Computers in the 1980s, ARM processors have...
Performance portability should be a major concern to those who wish to effectively implement AI and MLIn the ever-evolving landscape of artificial intelligence (AI) and machine learning (ML), performance portability has emerged as a critical factor influencing the...
Overview With CentOS 7 reaching end of life on June 30th, 2024 and CentOS 8 already discontinued in favor of CentOS Stream, users of open source SingularityCE might find themselves in a situation where a migration to another open source operating system is necessary....