Response to CVE-2023-30549
The Apptainer project has recently published CVE-2023-30549. The associated
Signing Singularity Containers to Ensure Trust and Reproducibility
By Staff
Ensuring Long-term Trust and Reliability of SIF Images: What You Need to Know
As container runtimes technologies become more prevalent, it presents a dilemma for government, public, and private sector organizations. They must consider the long-term implications of using the same containers and data for future experiments, particularly in the realm of AI and ML. The concern is that aging containers may introduce significant security risks over the container’s lifetime. How can we balance the benefits of containerization with the need for ongoing security and risk management?With long-term signature verification growing increasingly vital to container image management, the Sylabs team recognizes the importance of addressing ongoing concerns around this issue and how the popular Singularity Image Format (SIF) format can help this problem. In this post, we shed light on the challenges associated with long-term signature verification of SIF images and emphasize the need for a robust solution. Additionally, we provide a brief overview of the available options for using X.509 certificates during the planned lifespan for a container image.
A federal push for improving software security
The 2021 Executive Order on Improving the Nation’s Cybersecurity brought the issue of trustworthiness of and transparency into digital infrastructure to the forefront of federal priorities. In the order, the administration outlines a set of standards and requirements designed to ensure that federal agencies are taking essential steps to “…identify, deter, protect against, detect, and respond to…” malicious cyber campaigns that threaten national security.Section 4(a) of the order, which focuses on enhancing software supply chain security, is particularly relevant to long-term considerations for containers. It states:
The security of software used by the Federal Government is vital to the Federal Government’s ability to perform its critical functions. The development of commercial software often lacks transparency, sufficient focus on the ability of the software to resist attack, and adequate controls to prevent tampering by malicious actors. There is a pressing need to implement more rigorous and predictable mechanisms for ensuring that products function securely, and as intended. The security and integrity of “critical software” — software that performs functions critical to trust (such as affording or requiring elevated system privileges or direct access to networking and computing resources) — is a particular concern. Accordingly, the Federal Government must take action to rapidly improve the security and integrity of the software supply chain, with a priority on addressing critical software.
You don’t have to look further than the SolarWinds hack or the security flaw in the ubiquitous Log4j framework to understand why this has taken on an increased urgency. Of course, verifying the authenticity and integrity of any software currently running on a system using X.509 certificates is a straightforward process. But digital signatures in container images that haven’t been used in many years introduce management challenges since digital signatures used to sign software and/or data containers may expire before a container’s usable lifetime. And there are many areas where long-term signing of containers is an important consideration.
It’s not just in federal agencies and departments, such as the DoD which is developing a new virtual data fabric for cyber testing, it’s also an issue across the private sector and academic and research organizations. Essentially, any organization doing repeatable science should be thinking about long-term signature verification of SIF images. That spans materials science, oil and gas discovery, chip design, shipbuilding, and many more industries. And it relates to ever-increasing numbers of scenarios and use cases from Kubernetes workflows to growing numbers of AI and ML users in HPC labs.
Putting the security considerations and dangers into perspective
The dangers of ignoring digital signatures in containers are clearcut; without signatures, it’s impossible to know whether a given container is authentic and what’s in it. This includes determining if the software is up to date with the latest patches. It’s also difficult to detect if someone modifies the container or adds a backdoor or some other exploit without your knowledge. The challenges and dangers of containers with expired signatures are another matter.Take the example of an academic institution or public research organization that needs to run a new experiment using a genomics container with long-expired expired signatures. If the organization has lost track of the key, they are simply out of luck and may not be able to reproduce the experiment using the same tools. And while it is possible for the organization to run the container if users still have all the key materials, the fact that the container has not been updated over time virtually guarantees that it includes a host of CVEs and opportunities for attackers.
Fortunately, it’s possible to avoid all these scenarios through several strategies for signature verification of SIF images when using X.509 certificates.
Long-term signature verification in SIF images
While signing a SIF image is a relatively straightforward process, signature expiration dates and other variables do complicate long-term container usage considerations. One of the biggest issues is that over time malicious actors or third parties may gain access to the private key material used to protect a given environment, so best practices are to either limit the lifetime of private keys or ensure there is a way to revoke key material in the event of a compromise. For long-lived SIF containers, the question of how to manage digital signatures if you choose to periodically rotate the private key materials is critically important. For the moment, Sylabs has identified the following three strategies as potential options.- Periodically re-issue certificates—With this option, the signing party uses long-lived private key material and periodically re-issues certificates that all contain public key material. In this model, the signing party does not periodically rotate its private key material except in the event of a compromise. Updated certificates are periodically published and must be distributed to all parties wishing to verify a container.
- Periodically resign containers—In this model, the signing party periodically rotates the private key it uses to sign containers. The new signature and the new public key material must then be distributed to all parties wishing to verify the container.
- “Keyless” certificates—This option is radically different from traditional PKI approaches and was pioneered by the SigStore project (Sigstore Security Model, 2023). In this model, the signing party authenticates with a zero-trust identity provider and uses ephemeral keys and certificates, which are signed automatically by a special root CA. Signatures and certificates are stored in a transparency log, ensuring they are globally visible, discoverable, and auditable. Certificates in this model are extremely short-lived, often on the order of minutes.
Learn more
For a deeper dive into each of the options, read 3 Strategies for Long-Term Signature Verification of SIF Images When Using X.509 Certificates: Examining contrasting methods for long-term validation of digital signatures in Singularity Image Format (SIF) containers. The white paper explores the following topics:- An overview of SIF and a brief history of certificate-related considerations in SIF containers
- High-level descriptions of asymmetric encryption and digital signatures to enhance the clarity of the verification options for readers with limited familiarity of these concepts
- Three options for supporting the long-term verification of digital signatures in SIF containers using X.509 certificates
- The pros and cons of each approach, based on environment types and other factors
Join Our Mailing List
Related Posts
read more
An Introduction to Singularity Containers
Enabling Portable and Secure Computing Environments for High-Performance Workloads.As part of their ongoing efforts to streamline workflows, enhance productivity, and save time, engineers, and developers in enterprises and high performance computing (HPC) focused...
Our Most Productive Year Ever? 5 Ways Sylabs Strengthened its Container Leadership in 2022.
What a difference a year can make; after a couple consecutive years of pandemic-related uncertainties, the world has finally turned the page. And as we kick off 2023, the team at Sylabs couldn’t be prouder of all the progress we made with everything from new offerings...