My name is Adam Hughes, and I’m here to provide some commentary to go along with the three Singularity vulnerabilities Sylabs publicly disclosed on July 14, 2020. My aim here isn’t to give you all of the nitty gritty details of the vulnerabilities, as those are captured in the CVEs and GitHub Security Advisories. Rather, I would like to give a glimpse into all of the human elements involved in discovering, assessing, and ultimately remediating software vulnerabilities. Some would prefer to hide vulnerabilities in the shadows. Personally, I’ve always felt that responsible and transparent disclosure leads to more informed developers and users, and ultimately to more secure software.
We’ve had some big changes at Sylabs in the past few months. Jason Tuschen, a retired Navy SEAL Command Master Chief took the helm as CEO, and Tony Gaughan began acting as CTO. I mention these individuals not because they were personally involved in uncovering flaws in Singularity, but rather because they were instrumental in creating a culture at Sylabs that allows the engineering team to prioritize critical work such as this. Early on in their tenure, they asked us to consider our core values, our strengths and weaknesses, what was important, and what was extraneous. You might think having a former SEAL at the helm would make for a company that does burpees at regular intervals and follows orders well. Instead, I’ve participated in a period of reflection and creativity that allowed us to chart our path forward as a team. We figure out what’s worth doing, and do it well. Burpees were at one point suggested, but to my knowledge never mandated.
Early on in this new environment, we realized that our engineering team was strong in two areas: high-performance computing, and computer security. In many ways, our work at Sylabs has always reflected this. Singularity grew up in HPC. Less obvious perhaps is the focus on security, though you don’t have to look too far to find that. The Singularity Image Format (SIF) was developed at Sylabs, and brought a different take on how digital signatures and encryption can be applied to software containerization. On the commercial side, SingularityPRO has a focus on back-porting fixes to keep our customers secure. Our Singularity Enterprise stack allows unprivileged users to perform container builds that would normally require elevated privilege in a safe way. In retrospect, we’ve always had a bit of a focus on security at Sylabs. Today, that focus is a conscious one, and one we choose to lean into as a team.
It was during our period of change that Tru Huynh, a member of the open source Singularity community, messaged on Slack that he’d found some interesting behaviour related to digital signatures. The sign/verify commands, when used in conjunction with a SIF container image that contained an overlay partition, would successfully verify regardless of the contents of the overlay. Effectively, this meant that a user could modify the contents of the root filesystem without causing the signature to fail to verify. Clearly, something was wrong here. In accordance with the Sylabs Security Policy, our engineering team immediately went to work to assess the scope and severity of the problem.
On the surface of it there were two bugs, and they looked simple enough:
--all option to verify returned a success code, despite displaying warnings about the missing signatures.I ended up taking the lead on the remediation work. I’d been looking for an excuse to get more involved in Singularity development, and anything related to security generally piques my interest. So the assessment began. I didn’t get too far in the investigation before it became clear that there was much more to fix than I’d expected.
For a bit of background, every SIF image is made up of three distinct sections:
SIF signatures are themselves data objects, and can be used to verify the integrity of a single data object, or a group of data objects. The idea is that once someone signs an image, anyone can verify the image with a publicly available key. If the image is tampered with, the signature won’t match, and verification will fail.
After a couple of hours getting familiar with the signing and verification code in Singularity, it became obvious that Singularity was signing the data objects only, ignoring the global header and data object descriptors completely. Effectively, this meant that no metadata within the image was integrity-checked. This means that all sorts of things can change without being detected by the Singularity verify command:
rm -fr $HOME).From a developer’s perspective, this is a difficult scenario, and not one that is possible to fix without breaking changes. Security fixes are developed under embargo, with limited chance for feedback from a small number of insiders. They are not afforded the usual release candidate exposure prior to making their way to a general release, and as a result, you generally want to keep the changes as small as possible. Introducing a new signature format was obviously not going to be a small change.
Around the time of this realization, I happened to be forced to self-isolate (thanks, COVID-19!). In retrospect, being stuck alone in an AirBNB for fourteen days with a tricky bit of code to write was pretty good timing. It wasn’t much fun being stuck indoors, but it certainly did give me plenty of time to focus on the code. Around the time I was permitted to be on my way, I had polished up most of what would become the SIF integrity package, as well as the related changes to Singularity.
The new signature format covers not only data objects, but integrity-critical fields of the global header and data descriptors as well. It is expressive enough to do all of this within a single signature, which helps mitigate attacks that take advantage of mixing data objects that weren’t intended to be mixed by the original author.
More information regarding the three vulnerabilities can be found at:
All of these vulnerabilities have been fixed in Singularity version 3.6, and all open-source users are encouraged to upgrade immediately. The fixes have also been applied to SingularityPRO 3.1-5 and SingularityPRO 3.5-2, and are available for immediate download to all current SingularityPRO customers.
Finally, I’d like to give a few shout-outs to people who helped throughout this work. Tru Huynh provided the thread of the sweater for the Sylabs team to pull on. David Trudgian was always quick to provide meaningful feedback, did the hard work of back-porting the new signature format for SingularityPRO customers, and the massive task of delivering the Singularity 3.6 release. Ian Kaneshiro and Cedric Clerget helped review the code prior to its release, and were extremely professional as always.
Finding and fixing vulnerabilities is always bittersweet. On the one hand, we all wish software was free of them from the get go. On the other hand, vulnerabilities are a reminder that there’s always room for improvement. They are a constant source of humility. Just like those burpees, I suspect.
Ensuring Long-term Trust and Reliability of SIF Images: What You Need to Know As container runtimes technologies become more prevalent, it presents a dilemma for government, public, and private sector organizations. They must consider the long-term implications of...
The Apptainer project has recently published CVE-2023-30549. The associated
Enabling Portable and Secure Computing Environments for High-Performance Workloads.As part of their ongoing efforts to streamline workflows, enhance productivity, and save time, engineers, and developers in enterprises and high performance computing (HPC) focused...