Sylabs Security Vulnerability Protocol
A lot rides on Singularity.
Some of the biggest HPC centers on the planet depend on Singularity containers every day to run mission-critical applications. As stewards of the open source Singularity project, we owe it to the community to establish a clear security procedure for the times when vulnerabilities are reported or discovered.
How to responsibly report a vulnerability
It’s easy to assume that your software is secure, but at Sylabs, we continuously test the current open source version of Singularity for vulnerabilities. If you discover a new vulnerability, reporting it responsibly is easy as sending a message to security@sylabs.io to tell us what you’ve found. Several members of the Sylabs team will follow up with you and work with you to build and test a patch.
The goals of a standardized security response procedure
Before we cover the procedure that Sylabs takes to mitigate newly discovered vulnerabilities, let’s talk about the goals we’re trying to achieve.
- Fast turnaround: Even vulnerabilities that have not been announced publicly are a potential source of danger because a savvy hacker may be able to discover and exploit them independently. Our procedures are designed to first understand and then quickly remediate as primary goals.
- Limited exposure: Before patches are developed and made available, our goal is to limit the spread of information until after a patch is available and key stakeholders are protected.
- Transparency: The open-source community must know exactly what Sylabs intends to do about vulnerabilities and how we are are carrying out our commitment to security. Vulnerabilities are documented using the Common Vulnerabilities and Exposures (CVE) system to provide a permanent searchable record allowing administrators to accurately judge the risks of running a particular version of Singularity within their environment.
- Enable stakeholders over malicious actors: When a new vulnerability is publicly announced, a race begins between system administrators and those with nefarious intent. SingularityPRO subscribers are provided security patches prior to security announcements as a crucial head start in the security race. Although patches are made available to SingularityPRO customers first, those patches are provided without releasing security-related information. This limits exposure to the open source community while still providing a way to remediation for SingularityPRO customers, with a level of proactive measure.
Sylabs vulnerability procedure
When a vulnerability is discovered, Sylabs takes the following steps:
- Perform due diligence to fully replicate and describe the scope and severity of the bug. (This step is expected to take hours, not days.)
- A CVE number is requested and embargoed until public release is made.
- Security patch(es) are confidentially developed. (This step is expected to take hours or days and will be carried out with appropriate urgency.)
- Security patches are merged into test versions of SingularityPRO and Sylabs’ testing commences. Bugs related to patch(es) are fixed and testing is repeated as necessary. (This process is expected to take days.)
- Once patch(es) are developed and fully tested, they are pushed to the main Singularity Pro repos.
- SingularityPRO customers will receive a standard notification that there is a new version of Singularity available and they should upgrade. This notice will NOT contain any sensitive information and will NOT disclose the presence of a security-related patch.
- Singularity Pro customers are given a reasonable amount of time to upgrade their installations so that when details of the exploit are revealed they are already protected.
- After a reasonable period of time has elapsed and SingularityPRO customers have likely upgraded (and on a Tuesday where possible as several administrators have suggested https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/FgHj7WhqIE8), the patches will be merged from the private development space into the public repository and a release will immediately be made. The release notes will do the following:
- Describe the issue in sufficient detail so that affected parties can judge whether to upgrade.
- If there is a mitigation or workaround detail it. If there is not explicitly say there is no known workaround.
- State whether a malicious user needs access to the system to exploit the vulnerability or whether it can be exploited remotely.
- State which versions of Singularity are affected and which OS-es/kernels are affected.
- Reference relevant CVE number(s).
- At the same time that a release is being made, the CVE(s) will be filled out with all relevant information and released from embargo.
- Announcements will be made on Slack and Google Group that a new version of Singularity is available with all relevant security information and links to release notes.
- SingularityPRO customers will receive follow-up notification that the last version contains security patches and they should double check that they have upgraded. This email should contain the same information that is in the release notes (along with notifications in Sylabs customer support portal.)
Continual Improvements
This security procedure is intended as a guideline and not a one-size fits all mandate. Situations will undoubtedly arise where these steps won’t apply cleanly. But the collective experience of the open source community and the Sylabs team, as well as our recent patching efforts lead us to believe this is a well-conceived plan.
Does the procedure make sense? Are we missing something? Tell us what you think!
