Maintaining Open Source Singularity Since 2017

SCS Login

Security Policy

Focus on Security

A lot rides on Singularity.

If you are an open source user, developer or Sylabs™ customer, we take security very seriously, our goal is to provide all Singularity users with the best-possible assistance regarding any issue that might affect the security of their systems.

Some of the biggest HPC centers on the planet depend on Singularity containers every day to run mission-critical applications. As stewards of the open source Singularity project, we owe it to the community to establish a clear security procedure for the times when vulnerabilities are reported or discovered.

Do you think you found a vulnerability?

It’s easy to assume that your software is secure, but at Sylabs™, we continuously test the current open source version of Singularity for vulnerabilities.

If you believe you have discovered a vulnerability in Singularity, let us know so we can research it! You can notify our team either:

We encourage people who contact Singularity Security team to use email encryption. Get our PGP public key and verify the fingerprint:

0x0EEDA03B2CF74611AE99A09FB20868209CA9540F

The goals of a standardized security response procedure

Before we cover the procedure that Sylabs™ takes to mitigate newly discovered vulnerabilities, let’s talk about the goals we’re trying to achieve.

Fast turnaround: Even vulnerabilities that have not been announced publicly are a potential source of danger because a savvy hacker may be able to discover and exploit them independently. Our procedures are designed to first understand and then quickly remediate as primary goals.

Limited exposure: Before patches are developed and made available, our goal is to limit the spread of information until after a patch is available and key stakeholders are protected.

Transparency: The open-source community must know exactly what Sylabs™ intends to do about vulnerabilities and how we are are carrying out our commitment to security. Vulnerabilities are documented using the Common Vulnerabilities and Exposures (CVE) system to provide a permanent searchable record allowing administrators to accurately judge the risks of running a particular version of Singularity within their environment.

Enable stakeholders over malicious actors: When a new vulnerability is publicly announced, a race begins between system administrators and those with nefarious intent. SingularityPRO subscribers are provided security patches prior to security announcements as a crucial head start in the security race. Although patches are made available to SingularityPRO customers first, those patches are provided without releasing security-related information. This limits exposure to the open source community while still providing a way to remediation for SingularityPRO customers, with a level of proactive measure.

Sylabs™ vulnerability procedure

When a vulnerability is discovered, Sylabs™ takes the following steps:

  1. Perform due diligence to fully replicate and describe the scope and severity of the bug. (This step is expected to take hours, not days.)
  2. A CVE number is requested and embargoed until public release is made.
  3. Security patch(es) are confidentially developed. (This step is expected to take hours or days and will be carried out with appropriate urgency.)
  4. Security patches are merged into test versions of SingularityPRO and Sylabs™’ testing commences. Bugs related to patch(es) are fixed and testing is repeated as necessary. (This process is expected to take days.)
  5. Once patch(es) are developed and fully tested, they are pushed to the main Singularity Pro repos.
  6. SingularityPRO customers will receive a standard notification that there is a new version of Singularity available and they should upgrade. This notice will NOT contain any sensitive information and will NOT disclose the presence of a security-related patch.
  7. Singularity Pro customers are given a reasonable amount of time to upgrade their installations so that when details of the exploit are revealed they are already protected.
  8. After a reasonable period of time has elapsed and SingularityPRO customers have likely upgraded, the patches will be merged from the private development space into the public repository and a release will immediately be made. The release notes will do the following:

    1. Describe the issue in sufficient detail so that affected parties can judge whether to upgrade.
    2. If there is a mitigation or workaround detail it. If there is not explicitly say there is no known workaround.
    3. State whether a malicious user needs access to the system to exploit the vulnerability or whether it can be exploited remotely.
    4. State which versions of Singularity are affected and which OS-es/kernels are affected.
    5. Reference relevant CVE number(s).
  9. At the same time that a release is being made, the CVE(s) will be filled out with all relevant information and released from embargo.
  10. Announcements will be made on Slack and Google Group that a new version of Singularity is available with all relevant security information and links to release notes.
  11. SingularityPRO customers will receive follow-up notification that the last version contains security patches and they should double check that they have upgraded. This email should contain the same information that is in the release notes (along with notifications in Sylabs™ customer support portal.)

Note: All security fixes that Sylabs™ creates will also be made available to the open source community when the full disclosure/CVE have been made public. This will include any patches that are created for legacy versions of Singularity that are no longer being actively maintained by the community.

Continual Improvements

This security procedure is intended as a guideline and not a one-size fits all mandate. Situations will undoubtedly arise where these steps won’t apply cleanly. But the collective experience of the open source community and the Sylabs™ team, as well as our recent patching efforts lead us to believe this is a well-conceived plan.

Does the procedure make sense? Are we missing something? Tell us what you think!