SIF — Containing Your Containers

One of Singularity’s design concepts in architecting a container solution is to  store a container as a single run-time file. This decision makes a container, which may consist of  several hundreds or thousands of files, easier to manage, and provides a conduit for some really interesting features, such as:  it is fast to access any segment, can easily be “classified” within a system that implements rigorous file access controls, and of course it has excellent mobility and reproducibility attributes whereby moving or copying a container means moving or copying one file.

With feature enhancements in continual development, and with the team’s desire to offer better overall security model to all Singularity users, it became clear that a container would soon be more than just one Operating System partition image in a file. We wanted to augment our containers with cryptographic signatures and cleverly store metadata outside of the OS partition, to  be reserved for runtime and application environments. This is how SIF, the new Singularity Image Format came to be!

Well what is it?!

SIF is a new file format tailored to container images. It allows for the storage of different parts that makes up a container. For example, our containers may include OS partitions images (read-only), user writable sections, recipes used to create the container, cryptographic signatures for data integrity and authenticity, and whatever else the community can think of… a SIF file resembles a general file system by its structure. A global header identifies the SIF file and holds information about what one can expect to find in the container file. Next to this is a list of data object descriptors that hold information for each region of data that a SIF file contains. The primary goal of the descriptors is to identify the kind of data stored and where to find it within the container file. Following those descriptors is the actual data for these descriptors namely the OS partition image, recipe, environment variables and signature blocks. The picture below demonstrate and example of what an actual SIF file would look like:

Even though SIF was designed and implemented for security features like signing, and the encryption of important data, many upcoming features were imagined after a working prototype was included in a development branch. The flexible and general format of SIF can be described as a container for containers. It’s easy to incorporate new data types inside a SIF file and access for reading and writing is direct. Here’s a few features that the Singularity developers would like to bring to our community and users; potentially as part of the 3.0 release planned for later this year :

• PGP signature for any data objects
• Block encryption for any data objects
• Support for multi-OS partitions and architecture autodetection for heterogeneous cluster systems and data centers
• Support for multi-container workflows via an array of ‘runscripts’
• Fast and direct access to metadata

Because security is central to our development, singularity developers decided not only to apply a security conscious approach at each step of software development, but also bring forth new security features and cryptography into Singularity. The immediately identifiable trait of Singularity being its use of a single file, combined with the new SIF makes cryptographic integration very powerful.

For example, suppose an OS partition image inlined as data object within SIF. A user can now generate a verification hash of this partition and sign it with his PGP key. The resulting signature added to SIF by Singularity is now part of the container. This configuration allows users to use this signed container without having to unarchive and tamper with container data. The execution of the container runtime happens from the SIF data object directly which have been verified if wanted, in the same workflow. With Singularity and SIF, it is now possible to use verified unaltered data as it’s been created and signed without sacrificing mobility and usage.

Well there you go, SIF described in a few paragraphs. We hope you like what we are thinking about and what we are doing. The community feedback is important to what Singularity is, and to what it can be. Please engage with us at via Slack, GitHub or Google Groups via  https://sylabs.io/support/