Now that we have readied the WSL2 environment with Singularity and the relevant CUDA libraries, it’s time to run the sample Keras workflow.
Sylabs Remote Build Service
At Sylabs, our mission is to drive development of the open source Singularity project and expand the surrounding ecosystem positioning Singularity as the de-facto choice for HPC and Enterprise Performance Computing (EPC) workflows. .
Today, we’re happy to share with you a preview of one of the products we have been working on: the Sylabs Remote Build Service, a cloud-based service (It’s not the most creative name, I know, but as a team of software developers, you’ll have to forgive us for our lack of marketing prowess!)
Why Build Remotely?
Singularity enables many different workflows by giving you the opportunity to use compute resources where privileges are limited. But if we step back and think about the process of building an image, elevated privileges are sometimes necessary. Specifically, if you want to build an image from a recipe or definition file, you need root access.
Right now, Singularity users need a Linux workstation or Virtual Machine where they have elevated privileges to build and test their container images. In many HPC and EPC environments, however, there are a couple of potential issues with this approach:
- Users may not have access to Linux workstations. It’s possible to work around this limitation using virtual machines or cloud computing, but some sites prohibit this for technical and/or policy reasons.
- Some facilities have policies that do not allow elevated privileges to be granted to non-admin users, which limits the types of Singularity images users can build.
Sylabs Remote Build Service addresses these challenges by removing the need for users to have elevated privileges, and potentially allowing users to build Singularity containers on non-Linux workstations.
Singularity 3.0 development is continuing at a fast pace, and new and exciting use cases are already popping up. For instance, using the the Remote Build System, it should be possible to build Singularity images using a Mac or Windows workstation; please stay tuned!
One of the goals of the Remote Build Service is for it to integrate seamlessly with Singularity so that current Singularity users don’t have to modify their workflow. In fact, with the addition of a single flag on the command line, the build can be done remotely:
$ singularity build --remote test.sif test.def
This causes Singularity to make a request to the Remote Build Service, which completes the build remotely. During the build process, output of build is streamed back to the console, so that the user can monitor its progress. Assuming the build completes successfully, the built SIF image file is transferred back to the user’s workstation, from which point it can be executed with Singularity.
Security as a Discipline
Singularity and Sylabs have been focussed on security from day one. The astute Singularity user will note that building images within a managed service still requires elevated privileges. It simply shifts the location where elevated privileges are utilized.
This shift allows the Remote Build Service to implement appropriate levels of isolation between the components performing the builds with elevated privileges, and the rest of the infrastructure. As a system administrator, you get a turn-key solution that empowers users to build Singularity images, providing you with centralized auditing and monitoring of Singularity builds occuring on-site.
Do you have a use case for the Sylabs Remote Build Service? Interested in learning more? We’d love to hear from you for topics to cover in future Lab Notes, or joining community discussions.
Join Our Mailing List
Signing the Container The Singularity 3.0 family introduced the ability to create (and manage) PGP keys to sign and verify containers. This provides a trusted method for Singularity users to share containers and ensures a bit-for-bit reproduction of the original...
Create an Account & Authentication Token Now that we have SingularityCE installed in WSL2, and NVIDIA GPU support is enabled, we will create a Singularity Container Services account and configure the local Singularity client, followed by building a remote...