Maintaining Open Source Singularity Since 2017
Supporting Local Files in a Singularity Remote Build
By Adam Hughes
Overview
This long awaited feature allows copying local files into remote builds via the %files section of the Singularity def file the same way as you would if you were building locally with singularity build.
No special command-line arguments or other user actions are required: simply add a %files section to your DEF file to be used with remote build and any referenced files will be automatically copied to the image.
Remote build files support is included in SingularityCE v3.10 (and later), scs-build 0.7.4 (and later), and will be back-ported to SingularityPRO v3.9 for an upcoming release.
scs-build (Singularity Container Services)
scs-build is the go-to utility for integrating Singularity builds into a CI/CD pipeline, such as GitHub Actions, GitLab, Jenkins, CircleCI, and others. It is the same code used in SingularityCE and SingularityPRO to call out to a remote build service. There are currently two means to support the remote building of a SIF image, either in Sylabs Singularity Enterprise or in Sylabs Singularity Container Services (cloud.sylabs.io).
It is imperative that the Docker command-line arguments are properly formed to mimic the Singularity build from being invoked on the local system. This ensures the current local directory is mirrored in the scs-build container environment.
docker run \
--workdir `pwd` \
-e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
-v `pwd`:`pwd` \
-u `id -u` \
sylabsio/scs-build build \
--skip-verify \
`pwd`/<def file in current local directory> \
<library ref>This command-line invocation ensures file privileges are maintained within the build container environment through the use of the -u argument.
Depending on the workflow, the DEF file might be in another directory. If that’s the case, it is necessary to (also) map that directory into the environment (using the docker run --volume argument) and reference it in place.
Singularity
No special command-line arguments or actions are required to perform a remote build using a DEF file containing a %files section. Invoke singularity build --remote as normal:
singularity build --remote <library ref> <def file spec>If the DEF file specified contains a %files section, the build context is automatically created and managed by the build operation.
Availability
Support for remote build files is enabled now on Sylabs Cloud (https://cloud.sylabs.io) and will be available in the next release of Singularity Enterprise, using a compatible Singularity runtime (see Overview)
Support
If you happen to experience any issues with the remote build files support, in either scs-build or singularity CLI please contact us via the community Slack channel (https://singularityce.slack.com) in the #sylabs-cloud channel.
Join Our Mailing List
Recent Posts
Related Posts
Laying the Groundwork for 2025: Sylabs’ Vision for Secure Multi-Cloud Solutions
With a legacy rooted in innovation through the work we’ve done with the Singularity platform, Sylabs has consistently worked to provide secure container technology to empower researchers, enterprises, and innovators worldwide. These efforts have paid dividends as we...
Creating New Levels of Security with Containers and Confidential Computing
Data breaches are becoming increasingly common. It's more important than ever to find robust solutions that protect sensitive information. Recent high-profile data breaches underscore the necessity for fresh approaches to data protection. Breaches like these are...
OCI-SIF Container Images: Unraveling Their Features and Benefits
Among the container industry, OCI (Open Container Initiative) and SIF (Singularity Image Format) stand out as two prominent formats, each offering unique features and benefits. Understanding their distinct characteristics is crucial for HPC developers seeking optimal...