Supporting Local Files in a Singularity Remote Build

By Staff

Overview 

This long awaited feature allows copying local files into remote builds via the %files section of the Singularity def file the same way as you would if you were building locally with singularity build.

No special command-line arguments or other user actions are required: simply add a %files section to your DEF file to be used with remote build and any referenced files will be automatically copied to the image.

Remote build files support is included in SingularityCE v3.10 (and later), scs-build 0.7.4 (and later), and will be back-ported to SingularityPRO v3.9 for an upcoming release.

scs-build (Singularity Container Services)

scs-build is the go-to utility for integrating Singularity builds into a CI/CD pipeline, such as GitHub Actions, GitLab, Jenkins, CircleCI, and others. It is the same code used in SingularityCE and SingularityPRO to call out to a remote build service. There are currently two means to support the remote building of a SIF image, either in Sylabs Singularity Enterprise or in Sylabs Singularity Container Services (cloud.sylabs.io).

It is imperative that the Docker command-line arguments are properly formed to mimic the Singularity build from being invoked on the local system. This ensures the current local directory is mirrored in the scs-build container environment.

docker run \
	--workdir `pwd` \
	-e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
	-v `pwd`:`pwd` \
	-u `id -u` \
	sylabsio/scs-build build \
	--skip-verify \
	`pwd`/<def file in current local directory> \
	<library ref>

This command-line invocation ensures file privileges are maintained within the build container environment through the use of the -u argument.

Depending on the workflow, the DEF file might be in another directory. If that’s the case, it is necessary to (also) map that directory into the environment (using the docker run --volume argument) and reference it in place.

Singularity

No special command-line arguments or actions are required to perform a remote build using a DEF file containing a %files section. Invoke singularity build --remote as normal:

singularity build --remote <library ref> <def file spec>

If the DEF file specified contains a %files section, the build context is automatically created and managed by the build operation.

Availability

Support for remote build files is enabled now on Sylabs Cloud (https://cloud.sylabs.io) and will be available in the next release of Singularity Enterprise, using a compatible Singularity runtime (see Overview)

Support

If you happen to experience any issues with the remote build files support, in either scs-build or singularity CLI  please contact us via the community Slack channel (https://singularityce.slack.com) in the #sylabs-cloud channel.

Related Posts