Maintaining Open Source Singularity Since 2017

SCS Login

Using Singularity Container Services Registry for CI/CD Workflows

By Staff

Aug 30, 2023 | How To Guides

Introduction

Open Container Initiative (OCI) registries offer extensibility and customization options. Users can define custom metadata, annotations, and labels for images, which aids in better categorization and organization. Furthermore, registries can be integrated with various Continuous Integration/Continuous Deployment (CI/CD) tools and pipelines, facilitating streamlined development and deployment processes.
In essence, OCI registries bolster container image management, security, distribution, and collaboration, serving as a robust foundation for container-based application development and deployment.

CI/CD Workflow Example Overview

Presented here is a basic CI/CD Workflow diagram. Each circle in the diagram symbolizes a process or a block of code, intended to be executed by specialized software like Gitlab, Github Actions, Jenkins, among others. Each mentioned software has its unique method of setting up a workflow. In this guide, we’ll craft a configuration for Gitlab, producing a template that can be repurposed for other settings.
Consider the hybrid workflow example below. Take a moment to comprehend the overall process, as our objective is to craft a final .gitlab-ci.yml file:
Start by generating an empty directory and an empty .gitlab-ci.yml file:
mkdir demo
touch .gitlab-ci.yml
Next, establish placeholders for each workflow phase within the .gitlab-ci.yml file. The cache segment determines which directory retains its content across stages. This directory serves as our workspace:
cache:
  - paths:
    - example

stages:
  - prepare
  - test
  - build
  - singularity-prepare
  - build-image-and-push
During the preparation stage, source code is typically fetched from the present repository, and occasionally from other sources. For this example, we’ll install git and store the source code in the “example” directory:
prepare:
  stage: prepare
  image: alpine:3.18.0
  script:
    - rm -rf example
    - apk update && apk add git
    - git clone https://github.com/golang/example
In every software development phase, it’s advisable and best practice to run unit tests against your code. For our next stage:
test:
  stage: test
  image: golang:1.20-alpine
  script:
    - ls -la
    - cd example/hello && go test
*Note: This example lacks test files, so messages indicating that can be disregarded.
Now we approach the application’s build stage. In this demonstration, building the binary is direct. However, intricate software often demands dependencies and development utilities. Feel free to modify the script section to cater to your specific build workflow:
build:
stage: build
image: golang:1.20-alpine
script:
- cd example/hello && go build
Remember, our source code is in Go, so a simple go build suffices for this demo.
The ensuing phase prepares singularity to push an OCI image to the Singularity Container Services. This action necessitates obtaining a token from SCS, followed by setting a variable in Gitlab named $SINGULARITY_TOKEN with the token as its value. Follow these steps:
Navigate to your project settings and select CI/CD.
Then, scroll to the “Variables” section, click the “Expand” button, and input the variable.
The source code for this stage is as follows:
singularity-prepare:
stage: singularity-prepare
image: almalinux:9
script:
- dnf install -y epel-release && dnf install -y singularity-ce
- echo $SINGULARITY_TOKEN>/tmp/token && singularity remote login --tokenfile /tmp/token && rm /tmp/token
- singularity remote get-login-password > example/dockerlogin
This code helps in obtaining a temporary password, subsequently utilized to push the image to SCS.

The final stage assembles the OCI image using Docker, auto-calculates the image’s tag, and ultimately pushes the image to SCS:

build-image-and-push:
stage: build-image-and-push
image: docker:latest
services:
- docker:dind
before_script:
- cat example/dockerlogin | docker login -u "USERNAME" --password-stdin registry.sylabs.io
script:
- |
if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
tag="latest"
echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
else
tag=":$CI_COMMIT_REF_SLUG"
echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
fi
- docker build --pull -t "registry.sylabs.io/USERNAME/hello:${tag}" .
- docker push "registry.sylabs.io/USERNAME/hello:${tag}"
after_script:
- rm example/dockerlogin
rules:
- if: $CI_COMMIT_BRANCH
exists:
- Dockerfile
With this workflow in place, an updated container image will be pushed to the Singularity Container Service registry each time there’s an update in the source code.

Join Our Mailing List

Categories

Related Posts