Create, run and manage your MPI containers in a few steps

Nov 25th, 2019

Create, run and manage your MPI containers in a few steps

High Performance Computing (HPC) can be very overwhelming for anyone how is just trying to gain some experience and slowly gain expertise. The Message Passing Interface (MPI) is a big part of that complexity: writing MPI applications is not really trivial; usually you end up on a system with a pre-installed MPI and then you are basically limited by what you can do since most implementations actually have an impact of how you will run your application. And who wants to figure out the “best way” to run your MPI application when you are just starting to learn about all this? Fortunately, we have a solution for you: SyMPI and SyContainerize. (If you are a power user, it might still be very useful, see the conclusion.)

Pre-requirements

Getting started is actually pretty easy: you just need Go, Git and make.

You will need to install go-1.13 or newer (it is important). For installation instructions, please refer to the following page: https://golang.org/doc/install

Then, the easiest way to install Git and make is to rely on binary packages from your Linux distribution. On Ubuntu and Debian, if you are not sure Git and make are already installed, simply run:

$ sudo apt -y install git make

On RPM based system such as CentOS, run:

$ sudo yum install -y git make

And that is it, you now have everything you need, you will not have to run a single additional `sudo` command.

Finally, you need to make sure that one environment variable is correctly set. If the command ‘echo $GOPATH’ returns something like ‘/home/me/go’, you are already all set. If not, run the following commands:

$ echo “export GOPATH=$HOME/go” >> $HOME/.bashrc $ echo “export PATH=$GOPATH/bin:$PATH” >> $HOME/.bashrc $ source $HOME/.bashrc && mkdir -p $GOPATH/bin

For the remaining of this article, we assume that we are using Ubuntu Disco but it also has been successfully tested on CentOS.

Installation of singularity-mpi

First create a directory where the code will be checked out:

$ mkdir -p $HOME/src && cd $HOME/src

Then, get the code from our Git repository:

$ git clone https://github.com/sylabs/singularity-mpi

Compile the code and install:

$ cd singularity-mpi && make install

Now, let’s make sure that your system is ready to create and run MPI containers. For that run the ‘sympi -config’ command. If you system is correctly setup, you will get the following output:

$ sympi -config 2019/11/22 14:20:55 [WARN] failed to find the Singularity binary 2019/11/22 14:20:55 * Slurm not detected 2019/11/22 14:20:55 * Infiniband not detected 2019/11/22 14:20:55 * Checking for Singularity fail 2019/11/22 14:20:55 * Checking for wget pass 2019/11/22 14:20:55 * Checking for gfortran pass 2019/11/22 14:20:55 * Checking for gcc pass 2019/11/22 14:20:55 * Checking for g++ pass 2019/11/22 14:20:55 * Checking for make pass 2019/11/22 14:20:55 * Checking for file pass 2019/11/22 14:20:55 * Checking for mksquashfs pass 2019/11/22 14:20:55 * Checking for bzip2 pass 2019/11/22 14:20:55 * Checking for newuidmap pass 2019/11/22 14:20:55 * Checking for tar pass

If your system is not correctly setup to build and run MPI containers, you will get something like:

$ sympi -config 2019/11/22 14:22:38 [WARN] failed to find the Singularity binary 2019/11/22 14:22:38 * Slurm not detected 2019/11/22 14:22:38 * Infiniband not detected 2019/11/22 14:22:38 * Checking for Singularity fail 2019/11/22 14:22:38 * Checking for wget fail The system is not correctly setup. On Debian based systems, the following commands can ensure that all required packages are install: sudo apt -y install build-essential \ libssl-dev \ uuid-dev \ libgpgme11-dev \ squashfs-tools \ libseccomp-dev \ wget \ pkg-config \ git \ cryptsetup \ tar bzip2 \ gcc gfortran g++ make \ squashfs-tools \ uidmap On RPM based systems: yum groupinstall -y 'Development Tools' && \ sudo yum install -y openssl-devel \ libuuid-devel \ libseccomp-devel \ wget \ squashfs-tools \ cryptsetup shadow-utils \ gcc gcc-gfortran gcc-c++ make \ On RPM systems, you may also want to run the following commands as root to enable fakeroot: grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" \ sudo echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf 2019/11/22 14:22:38 System not setup properly: wget not found: exec: "wget": executable file not found in $PATH

These directions will ensure that your system is ready to create and run MPI containers. Note that this is not specific the SyMPI and SyContainerizetools, which is the focus of this article, but rather necessary steps to be able to run any containers.

Install Singularity

Before we can create any container, we need to install Singularity. Nothing more simple, it only requires two commands:

$ sympi_init $ symp -install singularity_master -no-suid

Why do we install the master version of Singularity? Because at the time of the redaction of this article, it is the only version that let us do everything as a user, without executing a single ‘sudo’ command (and SyMPI has a few special requirements). In HPC, running privileged commands is just something that cannot be done on most systems. And this is what the ‘-no-suid’ options does: ensure that everything is setup to be used as a user.

The installation takes a little time but once it is completed, you can check its availability:

$ sympi -list singularity Available Singularity installation(s) on the host: singularity:master [no-suid]

At this point, Singularity is installed but not active:

$ singularity Command 'singularity' not found, but can be installed with: sudo apt install singularity # version 0.30c-1, or sudo apt install singularity-container # version 2.6.1-2

To load it, use the following command:

$ sympi -load singularity:master

And now, if you list again the versions of Singularity that are available, you can see a ‘(L)’ next to it which means that it is the version of Singularity that is currently usable. It is especially useful if you later want to use various versions of Singularity and switch back-and-forth between these versions.

$ sympi -list singularity Available Singularity installation(s) on the host: singularity:master [no-suid] (L)

That’s it! Singularity is correctly installed and setup, we can now create our first MPI container.

Create our first MPI container

For illustration purposes, let’s first create a container for a simple helloworld. Create the ‘$HOME/src/helloworld.c’ file and copy the following code in it:

#include <mpi.h> #include <stdio.h> #include <stdlib.h> int main (int argc, char **argv) { int rc; int size; int myrank; rc = MPI_Init (&argc, &argv); if (rc != MPI_SUCCESS) { fprintf (stderr, "MPI_Init() failed"); return EXIT_FAILURE; } rc = MPI_Comm_size (MPI_COMM_WORLD, &size); if (rc != MPI_SUCCESS) { fprintf (stderr, "MPI_Comm_size() failed"); goto exit_with_error; } rc = MPI_Comm_rank (MPI_COMM_WORLD, &myrank); if (rc != MPI_SUCCESS) { fprintf (stderr, "MPI_Comm_rank() failed"); goto exit_with_error; } fprintf (stdout, "Hello, I am rank %d/%d\n", myrank, size); MPI_Finalize(); return EXIT_SUCCESS; exit_with_error: MPI_Finalize(); return EXIT_FAILURE; }

Then let’s create a configuration file that will be used to create a Singularity container for it. So we can refer to it through this article, let’s assume that file is ‘$HOME/src/helloworld.conf’:

app_name = helloworld-openmpi-4.0.2-bind app_url = file:///home/me/src/helloworld.c app_exe = helloworld app_compile_cmd = mpicc -o helloworld helloworld.c mpi_model = bind mpi = openmpi:4.0.2 distro = ubuntu:disco

A few explanations about what it means.

That is all you need for your first MPI container: some high-level information about your application, SyContainerize will figure out and perform all the required technical steps. So you are ready to create your first container!

$ sycontainerize -conf $HOME/src/helloworld.conf

Creating a container takes time. But once completed, it appears when using SyMPI. Yes, it is that simple.

$ sympi -list Available Singularity installation(s) on the host: singularity:master [no-suid] (L) Available MPI installation(s) on the host: openmpi:4.0.2 Available container(s): helloworld-openmpi-4.0.2-bind

As you can also see SyContainerize also installed Open MPI 4.0.2 on the host (and if you need it later, it will used to installation already available). You want to run it? No problem, use SyMPI.

$ sympi -run helloworld-openmpi-4.0.2-bind Analyzing /home/me/.sympi/mpi_container_helloworld-openmpi-4.0.1-bind/helloworld-openmpi-4.0.2-bind.sif to figure out the correct configuration for execution... Container based on openmpi 4.0.2 Looking for available compatible version... openmpi 4.0.2 was found on the host as a compatible version Container is in bind mode Binding/mounting openmpi 4.0.2 on host -> /home/me/.sympi/mpi_install_openmpi-4.0.2 Execution successful! Stdout: INFO: Convert SIF file to sandbox... INFO: Convert SIF file to sandbox... INFO: Cleaning up image... INFO: Cleaning up image... Stderr: Hello, I am rank 0/1 Hello, I am rank 0/1

That is it! As you can see on the output, SyMPI is using some metadata that SyContainerize included in the image when it created the container, and uses it to find the correct configuration and run the container.

Using containers on different systems

But what if the environment does not have an MPI version available to run the container? For example, what would happen if we want to run our container on a different system, let’s say with CentOS 6 instead of Ubuntu Disco? Not a problem, you can export your container.

$ sympi -export helloworld-openmpi-4.0.2-bind Container successfully exported: /tmp/helloworld-openmpi-4.0.2-bind.sif

Then you can copy it to your other system and import it:

$ sympi -import ./helloworld-openmpi-4.0.2-bind.sif

And then, run it as presented earlier on the initial system. If the target MPI, or a compatible MPI, cannot be found, SyMPI will install the appropriate MPI and use it!

Want to try a more realistic MPI workload?

If you want to try a more realistic MPI workload, i think a good candidate is NetPIPE. NetPIPE is commonly used benchmark to evaluation point-to-point communications between two MPI ranks. In other terms, it does a bunch of fancy send and receive operations between the two ranks in a way that you can have, at the end, a good idea of what to expect in terms of network bandwidth and latency.

And NetPIPE is a more substantial piece of software. For instance, preparing NetPIPE relies on a Makefile and it will run for a little while.

Anyway, if you are interested, just repeat the steps previously described but with the following configuration file for SyContainerize:

app_name = ubuntu-disco-openmpi-4.0.2-netpipe-5.1.4-bind app_url = http://netpipe.cs.ksu.edu/download/NetPIPE-5.1.4.tar.gz app_exe = NPmpi app_compile_cmd = make mpi mpi_model = bind mpi = openmpi:4.0.2 distro = ubuntu:disco

That will create a container for NetPIPE 5.1.4, Open MPI 4.0.2 on Ubuntu Disco. And you can export and import that image on CentOS 7 for example. It will just work!

Conclusion

If you took the time to go through all the steps of this article, you now have 2 containers without requiring any special expertise and even if you do have the expertise, without dealing with tons of technical details. And one of these two containers is a container with NetPIPE that you can just bring with you and run on different systems.

If you are a user who wants to gain MPI expertise, you can now focus on it and have a easy and quick way to create a container for your code as you write it and run it pretty much anywhere.

For power users, it is also a beneficial environment, here is the output from my CentOS 7 system:

$ sympi -list Available Singularity installation(s) on the host: singularity:3.4.2 singularity:3.5.0 singularity:master [no-suid] Available MPI installation(s) on the host: mpich:3.0 mpich:3.0.4 mpich:3.1 mpich:3.1.4 mpich:3.2 mpich:3.2.1 mpich:3.3 mpich:3.3.2 openmpi:3.0.4 openmpi:3.1.0 openmpi:3.1.4 openmpi:4.0.0 openmpi:4.0.1 openmpi:4.0.2 Available container(s): centos-7-mpich-3.0-IMB-hybrid centos-7-mpich-3.0-NetPIPE-5.1.4-hybrid centos-7-mpich-3.0-helloworld-hybrid centos-7-mpich-3.0.4-IMB-hybrid centos-7-mpich-3.0.4-NetPIPE-5.1.4-hybrid centos-7-mpich-3.0.4-helloworld-hybrid centos-7-mpich-3.1-IMB-hybrid centos-7-mpich-3.1-NetPIPE-5.1.4-hybrid centos-7-mpich-3.1-helloworld-hybrid centos-7-mpich-3.1.4-IMB-hybrid centos-7-mpich-3.1.4-NetPIPE-5.1.4-hybrid centos-7-mpich-3.1.4-helloworld-hybrid centos-7-mpich-3.2-IMB-hybrid centos-7-mpich-3.2-NetPIPE-5.1.4-hybrid centos-7-mpich-3.2-helloworld-hybrid centos-7-mpich-3.2.1-IMB-hybrid centos-7-mpich-3.2.1-NetPIPE-5.1.4-hybrid centos-7-mpich-3.2.1-helloworld-hybrid centos-7-mpich-3.3-IMB-hybrid centos-7-mpich-3.3-NetPIPE-5.1.4-hybrid centos-7-mpich-3.3-helloworld-hybrid centos-7-mpich-3.3.2-IMB-hybrid centos-7-mpich-3.3.2-NetPIPE-5.1.4-hybrid centos-7-mpich-3.3.2-helloworld-hybrid centos-7-openmpi-3.0.4-IMB-hybrid centos-7-openmpi-3.0.4-NetPIPE-5.1.4-hybrid centos-7-openmpi-3.0.4-helloworld-bind centos-7-openmpi-3.0.4-helloworld-hybrid centos-7-openmpi-3.1.0-IMB-hybrid centos-7-openmpi-3.1.0-NetPIPE-5.1.4-hybrid centos-7-openmpi-3.1.0-helloworld-hybrid centos-7-openmpi-3.1.4-IMB-hybrid centos-7-openmpi-3.1.4-NetPIPE-5.1.4-hybrid centos-7-openmpi-3.1.4-helloworld-hybrid centos-7-openmpi-4.0.0-IMB-hybrid centos-7-openmpi-4.0.0-NetPIPE-5.1.4-hybrid centos-7-openmpi-4.0.0-helloworld-hybrid centos-7-openmpi-4.0.1-IMB-hybrid centos-7-openmpi-4.0.1-NetPIPE-5.1.4-hybrid centos-7-openmpi-4.0.1-helloworld-hybrid centos-7-openmpi-4.0.2-IMB-hybrid centos-7-openmpi-4.0.2-NetPIPE-5.1.4-hybrid centos-7-openmpi-4.0.2-helloworld-hybrid test_helloworld ubuntu-disco-openmpi-4.0.1-netpipe-5.1.4-hybrid

As you can see, i use SyMPI on a daily basis to run tests with a lot of different configurations, meaning different MPIs, different applications and different Linux distributions. And it saves me tons of time!

What is next?

Well as you saw, this article is about running MPI containers on a single system... which is a little lame since MPI is designed to do parallel computing. The good news is that the code is actually ready to interface with job managers (SLURM at the moment) and many other details. So you can actually already use it on a cluster… but that will be the topic of another article! Stay tuned!