Create, run and manage your MPI containers in a few steps
High Performance Computing (HPC) can be very overwhelming for anyone how is just trying to gain some experience and slowly gain expertise. The Message Passing Interface (MPI) is a big part of that complexity: writing MPI applications is not really trivial; usually you end up on a system with a pre-installed MPI and then you are basically limited by what you can do since most implementations actually have an impact of how you will run your application. And who wants to figure out the “best way” to run your MPI application when you are just starting to learn about all this? Fortunately, we have a solution for you: SyMPI and SyContainerize. (If you are a power user, it might still be very useful, see the conclusion.)
Pre-requirements
Getting started is actually pretty easy: you just need Go, Git and make.
You will need to install go-1.13 or newer (it is important). For installation instructions, please refer to the following page: https://golang.org/doc/install
Then, the easiest way to install Git and make is to rely on binary packages from your Linux distribution. On Ubuntu and Debian, if you are not sure Git and make are already installed, simply run:
$ sudo apt -y install git make
On RPM based system such as CentOS, run:
$ sudo yum install -y git make
And that is it, you now have everything you need, you will not have to run a single additional `sudo` command.
Finally, you need to make sure that one environment variable is correctly set. If the command ‘echo $GOPATH’ returns something like ‘/home/me/go’, you are already all set. If not, run the following commands:
$ echo “export GOPATH=$HOME/go” >> $HOME/.bashrc
$ echo “export PATH=$GOPATH/bin:$PATH” >> $HOME/.bashrc
$ source $HOME/.bashrc && mkdir -p $GOPATH/bin
For the remaining of this article, we assume that we are using Ubuntu Disco but it also has been successfully tested on CentOS.
Installation of singularity-mpi
First create a directory where the code will be checked out:
$ mkdir -p $HOME/src && cd $HOME/src
Then, get the code from our Git repository:
$ git clone https://github.com/sylabs/singularity-mpi
Compile the code and install:
$ cd singularity-mpi && make install
Now, let’s make sure that your system is ready to create and run MPI containers. For that run the ‘sympi -config’ command. If you system is correctly setup, you will get the following output:
$ sympi -config
2019/11/22 14:20:55 [WARN] failed to find the Singularity binary
2019/11/22 14:20:55 * Slurm not detected
2019/11/22 14:20:55 * Infiniband not detected
2019/11/22 14:20:55 * Checking for Singularity fail
2019/11/22 14:20:55 * Checking for wget pass
2019/11/22 14:20:55 * Checking for gfortran pass
2019/11/22 14:20:55 * Checking for gcc pass
2019/11/22 14:20:55 * Checking for g++ pass
2019/11/22 14:20:55 * Checking for make pass
2019/11/22 14:20:55 * Checking for file pass
2019/11/22 14:20:55 * Checking for mksquashfs pass
2019/11/22 14:20:55 * Checking for bzip2 pass
2019/11/22 14:20:55 * Checking for newuidmap pass
2019/11/22 14:20:55 * Checking for tar pass
If your system is not correctly setup to build and run MPI containers, you will get something like:
$ sympi -config
2019/11/22 14:22:38 [WARN] failed to find the Singularity binary
2019/11/22 14:22:38 * Slurm not detected
2019/11/22 14:22:38 * Infiniband not detected
2019/11/22 14:22:38 * Checking for Singularity fail
2019/11/22 14:22:38 * Checking for wget fail
The system is not correctly setup.
On Debian based systems, the following commands can ensure that all required packages are install:
sudo apt -y install build-essential \
libssl-dev \
uuid-dev \
libgpgme11-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup \
tar bzip2 \
gcc gfortran g++ make \
squashfs-tools \
uidmap
On RPM based systems:
yum groupinstall -y 'Development Tools' && \
sudo yum install -y openssl-devel \
libuuid-devel \
libseccomp-devel \
wget \
squashfs-tools \
cryptsetup shadow-utils \
gcc gcc-gfortran gcc-c++ make \
On RPM systems, you may also want to run the following commands as root to enable fakeroot:
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" \
sudo echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf
2019/11/22 14:22:38 System not setup properly: wget not found: exec: "wget": executable file not found in $PATH
These directions will ensure that your system is ready to create and run MPI containers. Note that this is not specific the SyMPI and SyContainerizetools, which is the focus of this article, but rather necessary steps to be able to run any containers.
Install Singularity
Before we can create any container, we need to install Singularity. Nothing more simple, it only requires two commands:
$ sympi_init
$ symp -install singularity_master -no-suid
Why do we install the master version of Singularity? Because at the time of the redaction of this article, it is the only version that let us do everything as a user, without executing a single ‘sudo’ command (and SyMPI has a few special requirements). In HPC, running privileged commands is just something that cannot be done on most systems. And this is what the ‘-no-suid’ options does: ensure that everything is setup to be used as a user.
The installation takes a little time but once it is completed, you can check its availability:
$ sympi -list singularity
Available Singularity installation(s) on the host:
singularity:master [no-suid]
At this point, Singularity is installed but not active:
$ singularity
Command 'singularity' not found, but can be installed with:
sudo apt install singularity # version 0.30c-1, or
sudo apt install singularity-container # version 2.6.1-2
To load it, use the following command:
$ sympi -load singularity:master
And now, if you list again the versions of Singularity that are available, you can see a ‘(L)’ next to it which means that it is the version of Singularity that is currently usable. It is especially useful if you later want to use various versions of Singularity and switch back-and-forth between these versions.
$ sympi -list singularity
Available Singularity installation(s) on the host:
singularity:master [no-suid] (L)
That’s it! Singularity is correctly installed and setup, we can now create our first MPI container.
Create our first MPI container
For illustration purposes, let’s first create a container for a simple helloworld. Create the ‘$HOME/src/helloworld.c’ file and copy the following code in it:
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
int main (int argc, char **argv) {
int rc;
int size;
int myrank;
rc = MPI_Init (&argc, &argv);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Init() failed");
return EXIT_FAILURE;
}
rc = MPI_Comm_size (MPI_COMM_WORLD, &size);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Comm_size() failed");
goto exit_with_error;
}
rc = MPI_Comm_rank (MPI_COMM_WORLD, &myrank);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Comm_rank() failed");
goto exit_with_error;
}
fprintf (stdout, "Hello, I am rank %d/%d\n", myrank, size);
MPI_Finalize();
return EXIT_SUCCESS;
exit_with_error:
MPI_Finalize();
return EXIT_FAILURE;
}
Then let’s create a configuration file that will be used to create a Singularity container for it. So we can refer to it through this article, let’s assume that file is ‘$HOME/src/helloworld.conf’:
app_name = helloworld-openmpi-4.0.2-bind
app_url = file:///home/me/src/helloworld.c
app_exe = helloworld
app_compile_cmd = mpicc -o helloworld helloworld.c
mpi_model = bind
mpi = openmpi:4.0.2
distro = ubuntu:disco
A few explanations about what it means.
-
The first line sets the name of your container: ‘app_name’. That is the name that will appear when displaying all the containers that SyMPI can execute so you can really set the name to whatever is the most convenient to you. When you end up using the tool, carefully choose the name of your containers so you can identify what they are just by reading the name.
-
The second line sets ‘app_url’, which is the URL to use to get the source code of the application for which we want to create a container. Here, it is a local file so the URL start with ‘file://’ (required) and then the full path to the source file (remember to update the path to match the location of the file on your system).
-
The third line sets ‘app_exe’, which is the executable that needs to be used to start the application, even outside containers. Here, we will compile the file using ‘mpicc’ so it is the name of the output binary that we will specify on the ‘mpicc’ command.
-
The fourth line sets ‘app_compile_cmd’, which is the command to compile and install the application. As stated during the previous bullet, here we directly call ‘mpicc’
-
The fifth lines sets ‘mpi_model’. Two models are currently supported: bind and hybrid. When using bind, no MPI is installed in the container and we rely on the MPI from the host to start and run the MPI application. With hybrid, the container includes its own MPI but rely on the MPI on the host to start the application. Which one is the best? It really depends on what you are trying to ultimately do and it is out-of-the-scope of this article. Here, we use bind because the image is smaller.
-
The sixth line set ‘mpi’, which specifies the implementation and version of MPI to use. Here, we want to use Open MPI 4.0.2, as specified in the name of the container.
-
Finally, the last line sets distro, which specifies the Linux distribution to use in the container. At first, i would advise to use the same Linux distribution as the host, for simplification purposes.
That is all you need for your first MPI container: some high-level information about your application, SyContainerize will figure out and perform all the required technical steps. So you are ready to create your first container!
$ sycontainerize -conf $HOME/src/helloworld.conf
Creating a container takes time. But once completed, it appears when using SyMPI. Yes, it is that simple.
$ sympi -list
Available Singularity installation(s) on the host:
singularity:master [no-suid] (L)
Available MPI installation(s) on the host:
openmpi:4.0.2
Available container(s):
helloworld-openmpi-4.0.2-bind
As you can also see SyContainerize also installed Open MPI 4.0.2 on the host (and if you need it later, it will used to installation already available). You want to run it? No problem, use SyMPI.
$ sympi -run helloworld-openmpi-4.0.2-bind
Analyzing /home/me/.sympi/mpi_container_helloworld-openmpi-4.0.1-bind/helloworld-openmpi-4.0.2-bind.sif to figure out the correct configuration for execution...
Container based on openmpi 4.0.2
Looking for available compatible version...
openmpi 4.0.2 was found on the host as a compatible version
Container is in bind mode
Binding/mounting openmpi 4.0.2 on host -> /home/me/.sympi/mpi_install_openmpi-4.0.2
Execution successful!
Stdout: INFO: Convert SIF file to sandbox...
INFO: Convert SIF file to sandbox...
INFO: Cleaning up image...
INFO: Cleaning up image...
Stderr: Hello, I am rank 0/1
Hello, I am rank 0/1
That is it! As you can see on the output, SyMPI is using some metadata that SyContainerize included in the image when it created the container, and uses it to find the correct configuration and run the container.
Using containers on different systems
But what if the environment does not have an MPI version available to run the container? For example, what would happen if we want to run our container on a different system, let’s say with CentOS 6 instead of Ubuntu Disco? Not a problem, you can export your container.
$ sympi -export helloworld-openmpi-4.0.2-bind
Container successfully exported: /tmp/helloworld-openmpi-4.0.2-bind.sif
Then you can copy it to your other system and import it:
$ sympi -import ./helloworld-openmpi-4.0.2-bind.sif
And then, run it as presented earlier on the initial system. If the target MPI, or a compatible MPI, cannot be found, SyMPI will install the appropriate MPI and use it!
Want to try a more realistic MPI workload?
If you want to try a more realistic MPI workload, i think a good candidate is NetPIPE. NetPIPE is commonly used benchmark to evaluation point-to-point communications between two MPI ranks. In other terms, it does a bunch of fancy send and receive operations between the two ranks in a way that you can have, at the end, a good idea of what to expect in terms of network bandwidth and latency.
And NetPIPE is a more substantial piece of software. For instance, preparing NetPIPE relies on a Makefile and it will run for a little while.
Anyway, if you are interested, just repeat the steps previously described but with the following configuration file for SyContainerize:
app_name = ubuntu-disco-openmpi-4.0.2-netpipe-5.1.4-bind
app_url = http://netpipe.cs.ksu.edu/download/NetPIPE-5.1.4.tar.gz
app_exe = NPmpi
app_compile_cmd = make mpi
mpi_model = bind
mpi = openmpi:4.0.2
distro = ubuntu:disco
That will create a container for NetPIPE 5.1.4, Open MPI 4.0.2 on Ubuntu Disco. And you can export and import that image on CentOS 7 for example. It will just work!
Conclusion
If you took the time to go through all the steps of this article, you now have 2 containers without requiring any special expertise and even if you do have the expertise, without dealing with tons of technical details. And one of these two containers is a container with NetPIPE that you can just bring with you and run on different systems.
If you are a user who wants to gain MPI expertise, you can now focus on it and have a easy and quick way to create a container for your code as you write it and run it pretty much anywhere.
For power users, it is also a beneficial environment, here is the output from my CentOS 7 system:
$ sympi -list
Available Singularity installation(s) on the host:
singularity:3.4.2
singularity:3.5.0
singularity:master [no-suid]
Available MPI installation(s) on the host:
mpich:3.0
mpich:3.0.4
mpich:3.1
mpich:3.1.4
mpich:3.2
mpich:3.2.1
mpich:3.3
mpich:3.3.2
openmpi:3.0.4
openmpi:3.1.0
openmpi:3.1.4
openmpi:4.0.0
openmpi:4.0.1
openmpi:4.0.2
Available container(s):
centos-7-mpich-3.0-IMB-hybrid
centos-7-mpich-3.0-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.0-helloworld-hybrid
centos-7-mpich-3.0.4-IMB-hybrid
centos-7-mpich-3.0.4-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.0.4-helloworld-hybrid
centos-7-mpich-3.1-IMB-hybrid
centos-7-mpich-3.1-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.1-helloworld-hybrid
centos-7-mpich-3.1.4-IMB-hybrid
centos-7-mpich-3.1.4-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.1.4-helloworld-hybrid
centos-7-mpich-3.2-IMB-hybrid
centos-7-mpich-3.2-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.2-helloworld-hybrid
centos-7-mpich-3.2.1-IMB-hybrid
centos-7-mpich-3.2.1-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.2.1-helloworld-hybrid
centos-7-mpich-3.3-IMB-hybrid
centos-7-mpich-3.3-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.3-helloworld-hybrid
centos-7-mpich-3.3.2-IMB-hybrid
centos-7-mpich-3.3.2-NetPIPE-5.1.4-hybrid
centos-7-mpich-3.3.2-helloworld-hybrid
centos-7-openmpi-3.0.4-IMB-hybrid
centos-7-openmpi-3.0.4-NetPIPE-5.1.4-hybrid
centos-7-openmpi-3.0.4-helloworld-bind
centos-7-openmpi-3.0.4-helloworld-hybrid
centos-7-openmpi-3.1.0-IMB-hybrid
centos-7-openmpi-3.1.0-NetPIPE-5.1.4-hybrid
centos-7-openmpi-3.1.0-helloworld-hybrid
centos-7-openmpi-3.1.4-IMB-hybrid
centos-7-openmpi-3.1.4-NetPIPE-5.1.4-hybrid
centos-7-openmpi-3.1.4-helloworld-hybrid
centos-7-openmpi-4.0.0-IMB-hybrid
centos-7-openmpi-4.0.0-NetPIPE-5.1.4-hybrid
centos-7-openmpi-4.0.0-helloworld-hybrid
centos-7-openmpi-4.0.1-IMB-hybrid
centos-7-openmpi-4.0.1-NetPIPE-5.1.4-hybrid
centos-7-openmpi-4.0.1-helloworld-hybrid
centos-7-openmpi-4.0.2-IMB-hybrid
centos-7-openmpi-4.0.2-NetPIPE-5.1.4-hybrid
centos-7-openmpi-4.0.2-helloworld-hybrid
test_helloworld
ubuntu-disco-openmpi-4.0.1-netpipe-5.1.4-hybrid
As you can see, i use SyMPI on a daily basis to run tests with a lot of different configurations, meaning different MPIs, different applications and different Linux distributions. And it saves me tons of time!
What is next?
Well as you saw, this article is about running MPI containers on a single system... which is a little lame since MPI is designed to do parallel computing. The good news is that the code is actually ready to interface with job managers (SLURM at the moment) and many other details. So you can actually already use it on a cluster… but that will be the topic of another article! Stay tuned!