Sug Talk: Harvard’s Francesco Pontiggia On Using Singularity To Enable Scalable Interactive Workflows

TL;DR: Can Singularity contain GUI-based interactive applications? Harvard’s experience on their research cluster is that indeed it can, and that Singularity delivers additional benefits in interactive application and workflow delivery. Their talk can be found here.

SUG Series Introduction

The inaugural meeting of the Singularity User Group (SUG) was held March 12-13, 2019, at the San Diego Supercomputer Center (SDSC). The event attracted diverse representation from the international advanced computing community as conveyed through the post-event press release issued jointly by SDSC and Sylabs.

Over the course of the two-day event, over 20 talks were presented by members of the Singularity user, developer, and provider community. Because SUG generated a significant amount of interest, even from those who were unable to attend, we are sharing online each of the talks presented.

SUG Talk Introduction

Responsiveness, as perceived by the user, is a critical UX success metric. When it comes to interactive applications and workflows on shared computational infrastructures, the criticality of this metric is heightened. In fact, users expect their GUI as well as their CLI applications and workflows to remain responsive.

You might expect that the requirement for a responsive interactive UX will only result in a laundry list of challenges for containerization platforms. However, as Harvard’s Francesco Pontiggia shares in his SUG talk, Singularity containers are well suited to this stringent and high-profile requirement. Moreover, by introducing Singularity containers, the overall solution addresses challenges that he and his team had previously in delivering a responsive GUI UX to their research community. The abstract for Francesco’s contributed talk entitled Using Singularity containers to enable scalable interactive workflows on Harvard Odyssey HPC cluster is as follows:

Harvard FAS Research Computing has recently deployed a web-portal to allow users to run on the HPC cluster Odyssey interactive GUI jobs,including remote desktop sessions, notebook oriented workflows, GUI desktop software and other applications with a web-based frontend like for example RStudio server. Trying to deploy and run those applications out of our standard cluster software repository resulted in numerous challenges.

We encountered substantial difficulties in terms of ease of development and we faced compatibility issues with some specific software packages not designed to run in multi-tenant HPC environments. Moreover we faced severe limitations in the application startup performance, due to some aspect of our storage infrastructure not originally designed to support those type of workflows. While a slow startup is not a particularly relevant factor in the context of typical batch jobs run on our system, it becomes an important factor in negatively impacting the user experience when scheduling and accessing interactive applications.

Using Singularity containers proved crucial for an efficient deployment of our new portal. It gave us flexibility in the application development and allowed us to circumvent intrinsic limitations of specific applications. Moreover the use of containers allowed us to leverage our high performance lustre storage, particularly suited fast read access of large size files, allowing a significant performance boost in the applications startup time.

 

Francesco’s talk from SUG can be found below and here. Enjoy!