Managing HPC Software Complexity with Spack

Modern scientific software stacks include thousands of packages, and use a wide variety of technologies, from C, C++, and Fortran libraries, to packages written in interpreted languages like Python and R. Scientists often need to run the same stack at different scales, ranging from their own laptop to the world’s largest supercomputers, and use different workflows for different tasks.

Developing a new feature on an application might require frequent rebuilds, and the ability to run unit tests, or some small set of end-to-end tests, with a quick turnaround. Some debug features would likely be turned on in this kind of scenario.

Building an application for a campaign of large production runs on an HPC cluster, instead, requires to leverage low-level and difficult-to-build libraries such as MPI, BLAS, and LAPACK, or integrating with vendor’s provided libraries, in order to achieve the best performance. In this case, applications are frequently built in a way that is tuned for the machine they will run on.

In all these cases, building the software stack is extremely challenging, and the complexity of managing software configurations, and installing dependencies, can be an obstacle to both deployment and development.

Spack is an open source package manager that simplifies building, installing, customizing, and sharing HPC software stacks. It provides a powerful and flexible dependency model, a simple Python syntax for writing package build recipes, and a repository of over 8,300 packages maintained by a community of over 1,400 contributors. It is currently used by end-users, developers, clouds, and the world’s largest HPC centers.

This tutorial provides an introduction to Spack’s capabilities: installing and authoring packages, integrating Spack with development workflows, and deploying tuned software at HPC facilities. Attendees will learn foundational skills for automating day-to-day tasks.