Reproducible Benchmarking Techniques for High-Performance Computing

Benchmarking is integral to overall practices of HPC, AI and ML. A hier-
archy of benchmarks allows comparison of machines to each other, vendors to
respond to new procurements, and compute facilities to evaluate offers when
selecting new platforms. Within a developer community, benchmarks allow an
application’s performance to be evaluated and improved over time. Despite the
importance of benchmarking, the process of performing a benchmark is chal-
lenging and requires many manual steps, such as researching the appropriate
workloads, determining how to compile the software, acquiring execution steps,
and extracting the correct metric for comparison. Collectively, these steps pose
a high barrier for new users to perform a benchmark and and even higher barrier
to executing these benchmarks across a wide set of compute facilities.

This tutorial will provide a detailed introduction to two tools for addressing
the current challenges in HPC system benchmarking. Ramble [1, 2] is a Python
experimentation framework for encoding reproducible run instructions of large
sets of experiments. Benchpark [3, 4] is an infrastructure-as-code project com-
bining a variety of open-source tools into a fully specified system for tracking
benchmark performance across a variety of systems, across multiple HPC cen-
ters, and across arbitrary choices of benchmarks. One of the open-source tools
leveraged by Benchpark is Spack, used for defining reproducible build instruc-
tions of applications. Ramble builds upon Spack for defining its run instructions.
Benchpark and Ramble are gaining popularity in the HPC community for en-
abling collaborative continuous benchmarking. Attendees of this tutorial will
leave with foundational skills in benchmarking systems, using these tools for
building and testing.