Research Software Engineering in High-Performance Computing - Tools and Techniques for Continuous Integration and Benchmarking

It has long been understood that there is a strong overlap between the fields of HPC and RSE. Although the two are not entirely congruent, the techniques used and the communities are closely interlinked. It is therefore not surprising that dedicated RSE events have already been held at major HPC conferences. In 2024, an RSE workshop was also held at ISC for the first time, where the symbiosis between the two communities and their mutual positive influence were examined. In this follow-up workshop, the focus will now be placed on more technical aspects. More precisely, this workshop and its participants will discuss the impact, usages, and implementations of continuous integration and benchmarking for heterogeneous, extreme-scale HPC systems.

When developing research software, it is often relevant to track its performance over time. It is even vital when targeting HPC architectures. Changes to the software itself, the used toolchains, or the system setup should not compromise how fast users obtain their results. Ideally, performance or scalability should only ever increase. Hence benchmarking should be an integral part of testing, in particular for HPC codes. At the same time, up-to-date benchmarks that are publicly available can advertise the code and the machines running them, informing users how to set-up the software in the most ideal way or whether they are achieving the expected performance. To limit the burden on developers, the aforementioned steps should be automated within continuous integration (CI) practices, introducing continuous benchmarking (CB) to it. For HPC, an added complexity is the requirement of more than the usual CI backends, with access to longer running steps, more resources than available on a single node, and a diverse range of architectures that the software needs to be tested on.

In this workshop, participants will be able to exchange ideas, approaches, good practices, but also obstacles on the way to continuous benchmarking on HPC systems. Driven by keynote talks which will set the stage, participants are encouraged to share their own experiences, both from a users and from an operations perspective. To foster this exchange, the second part of the workshop will offer a stage for participants to contribute their expertise, and experience with lightning talks. In addition, the event will have a part where open questions to the audience will be posed, helping everyone to add their thoughts to this event.

Topics include, but are not limited to:
- User’s perspective: How do users experience the existing solutions? Is there something missing, from technological solutions to convenience?
- Operations’ perspective: What does it take for hosting sites to start and to maintain an CI/CB infrastructure? What are benefits, what are burdens?
- Success stories and scary tales: where did CI/CB help to identify problems? Where could it have helped? And what did hinder the adaptation of CI/CB?
- Past, present, future: what are lessons learned and what does a wishlist look like?

---

Organizers:

• René Caspart
• Robert Speck