The Future of Benchmarks in Supercomputing

As supercomputing welcomes new workflows of simulations, data science and artificial intelligence in the Exascale era, the goal of this workshop is to pose, engage, debate, and address the question - "How should the SC community evolve performance benchmarks?". The session will be organized as presentations and panel discussions with audience participation that will invite active members of the Top500, HPCG, MLPerf, TeraSort, etc. and key personnel from industry, academia, and government to discuss the value, need and desire for evolving the benchmark suite that is inclusive and accommodative of emerging applications to guide future supercomputing system design and architecture.

This workshop is a follow-on to well-attended (>100 participants) birds-of-a-feather sessions at SC23 and SC24 under the same name and a previous workshop at ISC24. Stalwarts from the community presented their viewpoints, and a significant majority of participants expressed a desire to continue deeper conversations on the following topics - (i) Design of benchmarks (representative, environmentally sustainable, cost-conscious, etc.), (ii) Choice of metrics (energy efficiency, I/O inclusive, etc.), (iii) Agility to evolve quickly (problem size, newer kernels, mini-apps, containerized harness etc.), and (iv) Articulation of purpose (ranking list, co-design, marketing).

This workshop will organize presentations and panels to further the community engagement around these aforementioned topics. Specifically this year, we will focus on the momentum and experience from several post-Exascale procurement benchmarks and the need for new benchmarks in a potential future for HPC workflows to thrive on AI-driven hardware (wafers, chiplets, reconfigurable arrays etc.) and software (Python-native, mixed-precision optimizations, etc.).

---

Organizers:

• Sreenivas Sukumar
• Jack Wells
• Roy Verghese
• Arti Garg