The Future of Benchmarks in Supercomputing

The goal of this workshop is to pose, engage, debate, and address the question - "How should the supercomputing community evolve performance benchmarks?”. The workshop will be organized as presentations and panel discussions with audience participation that will invite active members of the Top500, HPCG, MLPerf, Semi Analysis, etc. and key personnel from industry, academia, and government to discuss the value, need and desire for evolving the benchmark suite that is inclusive and accommodative of emerging applications in HPC, AI and data analytics to guide future supercomputing system design and architecture.

This workshop is a follow-on to well-attended (>100 participants) sessions at SC23, SC24, ISC24, and ISC25 under the same name. The past workshops are archived at https://sites.google.com/view/thefutureofbenchmarks. Stalwarts from the community have presented their viewpoints, and a significant majority of participants continue to express a strong desire for deeper conversations on the following topics - (i) Design of benchmarks (representative, cost-conscious etc.), (ii) Choice of metrics (energy efficiency, I/O inclusive, etc.), (iii) Agility to evolve (problem size, newer kernels, mini-apps, containerized harness etc.), and (iv) Articulation of purpose (ranking list, co-design, marketing).

---

Organizers:

• Jack Wells
• Sreenivas Sukumar
• Michael Woodacre