Modern Mixed-Precision Methods: Hardware Perspectives, Algorithms, Kernels, and Solvers

This tutorial will expose the audience to the rapidly expanding landscape of mixed- and multi-precision methods in modern scientific computing. The ongoing cross-pollination between high-performance computing (HPC) and machine learning (ML) or artificial intelligence (Al) is leading to automated computational steering of large-scale simulations at increasing scales and data models, foundational, and ab-initio models share the same system resources. More importantly for this tutorial, sharing of the hardware platforms and exploiting their wide range of computational modes has led to proliferation of multiple representations of floating-point data and increased interest in new methods that exploit them. Against the backdrop of the high-performance libraries produced by internet-scale companies, hardware vendors, national laboratories, and academic institutions, we will show the recent algorithmic progress in exploiting multiple precisions for increased efficiency in performance, communication bandwidth, and/or storage overhead. The computational techniques presented in this tutorial employ a combination of floating-point representations such as low/limited precision, quantized integers, and modular precision ecosystems. Note that lossless or lossy compression techniques are not covered. A portion of this tutorial will cover the fundamentals of the HPC software development in order to introduce the audience to some of the low-level details of coding for multiple precisions on modern hardware, including GPU accelerators. The hardware focus of the tutorial will feature floating-point representation and performance of the recent supercomputing and industrial computing CPUs and accelerators, as well as less mainstream devices meant for more specific tasks often at limited power envelopes.