Programming Novel AI Accelerators for Scientific Computing

Scientific applications are increasingly adopting Artificial Intelligence (AI) techniques to advance science. The scientific community is taking advantage of specialized hardware accelerators to push the limits of scientific discovery possible with traditional processors like CPUs and GPUs as demonstrated by the winners of ACM Gordon Bell prize recipients in recent years. The AI accelerator landscape can be daunting for the broader scientific community, particularly for those who are just beginning to engage with these systems. The wide diversity in hardware architectures and software stacks makes it challenging to understand the differences between these accelerators, their capabilities, programming approaches, and performance.
In this tutorial, we will cover an overview of the AI accelerators available for allocation at Argonne Leadership Computing Facility (ALCF): SambaNova, Cerebras, Graphcore, and Groq, focusing on their architectural features and software stacks, including chip and system design, memory architecture, precision handling, programming models, and software development kits (SDKs). Through hands-on exercises, attendees will gain practical experience in refactoring code and running models on these systems, focusing on use cases of pre-training and fine-tuning open-source Large Language Models (LLMs) and deploying AI inference solutions relevant to scientific contexts. Additionally, the sessions will cover the low-level HPC software stack of these accelerators using simple HPC kernels. By the end of this tutorial, participants will have a solid understanding of the key capabilities of emerging AI accelerators and their performance implications for scientific applications, equipping them with the knowledge to leverage these technologies effectively in their research endeavors.