Leveraging and Evaluating LLMs for HPC Research

Large Language Models (LLMs) are progressing at an impressive pace. They are becoming capable of solving complex problems while presenting the opportunity to leverage their capabilities for High-Performance Computing (HPC) research. Despite their progress, even the most sophisticated models can struggle with simple reasoning tasks and make mistakes, necessitating careful verification of their outputs. This tutorial focuses on these two important aspects: (1) leveraging LLMs to assist and advance HPC research, and (2) establishing best practices for evaluating LLMs within the HPC context. Designed specifically for students, postdocs, researchers, engineers, and practitioners, the tutorial will guide participants through methods and techniques for utilizing and testing LLMs for HPC research at basic and intermediate levels. Attendees will first explore the fundamentals of LLM design, development, application, and evaluation while focusing on HPC-specific uses. Six HPC research scenarios will be detailed: sequential and parallel code generation/translation, domain-specific AI surrogates, HPC performance prediction, checkpointing optimization, research assistants, and hardware design. Participants will also learn various complementary methods to rigorously evaluate LLM responses using automated Q&A benchmarks (e.g., multiple-choice questions, open response) and frameworks like LM Eval Harness, HELM, and DecodingTrust; more exploratory evaluation methods such as "lab-style" and "in-the-wild" approaches which emphasize the role of LLMs more as HPC research assistants than mere chatbots. The tutorial features a hands-on session where participants use an LLM to solve a provided HPC research problem while presenters assist with multi-turn prompting. The tutorial covers the following seven topics at basic [B] and intermediate [I] levels: (1) [B] Basics of LLMs, (2) [B] Uses cases of LLMs in HPC context, (3) [B] Importance of prompting and performance of different LLMs for HPC research (4) [B] Basic of LLM evaluation, (4) [I] Evaluation of LLMs for science and engineering, (6) [I] Evaluation techniques of LLMs for Science and Engineering. (7) [I] Hands-on. At the end of the tutorial, participants will be equipped with solid foundations, knowledge, techniques, and practical experience to leverage and evaluate LLMs for HPC research, equipped to transform theoretical insights into actionable solutions.