Optimizing OpenFOAM for Systems with CPUs and GPUs and Unified Memory Architecture

OpenFOAM{\textregistered} is an open-source C++ library widely used for computational fluid dynamics (CFD) simulations. While extensively optimized for distributed memory computing over decades, adapting OpenFOAM to leverage modern GPU accelerators remains challenging. This paper examines the evolution of OpenFOAM towards GPU acceleration using OpenMP offloading on AMD Instinct$^{TM}$ MI300A APU systems. We identify critical performance bottlenecks inherent in GPU-offloaded OpenFOAM implementations, including excessive kernel launch overhead, excessive use of atomics, and inefficient memory management patterns. To address these challenges, we introduce several optimization techniques: kernel fusion via expression templates, multi-container indexing for boundary patch evaluation, face-to-cell loop transformations, and memory pooling strategies. Our approach achieves substantial performance improvements on standard CFD benchmarks—the HPC MotorBike and OpenFOAM HPC Challenge problems—delivering significant speedups over CPU baselines while requiring minimal modifications to the existing codebase.

---

Contributors:

• Leopold Grinberg
• Mattijs Janssens
• Andrew Heather