RLTuner: Floating-Point Precision Tuning Using Deep Reinforcement Learning

Floating-point precision tuning is essential for optimizing numerical software, involving the selection of the ideal precision for floating-point variables to balance performance and accuracy. Identifying the optimal precision configuration is challenging due to the large number of possible mixed-precision configurations and the interactions between precision and speed. Existing works could get stuck in a local optima, removing a variable and/or a function call from the search space immediately if it fails to meet a given error threshold. In this paper, we introduce RLTuner, a novel automated tuning framework that reformulates precision tuning as a Markov Decision Process (MDP). By employing a Deep Q-Network (DQN), RLTuner leverages deep neural networks as function approximators to identify superior precision configurations that traditional pruning-based methods overlook. We evaluate RLTuner on 20 HPC benchmarks, ranging from kernels to large-scale applications like LULESH. RLTuner outperforms state-of-the-art baselines in 17 out of 20 cases, achieving speedups of up to 49.14%. Notably, it identifies higher-quality configurations—yielding both superior performance and lower error—in 65% of benchmarks. These results demonstrate that deep reinforcement learning is a robust, scalable approach for navigating complex precision-performance trade-offs in scientific computing.

---

Contributors:

• Yutong Wang
• Cindy Rubio-González