HPC-Coder-v2: Studying Code LLMs Across Low-Resource Parallel Languages

Large Language Model (LLM) based coding tools have been tremendously successful
as software development assistants, yet they are often designed for general
purpose programming tasks and perform poorly for more specialized domains such
as high performance computing. Creating specialized models and tools for these
domains is crucial towards gaining the benefits of LLMs in areas such as HPC.
While previous work has explored HPC-specific models, LLMs still struggle to
generate parallel code and it is not at all clear what hurdles are still holding
back these LLMs and what must be done to overcome them. In this work, we conduct
an in-depth study along the many axes of fine-tuning a specialized HPC LLM in
order to better understand the challenges. Based on our findings we fine-tune
and evaluate a specialized HPC LLM that is shown to be the best performing
open-source code LLM for parallel code generation to date.

---

Contributors:

• Aman Chaturvedi
• Siddharth Singh
• Abhinav Bhatele