Flux: Next-Generation Workload Management for High Performance Computing and Cloud

Flux, a next-generation workload manager, is the primary means to run workloads on the first National Nuclear Security Administration (NNSA) exascale system El Capitan deployed this year. Flux is unique in that it combines graph-based scheduling with fully hierarchical resource management, and enables portability, performance, and manageability of complex workflows. Along with serving the user-base of the U.S. Department of Energy National Laboratories, Flux is making its mark on centers internationally, and is a predominant player in the emerging space of HPC technologies in the cloud, offering Kubernetes operators and custom schedulers. Given Flux’s emergence as a production resource manager for El Capitan, the HPC community can benefit from an introduction to its unique capabilities for other exascale-class machines and cloud centers.

Given the flexibility to run under another workload manager, Flux is a vehicle that promises portability of workflows not just between traditional centers, but also between centers and cloud. Flux can not only run as a system workload manager, it can be run in containers, on users’ local devices, or under other workload managers. This tutorial will include practical training for users on how Flux can run under other resource managers and containers for their workload needs. Utilizing Jupyter Notebooks deployed in the cloud, the tutorial will demonstrate the capabilities of Flux from the command line or from the Flux Python API. The tutorial will start with project background, demonstrate simple job submission and management, and conclude by providing resources for interested users to continue engaging and developing with Flux.