HPSF Conference 2025

Spack team members manage a complex continuous integration (CI) infrastructure to compile and test software distributed via a centralized build cache. The workflows are executed in virtualized cloud environments, an approach that offers considerable flexibility in resource provisioning, resulting in quick feedback on pull requests submitted by the Spack community. However, fixed resource allocations for build jobs lead to job failures, slowdowns, and suboptimal resource usage.

spack-gantry aims to address these challenges by providing resource allocation predictions to build pipelines. We use historical build data to forecast future utilization and set resource limits. We will present strategies used to find optimal predictions, as well as experiments to create an online prediction service modeled after a genetic algorithm. The results of this work include improvements in usage efficiency and cost-per-job metrics.

Because spack-gantry involves modifications to the current CI system, we will discuss ramifications for users and developers and solicit feedback about how to make the infrastructure more user-friendly, reliable, and functional for the Spack community.