HPSF Conference 2025

In this talk, I will present about how we use Spack at Google both with customers and internally. This portion will include an overview of our public cache, along with our use of Spack for benchmarking.

Additionally, this talk will discuss some open-sourced tooling we have developed, Ramble, which was built on top of Spack's infrastructure. This talk will discuss how we use Ramble and Spack together to construct complex end-to-end performance benchmarking studies for HPC and AI applications, and how this process can use these tools to create reproducible experiments that are shareable with external users.

This talk presents our implementation of a build cache of pre-optimized HPC applications using Spack. By implementing architecture-specific enhancements for both x86 and ARM platforms during the build process, we created a set of stacks of optimized software accessible through build caches. Using application builds from the cache, users can reduce compute resource requirements without requiring specialized tuning expertise.
We'll demonstrate how teams can quickly deploy HPC clusters using these stacks and discuss the substantial advantages compared to building from source. We'll present comparisons to traditional builds, showing significant time-to-solution improvements. This work represents a step toward enabling the HPC community to focus on scientific discovery rather than software compilation and tuning.

The PDI data interface is a library that supports loose coupling of simulation codes with data handling libraries: the simulation code is annotated in a library-agnostic way, and data management through external libraries is described in a YAML "data handling specification tree". Access to each data handling tool or library (HDF5, NetCDF, Python, compiled functions, Dask/Deisa, libjson, MPI, etc.) is provided through a dedicated plugin. Testing, packaging and distributing PDI is a complex problem as each plugin comes with its own dependencies, some of wich are typically not provided by supercomputer administrators. In the last five years, we have managed to devise solutions to test & validate, package & distribute the library and its plugins, largely based on spack.

In this talk, we will describe PDI, the specific problems we encounter and how we tackled them with a mix of cmake, spack, and containers. We specifically focus on the creation of a large family of spack-based container images used as test environment of the library, and on the efforts deployed to ensure easy installation on the wide range of supercomputers our downstream application rely on.

The High-Performance Computing Services group at Lawrence Berkeley National Laboratory delivers extensive computing resources to Berkeley Lab and the University of California at Berkeley, supporting approximately 4,000 users and nearly 600 research projects across diverse scientific disciplines.

Over the past year and a half, we have modernized our primarily manual software build process using Spack, enabling us to meet the growing application and workflow demands of the HPC software stack.

This presentation will highlight how we leverage Spack’s features—such as environments, views, and module sets—to meet our specific needs and requirements. Additionally, we will discuss how, over the past year, our Spack pipeline, integrated with Reframe (a testing framework), has enabled our larger infrastructure team to efficiently plan and execute large-scale OS migrations across multiple scientific clusters in a short timeframe.

Modern scientific applications increasingly produce extremely large volumes of data while the scalability of I/O systems has not increased at the same rate. Lossy data compression has helped many applications address these limitations, but to meet the needs of the most demanding applications, specialized compression pipelines are needed. The FZ project helps users and compression scientists collaborate to meet the I/O needs of exascale applications by making it easier to implement custom compression tools and integrate them with applications. However, to fulfill the complex needs of this diverse ecosystem of software and systems, the FZ project uses Spack to manage the complexity of developing, distributing, and deploying specialized compression pipelines to meet the needs of its developers and users.

Spoken from the perspective of someone who has tried nearly every new spack feature in the last 5 years, and who maintains over 50 packages. This talk tells the story of how the FZ project tackled that complexity with spack, and where spack can grow to meet its future challenges coupled with tips and tricks we've learned along the way.

Personal computing devices sold today increasingly include AI hardware accelerators such as neural processing units and graphics cards with compute capability. However, scientific libraries packaged for laptop and desktop computers focus first on broad instruction set compatibility. Yet, hardware optimized libraries and behaviors can be applied at runtime as widely used by Intel MPI environmental variables. This session discusses the unique use case of the R package system for vendor neutral hardware acceleration using vendor agnostic SYCL / Kokkos. The goal is to allow scientific package developers to quickly and easily write vendor independent accelerator code with deep control and tuning capabilities that use hardware acceleration capabilities as well on laptop / desktop machines as HPC clusters. Although R is specifically discussed, ideas from this session translate to Python and other high-level language packages used in scientific computing. Additionally, this session raises technical challenges directly using Kokkos as well as Apptainer for continuous integration that would greatly benefit from early-stage feedback of those audience members at this conference.

Despite Spack's strengths as a feature-rich HPC package manager generating fast executables for HPC apps, its adoption remains limited partly due to a steep learning curve and its perception as primarily a sysadmin tool.

We propose a set of opinionated defaults that help new users quickly adopt best practices with guaranteed reproducibility and architecture compatibilities. The approach draws from conventions used in popular userspace Python package managers like pip and conda which was proven to be effective

Unlike Python, Spack is a source-distribution system, compilation errors are a common challenge. We experimented with smoke-testing compatibility compatibility across compilers, libraries, and x86_64 architectures. Results are encoded into conflict rules into the defaults, such practice can be helpful to avoid many common build failures.

We successfully deployed this approach on x86_64 platforms with substantially different purpose (DL vs HPC), demonstrating its transferability and proving current Spack features sufficient for implementation. Additional DX enhancements will be discussed. The defaults are available as an open-source repository.

The Data Acquisition systems of particle physics experiments regularly push the boundaries of high-throughput computing, demanding low-latency collection of data from thousands of devices, collating data into time-sliced events, processing these events and making trigger decisions, and writing the selected data streams to disk. To accomplish these tasks, the DAQ Engineering and Operations department at Fermilab leverages multiple software libraries and builds reusable DAQ frameworks on top. These libraries must be delivered in well-defined bundles and are thoroughly tested for compatibility and functionality before being deployed to live detectors. We have several techniques used to ensure that a consistent set of dependencies can be delivered and re-created at need. We must also support active development of DAQ software components, ideally in an environment as close as possible to that of the detectors. This development often occurs across multiple packages which have to be built in concert and features tested in a consistent and reproducible manner.
I will present our scheme for accomplishing these goals using Spack environments, bundle packages, and Github Actions-based CI.

The National Supercomputing Mission (NSM) advances India’s research by providing HPC infrastructure across institutions. However, managing software on diverse HPC systems is challenging due to hardware variations, dependencies, and version control.

Spack, a flexible package manager, addresses these issues by enabling seamless software deployment and dependency management across clusters. This study examines Spack’s implementation on 17 NSM HPC systems, improving software availability and consistency.

Spack simplifies this through customized installations, automated dependency handling, and reproducible builds, ensuring compatibility.

Implementation involved a centralized repository, automated builds, user training, software optimization, and continuous refinement. This improved research productivity, reduced support overhead, and standardized environments.

Key benefits include reproducibility, faster issue resolution, and better collaboration. Future plans involve expanding Spack repositories, integrating containers, automating updates, and training. This presentation covers our implementation, challenges, and best practices.

We aim to use Spack to deploy a large software stack at a German national HPC center. In this talk, we will give some background related to the size of the software stack, its deployment frequency, and constraints arising from the operational environment. Next, we will briefly outline some of the challenges and obstacles that we encountered such as configuration issues, interaction with Cray Programming Environment, and unexpected outcomes of the concretizer. We end the talk with a current status and next steps.

Fugaku, Japan’s flagship supercomputer, serves a diverse range of scientific disciplines, requiring extensive open-source software (OSS) support. However, managing OSS on Fugaku presents unique challenges due to its A64FX-based Arm architecture and Fujitsu’s proprietary compilers and libraries. Our team has been leveraging Spack to efficiently manage and maintain OSS. In this talk, we will share our experience using Spack on Fugaku, highlighting how it has enabled a robust and up-to-date OSS environment. We will discuss the practical benefits of Spack, including streamlined software deployment and simplified package management, and reflect on lessons learned from maintaining software in a large-scale HPC system. By sharing our insights, we aim to contribute to the broader Spack community and reinforce its role as a key tool for HPC software management.

The trials and tribulations of using Spack to construct, build, and maintain a facility-specific programming environment at LLNL. Working with the Spack developers to ensure that Spack is able to do what it claims it can do in a large production environment.

The exascale Aurora supercomputer at the Argonne Leadership Computing Facility posed numerous challenges during its development due to its novel scale. One such challenge was in creating a scalable and maintainable scientific software environment. Typical software deployment methods failed to scale and were difficult to maintain over time, necessitating a new way of thinking about software integration. In this talk we present our work on the Aurora Programming Environment, a bespoke scientific programming environment which optimizes for scale and leverages Spack for its strengths in reproducibility, automation, and multiplicative build combinations. We discuss details of the containerized build process and read-only image deployment strategy as well as existing pain points and workarounds. We also examine the future possibilities that our approach opens up, including tightly integrated CI/CD flows and portable containerized access to the PE. We believe this approach is generalizeable and may benefit facilities where traditional software integration methods fall short of needs.

Spack makes it easy to install dependencies for our software on multiple HPC platforms. However, there is little guidance on how to structure Spack environments for larger projects, share common Spack installations with code teams and utilize them in an effective way for continuous integration and development.

This presentation will share some of the lessons learned from deploying chained Spack installations for multiple code teams at LANL on various HPC platforms both on site and on other Tri-Lab systems, how to structure such deployments for reusability and upgradability, and make them deployable even on air-gapped systems. It will also show how we utilize Spack's build facilities to drive CMake-based projects on GitLab for continuous integration, without having to replicate build configuration logic in GitLab files, while giving developers an easy-to-follow workflow for recreating CI runs in various configurations.

In large-scale software development efforts, effective communication between projects is essential to ensure consistency, reproducibility, and efficiency. This presentation explores strategies to improve coordination among software teams by leveraging Continuous Integration (CI) for relaying crucial build configurations while maintaining security for proprietary project sources. We will demonstrate best practices for sharing build configurations with upstream projects without exposing proprietary code.

A key focus will be optimizing the use of Spack, particularly in reducing the number of Spack Package Repositories utilized across multiple teams. This will simplify maintenance, harden builds, and avoiding duplication. Additionally, we will highlight the benefits of heavily integrating Spack CI to generate build caches, which will reduce rebuild times, and enhance software portability. By adopting these approaches, teams can achieve better collaboration, streamlined workflows, and improved software sustainability.

The WEAVE team at LLNL has created a Spack-based virtual environment (accessible to Livermore Computing community) that has a rich set of open-source tools to create a workflow for any HPC applications, commonly used Python packages and several commonly used ML and AI packages.
The goal is to provide a stable, well tested environment that users can activate and use directly across Livermore Computing's vast array of machines/OSes and networks.
We also provide the capability for users to create a local environment based on the WEAVE environment.

Using Spack allows us to install the same set of software across different platforms in LC. It also allows us to use the same Spack environment file to recreate the exact same ecosystem across networks within the Lab.
We leverage GitLab CI for our DevOps platform to automate Spack install, creating test environment, running tests and deploying the environment.
We also leverage LLNL Nexus Repository to sync our build files across networks within the lab.

WEAVE team also has implemented the "WEAVE Badging Program" where the community can submit a request to have a tool to be integrated into the WEAVE environment.

In the realm of large software projects, the choice between a monolithic repository and several distributed repositories presents significant trade-offs that can impact development efficiency, collaboration, and maintainability. Monolithic repositories, while offering centralized management and streamlined dependency handling, can become unwieldy as project size increases. Conversely, distributed repositories provide modularity and flexibility but may lead to challenges in integration and version control. This presentation will delve into the ongoing research conducted by Sandia National Laboratories, where researchers are exploring innovative solutions to harness the strengths of both repository models through the use of Spack, a package manager designed for scientific computing. We will outline the methodology employed in this exploration, highlighting the performance trade-offs i dentified thus far, including aspects such as build times, dependency resolution, and ease of collaboration. Attendees will gain insights into the implications of repository structure on software development practices and the potential for hybrid approaches to optimize project outcomes.