HPSF Conference 2025

Modern scientific applications increasingly produce extremely large volumes of data while the scalability of I/O systems has not increased at the same rate. Lossy data compression has helped many applications address these limitations, but to meet the needs of the most demanding applications, specialized compression pipelines are needed. The FZ project helps users and compression scientists collaborate to meet the I/O needs of exascale applications by making it easier to implement custom compression tools and integrate them with applications. However, to fulfill the complex needs of this diverse ecosystem of software and systems, the FZ project uses Spack to manage the complexity of developing, distributing, and deploying specialized compression pipelines to meet the needs of its developers and users.

Spoken from the perspective of someone who has tried nearly every new spack feature in the last 5 years, and who maintains over 50 packages. This talk tells the story of how the FZ project tackled that complexity with spack, and where spack can grow to meet its future challenges coupled with tips and tricks we've learned along the way.