HPSF Conference 2025

The PDI data interface is a library that supports loose coupling of simulation codes with data handling libraries: the simulation code is annotated in a library-agnostic way, and data management through external libraries is described in a YAML "data handling specification tree". Access to each data handling tool or library (HDF5, NetCDF, Python, compiled functions, Dask/Deisa, libjson, MPI, etc.) is provided through a dedicated plugin. Testing, packaging and distributing PDI is a complex problem as each plugin comes with its own dependencies, some of wich are typically not provided by supercomputer administrators. In the last five years, we have managed to devise solutions to test & validate, package & distribute the library and its plugins, largely based on spack.

In this talk, we will describe PDI, the specific problems we encounter and how we tackled them with a mix of cmake, spack, and containers. We specifically focus on the creation of a large family of spack-based container images used as test environment of the library, and on the efforts deployed to ensure easy installation on the wide range of supercomputers our downstream application rely on.