HPSF Conference 2025

HPSF Conference 2025

May 5-8, 2025
Chicago, IL
View More Details & Registration

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for the event to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to find out more information.

This schedule is automatically displayed in Central Time (UTC/GMT -6 hours). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date."

IMPORTANT NOTE: Timing of sessions and room locations are subject to change.

9:00am CDT

Introduction to Containers and Charliecloud - Reid Priedhorsky, Los Alamos National Laboratory

Wednesday May 7, 2025 9:00am - 9:40am CDT

Charliecloud is a lightweight, fully unprivileged, open source container implementation for HPC applications. It can handle the entire container workflow, including building images, pushing/pulling to registries, and dealing with accelerators like GPUs. We take a unique approach to containerization while remaining compatible with the broader container ecosystem where it matters.

Notable distinctions include: fully unprivileged end-to-end workflow, small code base with minimal dependencies, delegation of all security boundaries, layer-free build cache, zero-consistency root emulation based on seccomp, and a spirit of openness and public service.

This talk will motivate containers for HPC applications, then explain Charliecloud’s design philosophy and how we address these needs. We’ll discuss Charliecloud’s “weirdness” when compared to other implementations (i.e., its design philosophy and key distinctions), provide a brief overview of the code’s structure, and summarize recent and upcoming changes of interest.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Wednesday May 7, 2025 9:00am - 9:40am CDT
Illinois River

Charliecloud

9:40am CDT

Charliecloud Workshop: Key Workflow Operation - Pull - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 9:40am - 10:00am CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will walk through the key workflow operation pull.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 9:40am - 10:00am CDT
Illinois River

Charliecloud

10:00am CDT

Charliecloud Workshop: Containers Are Not Special Alpine via Tarball - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 10:00am - 10:20am CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will explain that containers are not special by running an Alpine container via tarball.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 10:00am - 10:20am CDT
Illinois River

Charliecloud

10:45am CDT

Charliecloud Workshop: Key Workflow Operation - Build from Dockerfile - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 10:45am - 11:05am CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will walk through the key workflow operation, building from a Dockerfile.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 10:45am - 11:05am CDT
Illinois River

Charliecloud

11:05am CDT

Charliecloud Workshop: Key Workflow Operation - Push - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 11:05am - 11:25am CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will walk through the key workflow operation push.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 11:05am - 11:25am CDT
Illinois River

Charliecloud

11:25am CDT

Charliecloud Workshop: MPI Hello World - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 11:25am - 11:45am CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will include a multi-node MPI hello world example.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 11:25am - 11:45am CDT
Illinois River

Charliecloud

11:45am CDT

Charliecloud Workshop: Wrap-up - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 11:45am - 12:05pm CDT

This workshop will provide participants with background and hands-on experience to use basic Charliecloud containers for HPC applications. Participants will build toy containers and a real HPC application, and then run them in parallel on a supercomputer. This will be a highly interactive workshop with lots of Q&A.

This section will wrap-up the workshop with final Q&A.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 11:45am - 12:05pm CDT
Illinois River

Charliecloud

1:35pm CDT

Key Charliecloud Innovation - Build Cache - Reid Priedhorsky, Los Alamos National Laboratory

Wednesday May 7, 2025 1:35pm - 1:55pm CDT

Container images are built by interpreting instructions in a machine-readable recipe, typically a Dockerfile, which is often faster with a build cache that stores instruction results for re-use. The standard approach (used e.g. by Docker and Podman) is a many-layered union filesystem, encoding differences between layers as tar archives.

This talk describes a new approach, as implemented in Charliecloud: store changing images in a Git repository. Our experiments show this performs similarly to layered caches on both build time and disk usage, with a considerable advantage for many-instruction recipes. Our approach also has structural advantages: better diff format, lower cache overhead, and better file de-duplication. These results show that a Git-based cache for layer-free container implementations is not only possible but may outperform the layered approach on important dimensions.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Wednesday May 7, 2025 1:35pm - 1:55pm CDT
Illinois River

Charliecloud

1:55pm CDT

Key Charliecloud Innovation - SIF - Krishna Chilleri, Los Alamos National Laboratory

Wednesday May 7, 2025 1:55pm - 2:15pm CDT

The Singularity Image Format (SIF) is a compressed read-only SquashFS filesystem that includes everything needed to run a containerized application. Leveraging its SquashFS-based architecture, Charliecloud has integrated support for the reading and execution of this image format. This talk will provide details about how the ch-run functionality identifies SIF files, analyzes the file header structure, and retrieves the SquashFS partition offset needed for container image execution.

Additionally, the talk is designed for anyone looking to gain a clearer understanding of SIF, especially if you find the upstream documentation challenging to navigate.

LA-UR-25-22166

Speakers

Krishna Chilleri

Student, Los Alamos National Laboratory

Wednesday May 7, 2025 1:55pm - 2:15pm CDT
Illinois River

Charliecloud

2:15pm CDT

Key Charliecloud Innovation - squashfs - Megan Phinney, Los Alamos National Laboratory

Wednesday May 7, 2025 2:15pm - 2:35pm CDT

The typical filesystem image formats for Charliecloud are SquashFS and tar archives. SquashFS is a compressed, read-only filesystem that unprivileged users can mount in user space with SquashFUSE; it is the preferred image format due to its various efficiencies. The previous SquashFS workflow was non-ideal due to user complexity and difficulties with HPC job schedulers.

We have designed a workflow that requires us to link SquashFUSE to Charliecloud to enable the new mount/unmount procedure of only needing a single user command. Also, an additional persistent process is needed to service the FUSE requests called the FUSE loop: once the containerized application process finishes, it unmounts the SquashFS and ends the FUSE loop. Our SquashFS workflow is currently in production with it being more user friendly, cleans up after itself and is more compatible with HPC job schedulers. We were able to reduce user commands from 3 to 1, increase reliability and decrease mount/unmount time by more than 50%.

LA-UR-25-22140

Speakers

Megan Phinney

Scientist, Los Alamos National Laboratory

Wednesday May 7, 2025 2:15pm - 2:35pm CDT
Illinois River

Charliecloud

2:35pm CDT

Key Charliecloud Innovation - seccomp - Reid Priedhorsky, Los Alamos National Laboratory

Wednesday May 7, 2025 2:35pm - 2:55pm CDT

Do Linux distribution package managers need the privileged operations they request to actually happen? Apparently not, at least when building container images for HPC applications. Charliecloud uses this observation to implement a root emulation mode using a Linux seccomp filter that intercepts some privileged system calls, does nothing, and returns success to the calling program. This approach provides no consistency whatsoever but is sufficient to build a wide selection of Dockerfiles, including some that Docker itself cannot build, simplifying fully-unprivileged workflows needed for HPC application containers. This talk will detail the approach along its advantages and disadvantages.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Wednesday May 7, 2025 2:35pm - 2:55pm CDT
Illinois River

Charliecloud

2:55pm CDT

Key Charliecloud Innovation - CDI - Reid Priedhorsky, Los Alamos National Laboratory

Wednesday May 7, 2025 2:55pm - 3:15pm CDT

An ongoing challenge for HPC containers is how to make host resources such as GPU devices and proprietary interconnects performantly available inside a container. In Charliecloud, the key requirement is placing shared libraries (.so files) into the container and then running ldconfig(8) to update its cache. (Other implementations also have to deal with device files and their associated permissions and symlinks, but because Charliecloud bind-mounts host /dev into the container, this is not needed).

Charliecloud has done this for some time using ch-fromhost(1), which is a large, reverse-engineered shell script that copies the needed files into a writeable image. It is difficult to maintain, does not support SquashFS or other write-only images, and adds a workflow step.

Other implementations typically use “OCI hooks”, which are arbitrary vendor-provided or custom programs run at various phases during container setup. These also present maintainability / bit-rot problems, can be opaque, and because their interface is solely “do it for me”, any invalid assumptions that hooks make can be difficult or impossible to work around.

A different approach is the emerging Container Device Interface (CDI) standard, with contributors from nVidia, Intel, Los Alamos, and others. This gives prescriptive JSON/YAML descriptions of what is needed. Charliecloud has implemented CDI in its runtime ch-run(1), bind-mounting requested files and using an unprivileged tmpfs overlay (available since Linux 5.11 in February 2021) to avoid modifying the image. This is a considerably simpler and more maintainable way to make host resources available inside a container.

This talk will provide an overview of CDI, adaptations of the standard to our fully unprivileged workflow, and our C implementation. We will also demonstrate the functionality for nVidia GPUs and HPC/Cray Slingshot interconnect.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Wednesday May 7, 2025 2:55pm - 3:15pm CDT
Illinois River

Charliecloud

3:40pm CDT

Charliecloud Office Hours - Reid Priedhorsky, Los Alamos National Laboratory

Wednesday May 7, 2025 3:40pm - 5:00pm CDT

Members of the Charliecloud team will be available for office hours to listen to feedback/suggestions, answer questions, and/or help debug issues.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Wednesday May 7, 2025 3:40pm - 5:00pm CDT
Illinois River

Charliecloud

9:00am CDT

Breaking Charliecloud News - Reid Priedhorsky, Los Alamos National Laboratory

Thursday May 8, 2025 9:00am - 9:20am CDT

This session will cover late-breaking developments in Charliecloud, such as recent/upcoming new features, notable bugs, and requests for feedback. The specific agenda is TBD.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Thursday May 8, 2025 9:00am - 9:20am CDT
Illinois River

Charliecloud

9:20am CDT

Deploying AI Chatbot Assistants with Charliecloud - Jemma Stachelek, Los Alamos National Laboratory

Thursday May 8, 2025 9:20am - 10:00am CDT

Additional Authors: Tolulope Olatunbosun, Phil Romero & Mike Mason, Los Alamos National Laboratory

Retrieval Augmented Generation (RAG) systems improve the response relevance of LLMs (Large Language Models) by limiting the context to a document corpus. RAG systems have seen broad deployment as document summarization engines and AI chatbots. However, deploying these systems often assumes a privileged and “cloudy” environment with multi-container orchestration (i.e. docker compose) and unfettered internet access to pull resources (e.g. software, data, and models) on-the-fly. As an alternative, we leveraged Charliecloud’s NVIDIA GPU support capabilities to deploy a RAG chatbot in an unprivileged HPC environment where resources are pre-staged. We demonstrate the deployment of AI Chatbots using Charliecloud on a variety of hardware and software versioning.

LA-UR-25-21968

Speakers

Jemma Stachelek

Scientist, Los Alamos National Laboratory

Thursday May 8, 2025 9:20am - 10:00am CDT
Illinois River

Charliecloud

10:00am CDT

Using Charliecloud to Wrap HTCondor Worker Nodes - Oliver Freyermuth, University of Bonn (Germany)

Thursday May 8, 2025 10:00am - 10:20am CDT

This talk will present a setup using Charliecloud to spawn virtual HTCondor compute nodes inside of jobs submitted to a SLURM cluster. The actual containers are distributed via CernVM-FS and mounted unprivilegedly at the HPC site using cvmfsexec. The spawned HTCondor nodes integrate into a larger overlay batch system to run High-Throughput compute jobs from the Worldwide LHC Computing Grid community.

Charliecloud allows to make this setup very portable with its lightweight design, minimal system dependencies and simplicity of use. CernVM-FS which is optimized for distribution of large numbers of small files of which only few might be accessed proves an ideal fit for distribution of directory-format container images. In combination with HTCondor which focuses on optimizing the total throughput and easily handles large numbers of jobs, compute resources can be used opportunistically by integrating them fully unprivilegedly into an overlay batch system. The workloads themselves can again use unprivileged containers to enable the use of user-defined software stacks.

Speakers

Oliver Freyermuth

Research Scientist for IT Operations and High Throughput Computing, University of Bonn (Germany)

Thursday May 8, 2025 10:00am - 10:20am CDT
Illinois River

Charliecloud

10:45am CDT

Charliecloud + Gitlab-CI: Building and Using System-Representative Base Containers - Nick Sly, Lawerence Livermore National Laboratory

Thursday May 8, 2025 10:45am - 11:25am CDT

Charliecloud is used in conjunction with Gitlab-CI to build out a matrix of system-representative containers that can be used for building target system-compatible binaries for automated building and testing production codes on NNSA lab machines. This presentation covers the method of generating the base containers as well as a couple of use cases where they have proven helpful.

Speakers

Nick Sly

Scientist, Lawerence Livermore National Laboratory

Thursday May 8, 2025 10:45am - 11:25am CDT
Illinois River

Charliecloud

11:25am CDT

Maintaining the Debian Charliecloud Package - Peter Wienemann, Independent

Thursday May 8, 2025 11:25am - 12:05pm CDT

Charliecloud has been in the Debian archive since the early development days of Charliecloud. It was initially packaged by Lucas Nussbaum at the end of 2017/beginning of 2018. The speaker joined the packaging effort in January 2018 and continuously contributed to it since then. This talk will give a brief introduction into Debian and describe how its tool set was useful to improve the Debian Charliecloud package and feed improvements back into the upstream project. But also the information flow from upstream authors to package maintainers has been exemplary. This presentation will provide a few examples showing this fruitful interplay.

Speakers

Peter Wienemann

Independent

Thursday May 8, 2025 11:25am - 12:05pm CDT
Illinois River

Charliecloud

1:35pm CDT

Key Charliecloud Innovation - Kubernetes - Angelica Loshak, Los Alamos National Laboratory

Thursday May 8, 2025 1:35pm - 1:55pm CDT

Kubernetes automates container deployment and management across environments. HPC users can benefit from Kubernetes to support the increasing demand for novel workflows, especially in AI. Kubernetes' declarative approach allows users to schedule, scale, and maintain metrics on containers while supporting multiple container runtimes. Charliecloud can enhance HPC workloads when integrated with Kubernetes. However, Kubernetes only supports container runtimes that implement the Container Runtime Interface (CRI), which Charliecloud does not. To address this, we developed a prototype CRI-compatible server for Charliecloud, allowing Kubernetes to manage pods and to create, start, and track Charliecloud containers. Despite Kubernetes expecting certain features that Charliecloud does not use, such as network namespaces, we show that the two systems can still communicate effectively. Our implementation requires 700 lines of new code, fewer than 50 lines of modification to Charliecloud, and no changes to Kubernetes. This demonstrates that Kubernetes and Charliecloud are compatible tools, advancing scientific workflows that require large compute power.

LA-UR-24-28252

Speakers

Angelica Loshak

Student, Los Alamos National Laboratory

Thursday May 8, 2025 1:35pm - 1:55pm CDT
Illinois River

Charliecloud

1:55pm CDT

BEE: Orchestrating Workflows with Containerized Applications Leveraging Charliecloud - Krishna Chilleri, Los Alamos National Laboratory

Thursday May 8, 2025 1:55pm - 2:15pm CDT

Build and Execution Environment (BEE) is a workflow orchestration system designed to build containerized HPC applications and orchestrate workflows across HPC and cloud systems. BEE integrates with existing tools and infrastructure in the scientific computing ecosystem, making it an ideal choice for large-scale scientific simulations. The use of these tools and standards allows for efficient management of the provisioning, scheduling, and monitoring of individual tasks, as well as providing flexibility, portability, and reproducibility of workflows. This presentation will highlight how BEE leverages Charliecloud—as one of its container runtimes—to facilitate unprivileged builds, pull images from registries, and run containerized applications.

LA-UR-25-22166

Speakers

Krishna Chilleri

Student, Los Alamos National Laboratory

Thursday May 8, 2025 1:55pm - 2:15pm CDT
Illinois River

Charliecloud

2:15pm CDT

Charliecloud Office Hours - Reid Priedhorsky, Los Alamos National Laboratory

Thursday May 8, 2025 2:15pm - 3:15pm CDT

Members of the Charliecloud team will be available for office hours to listen to feedback/suggestions, answer questions, and/or help debug issues.

LA-UR-25-22140

Speakers

Reid Priedhorsky

Scientist, Los Alamos National Laboratory

I am a staff scientist at Los Alamos National Laboratory. Prior to Los Alamos, I was a research staff member at IBM Research. I hold a Ph.D. in computer science from the University of Minnesota and a B.A., also in computer science, from Macalester College.My work focuses on large-scale... Read More →

Thursday May 8, 2025 2:15pm - 3:15pm CDT
Illinois River

Charliecloud