Skip to content

June 20, 2019 • qnib • 5 min read

ISC2019: 5th Annual High Performance Container Workshop

100% human-written — no AI tools are used to write these posts.

container workshop isc

The ‘5th Annual High Performance Container Workshop’ workshop was held as part of the International Supercomputing Conference in Frankfurt on June 20nd from 9AM to 6PM at the Marriott Hotel.

Agenda

The first half of the day was spend with introducing the speakers, providing an overview and discuss the topics which are not exclusively HPC specific, but are fundamentals that are also important in non-HPC use cases: Which runtime fits my use-case? How to build my container image? How to distribute the artefacts? Depending on my use-case, discipline, vertical - what should I focus on and what is less important?

Segments

A complete Youtube playlist can be found here

Intro

Youtube Video / Combined Slides

#StartTitleSpeakerCompanyLinks
009:00WelcomeChristian KniepQNIB SolutionsVideo/Slides
1Intro UberCloudBurak YenierUberCLoudVideo/Slides
209:10Intro NVIDIACJ NewburnNVIDIAVideo/Slides
309:15Intro SylabsMichael BauerSylabsVideo/Slides
409:20Intro AWSArthur PetitpierreAWSVideo/Slides
509:25Intro MellanoxDror GoldenbergMellanoxVideo/Slides
609:30Intro RedHatValentin RothbergRedHatVideo/Slides
709:35Workshop Overview, Segments and PersonasChristian KniepQNIB SolutionsVideo/Slides

Runtime

Youtube Video / Combined Slides

| # | Start | Title | Speaker | Company | Links | | --- | ----- | ---------------------------------- | ----------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------- | --- | --- | | 0 | 09:40 | Introduction and Scope | Christian Kniep | QNIB Solutions | Video/Slides | | 1 | 09:45 | Current State of root-less dockerd | Akihiro Suda | NTT | Video/Slides | | 2 | 09:50 | The podman runtime | Valentin Rothberg | RedHat | Video/Slides | | 3 | 10:00 | The Singularity runtime | Michael Bauer | Sylabs | Video/Slides | | 4 | 10:05 | The SARUS runtime | Lucas Benedicic | CSCS | Video/Slides | | 5 | 10:10 | PANEL: Q&A | | | Video | | | 11:00 | Coffee Break | | | | | |

Build

Youtube Video / Combined Slides

#StartTitleSpeakerCompanyLinks
011:30Introduction and ScopeChristian KniepQNIB SolutionsVideo/Slides
111:35Rootless build with BuildKitAkihiro SudaNTTVideo/Slides
211:40Buildah, a tool that facilitates building OCI imagesValentin RothbergRedHatVideo/Slides
311:45Singularity buildMichael BauerSylabsVideo/Slides
411:50Optimize for hardware again!Christian KniepQNIB SolutionsVideo/Slides
511:55Tools: NVIDIA HPC Container MakerCJ NewburnNVIDIAVideo/Slides
612:00Build Tools like SPACK/EasyBuildMassimiliano CulpoEPFLVideo/Slides
712:05Panel: Q&AVideo

Distribute

Youtube Video / Combined Slides

| # | Start | Title | Speaker | Company | Links | | --- | ----- | ---------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | 0 | 12:20 | 12:20 | Introduction and Scope | Christian Kniep | QNIB Solutions | Video/Slides | | 1 | 12:25 | OCI Image Spec | Akihiro Suda | Video/Slides | | 2 | 12:30 | Singularity Image Format | Michael Bauer | Sylabs | Video/Slides | | 3 | 12:35 | Skopeo Distribution Tool | Valentin Rothberg | RedHat | Video/Slides | | 4 | 12:40 | Hardware Optimized Images via MetaHub Registry Proxy | Christian Kniep | QNIB Solutions | Video/Slides | | | 12:45 | PANEL: Q&A | | | Video | | | 13:00 | Lunch Break | | | |

Orchestration/Scheduling

Youtube Video / Combined Slides

#StartTitleSpeakerCompanyLinks
014:00Introduction and ScopeChristian KniepQNIB SolutionsVideo/Slides
114:01Simple Orchestration with SWARMAbdulrahman AzabUniversity of OsloVideo/Slides
214:05Recap on KubernetesDaniel GruberUberCloudVideo/Slides
314:15Nextflow to model (bioinformatic) workloadsPaolo Di TommasoCRGVideo/Slides
414:20Lustre within KubernetesArthur PetitpierreAWSVideo/Slides
514:23Using K8s operators for containerized RDMA workloadsDror GoldenbergMellanoxVideo/Slides
614:25Slurm Operator for KubernetesMichael BauerSylabsVideo/Slides
714:30AWS BatchArthur PetitpierreAWSVideo/Slides
14:35PANEL: Q&AVideo

Infrastructure

Youtube Video / Combined Slides

#StartTitleSpeakerCompanyLinks
015:15Introduction and ScopeChristian KniepQNIB SolutionsVideo/Slides
115:16OpenStack Update and DirectionMartial MichelData Machines CorpVideo/Slides
215:20Dynamic HPC in a cloud environmentArthur PetitpierreAWSVideo/Slides

HPC Specific

Youtube Video / Combined Slides

| # | Start | Title | Speaker | Company | Links | | --- | ----- | --------------------------------------------------------- | ------------------ | -------------- | ------------------------------------------------------------------------------------------------------------ | --- | | 0 | 15:25 | Introduction and Scope | Christian Kniep | QNIB Solutions | Video/Slides | | 1 | 15:27 | How AWS blends fast POSIX (Lustre) and object stores (S3) | Arthur Petitpierre | AWS | Video/Slides | | 2 | 15:30 | RDMA Device Isolation | Dror Goldberg | Mellanox | Video/Slides | | 3 | 15:35 | PANEL: Q&A | | | Video | | | 16:00 | Coffee Break | | | | |

Use-Cases / Conclusions

Youtube Video / Combined Slides

#StartTitleSpeakerCompanyLinks
116:30RDMA-GPU use-caseDror GoldenbergMellanoxVideo/Slides
216:40Mellanox Containerization JourneyDror GoldenbergMellanoxVideo/Slides
316:55Looking back on 5y of containerizationBurak YenierUberCloudVideo/Slides
417:15NERSC: Looking backShane CanonNERSCVideo/Slides
517:30NVIDIAs journey with ContainersCJ NewburnNVIDIAVideo/Slides
17:50PANEL: Q&AVideo
18:00Workshop Ending

Abstract

Linux Containers continue to gain momentum within data centers all over the world. They are able to benefit legacy infrastructures by leveraging the lower overhead compared to traditional, hypervisor-based virtualization. But there is more to Linux Containers, which this workshop will explore. Their portability, reproducibility and distribution capabilities outclass all prior technologies and disrupt former monolithic architectures, due to sub-second life cycles and self-service provisioning.

This workshop will outline the current state of Linux Containers in HPC/AI, what challenges are hindering the adoption in HPC/BigData and how containers can foster improvements when applied to the field of HPC, Big Data and AI in the mid- and long-term. By dissecting the different layers within the container ecosystem (runtime, supervision, engine, orchestration, distribution, security, scalability) this workshop will provide a holistic and a state-of-the-container overview, so that participants can make informed discussions on how to start, improve or continue their container adoption.

Workshop History