Skip to content

Blog

HPC3: Expected Container Behaviour

The last blog post introduced the HPC Container Conformance (HPC3) project - a project to provide guidance on how to build and annotate container images for HPC use cases.

For HPW we'll need to cover two main parts (IMHO) first:

  1. Entrypoint/Cmd relationship: How do interactive users and batch systems expect a container to behave. We need to make sure that a container works with docker run, singularity run and podman run out of the box (engine configuration already done)
  2. Annotation Baseline: Which annotations do we need and want, some are mandatory and some are optional.

This blob post is going to set a baseline in Terms of Expected Container Behavior to make sure that we can switch HPC3 conformant images of the same application and - ideally - have the job ran in the same way.

The HPC Container Conformance Project

A lot of HPC sites have an unwritten understanding of how to use them:

  1. You need a user to log in. Either on the submithost or when a job launches on the compute nodes,
  2. Once logged in, your environment is setup with default apps, specific application are available using module load from a central software share,
  3. you might be even able to install new software in your home directory.

A central software-share with curated sets of scientific applications and libraries is/was a powerful concept. But with containers this falls a part to some degree... 😦

Buildkit Dockerfile Frontend Caching

Ok, where was I? In the last blog post (BuildKit Dockerfile Frontend) scratching the surface by introcuding here-docs in Dockerfiles. The original inspiration for the post - series, as it turned out - was my to cut the cristiangreco/docker-pdflatex image down by a GB or so.
I went from this:

COPY install.sh /install.sh
RUN sh /install.sh && rm /install.sh

to this:

RUN <<eot bash
  apt-get update
  apt-cache depends texlive-full \
  | grep "Depends:" \
  | grep -v "doc$" \
  | egrep -v "texlive-(games|music)" \
  | egrep -v "texlive-lang-(arabic|cjk|chinese|cyrillic|czechslovak|european|french|german|greek|italian|japanese|korean|other|polish|portuguese|spanish)$" \
  | cut -d ' ' -f 4 \
  | xargs apt-get install --no-install-recommends -y
  apt-get autoremove
eot

BuildKit Dockerfile Frontend

I am a big fan of pandoc to generate documents instead of using a binary format like MS Word. Feels more natural to have a source and compile it into a delivarable.

Dockerfile Improvement

Installing LaTex and all the dependencies for pandoc is a hassle and the natural way of dealing with that is (of course) to create a container. I found the container cristiangreco/docker-pdflatex on the interweb.

But the image was rather big (4.4GB) and I thought:

Let's see if we can shave of a GB or so and while we are at it make the Dockerfile more readable

Blog Relaunched!

In one of my older post from 2019 I wrote:

I won't say "Long time, no post" - but...

After a long(-ish) neglect of the blog I am happy to relaunch the blog now! I am intending to write blog post regulary and blog about container-in-HPC, monitoring, GOLANG and what else comes into mind. I migrated everything from Jekyll to mkdocs-material and had to touch every blog post. What a journey I want through the last ten years. The first blog post is from 2012...

Let's see what the next 10 years are going to be! 🥳

ISC2021: 7th Annual High Performance Container Workshop

The workshop took place on July 2nd.

The High Performance Container Workshop series assembles thought leaders to provide the 'state of containers' and the latest trends.

Virtual HPCW 2021

As ISC2021 moved to a virtual event, we went virtual as well with a mix of pre-recorded short videos and a live session. Last year we dove deep into each aspect (runtime, build, distribute, schedule, HPC specifics, outlook) in three sittings. We highly recommend going through the youtube recordings. They can be found in last years post: qnib.org/2020/06/17/isc2020-hpcw.

ISC2020: 6th Annual High Performance Container Workshop

The High Performance Container Workshop series assembles thought leaders to provide the 'state of containers' and the latest trends.

Virtual Workshop 2020

As ISC20 moved to a virtual event, we went virtual as well.

Instead of having one long sitting, we broke the workshop into three sessions that shared a similar structure:

  • Firstly, a hand full experts and thought leaders provided their insights and expertise in a short and crisp 10min lightning talk. This filled the first half (~45min) and went back to back, without time to do Q&A.
  • Second - and this was really the important and insightful part - all speakers were available for a panel and virtual Q&A.

ISC2019: 5th Annual High Performance Container Workshop

The '5th Annual High Performance Container Workshop' workshop was held as part of the International Supercomputing Conference in Frankfurt on June 20nd from 9AM to 6PM at the Marriott Hotel.

Agenda

The first half of the day was spend with introducing the speakers, providing an overview and discuss the topics which are not exclusively HPC specific, but are fundamentals that are also important in non-HPC use cases: Which runtime fits my use-case? How to build my container image? How to distribute the artefacts? Depending on my use-case, discipline, vertical - what should I focus on and what is less important?

Metahub: Dynamic Registry Proxy

I won't say "Long time, no post" - but...

As I had some time at my hands the last couple of months, I was iterating on my idea on hardware optimization using manifest list from the last post Match Node-Specific Needs Using Manifest Lists.

ReCap

The gist is that hardware optimization with containers is a bit of a step back, as the kernel virtualization (aka containers) promises to provide isolation on-top of a Linux (or Windows) kernel without caring to much about the underlying host configuration.