Setup containerized Jupyter server for Jupyter-JSC

This Jupyter notebook will explain how to setup a containerized Jupyter server at Jupyter-JSC. It makes use of the expert features described on page 22 (as of November 25th, 2020) of the training material available here. Please note, that setting up a containerized Jupyter server for the JupyterHub at JSC might introduce certain drawbacks to your Jupyter-JSC experience. Specifically, you will be restricted to the software environment that is installed in your container environment only, which might introduce unwanted side-effects to your JupyterLab-based workflows on the JSC HPC systems. For example, usage of the SLURM batch scheduler commands is not possible, because the SLURM libraries are not visible from the container environment per default. Also, you won't be able to use the Lmod software environment modules provided by JSC. Please note, that if these kind of side-effects are not acceptable, you might rather use a containerized Jupyter kernel as described here. You could also setup your own non-containerized JupyterLab server.

Please note, you can switch back to the default Jupyter-JSC server environment anytime by deleting $HOME/.jupyter/start_jupyter-jsc.sh after login to the JSC systems via SSH.

Requirements:

container group access for the JSC systems as described here in the docs
Jupyter server environment in a Docker (or Singularity) container
- install at least the python packages jupyterhub and jupyterlab.

Specify the filesystem location that stores the Singularity container image,

IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook

Convert the example Jupyter base-notebook (that is available via Dockerhub) into a Singularity container image,

mkdir -p ${IMAGE_TARGET_DIR}

singularity pull --force ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log

cat singularity.log | grep -v warn

INFO:    Using cached SIF image

Check that the Singularity image is available,

ls -lah ${IMAGE_TARGET_DIR}

total 177M
drwxr-sr-x 2 hoeflich1 cesmtst 4.0K Jan 19 18:50 .
drwxr-sr-x 5 hoeflich1 cesmtst 4.0K Jan 19 18:05 ..
-rwxr-xr-x 1 hoeflich1 cesmtst 183M Jan 19 18:50 jupyter-base-notebook.sif

Now, manually (!) specify the Singularity image filesystem location in the start_jupyter-jsc.sh script and check that the specified path is correct,

cat start_jupyter-jsc.sh

#!/bin/bash

# Author: Katharina Höflich
# Repository: https://github.com/FZJ-JSC/jupyter-jsc-notebooks

SINGULARITY_IMAGE=/p/project/cesmtst/hoeflich1/jupyter-base-notebook/jupyter-base-notebook.sif
JUPYTERJSC_USER_CMD="singularity exec ${SINGULARITY_IMAGE} jupyterhub-singleuser --config ${JUPYTER_LOG_DIR}/config.py"

And copy the start_jupyter-jsc_singularity.sh script to the filesystem location expected by Jupyter-JSC (Link to start_jupyter-jsc_singularity.sh)

cp start_jupyter-jsc_singulartiy.sh $HOME/.jupyter/start_jupyter-jsc.sh

Finally, opening a new Jupyter session via the Jupyter-JSC control panel should now load the containerized Jupyter server that was setup here.