This Jupyter notebook will walk you through the installation of a containerized Jupyter kernel (for use at Jupyter-JSC, but it should actually work with any Jupyter server on a system where Singularity is installed). Considerable performance improvements (especially with respect to kernel start-up times) over e.g. conda-based Jupyter kernels on distributed filesystems, as are typically installed on HPC systems, might be experienced. In the example below, the base-notebook from the Jupyter docker stacks is used as an IPython kernel (already having the required ipykernel package installed), the approach presented here might be extended to any other Jupyter kernel compatible programming language, though.
Requirements:
- Python environment with an installed
ipykernelpackage in a Docker (or Singularity) container containergroup access for the JSC systems as described here in the docs
Check that the Singularity container runtime is available via the JupyterLab environment,
singularity --version
Specify the filesystem location that stores the Singularity container image,
IMAGE_TARGET_DIR=/p/project/cesmtst/hoeflich1/jupyter-base-notebook
Optional, if you already have a Singularity container image available at the above location: Convert a containerized Python environment (e.g. the Jupyter base-notebook that is available via Dockerhub) into a Singularity container image to be used as an example here,
mkdir -p ${IMAGE_TARGET_DIR}
Note that pulling and converting the Dockerhub image will take a bit of time,
singularity pull ${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif docker://jupyter/base-notebook &> singularity.log
cat singularity.log | grep -v warn
Check that the Singularity image is available,
ls -lah ${IMAGE_TARGET_DIR}
Now, setup a Jupyter kernel specification with the install-jupyter-kernel.sh script from this repository (which basically writes a kernel.json file to the home directory location that Jupyter expects for user-specific kernels),
KERNEL_DISPLAY_NAME=Singularity-Python # don't use whitespaces here!
SINGULARITY_IMAGE=${IMAGE_TARGET_DIR}/jupyter-base-notebook.sif
./install-singularity-jupyter-kernel.sh ${KERNEL_DISPLAY_NAME} ${SINGULARITY_IMAGE}
Check that the Jupyter kernel specification was written,
cat ${HOME}/.local/share/jupyter/kernels/${KERNEL_DISPLAY_NAME}/kernel.json
And that the above Singularity-Python kernel is visible by the Jupyter server,
jupyter kernelspec list
If so, you should be able to choose and connect to the containerized Python kernel from the drop down menu and/or the kernel launcher tab (a reload of the JupyterLab web page might be necessary).