SSH port forward as an alternative to mesh/wstunnel for reaching services on air gapped HPC

[matbun]

What we are trying to do

We use interLink to transparently offload Kubeflow workloads (notebooks and KFP
pipeline steps) from a Kubernetes cluster to two HPC systems, CERN HPC and
MareNostrum5 (BSC). A common need is reaching a service that runs inside an
offloaded pod on an HPC compute node, for example a Jupyter server, from the
user browser.

The current answer: mesh / wstunnel

interLink ships a mesh setup (WireGuard over wstunnel) or just wstunnel for explicit port forward. As we
understand it, that design assumes two things:

• the HPC compute node has outbound internet to reach the tunnel server, and
• the Kubernetes cluster can expose a public ingress gateway.

In our environment neither holds:

• MareNostrum5 is air gapped. No node has internet, not even the login nodes. Also other HPCs rely on network-segregated compute nodes.
• Our Kubernetes cluster does not expose any public endpoint.

Our approach: SSH port forwarding

Instead of the HPC pod dialing out to a public gateway, we let the cluster dial
in to the HPC. We reuse interLink's existing shadow pod rendering, but replace the
mesh workload with a plain ssh -L from a cluster pod to the HPC login node,
forwarding on to the compute node:

browser → Kubeflow UI → K8s Service → shadow pod (ssh -L) → HPC login node → compute node (Jupyter)

The only requirement is outbound SSH from the k8s cluster to the HPC login node. No
compute node internet, no public ingress. The shadow init container also learns
which compute node the job landed on (from a small file the job writes) and
points the forward at it.

Tradeoffs

• It works when the cluster initiates the connection: notebooks and web UIs,
  which is the common case.
• It does not cover an offloaded pod calling back into Kubernetes (for example
  object storage). Not a priority for us today.
• It breaks if the login node enforces 2FA. That would need a trusted endpoint
  that accepts connections without 2FA.

Validated in a fully in cluster deployment

We run this as a fully in cluster interLink deployment: every component (the
virtual node, the interLink API server, the SLURM plugin, and the SSH and sshfs
sidecars) runs as pods on the Kubernetes side. Nothing we install runs on the HPC.
The plugin sbatch, squeue, scancel and sinfo calls are thin shims that
ssh to the login node on demand, and the SSH port forward is a connection opened
from the cluster (alternatively, we could submit SLURM commands to a remote
SLURM cluster using SLURM's REST API).

On top of that deployment, users offload to HPC with no code changes:

• single pod offload to CERN HPC,
• Jupyter notebooks offloaded and reached exactly through this SSH port forward
  shadow,
• KFP pipeline steps offloaded via an @hpc_offload decorator,
• a single node, multi GPU PyTorch TrainJob (Kubeflow Trainer v2) offloaded to
  MareNostrum5 (2 H100 GPUs, DDP).

The in-cluster model is itself an adoption advantage: onboarding a new HPC needs
only a user account and outbound SSH to the login node. There is no long running
service to install, run, or keep alive on the login node, and therefore no lengthy
negotiation with the HPC sysadmins to deploy and maintain a service. That is a low
barrier to bringing a new site online, and the SSH port forward preserves it: the
connectivity path adds no HPC side service either.

Question for the maintainers

Would interLink consider supporting an SSH tunnel connectivity mode as a first
class alternative to mesh / wstunnel, for restricted or air gapped sites?

Thank you!

[antoinetran]

Just a message to say that our project LISA, which will also rely on the BSC airgap cluster, are definitely interested. Our use-case is a bit different though: most of our services are deployed on kubernetes cluster, and we offload scientific jobs to compute node. Normally we don't expect jupyter notebook to run in compute nodes, but I can see this might be something we have to think of.

It does not cover an offloaded pod calling back into Kubernetes (for example
object storage). Not a priority for us today.

For us we do use object storage / message bus service, and since BSC does not provide one, these are deployed in kubernetes and needs to be reached from compute nodes.

[antoinetran]

@matbun if a ssh forward tunnel works (if allowed) from login to compute, what would prevent one to do a ssh reverse tunnel from compute to login, and login to Kubernetes cluster, so that the wireguard can access the interlink api? That would allow the full Interlink VPN to work, so bi-directional access to services.

[dciangot]

I'd really like to see this as an alternative to the edge deployment. This way, there wiil be only to options we'll guarantee to be maintainer:

• edge
• ssh

covering most of the use cases.
About the plan. I propose to start talking about this at the next Community hour, Mon 13 at 11:00 with whoever will be able to attend.

Then we can start dropping a list of action items here with related issues. Thumbs up if agreed! @antoinetran @matbun