Explore/mps enforcement - #2
Draft
cgetzen wants to merge 2 commits into
Draft
Conversation
Add opt-in NVIDIA MPS VRAM (and optional SM%) enforcement for KAI fractional GPU pods, scoped so ONLY workload/fraction pods use MPS while reservation and Slurm pods keep using the physical device directly. Shared contract (pkg/common/mps): pipe/log dir layout keyed by gpu-group, CUDA_MPS_* env names, and value formatters (pinned mem = fraction x node VRAM; active-thread % ~ fraction x 100). Client wiring (mirrors hamicore), gated on RequestsGPUFraction AND the existing `mps: "true"` annotation: - admission plugin (pkg/admission/webhook/v1alpha2/mps): injects CUDA_MPS_* env (ConfigMapKeyRef, optional) + hostPath pipe-dir mount; enabled via --mps-enabled. - binder plugin (pkg/binder/plugins/mps): at PreBind writes pipe dir (from bindRequest SelectedGPUGroups) and pinned-mem limit into the shared-gpu ConfigMap. Registered as a default (disabled) plugin at priority 40; requires gpusharing to run before it. Daemon host = the GPU reservation pod (its lifecycle == "GPU in fractional use"; it already holds the whole GPU): --reservation-mps-enabled adds a hostPath mount + env, and the reservation entrypoint starts a per-GPU nvidia-cuda-mps-control daemon (pkg/resourcereservation/mps). Compute mode left at DEFAULT so MPS stays opt-in per client. Operator threads --mps-enabled / --reservation-mps-enabled from the mps plugin enablement. Unit tests for all new packages; go build ./... and touched-package tests pass (envtest integration suite still needs KUBEBUILDER_ASSETS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion image
The reservation ("capacity") pod hosts the per-GPU MPS control daemon
(nvidia-cuda-mps-control -d) so KAI fraction pods can connect as MPS clients
and get a hard VRAM cap (CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). Two fixes surfaced
by the slurm-staging E2E:
- applyMPSHosting now overrides the reservation container's security context to
root (RunAsUser 0, RunAsNonRoot false, AllowPrivilegeEscalation true). The
daemon must mkdir its per-GPU pipe/log dirs on the shared root-owned hostPath
and serve client contexts; under the default non-root context (UID 10000, all
caps dropped) it failed with "mkdir .../pipe: permission denied". Scoped to
MPS-hosting reservations only — non-MPS reservations keep the configured
non-root context.
- Dockerfile.rrmps builds the resourcereservation binary on a glibc (ubuntu)
base instead of the distroless default, so the dynamically-linked,
toolkit-mounted nvidia-cuda-mps-control can run. The binary is CGO (go-nvml),
built with CGO_ENABLED=1.
E2E on slurm-staging: a gpu-fraction:0.5 + mps:true pod connects to the
KAI-orchestrated daemon (CUDA free/total 7580/15095 MiB) and OOMs at 7 GiB;
two fractions share one GPU/reservation/daemon; teardown reaps the daemon with
the reservation. Validated with the equivalent root hand-patch to the binder's
--resource-reservation-container-security-context flag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issues
Fixes #
Checklist
make changelog(or applied theskip-changeloglabel). Do not editCHANGELOG.mddirectly — pending fragments are folded into it at release time.Breaking Changes
Additional Notes