Kubernetes test infrastructure for NVIDIA GPU software — mock GPU environments, CI tooling, and testing utilities.
Turn any Kubernetes cluster into a multi-GPU environment for testing. No physical NVIDIA hardware required.
# 1. Create cluster
kind create cluster --name mokka
# 2. Load the published image (or build locally with: docker build -t nvml-mock:local -f deployments/nvml-mock/Dockerfile .)
# The published image is multi-arch. `kind load docker-image` cannot load a
# multi-arch image from Docker Desktop's containerd store and fails with
# "ctr: content digest ...: not found", so save one platform first.
ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
docker pull ghcr.io/nvidia/nvml-mock:latest
docker save --platform "linux/${ARCH}" ghcr.io/nvidia/nvml-mock:latest -o nvml-mock.tar
kind load image-archive nvml-mock.tar --name mokka
# 3. Install
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mockAfter install, deploy a consumer to test:
| Consumer | Guide |
|---|---|
| NVIDIA Device Plugin | Quick Start |
| NVIDIA DRA Driver | Quick Start |
| NVIDIA GPU Operator | Quick Start |
Full documentation: nvml-mock Helm chart README
The nvml-mock Go E2E workflow gates standalone, DRA, GPU Operator, multi-node,
node-wide NRI, and NFD label-provenance coverage. Run manually via
workflow_dispatch or automatically on PRs.
| Test Suite | What It Validates | Profiles |
|---|---|---|
| Standalone Demo | nvml-mock chart install, nvidia-smi, NVLink/fabricmanager, InfiniBand, PCI sysfs, and cross-node checks |
Workflow-selected profiles |
| Failure Injection | Healthy, ECC, lost, and fallen-off-bus modes | Workflow-selected profiles |
| DRA Driver | Mock driver files, nvidia-smi, ResourceSlices, and DRA ResourceClaim scheduling |
Workflow-selected profiles |
| GPU Operator | GPU Operator install, validator pod startup, GFD labels, and allocatable GPUs | Workflow-selected profiles |
| Multi-Node Fleet | Heterogeneous A100/T4 workers, mock files, InfiniBand behavior, device plugin resources, and GPU workload scheduling | Fixed multi-node topology |
| Node-Wide NRI Injection | Ambient mock GPU injection into ordinary pods without GPU requests or hostPath mounts | Workflow-selected profiles |
| NFD Label Provenance | That NFD creates feature.node.kubernetes.io/pci-10de.present from the feature file nvml-mock writes, and that nvml-mock does not write the label itself |
Pinned to a100 — the label is vendor-only and byte-identical across profiles |
Manual dispatch accepts a JSON array of GPU profiles; local runs default to
gb200.
See .github/workflows/nvml-mock-e2e-go.yaml for details.
The underlying CGo-based mock libnvidia-ml.so that powers nvml-mock.
Use standalone for local development and CI pipelines.
| Document | Description |
|---|---|
| Overview | Project overview, components, GPU profiles |
| Quick Start | Build and run in 5 minutes |
| Configuration | YAML configuration reference |
| Architecture | System design and components |
| CUDA Mock | Mock CUDA library overview |
| Development | Contributing and extending the library |
| Examples | Usage patterns and scenarios |
| Troubleshooting | Common issues and solutions |
| Integration | Description | Guide |
|---|---|---|
| fake-gpu-operator | Run:ai's K8s-level GPU simulation | Integration Guide |
| Demo | Description |
|---|---|
| Standalone | nvml-mock with FGO-style labels on Kind |
| With fake-gpu-operator | Full FGO + nvml-mock integration |
- Logo designed by Roman Hlushko with the assistance of OpenAI's ChatGPT.
Apache License 2.0 — see LICENSE.