-
Notifications
You must be signed in to change notification settings - Fork 35
refactor(providers): share Kubernetes accelerator discovery #462
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,7 +1,8 @@ | ||||||||||||||
| # DRA Topology Provider | ||||||||||||||
|
|
||||||||||||||
| The DRA provider reads existing `nvidia.com/gpu.clique` Kubernetes node labels | ||||||||||||||
| generated by the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)'s | ||||||||||||||
| The DRA provider reads accelerator-domain IDs from an existing Kubernetes Node | ||||||||||||||
| label and defaults to `nvidia.com/gpu.clique`, generated by the | ||||||||||||||
| [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)'s | ||||||||||||||
| [GPU Feature Discovery (GFD)](https://github.com/NVIDIA/k8s-device-plugin/blob/main/docs/gpu-feature-discovery/README.md), | ||||||||||||||
| specifically its IMEX labeler, and groups nodes by NVLink partition. Its | ||||||||||||||
| supported use is generating Slurm `topology/block` data with the Slinky engine. | ||||||||||||||
|
|
@@ -25,41 +26,60 @@ The DRA provider supplies the label-to-Slinky block-topology bridge for this eco | |||||||||||||
| Use the DRA provider only when all of the following are true: | ||||||||||||||
|
|
||||||||||||||
| - You are using Slinky (Slurm-on-Kubernetes) with `topology/block` | ||||||||||||||
| - Every participating node already has a valid `nvidia.com/gpu.clique` label | ||||||||||||||
| - Every participating node already has a valid value in the configured accelerator-domain label | ||||||||||||||
| - Workloads fit within one NVLink partition, or you accept that placement across multiple partitions will not account for backend-fabric locality | ||||||||||||||
|
|
||||||||||||||
| The label is deployment- and state-dependent; it is not guaranteed to exist in every MNNVL GPU Operator installation. If it is absent, the provider cannot derive partition membership. If you need the switch hierarchy or topology-aware selection across partitions, use the [InfiniBand](./infiniband.md) or [NetQ](./netq.md) provider instead. | ||||||||||||||
|
|
||||||||||||||
| ## How It Works | ||||||||||||||
|
|
||||||||||||||
| The DRA provider does not create `nvidia.com/gpu.clique`. Before selecting this provider, verify that the GPU Operator exposes the label on every participating node. | ||||||||||||||
| The DRA provider does not create its source label. Before selecting this provider, verify that the configured label exists on every participating node. When `provider.params.accelerator` is omitted, the source label defaults to `nvidia.com/gpu.clique` for backward compatibility. | ||||||||||||||
|
|
||||||||||||||
| Topograph reads these labels from the Kubernetes API: | ||||||||||||||
|
|
||||||||||||||
| 1. Lists all nodes (filtered by `nodeSelector` if provided) | ||||||||||||||
| 2. For each node with a `nvidia.com/gpu.clique` label, reads the clique ID and groups nodes by domain | ||||||||||||||
| 2. Uses the shared Kubernetes-label accelerator discoverer to read the configured Node label and group nodes by domain | ||||||||||||||
| 3. Returns the NVLink domain map as block topology | ||||||||||||||
|
|
||||||||||||||
| If no nodes with matching labels are found, Topograph returns a `502` error with a diagnostic message indicating which label and annotations to check. | ||||||||||||||
|
|
||||||||||||||
| ## Prerequisites | ||||||||||||||
|
|
||||||||||||||
| - A Slinky (Slurm-on-Kubernetes) cluster configured to use `topology/block` | ||||||||||||||
| - A valid `nvidia.com/gpu.clique` label already present on every participating Kubernetes node | ||||||||||||||
| - A valid accelerator-domain label already present on every participating Kubernetes node | ||||||||||||||
|
|
||||||||||||||
| ## Parameters | ||||||||||||||
|
|
||||||||||||||
| | Parameter | Type | Required | Description | | ||||||||||||||
| |---|---|---|---| | ||||||||||||||
| | `nodeSelector` | `map[string]string` | No | Label selector to filter which nodes participate in topology discovery | | ||||||||||||||
| | `accelerator` | `object` | No | Shared accelerator discovery configuration. When omitted, DRA reads `nvidia.com/gpu.clique`. | | ||||||||||||||
| | `accelerator.source` | `string` | With `accelerator` | Must be `kubernetes-label`; DRA is an accelerator-only label provider. | | ||||||||||||||
| | `accelerator.kubernetesLabel.key` | `string` | With `accelerator` | Kubernetes Node label read as the accelerator-domain ID. | | ||||||||||||||
|
|
||||||||||||||
| ## Configuration | ||||||||||||||
|
|
||||||||||||||
| Deploy Topograph with the Helm chart and select the DRA provider and Slinky | ||||||||||||||
| engine through the chart's `provider` and `engine` values. Set the optional | ||||||||||||||
| `nodeSelector` under `provider.params`. The chart manages the Topograph | ||||||||||||||
| configuration and topology request payload; they do not need to be supplied | ||||||||||||||
| separately. | ||||||||||||||
| source label and `nodeSelector` under `provider.params`. The chart manages the | ||||||||||||||
| Topograph configuration and topology request payload, so you do not need to | ||||||||||||||
| supply either one separately. | ||||||||||||||
|
Comment on lines
+64
to
+66
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Describe the optional configuration accurately. Line 64 calls this an “optional source label.” Proposed wording-Set the optional source label and `nodeSelector` under `provider.params`.
+Set the optional accelerator configuration and `nodeSelector` under `provider.params`.Cross-file evidence: 📝 Committable suggestion
Suggested change
🧰 Tools🪛 LanguageTool[grammar] ~64-~64: Ensure spelling is correct (QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1) 🤖 Prompt for AI Agents |
||||||||||||||
|
|
||||||||||||||
| ```yaml | ||||||||||||||
| provider: | ||||||||||||||
| name: dra | ||||||||||||||
| params: | ||||||||||||||
| accelerator: | ||||||||||||||
| source: kubernetes-label | ||||||||||||||
| kubernetesLabel: | ||||||||||||||
| key: nvidia.com/gpu.clique | ||||||||||||||
| nodeSelector: | ||||||||||||||
| nvidia.com/gpu.present: "true" | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| The `accelerator` object has the same shape as the Kubernetes-label source for | ||||||||||||||
| `infiniband-k8s`. DRA supports only that source because it intentionally | ||||||||||||||
| produces accelerator block domains without discovering a network fabric. | ||||||||||||||
|
|
||||||||||||||
| Configure Kubernetes client limits with the chart-wide settings: | ||||||||||||||
|
|
||||||||||||||
|
|
@@ -85,13 +105,13 @@ for the available Helm values. | |||||||||||||
|
|
||||||||||||||
| ## Verifying the Output | ||||||||||||||
|
|
||||||||||||||
| Before triggering topology generation, verify that clique labels exist on all participating nodes: | ||||||||||||||
| Before triggering topology generation, verify that the configured labels exist on all participating nodes. For the default label: | ||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
| kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"]}' | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| If topology generation returns a `502` error, check that the expected nodes have the `nvidia.com/gpu.clique` label and the `topograph.nvidia.com/region` / `topograph.nvidia.com/instance` annotations (the latter two are set by Topograph itself during topology discovery): | ||||||||||||||
| If topology generation returns a `502` error, check that the expected nodes have the configured source label and the `topograph.nvidia.com/region` / `topograph.nvidia.com/instance` annotations (the latter two are set by Topograph itself during topology discovery). For the default label: | ||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
| kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"], region: .metadata.annotations["topograph.nvidia.com/region"], instance: .metadata.annotations["topograph.nvidia.com/instance"]}' | ||||||||||||||
|
|
||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
Add positive DRA validation coverage and run the chart tests.
These tests cover only rejection paths. Add or verify cases for omitted
provider.params.accelerator, which must preserve thenvidia.com/gpu.cliquedefault, and explicitsource: kubernetes-labelwithkubernetesLabel.key, which must pass.As per coding guidelines, “Run Helm chart tests when changing
charts/topograph/; review snapshot changes before committing updates.”Also applies to: 136-145
🤖 Prompt for AI Agents
Source: Coding guidelines