Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Added

- The DRA provider now accepts the shared `provider.params.accelerator` Kubernetes-label configuration used by `infiniband-k8s`, allowing a custom Node label to supply accelerator domains while preserving `nvidia.com/gpu.clique` as the default when the section is omitted.
- Pluggable accelerator-domain discovery for InfiniBand providers, independently selectable from fabric discovery with `nvidia-smi`, an explicitly configured Kubernetes Node label, or no accelerator source. Discovery is disabled when `accelerator` is omitted or empty; a non-empty section must set `source` explicitly. Helm defaults the `nvidia-smi` workload location to the `gpu-operator` namespace and `nvidia-device-plugin-daemonset` DaemonSet when those values are omitted.
- Helm `kubeClient.qps` and `kubeClient.burst` values for tuning the DRA provider and the Kubernetes, NFD, and Slinky engine clients through deployment-level `KUBE_QPS` and `KUBE_BURST` settings.
- The Kubernetes engine now publishes `accelerator.topograph.run/sub-domain` when a provider supplies `InstanceTopology.XclrSubDomainID`.
Expand Down Expand Up @@ -70,6 +71,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Removed

- **BREAKING:** Removed the exported Go constant `topology.KeyNvidiaGPUClique`. The GPU Operator's `nvidia.com/gpu.clique` label is now a DRA provider default rather than part of the canonical topology API; configure label-backed accelerator discovery through `provider.params.accelerator.kubernetesLabel.key`.
- Periodic node annotation refreshes, including the `--refresh-interval` broker flag and `nodeDataBroker.refreshInterval` Helm value. The broker now applies annotations once at startup.

### Security
Expand Down
8 changes: 6 additions & 2 deletions charts/topograph/templates/_validation.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
{{- fail "env.KUBE_BURST is managed by the chart; configure kubeClient.burst instead" }}
{{- end }}

{{- if or (eq .Values.provider.name "infiniband-k8s") (eq .Values.provider.name "infiniband-bm") }}
{{- if or (eq .Values.provider.name "infiniband-k8s") (eq .Values.provider.name "infiniband-bm") (eq .Values.provider.name "dra") }}
{{- $params := default dict .Values.provider.params }}
{{- $acceleratorValue := get $params "accelerator" }}
{{- $accelerator := default dict $acceleratorValue }}
Expand All @@ -49,7 +49,7 @@
{{- fail (printf "unsupported provider.params.accelerator.source %q" $source) }}
{{- end }}

{{- if and (eq .Values.provider.name "infiniband-k8s") (eq $source "kubernetes-label") }}
{{- if and (or (eq .Values.provider.name "infiniband-k8s") (eq .Values.provider.name "dra")) (eq $source "kubernetes-label") }}
{{- $kubernetesLabel := default dict (get $accelerator "kubernetesLabel") }}
{{- $key := trim (toString (get $kubernetesLabel "key")) }}
{{- if eq $key "" }}
Expand All @@ -61,6 +61,10 @@
{{- fail "provider.params.accelerator.source kubernetes-label is not supported by infiniband-bm" }}
{{- end }}

{{- if and (eq .Values.provider.name "dra") (hasKey $params "accelerator") (ne $source "kubernetes-label") }}
{{- fail "provider.params.accelerator.source must be kubernetes-label for the dra provider" }}
{{- end }}

{{- end }}

{{- if eq .Values.provider.name "gcp" }}
Expand Down
22 changes: 22 additions & 0 deletions charts/topograph/tests/validation_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,25 @@ tests:
asserts:
- failedTemplate:
errorMessage: "provider.params.accelerator.kubernetesLabel.key must be set for source kubernetes-label"

- it: rejects a DRA label accelerator source without a key
set:
provider:
name: dra
params:
accelerator:
source: kubernetes-label
asserts:
- failedTemplate:
errorMessage: "provider.params.accelerator.kubernetesLabel.key must be set for source kubernetes-label"
Comment on lines +125 to +134

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add positive DRA validation coverage and run the chart tests.

These tests cover only rejection paths. Add or verify cases for omitted provider.params.accelerator, which must preserve the nvidia.com/gpu.clique default, and explicit source: kubernetes-label with kubernetesLabel.key, which must pass.

As per coding guidelines, “Run Helm chart tests when changing charts/topograph/; review snapshot changes before committing updates.”

Also applies to: 136-145

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@charts/topograph/tests/validation_test.yaml` around lines 125 - 134, Add
positive validation cases in the chart tests for an omitted
provider.params.accelerator preserving the nvidia.com/gpu.clique default and for
source kubernetes-label with kubernetesLabel.key passing validation. Run the
Helm chart tests and review any snapshot changes before committing.

Source: Coding guidelines


- it: rejects a non-label accelerator source for DRA
set:
provider:
name: dra
params:
accelerator:
source: none
asserts:
- failedTemplate:
errorMessage: "provider.params.accelerator.source must be kubernetes-label for the dra provider"
12 changes: 7 additions & 5 deletions charts/topograph/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@ provider:
# params:
# accelerator:
# # Accelerator-domain discovery is independent of fabric discovery.
# # Sources supported by infiniband-k8s: nvidia-smi,
# # kubernetes-label, and none. Omitting the accelerator section disables
# # accelerator discovery; an empty accelerator section is equivalent to
# # source: none. A non-empty section must set source explicitly.
# # Sources supported by infiniband-k8s: nvidia-smi, kubernetes-label,
# # and none. The dra provider supports kubernetes-label and defaults to
# # nvidia.com/gpu.clique when this section is omitted. For InfiniBand,
# # omitting accelerator disables discovery; an empty section is
# # equivalent to source: none. A non-empty section must set source.
# source: kubernetes-label
# kubernetesLabel:
# # Required for the kubernetes-label source; there is no default key.
# # Required for the kubernetes-label source. DRA defaults this key to
# # nvidia.com/gpu.clique only when the accelerator section is omitted.
# key: nvidia.com/gpu.clique
# nvidiaSmi:
# # Optional; defaults used by the Helm-managed node-data-broker are
Expand Down
4 changes: 4 additions & 0 deletions demos/dra-slinky/values.dra-slinky.kwok.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
provider:
name: dra
params:
accelerator:
source: kubernetes-label
kubernetesLabel:
key: nvidia.com/gpu.clique
nodeSelector:
kwok.x-k8s.io/node: "fake"
engine:
Expand Down
2 changes: 2 additions & 0 deletions demos/oci-sim-slinky/demo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ cd "$demo_dir/../.."

source demos/utils.sh

step "make build TARGETS=kwok-nodes"

step "delete_cluster"

step "kind create cluster --name \"${KUBE_CONTEXT#kind-}\" --wait 120s"
Expand Down
6 changes: 3 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ Topograph exposes three endpoints for interacting with the service. Below are th
- **name**: (optional) A string specifying the Service Provider, such as `aws`, `oci`, `gcp`, `nebius`, `nscale`, `netq`, `dra`, `infiniband-k8s`, `infiniband-bm` or `test`. This parameter will override the provider set in the topograph config.
- **creds**: (optional) A key-value map with provider-specific parameters for authentication.
- **params**: (optional) A key-value map with provider-specific parameters. The `test` provider uses these parameters for response simulation; for complete behavior and examples, see [Test Mode and Test Provider](./providers/test.md).
- **accelerator**: (optional) Used in: [`infiniband-k8s`, `infiniband-bm`]. Configures accelerator-domain discovery independently of network-fabric discovery. Omitting this section or setting it to an empty object disables accelerator-domain discovery.
- **source**: (required when `accelerator` is non-empty) `nvidia-smi`, `kubernetes-label` (`infiniband-k8s` only), or `none`. An empty object is equivalent to `source: none`.
- **kubernetesLabel.key**: (required for `kubernetes-label`) Kubernetes Node label read as the accelerator-domain ID. No default is assumed.
- **accelerator**: (optional) Used in: [`dra`, `infiniband-k8s`, `infiniband-bm`]. Configures accelerator-domain discovery independently of network-fabric discovery. For InfiniBand, omitting this section or setting it to an empty object disables accelerator-domain discovery. DRA supports only `kubernetes-label` and retains its legacy `nvidia.com/gpu.clique` default when the section is omitted.
- **source**: (required when `accelerator` is non-empty) `nvidia-smi`, `kubernetes-label` (`dra` and `infiniband-k8s`), or `none`. DRA accepts only `kubernetes-label`.
- **kubernetesLabel.key**: (required for `kubernetes-label`) Kubernetes Node label read as the accelerator-domain ID. No default is assumed for an explicit section.
- For `infiniband-k8s`, a request with `source: nvidia-smi` reads accelerator-domain annotations previously collected by the node-data-broker. The request does not run `nvidia-smi` or reconfigure the broker. Deploy the broker with the same accelerator source before sending the request; see [Helm node-data-broker settings](./providers/infiniband.md#helm-node-data-broker-settings).
- **engine**: (optional) Selects the topology output and provides any engine-specific parameters.
- **name**: (optional) A string specifying the topology output, either `slurm`, `k8s`, `nfd`, `slinky`, or `graph`. This parameter will override the engine set in the topograph config.
Expand Down
4 changes: 2 additions & 2 deletions docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Currently supported providers:
- [Nscale](./providers/nscale.md)
- [Lambda](./providers/lambdai.md)
- [NetQ](./providers/netq.md)
- [DRA](./providers/dra.md) — provides Slinky block topology from pre-existing `nvidia.com/gpu.clique` labels; it does not discover the backend switch fabric
- [DRA](./providers/dra.md) — provides Slinky block topology from a configured pre-existing Node label (default `nvidia.com/gpu.clique`); it does not discover the backend switch fabric
- [InfiniBand (bare-metal)](./providers/infiniband.md#infiniband-bm-bare-metal)
- [InfiniBand (Kubernetes)](./providers/infiniband.md#infiniband-k8s-kubernetes)
- [Test](./providers/test.md) - simulates Topograph success, pending, and error responses for integration testing
Expand All @@ -66,7 +66,7 @@ Currently supported engines:
| InfiniBand fabric, no NetQ, Kubernetes | [InfiniBand (Kubernetes)](./providers/infiniband.md) |
| Client integration and regression testing | [Test](./providers/test.md) |

The DRA provider is a narrow Slinky integration, not a general Kubernetes MNNVL topology provider. It groups nodes into Slurm `topology/block` domains from existing `nvidia.com/gpu.clique` labels. Those labels are not present in every GPU Operator deployment, and DRA does not discover the backend switch fabric between NVLink partitions. Consequently, if a workload cannot fit in one partition, selection of additional partitions is not informed by network proximity. Use NetQ or `infiniband-k8s` when cross-partition fabric locality is required.
The DRA provider is a narrow Slinky integration, not a general Kubernetes MNNVL topology provider. It groups nodes into Slurm `topology/block` domains from an existing Node label, defaulting to `nvidia.com/gpu.clique`. That label is not present in every GPU Operator deployment, and DRA does not discover the backend switch fabric between NVLink partitions. Consequently, if a workload cannot fit in one partition, selection of additional partitions is not informed by network proximity. Use NetQ or `infiniband-k8s` when cross-partition fabric locality is required.

For non-MNNVL GPU clusters (such as DGX B200 or B300 SuperPODs), `nvidia.com/gpu.clique` is not set — Topograph with an InfiniBand provider is the only source of network topology for scheduling decisions on these systems.

Expand Down
42 changes: 31 additions & 11 deletions docs/providers/dra.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# DRA Topology Provider

The DRA provider reads existing `nvidia.com/gpu.clique` Kubernetes node labels
generated by the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)'s
The DRA provider reads accelerator-domain IDs from an existing Kubernetes Node
label and defaults to `nvidia.com/gpu.clique`, generated by the
[NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html)'s
[GPU Feature Discovery (GFD)](https://github.com/NVIDIA/k8s-device-plugin/blob/main/docs/gpu-feature-discovery/README.md),
specifically its IMEX labeler, and groups nodes by NVLink partition. Its
supported use is generating Slurm `topology/block` data with the Slinky engine.
Expand All @@ -25,41 +26,60 @@ The DRA provider supplies the label-to-Slinky block-topology bridge for this eco
Use the DRA provider only when all of the following are true:

- You are using Slinky (Slurm-on-Kubernetes) with `topology/block`
- Every participating node already has a valid `nvidia.com/gpu.clique` label
- Every participating node already has a valid value in the configured accelerator-domain label
- Workloads fit within one NVLink partition, or you accept that placement across multiple partitions will not account for backend-fabric locality

The label is deployment- and state-dependent; it is not guaranteed to exist in every MNNVL GPU Operator installation. If it is absent, the provider cannot derive partition membership. If you need the switch hierarchy or topology-aware selection across partitions, use the [InfiniBand](./infiniband.md) or [NetQ](./netq.md) provider instead.

## How It Works

The DRA provider does not create `nvidia.com/gpu.clique`. Before selecting this provider, verify that the GPU Operator exposes the label on every participating node.
The DRA provider does not create its source label. Before selecting this provider, verify that the configured label exists on every participating node. When `provider.params.accelerator` is omitted, the source label defaults to `nvidia.com/gpu.clique` for backward compatibility.

Topograph reads these labels from the Kubernetes API:

1. Lists all nodes (filtered by `nodeSelector` if provided)
2. For each node with a `nvidia.com/gpu.clique` label, reads the clique ID and groups nodes by domain
2. Uses the shared Kubernetes-label accelerator discoverer to read the configured Node label and group nodes by domain
3. Returns the NVLink domain map as block topology

If no nodes with matching labels are found, Topograph returns a `502` error with a diagnostic message indicating which label and annotations to check.

## Prerequisites

- A Slinky (Slurm-on-Kubernetes) cluster configured to use `topology/block`
- A valid `nvidia.com/gpu.clique` label already present on every participating Kubernetes node
- A valid accelerator-domain label already present on every participating Kubernetes node

## Parameters

| Parameter | Type | Required | Description |
|---|---|---|---|
| `nodeSelector` | `map[string]string` | No | Label selector to filter which nodes participate in topology discovery |
| `accelerator` | `object` | No | Shared accelerator discovery configuration. When omitted, DRA reads `nvidia.com/gpu.clique`. |
| `accelerator.source` | `string` | With `accelerator` | Must be `kubernetes-label`; DRA is an accelerator-only label provider. |
| `accelerator.kubernetesLabel.key` | `string` | With `accelerator` | Kubernetes Node label read as the accelerator-domain ID. |

## Configuration

Deploy Topograph with the Helm chart and select the DRA provider and Slinky
engine through the chart's `provider` and `engine` values. Set the optional
`nodeSelector` under `provider.params`. The chart manages the Topograph
configuration and topology request payload; they do not need to be supplied
separately.
source label and `nodeSelector` under `provider.params`. The chart manages the
Topograph configuration and topology request payload, so you do not need to
supply either one separately.
Comment on lines +64 to +66

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Describe the optional configuration accurately.

Line 64 calls this an “optional source label.” provider.params.accelerator is optional, but DRA requires both accelerator.source and accelerator.kubernetesLabel.key when that object is present. Users can otherwise treat these fields as independently optional and submit a configuration that Helm rejects.

Proposed wording
-Set the optional source label and `nodeSelector` under `provider.params`.
+Set the optional accelerator configuration and `nodeSelector` under `provider.params`.

Cross-file evidence: charts/topograph/templates/_validation.tpl:43-67 requires the accelerator source and Kubernetes label key for this DRA configuration.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
source label and `nodeSelector` under `provider.params`. The chart manages the
Topograph configuration and topology request payload, so you do not need to
supply either one separately.
Set the optional accelerator configuration and `nodeSelector` under `provider.params`. The chart manages the
Topograph configuration and topology request payload, so you do not need to
supply either one separately.
🧰 Tools
🪛 LanguageTool

[grammar] ~64-~64: Ensure spelling is correct
Context: ...provider.params. The chart manages the Topograph configuration and topology request payl...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/providers/dra.md` around lines 64 - 66, Update the DRA provider
documentation to describe provider.params.accelerator as optional only as a
complete object, while stating that when present it must include both
accelerator.source and accelerator.kubernetesLabel.key. Remove wording that
implies the source label or nodeSelector fields are independently optional, and
keep the existing guidance that the chart manages the related Topograph
configuration and topology request payload.


```yaml
provider:
name: dra
params:
accelerator:
source: kubernetes-label
kubernetesLabel:
key: nvidia.com/gpu.clique
nodeSelector:
nvidia.com/gpu.present: "true"
```

The `accelerator` object has the same shape as the Kubernetes-label source for
`infiniband-k8s`. DRA supports only that source because it intentionally
produces accelerator block domains without discovering a network fabric.

Configure Kubernetes client limits with the chart-wide settings:

Expand All @@ -85,13 +105,13 @@ for the available Helm values.

## Verifying the Output

Before triggering topology generation, verify that clique labels exist on all participating nodes:
Before triggering topology generation, verify that the configured labels exist on all participating nodes. For the default label:

```bash
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"]}'
```

If topology generation returns a `502` error, check that the expected nodes have the `nvidia.com/gpu.clique` label and the `topograph.nvidia.com/region` / `topograph.nvidia.com/instance` annotations (the latter two are set by Topograph itself during topology discovery):
If topology generation returns a `502` error, check that the expected nodes have the configured source label and the `topograph.nvidia.com/region` / `topograph.nvidia.com/instance` annotations (the latter two are set by Topograph itself during topology discovery). For the default label:

```bash
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"], region: .metadata.annotations["topograph.nvidia.com/region"], instance: .metadata.annotations["topograph.nvidia.com/instance"]}'
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/node-labels.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ provider discovery.

Some GPU Operator deployments expose `nvidia.com/gpu.clique` on nodes with
Multi-Node NVLink (MNNVL) GPUs; operators may select it explicitly as either an
engine source label or an `infiniband-k8s` provider discovery label. The `netq`
engine source label or a DRA/`infiniband-k8s` provider discovery label. The `netq`
provider instead uses a `DomainUUID` from the NMX management API—a different
identifier that refers to the same physical domain but cannot be compared as a
string.
Expand Down
19 changes: 19 additions & 0 deletions internal/k8s/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,28 @@ import (
"k8s.io/client-go/tools/remotecommand"
"k8s.io/klog/v2"

internalconfig "github.com/NVIDIA/topograph/internal/config"
"github.com/NVIDIA/topograph/pkg/topology"
)

type nodeSelectorConfig struct {
NodeSelector map[string]string `mapstructure:"nodeSelector"`
}

// NodeListOptions decodes a provider's optional nodeSelector into Kubernetes
// list options. Other provider parameters are intentionally ignored.
func NodeListOptions(params map[string]any) (*metav1.ListOptions, error) {
config := nodeSelectorConfig{}
if err := internalconfig.Decode(params, &config); err != nil {
return nil, err
}
if len(config.NodeSelector) == 0 {
return nil, nil
}

return &metav1.ListOptions{LabelSelector: labels.Set(config.NodeSelector).String()}, nil
}

func GetNodes(ctx context.Context, client kubernetes.Interface, opt *metav1.ListOptions) (*corev1.NodeList, error) {
if opt == nil {
opt = &metav1.ListOptions{}
Expand Down
38 changes: 38 additions & 0 deletions internal/k8s/utils_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (

"github.com/stretchr/testify/require"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func TestIsPodReady(t *testing.T) {
Expand Down Expand Up @@ -81,6 +82,43 @@ func TestIsPodReady(t *testing.T) {
}
}

func TestNodeListOptions(t *testing.T) {
tests := []struct {
name string
params map[string]any
want *metav1.ListOptions
err string
}{
{name: "no parameters"},
{
name: "other provider parameters are ignored",
params: map[string]any{"accelerator": map[string]any{"source": "none"}},
},
{
name: "node selector",
params: map[string]any{"nodeSelector": map[string]string{"key": "value"}},
want: &metav1.ListOptions{LabelSelector: "key=value"},
},
{
name: "invalid node selector",
params: map[string]any{"nodeSelector": 0.1},
err: "could not decode configuration",
},
}

for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got, err := NodeListOptions(test.params)
if test.err != "" {
require.ErrorContains(t, err, test.err)
return
}
require.NoError(t, err)
require.Equal(t, test.want, got)
})
}
}

func TestValidateLabelKey(t *testing.T) {
testCases := []struct {
name string
Expand Down
Loading
Loading