Skip device plugin alert when devicePlugin is disabled in ClusterPolicy

[harche]

Summary

When devicePlugin.enabled is set to false in the ClusterPolicy, the nvidia-node-status-exporter still monitors gpu_operator_node_device_plugin_devices_total, which reports 0 (since no device plugin pods are running). This triggers a false positive GPUOperatorNodeDeploymentFailed alert after 30 minutes.

This is a valid configuration — for example, when GPU allocation is managed externally via MIG partitioning through a third-party operator.

Changes

• controllers/object_controls.go: The operator now injects a DEVICE_PLUGIN_ENABLED env var into the node-status-exporter daemonset based on config.DevicePlugin.IsEnabled().
• cmd/nvidia-validator/metrics.go: When DEVICE_PLUGIN_ENABLED=false, the exporter skips device plugin validation and sets the gauge to -1 (documented sentinel value), preventing the == 0 alert from firing.
• assets/state-node-status-exporter/0800_prometheus_rule_openshift.yaml: Updated comment to document the behavior.
• controllers/transforms_test.go: Unit tests for env var injection (enabled + disabled).
• tests/e2e/helpers/clusterpolicy.go: EnableDevicePlugin/DisableDevicePlugin helpers.
• tests/e2e/suites/clusterpolicy_test.go: E2E tests verifying env var propagation to node-status-exporter daemonset.

Test plan

• Unit tests pass (TestTransformNodeStatusExporter)
• Code compiles (go build ./cmd/nvidia-validator/ and go build ./controllers/)
• E2E tests compile (go build ./tests/e2e/...)
• Manual validation on hardware with MIG-capable GPUs and devicePlugin.enabled: false

Fixes: #2237

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[harche]

Initial code review looks good and unit tests pass. Keeping this as a draft, hardware testing with MIG-capable GPUs and devicePlugin.enabled: false is pending before marking ready for review.

[harche]

Verified the fix on OpenShift 4.21.5 / GCP with NVIDIA T4

Environment:

• OpenShift 4.21.5 on GCP (us-central1)
• GPU Operator v26.3.1 baseline, patched with this PR
• NFD 4.21.0-202604200440 (redhat-operators)
• Node: n1-standard-4 with 1x NVIDIA Tesla T4

How I tested:

1. Built the PR branch (6b55de9) on-cluster using an OpenShift BuildConfig — multi-stage build replacing gpu-operator and nvidia-validator binaries in the v26.3.1 base image.

2. Reproduced the bug first with the unpatched operator:

   • Set devicePlugin.enabled: false in ClusterPolicy
   • Metric dropped to 0, alert entered pending state:

     gpu_operator_node_device_plugin_devices_total{node="...gpu-worker-a-zdd2q"} 0
     GPUOperatorNodeDeploymentFailed: state=pending
     

3. Deployed the patched image to the nvidia-node-status-exporter daemonset with DEVICE_PLUGIN_ENABLED=false env var injected:

   gpu_operator_node_device_plugin_devices_total{node="...gpu-worker-a-zdd2q"} -1
   GPUOperatorNodeDeploymentFailed: state=inactive, activeAlerts=0
   

Results:

	Before fix	After fix
DEVICE_PLUGIN_ENABLED env var	absent	false
gpu_operator_node_device_plugin_devices_total	0	-1 (sentinel)
Alert expr == 0 matches	yes	no
GPUOperatorNodeDeploymentFailed	pending	inactive

Fix works as expected. No operator errors observed during testing.

[shivamerla]

Alternatively we can also check for both gpu_operator_node_plugin_ready == 1 and gpu_operator_node_device_plugin_devices_total == 0 before firing an alert, but the caveat is that if the plugin pod itself is crashing, then we would not raise an alert which is not desired. So current approach looks good.

[shivamerla]

/ok-to-test 71be3d1

[copy-pr-bot]

/ok-to-test

@shivamerla, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

[shivamerla]

/ok-to-test 71be3d1