Add DCGM exporter pod metadata enrichment API

[karthikvetrivel]

Description

Resolves #2009.

Previously, there was no way of enabling pod metrics when using DCGM exporter through GPU Operator. This fix introduces introduces first-class Helm values that provision RBAC and allow pod metrics to include Pod UID & Pod Label.

Design Choices:

• Introduced enabling pod metrics as first-class Helm values instead of making arbitrary env vars provision RBAC to match the upstream DCGM metrics helm chart.
• I intentionally did not add resourceSlices RBAC. Standalone dcgm-exporter Helm includes resourceSlices because it also supports DRA-related Kubernetes enrichment. GPU Operator is only exposing pod metadata enrichment here because it does not support DRA yet. I am open to changing this.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Added unit tests.
Confirmed the operator reconciled the expected DCGM exporter resources:

• nvidia-dcgm-exporter DaemonSet rolled out successfully.
• DCGM exporter container received:
  - DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS=true
  - DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID=true
  - DCGM_EXPORTER_KUBERNETES_POD_LABEL_ALLOWLIST_REGEX=^gpu_test_label$
  • Pod template had automountServiceAccountToken: true.
  • nvidia-dcgm-exporter-read-pods ClusterRole and ClusterRoleBinding were created.
• Exporter service account could get, list, and watch pods cluster-wide.

Scraped DCGM exporter metrics from a labeled GPU pod and confirmed metrics included gpu_test_label="live-test" and pod_uid="..."

[rahulait]

LGTM

[tariq1890]

Is this a TODO comment?

[karthikvetrivel]

This was just to explain why there isn't parity with the upstream DCGM Exporter clusterrole, which contains resource slices. Can remove if you'd like?

[tariq1890]

Yes, either add a TODO comment with the intended change in the future or just remove

[karthikvetrivel]

Updated to be a TODO!

[tariq1890]

This feels a bit too terse. Can we expand on this a bit more? Let's also clarify that this setting adds pod label as a label dimension to the DCGM exporter prometheus metrics.

Also, the "Requires cluster-level read access to pods" is a bit unnecessary IMO. If we do want to mention this, can we enclose it in parenthesis?

[karthikvetrivel]

Yep, expanded on the label and enclosed "Requires cluster-level read access to pods" in parentheses.