Observability: dashboards, alerts, and a way to scrape at all

[wrkode]

Closes #7.

The twelve gpucell_* series existed and were documented — and nothing could consume them. There was no Service in front of the metrics port and no container port declared, so the endpoint was unreachable. The pack starts by fixing that.

Then, behind monitoring.enabled (off by default, because it needs the Prometheus Operator CRDs and a default install should not depend on them):

• a ServiceMonitor
• a four-row Grafana dashboard, delivered as a sidecar-discovered ConfigMap
• eight warning-biased alerts

All three are individually disableable, and the value plumbing is nil-safe so an explicit enabled: false still wins under helm upgrade --reuse-values (where the subtree is not merged and sprig's default would swallow the false).

The alerts keep the two layers apart

This is the whole reason the metrics are shaped the way they are:

GPUCellPoolNoFreePhysicalGPU	the infrastructure cluster has no whole device left and this pool wants one
GPUCellPoolSharedGPUExhausted	the cells are healthy and their GPU is entirely handed out to HAMi workloads

Different problems, different remedies; an alert that merged them would be useless. The physical one is deliberately conjoined with "the pool actually wants a cell" — a saturated cluster whose pools are all satisfied is not a fault, and alerting on it trains people to ignore the rule.

GPUCellPoolCapacityStale earns its place for a subtler reason: a failed capacity read retains the previous values rather than reporting zero, because "0 GPUs, plenty free" from a failed scrape would be the worst possible lie. When it fires, every capacity number on the dashboard is untrustworthy.

Verified, not eyeballed

• The chart renders in all three modes — monitoring off (zero monitoring.coreos.com objects), on, and explicitly-partial.
• Every metric name, label selector and by() grouping across all 25 alert and panel expressions was cross-checked against what internal/metrics/metrics.go actually declares. This is a real trap, not a theoretical one: the design doc had already drifted to {pool} where the code emits {pool, namespace}.
• Dashboard JSON is generated, so it is valid by construction.

Dashboards live in config/grafana/ with the chart copy synced by make manifests (Helm can only package files inside the chart directory); drift between them fails make verify.

One security claim corrected

values.yaml said the metrics endpoint serves "HTTPS with authn/authz". It does not — no FilterProvider is set, so it is HTTPS with a self-signed certificate and no authorization at all. The claim is corrected and the real posture documented (plus how to restrict it).

I did not just enable the filter: controller-runtime/pkg/metrics/filters imports k8s.io/apiserver, which pulls a large dependency tree into an operator whose first design principle is avoiding exactly that. That is a trade to make deliberately, so it is filed as #11 rather than slipped into a dashboards PR.

make test, make lint and make verify pass.

🤖 Generated with Claude Code