feat(observability): restore Flux reconciliation alerts in Coroot profile

[devantler]

🤖 Generated by the Agentic Engineer (Codex instance)

Evidence

The production Coroot profile still renders kube-prometheus-stack, and its validator and operator guide say that Flux reconciliation alerting remains dependent on that legacy stack. The current reference Platform has already replaced the Kustomization half of FluxKustomizationNotReady with Flux notification-controller resources that reuse the existing Slack webhook. The template has no matching flux-notifications resources.

Audience and problem

Operators who select the production Coroot profile need to learn when a Flux Kustomization cannot reconcile. Coroot observes workloads but cannot see failed GitOps builds, applies, dependencies, or readiness waits. Until this safety net exists independently, the profile cannot retire the legacy Prometheus Kustomization alert without losing a deployment-failure signal.

Hypothesis and smallest useful change

Add a production-Coroot-only Flux notification profile through the existing alertmanager_webhook_url. One Alert forwards every Kustomization error event; a second, narrowly filtered info-severity Alert covers dependency waits, which Flux does not emit as errors. Apply both in the controller layer so the path exists before that layer finishes waiting, and narrow the legacy Prometheus rule to HelmRelease readiness so a Kustomization failure is not sent twice.

Acceptance criteria

• The ordinary local and production profiles remain byte-for-byte equivalent in rendered inventory and do not gain Flux notification objects.
• The local Coroot profile remains notification-free.
• The production Coroot controller profile renders exactly one Slack Provider, one webhook Secret, one all-Kustomization error Alert, and one dependency-wait-only info Alert.
• The notification objects are applied in the controller layer, before that Flux Kustomization finishes waiting for controller readiness.
• The Provider references only the rendered webhook Secret, whose address comes from the existing alertmanager_webhook_url substitution with a validation-safe placeholder.
• The production Coroot infrastructure profile narrows the legacy Flux Prometheus rule to HelmRelease readiness; no Kustomization failure is routed through both paths.
• Validation rejects missing, duplicated, broadened, relocated, or miswired notification resources and proves both Coroot provider renders still build.
• Operator documentation explains both Flux failure paths, the unchanged secret path, the deduplication boundary, and the remaining reason kube-prometheus-stack is still transitional.

Success signal and window

At merge, both Coroot profile renders pass the fail-closed observability contract while default renders remain unchanged. This closes one named blocker to retiring kube-prometheus-stack; parent #65 stays open for the remaining alert-parity and retirement slices.

Rough size

S

[coderabbitai]

🔗 Related PRs

#62 - feat(coredns): migrate docker-provider CoreDNS from Deployment to HelmRelease [merged]
#86 - feat(cluster-policies): consume shared helm remediation policy [merged]
#99 - feat(observability): preserve Coroot incident notifications [merged]
#101 - feat(observability): stage Coroot cluster heartbeat [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!