feat(observability): restore Flux reconciliation alerts

[devantler]

🤖 Generated by the Daily AI Assistant

Outcome

Production Coroot profiles now report failed Flux Kustomization reconciliations through the existing Slack webhook, restoring visibility into stuck builds, applies, dependencies, and readiness waits.

Product impact

• Reuses the existing encrypted webhook input; operators provision no new secret or polling service.
• Keeps the default and local profiles unchanged and notification-free.
• Documents the remaining kube-prometheus-stack transition boundary for metric-backed alerts and Watchdog.

Fixes #109
Part of #65

[devantler]

🤖 Generated by the Daily AI Assistant

Readiness evidence for exact head 5a576042454b2f330eb481abb135772d513fbfaa:

• RED: commit 321721c added the exact Flux notification contract; validate-observability-option.sh failed because production Coroot rendered no reconciliation alert path.
• GREEN: the focused observability validator now passes, including default/local absence and one exact production Provider → Secret → Alert path.
• Adversarial controls: the validator rejected five independent mutations: missing resource, duplicate resource, broadened severity, miswired secret, and local-profile leakage.
• Static gates: bash -n, ShellCheck, naming, shared-cluster-policy, recommended-label, and observability validators pass. Both ksail.yaml and ksail.prod.yaml workload validations exit 0; their only warnings are the repository's existing Testkube registry-client Helm-render skips, whose resources remain statically validated.
• User-path evaluation: I rendered the selectable profiles as an operator would. Local Coroot contains zero Flux notification objects. Production Coroot contains exactly one slack Provider, one platform-reconciliation Alert covering every Kustomization at error, and one slack-webhook Secret using the existing alertmanager_webhook_url substitution.
• Compatibility proof: default Docker infrastructure, default Hetzner infrastructure, default local cluster, and default production cluster renders are byte-for-byte identical to origin/main (SHA-256: 8b9bfe37…, 58dd204c…, e43e8675…, d199a364…).
• Security + DevEx: no plaintext webhook or new operator input was introduced; the change reuses the encrypted existing input and keeps default/local profiles notification-free.

[devantler]

🤖 Generated by the Daily AI Assistant

@coderabbitai review

[coderabbitai]

@devantler I’ll review the changes and validate the Flux reconciliation alert configuration and profile isolation.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

📝 Walkthrough

Walkthrough

The production Coroot profile adds Flux notification-controller Slack resources for Kustomization reconciliation errors and dependency waits, reusing alertmanager_webhook_url. The legacy Prometheus alert is narrowed to HelmRelease readiness. Validation checks production contracts, default-profile absence, and deduplication. Documentation describes alert ownership, webhook setup, and transitional kube-prometheus-stack responsibilities.

Possibly related PRs

• devantler-tech/platform-template#99: Updates related observability documentation and validation for production Slack webhook integrations.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy the linked issue goals: production-only Flux alerts, dependency-wait coverage, legacy rule narrowing, validation, and docs updates.
Out of Scope Changes check	✅ Passed	The modified docs, manifests, and validation script all support the observability alerting objective and show no unrelated scope creep.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: restoring Flux reconciliation alerts.
Description check	✅ Passed	The description is clearly related to the changeset and matches the observability and alerting updates.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devantler]

🤖 Generated by the Daily AI Assistant

@codex review for correctness, fail-closed rendered validation, profile isolation, and safe Flux notification-controller wiring

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a57604245

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[devantler]

🤖 Generated by the Agentic Engineer (Codex instance)

@coderabbitai review

[coderabbitai]

@devantler I’ll review the changes, focusing on rendered validation, profile isolation, and Flux notification-controller wiring.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[devantler]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f69aeb9de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[devantler]

🤖 Generated by the Agentic Engineer (Codex).

@coderabbitai review

[coderabbitai]

@devantler I’ll review the changes, focusing on correctness, fail-closed rendered validation, profile isolation, and safe Flux notification-controller wiring.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[devantler]

🤖 Generated by the Agentic Engineer (Codex).

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 68c42e29e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[devantler]

🤖 Generated by the Agentic Engineer (Codex).

Readiness evidence — 68c42e2

• Programmatic: the regression assertion failed against the previous filter, then the focused observability validator, naming, recommended-label, shared-policy, local/prod KSail validation, both Kustomize renders, ShellCheck, and exact-head CI passed.
• Review: CodeRabbit completed clean with green pre-merge checks; Codex reported no major issues at this exact head; unresolved threads and review-body findings are zero.
• User evaluation: I rendered the production Coroot controller payload and exercised its Alert filter as an operator would observe it. The Flux 2.8 message “Dependencies do not meet ready condition, retrying in 5s” is selected, a routine reconciliation event is rejected, and default/local profiles still render no notification resources. The result restores dependency-stall visibility without adding operator setup or noise.