fix(redis): identify master by FQDN + self-labeling sidecar to prevent roll split-brain

[sd2k]

Draft / RFC. This is a complete proposal for discussion, not a merge-ready change. See Blast radius & splitting below — it almost certainly should land as several smaller PRs, and needs validation on a real cluster first.

Summary

Rolling the redis StatefulSet from this lib repeatedly caused production outages for the Grafana ML team: the redis-master Service ended up with zero endpoints (clients got connection refused) and, in the worst cases, a two-live-master data split-brain (one even served reverted data that crashlooped a downstream service). This PR fixes the chart's two root causes.

Context / incident write-up: grafana/machine-learning#6261 (ML have also filed it there because their image pin makes them the most-affected consumer).

Root causes

1. Identity by raw pod IP. init.sh/sentinel.conf monitor and announce by pod IP, with no announce-ip. A roll changes every pod IP, so Sentinel keeps monitoring dead "ghost" IPs → spurious failovers and split-brain.
2. Fragile label-based routing. The redis-master Service selects on a redis-role pod label that is set once at init and otherwise mutated cross-pod by a Sentinel client-reconfig-script (label.sh). When a reconfig event resolved to a stale/ghost IP, label.sh demoted the old master and then failed to promote the new one (the IP matched no pod) → zero pods labelled master → Service stranded.
3. Self-promotion on restart. init.sh could let a restarting pod (esp. index 0) self-promote when it couldn't reach the master during init → two masters.

What this changes

• Identify everything by stable StatefulSet FQDN, not pod IP. replica-announce-ip $(hostname -f), sentinel announce-ip $(hostname -f), sentinel resolve-hostnames yes / announce-hostnames yes, and sentinel monitor <group> <master-FQDN> …. FQDNs survive pod-IP churn, so rolls no longer create ghosts. Requires Redis ≥ 6.2 (the default image here is already redis:6.2-bullseye).
• No self-promotion on restart. init.sh discovers the current master's FQDN (Sentinel if reachable, else index 0) and configures the pod as replicaof it unless it is the master. A restarting pod rejoins as a replica instead of self-promoting.
• Self-labeling sidecar replaces label.sh. New redis-label-reconciler sidecar: each pod continuously labels its own pod's redis-role from the ground-truth redis-cli role. Because a pod only ever writes its own label from its own actual role, the Service endpoint always converges on the true master within a few seconds and can never be stranded with zero endpoints, regardless of Sentinel state. label.sh and the client-reconfig-script are removed.

The redis-master/redis-slave Services and their redis-role selectors are unchanged — only the mechanism that maintains the label is replaced.

⚠️ Blast radius & splitting

This lib is vendored by multiple teams (within Grafana: machine-learning, fleet-management, agent-management, grafana_com, play-grafana, …). This is a behavioural change to redis HA — please don't merge as-is. Suggested split into reviewable, independently-shippable pieces:

1. Redis ≥ 6.2 prerequisite — confirm/raise each consumer's pinned image (e.g. ML pins 6.0). The FQDN bits hard-require 6.2.
2. label.sh hardening or removal — strictly safer on its own; could ship first.
3. Self-labeling sidecar (F2) — the structural routing fix; adds a container + uses the existing pods: get/list/patch RBAC.
4. FQDN Sentinel/replication (F3) — the identity change; biggest behavioural delta.
5. No-self-promotion init logic (F4).

Other open questions for reviewers:

• The init container still apt installs kubectl into a shared volume purely for the sidecar — worth replacing with a slim image that ships kubectl.
• Migration path: deploying this triggers a roll, which is the exact operation that surfaced the bug. Needs a careful per-cluster rollout (the new pods must come up cleanly alongside old-version pods mid-roll).
• OnDelete (the default here) vs RollingUpdate (some consumers override) changes how this rolls out.

Validation done

• jsonnet evaluates the full lib with a minimal config.
• Rendered init.sh / label-reconciler.sh pass shellcheck clean; %%→% escaping verified in the rendered output.
• Rendered StatefulSet has the 5th container wired correctly (mounts, downward-API POD_NAME), and the dead label.sh/kubectl mounts are removed from the sentinel container.

Not yet done: deployed/tested on a live cluster. That's the key gap before this is more than a draft.

---

🤖 Generated with Claude Code

[cla-assistant]

All committers have signed the CLA.