mesh-granularity=location: silently degraded leader blackholes relayed traffic with no failover

[kvaps]

Summary

With --mesh-granularity=location, all cross-location traffic to non-leader
nodes is relayed through the location's WireGuard leader. Kilo only re-elects
a leader when the previous one is fully removed from the cluster
(NotReady/deleted). It has no mechanism to detect a leader whose Kubernetes
state is Ready but whose underlying packet forwarding is broken. When this
happens, every non-leader node in the location becomes unreachable from other
locations until the failing node is fully removed — typically several minutes.

Reproducer / observed scenario

Hybrid cluster: on-prem location (3 nodes) + cloud-burst location (Azure VMSS,
2-3 nodes). Cilium with VXLAN overlay as CNI, Kilo for the inter-location
WireGuard mesh, --mesh-granularity=location, --compatibility=cilium.

The cloud leader experienced a kernel/hypervisor-level VXLAN forwarding fault:

• WireGuard tunnel up, handshakes OK
• kubelet/API server responsive (Ready)
• ICMP ping to the leader and through it: OK
• Cilium control-plane health: OK (existing flows)
• New VXLAN-encapsulated flows: enter cilium_vxlan on the leader but never
  appear on the underlying NIC

Effect: every non-leader pod in that location was unreachable from the other
location for the full window (~5–15 min) until the autoscaler removed the node
because it eventually went NotReady. From Kilo's point of view the leader
was healthy throughout, so no re-election happened.

Why the current model doesn't catch it

Leader election is driven by Kubernetes node readiness, not by the actual
forwarding capability of the leader. There's no equivalent of a periodic
forwarding probe between leaders/peers, so any fault below the kubelet level
(NIC driver, hypervisor, kernel datapath) can blackhole relayed traffic
indefinitely while the node looks Ready.

Possible directions (open for discussion)

• Active forwarding probe between location leaders (and from leaders to
  non-leaders) — promote a non-leader if the current leader fails the probe,
  independent of node readiness.
• Allow per-location leader redundancy: e.g. multiple leaders per location
  with ECMP/anycast on the WG endpoint, so a degraded one is bypassed by
  routing rather than re-election.
• A hybrid granularity: leaders per location for non-leader pods that don't
  have a public IP, plus direct tunnels between any nodes that do have one.

Environment

• Kilo 0.7.0 (upstream)
• Cilium with VXLAN overlay, kubeProxyReplacement=true
• Talos Linux on cloud-burst VMSS, on-prem control planes
• --mesh-granularity=location, --compatibility=cilium,
  --encapsulate=crosssubnet, --mtu=auto, --internal-cidr=$(NODE_IP)/32

[kvaps]

Just realized #328 (open since 2022) introduces --mesh-granularity=cross,
which gives every pair of nodes across different locations a direct
WireGuard tunnel while keeping intra-location traffic on the CNI overlay.
That eliminates the leader-as-relay path entirely for cross-location
traffic — which is the SPOF this issue describes.

It doesn't address the more general problem of detecting silent forwarding
faults (the "possible directions" above), but for the cross-site relay
case it sidesteps the failure mode by removing the relay altogether.

If finalizing #328 is the preferred direction here, happy to help test
with --compatibility=cilium and chase down whatever's blocking merge.

[kvaps]

One concrete shape for the "active forwarding probe" direction, in case
it helps frame the discussion:

Today findLeader in pkg/mesh/topology.go is purely deterministic
(annotation + public IP). A health-aware variant could combine a
coordination.k8s.io/v1.Lease per location with an active forwarding
probe inside each kg pod:

• A kg pod will only acquire / renew the location lease while a
  background probe is succeeding (e.g. forwarding a synthetic packet
  to a known non-leader peer in its own location, or to a leader in
  another location over WG).
• On N consecutive probe failures the current holder voluntarily
  drops the lease.
• Peers select the lease holder as the location's WG endpoint instead
  of findLeader's annotation-only logic.

This addresses the silent-degradation case because the leader renounces
the lease itself when its datapath is broken, even while the node stays
Ready. A plain Lease without a probe wouldn't help here — in the
incident we observed, kubelet/apiserver/WG control plane were all
healthy, only the VXLAN forward path was failing, so Lease renewal
would have continued unaffected.

Probably out of scope for #328 (which addresses the topology side), but
worth tracking as a separate piece for the location/cross granularity
modes that still elect a relay.

[squat]

Yes exactly, #328 is the ideal direction I want to take to mitigate the SPOF architecture in Kilo. It should ideally become the default topology in a future breaking release. One of the rings m things to consider before deciding on a final approach is the proposal for a more generic subnet-controlled approach as described here: #328 (comment)

I appreciate the relative simplicity (both in code and in mental model) of #328 and world line to move forward with that. It doesn't necessarily eliminate the possibility of implementing the subnet-controlled topology on the future. The main blocked here is lack of tests. It needs a suite of e2e tests like those for full-mesh and local-mesh. Once that's in, I'd be happy to merge!

[kvaps]

Got it — taking e2e tests as the merge blocker for #328.

Plan, mirroring e2e/full-mesh.sh and e2e/location-mesh.sh:

• Add e2e/cross-mesh.sh with:
  • setup_suite: annotate kind nodes into two locations
    (e.g. control-plane + one worker as loc1, second worker as
    loc2) and patch the DaemonSet with
    --mesh-granularity=cross
  • test_cross_mesh_connectivity — pings + check_adjacent
    (same as full/location)
  • test_cross_mesh_peer — check_peer wg99 e2e 10.5.0.1/32 cross
  • test_mesh_granularity_auto_detect —
    _kgctl graph == _kgctl graph --mesh-granularity cross
  • test_cross_intra_location_no_wg — assert
    kgctl showconf node <loc1-node> lists peers only for
    nodes in other locations (sanity that cross differs from
    full in the right way)
• Append ./e2e/cross-mesh.sh to the e2e: target in Makefile

I'll send this as a follow-up commit on the #328 branch (or a
separate PR rebased on it if preferred), and rebase against current
main while at it. Anything else you'd want covered?

[squat]

Amazing 👍 I'd say create a new PR with #328 as the base so we can attribute the original author and merge with your patches, then we can close that PR referencing your changes