Design: explicit budget-epoch/window reset for tenant_spend, decoupled from SWIM membership eviction

[jordigilh]

Context

grid#47's review (#47 (comment)) caught a correctness bug: OriginStateHandle::remove_origin was wiping a departed origin's tenant_spend slot on ordinary SWIM membership eviction (a site marked Suspect/Dead past SUSPECT_MEMBER_TTL/dead_member_ttl — see operator::swim_runtime::prune_tracked_members), not just on permanent retirement. Since tenant_spend is a cumulative grow-only GCounter, this let a tenant's spendRatio drop — and an already-exhausted budget reopen — on a mere restart or transient partition.

The fix (landing in grid#47) stops pruning tenant_spend on membership eviction. That's the correct behavior for the common case, but it trades away the growth bound the original (buggy) code accidentally provided: today, nothing ever removes a site's slot from a tenant's GCounter once recorded, whether that site is a legitimate one that will never rejoin (permanently decommissioned) or, in a compromised-key threat model, a malicious/churning origin repeatedly claiming new site identities.

Note this is a different axis from MAX_TRACKED_TENANTS (crdt/src/grid_state.rs), which already bounds the number of distinct tenant_id keys. There is currently no bound on the number of distinct site-slots accumulated within a single tenant's GCounter over the cluster's lifetime.

Proposal

Design an explicit, opt-in budget-epoch / spend-window reset mechanism, decoupled from SWIM membership events entirely — e.g.:

1. A periodic or admin-triggered "epoch roll" that resets tenant_spend (or archives it and starts a fresh counter) on a schedule independent of any single site's membership status, so a legitimate long-term-departed site's slot can eventually be reclaimed without being tied to a churn/restart signal.
2. A separate, explicit hard cap on distinct site-slots per tenant (mirroring MAX_TRACKED_TENANTS's bound on tenant count) as defense-in-depth against a compromised-key/malicious-origin threat, independent of (1).

Non-goals

• Not proposing to resurrect the membership-eviction-triggered clearing this issue is a reaction to — that was the bug.
• Not blocking grid#47 — that PR's fix is correct and self-contained; this is deliberately deferred follow-up work, called out explicitly in remove_origin's doc comment.

References

• grid#47: feat(operator,crdt,swim): tenant-budget plumbing via SWIM-gossiped GCounter (grid#40) #47
• grid#40 (parent tenant-budget plumbing issue): feat: tenant-budget Grid-side plumbing for MVP 1b (app/tenant-based routing) #40

[jordigilh]

Split this into its two independent proposal items so each can move at its own pace:

• Item 2 (site-slot cap): fix(crdt): cap distinct origin site-slots per tenant_spend counter #63 — grounded in code precedent, implemented, TDD'd (unit + integration tiers; E2E deliberately skipped with rationale — mirrors MAX_TRACKED_TENANTS's own coverage shape), all lint clean. Ready for review.
• Item 1 (epoch/window reset): Design: budget-epoch/window reset for tenant_spend (grid#52 item 1, split from site-slot cap) #64 — genuinely harder (monotonic GCounter re-merge means a naive reset gets silently revived by a lagging peer; needs CRD schema changes since BudgetPolicyConfig denies unknown fields). Sketched a merge-time epoch-fencing approach there as a starting point, not a final design.

Leaving this issue open until #64 lands too.