feat: tenant-budget Grid-side plumbing for MVP 1b (app/tenant-based routing)

[jordigilh]

Context

Part of closing the "app/tenant-based routing" third of MVP 1 (multi-cluster routing: model-based, app-based, standard). Model-based and standard routing are covered today (grid-glb-demo, grid-workload-inference, grid-combined-site). Tenant/app-based routing has zero demoable evidence at the Grid (multi-cluster) layer today.

The tenant-identification and cost-metering work is tracked on the ai side:

• ai#130 — tenant-ID extraction (open, unassigned)
• ai#104 — per-tenant routing wiring at the single-gateway layer (open, unassigned)
• ai#191 — budget-enforcement engine, cascading org→team→user→agent limits (open, unassigned)
• ai#577 / ai-pr-581 — per-request cost-metering / balance-check filter (open PR, assigned to @noyitz)
• ai#301 — parent epic, "Multi-Tenancy Support"

None of that work extends the budget/tenant concept across cluster boundaries. This issue tracks the Grid-side plumbing needed to make a per-tenant budget cap demoable at the multi-cluster layer.

Scope

• CRD field for a per-tenant budget on the routing overlay / policy resource
• Per-tenant-keyed GCounter (today's grid GCounter CRDT is a single global counter — needs to become tenant-keyed)
• SWIM broadcast wiring so tenant spend state propagates across sites
• A budget-aware scoring gate that degrades routing (e.g. cheaper backend) as spend approaches the cap, and hard-stops at 100%

Open design question

GCounter is monotonic by construction — it can't natively express a rate-based budget ("$X per hour") without a reset/windowing design. This needs a design call before implementation, not after.

Estimate

Grid-side plumbing alone is well-precedented (~7-8 days) once the design call above is made; the full MVP 1b close (including the ai-side work above) is a 2-3 week cross-repo effort.

[jordigilh]

Design options for the GCounter rate-window question

Spiked this rather than leaving it as an open question. Read crdt/src/gcounter.rs, crdt/src/grid_state.rs, swim/src/state_broadcast.rs, scoring/src/scoring.rs, and re-read ai#191/ai#577 closely. One fact changes the shape of this decision: ai#191's own issue body already says the quiet part out loud —

Budget state integrates with the orchestration layer's G-Counter CRDTs for cross-grid visibility. Gateway-local tracking for single-site deployments.

That is: the architecture already assumes enforcement/windowing happens gateway-local (single-writer, so it can use anything — an in-memory sliding window, Redis TTL, whatever ai#577's metering filter ends up using). Grid's G-Counter was never meant to be the rate limiter. That reframes "GCounter can't do rate limits" from a blocker into a scoping question: what does Grid's counter need to represent?

Option A (recommended) — Grid tracks cumulative spend; windowing stays where it already lives

Grid's cross-site G-Counter continues to represent cumulative spend (lifetime, or "for the life of this budget period" if the CR ever gets a reset field later) — the one thing a G-Counter is naturally, provably good at (this repo already has 8 passing CRDT-law tests for exactly that: commutative/associative/idempotent merge). The scoring gate degrades routing (cheaper backend, then hard-stop) as cumulative spend approaches a cap defined on the CRD. Any genuine "$X per hour" enforcement decision happens at the ai-gateway layer per ai#191's own design — which is single-writer-per-request-path there, so it's free to do real windowing without touching Grid's CRDT at all.

• Cost: zero new engineering on the CRDT primitive. The existing ~7-8 day Grid-side estimate (CRD field, SWIM broadcast wiring, scoring gate) holds unchanged.
• Trade-off: if the customer's literal ask is "$X/hour, enforced grid-wide," this doesn't deliver that — it delivers "$X total, visible grid-wide" instead. Worth confirming which one is actually being asked for before committing to Option B's extra cost.

Option B — Windowed G-Counter via epoch-keyed buckets

If a real cross-site rate window is a hard requirement: key the counter map by a coarse epoch id (e.g. floor(unix_hour)) instead of a single running total — BTreeMap<epoch_id, GCounter>. Each site increments only the current epoch's counter; merge only combines matching epoch keys (same max-per-slot rule, unchanged); epochs older than 1-2 windows get pruned on the next reconcile tick (no decrement operation ever needed — a new epoch just starts at zero). Budget check reads only the current epoch's total.

• Cost: small, bounded addition on top of Option A's plumbing — maybe +1-2 days for the epoch-keying, merge-scoping, and prune-sweep logic. Reuses the existing, already-tested per-slot max-merge rule unchanged; this is a generalization of a pattern already proven in this repo, not new CRDT theory.
• Trade-off: standard fixed-window boundary effect (a burst straddling the epoch boundary can momentarily allow ~2x the nominal cap) — the same trade-off AWS API Gateway/Stripe-style fixed-window limiters accept every day. Needs picking a window granularity (hour vs. day) as a config knob, and rough wall-clock agreement across sites (already implicitly required for TLS cert validation, so not a new constraint).

Option C — PN-Counter with explicit decay (not recommended now)

Extend to a PN-Counter (two G-Counters: increments + decrements) so sites can broadcast a real decay operation instead of discrete epoch buckets. More general (smooth decay curves, not just windows) but meaningfully heavier: needs cross-site agreement on a decay rate, and is harder to reason about under partition (a site rejoining after being offline for hours could apply a large stale decrement). No precedent for this anywhere in the current CRDT crate. Worth keeping in the back pocket if Option B's boundary effect ever becomes an actual operational complaint, not something to build speculatively now.

Recommendation

Start with Option A. It costs nothing extra, is consistent with ai#191's own stated design, and turns Demo 6 back into "well-precedented, ~7-8 days" instead of "blocked on an open design question." Only reach for Option B if there's an explicit customer requirement for grid-wide rate enforcement (not just visibility) that Option A's "cumulative + local windowing" framing doesn't satisfy — and even then, it's a bounded, well-scoped add-on, not a rework.