Spike: request-time selection across equivalent Grid providers

[nerdalert]

Problem

Grid currently ranks eligible providers, and Praxis selects the first viable candidate. When several providers are equally suitable, a deterministic identity tie-break keeps the same provider first. The others remain available for failover but do not actively share new traffic.

Grid ranks A, B, C
  -> Praxis always selects A
  -> B and C remain idle until A is unavailable

Ranking answers which providers are preferred. It does not by itself provide load balancing among equally preferred providers.

This spike investigates a clean division of responsibility:

• Grid discovers providers, evaluates health and admission, applies routing and scoring policy, and publishes groups of equally eligible candidates.
• Praxis reads the accepted overlay from memory and locally selects a provider within the best viable group.

Grid, Kubernetes, the operator, ConfigMaps, overlay-sync, metrics endpoints, and remote policy services must remain outside the request hot path.

Focused demo narrative

The spike is exercised through a deliberately small provider-traffic demo:

Client
  |
  v
One consumer gateway
  |
  v
Praxis intelligent_route
  |
  +--> Provider Gateway A --> backend A
  +--> Provider Gateway B --> backend B
  `--> Provider Gateway C --> backend C

All three provider gateways advertise the same capability and are equally eligible. Grid discovers them across the three sites and publishes them in the same active selection group. The consumer gateway then applies a local picker to new, unbound requests.

The focused proof should demonstrate:

1. one consumer gateway receives all measured client traffic;
2. three provider gateways are discovered and accepted in one active group;
3. roundRobin produces a repeating A -> B -> C sequence;
4. 60 measured requests produce an exact 20/20/20 split;
5. every request succeeds and reports the selected provider;
6. the serving overlay remains unchanged during the measured window;
7. traces show the consumer-to-selected-provider path;
8. request processing causes no control-plane call or overlay write.

This isolates provider selection. It intentionally excludes DNS/GTM behavior, multiple active consumer gateways, metric-derived weights, and fallback-group demonstration.

Proposed routing contract

Keep the following concepts separate:

admission
  -> may this provider receive new or existing sessions?

routing policy
  -> which eligibility boundary is preferred?

scoring policy
  -> which asynchronous signal affects provider preference?

selection group
  -> which providers may actively compete with one another?

selection policy
  -> how does Praxis choose within the active group?

An initial overlay extension can use additive fields:

selection_policy: round_robin
candidates:
  - cluster: provider-a
    selection_group: 0
  - cluster: provider-b
    selection_group: 0
  - cluster: provider-c
    selection_group: 0

Group 0 is the best active group. Lower groups are fallback candidates and must not receive new traffic while a better group remains viable.

Implementation boundary

The Praxis-side changes for this spike should remain constrained to the praxis-proxy/ai routing filters, primarily intelligent_route and its AI-owned overlay, grouping, and picker modules.

Praxis core's generic HTTP/TCP load balancers already select endpoints within a configured cluster. They should not gain Grid-specific concepts such as provider admission, selection groups, overlay revisions, or provider affinity.

praxis-proxy/grid
  -> computes provider eligibility and selection groups
  -> publishes the overlay contract

praxis-proxy/ai intelligent_route
  -> consumes the accepted overlay snapshot
  -> applies affinity and the configured group picker
  -> selects the provider cluster

Praxis core load_balancer
  -> selects an endpoint within that already-selected cluster

This keeps Grid semantics in the AI extension, preserves the reusable Praxis core boundary, and limits request-path changes to the filter that already owns model/provider routing.

Picker policies

The first increment should evaluate:

• deterministic: preserve the historical first-viable behavior;
• roundRobin: take turns across viable candidates in the active group;
• random: choose uniformly within the active group.

Round-robin state is process-local. The goal is balanced behavior at each Praxis gateway, not a globally synchronized counter across a fleet.

The request sequence is:

valid existing session binding?
  |
  +-- yes -> reuse the bound provider if admission still permits it
  |
  `-- no  -> identify the best viable group
            -> run its configured picker
            -> create an affinity binding when affinity is enabled

Compatibility requirements

• Existing overlays without group or picker metadata retain deterministic, ordered-first behavior.
• Omitted policy must not silently change an upgraded deployment's behavior.
• New-install configuration may explicitly choose roundRobin.
• Unknown picker values fail validation rather than silently changing routing.
• Additive overlay fields must survive operator, overlay-sync, and Praxis digest validation.
• Invalid reloads retain the last-known-good serving snapshot.
• In-flight requests continue using the immutable snapshot they started with.
• Picker state must not reset merely because unchanged overlay content is republished.

Candidate grouping questions

The spike must define a deterministic and transitive grouping rule for each routing policy.

Candidate identity must not split otherwise equivalent providers. Admission, freshness, and required policy boundaries must split groups. Metric score differences should not automatically prevent future weighted distribution inside a group.

Do not use pairwise epsilon comparisons in sorting. They can be non-transitive and make output depend on input order.

The spike should verify behavior for:

• input permutations;
• multiple capabilities;
• local and remote candidates;
• mixed admission states;
• stale candidates;
• grouped and legacy ungrouped overlays;
• duplicate or conflicting group metadata;
• overlay reloads with the same and different semantic revisions.

Admission, affinity, retry, and fallback

The design must preserve:

• new_and_existing: eligible for new and already-bound sessions;
• existing_only: excluded from new selection but valid for an existing binding;
• excluded: never selected;
• valid affinity before picker execution;
• retry within the selected group before falling through to a lower group;
• bounded behavior when group membership changes during reload.

The focused three-provider demo proves equal active-group distribution. A separate scenario is still needed to prove draining, retry order, lower-group fallback, and recovery without weakening the focused narrative.

Observability

Each route decision should expose bounded, privacy-safe evidence:

• picker policy;
• selection group;
• selected provider, site, cluster, and stable ID;
• overlay accepted and serving revision;
• admission state;
• whether affinity was created, reused, or failed over;
• whether retry stayed within the group or fell through.

Provider and session identifiers must not become unbounded metric labels. Tracing may carry bounded decision attributes, while aggregate metrics should use controlled dimensions.

How this leads to weighting

Equal round-robin selection closes the immediate winner-take-all gap for equivalent providers. It also creates the correct boundary for a later weighted policy:

Grid control plane
  -> filters ineligible providers
  -> forms the active selection group
  -> normalizes asynchronous capacity signals
  -> publishes bounded traffic weights

Praxis data plane
  -> reads the immutable local snapshot
  -> preserves valid affinity
  -> performs local weighted selection within the active group

Weighting must be a separate, explicit selection policy. Scores, ranks, queue depth, or KV-cache pressure must not silently become traffic percentages.

A later weighted increment must define:

• traffic_weight schema and valid range;
• missing, zero, invalid, and stale weight behavior;
• normalization across providers with different capacities;
• whether weights distribute requests, new sessions, or both;
• deterministic tests using an injected random source;
• statistical runtime tolerances over sufficiently large samples;
• affinity behavior as weights change;
• removal and recovery without resetting unrelated picker state;
• protection against noisy metrics and excessive overlay churn.

Deliverables

• Grid/Praxis responsibility boundary.
• Additive overlay schema and compatibility behavior.
• Deterministic grouping rules for supported routing policies.
• Picker ownership and configuration shape.
• Immutable snapshot and picker-state lifecycle.
• Admission, affinity, retry, fallback, and recovery semantics.
• Bounded telemetry contract.
• Focused one-consumer/three-provider demo and evidence format.
• Phased implementation plan from equal selection to optional weighting.
• Risks and unresolved questions.

Out of scope for this spike

• Fleet-global round-robin coordination.
• Per-request calls to Grid or Kubernetes.
• Treating scores as implicit weights.
• Production metric-derived weighting.
• DNS, Anycast, or global entry-point balancing.
• Tenant, billing, quota, or cost-policy implementation.
• Replacing existing Grid scoring strategies.

Exit criteria

The spike is ready to conclude when it provides a compatibility-preserving design and focused runtime evidence that one Praxis consumer can distribute new traffic across three equally eligible provider gateways using only its local in-memory snapshot.

The spike must also leave a concrete, separately scoped path to weighted selection without claiming that equal round-robin distribution is weighted or metric-aware routing.

[jordigilh]

Spiked the Grid-side half of this against the current codebase (not implementing — just grounding the open design questions in what already exists).

Grid already has most of the needed building blocks

RoutingCandidate (operator/src/resources/routing_overlay.rs) already carries rank, score, score_breakdown, and admission_state per candidate. render_routing_overlay's final sort (ScoreFirst policy) is:

admission_sort_key → fresh (desc) → score (desc, exact float) → locality_sort_key → site → name → cluster

The last three (site/name/cluster) are pure determinism tiebreakers — they carry no routing-relevant meaning, they just exist so the sort is total and stable. GeographyFirst follows the same shape with locality promoted ahead of score.

This means adding selection_group is genuinely additive, not new infrastructure: the "candidate grouping" question in this issue already has a clean, transitive answer sitting in the existing sort key.

Answering the "deterministic and transitive grouping rule" question

Define the group boundary as: every sort key up to (but excluding) the pure tiebreakers. Concretely, for ScoreFirst: group by exact equality of (admission_sort_key, fresh, score, locality_sort_key); bump the group counter each time that tuple changes walking the already-sorted list. For GeographyFirst, same idea with the key order that policy already sorts by.

Why this satisfies the constraints this issue lists:

• Transitive, no pairwise epsilon — it's equality on already-totally-ordered discrete/exact fields, computed once per adjacent pair in a single pass, not a pairwise all-to-all comparison.
• Mixed admission states split groups — different admission_sort_key never merges.
• Stale candidates split groups — fresh is already in the tuple.
• Score differences don't split groups going forward — this rule only groups on exact score equality today (deterministic-first-viable-safe), but the tuple is exactly where a later quantized/banded score would slot in without touching anything else.
• Grouped and legacy ungrouped overlays — selection_group would be #[serde(skip_serializing_if = "Option::is_none")] like every other enrichment field on RoutingCandidate (admission_state, score, rank all follow this pattern already), so existing overlays/consumers see no schema change until a picker policy is configured.

Effort estimate

• Grid side (this repo): small. One new Option<u32> field on RoutingCandidate, one pass over the already-sorted candidates slice (right where rank is assigned today, routing_overlay.rs:~1198-1204) to derive group boundaries from the existing key tuple. No new data sources, no new CRD fields needed for the grouping computation itself.
• Praxis side (praxis-proxy/ai, intelligent_route): this is where the real complexity lives — picker policy (deterministic/roundRobin/random), per-process picker state lifecycle across overlay reloads, session-affinity interaction, retry-within-group-before-fallthrough, and the observability contract this issue specifies. Consistent with this issue's own "Implementation boundary" section, that's out of this repo's scope.

Net: this issue isn't blocked on Grid needing new capability — the grouping computation is a small, well-defined addition to an existing pipeline. The bulk of the design/implementation effort is on the ai side. Happy to pick up the Grid-side field + tests if/when the ai-side picker design is ready to consume it.

[nerdalert]

Grid AI Provider Load-Balancing Demo

Spike Demo Video => https://github.com/user-attachments/assets/e580a9c6-c1e9-446f-bab9-e4c7c31c

  Client
    |
    v
  One consumer gateway
    |
    v
  intelligent_route
    |
    +--> Provider Gateway A --> VCR backend A
    +--> Provider Gateway B --> VCR backend B
    `--> Provider Gateway C --> VCR backend C

The demo uses:

  routingPolicy: scoreFirst

  scoringPolicy:
    strategy: noMetrics

  selectionPolicy:
    picker: roundRobin

Grid discovers the three provider gateways, validates their availability, places them in the active selection group, and publishes the routing overlay.

The AI gateway then performs request-time round-robin selection inside that group. It uses the already-loaded in-memory routing snapshot, so the request hot path does not call Kubernetes, Grid, EPP, Prometheus, or a scoring service.

For 60 new sessions, the validated result is:

  Provider A: 20
  Provider B: 20
  Provider C: 20

The selected provider sequence repeats in order. Existing session affinity is checked before round-robin selection, so a session remains pinned to its original provider.

Tracing

Each request is sent through the single consumer gateway and produces correlated OpenTelemetry evidence:

  Client
    |
    v
  consumer-gateway
    |
    v
  routing.select
    |
    v
  provider-gateway
    |
    v
  VCR backend

The tracing UI shows:

• The selected provider.
• The observed consumer-to-provider path.
• Provider attribution counts.
• Request-by-request results.
• Jaeger trace links.
• The routing selection group and picker policy.
• Five-span traces across the consumer and provider gateways.

This spike (demo above) adds provider-level round-robin balancing through the AI gateway’s intelligent_route filter, filling a gap in the existing Grid demo coverage:

• Provider traffic (https://github.com/praxis-proxy/demos/tree/main/demos/grid-provider-traffic): Equal round-robin distribution across provider gateways without inference metrics.
• llm-d pool metrics (https://github.com/praxis-proxy/demos/tree/main/demos/grid-llmd-pool-metrics): EPP metric-driven scoring, ranking, pressure-based failover, and recovery.
• GLB (https://github.com/praxis-proxy/demos/tree/main/demos/grid-glb-demo): Multi-hop global routing through GTM, edge gateways, and provider gateways.

Together, these demos cover local provider balancing, metric-aware routing, and global multi-hop traffic flow.