feat(routing): publish provider selection modes

[nerdalert]

Summary

This PR adds an explicit provider-selection contract to Grid. It accompanies the research spike in grid#31 (#31) to gather feedback and converge on an approach to provider-level load distribution.

Its scope is selecting among provider gateways, endpoint selection within a provider remains the responsibility of that provider’s serving stack.

Grid determines which provider gateways are eligible, organizes equivalent providers into selection groups, and publishes the selected distribution mode in the routing overlay. Praxis consumes that immutable overlay and performs request-time selection locally.

  Client
    |
    v
  Consumer Gateway
    |
    v
  intelligent_route
    |
    +--> Provider Gateway A
    +--> Provider Gateway B
    `--> Provider Gateway C

This operates at the provider-gateway layer. Endpoint selection within a provider, including llm-d EPP backend selection, remains the responsibility of that provider's serving stack.

Architecture

Provider routing is separated into four stages:

  Health, authorization, admission, and freshness
                          |
                          v
                  Routing policy
                          |
                          v
                   Scoring policy
                          |
                          v
                 Selection groups
                          |
                          v
                  Selection mode

Routing Policy

geographyFirst groups providers by admission, locality, and freshness. Praxis can balance across providers in the closest viable locality tier while keeping remote tiers available as fallback.

scoreFirst groups providers by admission and freshness. Eligible providers from multiple sites can therefore participate in the same active group.

Scoring Policy

The scoring strategies remain independent of request distribution:

• noMetrics: No inference metrics are required; routing continues to use health, admission, freshness, locality, and authorization.
• queueDepth: Grid retrieves queue-depth metrics from the provider’s llm-d EPP metrics endpoint and uses them to rank provider gateways by queue pressure.
• kvCachePressure: Grid retrieves KV-cache utilization metrics from the provider’s llm-d EPP metrics endpoint and uses them to rank provider gateways by available cache capacity.

A score difference does not become a traffic ratio and does not split otherwise equivalent providers into different selection groups.

Selection Policy

spec:
routingPolicy: scoreFirst
scoringPolicy:
strategy: noMetrics
selectionPolicy:
mode: roundRobin

deterministic selects the first viable provider in the active group. This is useful when a deployment requires a strict preferred provider.

roundRobin rotates equally among viable providers in the active group. It provides active/active provider distribution without requiring EPP, Prometheus, or inference-specific metrics.

Selection policy is explicit. Grid does not infer it from scores, provider count, routing policy, or the availability of metrics.

Responsibilities

Grid remains the asynchronous control plane:

• observes provider health, admission, freshness, and configured metrics;
• applies routing and scoring policies;
• creates deterministic selection groups;
• publishes immutable routing overlays;
• removes or restricts unavailable providers through admission state.

Praxis remains the request-time data plane:

• loads and validates an overlay snapshot;
• evaluates session affinity before new-provider selection;
• selects locally within the first viable group;
• preserves existing bindings when admission permits existing traffic;
• falls through to the next viable group when the active group has no eligible provider.

The request path does not call Grid, Kubernetes, EPP, Prometheus, ConfigMaps, or another remote control-plane service.

Multiple Consumer Gateways

The contract supports multiple consumer gateways. Each consumer receives the same Grid-produced provider ordering and group metadata, then performs selection from its local in-memory snapshot.

Round-robin counters are local to each gateway process. A single gateway produces a predictable local sequence, but multiple replicas do not share a global counter. Aggregate traffic should converge toward an even distribution over a sufficient request volume, but strict global request-by-request ordering is not guaranteed.

Session affinity can also make the observed distribution intentionally uneven because an existing session remains bound to its selected provider.

Compatibility

selectionPolicy is optional.

When it is omitted, Grid omits selection_policy from the generated overlay. Consumers that do not receive an explicit mode retain deterministic selection
behavior.

New Helm installations can explicitly select roundRobin, while existing resources with an omitted policy are not silently changed.

The field is additive across the Grid, overlay-sync, and Praxis contracts.

Overlay Contract

Grid publishes:

  {
    "selection_policy": {
      "mode": "roundRobin"
    },
    "candidates": [
      {
        "cluster": "provider-a",
        "selection_group": 0
      },
      {
        "cluster": "provider-b",
        "selection_group": 0
      },
      {
        "cluster": "provider-c",
        "selection_group": 0
      }
    ]
  }

Selection groups are:

• deterministic under candidate input permutations;
• scoped independently by capability;
• contiguous and ordered by preference;
• separated by admission and routing-policy boundaries;
• unaffected by candidate identity alone;
• unaffected by metric score differences within an otherwise equivalent group.

The selection policy and group metadata are included in semantic overlay validation and digest calculation.

Components

• Grid feature branch (https://github.com/nerdalert/grid/tree/feat/provider-selection-groups)

• Provider selection architecture
  (https://github.com/nerdalert/grid/blob/feat/provider-selection-groups/docs/architecture/provider-selection-and-load-balancing.md)

• Companion Praxis AI selection branch (https://github.com/nerdalert/ai/tree/feat/provider-load-balancing)

• Grid provider-selection research spike and demo video (Spike: request-time selection across equivalent Grid providers #31)

• Public Grid demos (https://github.com/praxis-proxy/demos)

• Praxis tracing visualization (https://github.com/nerdalert/praxis-tracing)

The companion AI change implements the in-memory group index and request-time selection used by intelligent_route. The Grid and AI changes preserve a clean
control-plane/data-plane boundary.

OpenTelemetry instrumentation and the tracing UI are separate components. They improve visualization and validation but are not required for provider selection to operate.

Scope

This PR includes:

• optional selectionPolicy.mode;
• deterministic and roundRobin modes;
• deterministic Grid-side selection groups;
• overlay contract and digest updates;
• overlay-sync validation;
• CRD, Helm, sample, and architecture documentation updates;
• provider-traffic demo orchestration.

This PR does not include:

• weighted request selection;
• metric-derived traffic weights;
• synchronized counters across gateway replicas;
• request-time metric queries;
• backend endpoint selection inside a provider;
• OpenTelemetry instrumentation.

Weighted selection remains a separate extension. It can build on the same active-group boundary without changing the distinction between scoring and request distribution.

Validation

Static validation passed:

• make lint
• make test
• Rust documentation with warnings denied
• focused operator tests
• focused overlay-sync tests
• focused xtask tests
• image-override tests
• CRD parity checks
• Helm rendering and schema rejection checks
• git diff --check

The focused cold Kind proof validated:

• three provider clusters;
• one consumer gateway;
• three provider gateways discovered through SWIM;
• scoreFirst + noMetrics + roundRobin;
• 60 of 60 successful requests;
• exact 20/20/20 provider distribution;
• a repeating three-provider sequence;
• stable overlay and serving revisions during traffic;
• successful teardown.

The demo narrative and recording are available in grid#31 (#31).

The focused provider-traffic demonstration uses one consumer gateway and three provider gateways:

  Client
    |
    v
  Consumer gateway
    |
    +--> Provider gateway A --> VCR backend A
    +--> Provider gateway B --> VCR backend B
    `--> Provider gateway C --> VCR backend C

Run it with the published images (a demo PR is not opened yet for reference but the validation images are published):

  - ghcr.io/nerdalert/praxis-ai:grid-provider-selection-otel-20260814-536534aba5c1
  - ghcr.io/nerdalert/grid-operator:provider-selection-20260814-d965540dad8d
  - ghcr.io/nerdalert/grid-overlay-sync:provider-selection-20260814-d965540dad8d
  - ghcr.io/nerdalert/praxis-tracing:provider-selection-20260814-fbadb3e2007a

  source demos/grid-provider-traffic/configs/images.env
  export GRID_XTASK_VCR_IMAGE=ghcr.io/neuralmagic/vllm-vcr:vllm0.23
  export GRID_XTASK_IMAGE_PULL_POLICY=IfNotPresent
  export GRID_REPO=/path/to/grid

  ./demos/grid-provider-traffic/run.sh --quick --teardown

The expected proof is 60 successful requests with exact 20/20/20 provider attribution and a repeating three-provider sequence. The demo uses noMetrics and roundRobin; it does not claim weighted routing or metric-derived traffic ratios.

Operational Considerations

Provider membership and eligibility are eventually consistent because Grid updates overlays asynchronously.

A gateway begins using new membership only after overlay-sync validates the overlay and Praxis accepts the corresponding serving revision. Selection state is scoped to the loaded snapshot and may restart when routing semantics genuinely change.

The operator avoids rewriting semantically unchanged overlay ConfigMaps. This prevents metadata-only reconciliation from resetting request-time selection state while retaining repair behavior for malformed, incomplete, or corrupted payloads.

Breaking Changes

No breaking change is intended.

Deployments that omit selectionPolicy retain deterministic behavior. Round-robin distribution is enabled only when explicitly configured.

Related Components

This change is part of a cross-repository spike. The following feature branches are not yet merged:

• Grid provider-selection branch (https://github.com/nerdalert/grid/tree/feat/provider-selection-groups)
  Defines selection policy, constructs provider groups, publishes the overlay contract, and supplies the demo orchestration.

• Praxis AI provider-selection branch (https://github.com/nerdalert/ai/tree/feat/provider-load-balancing)
  Consumes the overlay, builds the in-memory group index, and performs request-time selection in intelligent_route.

The following resources provide supporting context and validation:

• Grid issue Spike: request-time selection across equivalent Grid providers #31 (Spike: request-time selection across equivalent Grid providers #31)
  Tracks the research spike and contains the focused demo narrative and video.

• Public Grid demos (https://github.com/praxis-proxy/demos)
  Contains the reusable demo framework. The provider-traffic additions associated with this spike may not yet be available on its default branch.

• Praxis tracing visualization (https://github.com/nerdalert/praxis-tracing) Provides optional visualization and trace inspection. It is not required for provider selection.

The Grid and Praxis AI changes must be evaluated together for the complete feature. This Grid PR defines and publishes the contract; the companion AI branch implements its request-time consumer. OpenTelemetry and the tracing UI remain optional supporting components.

[jordigilh]

Comprehensive GA-readiness pass on the routing/grouping contract, test coverage, and Helm templating. Nothing here blocks the design; flagging a few things worth closing before this leaves draft.

[jordigilh]

This reorders GeographyFirst candidate ranking (freshness now outranks score) for every existing GridNetwork, not just ones opting into selectionPolicy. The reordering looks necessary — assign_selection_groups needs freshness-sorted input to produce contiguous groups — but since the default selection mode is deterministic (picks candidates[0]), this can change which provider receives new traffic on upgrade. The PR's "Breaking Changes" section says "No breaking change is intended" — could we make this ranking-order change explicit there, even though the mode itself is unaffected?

[jordigilh]

This comment still describes the pre-PR ordering ("locality sorts above freshness" / local ranks first). The assertion below it (line 2564) was correctly updated to the new expectation ("freshness precedes score", provider-api ranks first) but this comment wasn't. Suggest updating it to match the assertion message.

[jordigilh]

charts/grid-site/tests/gridnetwork_test.yaml already has a helm-unittest suite for this template but doesn't cover the new selectionPolicy default logic (and isn't touched by this PR at all). At least the "no existing CR + no configured value → defaults to roundRobin" branch is testable today since lookup returns empty under helm-unittest — worth adding a case for it. The "preserve existing CR's policy" branch genuinely needs a live cluster; fine to leave that one uncovered here as long as it's called out.

[jordigilh]

docs/conventions.md asks for benchmarks on "significant changes," or an explanation in the PR description of why that's impractical. This is an O(n) pass in the async control-plane render path so risk looks low — a one-line note in the PR description addressing this would close the loop per that convention explicitly.