Skip to content

fix(xtask): set GridNetwork gatewayRefs[].localSiteName in E2E fixtures - #61

Merged
nerdalert merged 5 commits into
praxis-proxy:mainfrom
jordigilh:fix/grid-60-scoring-order-fixture-localsitename
Aug 16, 2026
Merged

fix(xtask): set GridNetwork gatewayRefs[].localSiteName in E2E fixtures#61
nerdalert merged 5 commits into
praxis-proxy:mainfrom
jordigilh:fix/grid-60-scoring-order-fixture-localsitename

Conversation

@jordigilh

Copy link
Copy Markdown
Contributor

Summary

xtask env verify-operator-reconcile failed deterministically with:

error: scoring order check failed: op-e2e-api-fallback (pos 0) must appear after site-a (pos 1); expected local > api_provider

Root-caused live on a fresh local kind cluster (see comment thread on #60 for the full investigation). Fixes #60.

Root cause

network_fixture_json (the shared GridNetwork E2E fixture builder) never set gatewayRefs[].localSiteName. render_routing_overlay's local_site therefore fell back to the network name (op-e2e-net), which matches no candidate's site field. Every candidate tied at LocalityTier::Unknown, so GeographyFirst ordering (the production default) fell through past locality to score.

Under the noMetrics scoring strategy (#38, now the default), both the local and API-provider candidates correctly score identically — "both providers must have equal dynamic scores under noMetrics" is asserted intentionally in routing_overlay.rs's own unit tests. With score tied too, the sort fell to the final deterministic tiebreak: (site, name, cluster) ascending — and "op-e2e-api-fallback" < "site-a" alphabetically.

This was previously masked by the old combined-weight scorer, which gave local backend_kind a static score advantage over api_provider even with no live metrics — so the tie-through-to-score path happened to still land on the right answer by coincidence, not because locality was genuinely exercised. noMetrics correctly removed that bias (per its own design/tests), which is what exposed the dormant fixture gap.

Confirmed pre-existing and unrelated to any of my recent grid#58/grid#59 changes: reproduces identically with those changes stashed, on main.

Fix

Thread the existing routing_cluster parameter ("site-a" by default) through to network_fixture_json as gatewayRefs[].localSiteName, so the healthy/degraded/metrics fixtures — which all use routingClusterRef = routing_cluster — genuinely resolve to LocalityTier::SameSite, and the API-fallback/invalid fixtures (no matching routingClusterRef) correctly resolve to Unknown.

apply_multi_provider_fixtures has no single "local" site across multiple provider sites, so it passes the network name explicitly, preserving its exact prior fallback behavior (that check validates per-site candidate presence, not locality ordering).

Verification

  • cargo test -p xtask: 459 passed, 0 failed
  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo +nightly-2026-03-28 fmt --all -- --check: clean
  • cargo machete: no unused deps
  • Live local kind cluster run:
    cargo run -p xtask -- env up --config tests/env/config.toml
    cargo run -p xtask -- env verify-operator-reconcile --config tests/env/config.toml
    ...
      [OK] scoring order: site-a (pos 0) before op-e2e-api-fallback (pos 2)
    verify-operator-reconcile: PASS
    

Test plan

  • verify-operator-reconcile passes locally end-to-end
  • Existing xtask unit tests for verify_scoring_order untouched and still pass (that function itself was never the bug)
  • apply_multi_provider_fixtures behavior intentionally unchanged (documented in code comment)

network_fixture_json never set localSiteName, so the operator's
local_site fallback (network name) matched no candidate's site. Every
overlay candidate tied at LocalityTier::Unknown, and GeographyFirst
ordering fell through to score -- tied under the noMetrics default
strategy (grid#38) -- then to the alphabetical (site, name, cluster)
tiebreak, which happens to rank "op-e2e-api-fallback" before "site-a".

This was previously masked by the old combined-weight scorer's
backend_kind bias giving local a higher static score even without real
locality data. noMetrics correctly removed that bias, exposing the
fixture gap as a scoring-order regression (grid#60).

Fix: pass the routingClusterRef used by the local-site fixtures
(healthy/degraded/metrics) as gatewayRefs[].localSiteName, so their
candidates genuinely resolve to LocalityTier::SameSite. The
multi-provider fixture path has no single local site, so it keeps the
prior fallback behavior unchanged.

Verified live: `xtask env verify-operator-reconcile` now passes the
scoring-order assertion, showing site-a (pos 0) before
op-e2e-api-fallback (pos 2).

Fixes praxis-proxy#60

Signed-off-by: Jordi Gil <jgil@redhat.com>
The routing business rule itself ("local ranks before remote/API-provider
under GeographyFirst") already has unit-tier behavioral coverage against
the real renderer in operator::resources::routing_overlay
(score_ordered_local_ranks_before_api_provider,
no_metrics_geography_first_still_prefers_local) -- that was never broken.

What broke was this E2E fixture's wiring, which is meta-tooling rather
than business logic. Add two small unit tests for network_fixture_json so
this specific regression (dropping/mis-wiring localSiteName) is caught by
`cargo test -p xtask` in milliseconds instead of only by a multi-minute
live-cluster E2E run. Verified RED (both tests fail against a
reintroduced copy of the pre-fix bug) before restoring GREEN.

Signed-off-by: Jordi Gil <jgil@redhat.com>

@praxis-bot praxis-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review

Summary: Fixes E2E fixture wiring — network_fixture_json now receives an explicit localSiteName parameter so GeographyFirst locality-tier ordering is genuinely exercised instead of silently falling through to alphabetical tiebreak.

Overall: Clean, well-scoped fix with clear root-cause analysis. The two new tests pin the fixture contract at the unit level. Two medium findings below.

Severity Count
Critical 0
Large 0
Medium 2

Findings without inline placement

[Medium] The PR's central thesis is that fixture wiring regressions should be caught by cargo test -p xtask in milliseconds rather than only by a live-cluster E2E run. The two new tests pin the single-provider path (apply_test_fixtures_for_cluster), but apply_multi_provider_fixtures (line 5540) has no corresponding contract test verifying that its localSiteName equals TEST_NETWORK (intentionally matching no candidate). If someone later changed that call to pass a real site name, locality ordering would silently re-engage in multi-provider validations with no unit-level signal. Add a test analogous to network_fixture_json_sets_local_site_name_from_argument that asserts apply_multi_provider_fixtures's network fixture uses TEST_NETWORK as localSiteName.

Comment thread xtask/src/env/operator.rs Outdated
…omment

Addresses grid#61 review feedback (praxis-bot):

- apply_multi_provider_fixtures's network fixture (TEST_NETWORK as
  localSiteName, intentionally matching no candidate) had no unit-level
  contract test, unlike the single-provider path. Extracted its JSON
  construction into multi_provider_network_fixture_json so it's directly
  testable without a live cluster, and added a test pinning that it
  stays TEST_NETWORK -- if a future change passed a real site name
  here, locality ordering would silently re-engage in multi-provider
  validations with no unit-level signal.
- Removed a test-body comment in
  apply_test_fixtures_for_cluster_wires_routing_cluster_as_local_site_name
  per the project convention (CLAUDE.md, Test Organization: "No
  comments in test bodies — use assertion messages"); the rationale was
  already redundant with the assertion message a few lines below.

Signed-off-by: Jordi Gil <jgil@redhat.com>
@jordigilh

Copy link
Copy Markdown
Contributor Author

Addressed both findings in a290858:

  • Extracted apply_multi_provider_fixtures's network fixture construction into a new multi_provider_network_fixture_json helper so it's directly unit-testable without a live cluster, and added apply_multi_provider_fixtures_network_uses_test_network_as_local_site_name pinning that it stays TEST_NETWORK.
  • Removed the test-body comment flagged inline.

@nerdalert
nerdalert merged commit 115f532 into praxis-proxy:main Aug 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: verify-operator-reconcile scoring order regression - api_provider ranks before local site-a

3 participants