feat(demo): add --kv-cache flag to llm-d pool-metrics demo

[jordigilh]

Summary

Adds a --kv-cache flag to grid-llmd-pool-metrics-demo that switches the
demo's routing signal from the default queue-depth scoring strategy to
llm-d's kv-cache-utilization signal, by setting
GridNetwork.spec.scoringPolicy.strategy: kvCachePressure on both sites
instead of the template's default queueDepth. This lets the demo show
either of Grid's two real, already-implemented scoring strategies without
touching the operator itself — ScoringStrategy::KvCachePressure and its
weights are pre-existing, unit-tested operator behavior
(operator/src/crd/grid_network.rs); this PR only wires the demo's config
templating and CLI surface to select it.

Both signals (queue_depth and kv_cache) are always shown in the live
scorecard regardless of the flag; only the strategy that actually produces
the score/rank driving the A→B failover changes.

Design

• ScoringFlavor enum (QueueDepth default / KvCachePressure) selected
  from the CLI flag, mapping to the scoringPolicy.strategy YAML value and
  to a pressure_phase_active predicate that keys the announced "pressure"
  phase off whichever signal is actually driving the active strategy
  (queue size vs. KV utilization), rather than always narrating queue
  pressure.
• materialize_config swaps the strategy: queueDepth anchor to
  strategy: kvCachePressure on both pool-a-site and pool-b-site when
  the flavor is selected — same checked_replace-based templating pattern
  already used for the mTLS transport transformations in this function.
• DemoContext.scoring_flavor threads the selection through setup and the
  live proof-scenario loop; evidence JSON records it as
  scoring_strategy: "kv-cache-pressure" / "queue-depth".

Testing

• 8 new unit tests: flavor→strategy-yaml mapping, the pressure-trigger
  predicate for both flavors, and both materialize_config outcomes
  (queue-depth leaves the template's default untouched; kv-cache swaps both
  sites, asserted against the same fixture the module's other
  config-templating tests already use).
• Manually confirmed the exact templating target
  (strategy: queueDepth, occurring exactly twice) matches the real
  tests/e2e/topologies/grid-llmd-pool-metrics/forge.yaml byte-for-byte,
  not just the synthetic test fixture.
• cargo test -p xtask: 466/466 passing. cargo clippy -p xtask --all-targets: clean. rustfmt: clean.
• Live-validated on a real 2-cluster Kind deployment (--kv-cache --teardown, full mode): environment converged with the kvCachePressure
  strategy applied on both sites, and the pressure/flip proof scenario
  showed the A→B rank flip driven specifically by the KV-cache signal — the
  queue/capacity pressure ratio stayed pegged at 1.00 for both pools
  throughout, while kv_cache rose 0.00 → 0.02 on pool-a and score
  dropped 1.00 → 0.98, flipping rank to pool-b. Recovery drained cleanly
  back to pool-a. All 4 proof scenarios (provenance, baseline,
  pressure_and_flip, recovery) passed; evidence JSON confirms
  "scoring_strategy": "kv-cache-pressure", "success": true.

Docs

Documented the new --kv-cache flag (and the existing --metrics-mtls
flag, previously undocumented) in the topology's README.

[jordigilh]

Ran a follow-up test-pyramid audit on this PR (per project convention: pyramid invariant, behavioral assurance against acceptance criteria, 100% unit coverage of testable logic):

• Unit coverage (measured with cargo-llvm-cov): every pure/decision-making line this feature adds — ScoringFlavor (enum + its 3 methods), pressure_phase_active, and materialize_config's new strategy-swap branch — is at 100% line coverage. Added one more unit test module (llmd_pool_metrics_demo_cli_tests, latest commit) to close the one remaining testable gap: the --kv-cache clap flag itself was previously untested (true of every flag on this command, not unique to this PR) — now asserts it defaults to false and parses to true via Cli::try_parse_from against the real Cli/Action tree.
• Pyramid shape: 9 fast pure-function unit tests (flavor mapping, pressure predicate, CLI parsing) at the base, 2 file-based config-templating tests in the middle (asserting real YAML output against the actual fixture), and the single live full-mode E2E run at the top (already described above) — broad base, narrow top, no inversion.
• Behavioral assurance: the pressure-predicate tests assert the actual acceptance criterion ("the announced pressure phase must key off whichever signal drives the active strategy, not always queue") rather than just structural coverage; the strategy_yaml test is a standing contract check against the operator's CRD serde rename so the two can't silently drift apart.
• The only intentionally-uncovered-by-unit-tests lines are the pre-existing imperative-shell dispatch (run(), prepare_setup(), the top-level env::run match arm) — consistent with the rest of this file's shape, and exercised instead by the live E2E run rather than mocked out.

[jordigilh]

Ran an independent security review of this diff (the --kv-cache flag and its config-templating path). Summary: no medium/high/critical issues — the flag is a boolean mapped through a 2-variant enum to fixed &'static str literals before being spliced into the Forge YAML via the existing checked_replace helper (fails closed on a match-count mismatch), so there's no path for attacker-controlled or free-form text to reach generated config through this flag. No new deserialization, credential handling, or logging of sensitive data on the changed path.

[jordigilh]

Automated review caught two real defects in the kvCachePressure path — pushed a fix:

1. Recovery proof ignored scoring flavor — proof_recovery's verification-probe gate was hardcoded to queue_size < 3.0 regardless of flavor. Under kvCachePressure, rank is driven by KV utilization, not queue depth, so this could produce flaky/false recovery-proof results in full-mode --kv-cache runs.
2. KV scrape missing metric fallback — scrape_epp_metrics read kv_cache from only inference_pool_average_kv_cache_utilization, unlike queue_size, which already falls back to the llm_d_router_epp_* series. An EPP exposing only that series would read kv_cache as a permanent 0.0.

Fixed via the same TDD discipline as the rest of this PR: 5 new RED tests against not-yet-existing parse_epp_metrics/recovery_condition_met functions → GREEN implementation → REFACTOR (extracted the recovery threshold into a named constant instead of leaving 3.0 as a magic literal). recovery_condition_met mirrors the existing pressure_phase_active flavor dispatch: QueueDepth keeps its original, already-live-validated threshold unchanged; KvCachePressure reuses the inverse of pressure_phase_active rather than inventing a second, uncalibrated KV constant.

100% line coverage of both new functions confirmed with cargo-llvm-cov (all branches of both matches hit). Full suite: 473 passed / 0 failed (was 468). make lint clean (clippy -D warnings, fmt, machete).

[praxis-bot]

Reviewed all three changed files. Clean PR -- no Critical, Large, or Medium issues found.

What looks good:

• ScoringFlavor enum is well-designed: from_kv_cache_flag / label / strategy_yaml cleanly separate CLI surface, display, and YAML-template concerns.
• pressure_phase_active and recovery_condition_met correctly dispatch on flavor, and the lack of a hysteresis gap for KvCachePressure recovery is explicitly documented as intentional ("reuses the inverse of pressure_phase_active rather than inventing an untested second KV constant").
• parse_epp_metrics extraction is a smart refactor -- separating the functional core from the I/O-bound scrape_epp_metrics makes the metric-name-fallback behavior unit-testable.
• materialize_config strategy swap uses checked_replace with count=2, which will fail loudly if the template ever changes its number of scoringPolicy.strategy occurrences.
• 10 new tests cover construction, labels, strategy YAML values, pressure detection for both flavors (including cross-signal independence assertions), metric parsing (primary, fallback, absent), recovery conditions with boundary values, and both materialize_config outcomes. Thorough.
• Existing tests updated to pass the new scoring_flavor parameter without losing coverage.
• README documents both --kv-cache and the previously-undocumented --metrics-mtls flag.

No changes requested.

[nerdalert]

@jordigilh LGTM, rebase at your leisure. Ty!