fix(operator): distinguish SecretMissing from KeyMissing in TLS Secret resolution

[jordigilh]

What

Fixes #58.

secret::read_secret_bytes collapsed two distinct cases into Ok(None): the referenced Secret not existing at all, and the Secret existing but lacking the requested key. endpoint_tls::read_secret_bytes_for_tls (shared by InferenceProvider's metrics/health-check TLS resolution, and verify_tls_accessible's reconcile-time check) could therefore only ever report KeyMissing when a key was present with an empty value — never when it was absent entirely — contradicting TlsFailureReason::KeyMissing's own doc comment and misleading operators diagnosing a live misconfiguration ("Secret doesn't exist" vs. "Secret exists, wrong/missing key name").

How

• Introduced SecretKeyLookup (Found(Vec<u8>) / SecretMissing / KeyMissing) and changed read_secret_bytes to return it directly instead of collapsing to Option<Vec<u8>>.
• read_secret_bytes_for_tls now matches on the three-way result instead of guessing.
• Removed the near-duplicate of this same distinction that verify_tls_accessible's private TlsSecretCheck/read_tls_secret_for_verify had already implemented independently against its own kube::Api call — it now delegates to the same (fixed) read_secret_bytes, so there's a single source of truth instead of two copies that could drift.
• grid_site.rs's gateway-probe path doesn't need the distinction (it maps everything to TrustMaterialMissing regardless), so it uses SecretKeyLookup::into_bytes() to keep its existing Option<Vec<u8>>-based control flow unchanged.

Testing

Added unit tests directly against a mocked kube::Client (tower::service_fn, matching the existing pattern from mcp_probe.rs):

• secret.rs: read_secret_bytes — key found, secret absent, secret with no data section, key absent from an existing secret's data (the bug), key present but empty.
• endpoint_tls.rs: resolve_tls_config and verify_tls_accessible — secret absent → SecretMissing, key absent from an existing secret → KeyMissing (the regression this PR fixes), no TLS configured → None.

cargo test -p operator --lib: 979/979 passing. make lint (workspace clippy -D warnings, nightly cargo fmt --check, cargo machete): clean.

Scope note

This is independent of the still-open #56 (AgentToolProvider) — #56 added a test in mcp_probe.rs (read_tls_material_key_absent_from_data_currently_misreported_as_secret_missing) that intentionally locked in this bug pending this fix, referencing this issue. I'll update that test to assert the corrected behavior when #56 rebases onto main post-merge, per the plan already noted in that test's doc comment.

[jordigilh]

Pushed a follow-up to close the pyramid-invariant gap flagged in review discussion: unit tests alone weren't enough to claim behavioral assurance for the SecretMissing-vs-KeyMissing fix, since nothing exercised it through reconcile()/resolve_phase_and_sites() or against a live cluster.

Added:

• Integration tier (operator/src/controller/inference_provider.rs): two new tests drive resolve_phase_and_sites — the function reconcile() actually calls — through a mocked kube::Client, asserting the real (ProviderPhase::Degraded, status.reason) pair for both the KeyMissing and SecretMissing cases.
• E2E tier (xtask/src/env/operator.rs + mod.rs): a new InferenceProvider fixture (op-e2e-tls-key-missing) with a CA Secret that exists but lacks the expected key, wired into verify-operator-reconcile. Verified locally against a live kind cluster:

  [OK] op-e2e-tls-key-missing phase=Degraded reason="HealthCheckTlsKeyMissing"
  

Full test suite (981 tests), clippy, fmt, and machete all clean.

Side finding (not in scope here): while validating the E2E fixture I hit a pre-existing, deterministic scoring-order failure in verify-operator-reconcile (api_provider candidate ranking ahead of the local one) — reproduced identically with this PR's changes reverted, so it predates and is unrelated to this fix. Filed as #60 rather than fixing it here.

[praxis-bot]

PR Review

Summary: Fixes grid#58 by introducing SecretKeyLookup to distinguish Secret-not-found from key-not-found in TLS Secret resolution, consolidating the duplicated TlsSecretCheck into the shared read_secret_bytes path, and adding thorough regression tests at unit, integration, and e2e tiers.

Overall assessment: Well-executed bug fix. The three-way enum is the right abstraction, the into_bytes() escape hatch keeps the grid_site path clean, and the deduplication of TlsSecretCheck vs SecretKeyLookup is a meaningful consolidation. Test coverage is excellent -- the bug case is exercised at every layer (direct unit, mocked kube client, mocked reconcile, and live e2e). Two medium-severity convention items below.

Severity	Count
Critical	0
Large	0
Medium	2