fix(routing): complete unknown cost-cap operator contract (#1181)

[Wibias]

Summary

• Continues and completes the Routing: define hard cost-cap behavior when runtime cost evidence is unknown #1181 work from fix(routing): define hard cost-cap behavior when cost evidence is unknown #1345 (foreign fork head is not fast-forward writable after rebase onto current dev).
• Keeps default onUnknownCost: \"allow\" fail-open eligibility, but stamps cost.capOutcome: \"unknown-allowed\" on dry-run/live traces so operators can distinguish known-under-cap from unknown cost.
• Opt-in exclude still emits cost-limit-unknown with capOutcome: \"unknown-excluded\"; documents + exposes onUnknownCost in routing docs and the Routing Profile Editor.

Addresses review on #1345

1. High — non-excluding capOutcome on the allow path (not an exclusion).
2. Medium — operator docs + dry-run/live integration coverage.
3. Merge gate — rebased onto current dev.

Credits the original #1345 implementation by @abhisheksharma2411.

Fixes #1181.

Test plan

• bun test tests/cost-cap-unknown-evidence.test.ts tests/cost-scoring.test.ts
• bun x tsc --noEmit
• bun test ./gui/tests/routing-profiles.test.tsx
• Exact-head CI green on this PR
• Confirm fix(routing): define hard cost-cap behavior when cost evidence is unknown #1345 can be closed once this lands (or author cherry-picks)

Summary by CodeRabbit

• New Features

  • Routing profiles can allow unknown cost estimates by default or exclude them with a fail-closed setting.
  • Live routing and dry-run results display cost-cap outcomes and detailed exclusion reasons.
  • Added localized cost-cap labels and exclusion messages across supported languages.

• Documentation

  • Updated routing guides and configuration references with the new setting, defaults, behavior, and examples.

• Bug Fixes

  • Improved consistency when handling incomplete, missing, or non-finite cost evidence.

[github-actions]

✅ Deterministic PR hygiene checks passed.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds configurable handling for unknown routing cost estimates. The default "allow" mode annotates evidence, while "exclude" rejects unknown-cost candidates. The behavior spans evaluation, traces, the editor, localization, tests, and documentation.

Changes

Unknown cost-cap policy

Layer / File(s)	Summary
Cost-cap contracts and normalization src/types.ts, src/routing/profile.ts, src/routing/trace.ts	Defines onUnknownCost and RouteCostCapOutcome, validates profile values, preserves normalized configuration, and filters invalid trace outcomes.
Evaluation and runtime coverage src/routing/evaluator.ts, tests/cost-cap-unknown-evidence.test.ts	Classifies finite and unknown estimates, applies allow or exclude behavior, annotates evidence, and tests evaluator, dry-run, and live routing paths.
Routing profile editor integration gui/src/routing-profile-editor-data.ts, gui/src/pages/RoutingProfiles.tsx, gui/src/i18n/*, gui/tests/routing-profiles.test.tsx	Adds draft and DTO support, serializes "exclude", provides an editor selector, and displays localized outcomes and exclusion reasons.
Configuration and editor documentation docs-site/src/content/docs/guides/routing-profile-editor.md, docs-site/src/content/docs/reference/configuration/routing.md	Documents defaults, trace outcomes, cost-limit-unknown, and the distinction from unknownEvidence.cost.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RoutingProfile
  participant routeModel
  participant RoutingEvaluator
  participant RouteTrace
  RoutingProfile->>routeModel: provide maxEstimatedCostUsd and onUnknownCost
  routeModel->>RoutingEvaluator: evaluate candidate cost evidence
  RoutingEvaluator->>RoutingEvaluator: classify known or unknown cost
  RoutingEvaluator->>RouteTrace: record capOutcome
  RoutingEvaluator-->>routeModel: return eligible or excluded candidate

Loading

Possibly related PRs

• lidge-jun/opencodex#1345: Extends the same unknown-cost policy and routing evaluation areas.
• lidge-jun/opencodex#1344: Modifies the same profile, evaluator, type, trace, and test areas.
• lidge-jun/opencodex#1108: Introduces the routing profile editor extended by this PR.

Suggested reviewers: ingwannu, lidge-jun

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the routing change and the completed unknown cost-cap operator contract.
Linked Issues check	✅ Passed	The changes implement the linked issue’s explicit unknown-cost policy across evaluation, traces, live and dry-run routing, documentation, and GUI configuration.
Out of Scope Changes check	✅ Passed	The documentation, localization, GUI, evaluator, profile, trace, type, and test changes directly support the unknown cost-cap contract.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/1345-cost-cap-contract

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ READY

• all PR quality gates passed.

UI screenshot waived by the gui-screenshot-waived label.

Hygiene

✅ Deterministic PR hygiene checks passed.

[abhisheksharma2411]

Thanks for carrying this forward, and for the credit — makes sense given the fork head wasn't fast-forward writable after the rebase. Confirming #1345 can be closed once this lands; no need for me to cherry-pick.

The capOutcome approach is a better answer to the operator-contract requirement than what I had. Keeping the non-excluding outcomes out of exclusions and whitelisting the values in parseCost both look right, and the four-state enum makes satisfied vs unknown-allowed legible in a way my version couldn't.

Two observations on the stamping in evaluatePolicyProfile:

1. Stamping limitUsd can change costScore()'s denominator for callers that didn't supply it.

costForCandidate now sets limitUsd: costLimit whenever a cap is configured, and cost is reassigned to it before scoring. costScore() picks its reference as:

const reference = typeof evidence.limitUsd === "number" && evidence.limitUsd > 0
  ? evidence.limitUsd
  : COST_SCORE_REFERENCE_USD;

So for any caller that passes cost evidence with a known estimatedUsd but no limitUsd, the reference silently moves from COST_SCORE_REFERENCE_USD (1.0) to the configured cap — which changes the score and therefore candidate ranking. Every in-tree caller (router.ts, routing-profile-routes.ts) already sets limitUsd via costEvidenceForCandidate, so I don't think this is reachable today. Flagging it as intent rather than a defect: if the normalization is deliberate, it might be worth a line in the comment saying so, since it's a scoring change hiding inside a trace-stamping change.

2. Synthesized cost evidence has no incomplete marker.

When a cap is set and the caller supplied no cost evidence at all, the candidate now gets cost: { limitUsd, capOutcome: "unknown-allowed" }. That block reads as present-but-unknown without incomplete: true, which costEvidenceForCandidate otherwise always sets for missing usage or price. Consider spreading in incomplete: true when synthesizing from nothing, so an operator (or anything consuming the trace) can't mistake it for evidence that simply lacks a price.

Neither blocks merge from my side. Happy to review again after the exact-head CI run, and thanks for the thorough review on #1345 — the fail-open ambiguity was a real gap.

[Wibias]

Supersedes #1345. Continues @abhisheksharma2411's #1181 cost-cap work on current dev with the remaining operator-contract fixes.

[abhisheksharma2411]

Re-reviewed at b944703. The docs alignment, the i18n mapping, and normalizeRouteDecisionTrace dropping capOutcome without a finite limitUsd all look right, and the dry-run/live parity tests answer the integration-coverage point cleanly.

One thing got sharper rather than resolved after c248690, and I think it needs a decision before merge.

Normalizing limitUsd to costLimit also moves costScore()'s denominator.

costForCandidate is used for two different purposes: it's stamped into the trace, and it's what gets passed to costScore() at the scoring step. CodeRabbit's finding was about the first (the trace must report the cap actually applied — agreed, and the new test at line 201 pins it). But the same object now feeds scoring, and costScore() picks its reference as:

const reference = typeof evidence.limitUsd === "number" && evidence.limitUsd > 0
  ? evidence.limitUsd
  : COST_SCORE_REFERENCE_USD;
return Math.max(0, Math.min(1, 1 - evidence.estimatedUsd / reference));

So for any caller whose evidence carries a limitUsd different from the profile cap, the cost score changes silently. Concretely, with maxEstimatedCostUsd: 0.5 and incoming evidence { estimatedUsd: 0.4, limitUsd: 1 }:

	reference	cost score
before this PR	1 (caller's)	1 - 0.4/1 = 0.60
at b944703	0.5 (profile cap)	1 - 0.4/0.5 = 0.20

The candidate stays eligible and correctly stamps capOutcome: "satisfied" in both cases — but its cost component drops by two thirds, and with optimize.cost weighted that can reorder candidates. The existing line-201 test doesn't catch it because 0.01 > 0.000001 makes that candidate ineligible anyway, so the score is never consulted.

I don't think this is reachable from router.ts or routing-profile-routes.ts today, since both build evidence via costEvidenceForCandidate({ limitUsd: profile.limits.maxEstimatedCostUsd }) and the two values already agree. But line 201 establishes divergent limitUsd as a supported input shape, which makes the scoring path reachable for anyone constructing evidence directly.

Two ways to close it, either fine by me:

• Separate the concerns — keep the trace copy as-is, and score against the caller's original evidence.cost, so a trace-reporting fix can't move ranking.
• Declare the normalization intentional — state in the comment that the profile cap is the authoritative scoring reference and a caller-supplied limitUsd is advisory, then add a test asserting the score for an eligible candidate whose incoming limitUsd differs from the cap (e.g. the 0.4/0.5/1 case above), so the behaviour is pinned rather than incidental.

The second is probably the better contract — the profile is the authority — but right now it's implied by a change made for a different reason, and nothing pins it.

Nothing else blocking from me; the rest of the contract work reads well.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/routing/evaluator.ts`:
- Around line 341-345: Update the cost normalization around costForCandidate and
the cap calculation so non-finite estimatedUsd values are treated as unknown,
not preserved as valid trace metadata. Apply the same finite-estimate rule used
near the cap logic, mark such evidence incomplete, and avoid emitting Infinity
or other non-finite estimates; add coverage for an evidence object containing a
non-finite estimatedUsd with incomplete false.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 7328e902-e196-4fa4-a8a6-87dfcd16f806

📥 Commits

Reviewing files that changed from the base of the PR and between b944703 and 6be0959.

📒 Files selected for processing (2)

• src/routing/evaluator.ts
• tests/cost-cap-unknown-evidence.test.ts

[abhisheksharma2411]

Just saw 6be09595 — that's exactly it, and much cleaner than either option I suggested. Scoring off the caller's evidence and keeping the stamped copy purely for the trace makes the split obvious to the next person reading it, and the comment you left there says it better than my write-up did. Nice catch on threading incomplete through the synthesized case too.

Looks good to me. Thanks for being so quick on all of this — it's been genuinely fun to follow, and I appreciate you taking the time to explain the reasoning on #1345 rather than just closing it. Looking forward to seeing it land 🙂

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs-site/src/content/docs/guides/routing-profile-editor.md`:
- Around line 33-34: Update the routing profile editor guide’s
`limits.onUnknownCost: "exclude"` documentation to state that evaluation emits
both `cost-limit-unknown` and `cost.capOutcome: "unknown-excluded"` in dry-run
and live traces, while preserving the existing exclusion behavior and related
unknown-cost guidance.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9fbede65-0e59-4d81-889b-ee2426da4952

📥 Commits

Reviewing files that changed from the base of the PR and between 6be0959 and 51b1bbe.

📒 Files selected for processing (13)

• docs-site/src/content/docs/guides/routing-profile-editor.md
• docs-site/src/content/docs/reference/configuration/routing.md
• gui/src/i18n/de.ts
• gui/src/i18n/en.ts
• gui/src/i18n/ja.ts
• gui/src/i18n/ko.ts
• gui/src/i18n/ru.ts
• gui/src/i18n/tr.ts
• gui/src/i18n/zh.ts
• gui/src/pages/RoutingProfiles.tsx
• gui/tests/routing-profiles.test.tsx
• src/routing/evaluator.ts
• tests/cost-cap-unknown-evidence.test.ts

[abhisheksharma2411]

Fixes look good to me — the scoring split in 6be09595 is cleaner than either option I suggested, and threading incomplete through the synthesized case ties it off nicely.

Happy with this one. Thanks for the quick turnaround 🙂

[Wibias]

Thanks @abhisheksharma2411 — both points were right. Separated scoring from the stamped trace copy (so limitUsd normalization can’t move costScore / ranking), and mark synthesized-from-nothing cost evidence incomplete: true. Covered in 6be09595.

[Wibias]

Thanks @abhisheksharma2411 — merging this.

Why it helps: your #1345 work correctly diagnosed that live routing builds cost evidence without usage, added the opt-in onUnknownCost: "exclude" path with cost-limit-unknown, and kept the default allow behavior non-breaking. #1404 finishes that operator contract on tip — cost.capOutcome on dry-run/live traces, docs/editor wiring, and scoring kept separate from trace stamping — so #1181 is complete.

Ship it.