diff --git a/.claude/skills/add-llm-adapter/SKILL.md b/.claude/skills/add-llm-adapter/SKILL.md index 4b0221ad..4e8a4e9e 100644 --- a/.claude/skills/add-llm-adapter/SKILL.md +++ b/.claude/skills/add-llm-adapter/SKILL.md @@ -1,7 +1,7 @@ --- name: add-llm-adapter description: > - Add a new LLM provider adapter behind the LLMProvider seam in packages/llm without leaking a single vendor SDK type across it: implement the seam interface, normalize system-prompt placement / tool schema / tool-call round-trip / streaming events / stop reasons / usage to the canonical Relavium types, register a pricing entry for cost, wire it into the fallback runner config, and add the per-provider conformance test (recorded fixtures on PR, live nightly). USE FOR: integrating a new model provider into @relavium/llm. DO NOT USE FOR: adding a new top-level workspace package (use ../add-package/SKILL.md), changing the seam *interface* itself (that supersedes ADR-0011 — use ../write-adr/SKILL.md), or restating the seam contract (it has one canonical home in docs/reference/shared-core/llm-provider-seam.md). + Add a new LLM provider adapter behind the LLMProvider seam in packages/llm without leaking a single vendor SDK type across it: implement the seam interface, normalize system-prompt placement / tool schema / tool-call round-trip / streaming events / stop reasons / usage to the canonical Relavium types, map the provider into the generated model catalog for cost, wire it into the fallback runner config, and add the per-provider conformance test (recorded fixtures on PR, live nightly). USE FOR: integrating a new model provider into @relavium/llm. DO NOT USE FOR: adding a new top-level workspace package (use ../add-package/SKILL.md), changing the seam *interface* itself (that supersedes ADR-0011 — use ../write-adr/SKILL.md), or restating the seam contract (it has one canonical home in docs/reference/shared-core/llm-provider-seam.md). --- # Add an LLM provider adapter @@ -11,7 +11,7 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a ## When to use - A new API-based provider must be callable from an agent node (a model id routes to it). - A provider you already support ships a new wire dialect that needs its own normalization path. -- Note the existing inventory is **three** adapters: a dedicated `AnthropicAdapter` (`@anthropic-ai/sdk`), a dedicated `GeminiAdapter` (`@google/genai`), and **one shared OpenAI-compatible adapter** (the `openai` SDK) serving both OpenAI and DeepSeek (DeepSeek via a custom `baseURL`). If the new provider speaks the OpenAI-compatible wire format, **extend that shared adapter with a new `baseURL` + pricing + capability entry — do not write a fourth adapter.** +- Note the existing inventory is **three** adapters: a dedicated `AnthropicAdapter` (`@anthropic-ai/sdk`), a dedicated `GeminiAdapter` (`@google/genai`), and **one shared OpenAI-compatible adapter** (the `openai` SDK) serving both OpenAI and DeepSeek (DeepSeek via a custom `baseURL`). If the new provider speaks the OpenAI-compatible wire format, **extend that shared adapter with a new `baseURL` + capability entry (pricing comes from the generated catalog, not a hand-typed table) — do not write a fourth adapter.** ## When not to use - You need to change the seam *interface* — add a method, a chunk variant, or let a vendor type through. That supersedes [ADR-0011](../../../docs/decisions/0011-internal-llm-abstraction.md); stop and run ../write-adr/SKILL.md. The seam is immovable; this skill works strictly behind it. @@ -25,12 +25,12 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a | Provider id | The literal added to the closed `ProviderId` enum — its **canonical home is `LLM_PROVIDERS`** in `packages/shared/src/constants.ts` (`ProviderId` aliases `LlmProviderId = (typeof LLM_PROVIDERS)[number]`, and `ProviderIdSchema = z.enum(LLM_PROVIDERS)`), never a vendor enum. The CLI mirrors it in `KNOWN_PROVIDER_IDS`/`KNOWN_PROVIDERS` (step 7). A genuinely-new arbitrary id **opens** the closed enum — a deliberate supersede per [ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md) §6, not a silent edit. | | SDK / transport | The official TS SDK to wrap, or `openai` + a custom `baseURL` if OpenAI-compatible. An SDK dependency stays strictly inside `packages/llm/src/adapters/*`. | | Capabilities | `{ tools, streaming, parallelToolCalls, vision, promptCache, reasoning }` — what this provider genuinely supports, for the `supports` capability flags. | -| Model ids + pricing | Canonical model ids this adapter serves and their per-token input/output (and cache) prices, for the pricing table. | +| Model ids + catalog key | Canonical model ids this adapter serves, and the provider's key in the upstream models.dev catalog (`CATALOG_PROVIDER_KEYS`) — pricing / context / limits are GENERATED from it, never hand-typed. | | Wire facts | The provider's native system-prompt placement, tool schema shape, tool-call/result round-trip, streaming events, stop reasons, and usage fields — the six things to normalize. | ## Workflow 1. **Confirm the seam doesn't have to move.** Read [llm-provider-seam.md](../../../docs/reference/shared-core/llm-provider-seam.md) end to end. Map every one of this provider's behaviors onto the *existing* canonical types (`LlmRequest`, `ContentPart`, `LlmResult`, `StopReason`, `Usage`, `StreamChunk`, `CapabilityFlags`). If something genuinely does not fit, that is an ADR (../write-adr/SKILL.md) superseding ADR-0011 — not an edit here. Reach provider-specific features only through the typed `providerOptions` escape hatch and the `supports` flags, never by widening the seam. -2. **Decide adapter vs. shared-OpenAI extension.** If the provider is OpenAI-wire-compatible, extend the shared OpenAI-compatible adapter with a new `baseURL` + capability + pricing entry (the DeepSeek pattern). Otherwise add a new file under `adapters/`: +2. **Decide adapter vs. shared-OpenAI extension.** If the provider is OpenAI-wire-compatible, extend the shared OpenAI-compatible adapter with a new `baseURL` + capability entry (the DeepSeek pattern; pricing comes from the generated catalog). Otherwise add a new file under `adapters/`: ```text packages/llm/src/ ├── types.ts # the seam — DO NOT edit to fit a provider @@ -39,7 +39,7 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a │ ├── openai-compatible-adapter.ts # OpenAI + DeepSeek (+ new wire-compatible providers) │ ├── gemini-adapter.ts │ └── -adapter.ts # NEW — only if not OpenAI-compatible - ├── pricing.ts # canonical-model-id → per-token price (add an entry) + ├── pricing.ts # catalog → price PROJECTION (generated; do NOT hand-edit — see Step 5) ├── fallback.ts # withFallback runner config (register the provider) ├── provider-factory.ts # id → adapter (register the id) └── conformance/ @@ -55,7 +55,22 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a 4. **Streaming events** — fold the native event stream into the one `StreamChunk` union (`text_delta` / `tool_call_start` / `tool_call_delta` / `tool_call_end` / `stop` / `error`). Concatenate tool-arg JSON deltas across `tool_call_delta` and **parse once at `tool_call_end`**. Some providers need an opt-in to emit final usage (OpenAI's `stream_options:{include_usage:true}`) — set it. 5. **Stop reasons** — map every native reason onto the five-value `StopReason` enum (`stop | length | tool_use | content_filter | error`). No native string leaks out. 6. **Usage** — map native token fields into `Usage.inputTokens`/`outputTokens` (+ `cacheReadTokens`/`cacheWriteTokens` where the provider exposes them). The final `stop` chunk always carries `stopReason` + `usage`. -5. **Register a pricing entry — cost is ours.** Add the canonical model id(s) and per-token prices to the pricing table (`pricing.ts`). `CostTracker` computes the cost from *our* table keyed on the **canonical model id** — never read a cost number from a provider response. This is the same `costMicrocents` that surfaces in the `cost:updated` run event ([sse-event-schema.md](../../../docs/reference/contracts/sse-event-schema.md)); store cost as integer micro-cents (1 micro-cent = 1e-8 USD), never a float. +5. **Map the provider into the catalog — do NOT hand-write prices** ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md)). Model metadata (price, context window, **max output**, and the **reasoning control's shape + accepted tiers**) is *generated*, not typed. Add **one line** to `CATALOG_PROVIDER_KEYS` (`packages/llm/src/catalog/`) mapping your `ProviderId` to the provider's key in the upstream catalog, then re-run `pnpm sync:models` and **review the generated diff** like any other change: + + ```ts + const CATALOG_PROVIDER_KEYS: Record = { + …, + yourprovider: 'their-upstream-key', // ← the whole integration + }; + ``` + + Two rules this replaces the old hand-typed table for, and why: + - **Never hand-type a price, a context window, or a max-output value.** The 12-row table this supersedes drifted silently (it claimed `claude-sonnet-4-6` maxed at 64k output; it is 128k) and priced only 12 of ~97 reachable models — so [ADR-0028](../../../docs/decisions/0028-workflow-resource-governance.md)'s cost cap silently did not apply to the rest. + - **Never hand-write `reasoning: true/false`.** The reasoning control's shape is **per model**, not per provider — `gemini-2.5-*` takes a `thinkingBudget` while `gemini-3.x` takes a `thinkingLevel`, and assuming one shape for a whole adapter is what produced a live bug ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md)'s dated correction note). Let the catalog say it; compute the accepted tiers with `acceptedTiers(provider, model)`. + + **If the upstream catalog does not cover your provider** (a bespoke or self-hosted endpoint), that is a *supported* case, not an error: its models simply arrive unpriced, exactly like a brand-new model, and a user prices them with `relavium models pricing` ([ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md)). Do not re-introduce a hand-typed table to work around it. + + Cost itself stays **ours**: `CostTracker` computes it from the catalog keyed on the **canonical model id** — never read a cost number from a provider response. It is the same `costMicrocents` that surfaces in the `cost:updated` run event ([sse-event-schema.md](../../../docs/reference/contracts/sse-event-schema.md)); store it as integer micro-cents (1 micro-cent = 1e-8 USD), never a float. 6. **Wire it into provider selection and the fallback runner.** Register the id in the provider factory and make it selectable by the `withFallback(providers)` runner so an agent's `fallback_chain` can list it. The chain is policy and lives outside the adapter — the adapter stays dumb. The `fallback_chain` field shapes (`model`, `provider`, `max_attempts`) are canonical in [agent-yaml-spec.md](../../../docs/reference/contracts/agent-yaml-spec.md); do not redefine them. Errors must surface as a classified `LlmError` (retryable vs. fatal per [error-handling.md](../../../docs/standards/error-handling.md)) so the runner knows when to fail over. 7. **Register the provider on the CLI so its onboarding + management surfaces light up (data-driven — no per-surface UI edit).** The id lives in **two** homes that mirror the seam's closed set; every CLI surface then derives from them: 1. **`LLM_PROVIDERS`** (`packages/shared/src/constants.ts`) — the canonical closed `ProviderId` enum (`ProviderId` aliases `LlmProviderId = (typeof LLM_PROVIDERS)[number]`; `ProviderIdSchema = z.enum(LLM_PROVIDERS)`; the **persisted** run-event `provider` field + authored agent YAML). Adding an *arbitrary* new id here opens the closed enum → an ADR ([ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md) §6 supersede), not a silent edit. @@ -77,8 +92,8 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a 10. **Commit** with ../commit-and-pr/SKILL.md scoped to the package: `feat(llm): add adapter behind the LLMProvider seam` with a `Refs: ADR-0011` trailer. ## Outputs -- A new `adapters/-adapter.ts` (or a new `baseURL`/capability/pricing entry on the shared OpenAI-compatible adapter) implementing `LlmProvider` with full six-axis normalization. -- A pricing-table entry per canonical model id; the provider registered in the factory and selectable in the fallback runner. +- A new `adapters/-adapter.ts` (or a new `baseURL`/capability entry on the shared OpenAI-compatible adapter) implementing `LlmProvider` with full six-axis normalization. +- One line in `CATALOG_PROVIDER_KEYS` + a re-run `pnpm sync:models` whose generated diff was reviewed; the provider registered in the factory and selectable in the fallback runner. - The provider added to the conformance matrix with committed fixtures (PR) wired into the nightly live run. - No vendor SDK type anywhere above the adapter; the boundary lint green. @@ -86,7 +101,7 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a - [ ] The seam interface was **not** changed; everything fits the existing canonical types, with provider-specific features only via `providerOptions`/`supports`. - [ ] SDK import is confined to `adapters/*`; no vendor type is re-exported or crosses into `packages/core`; `raw` is `unknown`. - [ ] All six normalizations implemented: system-prompt placement, tool schema (with unsupported-keyword stripping if restricted), tool-call/result round-trip (with id synthesis if the provider has no ids), streaming → `StreamChunk`, stop reasons → the 5-value enum, usage → `Usage`. -- [ ] Pricing entry added; cost computed from our table on the canonical model id and stored as integer micro-cents (`costMicrocents`), never read from the provider or stored as a float. +- [ ] Provider mapped into the catalog (`CATALOG_PROVIDER_KEYS` + `pnpm sync:models`) and the generated metadata reviewed; cost computed from the catalog on the canonical model id and stored as integer micro-cents (`costMicrocents`), never read from the provider or stored as a float. - [ ] Provider registered in the factory and usable in a `fallback_chain`; errors classified as `LlmError` (retryable/fatal). - [ ] Id registered in **both** `LLM_PROVIDERS` (`@relavium/shared`) and `KNOWN_PROVIDER_IDS`/`KNOWN_PROVIDERS` (the CLI, with a `testModel`) so the wizard / `provider` / `/doctor` / `/models` surface it data-driven; the `LLM_PROVIDERS`↔`KNOWN_PROVIDER_IDS` lock-step guard test (`providers.test.ts`) is green. A genuinely-new arbitrary id opening the closed enum is an ADR-0065 §6 supersede, not a silent edit. - [ ] Key handling host-aware (ADR-0018): a resolved key attached in-adapter on the Node-style hosts; a key *reference* passed to the Rust `llm_stream` egress on desktop (raw key never in the WebView); never logged/checkpointed/sent to the frontend; `AbortSignal` threaded through. @@ -99,7 +114,7 @@ Integrate a new model provider into `packages/llm` (`@relavium/llm`) as a thin a - **Leaking a vendor type across the seam** — typing `raw` as a vendor shape, re-exporting an SDK enum, or pattern-matching a vendor chunk in `packages/core`. This is the one failure ADR-0011 exists to prevent. - Widening the seam interface to fit a provider instead of using `providerOptions` — that is an ADR, not an edit. - Writing a fourth adapter for an OpenAI-compatible provider instead of extending the shared one with a `baseURL`. -- Trusting a provider's own cost field instead of computing the cost from our pricing table on the canonical model id; or storing cost as a float instead of integer micro-cents. +- Trusting a provider's own cost field instead of computing the cost from the generated catalog on the canonical model id; or storing cost as a float instead of integer micro-cents. - Forgetting the provider-specific stream quirk (e.g. OpenAI's `include_usage` opt-in) so the final `stop` chunk has no usage. - Passing a restricted-provider tool schema through without stripping unsupported keywords, or losing tool-call ids on a no-id provider. - Hand-editing a recorded fixture instead of regenerating it; committing a live API key or logging the key. diff --git a/.github/workflows/models-catalog.yml b/.github/workflows/models-catalog.yml new file mode 100644 index 00000000..f686814d --- /dev/null +++ b/.github/workflows/models-catalog.yml @@ -0,0 +1,115 @@ +# The model-catalog guards (ADR-0071 §9). Two scheduled lanes, neither of which gates a PR: +# +# • weekly-catalog-check — did any ALREADY-SHIPPED model's price MOVE or VANISH? A moved price feeds the +# ADR-0028 cost cap, so it is a HUMAN decision surfaced as a red check, never a silent bot commit. `pnpm +# sync:models` exits non-zero on a moved/vanished shipped price; benign additive drift stays green (a weekly +# red-no-matter-what trains the maintainer to ignore it — the erosion sync.mjs's own comments warn against). +# +# • nightly-effort-conformance — the ONLY mechanism that catches a stale catalog re-introducing the reasoning +# bug: for each shipped reasoning model it asks the REAL provider whether it accepts every tier the catalog +# claims. A one-off manual probe proves a fact once; the catalog drifts continuously, so this runs nightly. +# +# Third-party actions are pinned to a full commit SHA (the `# vX.Y.Z` comment tracks the release) — the same +# convention ci.yml uses, so a moved tag cannot inject unreviewed code. +name: model-catalog + +on: + schedule: + # Monday 06:00 UTC — the drift check. Ahead of a typical review week, so a red one is seen on Monday. + - cron: '0 6 * * 1' + # 07:00 UTC nightly — the live effort conformance (key-gated; skips with no key). + - cron: '0 7 * * *' + # Manual trigger for both, so a maintainer can run either on demand without waiting for the cron. + workflow_dispatch: {} + +permissions: + contents: read + +concurrency: + group: model-catalog-${{ github.event_name }} + cancel-in-progress: false + +jobs: + weekly-catalog-check: + name: shipped-price-change guard + # Only the weekly cron (and a manual run); the nightly cron skips this job. + if: github.event.schedule != '0 7 * * *' + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + persist-credentials: false + - name: Set up pnpm + uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + - name: Set up Node + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version-file: .nvmrc + cache: pnpm + - name: Install (frozen lockfile) + run: pnpm install --frozen-lockfile + # `pnpm sync:models` builds `@relavium/llm` AND its `@relavium/shared` dependency first (via turbo `^build`) — + # the `@relavium/*` `exports` resolve only to `dist/`, which is gitignored, so a fresh runner has no built + # `@relavium/shared` and the sync's own `tsc`/dist import would otherwise die on TS2307 before any guard runs. + # `pnpm sync:models` fetches models.dev, normalizes it, and applies the money guards. It fails RED on the ONE + # thing that must be a human decision: a MOVED or VANISHED price on a model we already ship (a rate that + # silently moves also silently moves how much the ADR-0028 cost cap protects). It does NOT red on benign + # additive drift — models.dev adds models in our four providers constantly, and a check that is red every + # week trains the maintainer to ignore it, which is exactly how the price-change protection would erode + # (sync.mjs says so in its own comments). New models "merge automatically" via the deferred auto-PR + # (deferred-tasks.md); until it lands they are simply picked up by the next local `pnpm sync:models`. The + # snapshot this job writes into the ephemeral CI checkout is discarded — only the exit code is the guard. + - name: No ALREADY-SHIPPED model's price moved or vanished + run: pnpm sync:models + + nightly-effort-conformance: + name: effort conformance (live) + # Only the nightly cron (and a manual run); the weekly cron skips this job. + if: github.event.schedule != '0 6 * * 1' + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + persist-credentials: false + - name: Set up pnpm + uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + - name: Set up Node + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version-file: .nvmrc + cache: pnpm + - name: Install (frozen lockfile) + run: pnpm install --frozen-lockfile + # Build the dependency closure first: the conformance suite imports `@relavium/shared`, whose `exports` resolve + # only to a gitignored `dist/`. Without this a fresh runner cannot resolve the cross-package entry and EVERY + # conformance file errors out before a single live probe — a red that looks identical to a real catalog drift. + # Turbo `^build` builds `@relavium/shared` ahead of `@relavium/llm` (mirrors ci.yml's pre-test build). + - name: Build workspace dependencies + run: pnpm turbo run build --filter=@relavium/llm + # Guard against a FALSE-GREEN nightly: the live cases are `it.skipIf( === '')`, so with NO provider secret + # configured EVERY probe skips and the suite passes — a green that means "nothing was checked", indistinguishable + # from "all tiers verified". Fail loudly instead, so a green nightly is proof that probes actually ran. + - name: Require at least one provider secret + run: | + if [ -z "${ANTHROPIC_API_KEY}${OPENAI_API_KEY}${GEMINI_API_KEY}${DEEPSEEK_API_KEY}" ]; then + echo "::error::No provider secret is configured — every live effort probe would SKIP, and a green run would falsely read as 'all tiers verified'. Configure at least one of ANTHROPIC/OPENAI/GEMINI/DEEPSEEK_API_KEY." + exit 1 + fi + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} + DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }} + # The conformance suite (fixtures + the `it.skipIf( === '')` live cases). With the provider keys set here + # the live cases execute — and a catalog that claims a tier a model rejects fails red — while a missing key + # simply skips that provider's lane. Scoped to `src/conformance` so a flaky live case cannot widen the red + # surface to the whole package's unit suite. + - name: Live effort conformance + run: pnpm --filter @relavium/llm exec vitest run src/conformance + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }} + DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }} diff --git a/CLAUDE.md b/CLAUDE.md index 37fe48b3..4471a8d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,8 +47,10 @@ inbound MCP client, and the full YAML lifecycle (`create`/`import`/`export`). Home, the two-registry command system (`/` palette + shell commands), per-tool approval/modes (ask/plan/accept-edits/auto), `@`-mention file injection, context compaction, the onboarding wizard + live model catalog, reasoning rendering, and regression -hardening. **Phase 2.6 (Conversational Authoring and the First-Class CLI) is next up** -— a full-screen Home-managed CLI with conversational workflow authoring, management +hardening. **Phase 2.6 (Conversational Authoring and the First-Class CLI) is in progress** — two +workstreams are merged to `main`: **2.6.F** (the platform floor + the full-screen TUI renderer, +PR #74, 2026-07-11) and **2.6.C** (the reseat transcript-carry + the `/cost` per-model breakdown, +PR #75, 2026-07-13). The phase is a full-screen Home-managed CLI with conversational workflow authoring, management browsers, competitor-breadth tools, settings/theming/`en`+`tr` localization, and the run-ops resume follow-up. diff --git a/apps/cli/src/chat/agent-source.test.ts b/apps/cli/src/chat/agent-source.test.ts index d3972d7b..52866c23 100644 --- a/apps/cli/src/chat/agent-source.test.ts +++ b/apps/cli/src/chat/agent-source.test.ts @@ -36,6 +36,33 @@ describe('resolveChatAgent', () => { expect(agent.provider).toBe('openai'); }); + it('passes opts.defaultProvider VERBATIM to the default agent — an unplaceable id starts (ADR-0059, Bug-3 wiring)', () => { + // The decisive Bug-3 hop: a persisted provider carried into the default agent so a live-discovered id the prefix + // map cannot place (`chat-latest`) still resolves. Without this passthrough it throws "cannot infer a provider". + const agent = resolveChatAgent(undefined, { + cwd: dir, + projectConfigDir: undefined, + defaultModel: 'chat-latest', + defaultProvider: 'openai', + }); + expect(agent.model).toBe('chat-latest'); + expect(agent.provider).toBe('openai'); + }); + + it('still throws a clean CliError for an unplaceable id when NO defaultProvider is persisted', () => { + let caught: unknown; + try { + resolveChatAgent(undefined, { + cwd: dir, + projectConfigDir: undefined, + defaultModel: 'chat-latest', + }); + } catch (err) { + caught = err; + } + expect(isCliError(caught)).toBe(true); + }); + it('falls back to DEFAULT_CHAT_MODEL when neither --agent nor [chat].default_model is set', () => { const agent = resolveChatAgent(undefined, { cwd: dir, diff --git a/apps/cli/src/chat/agent-source.ts b/apps/cli/src/chat/agent-source.ts index d608843a..c238ee80 100644 --- a/apps/cli/src/chat/agent-source.ts +++ b/apps/cli/src/chat/agent-source.ts @@ -1,5 +1,5 @@ import { parseAgent, type AgentDefinition } from '@relavium/core'; -import type { ReasoningEffort } from '@relavium/shared'; +import type { LlmProviderId, ReasoningEffort } from '@relavium/shared'; import { resolveYamlSource } from '../workflows/resolve.js'; import { buildDefaultChatAgent, DEFAULT_CHAT_MODEL } from './default-agent.js'; @@ -9,6 +9,9 @@ export interface ResolveChatAgentOptions { readonly projectConfigDir: string | undefined; /** The resolved `[chat].default_model` — used only to build the default agent when no `--agent` is given. */ readonly defaultModel: string | undefined; + /** The resolved `[chat].default_provider` (ADR-0059) — persisted at pick time, used verbatim for the DEFAULT + * agent so a live-discovered id whose prefix the inference cannot place still resolves. Absent ⇒ inference. */ + readonly defaultProvider?: LlmProviderId; /** The resolved `[chat].reasoning_effort` (ADR-0066) — baked onto the DEFAULT agent only (an authored `--agent` * owns its own `reasoning_effort`, never overridden by config). Absent ⇒ no reasoning control. */ readonly reasoningEffort?: ReasoningEffort; @@ -28,7 +31,11 @@ export function resolveChatAgent( opts: ResolveChatAgentOptions, ): AgentDefinition { if (agentRef === undefined) { - return buildDefaultChatAgent(opts.defaultModel ?? DEFAULT_CHAT_MODEL, opts.reasoningEffort); + return buildDefaultChatAgent( + opts.defaultModel ?? DEFAULT_CHAT_MODEL, + opts.reasoningEffort, + opts.defaultProvider, + ); } const source = resolveYamlSource(agentRef, { cwd: opts.cwd, diff --git a/apps/cli/src/chat/default-agent.test.ts b/apps/cli/src/chat/default-agent.test.ts index e3fad0d5..762d2fb2 100644 --- a/apps/cli/src/chat/default-agent.test.ts +++ b/apps/cli/src/chat/default-agent.test.ts @@ -28,6 +28,14 @@ describe('inferProviderFromModel', () => { expect(inferProviderFromModel('llama-3')).toBeUndefined(); expect(inferProviderFromModel('')).toBeUndefined(); }); + + it('resolves a catalog-known id via the catalog branch (ADR-0071 hardening)', () => { + // Every SHIPPED id ALSO matches a prefix, so the catalog and the heuristic agree — this cannot prove the catalog + // is consulted FIRST, only that the catalog branch is wired and returns the right provider. The catalog-first + // ORDER only bites for a live-only id the prefix cannot place (e.g. `chatgpt-4o-latest`), which no shipped + // catalog id exercises; there the PERSISTED provider (see buildDefaultChatAgent) is the decisive fix. + expect(inferProviderFromModel('gpt-5-chat-latest')).toBe('openai'); // a real catalog id + }); }); describe('buildDefaultChatAgent', () => { @@ -56,6 +64,21 @@ describe('buildDefaultChatAgent', () => { expect(buildDefaultChatAgent('gemini-2.5-pro').provider).toBe('gemini'); }); + it('uses a persisted knownProvider VERBATIM, skipping inference — the Bug-3 fix (ADR-0059)', () => { + // `chat-latest` is a live OpenAI id `keepOpenAiModelId` admits; it has no gpt/o-digit prefix and is not in the + // catalog, so inference returns undefined and the chat used to crash "cannot infer a provider". A provider + // persisted at pick time is used as-is, so the chat starts on the right provider. + const agent = buildDefaultChatAgent('chat-latest', undefined, 'openai'); + expect(agent.provider).toBe('openai'); + expect(agent.model).toBe('chat-latest'); + }); + + it('a knownProvider makes the throw path unreachable even for an id inference cannot place', () => { + expect(buildDefaultChatAgent('mystery-model-9', undefined, 'anthropic').provider).toBe( + 'anthropic', + ); + }); + it('bakes the [chat].reasoning_effort default onto the agent, and OMITS it when absent (ADR-0066)', () => { expect(buildDefaultChatAgent('claude-opus-4-8', 'high').reasoning_effort).toBe('high'); // Absent ⇒ the key is not present (never an explicit `undefined` under exactOptionalPropertyTypes). diff --git a/apps/cli/src/chat/default-agent.ts b/apps/cli/src/chat/default-agent.ts index 9be6352b..38f02e20 100644 --- a/apps/cli/src/chat/default-agent.ts +++ b/apps/cli/src/chat/default-agent.ts @@ -1,4 +1,4 @@ -import type { ProviderId } from '@relavium/llm'; +import { catalogModel, type ProviderId } from '@relavium/llm'; import type { Agent, ReasoningEffort } from '@relavium/shared'; import { CliError } from '../process/errors.js'; @@ -35,17 +35,27 @@ const DEFAULT_CHAT_SYSTEM_PROMPT = export const DEFAULT_CHAT_TOOLS: readonly string[] = ['read_file', 'list_directory', 'git_status']; /** - * Infer the provider that serves a model id from its well-known prefix. Phase-1 has no model→provider - * catalog lookup wired for the chat path, and a default agent must name a provider ({@link Agent.provider}); - * this covers the four `@relavium/llm` seam providers (ADR-0011) and returns `undefined` for an unrecognized - * model so the caller fails with a clear "bind an explicit `--agent`" message rather than guess wrong. + * Infer the provider that serves a model id — **catalog first** (ADR-0071), then a well-known-prefix fallback. + * + * A default agent must name a provider ({@link Agent.provider}). The catalog knows the provider of every model it + * carries AUTHORITATIVELY, regardless of how the id is spelled — `chatgpt-4o-latest` has no `gpt`/`o` prefix + * yet is unmistakably openai — so a catalog hit is trusted before the heuristic. The prefix map remains the fallback + * for an id the catalog does not carry (a brand-new model, or a custom `base_url` endpoint), covering the four + * `@relavium/llm` seam providers (ADR-0011). Returns `undefined` for an id neither knows, so the caller fails with a + * clear "bind an explicit `--agent`" message rather than guess wrong. + * + * The truly robust source is a PERSISTED provider from pick time (the picker knows it authoritatively — the live + * `/models` list is per-provider); {@link buildDefaultChatAgent} prefers that and only falls back to this inference. */ export function inferProviderFromModel(model: string): ProviderId | undefined { + // Normalize ONCE — catalog ids are lowercase, and the prefix fallback lowercases too, so a mixed-case id + // (`Claude-Opus-4-8`) resolves via the CATALOG (authoritative) instead of only via the prefix heuristic. const m = model.toLowerCase(); + const cataloged = catalogModel(m)?.provider; + if (cataloged !== undefined) return cataloged; if (m.startsWith('claude')) return 'anthropic'; // OpenAI's GPT family + the o-series reasoning models. `/^o\d/` matches the whole o-series (o1/o3/o4 and - // future o5+) in one expression rather than enumerating each prefix. (A model→provider catalog lookup is - // the eventual robust source; this prefix map is the deliberate Phase-1 zero-config default.) + // future o5+) in one expression rather than enumerating each prefix. if (m.startsWith('gpt') || /^o\d/.test(m)) return 'openai'; if (m.startsWith('gemini')) return 'gemini'; if (m.startsWith('deepseek')) return 'deepseek'; @@ -54,19 +64,28 @@ export function inferProviderFromModel(model: string): ProviderId | undefined { /** * Build the built-in default chat agent over `model` (the resolved `[chat].default_model`, or - * {@link DEFAULT_CHAT_MODEL}). Throws a clean exit-2 {@link CliError} when the provider cannot be inferred - * from the model id — guiding the user to set a known `[chat].default_model` or bind an explicit `--agent`. - * `reasoningEffort` (the resolved `[chat].reasoning_effort`, ADR-0066) is baked onto the agent so the default - * chat honors the config default; absent ⇒ no reasoning control (the provider default). + * {@link DEFAULT_CHAT_MODEL}). + * + * `knownProvider` — the provider PERSISTED alongside the model (`[chat]`/`[preferences].default_provider`, written by + * the picker/wizard at pick time, ADR-0059's "the provider is authoritative, never re-inferred"). When present it is + * used verbatim and inference is SKIPPED, so a model whose id the prefix map cannot place — a live-discovered + * `chatgpt-4o-latest`, a custom-endpoint id — still starts. Only when it is absent does this fall back to + * {@link inferProviderFromModel}; a clean exit-2 {@link CliError} then guides the user to a known model or `--agent`. + * `reasoningEffort` (the resolved `[chat].reasoning_effort`, ADR-0066) is baked onto the agent so the default chat + * honors the config default; absent ⇒ no reasoning control (the provider default). */ -export function buildDefaultChatAgent(model: string, reasoningEffort?: ReasoningEffort): Agent { - const provider = inferProviderFromModel(model); +export function buildDefaultChatAgent( + model: string, + reasoningEffort?: ReasoningEffort, + knownProvider?: ProviderId, +): Agent { + const provider = knownProvider ?? inferProviderFromModel(model); if (provider === undefined) { throw new CliError( 'invalid_invocation', `cannot infer a provider for chat model '${model}'. Set [chat].default_model to a known model ` + - `(claude-*, gpt-*, gemini-*, deepseek-*), or bind an explicit agent with ` + - `'relavium chat --agent '.`, + `(claude-*, gpt-*, gemini-*, deepseek-*) — or set [chat].default_provider, or bind an explicit agent ` + + `with 'relavium chat --agent '.`, ); } return { diff --git a/apps/cli/src/chat/effort-notice.test.ts b/apps/cli/src/chat/effort-notice.test.ts new file mode 100644 index 00000000..1a849d9d --- /dev/null +++ b/apps/cli/src/chat/effort-notice.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it } from 'vitest'; + +import type { EffortGateResult } from '@relavium/core'; + +import { + capUsd, + effortCappedNote, + effortRejectedNote, + effortUnavailableNote, + effortWithheldNote, + onceEffortNotice, + reasoningWithheldByCapFor, + unpricedModelNote, +} from './effort-notice.js'; + +/** + * The withheld-tier notice channel (ADR-0071 §6) — the sentence, and the promise that it is said EXACTLY once. + * + * The gate turned a loud provider 400 into a quiet no-op. That is the better failure only if it is not quiet, and + * an adversarial review found this channel wired at two of the nine places that build a session or an engine: a + * `/clear`, a `/models` reseat, the bare Home, `agent run`, and `gate` all withheld in perfect silence. + */ +describe('effortRejectedNote — what the model refused, and what it would take', () => { + it('names the refused tier AND the accepted ones, in canonical order', () => { + const note = effortRejectedNote('gpt-5.4-pro', 'low', ['medium', 'high', 'max']); + expect(note).toContain("does not accept reasoning effort 'low'"); + expect(note).toContain('it takes medium, high, max'); + expect(note).toContain('No tier is sent.'); // the consequence, stated — not left for the bill to reveal + }); + + it("re-sorts the gate's own list — its Set puts `off` LAST, the rows must not", () => { + // `EffortGateResult.rejected.accepted` is `[...acceptedTiers(...)]`, and that Set adds `off` after the graded + // tiers (it rides a different axis on three of the four providers). Printed raw it would read "high, off". + expect(effortRejectedNote('claude-opus-4-8', 'max', ['high', 'off', 'low'])).toContain( + 'it takes off, low, high', + ); + }); + + it('degrades to the unavailable note when NOTHING is accepted — never "it takes " with an empty list', () => { + expect(effortRejectedNote('deepseek-reasoner', 'high', [])).toBe( + effortUnavailableNote('deepseek-reasoner'), + ); + }); + + it('SANITIZES the model id — it comes from an authored YAML, which only checks it is non-empty', () => { + // `agent.model` is a bare `nonEmptyString` in the schema: no charset restriction. Two of the sinks write this + // note RAW to a terminal (`relavium run` / `agent run` → stderr; a non-interactive `chat` → stderr), so an + // escape sequence in a crafted workflow's model id would reach the terminal uninterpreted. `models.ts` already + // guards the same class of input for the same reason (CWE-150). + const note = effortRejectedNote('evil\u001b[31m\u0007model\nsecond-line', 'low', ['high']); + expect(note).not.toContain('\u001b'); // no ESC — the CSI that would recolour the user's terminal + expect(note).not.toContain('\u0007'); // no BEL + expect(note).not.toContain('\n'); // and no newline, which would forge a second warning line of its own + // ...and the LEGIBLE content survives (only control bytes are stripped), so it still names the model. + expect(note).toContain('evil'); + expect(note).toContain('second-line'); + }); +}); + +describe('effortUnavailableNote — the two empty lists are NOT the same sentence', () => { + it('a model with no published knob will never have one — say so', () => { + // `deepseek-reasoner` ships `reasoning: {}`: it reasons, and upstream describes no controllable tier. + expect(effortUnavailableNote('deepseek-reasoner')).toContain( + 'publishes no controllable reasoning tier', + ); + }); + + it('a model MISSING from the catalog might just be newer than the snapshot — offer the fix', () => { + // Same empty tier list, a different ACTION. The old id heuristic said "no reasoning control" for both, which + // tells a user running a brand-new model nothing they can do about it. + const note = effortUnavailableNote('some-custom-endpoint-model'); + expect(note).toContain('not in Relavium'); + expect(note).toContain('models refresh'); + }); +}); + +describe('effortCappedNote — the cap, not the model, was the blocker (review M6)', () => { + it('names max_tokens as the fix and reads the on/off label for a budget model', () => { + const note = effortCappedNote('claude-haiku-4-5', 'medium', 500); + expect(note).toContain('max_tokens'); + expect(note).toContain('500'); + expect(note).toContain('claude-haiku-4-5'); + expect(note).toContain('on'); // a budget model's `medium` reads "on" + }); + + it('effortWithheldNote routes a `capped` verdict to it', () => { + const capped: EffortGateResult = { kind: 'capped', requested: 'medium', maxTokens: 256 }; + expect(effortWithheldNote(capped, 'claude-haiku-4-5')).toBe( + effortCappedNote('claude-haiku-4-5', 'medium', 256), + ); + }); + + it('sanitizes a hostile model id (the display boundary, like its sibling notices)', () => { + const note = effortCappedNote('evilmodel', 'high', 100); + expect(note).not.toContain(''); + expect(note).not.toContain(''); + }); +}); + +describe('reasoningWithheldByCapFor — the CLI cap-check the hosts inject (review M6)', () => { + it('true for a budget model under a tight cap, false when roomy', () => { + expect(reasoningWithheldByCapFor('claude-haiku-4-5', 'medium', 500)).toBe(true); + expect(reasoningWithheldByCapFor('claude-haiku-4-5', 'medium', 8000)).toBe(false); + }); + + it('false for `off` (never budgeted) and for an uncatalogued model (no budget to exceed)', () => { + expect(reasoningWithheldByCapFor('claude-haiku-4-5', 'off', 1)).toBe(false); + expect(reasoningWithheldByCapFor('some-custom-endpoint-model', 'medium', 1)).toBe(false); + }); +}); + +describe('onceEffortNotice — a standing condition is not an event', () => { + it('says it ONCE, however many turns consult the gate', () => { + // The gate runs on every turn and every agent-node execution. A stale `off` bound on `gemini-2.5-pro` (which + // cannot disable thinking at all) is withheld on turn one and on turn fifty — without this the transcript grows + // a fresh copy of the same sentence every turn, and a workflow agent inside a `loop` prints it per iteration. + const said: string[] = []; + const sink = onceEffortNotice((note) => said.push(note)); + const note = effortUnavailableNote('gemini-2.5-pro'); + sink(note); + sink(note); + sink(note); + expect(said).toEqual([note]); + }); + + it('…but a DIFFERENT condition still speaks up — it is a de-dup, not a mute', () => { + // A `/models` reseat binds a new model; the tier that was fine a moment ago may not be. Suppressing that would + // reintroduce the silence this whole channel exists to remove. + const said: string[] = []; + const sink = onceEffortNotice((note) => said.push(note)); + sink(effortRejectedNote('gpt-5.4-pro', 'low', ['medium', 'high'])); + sink(effortRejectedNote('gpt-5.4-pro', 'low', ['medium', 'high'])); // same → suppressed + sink(effortRejectedNote('gpt-5-pro', 'low', ['high'])); // different model → spoken + sink(effortRejectedNote('gpt-5.4-pro', 'off', ['medium', 'high'])); // different tier → spoken + expect(said).toHaveLength(3); + }); +}); + +describe('capUsd + unpricedModelNote — the cost-cap notice (ADR-0071 §K7)', () => { + it('formats a cap as fixed-decimal USD, never scientific notation', () => { + expect(capUsd(5_000_000)).toBe('$0.05'); // the workflow-yaml-spec's own example cap + expect(capUsd(1_000_000_000)).toBe('$10.00'); + expect(capUsd(1)).toBe('$0.00'); // a sub-cent cap rounds to $0.00 — NOT `$1e-8`, the bug this fixed + }); + + it('names the model, the cap, the fix, AND the surface-specific strict setting', () => { + const chat = unpricedModelNote('my-local-model', 5_000_000, '[chat] strict_cost_cap'); + expect(chat).toContain('my-local-model'); + expect(chat).toContain('$0.05'); + expect(chat).toContain('models pricing my-local-model'); + expect(chat).toContain('[chat] strict_cost_cap'); // a chat user edits config.toml… + + const workflow = unpricedModelNote('my-local-model', 5_000_000, 'budget.strict_cost_cap'); + expect(workflow).toContain('budget.strict_cost_cap'); // …a workflow user edits the YAML + }); + + it('SANITIZES a hostile model id (the display boundary, like every sibling effort notice)', () => { + const note = unpricedModelNote('evilid\nsecond', 5_000_000); + expect(note).not.toContain(''); // no raw ESC + expect(note).not.toContain(''); // no BEL + expect(note).not.toContain('\n'); // collapsed to one line + }); +}); diff --git a/apps/cli/src/chat/effort-notice.ts b/apps/cli/src/chat/effort-notice.ts new file mode 100644 index 00000000..0e9a9950 --- /dev/null +++ b/apps/cli/src/chat/effort-notice.ts @@ -0,0 +1,254 @@ +import type { EffortGateResult } from '@relavium/core'; +import { + catalogModel, + effortTiersFor as seamEffortTiersFor, + reasoningControlShape, + reasoningWithheldByCap, + wireValueFor, + CANONICAL_ON_TIER, +} from '@relavium/llm'; +import { EFFORT_TIER_HINT, REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; + +import { sanitizeInline } from '../render/sanitize.js'; + +/** + * What we TELL the user when a reasoning tier is withheld + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * The gate replaced a loud provider 400 with a quiet no-op — the turn runs, the field is dropped, and the bill + * lands at the provider's default tier. That is the better failure ONLY if it is said out loud; a rejection the + * user cannot see is worse than the error it replaced. Every surface that can withhold (the `/effort` command, the + * session seed, the workflow runner, the in-Home chat) says it with the sentences below, so the wording cannot + * drift between them. + * + * It lives here rather than beside the picker because the picker is one CONSUMER of it: the engine host + * (`build-engine.ts`) and the session host both need it too, and neither should be reaching into `render/tui/`. + */ + +/** The model id, safe to write to a terminal — it comes from an authored YAML and is only `nonEmptyString` there. */ +function safeModel(model: string): string { + return sanitizeInline(model); +} + +/** + * Why this model has no effort control at all — the sentence a surface shows instead of a dead overlay. + * + * The two causes need different words because they need different ACTIONS. A model with no catalog row may simply + * be newer than our snapshot, and a refresh could give it its control back; a model that publishes no knob will + * never have one, however often we refresh. "No reasoning control" for both — which is all the old id heuristic + * could say — tells the user nothing they can act on. + */ +export function effortUnavailableNote(model: string): string { + return catalogModel(model) === undefined + ? `${safeModel(model)} is not in Relavium's model catalog, so its reasoning control is unknown — no tier is sent. Run \`relavium models refresh\` if the model is newer than the catalog.` + : `${safeModel(model)} publishes no controllable reasoning tier — a tier would be ignored.`; +} + +/** + * Why THIS tier is not available, and which ones are — the sentence for a tier the model rejects. + * + * `accepted` is what the engine's gate already computed (`{kind:'rejected', requested, accepted}`), re-sorted into + * canonical tier order: the gate builds its array by spreading a `Set` whose insertion order puts `off` last (it + * rides a different axis), and the rows must read `off → low → medium → high → max` wherever they appear. + */ +export function effortRejectedNote( + model: string, + requested: ReasoningEffort, + accepted: Iterable, +): string { + const offered = new Set(accepted); + const list = REASONING_EFFORTS.filter((tier) => offered.has(tier)); + return list.length === 0 + ? effortUnavailableNote(model) + : `${safeModel(model)} does not accept reasoning effort '${requested}' — it takes ${list.join(', ')}. No tier is sent.`; +} + +/** + * The reasoning-effort ROWS a picker should offer for a model, in canonical order — the PRESENTATION projection of + * `@relavium/llm`'s wire-accurate {@link acceptedTiers}, deduped so no two rows produce the SAME provider behavior + * (ADR-0066 amendment). By the model's control SHAPE ({@link reasoningControlShape}), never per-model: + * + * - **graded** (a real effort ladder): the accepted tiers, DEDUPED by distinct wire value. DeepSeek's + * low/medium/high all send `high`, so they collapse to ONE row (`off/high/max`); Gemini's `max` coarsens onto + * `high` (`low/medium/high`). A model whose rungs are all distinct (`claude-opus-4-8`, `gpt-5.4-pro`) is + * unchanged. The representative kept per wire is the tier whose own NAME matches the wire (so DeepSeek's + * `high`-wire row reads "high", not "low" or "max"), else the highest tier for that wire. + * - **budget** (a continuous token budget, no ladder — `claude-haiku-4-5`): a two-row **off/on**, "on" = + * {@link CANONICAL_ON_TIER}. A budget has no meaningful discrete rungs (Claude-Code parity). Only a GENUINE + * two-state choice opens the overlay: a model that cannot be turned off (`gemini-2.5-pro`) has nothing to + * toggle, so the list is empty. + * - **none** (`deepseek-reasoner`'s `{}`, a custom/uncatalogued model, or a model with no usable knob): empty. + * + * Every emitted tier is a member of the seam's accepted set, so the accept sends a value the engine gate accepts + * verbatim. Empty for all three "nothing to offer" cases; the surfaces distinguish WHY in what they SAY (see + * {@link effortUnavailableNote}). The seam ({@link acceptedTiers}) is unchanged — this is presentation only. + */ +export function effortTiersFor(model: string): readonly ReasoningEffort[] { + const entry = catalogModel(model); + const accepted = seamEffortTiersFor(model); + const shape = reasoningControlShape(entry?.reasoning); + if (shape === 'none') return []; + if (shape === 'budget') { + // A continuous budget → off/on. BOTH states must be reachable for a real choice; else nothing to pick. + return accepted.has('off') && accepted.has(CANONICAL_ON_TIER) ? ['off', CANONICAL_ON_TIER] : []; + } + // graded: keep one representative tier per distinct WIRE value (see the doc for how the representative is chosen). + const provider = entry?.provider ?? 'openai'; + const repByWire = new Map(); + for (const tier of REASONING_EFFORTS) { + if (tier === 'off' || !accepted.has(tier)) continue; + const wire = wireValueFor(provider, tier); + if (wire === undefined) continue; + const cur = repByWire.get(wire); + // Ascending order, so a later tier is HIGHER. Overwrite unless the current rep is already the name-match for + // this wire (then keep it — the name-match reads truest, e.g. Gemini's `high` over its coarsened `max`). + if (cur === undefined || (cur as string) !== wire) repByWire.set(wire, tier); + } + const reps = new Set(repByWire.values()); + return REASONING_EFFORTS.filter((t) => (t === 'off' ? accepted.has('off') : reps.has(t))); +} + +/** + * The display label + hint for one effort ROW (or the bound tier in the footer/notice) — so a budget model's + * canonical-on tier reads "on" everywhere, while every graded tier reads its own name. Keeps the picker, the footer + * and the notices from disagreeing about what a tier is called. + */ +export function effortRowLabel( + model: string, + tier: ReasoningEffort, +): { label: string; hint: string } { + if ( + tier === CANONICAL_ON_TIER && + reasoningControlShape(catalogModel(model)?.reasoning) === 'budget' + ) { + return { label: 'on', hint: 'reasoning on' }; + } + return { label: tier, hint: EFFORT_TIER_HINT[tier] }; +} + +/** + * Project an arbitrary tier onto the SURVIVING picker row that represents it (ADR-0066 amendment) — so an + * opening-highlight or ✓ lands on a real row even when that tier was deduped away or collapsed to on/off. Without + * it, `rows.indexOf(a-collapsed-tier)` is `-1` and a fresh `/models` accept on the neutral highlight silently wrote + * `off` for a graded-collapsed model (`effortTiersFor('deepseek-v4-pro')` = ['off','high','max'], and the neutral + * `medium` is not a row). A budget model folds any non-off tier onto the canonical-on row; a graded model folds a + * tier onto the surviving row with the SAME wire value. `undefined` ⇒ nothing represents it (empty list, or `off` + * on a can't-disable model) — the caller clamps to the first row. + */ +export function projectEffortToRow( + model: string, + rows: readonly ReasoningEffort[], + tier: ReasoningEffort, +): ReasoningEffort | undefined { + if (rows.includes(tier)) return tier; // already a surviving row + if (tier === 'off') return undefined; // its own axis — if `off` is not a row, nothing represents it + const entry = catalogModel(model); + if (reasoningControlShape(entry?.reasoning) === 'budget') { + return rows.includes(CANONICAL_ON_TIER) ? CANONICAL_ON_TIER : undefined; + } + const provider = entry?.provider ?? 'openai'; + const wire = wireValueFor(provider, tier); + return rows.find( + (r): r is Exclude => r !== 'off' && wireValueFor(provider, r) === wire, + ); +} + +/** + * The single note for a WITHHELD reasoning tier (ADR-0071 §6) — the one place the engine host (`build-engine.ts`) + * and the session host map an {@link EffortGateResult} to words, so neither carries its own inline ternary and the + * wording cannot drift between them. The core invokes the `onEffortWithheld` sink for `rejected`, `uncontrollable`, + * and `capped` (agent-runner / agent-session), so those three are the outcomes with a sentence; the exhaustive + * default guards a future gate kind against silently taking one of the existing sentences. + */ +export function effortWithheldNote(result: EffortGateResult, model: string): string { + switch (result.kind) { + case 'rejected': + return effortRejectedNote(model, result.requested, result.accepted); + case 'uncontrollable': + return effortUnavailableNote(model); + case 'capped': + return effortCappedNote(model, result.requested, result.maxTokens); + default: + throw new Error(`effortWithheldNote: '${result.kind}' is not a withheld outcome`); + } +} + +/** + * The note for a tier the model ACCEPTS but the request's `max_tokens` withholds (review M6). A budget-shaped model + * (`claude-haiku-4-5`) needs room for its minimum thinking budget under the output cap; a tight `max_tokens` leaves + * none, so the adapter drops thinking. The blocker is the CAP — so the sentence names `max_tokens`, not the tier. + */ +export function effortCappedNote( + model: string, + requested: ReasoningEffort, + maxTokens: number, +): string { + const { label } = effortRowLabel(model, requested); + return `reasoning ${label} needs a larger max_tokens than ${maxTokens} on ${sanitizeInline(model)} — it was withheld this turn.`; +} + +/** The {@link import('@relavium/core').ReasoningCapCheck} for the CLI hosts: reads the catalog for the bound model's + * budget range and asks whether the adapter would drop thinking under this `max_tokens` (review M6). `off` is never + * budgeted, and an uncatalogued model has no budget to exceed — both are `false` (nothing is cap-withheld). */ +export function reasoningWithheldByCapFor( + model: string, + tier: ReasoningEffort, + maxTokens: number, +): boolean { + if (tier === 'off') return false; + const entry = catalogModel(model); + if (entry?.reasoning === undefined) return false; + return reasoningWithheldByCap(entry.provider, entry.reasoning, tier, maxTokens); +} + +/** + * Wrap a notice sink so a PERSISTENT condition is reported once, not once per turn. + * + * The gate is consulted on every turn and every agent-node execution, and a withheld tier is not a transient + * event — a stale `off` bound on `gemini-2.5-pro` (which cannot disable thinking at all) is withheld on turn one + * and on turn fifty. Without this the transcript grows a fresh copy of the same sentence every turn, and a + * workflow agent inside a `loop` prints the same stderr warning on every iteration. `BudgetGovernor`'s warning + * has the same shape and is threshold-gated for the same reason. + * + * Keyed on the NOTE, so a genuinely new condition — a different tier, a different model after a reseat — still + * speaks up. The set lives for the life of the sink, which is the life of the session or run. + */ +export function onceEffortNotice(sink: (note: string) => void): (note: string) => void { + const said = new Set(); + return (note: string): void => { + if (said.has(note)) return; + said.add(note); + sink(note); + }; +} + +/** + * Integer micro-cents → a plain USD string (1 USD = 1e8 micro-cents), fixed-decimal so a tiny cap reads `$0.00` + * rather than `$1e-8`. Two decimals is the granularity a cost cap is ever set at; a sub-cent cap rounds to `$0.00`, + * which is honest — it is effectively "block everything". + */ +export function capUsd(microcents: number): string { + return `$${(microcents / 100_000_000).toFixed(2)}`; +} + +/** + * What we tell a user when a turn ran on a model we could not PRICE (ADR-0071 §K7). + * + * A cost cap on an unpriced model is a hole: the governor cannot estimate a turn's cost, so it degrades to `allow` + * — the right trade for a self-hosted model with ~no metered cost, but a false sense of safety if said in silence. + * One sentence, everywhere, so the chat transcript, `relavium run`, `agent run`, and the resumed `gate` all say it + * the same way. `strict_cost_cap` is named as the block-instead escape hatch, and `models pricing` as the fix. + */ +export function unpricedModelNote( + model: string, + capMicrocents: number, + // The exact key a user edits to turn strict on, which DIFFERS by surface: a chat user sets `[chat] strict_cost_cap` + // in config.toml, a workflow user sets `budget.strict_cost_cap` in the YAML. A generic "strict_cost_cap" makes each + // guess which file. Default to the bare name for a caller that has no better spelling. + strictSetting = 'strict_cost_cap', +): string { + // Sanitize the provider-controlled model id at this display boundary, exactly as the sibling effort notices do — + // a crafted id must not smuggle a terminal escape into the transcript line (it appears twice, incl. a command). + const safeModel = sanitizeInline(model); + return `${safeModel} has no price, so the cost cap (${capUsd(capMicrocents)}) does not apply to it. Price it with \`relavium models pricing ${safeModel}\`, or set ${strictSetting} to refuse an unpriced model.`; +} diff --git a/apps/cli/src/chat/persister.test.ts b/apps/cli/src/chat/persister.test.ts index 4d65ed74..3755dc27 100644 --- a/apps/cli/src/chat/persister.test.ts +++ b/apps/cli/src/chat/persister.test.ts @@ -21,6 +21,7 @@ import type { ProviderResolver } from '../engine/providers.js'; const EMPTY_CHAT: ResolvedChatConfig = { defaultModel: undefined, + defaultProvider: undefined, fsScope: undefined, maxTurns: undefined, maxMessages: undefined, @@ -28,6 +29,7 @@ const EMPTY_CHAT: ResolvedChatConfig = { compactThreshold: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, allowedCommands: undefined, allowedCommandGlobs: undefined, reasoningEffort: undefined, diff --git a/apps/cli/src/chat/session-host.test.ts b/apps/cli/src/chat/session-host.test.ts index 3183571b..d6649666 100644 --- a/apps/cli/src/chat/session-host.test.ts +++ b/apps/cli/src/chat/session-host.test.ts @@ -82,6 +82,7 @@ function capturingResolver(scripts: StreamChunk[][]): { const EMPTY_CHAT: ResolvedChatConfig = { defaultModel: undefined, + defaultProvider: undefined, fsScope: undefined, maxTurns: undefined, maxMessages: undefined, @@ -89,6 +90,7 @@ const EMPTY_CHAT: ResolvedChatConfig = { compactThreshold: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, allowedCommands: undefined, allowedCommandGlobs: undefined, reasoningEffort: undefined, @@ -124,6 +126,18 @@ describe('buildChatSession', () => { expect(built.context.fsScopeTier).toBe('sandboxed'); // default when [chat].fs_scope is unset }); + it('carries [chat].default_provider onto the default agent — an unplaceable id BUILDS (ADR-0059, Bug-3 hop)', async () => { + // The session-host wiring hop: buildChatSession threads chat.defaultProvider into resolveChatAgent, so a default + // agent over a live-discovered id the prefix cannot place (`chat-latest`) builds instead of throwing "cannot + // infer a provider". Drop the spread at session-host.ts:326-328 and this fails; the agent-source test alone + // (which calls resolveChatAgent directly) would not catch that. + const built = await build({ + chat: { ...EMPTY_CHAT, defaultModel: 'chat-latest', defaultProvider: 'openai' }, + }); + expect(built.agent.model).toBe('chat-latest'); + expect(built.agent.provider).toBe('openai'); + }); + it('honors [chat].fs_scope on the SessionContext', async () => { const built = await build({ chat: { ...EMPTY_CHAT, fsScope: 'project' } }); expect(built.context.fsScopeTier).toBe('project'); @@ -837,6 +851,40 @@ describe('buildGovernorWiring', () => { expect(typeof wiring?.updateCost).toBe('function'); }); + it('an UNPRICED model degrades to allow AND forwards the note to onUnpriced (ADR-0071 §K7)', async () => { + const notes: string[] = []; + const wiring = buildGovernorWiring( + { ...EMPTY_CHAT, maxCostMicrocents: 1_000_000, onExceed: 'fail' }, + undefined, + undefined, + undefined, + (note) => notes.push(note), + ); + // A model neither the catalog nor a user prices — the pre-egress estimate throws, and the governor degrades to + // allow. It must not reject (an unpriced self-hosted model is not a failure) but it must SAY so, once. + await expect( + wiring?.preEgress({ model: 'my-self-hosted-model', maxTokens: 1000 }), + ).resolves.toBeUndefined(); + await expect( + wiring?.preEgress({ model: 'my-self-hosted-model', maxTokens: 1000 }), + ).resolves.toBeUndefined(); + expect(notes).toHaveLength(1); // deduped per model + expect(notes[0]).toContain('my-self-hosted-model'); + expect(notes[0]).toContain('has no price'); + }); + + it('strict_cost_cap BLOCKS an unpriced model instead of degrading (ADR-0071 §K7)', async () => { + const wiring = buildGovernorWiring({ + ...EMPTY_CHAT, + maxCostMicrocents: 1_000_000, + onExceed: 'warn', // strict overrides on_exceed — a hard fail regardless + strictCostCap: true, + }); + await expect( + wiring?.preEgress({ model: 'my-self-hosted-model', maxTokens: 1000 }), + ).rejects.toBeInstanceOf(BudgetExceededError); + }); + it('on_exceed:fail — preEgress rejects with BudgetExceededError once the cap is exceeded', async () => { const wiring = buildGovernorWiring({ ...EMPTY_CHAT, maxCostMicrocents: 1, onExceed: 'fail' }); wiring?.updateCost(999_999); // cumulative now far past the 1-microcent cap diff --git a/apps/cli/src/chat/session-host.ts b/apps/cli/src/chat/session-host.ts index 9348b148..0b52bf88 100644 --- a/apps/cli/src/chat/session-host.ts +++ b/apps/cli/src/chat/session-host.ts @@ -12,11 +12,17 @@ import { type SessionDeps, type SessionEventSink, type SessionHandle, + type EffortGateResult, type SessionResumeState, type ToolDef, type ToolHost, } from '@relavium/core'; -import { modelSupportsReasoning, type PricingOverlay, type ProviderId } from '@relavium/llm'; +import { + effortTiersFor, + type EndpointKind, + type PricingOverlay, + type ProviderId, +} from '@relavium/llm'; import type { ManagerSkippedTool, McpClient, McpServerConfig } from '@relavium/mcp'; import type { AgentSessionRecord, @@ -34,6 +40,11 @@ import { createProviderResolver, type ProviderResolver } from '../engine/provide import { assembleToolEnv, clampChatTier, wiredToolIds } from '../engine/tool-host/assemble.js'; import { CliError } from '../process/errors.js'; import type { McpSecretResolver } from '../secrets/mcp-secret.js'; +import { + effortWithheldNote, + reasoningWithheldByCapFor, + unpricedModelNote, +} from './effort-notice.js'; import { resolveChatAgent } from './agent-source.js'; /** @@ -105,11 +116,23 @@ export interface BuildChatSessionOptions { * one-line notice. Absent ⇒ a no-op (the warn stays non-blocking either way). */ readonly onBudgetWarning?: (warning: ChatBudgetWarning) => void; + /** + * Sink for a turn on an UNPRICED model (ADR-0071 §K7) — the cost cap could not apply to it. Same channel shape as + * {@link BuildChatSessionOptions.onBudgetWarning}; the governor already dedups per-model, so this fires once each. + */ + readonly onUnpriced?: (note: string) => void; + /** + * Sink for a WITHHELD reasoning tier (ADR-0071 §6) — the bound model does not accept the effective tier, so the + * field is not sent. Same channel shape as {@link BuildChatSessionOptions.onBudgetWarning}: a session has no + * event for it, so the surface is told directly and puts the sentence in the transcript's notice channel. + * Absent ⇒ a no-op, and the tier is still withheld (never guessed at). + */ + readonly onEffortWithheld?: (note: string) => void; /** * The ADR-0065 §2 user-pricing overlay (2.5.G S10) — a `ReadonlyMap` the command projects * from the `model_catalog` `source='user'` rows (via `buildUserPricing`). It flows into BOTH the pre-egress * governor (so a user-priced model is enforced by `[chat].max_cost_microcents`) AND `SessionDeps.resolvePrice` - * (so the realized cost of the same model is tracked). Static `MODEL_PRICING` still wins for a known id. Absent ⇒ + * (so the realized cost of the same model is tracked). The USER outranks the catalog (ADR-0071 §1). Absent ⇒ * unknown models degrade cost governance to `allow` loudly, unchanged. */ readonly resolvePrice?: PricingOverlay; @@ -166,7 +189,14 @@ const DEFAULT_FS_SCOPE = 'sandboxed' as const; /** The fields {@link buildSessionRuntime} reads — the platform-capability inputs shared by a fresh + resumed session. */ type SessionRuntimeOptions = Pick< BuildChatSessionOptions, - 'chat' | 'now' | 'providers' | 'toolHost' | 'onBudgetWarning' | 'resolvePrice' + | 'chat' + | 'now' + | 'providers' + | 'toolHost' + | 'onBudgetWarning' + | 'onUnpriced' + | 'onEffortWithheld' + | 'resolvePrice' >; /** @@ -218,7 +248,13 @@ function buildSessionRuntime( ? {} : { allowedCommandGlobs: opts.chat.allowedCommandGlobs }), }; - const governor = buildGovernorWiring(opts.chat, opts.onBudgetWarning, opts.resolvePrice); + const governor = buildGovernorWiring( + opts.chat, + opts.onBudgetWarning, + opts.resolvePrice, + providers.endpointKind, + opts.onUnpriced, + ); // The session event sink (1.W): a draft → bus → stamped sequenceNumber/timestamp. Hoisted so a SURFACE // event (the in-REPL `/export`'s `session:exported`, 2.Q) can ride the same monotonic per-session counter. const emit = createSessionEventSink(bus, sessionId); @@ -226,9 +262,24 @@ function buildSessionRuntime( const deps: SessionDeps = { resolveProvider: providers.resolveProvider, keyFor: providers.keyFor, - // ADR-0066: the per-model reasoning capability (static registry projection) — gates whether the authored - // reasoning_effort tier is sent (a non-reasoning / custom model returns false, so the field is withheld). - resolveReasoning: modelSupportsReasoning, + // ADR-0071 §6: the host projects WHICH TIERS the model accepts, not merely whether it reasons. `gpt-5.4-pro` + // reasons and rejects `low`; the boolean this replaced said `true` and let that straight through to a 400. + // The seam's `effortTiersFor` IS the projection — passed by reference, not re-derived, so this host cannot + // drift from the picker that renders the same answer. + resolveEffortTiers: effortTiersFor, + // A budget-shaped model can accept a tier yet still have the adapter drop thinking when this turn's `max_tokens` + // leaves no room for its budget floor (review M6). Inject the catalog-backed check so the gate SURFACES that as + // a `capped` verdict instead of the adapter dropping it in silence. + withheldByCap: reasoningWithheldByCapFor, + // …and when the gate withholds, SAY SO. The engine cannot print; it hands back the verdict (which carries the + // tiers the model would take) and the surface turns it into the one sentence every path uses. + ...(opts.onEffortWithheld === undefined + ? {} + : { + onEffortWithheld: (result: EffortGateResult, model: string) => { + opts.onEffortWithheld?.(effortWithheldNote(result, model)); + }, + }), registry, tools, sleep: (ms) => new Promise((resolveSleep) => setTimeout(resolveSleep, ms)), @@ -277,6 +328,11 @@ export async function buildChatSession(opts: BuildChatSessionOptions): Promise void; + /** Unpriced-model notice (ADR-0071 §K7) — see {@link BuildChatSessionOptions.onUnpriced}. */ + readonly onUnpriced?: (note: string) => void; /** The loaded session record (its frozen `agentSnapshot` + `context` rebind the session). */ readonly record: AgentSessionRecord; /** The session's persisted transcript, in any order ({@link reconstructSessionState} sorts it). */ @@ -534,18 +598,36 @@ export function buildGovernorWiring( chat: ResolvedChatConfig, onWarning?: (warning: ChatBudgetWarning) => void, resolvePrice?: PricingOverlay, + endpointKind?: (id: ProviderId) => EndpointKind, + onUnpriced?: (note: string) => void, ): GovernorWiring | undefined { const cap = chat.maxCostMicrocents; if (cap === undefined || cap <= 0) return undefined; const budget: Budget = { max_cost_microcents: cap, on_exceed: chat.onExceed ?? 'pause_for_approval', + // ADR-0071 §K7: refuse a turn on a model we cannot price, instead of the silent degrade-to-allow. Default off. + ...(chat.strictCostCap ? { strict_cost_cap: true } : {}), }; const governor = new BudgetGovernor({ budget, // The ADR-0065 §2 user-pricing overlay — so the PRE-EGRESS estimate can price a user-priced (otherwise // unknown) model and enforce the cost cap on it. Omit ⇒ an unknown model degrades to `allow` loudly. ...(resolvePrice === undefined ? {} : { resolvePrice }), + // ADR-0071 §7: the adapter clamps an authored `max_tokens` to the model's ceiling on an OFFICIAL endpoint and + // not on a custom one. The estimate must make the same call — assume official on a gateway and it lands BELOW + // what the wire can spend, so the governor under-authorizes and waves through the call it exists to stop. + // Keyed on the ROUTING provider the governor threads per attempt, not the model's catalog provider — a custom + // gateway serving another provider's model id would otherwise be mis-read as official and under-clamped (M2). + ...(endpointKind === undefined ? {} : { resolveEndpoint: endpointKind }), + // ADR-0071 §K7: a turn ran on a model we could not price, so the cap did not apply to it. Say so, once — a cost + // cap that silently does not apply is a false sense of safety. `strict_cost_cap` is the block-instead option. + ...(onUnpriced === undefined + ? {} + : { + onUnpriced: (model: string, capMicrocents: number) => + onUnpriced(unpricedModelNote(model, capMicrocents, '[chat] strict_cost_cap')), + }), emit: (event) => { // `warn` is non-blocking BY CONTRACT. A misbehaving warn surface must never reject this emit — a // rejection would propagate as an `internal` turn error and break sendMessage — so swallow a sync throw. @@ -563,7 +645,7 @@ export function buildGovernorWiring( }); return { preEgress: (info) => - governor.checkPreEgress(info.model, info.maxTokens, info.mediaUnitsEstimate), + governor.checkPreEgress(info.model, info.maxTokens, info.mediaUnitsEstimate, info.provider), updateCost: (cumulative) => governor.updateCost(cumulative), }; } diff --git a/apps/cli/src/commands/agent-run.ts b/apps/cli/src/commands/agent-run.ts index a212c726..8e2b50ba 100644 --- a/apps/cli/src/commands/agent-run.ts +++ b/apps/cli/src/commands/agent-run.ts @@ -6,6 +6,7 @@ import type { SessionStreamHandleEvent } from '@relavium/core'; import { nonInteractiveApprovalPrompt } from '../chat/chat-mode.js'; import { applyChatMode, makeChatModeEnv } from '../chat/chat-mode-host.js'; import { cassetteResolver, loadCassette } from '../chat/fixture.js'; +import { onceEffortNotice } from '../chat/effort-notice.js'; import { buildChatSession, type BuiltChatSession } from '../chat/session-host.js'; import { loadResolvedConfig } from '../config/load.js'; import { surfaceMcpSkipped } from '../engine/mcp-servers.js'; @@ -82,6 +83,11 @@ export async function agentRunCommand( // FULLY offline: no `[[mcp_servers]]` registrations and an env-only secret resolver (never the keychain). const built = await (deps.buildSession ?? buildChatSession)({ chat: config.chat, + // ADR-0071 §6: a one-shot invoke has no transcript and no picker, so a withheld tier would otherwise vanish + // completely — the turn runs, the authored knob does nothing, and the bill arrives at the provider's default. + // STDERR, never stdout: `--json` owns stdout, and a warning line mid-stream is a parse error downstream. + onEffortWithheld: onceEffortNotice((note) => deps.io.writeErr(`warning: ${note}\n`)), + onUnpriced: (note) => deps.io.writeErr(`warning: ${note}\n`), agentRef: args.agent, cwd: deps.global.cwd, projectConfigDir, diff --git a/apps/cli/src/commands/chat.test.ts b/apps/cli/src/commands/chat.test.ts index c974f397..8bc4ec71 100644 --- a/apps/cli/src/commands/chat.test.ts +++ b/apps/cli/src/commands/chat.test.ts @@ -53,6 +53,7 @@ import { const EMPTY_CHAT: ResolvedChatConfig = { defaultModel: undefined, + defaultProvider: undefined, fsScope: undefined, maxTurns: undefined, maxMessages: undefined, @@ -60,6 +61,7 @@ const EMPTY_CHAT: ResolvedChatConfig = { compactThreshold: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, allowedCommands: undefined, allowedCommandGlobs: undefined, reasoningEffort: undefined, diff --git a/apps/cli/src/commands/chat.ts b/apps/cli/src/commands/chat.ts index 6740e667..c330bd7c 100644 --- a/apps/cli/src/commands/chat.ts +++ b/apps/cli/src/commands/chat.ts @@ -8,13 +8,8 @@ import { type SessionStreamHandleEvent, type UserCommandOutcome, } from '@relavium/core'; -import { modelSupportsReasoning, type ProviderId } from '@relavium/llm'; -import { - EFFORT_TIER_HINT, - REASONING_EFFORTS, - type AgentSessionRecord, - type ReasoningEffort, -} from '@relavium/shared'; +import type { ProviderId } from '@relavium/llm'; +import { REASONING_EFFORTS, type AgentSessionRecord, type ReasoningEffort } from '@relavium/shared'; import { exportSession } from '../chat/export.js'; import { formatDoctorReport, runDoctorChecks, type DoctorProbes } from '../chat/doctor.js'; import { assembleDoctorProbes } from '../chat/doctor-host.js'; @@ -44,6 +39,14 @@ import { type ChatMode, } from '../chat/chat-mode.js'; import { applyChatMode, makeChatModeEnv } from '../chat/chat-mode-host.js'; +import { + effortRejectedNote, + effortRowLabel, + effortTiersFor, + effortUnavailableNote, + onceEffortNotice, + projectEffortToRow, +} from '../chat/effort-notice.js'; import { createSessionPersister, makeCatalogIdResolver, @@ -523,6 +526,11 @@ export async function chatCommand(args: ChatCommandArgs, deps: ChatCommandDeps): mcpRegistrations: config.mcpServers, ...(resolvePrice === undefined ? {} : { resolvePrice }), onBudgetWarning: (warning) => emitLiveNotice(deps.io, budgetWarningText(warning)), + // ADR-0071 §6: a tier the bound model does not take is WITHHELD at send — and said out loud. Without this the + // turn runs, the field is gone, and the user is billed at the provider's default tier with nothing to explain + // why the knob they set did nothing. + onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(deps.io, note)), + onUnpriced: (note) => emitLiveNotice(deps.io, note), }); // The session now OWNS the live MCP connections (built.closeMcp). `runReplLoop`'s finally is the steady-state // teardown, but the build→loop window (opening history.db can throw) runs first — guard it so a pre-loop fault @@ -682,6 +690,11 @@ export async function chatResumeCommand( mcpRegistrations: config.mcpServers, resolvePrice, onBudgetWarning: (warning) => emitLiveNotice(deps.io, budgetWarningText(warning)), + // ADR-0071 §6: a tier the bound model does not take is WITHHELD at send — and said out loud. Without this the + // turn runs, the field is gone, and the user is billed at the provider's default tier with nothing to explain + // why the knob they set did nothing. + onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(deps.io, note)), + onUnpriced: (note) => emitLiveNotice(deps.io, note), }); closeMcp = resumed.closeMcp; surfaceMcpSkipped(deps.io, resumed.mcpSkipped); @@ -894,11 +907,27 @@ export function createChatModeControl( prompt, }); applyChatMode(modeEnv, store.getSnapshot().mode); - // ADR-0066: seed the footer's effort indicator from the session's initial effective tier (override ?? agent), - // shown only when the bound model is reasoning-capable (a non-reasoning model has no controllable tier, so its - // baked config default is not surfaced). Mirrors applyChatMode's initial-mode seed above. - const capable = modelSupportsReasoning(built.agent.model); - store.setReasoningEffort(capable ? built.session.reasoningEffort : undefined); + // ADR-0066/0071: seed the footer's effort indicator from the session's initial effective tier (override ?? + // agent), shown only when the bound model actually has a tier to control. Mirrors applyChatMode's seed above. + // + // The predicate is `effortTiersFor` — the SAME one the picker, the `/effort` command and the engine's gate ask. + // It used to be `modelSupportsReasoning`, an id heuristic over the hand-typed pricing table, and the two + // disagreed on sixteen shipped models: on `claude-sonnet-4-5` the picker offered five tiers and the engine sent + // the chosen one, while this line decided the model was not capable and suppressed the footer — so the user was + // billed for extended thinking with no indicator that it was on, and `/effort` could never show a ✓ on the tier + // it had itself just bound. + const tiers = effortTiersFor(built.agent.model); + const seeded = built.session.reasoningEffort; + const seedAccepted = seeded !== undefined && tiers.includes(seeded); + store.setReasoningEffort(seedAccepted ? seeded : undefined); + if (seeded !== undefined && !seedAccepted) { + // A tier the CURRENT model will not take — and the config keeps a `reasoning_effort` across a model-only + // default write, so this is the ordinary way to arrive here: set `off` on a model that could disable, switch + // the default model, and the tier stays behind. It is inert now, and being inert SILENTLY is the expensive + // half: a user who deliberately turned reasoning off on `gemini-2.5-pro` — which cannot disable thinking at + // all — would otherwise see `effort: off` in the footer while Google thought, and billed, on every turn. + store.notice(effortRejectedNote(built.agent.model, seeded, tiers)); + } return { onAbort: () => { built.session.abort(); // void-returning: block body so it never forwards abort()'s return value @@ -913,7 +942,11 @@ export function createChatModeControl( // ReseatTarget); so on a non-reasoning model the tier is stored but inert until the very next same-model turn. onSetEffort: (effort) => { built.session.setReasoningEffort(effort); - store.setReasoningEffort(capable ? effort : undefined); + // The footer shows the tier ONLY if the model will actually take it. Every caller of this setter already + // filters — the overlay lists only accepted tiers, `/effort ` refuses the rest — so this is the + // belt to their braces, and it is the line that used to lie: it asked a boolean that said `true` for + // `claude-sonnet-4-5` and `false` for models the picker was happily binding tiers on. + store.setReasoningEffort(effort !== undefined && tiers.includes(effort) ? effort : undefined); }, }; } @@ -1167,32 +1200,59 @@ export function createChatLineHandler( // non-reasoning model the tier is stored but gated off at send, so the note says it will be ignored. setReasoningEffort: (effortArg) => { const requested = effortArg.trim(); - const capable = modelSupportsReasoning(built.agent.model); + // The tiers THIS model takes — the same list the overlay renders, so the typed form and the interactive form + // cannot disagree. They did: the overlay was migrated to the catalog while `/effort ` kept validating + // against the fixed five, and on twenty-four shipped models it would happily accept a tier the engine then + // silently dropped. `/effort low` on `gpt-5.4-pro` (ladder: medium, high, xhigh) said "applies to your next + // message", showed `low` in the footer, and sent nothing — the user paid for the provider's default. + const offered = effortTiersFor(built.agent.model); if (requested.length === 0) { - // Bare `/effort`: show the current tier + EXPLAIN each one (a discovery affordance — the palette submits - // this bare form), marking the active one. On a non-reasoning model, say so plainly instead of a tier. + // Bare `/effort`: show the current tier + EXPLAIN each one it can actually take (a discovery affordance — + // the palette submits this bare form), marking the active one. const current = store.getSnapshot().reasoningEffort; - const rows = REASONING_EFFORTS.map( - (e) => ` ${e.padEnd(8)} ${EFFORT_TIER_HINT[e]}${e === current ? ' (current)' : ''}`, - ); - const header = capable - ? `reasoning effort: ${current ?? 'default (provider)'}` - : `reasoning effort: ${built.agent.model} has no controllable reasoning tier — a tier would be ignored`; - emitOutput(`${header}\n${rows.join('\n')}`); + if (offered.length === 0) { + emitOutput(`reasoning effort: ${effortUnavailableNote(built.agent.model)}`); + return; + } + // Label + mark through the SAME projection the picker uses, so a budget model reads "on" and a bound tier + // that was deduped away still marks its representative row. + const currentRow = + current === undefined + ? undefined + : projectEffortToRow(built.agent.model, offered, current); + const rows = offered.map((e) => { + const { label, hint } = effortRowLabel(built.agent.model, e); + return ` ${label.padEnd(8)} ${hint}${e === currentRow ? ' (current)' : ''}`; + }); + const currentLabel = + current === undefined + ? 'default (provider)' + : effortRowLabel(built.agent.model, current).label; + emitOutput(`reasoning effort: ${currentLabel}\n${rows.join('\n')}`); return; } - const tier = REASONING_EFFORTS.find((e) => e === requested); + // Accept EITHER a canonical tier name OR the DISPLAY label the discovery list above printed — a budget model + // advertises an "on" row, so `/effort on` must resolve to that offered tier rather than reading as an unknown + // tier (an affordance the list showed but the parser rejected). Label-match is scoped to `offered`, so "on" + // on a graded model — which shows no "on" row — still falls through to the raw match and is refused. + const tier = + offered.find((e) => effortRowLabel(built.agent.model, e).label === requested) ?? + REASONING_EFFORTS.find((e) => e === requested); if (tier === undefined) { emitOutput( `/effort: unknown tier '${requested.replace(/[^\x20-\x7e]/g, '?').slice(0, 16)}'`, ); return; } + if (!offered.includes(tier)) { + // A REAL tier that THIS model rejects. Refuse it here rather than binding it and letting the gate drop it + // at send: a tier that is set, displayed, and never sent is the silent-billing bug this work removes. + emitOutput(effortRejectedNote(built.agent.model, tier, offered)); + return; + } modeControl.onSetEffort(tier); emitOutput( - capable - ? `reasoning effort: ${tier} — applies to your next message.` - : `reasoning effort: ${tier} set, but ${built.agent.model} has no reasoning control — it will be ignored.`, + `reasoning effort: ${effortRowLabel(built.agent.model, tier).label} — applies to your next message.`, ); }, // `/thinking` (2.5.H): toggle the collapsible reasoning panel — a pure store-view flip (no session/engine @@ -1366,6 +1426,10 @@ interface FreshChatWiringDeps { readonly opened: OpenedSessionStore; readonly buildSession: typeof buildChatSession; readonly onBudgetWarning: NonNullable; + /** Withheld-tier sink (ADR-0071 §6) — threaded exactly like {@link FreshChatWiringDeps.onBudgetWarning}, because a + * `/clear` rebuild binds a NEW session and a session with no sink withholds a tier in silence. */ + readonly onEffortWithheld: NonNullable; + readonly onUnpriced: NonNullable; /** `[preferences].alt_screen` (2.6.F, ADR-0068 §e) — carried into the rebuilt `ReplWiring` so a `/clear` re-drive * keeps the full-screen render mode (else the mode reverts to the phase default mid-conversation). */ readonly altScreen?: boolean | undefined; @@ -1390,6 +1454,8 @@ async function buildFreshChatWiring(deps: FreshChatWiringDeps, intro: string): P ...(deps.mcpRegistrations === undefined ? {} : { mcpRegistrations: deps.mcpRegistrations }), ...(resolvePrice.size === 0 ? {} : { resolvePrice }), onBudgetWarning: deps.onBudgetWarning, + onEffortWithheld: deps.onEffortWithheld, + onUnpriced: deps.onUnpriced, }); // The SAME signals `runReplLoop` used for the hoist (`deps.altScreen` is `[preferences].alt_screen`, carried here // precisely so a `/clear` re-drive keeps the mode) — so a rebuilt session cannot silently re-acquire the 4000-char @@ -1497,6 +1563,8 @@ function createClearRebuild(params: { buildSession: params.buildSession, altScreen: params.altScreen, onBudgetWarning: (warning) => emitLiveNotice(params.io, budgetWarningText(warning)), + onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(params.io, note)), + onUnpriced: (note) => emitLiveNotice(params.io, note), }; return (oldSessionId) => buildFreshChatWiring(wiringDeps, clearedNotice(oldSessionId)); } @@ -1569,6 +1637,10 @@ interface ReseatWiringDeps { readonly opened: OpenedSessionStore; readonly buildResumedSession: typeof buildResumedChatSession; readonly onBudgetWarning: NonNullable; + /** Withheld-tier sink (ADR-0071 §6) — a `/models` reseat binds a DIFFERENT model, which is precisely when a tier + * that was fine a moment ago stops being accepted. Threaded like {@link ReseatWiringDeps.onBudgetWarning}. */ + readonly onEffortWithheld: NonNullable; + readonly onUnpriced: NonNullable; /** `[preferences].alt_screen` (2.6.F, ADR-0068 §e) — carried into the rebuilt `ReplWiring` so a `/models` reseat * keeps the full-screen render mode (else it reverts to the phase default after a mid-session model switch). */ readonly altScreen?: boolean | undefined; @@ -1626,6 +1698,8 @@ async function buildReseatWiring( ...(deps.mcpRegistrations === undefined ? {} : { mcpRegistrations: deps.mcpRegistrations }), ...(resolvePrice.size === 0 ? {} : { resolvePrice }), onBudgetWarning: deps.onBudgetWarning, + onEffortWithheld: deps.onEffortWithheld, + onUnpriced: deps.onUnpriced, }); let seeded: { store: ChatStoreController; persister: SessionPersister }; try { @@ -1715,6 +1789,8 @@ function createReseatRebuild(params: { buildResumedSession: params.buildResumedSession, altScreen: params.altScreen, onBudgetWarning: (warning) => emitLiveNotice(params.io, budgetWarningText(warning)), + onEffortWithheld: onceEffortNotice((note) => emitLiveNotice(params.io, note)), + onUnpriced: (note) => emitLiveNotice(params.io, note), }; return (oldSessionId, target, carriedTranscript) => buildReseatWiring(wiringDeps, oldSessionId, target, carriedTranscript); diff --git a/apps/cli/src/commands/dispatch.ts b/apps/cli/src/commands/dispatch.ts index 80bfaf62..862ba398 100644 --- a/apps/cli/src/commands/dispatch.ts +++ b/apps/cli/src/commands/dispatch.ts @@ -13,6 +13,7 @@ import { } from '../engine/providers.js'; import { openHistoryStore } from '../history/open.js'; import { openSessionStore } from '../history/session-open.js'; +import { refreshCatalog, type CatalogRefreshResult } from '../engine/catalog-refresh.js'; import { CliError } from '../process/errors.js'; import { type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; @@ -220,6 +221,21 @@ export function buildModelsPricingArgs(input: CommandInput): ModelsPricingComman if (provider === undefined) { throw new CliError('invalid_invocation', 'missing required option --provider .'); } + // `--clear` RETIRES an override (ADR-0071 §5) — the only way back from a price the user regrets. Before it existed + // there was none: a mispriced model could be corrected but never un-priced, so a user who overrode a catalog model + // by mistake was stuck with their own number for good. It takes no price flags, and rejects them rather than + // quietly ignoring half an invocation. + if (input.options['clear'] === true) { + for (const flag of ['input', 'output', 'cached'] as const) { + if (optString(input.options[flag]) !== undefined) { + throw new CliError( + 'invalid_invocation', + `--clear removes the price; it takes no --${flag}. Nothing written.`, + ); + } + } + return { model: reqPositional(input, 0, 'model'), provider, clear: true }; + } const rawInput = optString(input.options['input']); const rawOutput = optString(input.options['output']); if (rawInput === undefined) { @@ -409,12 +425,19 @@ export interface ModelsDbPorts { io: CliIo, providerStore: Pick, ) => Pick; + /** + * Fetch models.dev and install it (ADR-0071 §4a). A PORT, exactly like `openDb` and `makeResolver`, and for the + * same reason: a `models refresh` now has a network leg, and a unit test must be able to drive the whole wiring + * without one. The production port is the real fetch. + */ + readonly refreshCatalog: (homeDir: string) => Promise; } const PRODUCTION_MODELS_PORTS: ModelsDbPorts = { openDb: openLocalDb, makeResolver: (io, providerStore) => createProviderResolver(io.env, createOsKeychainStore(), { providerStore }), + refreshCatalog: (homeDir) => refreshCatalog({ homeDir }), }; /** @@ -428,7 +451,7 @@ export async function withModelsDeps( args: ModelsCommandArgs, ports: ModelsDbPorts = PRODUCTION_MODELS_PORTS, ): Promise { - const { homeDir } = loadResolvedConfig({ + const { homeDir, config } = loadResolvedConfig({ cwd: ctx.global.cwd, configPath: ctx.global.configPath, }); @@ -452,6 +475,11 @@ export async function withModelsDeps( global: ctx.global, catalog: catalogStore, refreshService, + // ADR-0071 §4a: the catalog half. The command holds a THUNK, not a socket — so a test drives it with a fake + // fetch, and the pure `@relavium/llm` catalog keeps taking data as an argument rather than reaching out itself. + refreshCatalog: () => ports.refreshCatalog(homeDir), + // Default OFF (ADR-0071 §4): absent config ⇒ no standing egress, and the shipped snapshot answers everything. + ...(config.catalogAutoRefresh ? { autoRefreshCatalog: true } : {}), providerSlug: createProviderSlugResolver(providerStore), }); } finally { @@ -460,8 +488,20 @@ export async function withModelsDeps( } const executeModels: CommandExecutor = (_input, ctx) => withModelsDeps(ctx, { refresh: false }); -const executeModelsRefresh: CommandExecutor = (_input, ctx) => - withModelsDeps(ctx, { refresh: true }); +const executeModelsRefresh: CommandExecutor = (input, ctx) => + withModelsDeps(ctx, { refresh: true, ...buildRefreshAxis(input) }); + +/** + * `--providers` / `--catalog` (ADR-0071 §4a) — which axis to refresh. Neither ⇒ BOTH, because "refresh what I know + * about models" is one intent. Both flags together is the same as neither, and saying so beats a pedantic error. + */ +function buildRefreshAxis(input: CommandInput): { axis?: 'providers' | 'catalog' } { + const providers = input.options['providers'] === true; + const catalog = input.options['catalog'] === true; + if (providers && !catalog) return { axis: 'providers' }; + if (catalog && !providers) return { axis: 'catalog' }; + return {}; +} /** * `models pricing ` (2.5.G S10, ADR-0065) — open the local db, build the catalog + provider stores over it, diff --git a/apps/cli/src/commands/gate.ts b/apps/cli/src/commands/gate.ts index f5cf2065..77400527 100644 --- a/apps/cli/src/commands/gate.ts +++ b/apps/cli/src/commands/gate.ts @@ -18,6 +18,7 @@ import { type BuildEngineOptions, } from '../engine/build-engine.js'; import { createHistoryCheckpointer } from '../engine/checkpointer.js'; +import { onceEffortNotice, unpricedModelNote } from '../chat/effort-notice.js'; import { createCliHost } from '../engine/host.js'; import { sweepHostMediaBestEffort as defaultSweepMedia, @@ -222,6 +223,16 @@ export async function gateCommand(args: GateCommandArgs, deps: GateCommandDeps): const resolvePrice = readUserPricingOverlay(opened.db); const engine = await (deps.buildEngine ?? defaultBuildEngine)({ providers, + // ADR-0071 §6: the far side of a gate re-runs agent nodes, so an authored tier the bound model rejects is + // withheld here too — and this surface has no other safety net (no picker, no footer, no client-side check). + // stderr, never stdout (`--json`). + onEffortWithheld: onceEffortNotice((note) => deps.io.writeErr(`warning: ${note}\n`)), + // ADR-0071 §K7: a resumed agent node on an unpriced model degrades to `allow`, so `budget.max_cost_microcents` + // did not apply to it — say so on stderr (never stdout, `--json`). `budget.strict_cost_cap` blocks instead. + onUnpriced: (model, capMicrocents) => + deps.io.writeErr( + `warning: ${unpricedModelNote(model, capMicrocents, 'budget.strict_cost_cap')}\n`, + ), // 2.5.A (ADR-0055): wire the SAME read+write fs + process ToolHost the `relavium run` path wires, jailed // to the ORIGINAL run's project root (`saveToRoot` — the original `runs.project_root` when it still exists // on this machine, else the resumer's cwd, exactly like the `save_to` root) at the resolved `fs_scope`. So a diff --git a/apps/cli/src/commands/manifest.ts b/apps/cli/src/commands/manifest.ts index 83d28c41..1301b398 100644 --- a/apps/cli/src/commands/manifest.ts +++ b/apps/cli/src/commands/manifest.ts @@ -291,14 +291,27 @@ const ENTRIES: readonly CommandManifestEntry[] = [ { id: 'models.refresh', label: 'Refresh models', - description: "Re-fetch each connected provider's live model list into the local cache.", + description: + "Re-fetch what we know about models: each connected provider's live list, and the models.dev catalog.", + args: [ + { + name: 'providers', + type: 'boolean', + description: "availability only — each connected provider's live model list", + }, + { + name: 'catalog', + type: 'boolean', + description: 'metadata only — prices, ceilings and reasoning tiers from models.dev', + }, + ], effect: 'write', }, { id: 'models.pricing', label: 'Set model pricing', description: - 'Set a user price for a model the registry does not know (custom / new provider models).', + 'Set your own price for a model — it overrides the catalog (you hold the invoice). --clear removes it.', args: [ { name: 'model', type: 'string', required: true, description: 'the model id to price' }, { @@ -308,21 +321,27 @@ const ENTRIES: readonly CommandManifestEntry[] = [ description: 'the provider that serves the model (must be registered)', }, { + // Not `required` at the manifest level, because `--clear` takes none of the three price flags. The command's + // arg builder enforces the real rule: exactly one of "set a price" or "--clear". name: 'input', type: 'string', - required: true, - description: 'input (prompt) price, USD per million tokens', + description: 'input (prompt) price, USD per million tokens (required unless --clear)', }, { name: 'output', type: 'string', - required: true, - description: 'output (completion) price, USD per million tokens', + description: 'output (completion) price, USD per million tokens (required unless --clear)', }, { name: 'cached', type: 'string', - description: 'cache-read price, USD per million tokens (default 0)', + description: + "cache-read price, USD per million tokens; omitted ⇒ the catalog's cache discount, applied to your input rate", + }, + { + name: 'clear', + type: 'boolean', + description: "remove your price for this model — it falls back to the catalog's", }, ], effect: 'write', diff --git a/apps/cli/src/commands/models-dispatch.test.ts b/apps/cli/src/commands/models-dispatch.test.ts index 57ace558..32322763 100644 --- a/apps/cli/src/commands/models-dispatch.test.ts +++ b/apps/cli/src/commands/models-dispatch.test.ts @@ -95,6 +95,9 @@ describe('withModelsDeps (2.5.G S5 — real-db wiring + lazy slug + close-on-fau // double-closes the better-sqlite3 handle. openDb: () => ({ db: client.db, close: () => onClose?.() }), makeResolver: () => resolver, + // NETWORK-FREE (ADR-0071 §4a). A `models refresh` now has a models.dev leg, and a unit test must never take it: + // this port is what keeps the whole wiring — including the close-on-fault lifecycle — testable offline. + refreshCatalog: () => Promise.resolve({ status: 'refreshed' as const, models: 80, added: 0 }), }; } @@ -138,7 +141,7 @@ describe('withModelsDeps (2.5.G S5 — real-db wiring + lazy slug + close-on-fau await expect( withModelsDeps( context(io, false), - { refresh: true }, + { refresh: true, axis: 'providers' }, testPorts(resolver, () => { closed = true; }), diff --git a/apps/cli/src/commands/models-pricing.test.ts b/apps/cli/src/commands/models-pricing.test.ts index fdcd46c2..a76fd011 100644 --- a/apps/cli/src/commands/models-pricing.test.ts +++ b/apps/cli/src/commands/models-pricing.test.ts @@ -7,7 +7,7 @@ import { type ModelCatalogStore, type ProviderStore, } from '@relavium/db'; -import { KNOWN_MODEL_IDS } from '@relavium/llm'; +import { priceModel } from '@relavium/llm'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { isCliError } from '../process/errors.js'; @@ -116,16 +116,48 @@ describe('modelsPricingCommand (2.5.G S10)', () => { inputCostPerMtokMicrocents: 300_000_000, outputCostPerMtokMicrocents: 900_000_000, cachedInputCostPerMtokMicrocents: 0, + // The catalog has never heard of `acme-custom-1` — the case the user tier was invented for. Nothing is + // overridden, so there is nothing to declare. + overriddenCatalogPrice: null, }); }); - it('REJECTS a canonical model id (the built-in price always wins) — nothing is written', () => { - const canonical = KNOWN_MODEL_IDS[0]; - if (canonical === undefined) throw new Error('test precondition: KNOWN_MODEL_IDS is non-empty'); - const err = runThrows({ ...baseArgs, model: canonical }); + it('--json DECLARES the catalog price an override replaces (ADR-0071 §5)', () => { + // The flip removed the guarantee that a user cannot misprice a shipped model. The condition on which it was + // removed is that the divergence is LOUD — for a machine consumer as much as for a human. Without this, a + // `--json` caller cannot tell a price that fills a gap from one that overrules a number we shipped. + const { code, out } = run({ ...baseArgs, model: 'gpt-5.5' }, true); + expect(code).toBe(EXIT_CODES.success); + const [rec] = parseNdjson(out); + const overridden = (rec as { overriddenCatalogPrice: { inputCostPerMtokMicrocents: number } }) + .overriddenCatalogPrice; + expect(overridden).not.toBeNull(); + expect(overridden.inputCostPerMtokMicrocents).toBeGreaterThan(0); + }); + + it('REFUSES a price under a provider the CATALOG anchors elsewhere — never stored-then-ignored', () => { + // The catalog anchors a model id to ONE provider. Both the merge and the cost overlay drop a user row that + // contradicts it, so writing one would store a price that silently never applies — the user believes they set + // it, and nothing reads it. Worse: before this guard the two DISAGREED — the merge dropped the row while + // `priceModel` billed it, so the picker kept showing the catalog price while the CostTracker charged the user's. + // `claude-opus-4-8` is Anthropic's; `openai` is the provider registered in this db. The FK guard passes (openai + // exists) and the CATALOG guard is the one that must catch it. + const err = runThrows({ ...baseArgs, model: 'claude-opus-4-8', provider: 'openai' }); expect(err.code).toBe('invalid_invocation'); - expect(err.message).toContain('built-in price'); - expect(catalog.listAll().find((m) => m.modelId === canonical)).toBeUndefined(); + expect(err.message).toContain("anthropic's model"); + expect(catalog.listAll().find((m) => m.modelId === 'claude-opus-4-8')).toBeUndefined(); + }); + + it('ACCEPTS a price for a model the CATALOG already knows — the override is the feature now', () => { + // This test asserted a REJECTION until ADR-0071 §1: the shipped table always won, so a user override would have + // been a silent no-op, and refusing it was the honest thing. Pricing resolves USER → CATALOG now, so a + // negotiated rate — or an enterprise discount, or a price our snapshot has not caught up with — takes effect. + // The user is the one holding the invoice. + expect(priceModel('gpt-5.5').inputPerMtokMicrocents).toBeGreaterThan(0); // the catalog does know it + expect(run({ ...baseArgs, model: 'gpt-5.5' }).code).toBe(0); + const written = catalog.listAll().find((m) => m.modelId === 'gpt-5.5'); + expect(written).toBeDefined(); + expect(written?.source).toBe('user'); }); it('REJECTS an unregistered provider (the catalog FK targets llm_providers) — nothing is written', () => { @@ -224,4 +256,45 @@ describe('modelsPricingCommand (2.5.G S10)', () => { } throw new Error('expected modelsPricingCommand to throw a CliError'); } + + it('--clear RETIRES an override — the only way back from a price the user regrets', () => { + // Before `--clear`, a mispriced model could be corrected but never UN-priced: a user who overrode a catalog model + // by mistake was stuck with their own number for good. The fix commit's own message told them to run this command + // — and it did not exist. + expect(run({ ...baseArgs, model: 'gpt-5.5' }).code).toBe(EXIT_CODES.success); + expect(catalog.listAll().find((m) => m.modelId === 'gpt-5.5')).toBeDefined(); + + const { code, out } = run({ model: 'gpt-5.5', provider: 'openai', clear: true }); + expect(code).toBe(EXIT_CODES.success); + expect(out).toContain('falls back to the catalog'); + // SOFT-deactivated, never deleted (`model_catalog.id` is an FK target from five tables), so the active-only + // reader stops seeing it and the model falls back to the catalog's price. + expect(catalog.listAll().find((m) => m.modelId === 'gpt-5.5')).toBeUndefined(); + }); + + it('--clear on a model with no user price is an honest no-op, never a lie', () => { + const { code, out } = run({ model: 'gpt-5.5', provider: 'openai', clear: true }); + expect(code).toBe(EXIT_CODES.success); + expect(out).toContain('no user price to clear'); + }); + + it('--clear --json reports `source` from the CATALOG price, matching the human line (bot fix)', () => { + // A cleared model the catalog PRICES falls back to it → source 'catalog'. + run({ ...baseArgs, model: 'gpt-5.5' }); // set a user price first + const catalogClear = run({ model: 'gpt-5.5', provider: 'openai', clear: true }, true); + expect(JSON.parse(catalogClear.out.trim())).toMatchObject({ + model: 'gpt-5.5', + cleared: true, + source: 'catalog', + }); + + // A cleared model the catalog does NOT price → source null (unpriced; `cleared ? 'catalog'` used to LIE here). + run({ ...baseArgs, model: 'acme-custom-1' }); // a user price on a non-catalog id + const nullClear = run({ model: 'acme-custom-1', provider: 'openai', clear: true }, true); + expect(JSON.parse(nullClear.out.trim())).toMatchObject({ + model: 'acme-custom-1', + cleared: true, + source: null, + }); + }); }); diff --git a/apps/cli/src/commands/models-pricing.ts b/apps/cli/src/commands/models-pricing.ts index 3f29033a..d861ba3b 100644 --- a/apps/cli/src/commands/models-pricing.ts +++ b/apps/cli/src/commands/models-pricing.ts @@ -1,5 +1,5 @@ import type { ModelCatalogStore, ProviderStore } from '@relavium/db'; -import { isCanonicalModelId } from '@relavium/llm'; +import { catalogModel, catalogPricing } from '@relavium/llm'; import { CliError } from '../process/errors.js'; import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; @@ -8,36 +8,9 @@ import type { GlobalOptions } from '../process/options.js'; import { writeRecordLines } from '../render/records.js'; import { stripTerminalControls } from '../render/tui/chat-projection.js'; -/** - * The `relavium models pricing ` capture command (workstream **2.5.G S10**, - * [ADR-0065](../../../../docs/decisions/0065-provider-economics-and-extensibility.md) §1/§2) — hand-enter the - * per-Mtok text-token price of a model the static registry does NOT know (a custom-endpoint model, or a new - * provider model not yet in `MODEL_PRICING`), so the cost cap (`max_cost_microcents`) can enforce it. The price is - * stored as a `source='user'` `model_catalog` row (integer micro-cents, never float); a live `models refresh` NEVER - * clobbers it (ADR-0065 §1). It closes the ADR-0064 §6 gap where an unknown model's cost governance degraded to - * `allow` because no price existed. - * - * Framework-free (no `commander`): parsed args + injected stores in, output via {@link CliIo}; a fault throws a - * typed {@link CliError} (exit 2). The command is PURE of secrets — it writes only a model id + provider + prices, - * never a key. It writes nothing on any validation failure (the reject precedes the upsert). - * - * Precedence guard: a **canonical** model id (one already in `MODEL_PRICING`) is REJECTED — the static registry - * always wins for a known id in both the merge and the cost path (ADR-0065 §2), so a user override would be - * silently ignored; failing loud is honest. Provider guard: the `` must be a REGISTERED provider (the catalog - * row's FK targets `llm_providers`), else the user is told to add it first. - */ - -export interface ModelsPricingCommandArgs { - /** The model id to price — a NON-canonical id (a custom / not-yet-registered model). */ - readonly model: string; - /** The provider slug that serves the model (e.g. `openai`) — must already be registered. */ - readonly provider: string; - /** Input (prompt) price, USD per million tokens. */ - readonly inputUsdPerMtok: number; - /** Output (completion) price, USD per million tokens. */ - readonly outputUsdPerMtok: number; - /** Cache-read price, USD per million tokens; omitted ⇒ `0` (no cache discount). */ - readonly cachedInputUsdPerMtok?: number; +/** Integer micro-cents/MTok → a USD string, for echoing the catalog price an override replaces (ADR-0071 §5). */ +function microcentsToUsd(microcents: number): string { + return (microcents / 100_000_000).toString(); } export interface ModelsPricingCommandDeps { @@ -46,7 +19,7 @@ export interface ModelsPricingCommandDeps { /** The catalog store — `upsert` writes the `source='user'` row (a pricing-only patch; the store preserves the * existing row's display/limits + media columns). `listAll` is read only to reject a cross-provider duplicate * (the overlay keys by model id, so the same id priced under two providers would be ambiguous). */ - readonly catalog: Pick; + readonly catalog: Pick; /** The provider registry — resolves the `` → its internal `llm_providers` UUID (the catalog FK). */ readonly providers: Pick; } @@ -76,18 +49,57 @@ function usdToMicrocents(usdPerMtok: number, flag: string): number { return Math.round(usdPerMtok * USD_PER_MTOK_TO_MICROCENTS); } +/** + * SET a price, or CLEAR one — a discriminated union, so a `--clear` invocation cannot carry a price and a set + * invocation cannot forget one. The two are different acts, and typing them as one optional-riddled shape is how a + * half-applied invocation gets written. + */ +export type ModelsPricingCommandArgs = SetPricingArgs | ClearPricingArgs; + +export interface SetPricingArgs { + readonly clear?: undefined; + /** The model id to price. Since ADR-0071 §1 this MAY be one the catalog already knows — the user outranks it. */ + readonly model: string; + /** The provider slug that serves the model — must be registered, AND must be the one the catalog anchors the model + * to (a contradicting price would never be applied anywhere, so it is refused rather than stored). */ + readonly provider: string; + /** Input (prompt) price, USD per million tokens. */ + readonly inputUsdPerMtok: number; + /** Output (completion) price, USD per million tokens. */ + readonly outputUsdPerMtok: number; + /** + * Cache-read price, USD per million tokens. + * + * Omitted ⇒ the row records that it was NOT stated (migration 0011), and a reader derives it: the catalog's cache + * DISCOUNT applied to the input rate the user gave. `--cached 0` is a real instruction and is believed — the flag + * is what lets a stored `0` mean "free" rather than "nobody said". + */ + readonly cachedInputUsdPerMtok?: number; +} + +/** `models pricing --provider

--clear` — retire the override; the model falls back to the catalog. */ +export interface ClearPricingArgs { + readonly clear: true; + readonly model: string; + readonly provider: string; +} + +export interface ModelsPricingCommandDeps { + readonly io: CliIo; + readonly global: GlobalOptions; + readonly catalog: Pick; + readonly providers: Pick; +} + export function modelsPricingCommand( args: ModelsPricingCommandArgs, deps: ModelsPricingCommandDeps, ): ExitCode { - // Precedence guard (ADR-0065 §2): a canonical id always resolves to `MODEL_PRICING`, so a user override would be - // a silent no-op. Reject BEFORE any write — nothing is stored on a rejected invocation. - if (isCanonicalModelId(args.model)) { - throw new CliError( - 'invalid_invocation', - `'${args.model}' already has a built-in price — a user override would never take effect (the static registry always wins). Nothing written.`, - ); - } + // The guard that lived here REFUSED a price for a model the shipped table already knew, because the table always + // won and the override would have been a silent no-op. It is gone with the table (ADR-0071 §1): pricing now + // resolves USER → CATALOG, so overriding a catalog model is not a mistake to be prevented — it is the feature. + // The user has a negotiated rate, or an enterprise discount, or simply a price our snapshot has not caught up + // with, and they are the one holding the invoice. // Provider guard: the catalog row's FK targets `llm_providers`, so the provider must already be registered. const providerRow = deps.providers.list().find((p) => p.name === args.provider); if (providerRow === undefined) { @@ -96,6 +108,14 @@ export function modelsPricingCommand( `unknown provider '${args.provider}' — register it first (e.g. \`relavium provider add ${args.provider}\` or \`relavium provider set-key ${args.provider}\`).`, ); } + + // `--clear` (ADR-0071 §5): retire the override and fall back to the catalog. The ONLY way back from a price the + // user regrets — before it existed, an override could be corrected but never removed, so a model priced by mistake + // stayed priced by the user for good. + if (args.clear === true) { + return runClearPricing(args, deps, providerRow.id); + } + // Cross-provider ambiguity guard (ADR-0065 §2): the cost overlay keys by MODEL ID (the runtime references a model // by id alone, no provider), so the SAME id user-priced under two providers can't be distinguished — the cap // would then apply an arbitrary one. Reject fail-loud rather than silently overwrite; nothing is written. @@ -112,11 +132,27 @@ export function modelsPricingCommand( `'${args.model}' is already user-priced under '${otherProvider}'. The cost cap keys by model id, so a second provider's price can't be distinguished — remove that price (re-price under '${otherProvider}') or use a distinct model id.`, ); } + // Provider-vs-CATALOG guard (ADR-0071 §1). The catalog anchors a model id to ONE provider, and both the merge + // and the cost overlay drop a row that contradicts it — so writing one would store a price that silently never + // applies, which is the worst of both worlds: the user believes they set it, and nothing reads it. + const anchored = catalogModel(args.model)?.provider; + if (anchored !== undefined && anchored !== args.provider) { + throw new CliError( + 'invalid_invocation', + `'${args.model}' is ${anchored}'s model, not ${args.provider}'s — the catalog anchors a model id to one provider, and a price under the wrong one would never be applied. Re-run with \`--provider ${anchored}\`. Nothing written.`, + ); + } // Convert + bounds-validate BEFORE the write (a bad `--cached` must not leave a partially-applied row). const inputCostPerMtokMicrocents = usdToMicrocents(args.inputUsdPerMtok, '--input'); const outputCostPerMtokMicrocents = usdToMicrocents(args.outputUsdPerMtok, '--output'); - // OMITTED `--cached` ⇒ `undefined` (not `0`): so the upsert can OMIT the column and the store PRESERVES an - // existing cached price on a re-price, rather than the `??`-passing `0` silently zeroing a hand-entered rate. + // OMITTED `--cached` ⇒ `undefined`, so the upsert OMITS the column and the store PRESERVES an existing rate: a + // re-price must never zero — or overwrite — a cache rate the user hand-entered earlier. + // + // `--cached 0` is a REAL instruction (a self-hosted endpoint whose cache genuinely costs nothing), and the money + // column cannot tell it apart from "never mentioned" — both are `0`. So the FACT of the statement rides in its own + // flag (migration 0011): stated ⇒ the number is theirs, zero included; not stated ⇒ a reader DERIVES it from the + // catalog's cache discount applied to their own input rate (see `rowToUserPricing`). Reading a stored `0` as + // "free" would bill a whole class of tokens at nothing; reading it as "unset" would discard what the user typed. const cachedInputCostPerMtokMicrocents = args.cachedInputUsdPerMtok === undefined ? undefined @@ -133,7 +169,11 @@ export function modelsPricingCommand( source: 'user', inputCostPerMtokMicrocents, outputCostPerMtokMicrocents, - ...(cachedInputCostPerMtokMicrocents === undefined ? {} : { cachedInputCostPerMtokMicrocents }), + // The rate AND the fact that it was stated ride together, or neither does. Writing the number without the flag + // would leave a reader deriving a value the user had explicitly typed. + ...(cachedInputCostPerMtokMicrocents === undefined + ? {} + : { cachedInputCostPerMtokMicrocents, cachedInputStated: true }), }); if (deps.global.json) { @@ -148,6 +188,19 @@ export function modelsPricingCommand( // The `--json` field stays present as `0` when `--cached` was omitted (unchanged contract) even though the // store now PRESERVES the existing cached rate rather than writing this `0` (see the upsert above). cachedInputCostPerMtokMicrocents: cachedInputCostPerMtokMicrocents ?? 0, + // The catalog price this override REPLACES (ADR-0071 §5) — `null` when the catalog does not price the model + // at all, which is the case the user tier was originally invented for. A machine consumer must be able to + // see the divergence for the same reason a human must: the flip removed the guard that made it impossible. + overriddenCatalogPrice: (() => { + const shipped = catalogPricing(args.model); + return shipped === undefined + ? null + : { + inputCostPerMtokMicrocents: shipped.inputPerMtokMicrocents, + outputCostPerMtokMicrocents: shipped.outputPerMtokMicrocents, + cachedInputCostPerMtokMicrocents: shipped.cachedInputPerMtokMicrocents, + }; + })(), }, ]); return EXIT_CODES.success; @@ -155,12 +208,53 @@ export function modelsPricingCommand( const cachedNote = args.cachedInputUsdPerMtok === undefined ? '' : `, cached $${args.cachedInputUsdPerMtok}/Mtok`; + // THE DIVERGENCE IS LOUD (ADR-0071 §5) — and it is the condition on which the "a user can never misprice a + // shipped model" guard was removed. The user outranks the catalog now; they get what they asked for. What they do + // NOT get is to do it in silence, so when their number disagrees with the one we shipped, we say both. + const shipped = catalogPricing(args.model); + const divergence = + shipped === undefined + ? '' + : `\n Overrides the catalog price for this model: input $${microcentsToUsd(shipped.inputPerMtokMicrocents)}/Mtok, output $${microcentsToUsd(shipped.outputPerMtokMicrocents)}/Mtok. Yours wins. Run \`relavium models pricing ${stripTerminalControls(args.model)} --clear\` to go back to the catalog's.`; // Strip any terminal-control byte from the (user-typed) model id before echo — parity with `renderModelList`'s // FIX 2. `ModelListingSchema` only requires min(1), so an id can carry a control byte; the JSON path is safe on // its own (JSON.stringify escapes them). The provider is a validated (kebab) ProviderId, and the prices are // numbers — both already safe. deps.io.writeOut( - `Set user pricing for ${stripTerminalControls(args.model)} (${args.provider}): input $${args.inputUsdPerMtok}/Mtok, output $${args.outputUsdPerMtok}/Mtok${cachedNote}. It applies to your next run/chat and survives \`models refresh\`.\n`, + `Set user pricing for ${stripTerminalControls(args.model)} (${args.provider}): input $${args.inputUsdPerMtok}/Mtok, output $${args.outputUsdPerMtok}/Mtok${cachedNote}. It applies to your next run/chat and survives \`models refresh\`.${divergence}\n`, ); return EXIT_CODES.success; } + +/** `--clear` (ADR-0071 §5): retire the user override for one model+provider and report the fallback. */ +function runClearPricing( + args: ClearPricingArgs, + deps: ModelsPricingCommandDeps, + providerId: string, +): ExitCode { + const cleared = deps.catalog.clearUserPricing(args.model, providerId); + if (deps.global.json) { + // `source` is the EFFECTIVE price source after the clear, decided by the catalog exactly as + // `clearedPricingMessage` decides its words — 'catalog' iff the catalog actually prices the model, else null + // (unpriced, the cost cap will not apply). `cleared ? 'catalog'` was wrong: a cleared model the catalog does + // NOT price reported 'catalog' while the human line said the opposite. + const source = catalogPricing(args.model) !== undefined ? 'catalog' : null; + writeRecordLines(deps.io, [{ model: args.model, provider: args.provider, cleared, source }]); + return EXIT_CODES.success; + } + deps.io.writeOut(clearedPricingMessage(args, cleared)); + return EXIT_CODES.success; +} + +/** The human line for a `--clear`, as a flat if/else so no nested ternary decides the message. */ +function clearedPricingMessage(args: ClearPricingArgs, cleared: boolean): string { + const safeModel = stripTerminalControls(args.model); + if (!cleared) { + return `${safeModel} (${args.provider}) has no user price to clear — nothing changed.\n`; + } + const shipped = catalogPricing(args.model); + if (shipped === undefined) { + return `Cleared the user price for ${safeModel} (${args.provider}). The catalog does not price this model, so its cost cap will not apply until you price it again.\n`; + } + return `Cleared the user price for ${safeModel} (${args.provider}). It falls back to the catalog: input $${microcentsToUsd(shipped.inputPerMtokMicrocents)}/Mtok, output $${microcentsToUsd(shipped.outputPerMtokMicrocents)}/Mtok.\n`; +} diff --git a/apps/cli/src/commands/models.test.ts b/apps/cli/src/commands/models.test.ts index b8c2c796..2035b39e 100644 --- a/apps/cli/src/commands/models.test.ts +++ b/apps/cli/src/commands/models.test.ts @@ -31,6 +31,7 @@ function modelRow( inputCostPerMtokMicrocents: 0, outputCostPerMtokMicrocents: 0, cachedInputCostPerMtokMicrocents: 0, + cachedInputStated: false, source, lastRefreshedAt: 1_700_000_000_000, isActive: true, @@ -68,7 +69,14 @@ function deps( json = false, providerSlug: (uuid: string) => string = (uuid) => uuid, ): ModelsCommandDeps { - return { io, global: globalOptions(json), catalog, refreshService, providerSlug }; + return { + io, + global: globalOptions(json), + catalog, + refreshService, + refreshCatalog: () => Promise.resolve({ status: 'refreshed' as const, models: 80, added: 0 }), + providerSlug, + }; } const REFRESHED: RefreshReport = { @@ -238,10 +246,130 @@ describe('modelsCommand — refresh', () => { deactivated: null, error: null, }, + // One record per SOURCE (ADR-0071 §4a) — the provider lists, then the catalog. A script can tell which half + // worked, because the two fail independently: an offline models.dev does not stop a provider refresh, and a + // keyless install can still refresh the catalog. + { source: 'catalog', status: 'refreshed', models: 80, added: 0 }, ]); }); - it('exits 2 on an explicit refresh with zero providers connected (no key at all)', async () => { + it('--providers refreshes availability ONLY — no models.dev egress at all', async () => { + const { io, out } = captureIo(); + const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; + let catalogFetched = false; + const d = deps(io, stubCatalog(rowsRef), stubRefresh(REFRESHED), true); + + await modelsCommand( + { refresh: true, axis: 'providers' }, + { + ...d, + refreshCatalog: () => { + catalogFetched = true; + return Promise.resolve({ status: 'refreshed' as const, models: 80, added: 0 }); + }, + }, + ); + expect(catalogFetched).toBe(false); // the flag is a promise about egress, not a display preference + expect(parseNdjson(out()).some((r) => (r as { source?: string }).source === 'catalog')).toBe( + false, + ); + }); + + it('--catalog refreshes metadata ONLY — and works with NO provider key at all', async () => { + // The form that makes default-OFF livable: a user with no key yet, who wants to see what things cost, types one + // command and gets them. Asking for a provider key here would be asking for the one thing they do not have. + const { io, out } = captureIo(); + const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; + let providersFetched = false; + const d = deps(io, stubCatalog(rowsRef), stubRefresh(REFRESHED), true); + + const code = await modelsCommand( + { refresh: true, axis: 'catalog' }, + { + ...d, + refreshService: stubRefresh(REFRESHED, () => { + providersFetched = true; + }), + }, + ); + expect(code).toBe(EXIT_CODES.success); + expect(providersFetched).toBe(false); + expect(parseNdjson(out())).toEqual([ + { source: 'catalog', status: 'refreshed', models: 80, added: 0 }, + ]); + }); + + it('[catalog] auto_refresh — a bare `models` refreshes the catalog FIRST when opted in', async () => { + // The opt-in standing-egress surface (ADR-0071 §4). Default OFF; when a user turns it on, a bare `models` (not a + // `refresh`) fetches the catalog before listing, because a stale price is the thing they came to look at. + let refreshedBeforeList = false; + let listed = false; + const catalog: ModelsCatalogReader = { + listAll: () => { + listed = true; + return []; + }, + }; + const { io } = captureIo(); + await modelsCommand( + { refresh: false }, + { + ...deps(io, catalog, stubRefresh(REFRESHED)), + autoRefreshCatalog: true, + refreshCatalog: () => { + refreshedBeforeList = !listed; // true iff refresh ran BEFORE the list read + return Promise.resolve({ status: 'refreshed' as const, models: 80, added: 0 }); + }, + }, + ); + expect(refreshedBeforeList).toBe(true); + }); + + it('[catalog] auto_refresh OFF (the default) — a bare `models` touches NO network', async () => { + const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; + let fetched = false; + const { io } = captureIo(); + await modelsCommand( + { refresh: false }, + { + ...deps(io, stubCatalog(rowsRef), stubRefresh(REFRESHED)), + refreshCatalog: () => { + fetched = true; + return Promise.resolve({ status: 'refreshed' as const, models: 80, added: 0 }); + }, + }, + ); + expect(fetched).toBe(false); // default OFF: no standing egress + }); + + it('a FAILED catalog refresh is a NOTE, never an error — the shipped snapshot still answers', async () => { + // ADR-0071 §4: additive only. A refresh that cannot reach models.dev must leave the product knowing exactly what + // it knew before — never a blank catalog, never a model that was priced yesterday and is not today. The cost cap + // is a safety control; it does not lapse because a third party had a bad deploy. + const { io, out, err } = captureIo(); + const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; + const d = deps(io, stubCatalog(rowsRef), stubRefresh(REFRESHED), false); + + const code = await modelsCommand( + { refresh: true, axis: 'catalog' }, + { + ...d, + refreshCatalog: () => + Promise.resolve({ + status: 'failed' as const, + models: 0, + added: 0, + reason: 'models.dev unreachable', + }), + }, + ); + expect(code).toBe(EXIT_CODES.success); // exit 0 — the command did what it could, and said so + expect(err()).toContain('models.dev unreachable'); + expect(err()).toContain('keeping the shipped catalog'); + expect(out()).not.toContain('models from models.dev'); + }); + + it('exits 2 on a PROVIDERS refresh with zero providers connected (no key at all)', async () => { const { io, out } = captureIo(); const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; const refresh = stubRefresh({ @@ -253,7 +381,10 @@ describe('modelsCommand — refresh', () => { let thrown: unknown; try { - await modelsCommand({ refresh: true }, deps(io, stubCatalog(rowsRef), refresh)); + await modelsCommand( + { refresh: true, axis: 'providers' }, + deps(io, stubCatalog(rowsRef), refresh), + ); expect.unreachable('should have thrown'); } catch (err) { thrown = err; @@ -265,6 +396,24 @@ describe('modelsCommand — refresh', () => { expect(out()).toBe(''); // stdout stays empty on a fault }); + it('a BOTH-axis refresh with no key SUCCEEDS when the catalog delivered — half a loaf is not a fault', async () => { + // A user with no key yet, running `models refresh` to see what things cost, HAS been served: the catalog is the + // half that does not need one. Telling them their command failed would be a lie, and it would hide the half that + // worked behind an error about the half that could not. + const { io, out } = captureIo(); + const rowsRef: { value: ModelCatalogListing[] } = { value: [] }; + const refresh = stubRefresh({ + providers: [ + { provider: 'anthropic', status: 'skipped-no-key' }, + { provider: 'openai', status: 'skipped-no-key' }, + ], + }); + + const code = await modelsCommand({ refresh: true }, deps(io, stubCatalog(rowsRef), refresh)); + expect(code).toBe(EXIT_CODES.success); + expect(out()).toContain('models from models.dev'); + }); + it('exits 0 when every connected provider FAILED — a per-provider failure is not a command failure (FIX 6)', async () => { // Every provider is `failed` (connected, but the fetch threw) — NOT `skipped-no-key`, so it is NOT the // zero-connected exit-2 case; the command prints the per-provider report and exits 0. diff --git a/apps/cli/src/commands/models.ts b/apps/cli/src/commands/models.ts index 32e961e5..e1b0ab07 100644 --- a/apps/cli/src/commands/models.ts +++ b/apps/cli/src/commands/models.ts @@ -5,6 +5,7 @@ import type { RefreshProviderResult, RefreshReport, } from '../engine/model-refresh.js'; +import type { CatalogRefreshResult } from '../engine/catalog-refresh.js'; import { CliError } from '../process/errors.js'; import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { CliIo } from '../process/io.js'; @@ -34,6 +35,15 @@ import { stripTerminalControls } from '../render/tui/chat-projection.js'; export interface ModelsCommandArgs { /** `true` for `models refresh` (force a live re-fetch); `false` for a bare `models` (list the cache). */ readonly refresh: boolean; + /** + * Which axis to refresh (ADR-0071 §4a). Absent ⇒ **BOTH**, because *"refresh what I know about models"* is one + * user intent, not two: the provider lists say what you can CALL, the catalog says what it COSTS, and a user who + * types `models refresh` wants both to be current. + * + * `'providers'` is ADR-0064's original behaviour, kept addressable for a script that only wants availability; + * `'catalog'` is metadata only, and is the one form that works with no provider key at all. + */ + readonly axis?: 'providers' | 'catalog'; } /** The narrow catalog reader the list path needs. */ @@ -46,6 +56,17 @@ export interface ModelsCommandDeps { readonly global: GlobalOptions; readonly catalog: ModelsCatalogReader; readonly refreshService: ModelRefreshService; + /** + * Fetch models.dev and install it (ADR-0071 §4a). Injected so the command stays testable without a network — and + * so the pure `@relavium/llm` catalog keeps taking data as an argument rather than reaching for a socket. + */ + readonly refreshCatalog: () => Promise; + /** + * `[catalog] auto_refresh` (ADR-0071 §4) — DEFAULT `false`, and the default is the design. Absent/`false` ⇒ a bare + * `models` never touches the network for metadata; the shipped snapshot answers. `true` ⇒ the list refreshes the + * catalog first, because a user who opted in wants the prices to be current where they can see them. + */ + readonly autoRefreshCatalog?: boolean; /** * Translate an internal `llm_providers` UUID (the FK `ModelCatalogListing.providerId` carries) → its provider * SLUG (e.g. `anthropic`) for the human table + the `--json` `provider` field, so the list path matches the @@ -60,33 +81,109 @@ export async function modelsCommand( deps: ModelsCommandDeps, ): Promise { if (args.refresh) { - return runRefresh(deps); + return runRefresh(deps, args.axis); } return runList(deps); } -/** `models refresh` — force a live re-fetch and report per-provider outcomes. */ -async function runRefresh(deps: ModelsCommandDeps): Promise { - const report = await deps.refreshService.refresh(); +/** + * `models refresh` — re-fetch, and report per-SOURCE outcomes (ADR-0071 §4a). + * + * Two axes, one command. The provider lists say which models a key can REACH; the catalog says what they COST and + * which reasoning tiers they take. A bare `models refresh` does both, because that is the intent behind the words. + * + * The two fail INDEPENDENTLY, and neither failure fails the other: an offline models.dev does not stop a provider + * refresh, and a keyless install can still refresh the catalog — which is the whole reason `--catalog` is separately + * addressable. Each is reported for what it did. + */ +async function runRefresh( + deps: ModelsCommandDeps, + axis: 'providers' | 'catalog' | undefined, +): Promise { + const wantProviders = axis !== 'catalog'; + const wantCatalog = axis !== 'providers'; + + // The two axes are independent I/O (models.dev vs the provider list APIs) and neither rejects — each captures its + // faults into its own result type — so start both and await together rather than serializing the round-trips. + const [catalogResult, report] = await Promise.all([ + wantCatalog ? deps.refreshCatalog() : Promise.resolve(undefined), + wantProviders ? deps.refreshService.refresh() : Promise.resolve(undefined), + ]); + + assertProviderRefreshInvocable(report, catalogResult); + + if (deps.global.json) { + writeRecordLines(deps.io, toRefreshRecords(report, catalogResult)); + return EXIT_CODES.success; + } + + if (report !== undefined) renderRefreshReport(deps.io, report); + if (catalogResult !== undefined) renderCatalogRefresh(deps.io, catalogResult); + return EXIT_CODES.success; +} + +/** + * A `models refresh` that reaches ZERO providers (no key at all) is an invocation fault — there was nothing it + * could fetch — UNLESS the catalog was also asked for and delivered: a keyless user running `models refresh` to + * see what things cost has been served, and calling that a failure would be a lie. No-op when providers were not + * refreshed at all (`--catalog`). + */ +function assertProviderRefreshInvocable( + report: RefreshReport | undefined, + catalogResult: CatalogRefreshResult | undefined, +): void { + if (report === undefined) return; const connected = report.providers.filter((p) => p.status !== 'skipped-no-key'); - if (connected.length === 0) { - // Nothing was connected — no key at all, so the refresh could fetch nothing. A clean, actionable exit-2 - // invocation fault (never echoes a key; names both ways to provide one). + if (connected.length === 0 && catalogResult?.status !== 'refreshed') { throw new CliError( 'invalid_invocation', 'no provider key configured — store one with `relavium provider set-key `, or set RELAVIUM__API_KEY.', ); } - if (deps.global.json) { - writeRecordLines(deps.io, report.providers.map(toRefreshJson)); - return EXIT_CODES.success; +} + +/** One `--json` record per SOURCE — the providers, then the catalog — so a script can tell which half worked. */ +function toRefreshRecords( + report: RefreshReport | undefined, + catalogResult: CatalogRefreshResult | undefined, +): unknown[] { + const records: unknown[] = report === undefined ? [] : report.providers.map(toRefreshJson); + if (catalogResult !== undefined) { + records.push({ + source: 'catalog', + status: catalogResult.status, + models: catalogResult.models, + added: catalogResult.added, + ...(catalogResult.reason === undefined ? {} : { reason: catalogResult.reason }), + }); } - renderRefreshReport(deps.io, report); - return EXIT_CODES.success; + return records; +} + +/** The catalog half of the refresh report — a failure is a NOTE, never an error: the shipped snapshot still answers. */ +function renderCatalogRefresh(io: CliIo, result: CatalogRefreshResult): void { + if (result.status === 'failed') { + io.writeErr( + `catalog: ${result.reason ?? 'refresh failed'} — keeping the shipped catalog (prices and limits are unchanged, not lost).\n`, + ); + return; + } + const added = result.added === 0 ? '' : `, ${result.added} new`; + io.writeOut(`catalog: ${result.models} models from models.dev${added}.\n`); } /** `models` (no sub) — list the cached catalog, refreshing first only when it is empty (first-run, ADR-0064 §5a). */ async function runList(deps: ModelsCommandDeps): Promise { + // `[catalog] auto_refresh` (ADR-0071 §4), DEFAULT OFF. When a user turns it on, THIS is where it fires: a command + // that is about to show prices, where a stale one would be the thing they came to look at. Never at boot, never on + // a `--help`, never behind a chat turn — a standing background fetch to a third party is what the default-OFF + // exists to refuse, and a user who opts in should still be able to see when it happens. + // + // A failure is silent here: the shipped snapshot answers, the list renders, and the user did not ask about the + // network. `models refresh --catalog` is the form that reports. + if (deps.autoRefreshCatalog === true) { + await deps.refreshCatalog(); + } let listings = deps.catalog.listAll(); let firstRunReport: RefreshReport | undefined; if (listings.length === 0) { diff --git a/apps/cli/src/commands/run.ts b/apps/cli/src/commands/run.ts index b8d02401..d5e06889 100644 --- a/apps/cli/src/commands/run.ts +++ b/apps/cli/src/commands/run.ts @@ -13,6 +13,7 @@ import { buildEngine as defaultBuildEngine, type BuildEngineOptions, } from '../engine/build-engine.js'; +import { onceEffortNotice, unpricedModelNote } from '../chat/effort-notice.js'; import { createCliHost } from '../engine/host.js'; import { connectWorkflowMcp, @@ -193,7 +194,28 @@ export async function runCommand(args: RunCommandArgs, deps: RunCommandDeps): Pr workspaceDir: deps.global.cwd, fsScopeTier: config.fsScope ?? 'sandboxed', }; - let engineOptions: BuildEngineOptions = { providers, toolEnv, ...mcpOption }; + // ADR-0071 §6: an authored `reasoning_effort` the bound model does not accept is WITHHELD — and reported. To + // STDERR, never stdout: `--json` owns stdout, and a warning line in the middle of the machine-readable stream + // would be a parse error for whatever is consuming the run. Silence was the alternative, and it is worse: the + // run succeeds, the knob does nothing, and the bill arrives at the provider's default tier. + // `onceEffortNotice`: the gate is consulted on EVERY agent-node execution, so an agent inside a `loop` would + // otherwise print the same warning on every iteration. A withheld tier is a standing condition, not an event. + const onEffortWithheld = onceEffortNotice((note: string): void => + deps.io.writeErr(`warning: ${note}\n`), + ); + // ADR-0071 §K7: a turn ran on a model we could not price, so `budget.max_cost_microcents` did not apply. The + // governor already dedups per model. STDERR, never stdout (`--json`). `budget.strict_cost_cap` blocks instead. + const onUnpriced = (model: string, capMicrocents: number): void => + deps.io.writeErr( + `warning: ${unpricedModelNote(model, capMicrocents, 'budget.strict_cost_cap')}\n`, + ); + let engineOptions: BuildEngineOptions = { + providers, + toolEnv, + onEffortWithheld, + onUnpriced, + ...mcpOption, + }; let mediaCasRoot: string | undefined; if (opened !== undefined) { const wiring = buildMediaEngineWiring(opened.db, homeDir, deps.global.cwd, config, (m) => @@ -213,6 +235,8 @@ export async function runCommand(args: RunCommandArgs, deps: RunCommandDeps): Pr engineOptions = { providers, toolEnv, + onEffortWithheld, + onUnpriced, host: createCliHost(opened.store, { media: wiring.media }), resolveMediaSurface: wiring.resolveMediaSurface, ...(wiring.mediaCostEstimate === undefined diff --git a/apps/cli/src/commands/specs-forwarding.test.ts b/apps/cli/src/commands/specs-forwarding.test.ts index 76211f41..025491b2 100644 --- a/apps/cli/src/commands/specs-forwarding.test.ts +++ b/apps/cli/src/commands/specs-forwarding.test.ts @@ -90,4 +90,32 @@ describe('commander action → executeCommand forwarding (S10)', () => { options: { provider: 'openai', input: '3', output: '9', cached: '0.1' }, }); }); + + it('models pricing forwards --clear (the ADR-0071 §5 retire path)', () => { + const { id, input } = drive([ + 'models', + 'pricing', + 'my-model', + '--provider', + 'openai', + '--clear', + ]); + expect(id).toBe('models.pricing'); + expect(input).toMatchObject({ + positionals: ['my-model'], + options: { provider: 'openai', clear: true }, + }); + }); + + it('models refresh forwards --providers (the ADR-0071 §4a axis flag)', () => { + const { id, input } = drive(['models', 'refresh', '--providers']); + expect(id).toBe('models.refresh'); + expect(input).toMatchObject({ positionals: [], options: { providers: true } }); + }); + + it('models refresh forwards --catalog (the ADR-0071 §4a axis flag)', () => { + const { id, input } = drive(['models', 'refresh', '--catalog']); + expect(id).toBe('models.refresh'); + expect(input).toMatchObject({ positionals: [], options: { catalog: true } }); + }); }); diff --git a/apps/cli/src/commands/specs.ts b/apps/cli/src/commands/specs.ts index e020efe7..8fbfeb29 100644 --- a/apps/cli/src/commands/specs.ts +++ b/apps/cli/src/commands/specs.ts @@ -455,16 +455,33 @@ function registerModels(program: Command, ctx?: CommandContext): void { .description('List the cached model catalog (refreshes on first run if empty).'); const refresh = models .command('refresh') - .description("Re-fetch each connected provider's live model list into the local cache."); + .description( + "Re-fetch what we know about models: each connected provider's live list, and the models.dev catalog.", + ) + .option('--providers', "availability only — each connected provider's live model list") + .option('--catalog', 'metadata only — prices, ceilings and reasoning tiers from models.dev'); const pricing = models .command('pricing ') .description( - 'Set a user price for a model the registry does not know (custom / new provider models).', + 'Set your own price for a model — it overrides the catalog (you hold the invoice). --clear removes it.', ) .requiredOption('--provider ', 'the provider that serves the model (must be registered)') - .requiredOption('--input ', 'input (prompt) price, USD per million tokens') - .requiredOption('--output ', 'output (completion) price, USD per million tokens') - .option('--cached ', 'cache-read price, USD per million tokens (default 0)'); + // NOT `requiredOption`: `--clear` takes none of the three price flags, and commander would reject the invocation + // before the command ever sees it. The real rule — exactly one of "set a price" or "--clear" — is enforced in + // `buildModelsPricingArgs`, which can express it; commander's required-flag check cannot. + .option( + '--input ', + 'input (prompt) price, USD per million tokens (required unless --clear)', + ) + .option( + '--output ', + 'output (completion) price, USD per million tokens (required unless --clear)', + ) + .option( + '--cached ', + "cache-read price, USD per million tokens; omitted ⇒ the catalog's cache discount, applied to your input rate", + ) + .option('--clear', "remove your price for this model — it falls back to the catalog's"); if (ctx === undefined) { models.action(() => { @@ -488,17 +505,23 @@ function registerModels(program: Command, ctx?: CommandContext): void { models.action(async () => { ctx.result.exitCode = await executeCommand('models', { positionals: [], options: {} }, ctx); }); - refresh.action(async () => { + refresh.action(async (opts: { providers?: boolean; catalog?: boolean }) => { ctx.result.exitCode = await executeCommand( 'models.refresh', - { positionals: [], options: {} }, + { positionals: [], options: { providers: opts.providers, catalog: opts.catalog } }, ctx, ); }); pricing.action( async ( model: string, - opts: { provider?: string; input?: string; output?: string; cached?: string }, + opts: { + provider?: string; + input?: string; + output?: string; + cached?: string; + clear?: boolean; + }, ) => { ctx.result.exitCode = await executeCommand( 'models.pricing', @@ -509,6 +532,7 @@ function registerModels(program: Command, ctx?: CommandContext): void { input: opts.input, output: opts.output, cached: opts.cached, + clear: opts.clear, }, }, ctx, diff --git a/apps/cli/src/config/load.ts b/apps/cli/src/config/load.ts index 1f5e6def..4da5bae1 100644 --- a/apps/cli/src/config/load.ts +++ b/apps/cli/src/config/load.ts @@ -80,13 +80,25 @@ export interface LoadedConfig { readonly homeDir: string; } +/** + * The `~/.relavium` root — WITHOUT parsing any config (ADR-0071 §4, folded from the step-7 Sonnet review). + * + * `loadResolvedConfig` returns this same value, but only after a project-dir walk and up to three TOML reads. The + * boot-time catalog seed needs the home dir and nothing else, and it runs on EVERY invocation — including bare + * Home and `--help`. Doing the full resolve there taxed the hottest path with work it throws away. One source of + * truth for "what is home", read the cheap way when that is all a caller needs. + */ +export function resolveHomeDir(options: Pick): string { + return options.home ?? homedir(); +} + /** * Discover and merge every config layer for the given cwd: the global `~/.relavium/config.toml` * (or `--config`), then the project `workspace.toml` + `project.toml` if a `.relavium/` is found * by walking up from cwd. Returns the resolved config and the discovered project dir. */ export function loadResolvedConfig(options: LoadConfigOptions): LoadedConfig { - const home = options.home ?? homedir(); + const home = resolveHomeDir(options); const globalFile = options.configPath ?? join(globalConfigDir(home), 'config.toml'); const global = loadConfigFile(globalFile, GlobalConfigSchema); diff --git a/apps/cli/src/config/resolve.test.ts b/apps/cli/src/config/resolve.test.ts index 60ff3732..33c574f0 100644 --- a/apps/cli/src/config/resolve.test.ts +++ b/apps/cli/src/config/resolve.test.ts @@ -7,6 +7,8 @@ describe('resolveConfig', () => { it('returns empty defaults with no layers', () => { expect(resolveConfig({})).toEqual({ updateChannel: undefined, + // ADR-0071 §4: default OFF. A local-first tool does not contact a third party unless asked. + catalogAutoRefresh: false, defaultModel: undefined, fsScope: undefined, maxTokensEstimate: undefined, @@ -20,6 +22,7 @@ describe('resolveConfig', () => { maxMessages: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, // ADR-0071 §K7: default off — the cost cap degrades to allow on an unpriced model allowedCommands: undefined, allowedCommandGlobs: undefined, }, @@ -94,6 +97,42 @@ describe('resolveConfig', () => { ).toBeUndefined(); }); + it('resolves default_model + default_provider as a COUPLED pair — the provider follows the model layer (ADR-0059)', () => { + // Both come from the SAME layer: a layer that sets the model owns (or omits) its provider. + const projectFull: ProjectConfig = { + chat: { default_model: 'gpt-4o', default_provider: 'openai' }, + }; + const rp = resolveConfig({ project: projectFull }).chat; + expect(rp.defaultModel).toBe('gpt-4o'); + expect(rp.defaultProvider).toBe('openai'); + + // THE REGRESSION GUARD (Bug-3 fold): a project pins ONLY the model (no provider — the common case, since the + // field is brand new and a claude id was always inferable), while a global [preferences] carries a stale + // provider. The provider must NOT leak across the layer boundary — it stays undefined so buildDefaultChatAgent + // falls back to inference (which correctly places the claude id), instead of binding an OpenAI agent → 404. + const global = { + preferences: { default_model: 'chat-latest', default_provider: 'openai' as const }, + }; + const projectModelOnly: ProjectConfig = { chat: { default_model: 'claude-sonnet-4-6' } }; + const coupled = resolveConfig({ global, project: projectModelOnly }).chat; + expect(coupled.defaultModel).toBe('claude-sonnet-4-6'); // project wins the model + expect(coupled.defaultProvider).toBeUndefined(); // NOT 'openai' — the provider does not leak across layers + + // No [chat] layer sets a model ⇒ the pair comes from global [preferences], TOGETHER. + const g = resolveConfig({ global }).chat; + expect(g.defaultModel).toBe('chat-latest'); + expect(g.defaultProvider).toBe('openai'); + + // A STRAY global provider with NO global model must NOT pair with the built-in DEFAULT model — it stays + // undefined (inference), closing the global-model-absent boundary of the same coupling. + const stray = resolveConfig({ global: { preferences: { default_provider: 'openai' } } }).chat; + expect(stray.defaultModel).toBeUndefined(); + expect(stray.defaultProvider).toBeUndefined(); + + // Absent everywhere ⇒ undefined (inference from the id). + expect(resolveConfig({}).chat.defaultProvider).toBeUndefined(); + }); + it('resolves [chat].auto_compact + compact_threshold (ADR-0062) last-writer-wins, per field', () => { const workspace: ProjectConfig = { chat: { auto_compact: false, compact_threshold: 0.7 } }; const project: ProjectConfig = { chat: { compact_threshold: 0.9 } }; diff --git a/apps/cli/src/config/resolve.ts b/apps/cli/src/config/resolve.ts index 762a82a8..cf4eac18 100644 --- a/apps/cli/src/config/resolve.ts +++ b/apps/cli/src/config/resolve.ts @@ -21,6 +21,10 @@ type ChatConfig = NonNullable; export interface ResolvedChatConfig { /** `[chat].default_model` — the model a chat session binds when its agent names none. */ readonly defaultModel: ChatConfig['default_model']; + /** `[chat].default_provider` — the provider serving `default_model`, persisted at pick time (ADR-0059) so the + * built-in default chat agent skips id inference. Resolves like `default_model` (project → workspace → global + * `[preferences].default_provider`). Absent ⇒ inference from the id. */ + readonly defaultProvider: ChatConfig['default_provider']; /** `[chat].fs_scope` — the filesystem permission tier for chat tool dispatch (same tiers as workflows). */ readonly fsScope: ChatConfig['fs_scope']; /** `[chat].max_turns` — the hard session turn cap → `SessionDeps.maxTurns` (absent ⇒ engine default 50; @@ -41,6 +45,8 @@ export interface ResolvedChatConfig { /** `[chat].on_exceed` — action when the cost cap trips (in an interactive REPL, `pause_for_approval` * degrades to a loud turn-end since the prompt itself is the approval gate). */ readonly onExceed: ChatConfig['on_exceed']; + /** `[chat] strict_cost_cap` (ADR-0071 §K7) — refuse a turn on a model we cannot price. Default false. */ + readonly strictCostCap: boolean; /** `[chat].allowed_commands` — the `!`-shell exact-match allowlist (→ engine `allowedCommands`; ADR-0061). * Absent/empty ⇒ `!`-shell disabled (the `empty ⇒ disabled` symmetry; no chat-specific relaxation). */ readonly allowedCommands: ChatConfig['allowed_commands']; @@ -53,6 +59,14 @@ export interface ResolvedChatConfig { export interface ResolvedConfig { readonly updateChannel: GlobalConfig['update_channel']; + /** + * `[catalog] auto_refresh` (ADR-0071 §4) — refresh the models.dev catalog automatically. **DEFAULT `false`.** + * + * A local-first tool that contacts a third party by default violates its own spirit even when the payload is + * innocuous, so the shipped snapshot answers everything offline and the standing egress is opt-in. An explicit + * `relavium models refresh` fetches regardless: a command the user typed IS consent. + */ + readonly catalogAutoRefresh: boolean; readonly defaultModel: string | undefined; readonly fsScope: FsScope; readonly maxTokensEstimate: number | undefined; @@ -97,6 +111,7 @@ export function resolveConfig(layers: ConfigLayers): ResolvedConfig { const { global, workspace, project } = layers; return { updateChannel: global?.update_channel, + catalogAutoRefresh: global?.catalog?.auto_refresh === true, defaultModel: project?.defaults?.model ?? workspace?.defaults?.model ?? global?.preferences?.default_model, fsScope: project?.defaults?.fs_scope ?? workspace?.defaults?.fs_scope, @@ -151,8 +166,33 @@ function resolveChat( // (ADR-0061). Only when the project sets NEITHER do both fall through to the workspace, per field. const projectSetsAllowlist = p?.allowed_commands !== undefined || p?.allowed_command_globs !== undefined; + // `default_model` + `default_provider` are a COUPLED pair — the provider must SERVE the model (ADR-0059). Resolve + // BOTH from the FIRST layer that sets a model, so a lower layer's stale provider never pairs with a higher layer's + // model: independent `??` fallbacks would let a project's claude `default_model` (no provider — a claude id was + // always inferable, so no pre-existing config sets one) inherit a global openai `default_provider` written by the + // wizard, and `buildDefaultChatAgent` would then bind an OpenAI agent over a Claude id (a 404 the pre-change id + // inference avoided). The layer that sets the model owns its provider (absent ⇒ inference); else global + // `[preferences]`. This is the same coupling `allowed_commands`/`allowed_command_globs` get just above. + let defaultModel: ChatConfig['default_model']; + let defaultProvider: ChatConfig['default_provider']; + if (p?.default_model !== undefined) { + defaultModel = p.default_model; + defaultProvider = p.default_provider; + } else if (w?.default_model !== undefined) { + defaultModel = w.default_model; + defaultProvider = w.default_provider; + } else if (global?.preferences?.default_model !== undefined) { + defaultModel = global.preferences.default_model; + defaultProvider = global.preferences.default_provider; + } else { + // No layer sets a MODEL — so no layer's provider is coupled to one. A stray global `default_provider` (a + // hand-edited config; the picker/wizard always write the pair) must NOT pair with the built-in DEFAULT model. + defaultModel = undefined; + defaultProvider = undefined; + } return { - defaultModel: p?.default_model ?? w?.default_model ?? global?.preferences?.default_model, + defaultModel, + defaultProvider, fsScope: p?.fs_scope ?? w?.fs_scope, maxTurns: p?.max_turns ?? w?.max_turns, maxMessages: p?.max_messages ?? w?.max_messages, @@ -160,6 +200,7 @@ function resolveChat( compactThreshold: p?.compact_threshold ?? w?.compact_threshold, maxCostMicrocents: p?.max_cost_microcents ?? w?.max_cost_microcents, onExceed: p?.on_exceed ?? w?.on_exceed, + strictCostCap: (p?.strict_cost_cap ?? w?.strict_cost_cap) === true, allowedCommands: projectSetsAllowlist ? p?.allowed_commands : w?.allowed_commands, allowedCommandGlobs: projectSetsAllowlist ? p?.allowed_command_globs : w?.allowed_command_globs, reasoningEffort: diff --git a/apps/cli/src/config/write.test.ts b/apps/cli/src/config/write.test.ts index e59688c1..d096f5d5 100644 --- a/apps/cli/src/config/write.test.ts +++ b/apps/cli/src/config/write.test.ts @@ -42,6 +42,14 @@ describe('writeGlobalDefaultModel', () => { expect(readBack(home)).toEqual({ preferences: { default_model: 'claude-sonnet-4-6' } }); }); + it('persists [preferences].default_provider alongside default_model, round-tripping both (ADR-0059)', () => { + // The provider is persisted at pick time so the next chat over an id the prefix cannot place still resolves. + writeGlobalPreferences({ defaultModel: 'chat-latest', defaultProvider: 'openai' }, home); + expect(readBack(home)).toEqual({ + preferences: { default_model: 'chat-latest', default_provider: 'openai' }, + }); + }); + it('preserves every other config key (update_channel, theme, mcp_servers) — merges only default_model', () => { // Seed an existing, valid global config carrying unrelated keys the write must NOT drop. mkdirSync(globalDir(home), { recursive: true }); diff --git a/apps/cli/src/config/write.ts b/apps/cli/src/config/write.ts index 89199ad8..b928187f 100644 --- a/apps/cli/src/config/write.ts +++ b/apps/cli/src/config/write.ts @@ -11,7 +11,12 @@ import { import { homedir } from 'node:os'; import { dirname, join } from 'node:path'; -import { GlobalConfigSchema, type GlobalConfig, type ReasoningEffort } from '@relavium/shared'; +import { + GlobalConfigSchema, + type GlobalConfig, + type LlmProviderId, + type ReasoningEffort, +} from '@relavium/shared'; import { parse as parseToml, stringify as stringifyToml } from 'smol-toml'; import type { ZodError } from 'zod'; @@ -29,7 +34,8 @@ import { ensureGlobalConfigDir, globalConfigDir } from './paths.js'; * Four load-bearing guarantees (the ADR-0063 contract; a security-reviewed surface): * 1. **Secret-free by construction.** The surface is a **typed setter** ({@link writeGlobalPreferences}, of which * {@link writeGlobalDefaultModel} is a thin single-key wrapper) — never a generic `writeKey(k, v)` — so it can - * only ever set the two non-secret `[preferences]` keys `default_model` / `reasoning_effort`. There is no API-key + * only ever set the three non-secret `[preferences]` keys `default_model` / `default_provider` / + * `reasoning_effort`. There is no API-key * field in the schema to write to (keys live only in the OS keychain, ADR-0006). And a schema-validation * failure on the write path is reported through the **same value-free formatter** the loader uses * ({@link formatZodError}) — never a raw `ZodError`, whose `.message` embeds the received value for several @@ -64,6 +70,10 @@ import { ensureGlobalConfigDir, globalConfigDir } from './paths.js'; * field absent from the object is left UNCHANGED (a partial merge), so a model-only write never clears the effort. */ export interface GlobalPreferenceWrite { readonly defaultModel?: string; + /** The provider serving {@link GlobalPreferenceWrite.defaultModel}, persisted so the next chat skips id inference + * (ADR-0059). The picker + wizard always write it TOGETHER with `defaultModel`; on its own it is still a real + * partial write (it updates just the provider, leaving the model unchanged), not a no-op. */ + readonly defaultProvider?: LlmProviderId; readonly reasoningEffort?: ReasoningEffort; } @@ -88,7 +98,13 @@ export function writeGlobalPreferences( ): void { // An all-absent write is a no-op: touch nothing rather than emit an empty `[preferences]` table where none // existed. Unreachable from the current callers (each passes ≥1 key), but keeps the typed setter footgun-free. - if (prefs.defaultModel === undefined && prefs.reasoningEffort === undefined) return; + if ( + prefs.defaultModel === undefined && + prefs.defaultProvider === undefined && + prefs.reasoningEffort === undefined + ) { + return; + } let target: string; let dir: string; if (targetPath === undefined) { @@ -119,6 +135,7 @@ export function writeGlobalPreferences( preferences: { ...existing.preferences, ...(prefs.defaultModel === undefined ? {} : { default_model: prefs.defaultModel }), + ...(prefs.defaultProvider === undefined ? {} : { default_provider: prefs.defaultProvider }), ...(prefs.reasoningEffort === undefined ? {} : { reasoning_effort: prefs.reasoningEffort }), }, }; diff --git a/apps/cli/src/engine/build-engine.ts b/apps/cli/src/engine/build-engine.ts index f537d17d..457e01b1 100644 --- a/apps/cli/src/engine/build-engine.ts +++ b/apps/cli/src/engine/build-engine.ts @@ -5,15 +5,17 @@ import { createStandardNodeExecutor, createToolRegistry, type AgentRunnerDeps, + type EffortGateResult, type ExecutionHost, type FsScopeTier, type McpCapability, type ToolDef, type ToolHost, } from '@relavium/core'; -import { modelSupportsReasoning, type PricingOverlay } from '@relavium/llm'; +import { effortTiersFor, type PricingOverlay } from '@relavium/llm'; import type { MediaCostEstimate, MediaSurface } from '@relavium/shared'; +import { effortWithheldNote, reasoningWithheldByCapFor } from '../chat/effort-notice.js'; import { createCliHost } from './host.js'; import { createProviderResolver, type ProviderResolver } from './providers.js'; import { assembleToolEnv } from './tool-host/assemble.js'; @@ -55,10 +57,23 @@ export interface BuildEngineOptions { * projects from the `model_catalog` `source='user'` rows. Threaded into BOTH the workflow PRE-EGRESS governor * (so a user-priced model is enforced by `budget.max_cost_microcents`) AND the agent node's realized * `AgentRunnerDeps.resolvePrice` (so the same model's realized cost is tracked, not thrown as `UnknownModel`). - * Static `MODEL_PRICING` still wins for a known id. Absent ⇒ an unknown model degrades cost governance to - * `allow` loudly, unchanged. + * The USER outranks the catalog (ADR-0071 §1). Absent ⇒ an unknown model degrades cost governance + * to `allow` loudly, unchanged. */ readonly resolvePrice?: PricingOverlay; + /** + * Sink for a WITHHELD reasoning tier (ADR-0071 §6) — an agent authored `reasoning_effort: ` that the bound + * model does not accept, so the field is not sent. The gate replaced a loud provider 400 with a quiet no-op, and + * on an authored workflow that no-op is the dangerous one: the run succeeds, the knob does nothing, and the bill + * lands at the provider's default tier. `run.ts` wires this to stderr — never stdout, which `--json` owns. + * Absent ⇒ silent (the tier is still withheld). + */ + readonly onEffortWithheld?: (note: string) => void; + /** + * Sink for an UNPRICED model turn (ADR-0071 §K7) — the cost cap could not apply. `run.ts` wires it to stderr, + * never stdout (`--json`). The governor already dedups per model. Absent ⇒ silent. + */ + readonly onUnpriced?: (model: string, capMicrocents: number) => void; } /** @@ -108,9 +123,22 @@ export async function buildEngine(options: BuildEngineOptions = {}): Promise { + options.onEffortWithheld?.(effortWithheldNote(result, model)); + }, + }), registry, tools, sleep: (ms) => new Promise((resolveSleep) => setTimeout(resolveSleep, ms)), @@ -136,11 +164,22 @@ export async function buildEngine(options: BuildEngineOptions = {}): Promise): string { + return JSON.stringify({ + openai: { + models: Object.fromEntries( + Object.entries(models).map(([id, m]) => [ + id, + { + id, + name: id, + limit: { context: 100_000, output: 10_000 }, + cost: { input: 1, output: 2 }, + ...(m as object), + }, + ]), + ), + }, + }); +} + +function respond(body: string, init: ResponseInit = {}): typeof globalThis.fetch { + return () => Promise.resolve(new Response(body, { status: 200, ...init })); +} + +beforeEach(() => { + home = mkdtempSync(join(tmpdir(), 'relavium-catalog-')); +}); + +afterEach(() => { + clearCatalogRefresh(); // module state — a leaked refresh would poison every later test in the process + rmSync(home, { recursive: true, force: true }); +}); + +describe('refreshCatalog — additive only, and the shipped snapshot is the floor', () => { + it('ADDS a model the snapshot never carried — the reason to run it at all', async () => { + expect(catalogModel('gpt-6-imaginary')).toBeUndefined(); // the premise + + const result = await refreshCatalog({ + homeDir: home, + fetch: respond(payload({ 'gpt-6-imaginary': {} })), + }); + + expect(result.status).toBe('refreshed'); + expect(result.added).toBe(1); + expect(catalogModel('gpt-6-imaginary')?.inputPerMtokMicrocents).toBe(100_000_000); // $1/MTok + }); + + it('an UNREACHABLE models.dev is a NO-OP — every shipped model is still priced', async () => { + const before = catalogModel('gpt-5.5'); + const result = await refreshCatalog({ + homeDir: home, + fetch: () => Promise.reject(new Error('ENOTFOUND')), + }); + + expect(result.status).toBe('failed'); + expect(result.reason).toBe('models.dev unreachable'); + expect(catalogModel('gpt-5.5')).toEqual(before); // unchanged — not blanked, not downgraded + expect(catalogModel('gpt-5.5')?.outputPerMtokMicrocents).toBeGreaterThan(0); + }); + + it("a MALFORMED payload is a no-op — a third party's bad deploy cannot unprice our models", async () => { + const result = await refreshCatalog({ homeDir: home, fetch: respond('{"openai":{"models":') }); + expect(result.status).toBe('failed'); + expect(catalogModel('gpt-5.5')?.outputPerMtokMicrocents).toBeGreaterThan(0); + }); + + it('a 5xx is a no-op, and says which one', async () => { + const result = await refreshCatalog({ + homeDir: home, + fetch: () => Promise.resolve(new Response('nope', { status: 503 })), + }); + expect(result.status).toBe('failed'); + expect(result.reason).toContain('503'); + }); + + it('the fetch→install path leaves a SHIPPED model untouched (§9 — the floor, end to end)', async () => { + // The exhaustive floor lives in `@relavium/llm`'s `lookup.test.ts` (source-direct, so its break-verify is + // reliable — this file imports across the package boundary, from `dist`). This is the integration leg: a real + // `refreshCatalog` over a payload that would DOWNGRADE `gpt-5.5` must leave its human-verified price standing. + const shipped = CATALOG_SNAPSHOT['gpt-5.5']; + expect(shipped?.outputPerMtokMicrocents).toBeGreaterThan(0); + + await refreshCatalog({ + homeDir: home, + fetch: respond( + JSON.stringify({ + openai: { + models: { + 'gpt-5.5': { + id: 'gpt-5.5', + name: 'GPT-5.5', + limit: { context: 8_000, output: 1_000 }, + cost: { input: 0, output: 0.00000001 }, // ~free — the cap-defeating price + }, + }, + }, + }), + ), + }); + + expect(catalogModel('gpt-5.5')?.outputPerMtokMicrocents).toBe(shipped?.outputPerMtokMicrocents); + }); + + it('reports `added` as what the FLOOR admitted, not what the payload offered', async () => { + // A models.dev payload can carry a NEW priced model and a NEW unpriced one (a preview row at `output: 0` passes + // Zod). Only the priced one is admitted — so `added` must be 1, not 2. Counting the payload's new ids by hand, + // host-side, drifted from what actually installed; the count now comes from `installCatalogRefresh` itself. + const result = await refreshCatalog({ + homeDir: home, + fetch: respond( + payload({ 'gpt-7-priced': {}, 'gpt-7-free': { cost: { input: 0, output: 0 } } }), + ), + }); + expect(result.added).toBe(1); + expect(catalogModel('gpt-7-priced')).toBeDefined(); + expect(catalogModel('gpt-7-free')).toBeUndefined(); + }); + + it('ABORTS an over-cap body instead of buffering it whole (ADR-0071 §8)', async () => { + // The cap must REFUSE an endless body, not read it all and then measure. A ReadableStream that would yield far + // more than 16 MB is cut off the moment the running count crosses the ceiling — the reader is never drained. + let chunksRead = 0; + const oversized = new ReadableStream({ + pull(ctrl) { + chunksRead += 1; + ctrl.enqueue(new Uint8Array(8 * 1024 * 1024)); // 8 MB a chunk — the cap is 16 MB + }, + }); + const result = await refreshCatalog({ + homeDir: home, + fetch: () => Promise.resolve(new Response(oversized, { status: 200 })), + }); + expect(result.status).toBe('failed'); + expect(result.reason).toContain('too large'); + expect(chunksRead).toBeLessThan(5); // aborted early — NOT the thousands an unbounded read would pull + }); +}); + +describe('refreshCatalog — one destination, and it stays there', () => { + it('REFUSES a redirect off models.dev — the destination IS the posture', async () => { + const result = await refreshCatalog({ + homeDir: home, + fetch: () => + Promise.resolve( + Object.defineProperty(new Response('{}', { status: 200 }), 'url', { + value: 'https://evil.example.com/api.json', + }), + ), + }); + expect(result.status).toBe('failed'); + expect(result.reason).toContain('off-host'); + }); + + it('a MANUAL-caught redirect (opaque, status 0) is named "off-host", not the misleading "returned 0"', async () => { + const result = await refreshCatalog({ + homeDir: home, + fetch: () => { + // `redirect: 'manual'` surfaces a caught 3xx as an OPAQUE response: type 'opaqueredirect', status 0. + const r = new Response('{}', { status: 200 }); + Object.defineProperty(r, 'status', { value: 0 }); + Object.defineProperty(r, 'type', { value: 'opaqueredirect' }); + return Promise.resolve(r); + }, + }); + expect(result.status).toBe('failed'); + expect(result.reason).toBe('models.dev redirected off-host'); // NOT "models.dev returned 0" + }); + + it('sends NO credentials and NO user data — an unauthenticated GET of a public file', async () => { + let sent: RequestInit | undefined; + let url: unknown; + await refreshCatalog({ + homeDir: home, + fetch: (u: unknown, init?: RequestInit) => { + url = u; + sent = init; + return Promise.resolve(new Response(payload({}), { status: 200 })); + }, + }); + + expect(url).toBe('https://models.dev/api.json'); // a compile-time constant — not user- and not model-supplied + expect(sent?.redirect).toBe('manual'); + expect(sent?.credentials).toBeUndefined(); + const headers = sent?.headers as Record | undefined; + expect(Object.keys(headers ?? {})).toEqual(['accept']); // no authorization, no cookie, no telemetry + }); +}); + +describe('the cache — a refresh survives the process that ran it', () => { + it('writes what it installed, and loads it back', async () => { + await refreshCatalog({ homeDir: home, fetch: respond(payload({ 'gpt-6-imaginary': {} })) }); + // The RAW payload, not the normalized rows: caching the normalized ones would freeze today's normalizer into the + // file, so a release that FIXED a normalization bug would go on serving the old wrong rows to everyone who had + // ever refreshed. Re-normalizing on load means an upgrade repairs the cache for free. + const cached: unknown = JSON.parse(readFileSync(catalogCachePath(home), 'utf8')); + expect(cached).toHaveProperty(['openai', 'models', 'gpt-6-imaginary']); + + clearCatalogRefresh(); + expect(catalogModel('gpt-6-imaginary')).toBeUndefined(); // gone from this process… + + loadCachedCatalog(home); + expect(catalogModel('gpt-6-imaginary')).toBeDefined(); // …and back, with no fetch + }); + + it('a MISSING or CORRUPT cache is silent — the shipped snapshot is the floor, not a fallback', () => { + expect(loadCachedCatalog(join(home, 'nowhere'))).toBeUndefined(); + expect(catalogModel('gpt-5.5')?.outputPerMtokMicrocents).toBeGreaterThan(0); + }); +}); diff --git a/apps/cli/src/engine/catalog-refresh.ts b/apps/cli/src/engine/catalog-refresh.ts new file mode 100644 index 00000000..bcf2987e --- /dev/null +++ b/apps/cli/src/engine/catalog-refresh.ts @@ -0,0 +1,282 @@ +import { mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; + +import { + CATALOG_PROVIDER_KEYS, + ModelsDevPayloadSchema, + catalogModelIds, + installCatalogRefresh, + normalizeCatalog, + type CatalogModel, +} from '@relavium/llm'; + +/** + * The models.dev catalog refresh (ADR-0071 §4) — the HOST half of the seam. + * + * `@relavium/llm` does no I/O: it takes a catalog as plain data and answers questions about it. Fetching a URL and + * writing a file are platform work, so they live here, exactly where the provider refresh already lives. The pure + * merge stays pure. + * + * **Default OFF.** `[catalog] auto_refresh` is `false`, and this module runs only when the user asks — either by + * typing `relavium models refresh`, which IS consent, or by turning the key on themselves. A local-first tool that + * contacts a third party by default violates its own spirit even when the payload is innocuous (CLAUDE.md rule 6), + * so the shipped snapshot answers everything, offline, until someone decides otherwise. + * + * **Additive only.** A refresh may add models and enrich thin ones; it may never leave a model less described than + * it shipped. A failed, unreachable, or malformed refresh is a NO-OP — the snapshot stands. The cost cap is a safety + * control, and it does not get to lapse because a third party had a bad deploy. + */ + +/** The one destination. A compile-time constant, not user- and not model-supplied — see ADR-0071 §8. */ +const SOURCE_URL = 'https://models.dev/api.json'; +const SOURCE_HOST = 'models.dev'; + +/** models.dev's payload is ~1 MB. A generous ceiling — and a REAL one: `readCapped` aborts the transfer the + * moment a body exceeds it, rather than buffering an endless one and checking its length too late. */ +const MAX_BYTES = 16 * 1024 * 1024; +const TIMEOUT_MS = 20_000; + +/** Where a refreshed catalog is cached, so the next process starts current without re-fetching. */ +export function catalogCachePath(homeDir: string): string { + return join(homeDir, 'catalog', 'models-dev.json'); +} + +/** What a `--catalog` refresh did, for the report and for `--json`. */ +export interface CatalogRefreshResult { + readonly status: 'refreshed' | 'failed'; + /** Models the refreshed catalog describes. `0` on a failure (the snapshot still answers). */ + readonly models: number; + /** Models the shipped snapshot did NOT carry — the reason to run this at all. */ + readonly added: number; + /** A short, actionable reason. Never a raw fetch error (it can carry a URL); never a key (there is none to leak). */ + readonly reason?: string; +} + +export interface CatalogRefreshDeps { + readonly homeDir: string; + /** Injectable for tests — the real one is global `fetch`. */ + readonly fetch?: typeof globalThis.fetch; +} + +/** + * Fetch models.dev, validate it, install it, and cache it to disk. + * + * Every failure mode lands on the same answer — **the shipped snapshot** — so this can never make the product worse + * than not running it. It returns the outcome rather than throwing, because "the network was down" is not a reason + * to fail a command the user ran to *look at their models*. + */ +export async function refreshCatalog(deps: CatalogRefreshDeps): Promise { + const fetched = await fetchModelsDevPayload(deps); + if (typeof fetched !== 'string') return fetched; // a classified network/HTTP failure — the snapshot answers + + // INSTALL is a SEPARATE failure domain from the network. A Zod / parse / normalization throw here means models.dev + // was REACHABLE but sent something we cannot use — reporting that as "unreachable" or "timed out" would point the + // user at the wrong problem. Its own generic reason, and NEVER the raw error text (a Zod message can echo the very + // payload fields we refuse to surface). + try { + return install(fetched, deps.homeDir); + } catch { + return { + status: 'failed', + models: 0, + added: 0, + reason: 'models.dev sent a catalog we could not read', + }; + } +} + +/** + * Fetch and read the models.dev body, CLASSIFIED. Returns the raw text on success, or a `failed` result naming the + * network / HTTP fault (timeout, unreachable, non-2xx, off-host redirect, over-cap). Never surfaces a raw upstream + * error string. Kept separate from {@link install} so a malformed-payload fault is not mistaken for a network one. + */ +async function fetchModelsDevPayload( + deps: CatalogRefreshDeps, +): Promise { + const doFetch = deps.fetch ?? globalThis.fetch; + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), TIMEOUT_MS); + try { + const response = await doFetch(SOURCE_URL, { + signal: controller.signal, + // NO credentials, NO cookies, NO user data — an unauthenticated GET of a public file (ADR-0071 §8). + headers: { accept: 'application/json' }, + // A redirect OFF models.dev is an error, not a hop: the destination is the point of the posture. `manual` + // surfaces the 3xx as a non-ok response rather than quietly following it somewhere else. + redirect: 'manual', + }); + // A `redirect: 'manual'` that CAUGHT a redirect yields an OPAQUE response (`type: 'opaqueredirect'`, `status: 0`) + // — that is the off-host redirect the ADR-0071 §8 posture forbids, not a generic non-2xx. Name it as such + // rather than the misleading "models.dev returned 0" the bare status check below would print. + if (response.type === 'opaqueredirect' || (response.status >= 300 && response.status < 400)) { + return { status: 'failed', models: 0, added: 0, reason: 'models.dev redirected off-host' }; + } + if (!response.ok) { + return { + status: 'failed', + models: 0, + added: 0, + reason: `models.dev returned ${response.status}`, + }; + } + // The response URL must still be models.dev — belt to the `redirect: 'manual'` braces, in case a fetch + // implementation resolves one anyway. + if (response.url !== '' && new URL(response.url).hostname !== SOURCE_HOST) { + return { status: 'failed', models: 0, added: 0, reason: 'models.dev redirected off-host' }; + } + const text = await readCapped(response, controller); + if (text === undefined) { + return { status: 'failed', models: 0, added: 0, reason: 'models.dev payload is too large' }; + } + return text; + } catch (err) { + // A timeout, a DNS failure, an offline laptop. The snapshot answers; the user is told, and nothing breaks. + return { + status: 'failed', + models: 0, + added: 0, + reason: + err instanceof Error && err.name === 'AbortError' + ? 'models.dev timed out' + : 'models.dev unreachable', + }; + } finally { + clearTimeout(timer); + } +} + +/** + * Read the body, ABORTING the moment it exceeds the cap — not after buffering the whole thing (ADR-0071 §8). + * + * `response.text()` reads everything into memory first and only then checks the length, so on a fast link a + * misbehaving host could deliver many times the cap inside the timeout before the check ever fired. `MAX_BYTES` is + * meant to REFUSE an endless body, and a check after the buffer does not refuse anything. Streaming the reader and + * aborting on overflow makes the ceiling real. `undefined` ⇒ over the cap (the caller degrades to the snapshot). + * + * A body with no reader (a test `Response`, an empty body) falls back to `text()` — bounded there by the cap check + * on the fully-read string, which is fine because such a body was never the streaming threat. + */ +async function readCapped( + response: Response, + controller: AbortController, +): Promise { + const body = response.body; + if (body === null) { + const text = await response.text(); + return text.length > MAX_BYTES ? undefined : text; + } + const decoder = new TextDecoder(); + let out = ''; + let bytes = 0; + const reader = body.getReader(); + for (;;) { + // The DOM lib types `ReadableStream` as `ReadableStream` here, so annotate the read result as `unknown` + // rather than assert the whole stream's generic (`body as ReadableStream`) — then NARROW the chunk + // at the use site: a byte stream yields `Uint8Array`, and `instanceof` keeps `.byteLength`/`decode` type-safe. + const chunk: { readonly done: boolean; readonly value: unknown } = await reader.read(); + if (chunk.done) break; + if (!(chunk.value instanceof Uint8Array)) continue; + bytes += chunk.value.byteLength; + if (bytes > MAX_BYTES) { + controller.abort(); // stop the transfer NOW — do not read another chunk + return undefined; + } + out += decoder.decode(chunk.value, { stream: true }); + } + return out + decoder.decode(); +} + +/** Load a previously-cached refresh, if there is one. Silent on absence or corruption — the snapshot answers. */ +export function loadCachedCatalog(homeDir: string): CatalogRefreshResult | undefined { + try { + // The SAME parse the fetch path runs, over the SAME raw payload — one normalizer, one shape, no second code path + // that could drift. A tampered or truncated cache fails Zod here and lands on the snapshot, exactly like a bad + // fetch does. + const models = parse(readFileSync(catalogCachePath(homeDir), 'utf8')); + const added = installCatalogRefresh(models); + return { status: 'refreshed', models: catalogModelIds().length, added }; + } catch { + // No cache, or a cache we cannot read. Not an error: the shipped snapshot is the floor, not a fallback. + return undefined; + } +} + +/** Parse + normalize a models.dev payload into Relavium's own types. Throws on a payload we cannot use. */ +function parse(text: string): Record { + // Zod at the boundary, never a cast: the payload is a third party's JSON, and it now feeds a safety control. + // `normalizeCatalog` is the SAME function the offline snapshot generator uses, so a refreshed row and a shipped row + // cannot end up shaped differently — the discipline ADR-0064 §1 already applies to a provider's `ModelListing`. + const payload = ModelsDevPayloadSchema.parse(JSON.parse(text)); + const { catalog } = normalizeCatalog(payload); + return catalog; +} + +/** + * Normalize, install, and cache — the RAW payload, not the normalized one. + * + * The cache holds exactly what models.dev sent. Caching the normalized rows instead would freeze TODAY's normalizer + * into the file: a Relavium release that fixes a normalization bug (a mis-read reasoning shape, a dropped cache + * rate) would go on serving the old, wrong rows to every user who had ever refreshed, and nothing would tell them. + * Re-normalizing on load means an upgrade repairs the cache for free. + * + * A cache-WRITE failure is not a refresh failure: the models are already live in this process. It just will not + * survive it. + */ +function install(text: string, homeDir: string): CatalogRefreshResult { + const models = parse(text); // throws on a payload we cannot use — the caller turns that into a no-op + const added = installCatalogRefresh(models); // the count the FLOOR admitted, not the count the payload offered + writeCache(text, homeDir); + return { + // TOTAL describable after the install — the shipped snapshot plus whatever the refresh added — and how many of + // those were new. `models: 200, added: 120` reads as "200 models, 120 the snapshot didn't carry". + status: 'refreshed', + models: catalogModelIds().length, + added, + }; +} + +/** + * Cache the payload, filtered to the FOUR providers we have an adapter for. + * + * models.dev ships 166 providers and ~5,600 models — the 2–3 MB blob ADR-0071 §3 explicitly declined to vendor. We + * consume four of them, and `seedCatalog` re-reads + re-parses this file on EVERY invocation, including `--help`. + * Caching the whole thing taxes the cheapest, hottest commands with a parse we throw almost all of away. Filtering + * to the mapped providers keeps the file small AND keeps it RAW-per-provider, so a future release that fixes a + * normalization bug still repairs the cache on the next load. + * + * A cache-WRITE failure is not a refresh failure: the models are already live in this process; the cache just will + * not outlive it. Written temp-then-rename so a concurrent reader never sees a torn file. + */ +function writeCache(text: string, homeDir: string): void { + const path = catalogCachePath(homeDir); + try { + const parsed: unknown = JSON.parse(text); + const filtered: Record = {}; + if (parsed !== null && typeof parsed === 'object') { + for (const key of Object.values(CATALOG_PROVIDER_KEYS)) { + const provider = (parsed as Record)[key]; + if (provider !== undefined) filtered[key] = provider; + } + } + mkdirSync(dirname(path), { recursive: true }); + // temp-then-rename: rename is atomic on the same filesystem, so a reader either sees the OLD cache or the NEW + // one, never a half-written body. `${pid}` keeps two concurrent refreshes from racing on the same temp name. + const tmp = `${path}.${process.pid}.tmp`; + try { + writeFileSync(tmp, JSON.stringify(filtered), 'utf8'); + renameSync(tmp, path); + } catch (err) { + // A failed write/rename must not leave the pid-suffixed temp behind to accumulate. Best-effort unlink, then + // re-throw into the outer catch (the refresh HELD regardless). A hard process kill between write and rename + // can still orphan it — unavoidable without a startup sweep — but the common failure path is now clean. + try { + unlinkSync(tmp); + } catch { + // the temp may never have been created (a write fault) — nothing to remove + } + throw err; + } + } catch { + // A read-only home, a full disk, an unparseable body we somehow got past `parse` — the refresh HELD regardless. + } +} diff --git a/apps/cli/src/engine/media-wiring.test.ts b/apps/cli/src/engine/media-wiring.test.ts index e13c85ec..dc890544 100644 --- a/apps/cli/src/engine/media-wiring.test.ts +++ b/apps/cli/src/engine/media-wiring.test.ts @@ -18,6 +18,7 @@ import { buildMediaEngineWiring } from './media-wiring.js'; const EMPTY_CONFIG: ResolvedConfig = { updateChannel: undefined, + catalogAutoRefresh: false, defaultModel: undefined, fsScope: undefined, maxTokensEstimate: undefined, @@ -25,6 +26,7 @@ const EMPTY_CONFIG: ResolvedConfig = { mediaGcGraceMs: undefined, chat: { defaultModel: undefined, + defaultProvider: undefined, fsScope: undefined, maxTurns: undefined, maxMessages: undefined, @@ -32,6 +34,7 @@ const EMPTY_CONFIG: ResolvedConfig = { compactThreshold: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, allowedCommands: undefined, allowedCommandGlobs: undefined, reasoningEffort: undefined, diff --git a/apps/cli/src/engine/model-catalog-view.test.ts b/apps/cli/src/engine/model-catalog-view.test.ts index 55c5e91e..1ed6c1a3 100644 --- a/apps/cli/src/engine/model-catalog-view.test.ts +++ b/apps/cli/src/engine/model-catalog-view.test.ts @@ -1,17 +1,17 @@ import type { ModelCatalogListing } from '@relavium/db'; -import { MODEL_PRICING } from '@relavium/llm'; +import { CATALOG_SNAPSHOT, catalogPricing, datedPinBase } from '@relavium/llm'; import { describe, expect, it } from 'vitest'; import { buildMergedCatalog, buildUserPricing } from './model-catalog-view.js'; /** Two known static anthropic model ids (derived from the registry so the test survives a pricing.ts edit). */ function twoAnthropicIds(): readonly [string, string] { - const ids = Object.entries(MODEL_PRICING) + const ids = Object.entries(CATALOG_SNAPSHOT) .filter(([, pricing]) => pricing.provider === 'anthropic') .map(([id]) => id); const [present, absent] = ids; if (present === undefined || absent === undefined) { - throw new Error('test precondition: MODEL_PRICING must carry ≥2 anthropic models'); + throw new Error('test precondition: the catalog must carry ≥2 anthropic models'); } return [present, absent]; } @@ -27,6 +27,7 @@ function row( inputCostPerMtokMicrocents: 0, outputCostPerMtokMicrocents: 0, cachedInputCostPerMtokMicrocents: 0, + cachedInputStated: false, isActive: true, ...partial, }; @@ -38,9 +39,27 @@ function slugResolver(map: Record): (uuid: string) => string { } describe('buildMergedCatalog', () => { - it('seeds the full static registry even with no live rows (never an empty picker)', () => { + it('seeds the full static registry (minus collapsed alias↔dated-pin duplicates) — never an empty picker', () => { const view = buildMergedCatalog({ rows: [], providerSlug: slugResolver({}), now: 0 }); - expect(view.entries).toHaveLength(Object.keys(MODEL_PRICING).length); + // Every static model is seeded EXCEPT an Anthropic dated pin whose rolling alias is also shipped — the pair + // collapses to the alias for the picker (ADR-0064 amendment). Derive the expected drop from the snapshot so the + // count stays correct as the catalog grows, and assert it is non-zero so the dedup is actually exercised. + const collapsed = Object.values(CATALOG_SNAPSHOT).filter((m) => { + if (m.provider !== 'anthropic') return false; + const base = datedPinBase(m.modelId); + return base !== undefined && CATALOG_SNAPSHOT[base]?.provider === 'anthropic'; + }).length; + expect(collapsed).toBeGreaterThan(0); // the shipped snapshot HAS such pairs — the assertion is meaningful + expect(view.entries).toHaveLength(Object.keys(CATALOG_SNAPSHOT).length - collapsed); + // …and NO surviving row is a collapsed dated pin. + expect( + view.entries.some( + (e) => + e.provider === 'anthropic' && + datedPinBase(e.modelId) !== undefined && + CATALOG_SNAPSHOT[datedPinBase(e.modelId) ?? '']?.provider === 'anthropic', + ), + ).toBe(false); expect(view.refreshedAt).toBeUndefined(); // With NO live data for any provider, every static model falls back to static presence (ADR-0064 §6). expect(view.entries.every((e) => e.available)).toBe(true); @@ -137,6 +156,7 @@ describe('buildUserPricing (2.5.G S10, ADR-0065 §2)', () => { inputCostPerMtokMicrocents: 300_000_000, outputCostPerMtokMicrocents: 900_000_000, cachedInputCostPerMtokMicrocents: 12_345, + cachedInputStated: true, contextWindowTokens: 32_000, maxOutputTokens: 4_000, }), @@ -228,3 +248,147 @@ describe('buildUserPricing (2.5.G S10, ADR-0065 §2)', () => { expect(entry?.pricing?.inputPerMtokMicrocents).toBe(300_000_000); }); }); + +/** + * A USER OVERRIDE OF A MODEL THE CATALOG KNOWS (ADR-0071 §1) — the case the whole precedence flip is FOR, and the + * one this file never tested. Every fixture above prices a model the catalog has never heard of (`acme-custom-1`, + * `no-limits`, `shared-id`), which is what the user tier was originally invented for and is no longer all it does. + * + * Three defects lived in exactly this gap, and a Sonnet review found all three: a partial override zeroed the + * catalog's limits, its cache rate and its context tiers; a cache hit could cost FIVE TIMES a cache miss; and a row + * whose provider contradicted the catalog was billed by the cost path while the picker showed the catalog's price. + */ +describe('a user override of a CATALOG model — a partial override must stay partial', () => { + const OPENAI = 'uuid-openai'; + const slugs = slugResolver({ [OPENAI]: 'openai' }); + // `undefined` when the model / field is UNKNOWN — kept distinct from a real `0` (an explicitly-free rate), so a + // caller cannot silently divide by a coerced zero and read "unknown" as "free". + const catalogRate = ( + id: string, + field: 'inputPerMtokMicrocents' | 'cachedInputPerMtokMicrocents', + ): number | undefined => catalogPricing(id)?.[field]; + + it('INHERITS the window, the ceiling and the cache DISCOUNT that the user never stated', () => { + // `models pricing gpt-5.5 --input 0.10 --output 1` — two flags, and nothing else said. The DB's columns are NOT + // NULL DEFAULT 0, so the window, the ceiling and the cache rate all arrive here as zeroes. + const overlay = buildUserPricing({ + rows: [ + row({ + modelId: 'gpt-5.5', + providerId: OPENAI, + source: 'user', + inputCostPerMtokMicrocents: 10_000_000, // $0.10/MTok — a negotiated rate + outputCostPerMtokMicrocents: 100_000_000, + }), + ], + providerSlug: slugs, + }); + const mine = overlay.get('gpt-5.5'); + expect(mine).toBeDefined(); + if (mine === undefined) return; + + // The limits come from the catalog — the picker showed a 0-token context window for GPT-5.5 before this. + expect(mine.contextWindowTokens).toBe(1_050_000); + expect(mine.maxOutputTokens).toBe(128_000); + + // The cache rate is the catalog's DISCOUNT, not its absolute number. Inheriting the absolute $0.50 against a + // $0.10 input would have made a cache HIT cost five times a cache MISS — a price nobody has ever been charged. + const cachedRate = catalogRate('gpt-5.5', 'cachedInputPerMtokMicrocents'); + const inputRate = catalogRate('gpt-5.5', 'inputPerMtokMicrocents'); + // A known priced model — assert both rates are REAL so a lost/unknown price fails loudly here, rather than + // silently producing a 0 ratio (the unknown-vs-zero distinction the helper now preserves). + expect(cachedRate).toBeGreaterThan(0); + expect(inputRate).toBeGreaterThan(0); + const ratio = (cachedRate ?? 0) / (inputRate ?? 1); + expect(mine.cachedInputPerMtokMicrocents).toBe(Math.round(10_000_000 * ratio)); + expect(mine.cachedInputPerMtokMicrocents).toBeLessThanOrEqual(mine.inputPerMtokMicrocents); + }); + + it('BELIEVES an explicit `--cached 0` — a stored zero means free only when the user said so', () => { + // One money column cannot hold both "the user typed 0" and "the user said nothing"; the FACT of the statement + // rides in its own flag (migration 0011). Without it, a self-hosted endpoint with a genuinely free cache could + // never be described — the zero was read as "unset" and silently replaced with the full input rate. + const overlay = buildUserPricing({ + rows: [ + row({ + modelId: 'gpt-5.5', + providerId: OPENAI, + source: 'user', + inputCostPerMtokMicrocents: 10_000_000, + outputCostPerMtokMicrocents: 100_000_000, + cachedInputCostPerMtokMicrocents: 0, + cachedInputStated: true, // they typed `--cached 0` + }), + ], + providerSlug: slugs, + }); + expect(overlay.get('gpt-5.5')?.cachedInputPerMtokMicrocents).toBe(0); + }); + + it('KEEPS the context tiers — as multipliers, so a long prompt still costs more', () => { + // A tiered model is tiered whoever is paying. Dropping the tiers because the user stated a flat price reopens + // the silent 2× under-bill on every long-context turn — the exact hole the tiers were wired up to close. + const overlay = buildUserPricing({ + rows: [ + row({ + modelId: 'gemini-2.5-pro', + providerId: 'uuid-gemini', + source: 'user', + inputCostPerMtokMicrocents: 100_000_000, // $1/MTok + outputCostPerMtokMicrocents: 800_000_000, + }), + ], + providerSlug: slugResolver({ 'uuid-gemini': 'gemini' }), + }); + const mine = overlay.get('gemini-2.5-pro'); + const shipped = catalogPricing('gemini-2.5-pro'); + expect(shipped?.contextTiers?.[0]?.aboveContextTokens).toBe(200_000); // the premise + + const tier = mine?.contextTiers?.[0]; + expect(tier?.aboveContextTokens).toBe(200_000); + // The catalog DOUBLES input above the threshold. So does the user's price — the multiple is a fact about the + // model; the absolute $2.50 is a fact about a price the user is no longer paying. + const multiple = + (shipped?.contextTiers?.[0]?.inputPerMtokMicrocents ?? 0) / + (shipped?.inputPerMtokMicrocents ?? 1); + expect(tier?.inputPerMtokMicrocents).toBe(Math.round(100_000_000 * multiple)); + expect(tier?.inputPerMtokMicrocents).toBeGreaterThan(mine?.inputPerMtokMicrocents ?? 0); + }); + + it('DROPS a row whose provider contradicts the catalog — the merge already did, the cost path did not', () => { + // The sharpest hole the flip opened. `mergeModelCatalog` dropped a cross-provider row (so the picker kept showing + // the catalog's price and said `pricingSource: 'catalog'`), while `priceModel` read the overlay unconditionally + // and BILLED it. One command could zero a shipped model's cost while the UI displayed $5/MTok. + const overlay = buildUserPricing({ + rows: [ + row({ + modelId: 'gpt-5.5', // OpenAI's model… + providerId: 'uuid-anthropic', // …priced under Anthropic + source: 'user', + inputCostPerMtokMicrocents: 1, + outputCostPerMtokMicrocents: 1, + }), + ], + providerSlug: slugResolver({ 'uuid-anthropic': 'anthropic' }), + }); + expect(overlay.has('gpt-5.5')).toBe(false); // …and it is not billed + }); + + it('a model the catalog does NOT know has nothing to inherit — its cache read still is not free', () => { + const overlay = buildUserPricing({ + rows: [ + row({ + modelId: 'acme-local-1', + providerId: OPENAI, + source: 'user', + inputCostPerMtokMicrocents: 300_000_000, + outputCostPerMtokMicrocents: 900_000_000, + }), + ], + providerSlug: slugs, + }); + const mine = overlay.get('acme-local-1'); + expect(mine?.cachedInputPerMtokMicrocents).toBe(300_000_000); // the input rate, not zero + expect(mine?.contextTiers).toBeUndefined(); + }); +}); diff --git a/apps/cli/src/engine/model-catalog-view.ts b/apps/cli/src/engine/model-catalog-view.ts index 8634c95b..655f196b 100644 --- a/apps/cli/src/engine/model-catalog-view.ts +++ b/apps/cli/src/engine/model-catalog-view.ts @@ -1,6 +1,10 @@ import type { ModelCatalogListing } from '@relavium/db'; import { + catalogModel, + catalogPricing, + collapseAliasDatedPinPairs, mergeModelCatalog, + type CatalogPriceTier, type ModelCatalogEntry, type ModelListing, type ModelPricing, @@ -15,15 +19,17 @@ import { LLM_PROVIDERS } from '@relavium/shared'; * It is the thin, PURE glue between the `@relavium/db` store (which speaks internal provider UUIDs + `source` * rows) and the pure `@relavium/llm` {@link mergeModelCatalog} (which speaks the `ProviderId` enum + plain tiers): * it builds the LIVE map from the `source='live'` rows (translating provider UUID → slug) and hands it to the - * merge, whose static tier is the in-code `MODEL_PRICING`. Keeping this in the host — not the store, not the merge + * merge, whose non-user tier is the generated catalog (ADR-0071). Keeping this in the host — not the store, not the merge * — is what lets `@relavium/llm`/`@relavium/core` stay platform-free while every surface reuses the one merge. * * The ADR-0065 USER-pricing tier is built by {@link buildUserPricing} (workstream **2.5.G S10**): it projects the * `source='user'` rows into the ONE `ReadonlyMap` that serves BOTH consumers — the merge's * `userPricing` slot (so the `/models` picker shows a user-priced model's cost) AND the cost path's * {@link PricingOverlay} (host-injected exactly like `keyFor`, so the budget governor enforces `max_cost_microcents` - * on an otherwise-unknown model). Static `MODEL_PRICING` still wins for a known id in both — the user tier only ever - * fills an UNKNOWN id (ADR-0065 §2), so a user can never silently misprice a shipped model. + * on an otherwise-unknown model). The USER tier OUTRANKS the catalog in both (ADR-0071 §1) — they hold the invoice, + * we hold a snapshot of a third-party aggregator. It cannot be done in SILENCE, though: `models pricing` echoes the + * catalog price the override replaces (§5), a partial override inherits every dimension it does not state, and a row + * whose provider contradicts the catalog's is refused rather than stored and quietly ignored. */ /** The merged catalog for the picker + the newest live-refresh stamp (the "last updated" freshness badge). */ @@ -54,7 +60,7 @@ function isProviderId(slug: string): slug is ProviderId { /** Map a stored catalog row → a seam {@link ModelListing} (the live-discovery half): id + the optional limits + * the live deprecation date as ISO (the store carries epoch-ms; the merge unions ISO dates). Pricing is NOT - * carried — the merge's pricing authority is the static registry / user tier, never a live row (ADR-0064 §6). */ + * carried — the merge's pricing authority is the user tier / the catalog, never a live row (ADR-0064 §6). */ function rowToListing(row: ModelCatalogListing): ModelListing { return { id: row.modelId, @@ -70,23 +76,115 @@ function rowToListing(row: ModelCatalogListing): ModelListing { } /** - * Map a `source='user'` catalog row → a seam {@link ModelPricing} (the ADR-0065 user tier). The DB stores integer - * micro-cents in the three `*_per_mtok_microcents` columns (NOT NULL, default `0`) — a captured price is a real - * value; a `0` means "not set for this dimension" and costs that dimension as free, which is the user's declared - * intent. Media output rates + cache-write are NOT user-capturable (no column), so they stay undefined — the cost - * fold degrades those to 0 (H4: never hard-fail on a missing rate). The context/output limits carry through so the - * merged picker and the footer context indicator can show them for an otherwise-unknown model. + * A LIMIT column's "not set" sentinel is `0` (the columns are NOT NULL DEFAULT 0) — read it as absent. + * + * Only for the limits. A zero context window or a zero output ceiling is never a value anyone means, so the sentinel + * is unambiguous there. It is NOT used for money: `--cached 0` is a thing a user can genuinely mean (a self-hosted + * endpoint with a free cache), and the ambiguity is resolved where it actually exists — at WRITE time, in + * `models pricing`, which knows whether the flag was passed. The column stores what the user meant. + */ +function statedLimit(value: number | undefined): number | undefined { + return value === undefined || value <= 0 ? undefined : value; +} + +/** + * The cache-read rate for a user who never stated one (ADR-0071 §10). + * + * The catalog's DISCOUNT, applied to the rate the user actually stated — not the catalog's absolute number. A user + * who negotiated $0.10/MTok input on `gpt-5.5` would otherwise inherit its $0.50 cache rate and pay FIVE TIMES MORE + * for a cache HIT than for a cache MISS, which is not a price anyone has ever been charged. A cache read is a + * discount off input; the discount is the part that survives an override, and it can never exceed the input rate it + * is a discount off. + * + * No catalog row to inherit from ⇒ their own input rate. A cache read is never free just because nobody said what it + * costs — and a user who knows it IS free says so with `--cached 0`, and is believed. + */ +function inheritedCacheRate(base: ModelPricing | undefined, input: number): number { + if (base === undefined || base.inputPerMtokMicrocents <= 0) return input; + const ratio = base.cachedInputPerMtokMicrocents / base.inputPerMtokMicrocents; + return Math.min(input, Math.round(input * ratio)); +} + +/** + * The catalog's context TIERS, re-expressed against the user's own rates (ADR-0071 §11). + * + * A tiered model is tiered whoever is paying: `gemini-2.5-pro` doubles its input rate above 200k context, and a user + * who negotiated a discount has not negotiated that away. Dropping the tiers because the user stated a flat price + * reopens the silent 2× under-bill on every long-context turn — the exact hole the tiers were wired up to close. + * + * So the tiers are inherited as MULTIPLIERS, not as absolute rates. The catalog's $2.50-above-$1.25 is a fact about + * the price the user is no longer paying; that it DOUBLES is a fact about the model. The CLI cannot express a + * per-tier price, so this is the honest reading of a flat price on a tiered model — and it is strictly safer than + * the alternative, because it can only ever bill more than flat, never less. + */ +function scaledTiers( + base: ModelPricing | undefined, + input: number, + output: number, + cachedInput: number, +): readonly CatalogPriceTier[] | undefined { + if (base?.contextTiers === undefined || base.inputPerMtokMicrocents <= 0) return undefined; + const baseIn = base.inputPerMtokMicrocents; + const baseOut = base.outputPerMtokMicrocents; + const baseCached = base.cachedInputPerMtokMicrocents; + return base.contextTiers.map((tier) => ({ + aboveContextTokens: tier.aboveContextTokens, + inputPerMtokMicrocents: Math.round(input * (tier.inputPerMtokMicrocents / baseIn)), + outputPerMtokMicrocents: + baseOut <= 0 ? output : Math.round(output * (tier.outputPerMtokMicrocents / baseOut)), + ...(tier.cachedInputPerMtokMicrocents === undefined || baseCached <= 0 + ? {} + : { + cachedInputPerMtokMicrocents: Math.round( + cachedInput * (tier.cachedInputPerMtokMicrocents / baseCached), + ), + }), + })); +} + +/** + * Map a `source='user'` catalog row → a seam {@link ModelPricing} (the ADR-0065 user tier, which OUTRANKS the + * catalog since ADR-0071 §1). + * + * **A PARTIAL OVERRIDE MUST BE PARTIAL.** The user types `models pricing gpt-5.5 --input 3 --output 12` to record + * their negotiated token rates. They have not said anything about the model's context window, its output ceiling, + * or its cache-read discount — and the DB's columns are `NOT NULL DEFAULT 0`, so "unsaid" arrives here as `0`. + * Reading those zeroes as VALUES was harmless while a user row could only describe a model the registry had never + * heard of: there was nothing to overwrite. Under user-first precedence it destroys verified data — the picker + * showed a 0-token context window for GPT-5.5, and every cached token on it billed at nothing. + * + * So each dimension the user did not state falls back to the CATALOG's, and only then to a safe floor. A cache read + * is never free: absent everywhere, it bills at the user's own input rate (ADR-0071 §10 — "no published cache rate" + * means the provider does not DISCOUNT cache reads, not that it gives them away). + * + * `--cached 0` cannot be told apart from an omitted `--cached`: one column, one sentinel. Treating an explicit zero + * as "not stated" is the safe reading of the ambiguity — the alternative bills a whole class of tokens at nothing. */ function rowToUserPricing(row: ModelCatalogListing, provider: ProviderId): ModelPricing { + const base = catalogPricing(row.modelId); // undefined for a model the catalog has never heard of + const input = row.inputCostPerMtokMicrocents; + const output = row.outputCostPerMtokMicrocents; + // STATED ⇒ their number, zero included. NOT stated ⇒ derive it, because the column's `0` is a default and not an + // instruction (ADR-0071 §10 — the flag exists precisely so the two can be told apart). + const cachedInput = row.cachedInputStated + ? row.cachedInputCostPerMtokMicrocents + : inheritedCacheRate(base, input); + const tiers = scaledTiers(base, input, output, cachedInput); return { provider, nativeId: row.modelId, displayName: row.displayName, - contextWindowTokens: row.contextWindowTokens ?? 0, - maxOutputTokens: row.maxOutputTokens ?? 0, - inputPerMtokMicrocents: row.inputCostPerMtokMicrocents, - outputPerMtokMicrocents: row.outputCostPerMtokMicrocents, - cachedInputPerMtokMicrocents: row.cachedInputCostPerMtokMicrocents, + contextWindowTokens: statedLimit(row.contextWindowTokens) ?? base?.contextWindowTokens ?? 0, + maxOutputTokens: statedLimit(row.maxOutputTokens) ?? base?.maxOutputTokens ?? 0, + inputPerMtokMicrocents: input, + outputPerMtokMicrocents: output, + cachedInputPerMtokMicrocents: cachedInput, + // Cache-WRITE has no user column at all, so it can only come from the catalog. Dropping it would bill Anthropic + // cache writes — the expensive half of prompt caching — at zero for any model the user has priced. + ...(base?.cacheWritePerMtokMicrocents === undefined + ? {} + : { cacheWritePerMtokMicrocents: base.cacheWritePerMtokMicrocents }), + ...(tiers === undefined ? {} : { contextTiers: tiers }), ...(row.deprecationDate !== undefined ? { deprecatedAt: new Date(row.deprecationDate).toISOString() } : {}), @@ -115,6 +213,16 @@ export function buildUserPricing(input: { // write win by UUID luck. `models pricing` REJECTS creating such a duplicate, so this is a defense-in-depth // floor for a legacy / directly-edited db, never the primary guard. if (map.has(row.modelId)) continue; + // …and the guard the merge ALREADY had, which the overlay did not: a user row whose provider contradicts the + // CATALOG's is dropped here too, so the two cannot disagree about which rows apply. + // + // They did, and it was the sharpest hole the flip opened. `mergeModelCatalog` drops a cross-provider row (the + // picker keeps showing the catalog's price and says `pricingSource: 'catalog'`), while `priceModel` read the + // overlay unconditionally and BILLED it. So `models pricing gpt-5.5 --provider anthropic --input 0.00000001` + // zeroed the cost of a shipped OpenAI model — realized fold and pre-egress estimate alike, making + // `max_cost_microcents` unenforceable on it — while the UI displayed $5/MTok. Not merely silent: actively wrong. + const anchored = catalogModel(row.modelId)?.provider; + if (anchored !== undefined && anchored !== slug) continue; map.set(row.modelId, rowToUserPricing(row, slug)); } return map; @@ -157,5 +265,8 @@ export function buildMergedCatalog(input: BuildMergedCatalogInput): MergedCatalo ...(input.keyedProviders === undefined ? {} : { keyedProviders: input.keyedProviders }), now: input.now, }); - return { entries, refreshedAt }; + // Collapse an Anthropic alias↔dated-pin PAIR to the rolling alias here, at the DISPLAY boundary (ADR-0064 + // amendment) — `mergeModelCatalog` keeps its full-fidelity output for any consumer that needs both rows; only the + // picker's projection drops the duplicate. Identity dedup, not an availability change (see the fn's contract). + return { entries: collapseAliasDatedPinPairs(entries), refreshedAt }; } diff --git a/apps/cli/src/engine/pricing-overlay.ts b/apps/cli/src/engine/pricing-overlay.ts index a723489d..fe920ace 100644 --- a/apps/cli/src/engine/pricing-overlay.ts +++ b/apps/cli/src/engine/pricing-overlay.ts @@ -12,8 +12,8 @@ import { buildUserPricing } from './model-catalog-view.js'; * with no static price, once user-priced, is enforced by `max_cost_microcents` (the cost-cap gap ADR-0064 §6 left * open). It is the DB-facing counterpart of the pure {@link buildUserPricing}: it projects the `model_catalog` * `source='user'` rows (translating each internal provider UUID → its slug) into the one overlay that serves both - * the pre-egress estimate and the realized fold. Static `MODEL_PRICING` still wins for a known id — the user tier - * only fills an UNKNOWN id — so a user can never silently misprice a shipped model. + * the pre-egress estimate and the realized fold. The USER tier OUTRANKS the catalog (ADR-0071 §1); what it may not + * do is override it SILENTLY, which is why `models pricing` names the catalog price it replaces. * * Living in the host (not `@relavium/core`/`@relavium/llm`) is what keeps the engine platform-free: the engine * receives a plain injected map, exactly like `keyFor`, and never imports `@relavium/db`. diff --git a/apps/cli/src/engine/providers.ts b/apps/cli/src/engine/providers.ts index 571df113..735d2696 100644 --- a/apps/cli/src/engine/providers.ts +++ b/apps/cli/src/engine/providers.ts @@ -5,6 +5,7 @@ import { defaultProviders, InvalidBaseUrlError, isRetryable, + type EndpointKind, LlmProviderError, type LlmProvider, type ProviderId, @@ -45,6 +46,20 @@ export interface ProviderResolver { * probe when this is absent; the real {@link createProviderResolver} always provides it, so production never falls back. */ readonly hasKey?: (id: ProviderId) => boolean; + /** + * Is this provider talking to its OWN API, or to a custom `base_url` + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §7)? + * + * The adapter clamps an authored `max_tokens` to the model's published ceiling on an official endpoint and + * deliberately does NOT on a custom one — a gateway may serve anything under a familiar model id. The pre-egress + * ESTIMATE has to make the same call, or it stops describing the request that is about to be sent: treat a + * gateway as official and the estimate lands below what the wire can spend, so the budget governor + * under-authorizes and waves through a call it should have stopped. + * + * Decided by HOST, not by "is there a stored row": a user who registers OpenAI with its own endpoint spelled out + * is still on the official API, however they spell it. OPTIONAL so a test stub can omit it; absent ⇒ official. + */ + readonly endpointKind?: (id: ProviderId) => EndpointKind; } /** @@ -312,7 +327,10 @@ export function createProviderResolver( // mapping lives in the seam package (`@relavium/llm`); a stored CUSTOM `base_url` (ADR-0065 §3) rebinds its // provider's adapter to a validated per-provider endpoint here. const adapters: Record = { ...defaultProviders() }; - applyCustomEndpoints(adapters, options); + // The provider ids that ended up pointed at a GENUINELY different host — not merely at a differently-spelled + // official one. Feeds `endpointKind`, which the pre-egress estimate reads (ADR-0071 §7). + const customEndpoints = new Set(); + applyCustomEndpoints(adapters, options, customEndpoints); // The ONE key-resolution path (keychain → env), returning `undefined` for genuine absence — shared by `keyFor` // (which throws on absence) and `hasKey` (which returns a boolean), so the two never drift (2.5.G key-awareness). const resolveKey = (id: ProviderId): string | undefined => { @@ -341,6 +359,7 @@ export function createProviderResolver( }; return { resolveProvider: (id) => adapters[id], + endpointKind: (id) => (customEndpoints.has(id) ? 'custom' : 'official'), keyFor: (id) => { const key = resolveKey(id); if (key === undefined) { @@ -360,6 +379,26 @@ export function createProviderResolver( }; } +/** + * Is this stored `base_url` a host OTHER than the provider's own API? + * + * By HOST, never by string: the CLI stores a `--base-url` VERBATIM, so `https://api.openai.com/v1/` (one trailing + * slash) and `https://api.openai.com/v1` are different strings for the same API. Mirrors `endpointKindFor` inside + * the adapter, which decides the same question for the wire — the two must agree, or the estimate stops describing + * the request. A trailing-dot FQDN (`api.openai.com.`) is the same host too, and DNS says so. + */ +function isCustomHost(id: ProviderId, baseUrl: string): boolean { + try { + const host = new URL(baseUrl).hostname.toLowerCase().replace(/\.$/, ''); + return host !== new URL(KNOWN_PROVIDERS[id].baseUrl).hostname.toLowerCase(); + } catch { + return true; // unparseable ⇒ treat as custom (the conservative side: no clamp, no dialect switch) + } +} + +/** One `providerStore.list()` row. Derived so no new import is needed. */ +type StoredProviderRow = ReturnType[number]; + /** * Rebind a provider's adapter to a validated CUSTOM-endpoint adapter when its stored row carries a `base_url` that * differs from the known default (2.5.G S9, ADR-0065 §3–4). Only **OpenAI-compatible** (`openai`/`deepseek`) is @@ -373,25 +412,43 @@ export function createProviderResolver( function applyCustomEndpoints( adapters: Record, options: ProviderResolverOptions, + custom: Set, ): void { const store = options.providerStore; if (store === undefined) return; // no registry ⇒ default endpoints only (the pre-S9 behavior) + // Built lazily, ONCE, and only if a row actually needs it — memoized in this closure so a row-level helper can + // reuse the same validated fetch without each row rebuilding it. let validatedFetch: FetchLike | undefined; + const getValidatedFetch = (): FetchLike => + (validatedFetch ??= options.validatedFetch ?? createValidatedFetch()); for (const row of store.list()) { - const id = KNOWN_PROVIDER_IDS.find((known) => known === row.name); - if (id === undefined) continue; // a non-enum name (provider add enforces the closed enum) — ignore - if (row.baseUrl === KNOWN_PROVIDERS[id].baseUrl) continue; // the default endpoint — keep the default adapter - if (id !== 'openai' && id !== 'deepseek') continue; // custom base_url is openai-compatible only this round (§3) - validatedFetch ??= options.validatedFetch ?? createValidatedFetch(); // built lazily, once, only when needed - try { - adapters[id] = createCustomOpenAiProvider({ - providerId: id, - baseURL: row.baseUrl, - fetch: validatedFetch, - }); - } catch (err) { - // A bad stored base_url (non-HTTPS / private / creds) — refuse the custom endpoint, keep the default adapter. - if (!(err instanceof InvalidBaseUrlError)) throw err; - } + applyCustomEndpointForRow(row, adapters, custom, getValidatedFetch); + } +} + +/** Rebind ONE stored row to a custom OpenAI-compatible adapter, or leave the default adapter standing. */ +function applyCustomEndpointForRow( + row: StoredProviderRow, + adapters: Record, + custom: Set, + getValidatedFetch: () => FetchLike, +): void { + const id = KNOWN_PROVIDER_IDS.find((known) => known === row.name); + if (id === undefined) return; // a non-enum name (provider add enforces the closed enum) — ignore + if (row.baseUrl === KNOWN_PROVIDERS[id].baseUrl) return; // the default endpoint — keep the default adapter + if (id !== 'openai' && id !== 'deepseek') return; // custom base_url is openai-compatible only this round (§3) + try { + adapters[id] = createCustomOpenAiProvider({ + providerId: id, + baseURL: row.baseUrl, + fetch: getValidatedFetch(), + }); + // Record it for the pre-egress estimate (ADR-0071 §7) — by HOST, so a row that merely SPELLS the official + // endpoint differently (a trailing slash, a missing `/v1`) is not mistaken for a gateway. The adapter makes + // the same call for the wire; this keeps the estimate describing the request the adapter will send. + if (isCustomHost(id, row.baseUrl)) custom.add(id); + } catch (err) { + // A bad stored base_url (non-HTTPS / private / creds) — refuse the custom endpoint, keep the default adapter. + if (!(err instanceof InvalidBaseUrlError)) throw err; } } diff --git a/apps/cli/src/harness/regression.e2e.test.ts b/apps/cli/src/harness/regression.e2e.test.ts index d3df3e55..d493e093 100644 --- a/apps/cli/src/harness/regression.e2e.test.ts +++ b/apps/cli/src/harness/regression.e2e.test.ts @@ -1,16 +1,17 @@ import { randomUUID } from 'node:crypto'; import { mkdtempSync, rmSync } from 'node:fs'; -import { tmpdir } from 'node:os'; +import { homedir, tmpdir } from 'node:os'; import { join } from 'node:path'; import { fileURLToPath } from 'node:url'; import type { WorkflowDefinition } from '@relavium/core'; import { createClient, createRunHistoryStore, runMigrations } from '@relavium/db'; import { RunEventSchema, type RunEvent } from '@relavium/shared'; -import { describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { gateCommand } from '../commands/gate.js'; import { runCommand } from '../commands/run.js'; +import { resolveHomeDir } from '../config/load.js'; import type { OpenedHistory } from '../history/open.js'; import { EXIT_CODES, type ExitCode } from '../process/exit-codes.js'; import type { GlobalOptions } from '../process/options.js'; @@ -236,6 +237,39 @@ function assertGapFreeSeq(events: readonly RunEvent[]): void { const argv = (...tokens: string[]): string[] => ['node', 'relavium', ...tokens]; describe('engine regression harness (2.K) — offline fixtures over `relavium run … --json`', () => { + // ISOLATE `~/.relavium` FOR THE WHOLE SUITE (deferred-tasks.md, 2026-07-13). The full-shell `run(argv)` case + // below drives the REAL CLI shell, which resolves the DEFAULT history db (`~/.relavium/history.db`) and RUNS + // MIGRATIONS against it — the developer's actual chat history under a real HOME. Point HOME at a throwaway dir + // for every test (a floor, so a future test cannot silently re-acquire the real path either), and GUARD it: if + // the redirect ever fails to take, refuse to run rather than migrate real user data (a suite that writes to real + // data both damages it and blinds CI, which is exactly how this was found — PR #75). + const realHome = homedir(); + let isolatedHome: string | undefined; + let prevHome: string | undefined; + let prevProfile: string | undefined; + + beforeEach(() => { + isolatedHome = mkdtempSync(join(tmpdir(), 'relavium-harness-home-')); + prevHome = process.env['HOME']; + prevProfile = process.env['USERPROFILE']; + process.env['HOME'] = isolatedHome; + process.env['USERPROFILE'] = isolatedHome; // Windows home + if (resolveHomeDir({}) === realHome) { + throw new Error( + 'e2e isolation failed: HOME still resolves to the real home — refusing to run migrations against ~/.relavium.', + ); + } + }); + + afterEach(() => { + if (prevHome === undefined) delete process.env['HOME']; + else process.env['HOME'] = prevHome; + if (prevProfile === undefined) delete process.env['USERPROFILE']; + else process.env['USERPROFILE'] = prevProfile; + if (isolatedHome !== undefined) rmSync(isolatedHome, { recursive: true, force: true }); + isolatedHome = undefined; + }); + for (const scenario of SCENARIOS) { const label = `${scenario.file} [${scenario.input.join(' ') || 'no input'}] → exit ${String(scenario.exit)}`; it(label, async () => { diff --git a/apps/cli/src/harness/session-chain.e2e.test.ts b/apps/cli/src/harness/session-chain.e2e.test.ts index cda89743..c4de03c3 100644 --- a/apps/cli/src/harness/session-chain.e2e.test.ts +++ b/apps/cli/src/harness/session-chain.e2e.test.ts @@ -33,6 +33,7 @@ import { CHAT_TEXT_CAPABILITY_FLAGS } from '../test-support.js'; const EMPTY_CHAT: ResolvedChatConfig = { defaultModel: undefined, + defaultProvider: undefined, fsScope: undefined, maxTurns: undefined, maxMessages: undefined, @@ -40,6 +41,7 @@ const EMPTY_CHAT: ResolvedChatConfig = { compactThreshold: undefined, maxCostMicrocents: undefined, onExceed: undefined, + strictCostCap: false, allowedCommands: undefined, allowedCommandGlobs: undefined, reasoningEffort: undefined, diff --git a/apps/cli/src/home/drive-home.test.ts b/apps/cli/src/home/drive-home.test.ts index b1663c81..ccd073d4 100644 --- a/apps/cli/src/home/drive-home.test.ts +++ b/apps/cli/src/home/drive-home.test.ts @@ -7,7 +7,7 @@ import { createClient, createSessionStore, runMigrations, type DbClient } from ' import { REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { buildChatSession } from '../chat/session-host.js'; +import { buildChatSession, buildResumedChatSession } from '../chat/session-host.js'; import { scriptedResolver, textTurn } from '../chat/test-support.js'; import type { ProviderResolver } from '../engine/providers.js'; import type { OpenedSessionStore } from '../history/session-open.js'; @@ -391,6 +391,49 @@ describe('driveHome (2.5.B / ADR-0054)', () => { expect(await drivePromise).toBe(EXIT_CODES.success); }); + it('startChat couples the fresh default_model to the provider read WITH it — never a stale startup provider (review M5)', async () => { + // The 404 trap: startup config binds default_model=gpt-4o + default_provider=openai. Mid-session the user + // hand-edits the config to a DIFFERENT-family model (claude-sonnet-4-6) and does NOT add a provider line. The + // next chat must NOT pair the fresh Claude model with the stale `openai` startup provider (which would dial the + // OpenAI adapter with a Claude id → 404). With no provider on the fresh read, it is left undefined ⇒ inferred. + const configFile = join(cwd, 'coupling-config.toml'); + writeFileSync( + configFile, + '[preferences]\ndefault_model = "gpt-4o"\ndefault_provider = "openai"\n', + ); + let captured: RootAppProps | undefined; + const builtModels: Array = []; + const builtProviders: Array = []; + const { deps } = makeDeps((p) => (captured = p), { + global: { ...global, configPath: configFile }, + buildSession: (args) => { + builtModels.push(args.chat.defaultModel); + builtProviders.push(args.chat.defaultProvider); + return buildChatSession(args); + }, + }); + const drivePromise = driveHome(deps); + const props = captured; + if (props === undefined) throw new Error('the injected render was never invoked'); + + // The user edits the global config mid-session: a new model of another family, no provider line. + writeFileSync(configFile, '[preferences]\ndefault_model = "claude-sonnet-4-6"\n'); + + type(props, 'hello'); + props.controller.handleKey('', ENTER); // submit ⇒ startChat ⇒ buildSession + await flush(); + + expect(builtModels.at(-1)).toBe('claude-sonnet-4-6'); // the fresh model took effect + // THE FIX: the provider is NOT the stale startup 'openai' — it is undefined (⇒ inferred to anthropic), coupled + // to the same read that supplied the model. Before M5 this was 'openai' and the turn 404'd. + expect(builtProviders.at(-1)).toBeUndefined(); + + props.controller.handleKey('c', CTRL_C); + await flush(); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + it('in-Home /models reseat: the REAL reseatChat resumes the session under the switched model, carrying the transcript (ADR-0059)', async () => { // Exercises the REAL drive-home reseatChat builder (loadFull → swapAgentModel → buildResumedChatSession → seeded // store) end-to-end — not the mocked controller-level test — pinning the build-first swap over the same sessionId. @@ -417,7 +460,10 @@ describe('driveHome (2.5.B / ADR-0054)', () => { props.controller.handleKey('', ENTER); await flush(); expect(props.controller.getSnapshot().modelPicker).toBeDefined(); - type(props, 'opus'); // filter to claude-opus-4-8 (registry-priced anthropic ⇒ available on the scripted key) + // Filter to EXACTLY one model. `'opus'` matched a single row while the shipped table had twelve; the generated + // catalog carries eighty, and several are Opus — so a substring filter now selects whichever sorts first, and + // the test would silently be asserting about a different model than it names (ADR-0071 §1). + type(props, 'claude-opus-4-8'); props.controller.handleKey('', ENTER); // opus is reasoning-capable ⇒ the ADR-0066 effort sub-step (not an // immediate reseat). The picker advanced to the effort phase over the pending model. const effortPicker = props.controller.getSnapshot().modelPicker; @@ -464,6 +510,55 @@ describe('driveHome (2.5.B / ADR-0054)', () => { expect(await drivePromise).toBe(EXIT_CODES.success); }); + it('a reseat notice fired DURING the build is buffered and flushed into the transcript, not lost to stderr (review M5/bot)', async () => { + // The store is seeded from the build (it needs `built.resumeState`), so it cannot exist yet when a governor/ + // effort callback fires mid-build. The old sink wrote such a notice to stderr, which the alt-screen renderer's + // next frame erases — and `onceEffortNotice` had already marked it delivered, so it never repeated: silently + // lost. Buffer + flush is the fix. Drive the REAL reseatChat, firing an `onUnpriced` note DURING the build. + const errs: string[] = []; + let captured: RootAppProps | undefined; + const { deps } = makeDeps((p) => (captured = p), { + io: { ...io, writeErr: (t: string) => errs.push(t) }, + providers: scriptedResolver([textTurn('sonnet reply'), textTurn('opus reply')]), + // Fire a notice WHILE the resumed session builds (before the store exists), then delegate to the real builder. + buildResumedSession: (args) => { + args.onUnpriced?.('UNPRICED-MID-BUILD'); + return buildResumedChatSession(args); + }, + }); + const drivePromise = driveHome(deps); + const props = captured; + if (props === undefined) throw new Error('the injected render was never invoked'); + + type(props, 'first'); + props.controller.handleKey('', ENTER); + await flush(); + const sessionId = props.controller.getSnapshot().session?.sessionId ?? ''; + expect(sessionId).not.toBe(''); + + props.controller.handleKey('/', {}); + type(props, 'models'); + props.controller.handleKey('', ENTER); + await flush(); + type(props, 'claude-opus-4-8'); + props.controller.handleKey('', ENTER); // reasoning-capable ⇒ effort sub-step + props.controller.handleKey('', ENTER); // apply the default tier ⇒ the REAL reseatChat (fires the mid-build note) + await flush(); + + const transcript = + props.controller.getSnapshot().session?.store.getSnapshot().state.transcript ?? []; + // THE FIX: the mid-build note reached the TRANSCRIPT (a `notice` entry), not erased stderr. + expect( + transcript.some((e) => e.role === 'notice' && e.text.includes('UNPRICED-MID-BUILD')), + ).toBe(true); + expect(errs.join('')).not.toContain('UNPRICED-MID-BUILD'); // …and it was NOT written to stderr + + props.controller.handleKey('c', CTRL_C); + await flush(); + props.controller.handleKey('c', CTRL_C); + expect(await drivePromise).toBe(EXIT_CODES.success); + }); + it('a KEY-LESS first run runs the onboarding wizard BEFORE mounting the Home (2.5.G S8)', async () => { // A scripted clack slice + a key-less resolver (keyFor throws for every provider) ⇒ the wizard triggers. const outros: string[] = []; diff --git a/apps/cli/src/home/drive-home.tsx b/apps/cli/src/home/drive-home.tsx index 1f2b2289..619466cf 100644 --- a/apps/cli/src/home/drive-home.tsx +++ b/apps/cli/src/home/drive-home.tsx @@ -19,6 +19,7 @@ import { type ChatBudgetWarning, } from '../chat/session-host.js'; import { assembleDoctorProbes } from '../chat/doctor-host.js'; +import { onceEffortNotice } from '../chat/effort-notice.js'; import type { DoctorProbes } from '../chat/doctor.js'; import { createSessionPersister, @@ -26,7 +27,7 @@ import { type SessionPersister, } from '../chat/persister.js'; import { loadResolvedConfig } from '../config/load.js'; -import { writeGlobalDefaultModel, writeGlobalPreferences } from '../config/write.js'; +import { writeGlobalPreferences } from '../config/write.js'; import { createModelCatalogPort } from '../engine/model-catalog-port.js'; import { readUserPricingOverlay } from '../engine/pricing-overlay.js'; import { assembleToolEnv } from '../engine/tool-host/assemble.js'; @@ -342,9 +343,13 @@ export async function driveHome(deps: HomeDeps): Promise { // Write to the SAME file the picker re-reads + the started session resolves (honors `--config`), so a `/models` // write is never a silent no-op to a different file (2.5.G S7). The effort rides the SAME atomic write; an // absent `reasoningEffort` (a non-reasoning model) leaves any prior effort default unchanged. - writeDefault: (modelId, reasoningEffort) => + writeDefault: (modelId, provider, reasoningEffort) => writeGlobalPreferences( - { defaultModel: modelId, ...(reasoningEffort === undefined ? {} : { reasoningEffort }) }, + { + defaultModel: modelId, + defaultProvider: provider, // ADR-0059: persist the provider so the next chat skips id inference + ...(reasoningEffort === undefined ? {} : { reasoningEffort }), + }, homeDir, deps.global.configPath, ), @@ -485,17 +490,27 @@ export async function driveHome(deps: HomeDeps): Promise { // agree BY CONSTRUCTION here — this pins that, loudly, if a future edit ever gives them separate sources. assertRenderStoreAgree(altScreenActive, store.getSnapshot().state.transcriptBound); // The ADR-0065 §2 user-pricing overlay (2.5.G S10), read FRESH per chat from the SAME `history.db` (empty map - // on a read fault). Static `MODEL_PRICING` still wins. + // on a read fault). The USER outranks the catalog (ADR-0071 §1). const resolvePrice = readUserPricingOverlay(opened.db); + // Re-read the EFFECTIVE `[chat]` FRESH per chat (not the load-once `config` snapshot) so a same-session + // `/models` write — the model (2.5.G S7) + its effort (ADR-0066 §6) — lights up the next chat. Read it ONCE: + // the model and the provider persisted WITH it (ADR-0059) must come from the SAME snapshot, or a fresh model + // paired with a stale startup provider dials the wrong adapter → 404 (review M5). So couple them — when the + // fresh read supplies a model, take ITS provider (undefined ⇒ infer from the model id, NEVER back-filled from + // startup); only with no fresh model (a genuinely unset default, or a malformed-config read fault) fall back + // to the startup PAIR. The other `[chat]` settings keep the startup snapshot. + const effectiveChat = readEffectiveChat(); + const freshModel = effectiveChat?.defaultModel; + const [defaultModel, defaultProvider] = + freshModel !== undefined + ? [freshModel, effectiveChat?.defaultProvider] + : [config.chat.defaultModel, config.chat.defaultProvider]; const built: BuiltChatSession = await (deps.buildSession ?? buildChatSession)({ - // Re-read the EFFECTIVE default model AND reasoning-effort FRESH per chat (not the load-once `config` - // snapshot) so a same-session `/models` write — the model (2.5.G S7) AND its effort sub-step (ADR-0066 §6) — - // takes effect on the very next chat started in this long-lived Home. A read fault degrades to the startup - // value. The other `[chat]` settings keep the startup snapshot. chat: { ...config.chat, - defaultModel: readEffectiveDefault() ?? config.chat.defaultModel, - reasoningEffort: readEffectiveEffort() ?? config.chat.reasoningEffort, + defaultModel, + defaultProvider, + reasoningEffort: effectiveChat?.reasoningEffort ?? config.chat.reasoningEffort, }, agentRef: undefined, // the built-in default agent (zero-config first run) cwd: deps.global.cwd, @@ -510,6 +525,11 @@ export async function driveHome(deps: HomeDeps): Promise { // exactly this reason (Step-4b-3 Sonnet fix): a raw write lands on the alt buffer, where ink's next frame // overwrites it — the user is warned about their spend on a line that survives a single frame. onBudgetWarning: (warning) => store.notice(budgetWarningText(warning)), + // Same channel, same reason (ADR-0071 §6): a tier the bound model will not take is withheld at send, and + // saying so on raw stderr would land on the alt buffer for one frame. `onceEffortNotice` keeps a standing + // condition — a stale `off` on a model that cannot disable thinking — from repeating every single turn. + onEffortWithheld: onceEffortNotice((note) => store.notice(note)), + onUnpriced: (note) => store.notice(note), }); return wireHomeChatSession(built, store, { open: true }); }; @@ -538,16 +558,21 @@ export async function driveHome(deps: HomeDeps): Promise { ); const record: AgentSessionRecord = { ...loaded.session, agentSnapshot: newAgent }; const resolvePrice = readUserPricingOverlay(opened.db); - // The store's SEED comes from the build, so it cannot be created first — yet `onBudgetWarning` closes over it and - // a pre-egress cap check can fire DURING the build. Hold it in a `let` and fall back to stderr until it exists, - // exactly as `emitLiveNotice` does on the standalone chat. Either way the warning is never written raw onto the - // alt buffer, where ink's next frame would erase it (Step-4b-3 Sonnet fix, carried here by the phase review). + // The store's SEED comes from the build (it needs `built.resumeState`), so it cannot exist before the build — + // yet a governor/effort callback could fire DURING it. Every notice sink here goes through this one indirection: + // render into the transcript once the store exists, otherwise BUFFER (review M5/bot) and flush after the store + // is wired. The old fallback wrote to stderr, which on the alt-screen renderer ink's next frame erases — so a + // withheld-effort notice that fired mid-build was lost, and `onceEffortNotice` had already marked it delivered + // (never to repeat). `onBudgetWarning` alone used to be guarded; the other two closed over the later `const + // store` and would TDZ-crash under the same condition it was protected against. const storeRef: { current?: ChatStoreController } = {}; - const noteBudget = (warning: ChatBudgetWarning): void => { - const text = budgetWarningText(warning); + const pendingNotes: string[] = []; + const noteToStore = (text: string): void => { if (storeRef.current !== undefined) storeRef.current.notice(text); - else deps.io.writeErr(`${text}\n`); + else pendingNotes.push(text); }; + const noteBudget = (warning: ChatBudgetWarning): void => + noteToStore(budgetWarningText(warning)); const built = await (deps.buildResumedSession ?? buildResumedChatSession)({ chat: config.chat, record, @@ -558,6 +583,10 @@ export async function driveHome(deps: HomeDeps): Promise { mcpRegistrations: config.mcpServers, ...(resolvePrice.size === 0 ? {} : { resolvePrice }), onBudgetWarning: noteBudget, + // A RESEAT binds a different model — precisely when a tier that was fine a moment ago stops being accepted. + // Through `noteToStore`, like every sink here, so none can TDZ on the store declared below. + onEffortWithheld: onceEffortNotice(noteToStore), + onUnpriced: noteToStore, }); // Seed the view store with the carried model + cost/turns — a resumed session never re-emits session:started, // so without this the footer shows nothing until the first new turn (mirrors chatResumeCommand). @@ -579,6 +608,10 @@ export async function driveHome(deps: HomeDeps): Promise { // agree BY CONSTRUCTION here — this pins that, loudly, if a future edit ever gives them separate sources. assertRenderStoreAgree(altScreenActive, store.getSnapshot().state.transcriptBound); storeRef.current = store; // from here a budget warning renders in the transcript, not on the alt buffer + // Flush any notice that fired DURING the build (an effort-withheld or unpriced note on the reseated model) + // into the transcript now that the store exists — it would otherwise have been written to erased stderr. + for (const note of pendingNotes) store.notice(note); + pendingNotes.length = 0; return wireHomeChatSession(built, store, { open: false, initialSequenceNumber: built.nextSequenceNumber, @@ -608,8 +641,12 @@ export async function driveHome(deps: HomeDeps): Promise { io: deps.io, // Reuse the SAME config-write target as the `/models` port (honors `--config`) so the wizard's starter // model + a later `/models` pick + the started session all agree on one file (2.5.G S7/S8). - writeDefaultModel: (modelId) => - writeGlobalDefaultModel(modelId, homeDir, deps.global.configPath), + writeDefaultModel: (modelId, provider) => + writeGlobalPreferences( + { defaultModel: modelId, defaultProvider: provider }, + homeDir, + deps.global.configPath, + ), ...(deps.onboardingPrompter === undefined ? {} : { prompter: deps.onboardingPrompter }), }); } diff --git a/apps/cli/src/onboarding/wizard.test.ts b/apps/cli/src/onboarding/wizard.test.ts index 921242a5..6d02a249 100644 --- a/apps/cli/src/onboarding/wizard.test.ts +++ b/apps/cli/src/onboarding/wizard.test.ts @@ -176,7 +176,10 @@ describe('runOnboardingWizard', () => { // ...the provider row + keychain-ref were registered... expect(s.get('anthropic')?.apiKeyKeychainRef).toBe(keychainAccount('anthropic')); // ...the chosen provider's starter model was set as the default (so the NEXT chat binds a model whose key exists)... - expect(writeDefaultModel).toHaveBeenCalledWith(KNOWN_PROVIDERS.anthropic.testModel); + expect(writeDefaultModel).toHaveBeenCalledWith( + KNOWN_PROVIDERS.anthropic.testModel, + 'anthropic', + ); // ...a "Connected" note shows only the key HINT (last 4), never the full key, and confirms + hands off. const all = [...notes, ...outros].join('\n'); expect(all).toContain('••••1234'); @@ -199,7 +202,7 @@ describe('runOnboardingWizard', () => { writeDefaultModel, }); // NOT the anthropic default — the openai starter, so a first chat doesn't try (and fail) an anthropic key. - expect(writeDefaultModel).toHaveBeenCalledWith(KNOWN_PROVIDERS.openai.testModel); + expect(writeDefaultModel).toHaveBeenCalledWith(KNOWN_PROVIDERS.openai.testModel, 'openai'); expect(KNOWN_PROVIDERS.openai.testModel).not.toBe(KNOWN_PROVIDERS.anthropic.testModel); }); @@ -389,7 +392,7 @@ describe('runOnboardingWizard', () => { ]), }); expect(keychain.store.get(keychainAccount('gemini'))).toBe('sk-offline-key'); - expect(writeDefaultModel).toHaveBeenCalledWith(KNOWN_PROVIDERS.gemini.testModel); + expect(writeDefaultModel).toHaveBeenCalledWith(KNOWN_PROVIDERS.gemini.testModel, 'gemini'); // The NETWORK branch pre-highlights 'continue' so a bare Enter saves anyway (an offline first-run isn't blocked). expect(selectCalls[1]?.initialValue).toBe('continue'); }); diff --git a/apps/cli/src/onboarding/wizard.ts b/apps/cli/src/onboarding/wizard.ts index e0aca63c..878ff13d 100644 --- a/apps/cli/src/onboarding/wizard.ts +++ b/apps/cli/src/onboarding/wizard.ts @@ -99,10 +99,11 @@ export interface OnboardingDeps { readonly keychain: KeychainStore; readonly resolver: ProviderResolver; readonly io: CliIo; - /** Persist the NEXT session's default model (writeGlobalDefaultModel via the Home's config target). The wizard - * sets a starter model of the CHOSEN provider so the first chat binds a model whose key was just stored — the - * built-in default (`claude-sonnet-4-6` → anthropic) would otherwise error for a user who picked another provider. */ - readonly writeDefaultModel: (modelId: string) => void; + /** Persist the NEXT session's default model AND its `provider` (ADR-0059 — authoritative at pick time) via the + * Home's config target. The wizard sets a starter model of the CHOSEN provider so the first chat binds a model + * whose key was just stored — the built-in default (`claude-sonnet-4-6` → anthropic) would otherwise error for a + * user who picked another provider — and persisting the provider means that first chat skips id inference. */ + readonly writeDefaultModel: (modelId: string, provider: ProviderId) => void; /** * LIVE-validate a just-entered key (2.5.G S8) — injected for tests (no network). Absent ⇒ the real bounded, * key-redacted {@link validateProviderKey} probe against the provider's cheap `testModel`. The wizard uses the @@ -238,7 +239,7 @@ export async function runOnboardingWizard(deps: OnboardingDeps): Promise { ? `Verified and stored your ${provider} key (${keyHint(keyToStore)}) in the OS keychain.` : `Saved your ${provider} key (${keyHint(keyToStore)}) — it couldn't be verified now. Run /doctor to re-check.`; try { - deps.writeDefaultModel(starterModel); + deps.writeDefaultModel(starterModel, provider); p.note( `${storedLine}\nYour default model is ${starterModel} — change it anytime with /models.`, 'Connected', diff --git a/apps/cli/src/render/sanitize.ts b/apps/cli/src/render/sanitize.ts new file mode 100644 index 00000000..1f5c5c05 --- /dev/null +++ b/apps/cli/src/render/sanitize.ts @@ -0,0 +1,62 @@ +/** + * Terminal-output sanitization primitives — the CLI's Trojan-Source / ANSI-injection floor. + * + * Pure string functions with NO render/TUI dependency, so every surface can share them without importing the + * ink projection layer: the TUI projections, the record/list commands, and — the reason this lives OUTSIDE + * `render/tui/` — the engine hosts (`build-engine.ts`, `session-host.ts`) that report a withheld reasoning tier + * through `chat/effort-notice.ts`. `render/tui/chat-projection.ts` re-exports both, so its own consumers are + * unaffected. SECURITY-SENSITIVE: keep behavior exact (covered by the strip tests). + */ + +/* eslint-disable no-control-regex */ +// The terminal-sanitizer matcher, composed from one named source fragment per ESC-introduced escape family so a +// future edit touches a single, clearly-scoped family rather than one dense expression. SECURITY-SENSITIVE: the +// alternation below is byte-for-byte the prior single literal — keep behavior exact (covered by the strip tests). +/** OSC (`ESC ]`) + DCS/PM/APC/SOS (`ESC P`/`^`/`_`/`X`) string sequences, sharing one body. The terminator (BEL + * or ST) is REQUIRED, so an UNterminated introducer does NOT match here — it falls through to {@link ESC_2BYTE} + * (only its 2-byte form stripped), leaving the following text visible. An optional terminator would instead + * swallow the whole remainder of the string, silently erasing legitimate model output. */ +const ESC_STRING_SEQ = String.raw`\x1b[\]P^_X][^\x07\x1b]*(?:\x07|\x1b\\)`; +/** CSI (`ESC [` colors/cursor) — the parameter / intermediate / final-byte form. */ +const ESC_CSI = String.raw`\x1b\[[0-?]*[ -/]*[@-~]`; +/** Any remaining 2-byte `ESC <0x40–0x5f>` escape (incl. an unterminated OSC/DCS/PM/APC introducer). */ +const ESC_2BYTE = String.raw`\x1b[@-Z\\-_]`; +/** Every ESC-introduced sequence, composed from the named families above. */ +const ESC_SEQUENCES = new RegExp(`${ESC_STRING_SEQ}|${ESC_CSI}|${ESC_2BYTE}`, 'g'); +/** Remaining C0/C1 control bytes — keep only TAB (\x09) and LINE FEED (\x0a). */ +const BARE_CONTROLS = /[\x00-\x08\x0b-\x1f\x7f-\x9f]/g; +/* eslint-enable no-control-regex */ + +/** + * Unicode bidirectional / directional FORMAT controls — the Trojan-Source floor (CVE-2021-42574; 2.5-close + * Step 14). These code points live ABOVE the C0/C1 range {@link BARE_CONTROLS} strips, so they survive it — yet + * they REORDER how a terminal visually renders a line, letting streamed model output or pasted input display in + * an order that differs from its logical bytes (a spoofed path/command in an approval prompt, a hidden argument, + * a reversed URL). The set is the standard Trojan-Source family: the embeddings/overrides U+202A–202E + * (LRE/RLE/PDF/LRO/RLO), the isolates U+2066–2069 (LRI/RLI/FSI/PDI), and the marks LRM (U+200E) / RLM (U+200F) / + * ALM (U+061C). ZWJ/ZWNJ (U+200D/U+200C) are deliberately NOT stripped — they are legitimate in emoji sequences + * and in Indic/Arabic/Persian shaping. Not in `no-control-regex`'s C0/C1 range, so no eslint-disable is needed. + * Written with `\u` escapes (never literal bidi bytes) so the source itself carries no Trojan-Source hazard. + */ +const BIDI_CONTROLS = /[\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]/g; + +/** + * Strip terminal control sequences from text that will be written to a terminal — so model output (or pasted + * input) cannot inject ANSI/OSC escapes (colors, a cursor jump, a window-title/clipboard/hyperlink write, a + * `\r` line-overwrite) NOR spoof the visual line order with Unicode bidi controls (the Trojan-Source floor). + * Applied at the **display** boundary only; the PERSISTED transcript keeps the raw text (it is user/model data, + * not displayed back through a shell). Keeps printable text plus tabs and newlines. + */ +export function stripTerminalControls(text: string): string { + return text.replace(ESC_SEQUENCES, '').replace(BARE_CONTROLS, '').replace(BIDI_CONTROLS, ''); +} + +/** + * Sanitize a single-line dynamic identifier (a tool id, the bound model name, a persisted session title) for + * terminal display: strip the ANSI/OSC/control bytes {@link stripTerminalControls} removes, then collapse any + * surviving tab/newline to a single space so the value cannot spoof extra terminal lines or columns inside a + * one-line annotation/footer/list row. + */ +export function sanitizeInline(text: string): string { + return stripTerminalControls(text).replace(/[\t\n]+/g, ' '); +} diff --git a/apps/cli/src/render/tui/chat-ink.tsx b/apps/cli/src/render/tui/chat-ink.tsx index 2ec89984..d83e28b3 100644 --- a/apps/cli/src/render/tui/chat-ink.tsx +++ b/apps/cli/src/render/tui/chat-ink.tsx @@ -28,6 +28,7 @@ import { initialEffortPickerState, type EffortPickerState, } from './effort-picker.js'; +import { effortRowLabel } from '../../chat/effort-notice.js'; import { EffortTierList } from './effort-tier-list.js'; import { foldModelPickerKey, @@ -887,6 +888,7 @@ export function ChatApp(props: Readonly): ReactElement { phase: 'model', effortStep: props.onSetEffort !== undefined, pending: undefined, + effortTiers: [], // no model pending yet — populated on the model→effort transition effortSelected: 0, currentEffort: props.store.getSnapshot().reasoningEffort, }); @@ -909,13 +911,15 @@ export function ChatApp(props: Readonly): ReactElement { ): void => { if (step.modelId === open.currentDefault) { if (step.reasoningEffort === undefined || step.reasoningEffort === open.currentEffort) { - const at = step.reasoningEffort === undefined ? '' : ` at effort ${step.reasoningEffort}`; + const at = + step.reasoningEffort === undefined + ? '' + : ` at effort ${effortRowLabel(step.modelId, step.reasoningEffort).label}`; props.store.note(`Already on ${step.displayName}${at}.`); } else { + const label = effortRowLabel(step.modelId, step.reasoningEffort).label; props.onSetEffort?.(step.reasoningEffort); - props.store.note( - `Reasoning effort set to ${step.reasoningEffort} — applies to your next message.`, - ); + props.store.note(`Reasoning effort set to ${label} — applies to your next message.`); } return; } @@ -976,17 +980,17 @@ export function ChatApp(props: Readonly): ReactElement { case 'close': applyEffortPicker(undefined); return; - case 'accept': + case 'accept': { applyEffortPicker(undefined); + const label = effortRowLabel(open.model, step.effort).label; // a budget model's `medium` reads "on" if (step.effort === open.current) { - props.store.note(`Already at reasoning effort ${step.effort}.`); + props.store.note(`Already at reasoning effort ${label}.`); } else { props.onSetEffort?.(step.effort); - props.store.note( - `Reasoning effort set to ${step.effort} — applies to your next message.`, - ); + props.store.note(`Reasoning effort set to ${label} — applies to your next message.`); } return; + } case 'state': applyEffortPicker(step.state); return; @@ -1548,6 +1552,8 @@ export function ChatApp(props: Readonly): ReactElement { {/* The standalone `/effort` overlay (ADR-0066 §6) — the shared tier list; `Esc` cancels (not a back-out). */} {effortPicker !== undefined && ( { + // GitHub Actions (and most CI) sets `CI`. The wall-clock SHAPE ratio below is GC-dominated on a shared runner and + // gives false positives there (see its comment); it runs on a warm local box, where the ratio measures the + // algorithm. The latency-budget test stays on both. A dedicated perf lane can still FORCE the shape guard under + // CI with `RUN_WRAP_SCALE_GUARD=1` (on a runner it controls), so the guard is available to automation, just not + // to the shared per-PR gate. + const SKIP_SHAPE_GUARD = + Boolean(process.env['CI']) && process.env['RUN_WRAP_SCALE_GUARD'] !== '1'; + const conversation = (turns: number): TranscriptEntry[] => Array.from({ length: turns }, (_, i) => [ { role: 'user' as const, text: `question ${i}: ${'x'.repeat(200)}` }, @@ -1143,30 +1151,30 @@ describe('the reseat carry is O(n) — the perf claim ADR-0059 makes', () => { expect(elapsed).toBeLessThan(100); // a user-initiated switch must feel instant }); - it('scales LINEARLY, not quadratically — 8x the conversation is not 64x the wrap', () => { - // The guard is the SHAPE. An accidental O(n^2) — a per-entry re-scan of everything before it — is the regression - // that makes a long chat unusable, and the budget above cannot catch it: quadratic at the budget's 200-message - // case is still only a few ms. This ratio is the only thing standing between that bug and a release. - // - // BOTH the scale factor and the threshold are MEASURED, not reasoned about. The realistic form of this bug does - // something CHEAP per pair (reads a length, compares an id), so the test's whole job is separating a cheap - // quadratic from linear — and at a 4x scale factor it cannot: injecting a real O(n^2) re-scan moved the ratio - // from 4.0 to only ~5, which any threshold loose enough to be stable waves straight through. At 8x they separate: - // - // real code ...... 4.60 4.60 4.68 4.76 4.80 4.81 (six runs — sub-8x, since the per-entry cost - // injected O(n^2) ................................ 16.62 amortizes as n grows; the SHAPE is what matters) - // - // 11 sits between them with 2.3x headroom over the worst clean reading and a decisive margin below the quadratic. - // A ratio of MINIMUMS (see `minWrapMs`) is also largely independent of how fast the runner is — numerator and - // denominator scale together — so this asserts something about the algorithm, not about the machine. - // - // NOTE for anyone re-verifying this by breaking it: an injected re-scan with NO side effect gets eliminated by V8 - // and the test will pass, proving nothing. Make the loop observable (accumulate into a value the function reads). - wrapTranscript(conversation(200), 80); // warm the JIT, not the entry cache - - const small = minWrapMs(() => conversation(200), 9); // 400 entries - const large = minWrapMs(() => conversation(1600), 9); // 8x: 3200 entries - - expect(large / small).toBeLessThan(11); - }); + // LOCAL-ONLY (skipped under CI). The guard is the SHAPE: an accidental O(n^2) — a per-entry re-scan of everything + // before it — makes a long chat unusable, and the latency budget above cannot catch it (quadratic at 200 messages + // is still a few ms). Only a scale RATIO can. But a ratio of wall-clock is a ratio of GC as much as of algorithm: + // the 8x-larger case allocates 8x as much and so pays GC the small case dodges, and on a shared CI runner NO large + // sample runs GC-free, so even a ratio of MINIMUMS is unbounded. This exact test read 11.47 and then 22.17 on two + // successive CI runs for PROVABLY-LINEAR code — the second above the ~16.6 the injected quadratic produces, i.e. a + // linear reading indistinguishable from the bug it exists to catch. A test that is red regardless of the code is + // not a guard, so it does not gate a PR; it runs where the ratio is stable (a warm local box: real ~4.8, injected + // O(n^2) ~16.6) and belongs on a dedicated nightly perf lane, not the per-PR gate. The CI-side regression signal + // for this same path is the latency budget above (`< 100ms`, 3–4x headroom, green on both CI runs). + // + // NOTE for anyone re-verifying this by breaking it: an injected re-scan with NO side effect gets eliminated by V8 + // and the test will pass, proving nothing. Make the loop observable (accumulate into a value the function reads). + it.skipIf(SKIP_SHAPE_GUARD)( + 'scales LINEARLY, not quadratically — 8x the conversation is not 64x the wrap', + () => { + wrapTranscript(conversation(200), 80); // warm the JIT, not the entry cache + + const small = minWrapMs(() => conversation(200), 15); // 400 entries + const large = minWrapMs(() => conversation(1600), 15); // 8x: 3200 entries + + // 13 sits between a warm-local clean reading (~5) and the ~16.6 quadratic signal — decisive on the box where the + // ratio is a measure of the algorithm rather than of the runner's memory pressure. + expect(large / small).toBeLessThan(13); + }, + ); }); diff --git a/apps/cli/src/render/tui/chat-projection.ts b/apps/cli/src/render/tui/chat-projection.ts index cbfd109c..4a575f0d 100644 --- a/apps/cli/src/render/tui/chat-projection.ts +++ b/apps/cli/src/render/tui/chat-projection.ts @@ -2,6 +2,8 @@ import type { ToolApprovalRequest } from '@relavium/core'; import type { ReasoningEffort } from '@relavium/shared'; import { MODE_LABEL, type ChatMode } from '../../chat/chat-mode.js'; +import { effortRowLabel } from '../../chat/effort-notice.js'; +import { sanitizeInline, stripTerminalControls } from '../sanitize.js'; import { formatCostUsd, formatDuration, formatElapsed, formatTokens } from './format.js'; import type { SessionViewState, @@ -19,58 +21,7 @@ import { wrapText, type DisplayLine } from './viewport.js'; * arranges them in `ink` `Box`/`Text`. */ -/* eslint-disable no-control-regex */ -// The terminal-sanitizer matcher, composed from one named source fragment per ESC-introduced escape family so a -// future edit touches a single, clearly-scoped family rather than one dense expression. SECURITY-SENSITIVE: the -// alternation below is byte-for-byte the prior single literal — keep behavior exact (covered by the strip tests). -/** OSC (`ESC ]`) + DCS/PM/APC/SOS (`ESC P`/`^`/`_`/`X`) string sequences, sharing one body. The terminator (BEL - * or ST) is REQUIRED, so an UNterminated introducer does NOT match here — it falls through to {@link ESC_2BYTE} - * (only its 2-byte form stripped), leaving the following text visible. An optional terminator would instead - * swallow the whole remainder of the string, silently erasing legitimate model output. */ -const ESC_STRING_SEQ = String.raw`\x1b[\]P^_X][^\x07\x1b]*(?:\x07|\x1b\\)`; -/** CSI (`ESC [` colors/cursor) — the parameter / intermediate / final-byte form. */ -const ESC_CSI = String.raw`\x1b\[[0-?]*[ -/]*[@-~]`; -/** Any remaining 2-byte `ESC <0x40–0x5f>` escape (incl. an unterminated OSC/DCS/PM/APC introducer). */ -const ESC_2BYTE = String.raw`\x1b[@-Z\\-_]`; -/** Every ESC-introduced sequence, composed from the named families above. */ -const ESC_SEQUENCES = new RegExp(`${ESC_STRING_SEQ}|${ESC_CSI}|${ESC_2BYTE}`, 'g'); -/** Remaining C0/C1 control bytes — keep only TAB (\x09) and LINE FEED (\x0a). */ -const BARE_CONTROLS = /[\x00-\x08\x0b-\x1f\x7f-\x9f]/g; -/* eslint-enable no-control-regex */ - -/** - * Unicode bidirectional / directional FORMAT controls — the Trojan-Source floor (CVE-2021-42574; 2.5-close - * Step 14). These code points live ABOVE the C0/C1 range {@link BARE_CONTROLS} strips, so they survive it — yet - * they REORDER how a terminal visually renders a line, letting streamed model output or pasted input display in - * an order that differs from its logical bytes (a spoofed path/command in an approval prompt, a hidden argument, - * a reversed URL). The set is the standard Trojan-Source family: the embeddings/overrides U+202A–202E - * (LRE/RLE/PDF/LRO/RLO), the isolates U+2066–2069 (LRI/RLI/FSI/PDI), and the marks LRM (U+200E) / RLM (U+200F) / - * ALM (U+061C). ZWJ/ZWNJ (U+200D/U+200C) are deliberately NOT stripped — they are legitimate in emoji sequences - * and in Indic/Arabic/Persian shaping. Not in `no-control-regex`'s C0/C1 range, so no eslint-disable is needed. - * Written with `\u` escapes (never literal bidi bytes) so the source itself carries no Trojan-Source hazard. - */ -const BIDI_CONTROLS = /[\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]/g; - -/** - * Strip terminal control sequences from text that will be written to a terminal — so model output (or pasted - * input) cannot inject ANSI/OSC escapes (colors, a cursor jump, a window-title/clipboard/hyperlink write, a - * `\r` line-overwrite) NOR spoof the visual line order with Unicode bidi controls (the Trojan-Source floor). - * Applied at the **display** boundary only; the PERSISTED transcript keeps the raw text (it is user/model data, - * not displayed back through a shell). Keeps printable text plus tabs and newlines. - */ -export function stripTerminalControls(text: string): string { - return text.replace(ESC_SEQUENCES, '').replace(BARE_CONTROLS, '').replace(BIDI_CONTROLS, ''); -} - -/** - * Sanitize a single-line dynamic identifier (a tool id, the bound model name, a persisted session title) for - * terminal display: strip the ANSI/OSC/control bytes {@link stripTerminalControls} removes, then collapse any - * surviving tab/newline to a single space so the value cannot spoof extra terminal lines or columns inside a - * one-line annotation/footer/list row. - */ -export function sanitizeInline(text: string): string { - return stripTerminalControls(text).replace(/[\t\n]+/g, ' '); -} +export { sanitizeInline, stripTerminalControls }; /** The max length of a typed denial reason (`[c]`, Step 14) — bounds what rides the `ToolDeniedByUserError` into * the turn outcome / `--json` / logs, so a pasted wall of text can't blow up one error line. */ @@ -648,7 +599,11 @@ export function formatSessionFooterWithMode( const base = formatSessionFooter(state); const modePart = `${MODE_LABEL[mode]} mode`; const withMode = base.length > 0 ? `${base} · ${modePart}` : modePart; - return reasoningEffort === undefined ? withMode : `${withMode} · effort: ${reasoningEffort}`; + if (reasoningEffort === undefined) return withMode; + // A budget model's bound `medium` reads "on" here, matching the picker (ADR-0066 amendment); a graded tier reads + // its own name. `state.model` absent (pre-session) ⇒ the label falls back to the tier name. + const effortLabel = effortRowLabel(state.model ?? '', reasoningEffort).label; + return `${withMode} · effort: ${effortLabel}`; } /** diff --git a/apps/cli/src/render/tui/effort-picker.test.ts b/apps/cli/src/render/tui/effort-picker.test.ts index f0dd67cf..9be0600b 100644 --- a/apps/cli/src/render/tui/effort-picker.test.ts +++ b/apps/cli/src/render/tui/effort-picker.test.ts @@ -1,6 +1,14 @@ import { REASONING_EFFORTS } from '@relavium/shared'; import { describe, expect, it } from 'vitest'; +import { + effortRejectedNote, + effortRowLabel, + effortTiersFor, + effortUnavailableNote, + projectEffortToRow, +} from '../../chat/effort-notice.js'; + import { canControlEffort, foldEffortPickerKey, @@ -9,7 +17,16 @@ import { } from './effort-picker.js'; function state(partial: Partial = {}): EffortPickerState { - return { selected: 0, current: undefined, model: 'deepseek-v4-flash', ...partial }; + // A SYNTHETIC full tier list, so the fold/navigation tests below exercise every arrow move regardless of what a + // real model's projection collapses to (that projection — dedupe, budget off/on — is covered by the + // `effortTiersFor` tests above). The `model` is only carried through the fold, not re-projected here. + return { + tiers: [...REASONING_EFFORTS], + selected: 0, + current: undefined, + model: 'deepseek-v4-flash', + ...partial, + }; } describe('canControlEffort', () => { @@ -21,21 +38,97 @@ describe('canControlEffort', () => { it('is false for a non-reasoning model, an unbound model, or an unwired setter', () => { expect(canControlEffort('gpt-4o', true)).toBe(false); // gpt-4o is not a reasoning model expect(canControlEffort('deepseek-chat', true)).toBe(false); // the legacy non-thinking alias + // …and a model that REASONS but publishes no controllable tier is also false: opening a five-row overlay for + // it (as the old `supportsReasoning` gate did) offers five rows the model does not take. + expect(canControlEffort('deepseek-reasoner', true)).toBe(false); expect(canControlEffort(undefined, true)).toBe(false); // no bound model yet (pre session:started) expect(canControlEffort('claude-opus-4-8', false)).toBe(false); // setter not wired (a non-interactive driver) }); }); +describe('effortTiersFor + initialEffortPickerState — the picker can no longer OFFER an illegal tier', () => { + it('THE BUG: gpt-5.4-pro is offered {medium, high, max} — NOT `low`, NOT `off`', () => { + // The picker used to show all five to every reasoning model. Picking `low` on this one produced an opaque + // provider 400 — the maintainer's report. The interactive path cannot produce it now, because it is not a row. + expect(effortTiersFor('gpt-5.4-pro')).toEqual(['medium', 'high', 'max']); + }); + + it('gpt-5-pro is offered ONE tier — a single-row list, which a boolean could never have produced', () => { + expect(effortTiersFor('gpt-5-pro')).toEqual(['high']); + }); + + it('a model with no controllable tier, and an unknown model, offer NOTHING', () => { + expect(effortTiersFor('deepseek-reasoner')).toEqual([]); // reasons; publishes no control + expect(effortTiersFor('some-custom-endpoint-model')).toEqual([]); // not in the catalog at all + }); + + it('the tiers are in CANONICAL order (off → max), never the accepted-set insertion order', () => { + expect(effortTiersFor('claude-opus-4-8')).toEqual(['off', 'low', 'medium', 'high', 'max']); + }); +}); + +describe('effortTiersFor — the GENERAL presentation rule: one row per distinct outcome (ADR-0066 amendment)', () => { + it('BUDGET model → off/on, never a fake 5-tier ladder (claude-haiku-4-5)', () => { + // A continuous token budget has no meaningful discrete rungs. Its picker is a two-row off/on; "on" is the + // canonical `medium` tier (a real member of the accepted set, so the accept sends a value the gate accepts). + expect(effortTiersFor('claude-haiku-4-5')).toEqual(['off', 'medium']); + // …and 'medium' READS "on" (the label the whole surface shows). + expect(effortRowLabel('claude-haiku-4-5', 'medium')).toEqual({ + label: 'on', + hint: 'reasoning on', + }); + expect(effortRowLabel('claude-haiku-4-5', 'off').label).toBe('off'); + }); + + it('BUDGET model that cannot be turned off → NO overlay (gemini-2.5-pro)', () => { + // Google: "N/A: Cannot disable thinking". With no `off` and no discrete ladder there is nothing to toggle, so + // the list is empty and the overlay never opens (was: five rows, one of them the rejected `off`). + expect(effortTiersFor('gemini-2.5-pro')).toEqual([]); + expect(canControlEffort('gemini-2.5-pro', true)).toBe(false); + }); + + it('GRADED model → deduped by distinct WIRE value: deepseek-v4-pro low/medium/high all send `high`', () => { + // DeepSeek publishes only two effort levels; Relavium's low/medium/high all map to `high`, so they collapse to + // ONE row — the representative reads "high" (the name that matches the wire), not "low". + expect(effortTiersFor('deepseek-v4-pro')).toEqual(['off', 'high', 'max']); + expect(effortRowLabel('deepseek-v4-pro', 'high').label).toBe('high'); // a graded tier keeps its own name + }); + + it('GRADED model → Gemini `max` coarsens onto `high`, so it collapses (gemini-3.5-flash)', () => { + // GEMINI_WIRE maps `max` onto `high`, so `high` and `max` are the same call — keep the name-match `high`. + expect(effortTiersFor('gemini-3.5-flash')).toEqual(['low', 'medium', 'high']); + }); + + it('GRADED model with all-distinct wires is UNCHANGED (claude-opus-4-8 full ladder, gpt-5.4-pro)', () => { + expect(effortTiersFor('claude-opus-4-8')).toEqual(['off', 'low', 'medium', 'high', 'max']); + expect(effortTiersFor('gpt-5.4-pro')).toEqual(['medium', 'high', 'max']); + }); +}); + describe('initialEffortPickerState', () => { - it('opens on the bound effort when set', () => { - // 'high' is index 3 in [off, low, medium, high, max]. - expect(initialEffortPickerState('m', 'high').selected).toBe(REASONING_EFFORTS.indexOf('high')); - expect(initialEffortPickerState('m', 'off').selected).toBe(REASONING_EFFORTS.indexOf('off')); + it('opens on the bound effort when the model accepts it', () => { + const s = initialEffortPickerState('claude-opus-4-8', 'high'); + expect(s.tiers[s.selected]).toBe('high'); }); - it('opens on a neutral middle tier (medium) when no effort is bound', () => { - const s = initialEffortPickerState('m', undefined); - expect(REASONING_EFFORTS[s.selected]).toBe('medium'); + it('opens on `medium` when nothing is bound and the model accepts medium', () => { + const s = initialEffortPickerState('claude-opus-4-8', undefined); + expect(s.tiers[s.selected]).toBe('medium'); + }); + + it('NEVER opens on a row the model does not have — the old `?? medium` default could', () => { + // gpt-5-pro accepts only `high`. The old code took `REASONING_EFFORTS.indexOf('medium')` = 2 and highlighted + // a row that, in a one-row list, does not exist. The highlight now always lands on a tier the model takes, + // because the list only contains tiers the model takes. + const s = initialEffortPickerState('gpt-5-pro', undefined); + expect(s.tiers).toEqual(['high']); + expect(s.selected).toBe(0); + expect(s.tiers[s.selected]).toBe('high'); + + // …and the same when the BOUND tier is one the model rejects (a model swap can leave a stale tier bound). + const stale = initialEffortPickerState('gpt-5.4-pro', 'low'); // gpt-5.4-pro rejects `low` + expect(stale.tiers).not.toContain('low'); + expect(stale.tiers[stale.selected]).toBe('medium'); // the first row, never an index into thin air }); it('carries the model + current through', () => { @@ -43,6 +136,58 @@ describe('initialEffortPickerState', () => { expect(s.model).toBe('deepseek-v4-pro'); expect(s.current).toBe('max'); }); + + // The Bug-1 opening-highlight regression: after the amendment collapsed graded/budget models to one row per + // DISTINCT outcome, `medium` is no longer necessarily a row. The old `tiers.indexOf(neutral)` returned -1 → + // clamped to 0 = the FIRST row, which for a disableable model is `off`. So a fresh picker highlighted `off`, and + // an immediate Enter WROTE `off` — silently turning reasoning off on a model the user opened to turn it up. + it('a collapsed graded model opens on the neutral tier’s REPRESENTATIVE row, never `off`', () => { + // deepseek-v4-pro rows = [off, high, max]; `medium` maps to the `high` wire, so it collapses onto that row. + const s = initialEffortPickerState('deepseek-v4-pro', undefined); + expect(s.tiers).toEqual(['off', 'high', 'max']); + expect(s.tiers[s.selected]).toBe('high'); + expect(s.tiers[s.selected]).not.toBe('off'); + }); + + it('a budget model opens on the `on` row (the canonical medium), never `off`', () => { + // claude-haiku-4-5 is a toggle → rows = [off, medium]; `medium` is displayed "on". It must open on `on`. + const s = initialEffortPickerState('claude-haiku-4-5', undefined); + expect(s.tiers).toEqual(['off', 'medium']); + expect(s.tiers[s.selected]).toBe('medium'); + }); + + it('a BOUND tier that was deduped away opens on its representative row', () => { + // deepseek bound to `low` (an authored agent / config default) — `low` is not a row; open on `high`, its twin. + const graded = initialEffortPickerState('deepseek-v4-pro', 'low'); + expect(graded.tiers[graded.selected]).toBe('high'); + // a budget model bound to `high` → the `on` row. + const budget = initialEffortPickerState('claude-haiku-4-5', 'high'); + expect(budget.tiers[budget.selected]).toBe('medium'); + }); +}); + +describe('projectEffortToRow — a tier maps to the row that REPRESENTS it (ADR-0066/0071 amendment)', () => { + it('returns the tier itself when it is a row', () => { + expect(projectEffortToRow('deepseek-v4-pro', ['off', 'high', 'max'], 'high')).toBe('high'); + expect(projectEffortToRow('deepseek-v4-pro', ['off', 'high', 'max'], 'off')).toBe('off'); + }); + + it('projects a deduped graded tier onto its wire twin', () => { + // `medium` and `low` both wire to `high` for deepseek → both project onto the `high` row. + expect(projectEffortToRow('deepseek-v4-pro', ['off', 'high', 'max'], 'medium')).toBe('high'); + expect(projectEffortToRow('deepseek-v4-pro', ['off', 'high', 'max'], 'low')).toBe('high'); + }); + + it('projects any on-tier of a budget model onto the canonical `on` row', () => { + expect(projectEffortToRow('claude-haiku-4-5', ['off', 'medium'], 'high')).toBe('medium'); + expect(projectEffortToRow('claude-haiku-4-5', ['off', 'medium'], 'max')).toBe('medium'); + expect(projectEffortToRow('claude-haiku-4-5', ['off', 'medium'], 'off')).toBe('off'); + }); + + it('returns undefined when the tier has no representative (nothing to highlight)', () => { + // a model whose rows do not include `off` and whose wire twin is absent → no row represents `off`. + expect(projectEffortToRow('gpt-5-pro', ['high'], 'off')).toBeUndefined(); + }); }); describe('foldEffortPickerKey', () => { @@ -96,3 +241,74 @@ describe('foldEffortPickerKey', () => { if (step.kind === 'state') expect(step.state).toBe(s); // same ref — no needless re-render/hint-wipe }); }); + +/** + * ONE PREDICATE, ASKED BY EVERY SURFACE (ADR-0071 §6). + * + * These lock the facts the two blockers turned on. The CLI used to carry four answers to "can the user set effort + * on this model": this list, the `/models` sub-step, the engine's gate, and — the odd one out — `modelSupportsReasoning`, + * an id heuristic over the hand-typed pricing table that the footer and the `/effort` command still read. An + * adversarial review computed the disagreement: **sixteen shipped models**, and on each one the user was either + * billed for reasoning with no indicator, or told a tier applied that the engine then silently dropped. + */ +describe('effortTiersFor — the one answer every surface reads', () => { + it('lists ONLY the tiers the model publishes — `gpt-5.4-pro` takes neither `low` nor `off`', () => { + // The maintainer's original bug report. The typed `/effort low` used to validate against the fixed five, say + // "applies to your next message", show `low` in the footer — and send nothing, because the gate dropped it. + const tiers = effortTiersFor('gpt-5.4-pro'); // catalog: effortValues ['medium','high','xhigh'] + expect(tiers).toEqual(['medium', 'high', 'max']); + expect(tiers).not.toContain('low'); + expect(tiers).not.toContain('off'); // on OpenAI, `off` IS an effort value ('none') — and this model omits it + }); + + it("is CANONICALLY ordered, never the accepted Set's insertion order", () => { + // `acceptedTiers` adds `off` LAST (it rides a different axis), so a raw spread of the Set would render + // `low, medium, high, max, off` — the rows must read in tier order, every time, on every model. + const tiers = effortTiersFor('claude-opus-4-8'); + expect(tiers).toEqual(REASONING_EFFORTS.filter((t) => tiers.includes(t))); + expect(tiers[0]).toBe('off'); + }); + + it('a model the OLD boolean called non-reasoning still has tiers — the sixteen-model disagreement', () => { + // `claude-sonnet-4-5` is not in the hand-typed `MODEL_PRICING` at all, so the heuristic answered `false` — while + // the picker offered five tiers and the engine sent the chosen one. The footer stayed blank and the user paid + // for extended thinking with nothing on screen to say it was on. + expect(effortTiersFor('claude-sonnet-4-5').length).toBeGreaterThan(0); + expect(canControlEffort('claude-sonnet-4-5', true)).toBe(true); + }); + + it('EMPTY for a model with no knob, and for one the catalog does not carry — but they are not the same sentence', () => { + expect(effortTiersFor('deepseek-reasoner')).toEqual([]); // reasoning: {} — reasons, publishes nothing + expect(effortTiersFor('some-custom-endpoint-model')).toEqual([]); // not in the catalog at all + expect(canControlEffort('deepseek-reasoner', true)).toBe(false); + + // Same empty list, different ACTION: one is fixable by a catalog refresh, the other never will be. The old + // heuristic said "no reasoning control" for both, which tells the user nothing they can do. + expect(effortUnavailableNote('deepseek-reasoner')).toContain( + 'publishes no controllable reasoning tier', + ); + expect(effortUnavailableNote('some-custom-endpoint-model')).toContain('models refresh'); + }); +}); + +describe('effortRejectedNote — a rejection the user cannot see is worse than the 400 it replaced', () => { + it('names the tier refused AND the tiers that would work, in canonical order', () => { + const note = effortRejectedNote('gpt-5.4-pro', 'low', effortTiersFor('gpt-5.4-pro')); + expect(note).toContain("does not accept reasoning effort 'low'"); + expect(note).toContain('it takes medium, high, max'); + expect(note).toContain('No tier is sent.'); // the consequence, stated — not left for the bill to reveal + }); + + it("takes the engine gate's Set as readily as the picker's array — one sentence, both callers", () => { + // `EffortGateResult.rejected` carries `accepted` as an array; the CLI's own list is an array too; the seam's + // `effortTiersFor` hands back a Set. All three reach this function, and all three must read the same. + const fromSet = effortRejectedNote('gpt-5.4-pro', 'off', new Set(['high', 'medium'] as const)); + expect(fromSet).toContain('it takes medium, high'); // canonical order, NOT the Set's ['high','medium'] + }); + + it('degrades to the unavailable note when NOTHING is accepted — never "it takes " with an empty list', () => { + expect(effortRejectedNote('deepseek-reasoner', 'high', [])).toBe( + effortUnavailableNote('deepseek-reasoner'), + ); + }); +}); diff --git a/apps/cli/src/render/tui/effort-picker.ts b/apps/cli/src/render/tui/effort-picker.ts index a35072b0..b56c11d3 100644 --- a/apps/cli/src/render/tui/effort-picker.ts +++ b/apps/cli/src/render/tui/effort-picker.ts @@ -1,23 +1,35 @@ -import { modelSupportsReasoning } from '@relavium/llm'; -import { REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; +import { type ReasoningEffort } from '@relavium/shared'; +import { effortTiersFor, projectEffortToRow } from '../../chat/effort-notice.js'; import type { ModelPickerKey } from './model-picker.js'; /** * The standalone `/effort` overlay ([ADR-0066](../../../../../docs/decisions/0066-normalized-reasoning-effort-control.md) §6) * — a keyboard-owning submode (like the `/models` picker + the `/` palette) that lists the reasoning-effort tiers and, * on Enter, pushes the chosen tier as the session's per-turn override via the surface's effort setter (NO reseat — - * effort changes neither provider, pricing, nor the plan). It is a FIXED five-row list: no catalog, filter, or - * refresh — so it needs no port and no async load, unlike the `/models` picker. + * effort changes neither provider, pricing, nor the plan). + * + * The rows come from the CATALOG, per model ([ADR-0071](../../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6) + * — it used to be a fixed five-row list, and that was the bug: `gpt-5-pro` accepts exactly one of them. + * It still needs no port and no async load; the catalog is a synchronous embedded snapshot. * * Two surfaces route the SAME fold (standalone `relavium chat` + the in-Home live chat); the accept is UNIFORM (call * `onSetEffort` + note), so — unlike the model picker's surface-divergent reseat-vs-default-write accept — no - * per-surface branching lives here. Offered ONLY when the bound model is reasoning-capable; a non-reasoning model - * never opens this overlay (the surface falls through to the `/effort` notice instead). The pure fold + state live - * here; the ink view is the shared {@link effort-tier-list.tsx} `EffortTierList`. + * per-surface branching lives here. Offered only when the model has a tier to offer; a model with none never opens + * this overlay (the surface shows `effortUnavailableNote` instead). The pure fold + state live here; the ink + * view is the shared {@link effort-tier-list.tsx} `EffortTierList`. */ export interface EffortPickerState { - /** The highlighted index into {@link REASONING_EFFORTS}. */ + /** + * The tiers THIS MODEL accepts, in canonical order ([ADR-0071](../../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * It used to be the fixed five. That is the F3 bug: `gpt-5.4-pro` rejects `low`, `gpt-5-pro` accepts only + * `high`, and `gemini-2.5-pro` cannot be turned off — yet every one of them was offered all five, and picking + * the wrong one produced an opaque provider 400. **The interactive path can no longer produce an illegal tier, + * because an illegal tier is not on the list.** + */ + readonly tiers: readonly ReasoningEffort[]; + /** The highlighted index into {@link EffortPickerState.tiers}. */ readonly selected: number; /** The session's currently-bound effort (the `✓` + the opening highlight); `undefined` ⇒ the provider default, so * the list opens on a neutral middle tier. */ @@ -34,15 +46,15 @@ export type EffortPickerStep = /** * Whether a surface should open the interactive `/effort` overlay: the per-turn effort setter must be wired AND the - * bound model reasoning-capable. A non-reasoning model (or no bound model yet / no setter) returns false, so the - * surface falls through to the informational `/effort` notice rather than opening a dead overlay. Shared by the - * standalone chat + the in-Home chat so the gate can never diverge between them. + * bound model must have at least one tier to OFFER. A model with none — no reasoning, no published knob, or no + * catalog row at all — returns false, so the surface shows {@link effortUnavailableNote} rather than a dead overlay. + * Shared by the standalone chat + the in-Home chat so the gate can never diverge between them. */ export function canControlEffort(model: string | undefined, setterWired: boolean): boolean { - return setterWired && model !== undefined && modelSupportsReasoning(model); + return setterWired && model !== undefined && effortTiersFor(model).length > 0; } -/** Clamp an index to `0..count-1` (or 0 when the list is empty — never for the fixed non-empty tier list). */ +/** Clamp an index to `0..count-1`, or 0 for an empty list (a model with no tiers never opens the overlay). */ function clampSelection(index: number, count: number): number { if (count <= 0) return 0; return Math.max(0, Math.min(index, count - 1)); @@ -53,12 +65,21 @@ export function initialEffortPickerState( model: string, current: ReasoningEffort | undefined, ): EffortPickerState { - return { selected: Math.max(0, REASONING_EFFORTS.indexOf(current ?? 'medium')), current, model }; + const tiers = effortTiersFor(model); + // The opening highlight lands on the bound tier, or the neutral `medium` when nothing is bound — but PROJECTED + // onto a surviving row first (ADR-0066 amendment): a graded-collapsed model (deepseek [off,high,max]) has no + // `medium` row, and a bare `indexOf('medium')` = -1 → 0 = `off` would silently open on reasoning-DISABLED. The + // projection folds `medium`→`high` (its wire twin) / a budget model's neutral→the `on` row, so the cursor never + // lands on `off` by accident. `undefined` (nothing represents it) still clamps to 0. + const target = projectEffortToRow(model, tiers, current ?? 'medium'); + const index = target === undefined ? 0 : tiers.indexOf(target); + return { tiers, selected: Math.max(0, index), current, model }; } /** * Fold one keystroke into the open effort overlay (the keyboard-owning contract, mirroring the model-picker fold). - * `Ctrl-C`/`Esc` cancel (nothing applied); `↑`/`↓` move over {@link REASONING_EFFORTS}; `Enter` accepts the + * `Ctrl-C`/`Esc` cancel (nothing applied); `↑`/`↓` move over {@link EffortPickerState.tiers} — the tiers THIS + * model accepts, not the fixed five; `Enter` accepts the * highlighted tier. It is a fixed list with no filter/refresh, so every other key is inert (returns the same state). */ export function foldEffortPickerKey( @@ -70,17 +91,17 @@ export function foldEffortPickerKey( if (key.upArrow === true) { return { kind: 'state', - state: { ...state, selected: clampSelection(state.selected - 1, REASONING_EFFORTS.length) }, + state: { ...state, selected: clampSelection(state.selected - 1, state.tiers.length) }, }; } if (key.downArrow === true) { return { kind: 'state', - state: { ...state, selected: clampSelection(state.selected + 1, REASONING_EFFORTS.length) }, + state: { ...state, selected: clampSelection(state.selected + 1, state.tiers.length) }, }; } if (key.return === true) { - const effort = REASONING_EFFORTS[clampSelection(state.selected, REASONING_EFFORTS.length)]; + const effort = state.tiers[clampSelection(state.selected, state.tiers.length)]; // Defensive: an out-of-range highlight (never expected — the fold clamps every move) closes rather than emitting // a malformed accept. return effort === undefined ? { kind: 'close' } : { kind: 'accept', effort }; diff --git a/apps/cli/src/render/tui/effort-tier-list.tsx b/apps/cli/src/render/tui/effort-tier-list.tsx index 07293348..aae0d18e 100644 --- a/apps/cli/src/render/tui/effort-tier-list.tsx +++ b/apps/cli/src/render/tui/effort-tier-list.tsx @@ -1,13 +1,14 @@ import { Box, Text } from 'ink'; import type { ReactElement } from 'react'; -import { EFFORT_TIER_HINT, REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; +import { type ReasoningEffort } from '@relavium/shared'; +import { effortRowLabel, projectEffortToRow } from '../../chat/effort-notice.js'; import { sanitizeInline } from './chat-projection.js'; import { colorProps, dimProps } from './projection.js'; /** - * The shared reasoning-effort tier list (ADR-0066) — the fixed five-row picker body used by BOTH the `/models` + * The shared reasoning-effort tier list (ADR-0066/0071) — the per-model picker body used by BOTH the `/models` * effort sub-step ({@link model-picker-view.tsx}'s `EffortSubList`, reached after choosing a reasoning model on a * reseat surface) and the standalone `/effort` overlay ({@link effort-picker.ts}). One canonical presentation so the * two entry points can never drift: each tier + its one-line hint, the highlighted row in cyan, and a `✓` on the @@ -17,6 +18,14 @@ import { colorProps, dimProps } from './projection.js'; * PURE: it owns no `useInput`; the surface routes keys to the fold and re-renders this from the resulting state. */ export interface EffortTierListProps { + /** + * The tiers to show — the ones THIS MODEL accepts (ADR-0071 §6), never the fixed five. `gpt-5.4-pro` rejects + * `low` and `gemini-2.5-pro` cannot be turned off; offering a row the provider would 400 on is the bug. + */ + readonly tiers: readonly ReasoningEffort[]; + /** The bound model id — decides a tier's DISPLAY label (a budget model's `medium` reads "on"; ADR-0066 amendment). + * Not shown itself; the header suffix names the model. */ + readonly model: string; /** The highlighted tier index (already clamped by the caller's fold, but re-clamped here for display safety). */ readonly selected: number; /** The session's currently-bound effort — the `✓` marker; `undefined` ⇒ no tier bound (the provider default). */ @@ -30,10 +39,13 @@ export interface EffortTierListProps { } export function EffortTierList(props: Readonly): ReactElement { - const { selected, current, labelSuffix, footer, color } = props; + const { tiers, model, selected, current, labelSuffix, footer, color } = props; // Re-clamp for display: a caller could pass an out-of-range index (a shrunk source, a stale render) — never index - // past the fixed five-row list. - const highlighted = Math.max(0, Math.min(selected, REASONING_EFFORTS.length - 1)); + // past the end of a list whose length now varies per model (`gpt-5-pro` has ONE row). + const highlighted = Math.max(0, Math.min(selected, tiers.length - 1)); + // The ✓ lands on the row that REPRESENTS the bound tier — projected, so a bound tier that was deduped away + // (deepseek `low` → the `high` row) or collapsed (a budget model's bound `high` → the `on` row) still marks a row. + const currentRow = current === undefined ? undefined : projectEffortToRow(model, tiers, current); const suffix = labelSuffix === undefined || labelSuffix.length === 0 ? '' : ` · ${labelSuffix}`; return ( @@ -41,14 +53,15 @@ export function EffortTierList(props: Readonly): ReactEleme Reasoning effort {sanitizeInline(suffix)} - {REASONING_EFFORTS.map((effort, index) => { + {tiers.map((effort, index) => { const isSelected = index === highlighted; - const isCurrent = effort === current; + const isCurrent = effort === currentRow; const rowColor = isSelected ? colorProps(color, 'cyan') : {}; + const { label, hint } = effortRowLabel(model, effort); return ( - {`${isSelected ? '›' : ' '} ${isCurrent ? '✓' : ' '} ${effort} · `} - {EFFORT_TIER_HINT[effort]} + {`${isSelected ? '›' : ' '} ${isCurrent ? '✓' : ' '} ${label} · `} + {hint} ); })} diff --git a/apps/cli/src/render/tui/home-app.test.tsx b/apps/cli/src/render/tui/home-app.test.tsx index c9aa7352..496bd6f5 100644 --- a/apps/cli/src/render/tui/home-app.test.tsx +++ b/apps/cli/src/render/tui/home-app.test.tsx @@ -97,11 +97,10 @@ function makeModelsPort( modelId, provider, displayName: modelId, - pricingSource: 'registry', + pricingSource: 'catalog', priceKnown: true, available: true, deprecated: false, - supportsReasoning: false, }; let written: string | undefined; return { diff --git a/apps/cli/src/render/tui/home-app.tsx b/apps/cli/src/render/tui/home-app.tsx index 19bcc1a8..617b289c 100644 --- a/apps/cli/src/render/tui/home-app.tsx +++ b/apps/cli/src/render/tui/home-app.tsx @@ -173,6 +173,8 @@ function ChatRegion( {/* The standalone `/effort` overlay in a live in-Home chat (ADR-0066 §6) — the shared tier list. */} {props.effortPicker !== undefined && ( ; refresh: ReturnType; writeDefault: ReturnType; + latestProvider: () => LlmProviderId | undefined; } { const entries = opts.entries ?? [pickerEntry({ modelId: 'a' }), pickerEntry({ modelId: 'b' })]; let written: string | undefined; + let writtenProvider: LlmProviderId | undefined; let writtenEffort: ReasoningEffort | undefined; const load = vi.fn(() => { if (opts.loadThrows === true) throw new Error('catalog read failed'); @@ -1713,11 +1714,14 @@ function makeModelsPort( }); const refreshIfStale = vi.fn(opts.refreshIfStale ?? (() => Promise.resolve(undefined))); const refresh = vi.fn(opts.refresh ?? (() => Promise.resolve({ providers: [] }))); - const writeDefault = vi.fn((modelId: string, reasoningEffort?: ReasoningEffort) => { - if (opts.writeThrows === true) throw new Error('config write failed'); - written = modelId; - if (reasoningEffort !== undefined) writtenEffort = reasoningEffort; - }); + const writeDefault = vi.fn( + (modelId: string, provider: LlmProviderId, reasoningEffort?: ReasoningEffort) => { + if (opts.writeThrows === true) throw new Error('config write failed'); + written = modelId; + writtenProvider = provider; + if (reasoningEffort !== undefined) writtenEffort = reasoningEffort; + }, + ); const port: HomeModelsPort = { load, refreshIfStale, @@ -1731,7 +1735,14 @@ function makeModelsPort( currentEffort: () => opts.currentEffort ?? writtenEffort, writeDefault, }; - return { port, load, refreshIfStale, refresh, writeDefault }; + return { + port, + load, + refreshIfStale, + refresh, + writeDefault, + latestProvider: () => writtenProvider, + }; } describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => { @@ -1780,21 +1791,37 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => const c = openPicker(port); await flush(); c.handleKey('', ENTER); // accept the selected (available) model - expect(writeDefault).toHaveBeenCalledWith('claude-x', undefined); // non-reasoning ⇒ model only, no effort + expect(writeDefault).toHaveBeenCalledWith('claude-x', 'anthropic', undefined); // model + provider, no effort expect(c.getSnapshot().modelPicker).toBeUndefined(); // the picker closed expect(c.getSnapshot().notice).toContain('Claude X'); // the confirmation names the model expect(c.getSnapshot().notice).toContain('next chat session'); // it is a NEXT-session action, not a live reseat }); + it("bare Home: the picked model's PROVIDER is persisted with it (ADR-0059 — the Bug-3 fix)", async () => { + // The provider is authoritative at pick time (the live `/models` list is per-provider). Persisting it means the + // next chat over a live-discovered id whose prefix the inference cannot place (an openai `chat-latest`) still + // resolves, instead of crashing "cannot infer a provider". The provider written is the picked ENTRY's provider. + const { port, writeDefault, latestProvider } = makeModelsPort({ + entries: [ + pickerEntry({ modelId: 'chat-latest', displayName: 'chat-latest', provider: 'openai' }), + ], + }); + const c = openPicker(port); + await flush(); + c.handleKey('', ENTER); + expect(writeDefault).toHaveBeenCalledWith('chat-latest', 'openai', undefined); + expect(latestProvider()).toBe('openai'); + }); + it('bare Home: a REASONING model offers the effort sub-step, writing BOTH model + effort defaults (ADR-0066 §6)', async () => { // ADR-0066 §6: in the bare Home a reasoning model advances to the effort sub-step (opened on the config effort // default), and accepting writes model + effort together — so the user sets both future-session defaults at once. const { port, writeDefault } = makeModelsPort({ entries: [ pickerEntry({ - modelId: 'deepseek-v4-flash', - displayName: 'DeepSeek V4 Flash', - supportsReasoning: true, + modelId: 'claude-opus-4-8', + displayName: 'Claude Opus 4.8', + provider: 'anthropic', }), ], currentEffort: 'low', // the existing effort default — the sub-list opens highlighted on it @@ -1808,9 +1835,9 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => c.handleKey('', { downArrow: true }); // medium → high c.handleKey('', ENTER); // accept the model + 'high' - expect(writeDefault).toHaveBeenCalledWith('deepseek-v4-flash', 'high'); // BOTH written (one atomic call) + expect(writeDefault).toHaveBeenCalledWith('claude-opus-4-8', 'anthropic', 'high'); // model+provider+effort, one atomic call expect(c.getSnapshot().modelPicker).toBeUndefined(); // closed - expect(c.getSnapshot().notice).toContain('DeepSeek V4 Flash'); + expect(c.getSnapshot().notice).toContain('Claude Opus 4.8'); expect(c.getSnapshot().notice).toContain('effort high'); // the notice names the written effort expect(c.getSnapshot().notice).toContain('next chat session'); }); @@ -1822,9 +1849,9 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => const { port, writeDefault } = makeModelsPort({ entries: [ pickerEntry({ - modelId: 'deepseek-v4-flash', - displayName: 'DeepSeek V4 Flash', - supportsReasoning: true, + modelId: 'claude-opus-4-8', + displayName: 'Claude Opus 4.8', + provider: 'anthropic', }), ], // no currentEffort → port.currentEffort() is undefined @@ -1836,10 +1863,34 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => expect(c.getSnapshot().modelPicker?.currentEffort).toBeUndefined(); // no config effort default c.handleKey('', ENTER); // immediate Enter on the opening highlight (the neutral 'medium') - expect(writeDefault).toHaveBeenCalledWith('deepseek-v4-flash', 'medium'); // the neutral default is written + expect(writeDefault).toHaveBeenCalledWith('claude-opus-4-8', 'anthropic', 'medium'); // the neutral default is written expect(c.getSnapshot().notice).toContain('effort medium'); }); + it('bare Home: a COLLAPSED reasoning model with no prior effort opens on a reasoning-ON row; immediate Enter never writes `off`', async () => { + // The Bug-1 regression guard the opus test above cannot catch (opus keeps all five tiers, so `medium` IS a row). + // deepseek-v4-pro collapses to rows [off, high, max] — `medium` is not a row. Before the projection fix the + // opening highlight fell to index 0 = `off`, so this immediate Enter WROTE `off`, silently disabling reasoning + // on a model the user opened to configure. It must open on `high` (the neutral tier's representative) instead. + const { port, writeDefault } = makeModelsPort({ + entries: [ + pickerEntry({ + modelId: 'deepseek-v4-pro', + displayName: 'DeepSeek V4 Pro', + provider: 'deepseek', + }), + ], + }); + const c = openPicker(port); + await flush(); + c.handleKey('', ENTER); // advance to the effort sub-step + expect(c.getSnapshot().modelPicker?.phase).toBe('effort'); + c.handleKey('', ENTER); // immediate Enter on the opening highlight + + expect(writeDefault).toHaveBeenCalledWith('deepseek-v4-pro', 'deepseek', 'high'); + expect(writeDefault).not.toHaveBeenCalledWith('deepseek-v4-pro', 'deepseek', 'off'); + }); + it('an honest notice when a project/workspace setting overrides the global write (no false success)', async () => { // The write lands on the global file, but the effective default stays the project/workspace override, so the // notice must NOT claim "applies to your next chat session" — it says the override still wins here. @@ -1850,7 +1901,7 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => const c = openPicker(port); await flush(); c.handleKey('', ENTER); - expect(writeDefault).toHaveBeenCalledWith('claude-x', undefined); // non-reasoning ⇒ model only, no effort + expect(writeDefault).toHaveBeenCalledWith('claude-x', 'anthropic', undefined); // model + provider, no effort expect(c.getSnapshot().notice).toContain('overrides it here'); expect(c.getSnapshot().notice).not.toContain('next chat session'); // no false claim of effect }); @@ -2085,9 +2136,7 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => const sessionA = makeSession({ sessionId: 'sess-A', store: boundStore, onSetEffort }); const reseatChat = vi.fn(() => Promise.resolve(makeSession().session)); const { port } = makeModelsPort({ - entries: [ - pickerEntry({ modelId: 'claude-opus-4-8', provider: 'anthropic', supportsReasoning: true }), - ], + entries: [pickerEntry({ modelId: 'claude-opus-4-8', provider: 'anthropic' })], }); const c = createHomeController({ doctorProbes: STUB_DOCTOR_PROBES, @@ -2129,9 +2178,7 @@ describe('the /models picker in the bare Home (2.5.G S7 / ADR-0064 §10)', () => const sessionA = makeSession({ sessionId: 'sess-A', store: boundStore, onSetEffort }); const reseatChat = vi.fn(() => Promise.resolve(makeSession().session)); const { port } = makeModelsPort({ - entries: [ - pickerEntry({ modelId: 'claude-opus-4-8', provider: 'anthropic', supportsReasoning: true }), - ], + entries: [pickerEntry({ modelId: 'claude-opus-4-8', provider: 'anthropic' })], }); const c = createHomeController({ doctorProbes: STUB_DOCTOR_PROBES, diff --git a/apps/cli/src/render/tui/home-controller.ts b/apps/cli/src/render/tui/home-controller.ts index d99130f3..7e721d48 100644 --- a/apps/cli/src/render/tui/home-controller.ts +++ b/apps/cli/src/render/tui/home-controller.ts @@ -16,6 +16,7 @@ import { } from './effort-picker.js'; import { foldModelPickerKey, partialFailureBanner, type ModelPickerState } from './model-picker.js'; import type { ReseatTarget } from '../../commands/chat.js'; +import { effortRowLabel } from '../../chat/effort-notice.js'; import { nextMode, type ChatMode } from '../../chat/chat-mode.js'; import { clearedNotice, modelSwitchNotice } from '../../chat/repl-info.js'; import { formatDoctorReport, runDoctorChecks, type DoctorProbes } from '../../chat/doctor.js'; @@ -206,10 +207,15 @@ export interface HomeModelsPort { /** The current resolved default reasoning-effort tier (ADR-0066 §6) — the `✓`/opening highlight of the bare-Home * effort sub-step; `undefined` ⇒ none set (the sub-list opens on a neutral middle tier). */ currentEffort: () => ReasoningEffort | undefined; - /** Persist the chosen model as the next session's default, and (ADR-0066 §6) — when the effort sub-step ran for a - * reasoning model — its effort tier too, in ONE atomic write (writeGlobalPreferences). An absent `reasoningEffort` - * leaves any prior effort default unchanged. Throws `ConfigError` on a bad write. */ - writeDefault: (modelId: string, reasoningEffort?: ReasoningEffort) => void; + /** Persist the chosen model as the next session's default — WITH its `provider` (ADR-0059: authoritative at pick + * time, so the next chat skips id inference) and (ADR-0066 §6) — when the effort sub-step ran for a reasoning + * model — its effort tier too, in ONE atomic write (writeGlobalPreferences). An absent `reasoningEffort` leaves + * any prior effort default unchanged. Throws `ConfigError` on a bad write. */ + writeDefault: ( + modelId: string, + provider: ReseatTarget['provider'], + reasoningEffort?: ReasoningEffort, + ) => void; } export interface HomeControllerDeps { @@ -717,6 +723,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { phase: 'model', effortStep, pending: undefined, + effortTiers: [], // no model pending yet — populated on the model→effort transition effortSelected: 0, currentEffort, }, @@ -736,7 +743,11 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { ): string => { // The effort (ADR-0066 §6) is written to the SAME layer atomically, so it shares the model's effectiveness — the // suffix just names what was set (a reasoning model went through the effort sub-step; a non-reasoning one did not). - const effort = reasoningEffort === undefined ? '' : ` at effort ${reasoningEffort}`; + // The label reads "on" for a budget model's canonical-on tier, matching the picker (ADR-0066 amendment). + const effort = + reasoningEffort === undefined + ? '' + : ` at effort ${effortRowLabel(modelId, reasoningEffort).label}`; if (effective === modelId) return `Default model set to ${displayName}${effort} — applies to your next chat session.`; if (effective === undefined) { @@ -751,12 +762,13 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { const writeNextSessionDefault = ( modelId: string, displayName: string, + provider: ReseatTarget['provider'], reasoningEffort?: ReasoningEffort, ): void => { const port = deps.models; if (port === undefined) return; try { - port.writeDefault(modelId, reasoningEffort); + port.writeDefault(modelId, provider, reasoningEffort); } catch { // A generic save-failure hint — the actual write target may be a `--config` override, not the canonical // `~/.relavium/config.toml`, so don't name a path the user may not be using. @@ -785,13 +797,15 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { displayName: string, reasoningEffort: ReasoningEffort, ): void => { + const label = effortRowLabel( + active.store.getSnapshot().state.model ?? '', + reasoningEffort, + ).label; if (reasoningEffort !== active.store.getSnapshot().reasoningEffort) { active.onSetEffort?.(reasoningEffort); - active.store.note( - `Reasoning effort set to ${reasoningEffort} — applies to your next message.`, - ); + active.store.note(`Reasoning effort set to ${label} — applies to your next message.`); } else { - active.store.note(`Already on ${displayName} at effort ${reasoningEffort}.`); + active.store.note(`Already on ${displayName} at effort ${label}.`); } set({ modelPicker: undefined }); }; @@ -849,7 +863,7 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { applyLiveSessionPick(active, modelId, displayName, provider, reasoningEffort); return; } - writeNextSessionDefault(modelId, displayName, reasoningEffort); + writeNextSessionDefault(modelId, displayName, provider, reasoningEffort); }; // The open `/models` picker owns every key (2.5.G S7) — parity with routeMentionKey. Returns whether the key was // consumed. A DIMMED (unavailable-on-your-key) model is non-selectable (ADR §6): accepting one shows a transient @@ -911,17 +925,20 @@ export function createHomeController(deps: HomeControllerDeps): HomeController { case 'close': set({ effortPicker: undefined }); break; - case 'accept': + case 'accept': { set({ effortPicker: undefined }); + const label = effortRowLabel( + active.store.getSnapshot().state.model ?? '', + step.effort, + ).label; if (step.effort === open.current) { - active.store.note(`Already at reasoning effort ${step.effort}.`); + active.store.note(`Already at reasoning effort ${label}.`); } else { active.onSetEffort?.(step.effort); - active.store.note( - `Reasoning effort set to ${step.effort} — applies to your next message.`, - ); + active.store.note(`Reasoning effort set to ${label} — applies to your next message.`); } break; + } case 'state': set({ effortPicker: step.state }); break; diff --git a/apps/cli/src/render/tui/model-picker-view.tsx b/apps/cli/src/render/tui/model-picker-view.tsx index 91f17f80..b8f803ac 100644 --- a/apps/cli/src/render/tui/model-picker-view.tsx +++ b/apps/cli/src/render/tui/model-picker-view.tsx @@ -1,7 +1,7 @@ import { Box, Text } from 'ink'; import type { ReactElement, ReactNode } from 'react'; -import type { ModelCatalogEntry } from '@relavium/llm'; +import { datedPinBase, type ModelCatalogEntry } from '@relavium/llm'; import { sanitizeInline } from './chat-projection.js'; import { EffortTierList } from './effort-tier-list.js'; @@ -73,6 +73,8 @@ function EffortSubList(props: Readonly<{ state: ModelPickerState; color: boolean const { state, color } = props; return ( ): ReactEle // One status line: the transient user-action `hint` (a dimmed/save note) takes priority over the async refresh // `banner` (partial-failure) so a completing refresh can never silently wipe the feedback the user just triggered. const status = state.hint ?? state.banner; + // The ✓ marks the effective default. A persisted DATED-PIN default whose row was collapsed onto its rolling alias + // (the alias↔dated-pin dedup, ADR-0064 amendment) marks the surviving alias instead of vanishing — the SAME model + // resolves either id server-side, so this normalizes the DISPLAY, it does not re-point the user's stored choice. + const presentIds = new Set(state.entries.map((e) => e.modelId)); + const markedDefault = + state.currentDefault !== undefined && !presentIds.has(state.currentDefault) + ? (datedPinBase(state.currentDefault) ?? state.currentDefault) + : state.currentDefault; const renderBody = (): ReactNode => { if (state.loading && visible.length === 0) { @@ -122,7 +132,7 @@ export function ModelPickerView(props: Readonly): ReactEle } return windowed.map((entry, index) => { const isSelected = start + index === selected; - const isDefault = entry.modelId === state.currentDefault; + const isDefault = entry.modelId === markedDefault; const ctx = formatContextWindow(entry.contextWindowTokens); const parts = [ sanitizeInline(entry.displayName), diff --git a/apps/cli/src/render/tui/model-picker.test.ts b/apps/cli/src/render/tui/model-picker.test.ts index 5ae618e2..580ec722 100644 --- a/apps/cli/src/render/tui/model-picker.test.ts +++ b/apps/cli/src/render/tui/model-picker.test.ts @@ -1,4 +1,5 @@ import type { ModelCatalogEntry } from '@relavium/llm'; +import { REASONING_EFFORTS } from '@relavium/shared'; import { describe, expect, it } from 'vitest'; import { @@ -18,11 +19,10 @@ function entry( return { provider: 'anthropic', displayName: partial.modelId, - pricingSource: 'registry', + pricingSource: 'catalog', priceKnown: true, available: true, deprecated: false, - supportsReasoning: false, ...partial, }; } @@ -37,6 +37,7 @@ function state(partial: Partial = {}): ModelPickerState { refreshedAt: undefined, banner: undefined, hint: undefined, + effortTiers: [], phase: 'model', effortStep: false, pending: undefined, @@ -172,8 +173,8 @@ describe('foldModelPickerKey — the ADR-0066 effort sub-step', () => { const reasoningState = (partial: Partial = {}): ModelPickerState => state({ entries: [ - entry({ modelId: 'claude-opus-4-8', displayName: 'Opus', supportsReasoning: true }), - entry({ modelId: 'deepseek-chat', displayName: 'DeepSeek', supportsReasoning: false }), + entry({ modelId: 'claude-opus-4-8', displayName: 'Opus' }), + entry({ modelId: 'deepseek-chat', displayName: 'DeepSeek' }), ], effortStep: true, ...partial, @@ -225,11 +226,55 @@ describe('foldModelPickerKey — the ADR-0066 effort sub-step', () => { expect(step.state.effortSelected).toBe(3); // index of 'high' }); - // A picker already parked in the effort phase over a pending model (selected tier = 'medium'). + it('THE BUG: choosing gpt-5.4-pro opens a sub-step with {medium, high, max} — `low` is not a row', () => { + // The sub-step used to list all five tiers for any reasoning model. `gpt-5.4-pro` REJECTS `low`, so a user who + // picked it got an opaque provider 400 — the maintainer's report. The list is now derived from what the model + // accepts, so the interactive path cannot produce an illegal tier at all. + const step = foldModelPickerKey( + '', + { return: true }, + reasoningState({ + entries: [ + entry({ modelId: 'gpt-5.4-pro', displayName: 'GPT-5.4 Pro', provider: 'openai' }), + ], + }), + ); + if (step.kind !== 'state') throw new Error('expected the effort sub-step to open'); + expect(step.state.effortTiers).toEqual(['medium', 'high', 'max']); + expect(step.state.effortTiers).not.toContain('low'); + expect(step.state.effortTiers).not.toContain('off'); + // …and the highlight lands on a row that EXISTS. The old default was `indexOf('medium')` over the fixed five. + expect(step.state.effortTiers[step.state.effortSelected]).toBe('medium'); + }); + + it('a model that reasons but has NO controllable tier accepts immediately — no empty overlay', () => { + // `deepseek-reasoner`. The old gate (`supportsReasoning`) opened a five-row overlay for it; every row was a + // tier the model does not take. + const step = foldModelPickerKey( + '', + { return: true }, + reasoningState({ + entries: [ + entry({ + modelId: 'deepseek-reasoner', + displayName: 'R', + provider: 'deepseek', + }), + ], + }), + ); + expect(step.kind).toBe('accept'); // straight through — there is nothing to ask + }); + + // A picker already parked in the effort phase over a pending model (selected tier = 'medium'). `effortTiers` is + // what the PENDING model accepts (ADR-0071 §6) — claude-opus-4-8 takes all five, so this fixture is still a + // five-row list. A narrower model (gpt-5.4-pro, gpt-5-pro) is exercised in the transition tests below, which is + // where the list is actually derived rather than asserted. const effortPhase = (partial: Partial = {}): ModelPickerState => reasoningState({ phase: 'effort', pending: { modelId: 'claude-opus-4-8', displayName: 'Opus', provider: 'anthropic' }, + effortTiers: [...REASONING_EFFORTS], effortSelected: 2, ...partial, }); @@ -278,7 +323,11 @@ describe('foldModelPickerKey — the ADR-0066 effort sub-step', () => { const step = foldModelPickerKey('', { escape: true }, effortPhase()); expect(step).toEqual({ kind: 'state', - state: effortPhase({ phase: 'model', pending: undefined }), + // `effortTiers` clears WITH `pending`. They are one fact — "the tiers of the model being confirmed" — and + // leaving the list behind makes `pending === undefined && effortTiers.length > 0` a representable state that + // the field's own invariant denies. Inert today (the view renders the sub-list only in the effort phase), + // and precisely the kind of half-cleared state that a later change reads as still-valid. + state: effortPhase({ phase: 'model', pending: undefined, effortTiers: [] }), }); }); diff --git a/apps/cli/src/render/tui/model-picker.ts b/apps/cli/src/render/tui/model-picker.ts index 5406ee01..7d27a38b 100644 --- a/apps/cli/src/render/tui/model-picker.ts +++ b/apps/cli/src/render/tui/model-picker.ts @@ -1,5 +1,7 @@ import type { ModelCatalogEntry, ProviderId } from '@relavium/llm'; -import { REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; +import type { ReasoningEffort } from '@relavium/shared'; + +import { effortTiersFor, projectEffortToRow } from '../../chat/effort-notice.js'; import { dropLastCodePoint } from './chat-input.js'; @@ -43,7 +45,7 @@ export interface ModelPickerState { /** * The picker's TWO-PHASE step ([ADR-0066](../../../../../docs/decisions/0066-normalized-reasoning-effort-control.md)): * `'model'` is the catalog list (the default); accepting a reasoning-capable model on an effort-offering surface - * ({@link effortStep}) advances to `'effort'` — a fixed sub-list of the reasoning-effort tiers for the chosen + * ({@link effortStep}) advances to `'effort'` — the sub-list of the tiers THE CHOSEN MODEL accepts, for the chosen * model. The two surfaces route the SAME fold, so the phase transition lives here, not in either host. */ readonly phase: 'model' | 'effort'; @@ -63,7 +65,12 @@ export interface ModelPickerState { readonly provider: ProviderId; } | undefined; - /** The highlighted index into {@link REASONING_EFFORTS} while in `'effort'` phase. */ + /** + * The tiers the PENDING model accepts (ADR-0071 §6) — never the fixed five. Empty while no model is pending. + * `gpt-5.4-pro` rejects `low`; offering it a row the provider would 400 on is the bug this closes. + */ + readonly effortTiers: readonly ReasoningEffort[]; + /** The highlighted index into {@link ModelPickerState.effortTiers} while in `'effort'` phase. */ readonly effortSelected: number; /** The session's currently-bound effort (the `✓` in the effort sub-list + the initial highlight); `undefined` ⇒ * no effort bound (the provider default), so the sub-list opens on a neutral middle tier. */ @@ -192,7 +199,11 @@ function acceptVisibleModel( ...(chosen.unavailableReason !== undefined ? { reason: chosen.unavailableReason } : {}), }; } - if (state.effortStep && chosen.supportsReasoning) { + // The sub-step opens only if the model has a tier to OFFER. `supportsReasoning` was the old gate, and it is the + // wrong question: `deepseek-reasoner` reasons and exposes no controllable tier, so the old gate opened an overlay + // with five rows the model does not take. + const effortTiers = effortTiersFor(chosen.modelId); + if (state.effortStep && effortTiers.length > 0) { return { kind: 'state', state: { @@ -203,7 +214,8 @@ function acceptVisibleModel( displayName: chosen.displayName, provider: chosen.provider, }, - effortSelected: initialEffortIndex(state.currentEffort), + effortTiers, + effortSelected: initialEffortIndex(effortTiers, chosen.modelId, state.currentEffort), hint: undefined, }, }; @@ -261,10 +273,10 @@ function foldModelPhaseKey( } /** - * The `'effort'` phase fold (ADR-0066): a fixed sub-list of the reasoning-effort tiers for the {@link ModelPickerState.pending} + * The `'effort'` phase fold (ADR-0066/0071): the sub-list of the tiers the {@link ModelPickerState.pending} model accepts — * model. `Esc` backs OUT to the model list (Ctrl-C, handled by the caller, is the hard cancel); `↑`/`↓` move over - * {@link REASONING_EFFORTS}; `Enter` accepts the chosen model + the highlighted tier. There is no filter or refresh - * here (a fixed five-item list), so every other key is inert. + * {@link ModelPickerState.effortTiers}; `Enter` accepts the chosen model + the highlighted tier. There is no filter or refresh + * here (the list is fixed once the model is chosen), so every other key is inert. */ function foldEffortPhaseKey( char: string, @@ -272,24 +284,30 @@ function foldEffortPhaseKey( state: ModelPickerState, ): ModelPickerStep { if (key.escape === true) { - return { kind: 'state', state: { ...state, phase: 'model', pending: undefined } }; + return { + kind: 'state', + state: { ...state, phase: 'model', pending: undefined, effortTiers: [] }, + }; } if (key.upArrow === true) { - const next = clampSelection(state.effortSelected - 1, REASONING_EFFORTS.length); + const next = clampSelection(state.effortSelected - 1, state.effortTiers.length); return { kind: 'state', state: { ...state, effortSelected: next } }; } if (key.downArrow === true) { - const next = clampSelection(state.effortSelected + 1, REASONING_EFFORTS.length); + const next = clampSelection(state.effortSelected + 1, state.effortTiers.length); return { kind: 'state', state: { ...state, effortSelected: next } }; } if (key.return === true) { const pending = state.pending; const effort = - REASONING_EFFORTS[clampSelection(state.effortSelected, REASONING_EFFORTS.length)]; + state.effortTiers[clampSelection(state.effortSelected, state.effortTiers.length)]; // Defensive: a missing pending model (never expected — set on the transition) or an out-of-range tier backs out // to the model list rather than emitting a malformed accept. if (pending === undefined || effort === undefined) { - return { kind: 'state', state: { ...state, phase: 'model', pending: undefined } }; + return { + kind: 'state', + state: { ...state, phase: 'model', pending: undefined, effortTiers: [] }, + }; } return { kind: 'accept', @@ -303,8 +321,18 @@ function foldEffortPhaseKey( } /** The effort sub-list's opening highlight: the session's bound effort, else a neutral middle tier (`'medium'`). */ -function initialEffortIndex(currentEffort: ReasoningEffort = 'medium'): number { - return Math.max(0, REASONING_EFFORTS.indexOf(currentEffort)); +function initialEffortIndex( + tiers: readonly ReasoningEffort[], + model: string, + currentEffort: ReasoningEffort | undefined, +): number { + // Open on the bound tier, or the neutral `medium` when nothing is bound — PROJECTED onto a surviving row first + // (ADR-0066 amendment): a graded-collapsed model (deepseek [off,high,max]) has no `medium` row, so a bare + // `indexOf('medium')` = -1 → 0 = `off` would silently open on reasoning-DISABLED and a fresh Enter would write + // it. The projection folds the tier onto its representative (medium→high / a budget neutral→the `on` row). + const target = projectEffortToRow(model, tiers, currentEffort ?? 'medium'); + const index = target === undefined ? 0 : tiers.indexOf(target); + return Math.max(0, index); } /* -------------------------------------------------------------------------------------------------- * diff --git a/apps/cli/src/run.ts b/apps/cli/src/run.ts index bb9d30dd..c7b02e49 100644 --- a/apps/cli/src/run.ts +++ b/apps/cli/src/run.ts @@ -1,5 +1,7 @@ import { CommanderError } from 'commander'; +import { resolveHomeDir } from './config/load.js'; +import { loadCachedCatalog } from './engine/catalog-refresh.js'; import { driveHome, type HomeDeps } from './home/drive-home.js'; import { shouldOpenHome } from './home/should-open-home.js'; import { CliError, toUserFacing } from './process/errors.js'; @@ -46,6 +48,16 @@ export async function run( return toUserFacing(err).exitCode; } + // THE CATALOG, BEFORE ANYTHING READS A PRICE (ADR-0071 §4). + // + // A previous `models refresh --catalog` cached models.dev to disk; load it so this process starts current. Absent, + // unreadable, or corrupt ⇒ silently nothing, and the SHIPPED SNAPSHOT answers — it is the floor, not a fallback, + // and a refresh that cannot be read must never leave the product knowing less than it shipped knowing. + // + // No fetch here, ever. `[catalog] auto_refresh` (default `false`) governs whether a background refresh may run at + // all, and even when it is on, the fetch belongs on a path the user can see — not in the boot of every `--help`. + seedCatalog(); + const result: { exitCode?: ExitCode } = {}; const program = buildProgram(io, { suppressErrorOutput: renderCtx.json, @@ -126,3 +138,20 @@ function stripErrorPrefix(message: string): string { } return message.slice(prefix.length).trimStart(); } + +/** + * Install a previously-refreshed models.dev catalog, if one is cached (ADR-0071 §4). + * + * TOTAL: it cannot fail the CLI. A missing cache, an unreadable one, a corrupt one, an unreadable home — every path + * lands on the shipped snapshot, which is a complete answer on its own. That is the point of embedding it: with no + * refresh ever run (the DEFAULT — `[catalog] auto_refresh = false`), Relavium prices every model it ships, offline, + * and contacts nobody. + */ +function seedCatalog(): void { + try { + loadCachedCatalog(resolveHomeDir({})); + } catch { + // An unreadable home, a torn cache. Not this seed's job to report — a cached catalog is not a reason to refuse + // to run `relavium --help`, and the shipped snapshot answers regardless. + } +} diff --git a/docs/decisions/0063-cli-config-write-contract.md b/docs/decisions/0063-cli-config-write-contract.md index d1b5fca4..42dc03a3 100644 --- a/docs/decisions/0063-cli-config-write-contract.md +++ b/docs/decisions/0063-cli-config-write-contract.md @@ -6,6 +6,8 @@ > **Note (2026-07-07): the typed setter gained a second key — `[preferences].reasoning_effort` (ADR-0066 §6).** The writer generalized from `writeGlobalDefaultModel` to `writeGlobalPreferences({ defaultModel?, reasoningEffort? })` (the former is now a thin wrapper). It writes ONE or BOTH of the two non-secret `[preferences]` keys in a single atomic, schema-round-tripped write; a field absent from the call is a **partial merge** (left unchanged), so a model-only pick never clears a prior effort default. The typed-setter secret-incapability guarantee is unchanged — both keys are non-secret enums/strings, and the strict `GlobalConfigSchema` still gates the emission. The `/models` picker's effort sub-step (bare Home) is the write's new driver, and `resolveChat` now reads `[preferences].reasoning_effort` as the global fallback below `[chat].reasoning_effort` — the exact shape §1 established for `default_model`. +> **Note (2026-07-13): the typed setter gained a third key — `[preferences].default_provider` ([ADR-0059](0059-cli-mid-session-model-reseat.md)).** The `/models` picker and the onboarding wizard now persist the provider ALONGSIDE `default_model`, in the same atomic `writeGlobalPreferences({ defaultModel?, defaultProvider?, reasoningEffort? })` write. The provider is authoritative at pick time (the live `/models` list is per-provider), and re-inferring it from an opaque id fails for a live-discovered id whose spelling the prefix map cannot place (`chatgpt-4o-latest` has no `gpt` prefix) — the "cannot infer a provider for chat model" crash this closes. `resolveChat` reads it as the global fallback below `[chat].default_provider`, the exact shape §1 established for `default_model`; `buildDefaultChatAgent` uses a persisted provider VERBATIM and only falls back to id inference (catalog first, then prefix) when none is stored. Still non-secret, still `GlobalConfigSchema`-gated, still a partial merge. + ## Context The CLI config layer is **read-only**. [load.ts](../../apps/cli/src/config/load.ts) imports `smol-toml` diff --git a/docs/decisions/0064-live-model-catalog.md b/docs/decisions/0064-live-model-catalog.md index d53e1714..0fac815a 100644 --- a/docs/decisions/0064-live-model-catalog.md +++ b/docs/decisions/0064-live-model-catalog.md @@ -34,6 +34,72 @@ > [database-schema.md](../reference/desktop/database-schema.md); the `0600`/`0700` guard is a documented Windows > no-op (ADR-0050), so the 2.5.I test lane gates POSIX-permission assertions off Windows, while the > `BEGIN IMMEDIATE` + retry mechanism behaves identically cross-OS. +> +> **Amended 2026-07-13 — two clauses are SUPERSEDED by [ADR-0071](0071-models-dev-as-the-model-metadata-source.md).** +> The architecture below stands; two of its sentences do not. +> +> 1. **"Pricing authority stays with the static registry."** The *split* this ADR drew — the live list decides +> **availability**, a static tier decides **economics** — is correct and is kept. Its **source** is not: the +> hand-maintained `MODEL_PRICING` proved unable to say what is true. It priced 12 of ~97 reachable models (so +> [ADR-0028](0028-workflow-resource-governance.md)'s cost cap silently did not apply to the rest), it carried +> two silent drifts, and its `reasoning: boolean` **cannot express** a per-model reasoning-control shape — +> which is a live bug, not a limitation: `gemini-2.5-*` takes `thinkingBudget`, not the `thinkingLevel` our +> adapter unconditionally sends. ADR-0071 replaces the table with a **generated, PR-reviewed catalog +> snapshot**. The static tier survives; the hand-typing does not. +> 2. **"This ADR adds no new egress surface."** ADR-0071's catalog refresh **does** — the first host Relavium +> contacts that is neither a model provider nor a user-supplied address. It is **default-OFF**, additive-only, +> a no-op offline, and recorded as the fifth path in +> [security-review.md](../standards/security-review.md). +> +> The Related line above points at [pricing.ts](../../packages/llm/src/pricing.ts) as the static registry's +> canonical home. After ADR-0071 that home is the generated snapshot under `packages/llm/src/catalog/` (rule 8). + + +> **Amendment (2026-07-13, [ADR-0071](0071-models-dev-as-the-model-metadata-source.md) §1).** §3's static tier is +> no longer the hand-typed `MODEL_PRICING`; it is the **generated catalog** (`packages/llm/src/catalog/snapshot.ts`, +> synced from models.dev). Read every "static registry" in this ADR as "the catalog". Three consequences: +> +> 1. **The cost-eligibility union widened.** §3's "union-in any id present in `MODEL_PRICING`" now unions the +> catalog's ids — eighty models, not twelve — still filtered by the non-chat deny-list (an embedding is priced +> and is still not something you can chat with). +> 2. **The pricing precedence FLIPPED.** §6 said the static registry always wins and the user tier fills an unknown +> id only. It now resolves **user → catalog**: the catalog is a snapshot of a third-party aggregator, and the +> user is the one holding the invoice. `pricingSource` reports `'catalog'` where it used to report `'registry'`. +> +> §7's deprecation is **unchanged in substance**: the date still comes from a Relavium-owned overlay +> (`packages/llm/src/catalog/deprecations.ts` — ADR-0071 §10), unioned with the live list and the user, earliest +> wins. It moved out of the price table, but it did not move into models.dev, which publishes a `status` flag and +> not a date. *(A first cut of the swap deleted it outright on the theory that the live list would carry it — no +> adapter populates `ModelListing.deprecatedAt`, so that would have made `deprecated` permanently `false` for every +> model. Caught in review.)* + + +> **Clarified 2026-07-14 (alias↔dated-pin availability equivalence — append-only, body unchanged):** §6's live-list +> availability is by EXACT model id. A provider that pins a rolling alias to a dated snapshot returns only ONE of the +> pair from `models.list()` (real Anthropic returns the dated `claude-haiku-4-5-20251001`, not the rolling +> `claude-haiku-4-5`), while the catalog (ADR-0071) ships BOTH as priced rows — so the id the list omits was dimming +> as `not-on-key` even though the SAME key calls it (both ids resolve server-side to the same model). The merge now +> RESCUES such a pair: a live-omitted model is `available` when its alias↔dated-pin sibling (`base` ↔ +> `base-YYYYMMDD`) IS in the live list AND is itself a shipped catalog row of the same provider — the catalog-row +> gate is what stops this fabricating availability for an arbitrary unpriced id. This **refines, not reverses**, §6: +> a model whose provider genuinely does not serve it (no live sibling, or a sibling that is not a catalog row) still +> dims `not-on-key`. Scoped to **Anthropic** this round (the maintainer's call — the OpenAI `gpt-4o` dated family is +> deliberately out of scope); the anthropic conformance fixture is corrected to return the dated pin so the suite +> exercises the rescued-alias case. +> +> **Amendment 2026-07-14 (alias↔dated-pin DISPLAY collapse — the second half of the same relationship):** the +> availability rescue above keeps BOTH ids callable, but the catalog still SHIPS both as priced rows, so the +> `/models` picker rendered TWO selectable rows for one model (`claude-opus-4-1` and `claude-opus-4-1-20250805`, +> byte-identical but for id/name). `collapseAliasDatedPinPairs` (`@relavium/llm`) now drops the dated pin at the +> DISPLAY boundary — in `buildMergedCatalog`'s host projection, NOT in `mergeModelCatalog` (whose full-fidelity +> output any other consumer keeps) — leaving only the rolling alias. It collapses ONLY when both members are present +> as catalog rows (a lone dated pin with no alias sibling stays visible, the same gate `hasLiveSibling` applies), and +> is Anthropic-only by the `-YYYYMMDD` regex shape. This is **identity dedup, not an availability judgement** — it is +> deliberately distinct from the picker's "never HIDE a dimmed/deprecated model" rule (that concerns a model you +> cannot USE; this removes a second copy of one you can). A persisted DATED-PIN `default_model` (from before the +> collapse or a hand-edit) NORMALIZES its `✓` onto the surviving alias row rather than vanishing — the same model +> resolves either id server-side, so the marker moves, the stored choice does not. + ## Context diff --git a/docs/decisions/0065-provider-economics-and-extensibility.md b/docs/decisions/0065-provider-economics-and-extensibility.md index 9f8b9681..00df2eb3 100644 --- a/docs/decisions/0065-provider-economics-and-extensibility.md +++ b/docs/decisions/0065-provider-economics-and-extensibility.md @@ -4,6 +4,25 @@ - **Date**: 2026-07-05 - **Related**: [ADR-0064](0064-live-model-catalog.md) (**this ADR extends its static/live merge with a USER tier**; append-only top-note added there) · [ADR-0011](0011-internal-llm-abstraction.md) (**this ADR amends the provider model — a `kind` protocol abstraction + building the adapter from the stored row; the id enum stays CLOSED**; append-only top-note added there) · [ADR-0028](0028-workflow-resource-governance.md) (the pre-egress budget governor whose "cost cap will not apply" gap this closes) · [ADR-0038](0038-agentrunner-llm-call-boundary.md) (host-injected resolution — the pricing overlay is injected exactly like `keyFor`) · [ADR-0006](0006-os-keychain-for-api-keys.md) + [ADR-0019](0019-cli-node-keychain-library.md) (keys stay in the keychain — user pricing is a **non-secret** storage class) · [ADR-0053](0053-mcp-network-transport-egress-security.md) + [ADR-0029](0029-tool-policy-hardening.md) (the one shared SSRF primitive a custom `base_url` reuses) · [ADR-0050](0050-cli-history-db-at-rest-posture.md) · [ADR-0056](0056-cli-in-app-slash-command-system-and-manifest.md) (the `models pricing` / `provider list --verify` commands). Canonical homes: the cost path → [cost-tracker.ts](../../packages/llm/src/cost-tracker.ts) + [budget-governor.ts](../../packages/core/src/engine/budget-governor.ts); the static registry → [pricing.ts](../../packages/llm/src/pricing.ts); the DB columns → [database-schema.md](../reference/desktop/database-schema.md); the commands → [commands.md](../reference/cli/commands.md). +> **Amended 2026-07-13 ([ADR-0071](0071-models-dev-as-the-model-metadata-source.md) §1) — §2's precedence is +> REVERSED to `user → catalog → throw`; the rest of this ADR stands.** +> +> §2's original rule — *"static `MODEL_PRICING` wins for known canonical ids; the overlay fills unknown ids only, +> so a user cannot **silently** misprice a shipped model"* — protected the user from mispricing a model *we* had +> verified by hand. That made sense while the table was ours. ADR-0071 retires it for a catalog GENERATED from +> models.dev (a third-party aggregator, ~97 models vs 12), and static-first would then be a trap: a user prices an +> id the old table did not know, the catalog later **learns** it, and their explicit override is silently taken +> over by a public list price. This ADR's own custom **`base_url`** endpoints (OpenRouter, Azure, LiteLLM, +> enterprise gateways) make the point sharpest — there the public price is simply **wrong**, and the user's +> negotiated or marked-up rate is the correct one; they hold the invoice. +> +> So `models pricing` no longer REFUSES a model the catalog already knows — overriding one is the point of the +> command. This ADR's *actual* intent (**not `silently`**) is preserved, not reversed: a user override that +> disagrees with the catalog is surfaced in the model picker, in `/cost`, and at `models pricing` time — the user +> gets what they asked for, just never in silence. The Related line's *"the static registry → +> [pricing.ts](../../packages/llm/src/pricing.ts)"* pointer now means the generated snapshot (rule 8). + + ## Context Two latent defects, surfaced while scoping 2.5.G, frame this decision: diff --git a/docs/decisions/0066-normalized-reasoning-effort-control.md b/docs/decisions/0066-normalized-reasoning-effort-control.md index dab0bda6..8e929b2f 100644 --- a/docs/decisions/0066-normalized-reasoning-effort-control.md +++ b/docs/decisions/0066-normalized-reasoning-effort-control.md @@ -11,6 +11,47 @@ > **Note (2026-07-07): interactive `/effort` overlay realized (§6).** §6 anticipated "a future `/effort`" for standalone effort changes; it is now shipped as a first-class **interactive tier-selector overlay** (not just a typed `/effort ` — that still works). Bare `/effort` (typed or selected from the `/` palette) opens a keyboard-owning overlay — a fixed off/low/medium/high/max list, opening on the bound tier, arrow+Enter to apply — in BOTH `relavium chat` and the in-Home live chat, sharing one pure fold ([effort-picker.ts](../../apps/cli/src/render/tui/effort-picker.ts)) and one view ([effort-tier-list.tsx](../../apps/cli/src/render/tui/effort-tier-list.tsx), also used by the `/models` effort sub-step). Applying calls the §5 per-turn `onSetEffort` — **no reseat**. It opens ONLY on a reasoning-capable bound model; a non-reasoning model falls through to the informational notice (` has no controllable reasoning tier`). This realizes §6's "standalone effort changes ride the same session override" — from a promise to a shipped surface. (§6's separate point, that the bare-Home picker writes only the config default and not a live session's effort, is unchanged here — a follow-up gives that picker its own effort sub-step.) > > **Note (2026-07-07): the bare-Home `/models` picker gained an effort sub-step (§6, superseding "config default only").** §6 said the bare-Home picker "writes the config default, not a live session's effort" and stays single-phase. It is now **two-phase for a reasoning model** on BOTH surfaces: picking a reasoning-capable model advances to the effort sub-step (opened on the current default tier), and accepting writes the model AND its effort tier as the NEXT session's defaults. This required a **new global config key**, `[preferences].reasoning_effort` (the effort counterpart of `default_model`), added to `GlobalConfigSchema`; the CLI config writer generalized to `writeGlobalPreferences({ defaultModel?, reasoningEffort? })` (ADR-0063 §note), writing both in one atomic, schema-round-tripped write (a partial write leaves the other key unchanged); and `resolveChat` now falls back to `[preferences].reasoning_effort` below the `[chat]` layers — the exact precedence `default_model` already has. So a user can set BOTH their default model and default effort from the Home without starting a chat. A live in-Home chat's `/models` effort sub-step still sets the per-turn override (no reseat, §5) rather than writing config — the accept action is surface-specific, off the one picker. +> +> **Note (2026-07-13): a PREMISE of this ADR is FALSE, and it shipped as a live bug — corrected by +> [ADR-0071](0071-models-dev-as-the-model-metadata-source.md).** +> +> §1 below states: *"All four currently-targeted providers control reasoning by a discrete **TIER, not a token +> budget** … Gemini (a thinking-level field) … The older 'token budget' shapes (pre-`output_config` Anthropic, +> **Gemini 2.5**) are **legacy**."* +> +> That sentence is defensible about the industry and **wrong about the models we actually ship**. Google's +> documentation for the `generateContent` API this project calls is explicit — *"Gemini 2.5 series models don't +> support `thinkingLevel`; use `thinkingBudget` instead"* — and `gemini-2.5-pro` **cannot disable thinking at +> all** (`thinkingBudget` 128–32768, no zero). `gemini-2.5-flash` and `gemini-2.5-pro` are the **only two Gemini +> rows in `MODEL_PRICING`**. So the shape this ADR set aside as "legacy" is the shape of **every Gemini model we +> ship**, and the adapter has been sending them a `thinkingLevel` they do not take. +> +> The error was **structural, not clerical**. This ADR made the native shape a property of the **adapter** and +> the capability a per-model `boolean`. A boolean cannot say *"this model takes a token budget in [128, 32768] +> and has no off switch"* — so the bug had nowhere to be caught, and a test could not have found it either. +> [ADR-0071](0071-models-dev-as-the-model-metadata-source.md) makes the control's **shape** and its **accepted +> tiers** per-model data from a catalog; the adapter then selects `thinkingLevel` vs `thinkingBudget` from that +> descriptor rather than assuming one for the whole provider. +> +> **What survives, unchanged:** the normalized five-tier vocabulary (`off|low|medium|high|max`), the +> canonical-wins-over-`providerOptions` precedence, the `/effort` overlay, and the host-gated design. What +> changes is only that a model now *declares* which tiers it accepts and which native shape carries them, +> instead of the adapter assuming both. + +> **Note (2026-07-14): the PICKER now presents "one row per distinct outcome", separate from the wire-accurate +> accepted set (§6 presentation refinement — append-only, the seam is unchanged).** `acceptedTiers` correctly +> answers what the WIRE takes, but it made a poor MENU: it offered five rows for models that expose fewer real +> choices, because several normalized tiers can collapse onto one provider value (DeepSeek's low/medium/high all +> send `high`; Gemini's `max` coarsens onto `high`) and because a continuous token BUDGET has no discrete rungs at +> all (`claude-haiku-4-5` — the maintainer's report: a five-tier ladder where the model is really on/off, à la +> Claude Code). A new PRESENTATION helper `reasoningControlShape(controls)` → `graded | budget | none` and a +> `CANONICAL_ON_TIER` (= `medium`) drive a CLI projection (`effortTiersFor`, `effortRowLabel` in +> `chat/effort-notice.ts`): a **graded** ladder is deduped by distinct wire value (the representative reads the +> name that matches the wire); a **budget** model is a two-row **off/on** ("on" = `medium`, a real member of the +> accepted set, so the accept sends a value the gate accepts verbatim — a model that cannot be turned off, like +> `gemini-2.5-pro`, has nothing to toggle, so no overlay opens); **none** shows nothing. This is presentation only: +> `acceptedTiers` (the wire truth the engine gate, failover chain, and four adapters read) is untouched, so every +> emitted row is still a valid accepted tier. The rule is GENERAL (by shape), never per-model. ## Context diff --git a/docs/decisions/0071-models-dev-as-the-model-metadata-source.md b/docs/decisions/0071-models-dev-as-the-model-metadata-source.md new file mode 100644 index 00000000..0e3af443 --- /dev/null +++ b/docs/decisions/0071-models-dev-as-the-model-metadata-source.md @@ -0,0 +1,485 @@ +# ADR-0071: models.dev as the model-metadata source; the hand-maintained registry is retired + +- **Status**: Accepted +- **Date**: 2026-07-13 +- **Related**: [ADR-0064](0064-live-model-catalog.md) (**supersedes** two of its clauses — "pricing authority stays with the static registry" and "adds no new egress surface"; every other part of its architecture is kept) · [ADR-0065](0065-provider-economics-and-extensibility.md) (**narrows** its user-pricing precedence — see §5 — and its custom `base_url` is the *reason* for that narrowing) · [ADR-0066](0066-normalized-reasoning-effort-control.md) (**corrects a false premise in it**: it assumed one fixed reasoning-control shape per *adapter*; the shape is per *model*, and that assumption is a live bug — §Context/3) · [ADR-0070](0070-durable-per-model-session-cost-attribution.md) (which forward-named this ADR: *"2.6.Q's pricing-enrichment decision … will be **ADR-0071**"*; its `session_costs` invariant and `unpriced_calls` counter are what made the unpriced long tail *visible*) · [ADR-0028](0028-workflow-resource-governance.md) (the cost cap this turns into a real control) · [ADR-0062](0062-context-compaction-and-cli-history-commands.md) (its `contextWindowForModel` reads the retired table — §7) · [ADR-0068](0068-full-screen-tui-renderer-ink7-harness.md) (the default-OFF-then-flip rollout convention this follows for the new egress) · [ADR-0043](0043-media-egress-failover-rematerialization-ssrf.md) (the outbound-URL posture this extends). Canonical homes: the generated snapshot → `packages/llm/src/catalog/`; the outbound-path inventory → [security-review.md](../standards/security-review.md); adding a provider → [add-a-provider.md](../runbooks/add-a-provider.md). + +## Context + +Relavium's model metadata — price, context window, max output, and reasoning capability — lives in +`MODEL_PRICING` ([pricing.ts](../../packages/llm/src/pricing.ts)), a **hand-maintained table of 12 models**. +[ADR-0064](0064-live-model-catalog.md) built a live catalog on top of it and drew the line explicitly: +*"the live list decides **availability**, the static registry stays the [pricing authority]"*. That line was +right about the split and wrong about the source. Three failures, all reported from real use, all with the +same root — **a hand-typed table cannot say what is true**: + +1. **Most models have no price.** `priceModel` is an exact-string lookup; an unknown id throws + `UnknownModelError`, and `BudgetGovernor.evaluatePreEgress` catches it and returns `allow`. + `max_cost_microcents` — a **safety control** — is silently skipped for every model outside the 12. The model + the maintainer actually hit, `gpt-5.4-pro`, is not in the table. + +2. **Reasoning capability is a `boolean`, and the truth is not a boolean.** `gateReasoningEffort` is a + pass-through, not a clamp: if the model reasons *at all*, whatever tier the user picked goes to the wire. + But the accepted tiers differ **per model** — `gpt-5.4-pro` takes `{medium, high, xhigh}` and **rejects + `low`**; `gpt-5-pro` takes only `{high}`; `gemini-3-pro-preview` has no `medium`. A boolean cannot say any + of that, so the picker offers all five tiers to every reasoning model and the provider 400s. + +3. **The control's SHAPE differs per model — and we get it wrong today. This is a live bug.** Our Gemini + adapter sends `thinkingConfig.thinkingLevel` for every reasoning model. + [ADR-0066](0066-normalized-reasoning-effort-control.md) allowed this: it assumed **one fixed native shape + per adapter** ("Gemini → thinking-level"). Google's documentation for the `generateContent` API we call is + unambiguous that the assumption is false: *"**Gemini 2.5 series models don't support `thinkingLevel`; use + `thinkingBudget` instead**"* — and `gemini-2.5-pro` **cannot disable thinking at all** (`thinkingBudget` + 128–32768, no zero). Those two models are the **only** Gemini rows we ship. So `/effort` on Gemini is, + today, sending a parameter the model does not take. This ADR therefore **corrects** ADR-0066's premise + rather than merely extending it. + +The table also **drifts, silently and in the safe direction**, which is exactly why nobody caught it: it says +`claude-sonnet-4-6` maxes at 64k output (it is 128k) and `gpt-5.5` has a 1,000,000 context (it is 1,050,000). + +A provider's `/models` endpoint cannot fix this — it returns **availability**, not economics. Our own code +already says so: *"the live tier is **never** a pricing authority — providers rarely return a price"*. The +missing half has to come from somewhere. + +[models.dev](https://models.dev) is that somewhere: an open, **MIT-licensed** catalog maintained by the +opencode project, carrying **per model** `cost` (input/output/cache-read/cache-write plus context-size +`tiers`), `limit` (context **and max output**), and `reasoning_options` — which encodes both the **shape** of +the reasoning control (`effort` / `budget_tokens{min,max}` / `toggle`) and its **accepted values**. + +**We checked it rather than trusting it:** + +- **All 12 of our hand-verified prices match it exactly** — including the awkward ones (`deepseek-v4-pro` + 0.435/0.87, `gpt-5.4-mini` 0.75/4.5). Zero disagreements; our ids are its keys, 1:1. +- Its `gemini-2.5-pro` `budget_tokens {min: 128, max: 32768}` matches **Google's own documentation exactly** — + on the very model where our code is broken. +- It has `gpt-5.4-pro`, with `effort: [medium, high, xhigh]` — which *is* the maintainer's bug report. + +So this is not a bet. On every axis we could independently verify, models.dev is more correct than we are. + +## Decision + +**Retire the hand-maintained `MODEL_PRICING`. Model metadata comes from a generated, repo-reviewed models.dev +snapshot that ships in the binary. Availability continues to come from the provider API. Reasoning capability +stops being a boolean and becomes a per-model control descriptor. Everything is driven by a provider table, so +adding a provider is an entry, not a rewrite.** + +### 1. Two axes, two sources, no overlap + +| Axis | Source | Answers | +|---|---|---| +| **Availability** | provider `listModels` (live, per user key) — [ADR-0064](0064-live-model-catalog.md) | *Can **I** call this model?* | +| **Metadata** | the generated snapshot | *What does it cost, what is its ceiling, what reasoning control does it take?* | + +Neither replaces the other. A model can be callable but unpriced (brand new), or priced but not callable (not +on your tier). The existing `available` / `priceKnown` flags already carry both. + +**ADR-0064's `MODEL_PRICING`-as-authority clause is superseded.** Its *split* — live for availability, a static +tier for economics — is **kept and strengthened**; only the static tier's **source** changes, from a hand-typed +table to a generated one. + +### 2. Provider-extensible by construction — adding a provider is an ENTRY, not a rewrite + +`ProviderId` stays the **closed enum** ([ADR-0064](0064-live-model-catalog.md) §6): it crosses the seam, the +persisted run-event `provider` field, and an exhaustive `Record`. Adding a provider is +therefore an adapter + an enum entry, as it always was. + +What this ADR must not do is bake "four providers" into the catalog machinery. So **one table drives +everything**: + +```ts +/** Our ProviderId → the provider's key in models.dev. The ONLY place the two vocabularies meet. */ +const CATALOG_PROVIDER_KEYS: Record = { + anthropic: 'anthropic', + openai: 'openai', + gemini: 'google', // ← their key is `google`, ours is `gemini` + deepseek: 'deepseek', +}; +``` + +The generator iterates `LLM_PROVIDERS`, not a literal list. **Adding a provider tomorrow is: write the +adapter, add the enum entry, add one line here, re-run the sync.** The runbook +([add-a-provider.md](../runbooks/add-a-provider.md)) is updated to say exactly that, as a numbered step. + +Two traps this table also closes: + +- **`google-vertex` is ignored.** It republishes the same Gemini ids at different prices; a naive flatten would + register every Gemini model twice. Only the mapped key is read. +- **models.dev providers we have no adapter for are ignored** — 162 of its 166. Importing a model we cannot + call would put an uncallable row in the picker. A provider appearing upstream is *not* a signal to add it. + +**A provider with no models.dev coverage is a supported case, not an error.** A self-hosted or bespoke endpoint +simply has no metadata rows; it degrades to the same "unpriced" path as a brand-new model (§6), and the user +can price it with `relavium models pricing` ([ADR-0065](0065-provider-economics-and-extensibility.md)). The +catalog tier is *additive*: absence of data is never a failure. + +### 3. The snapshot is GENERATED and SHIPPED — the floor is offline + +`scripts/sync-models-dev.ts` fetches `api.json`, keeps only the providers in `CATALOG_PROVIDER_KEYS`, +Zod-validates, normalizes to Relavium types, and emits a data module under `packages/llm/src/catalog/` plus the +upstream body's SHA-256. It is a **committed artifact**, reviewed in a PR like any other change. + +*Considered fetching at runtime and shipping nothing (rejected — this is the load-bearing rejection):* on first +run, offline, or with models.dev unreachable, there would be **no prices at all**, so `max_cost_microcents` +would not apply. **A cost cap that works only when a third-party host is reachable is not a safety control**, +and "the cap silently does not apply" is precisely the bug this ADR exists to close. The snapshot is the +**floor**: correct, offline, from the moment the binary is installed. + +*Considered keeping the hand-maintained table as the top-precedence tier (rejected):* it puts one artifact in +two homes (rule 8) and makes the **less** accurate source authoritative — the table is the thing that is wrong. +"Verified" does not come from a human *typing* the number; it comes from a human **reviewing the diff**. A +generated file preserves the review gate and deletes the duplication. + +*Considered vendoring the full `api.json` (rejected):* 3.17 MB, 166 providers, 5,669 models, of which 162 +providers are not callable by any adapter. Filtered to ours: **97 models**, and of the fields we consume, +**~22 KB**. + +**No new dependency.** `fetch` is native on the Node floor ([ADR-0067](0067-node-supported-floor-22-reaffirm-better-sqlite3.md)), +Zod is already a dependency, and SHA-256 is `node:crypto`. Nothing here needs an npm package. + +### 4. The refresh is OPT-IN and DEFAULT-OFF, and it can only ADD + +An **optional** refresh keeps a long-lived install current between releases. It is governed by three rules, in +order of importance: + +1. **Default OFF.** [ADR-0068](0068-full-screen-tui-renderer-ink7-harness.md) established this project's + convention for a new, risk-bearing surface: *ship opt-in default-OFF, validate, then flip*. Local-first is + non-negotiable (rule 6), and a local-first tool that contacts a third party **by default** violates its + spirit even when the payload is innocuous. The config key is `[catalog] auto_refresh` (default `false`). + A **user-initiated** `relavium models refresh` always may fetch — an explicit command *is* consent. Flipping + the default to `true` is a later, separate decision, once the surface has been validated in the field. +2. **Additive only.** A refresh may add models and enrich models the shipped snapshot does not pin, but it + **can never leave a model less priced than the shipped snapshot did**. A failed, unreachable, or malformed + refresh is a **no-op** — never a downgrade, never a blank catalog. +3. **The shipped snapshot is the floor.** With the refresh off (the default), behaviour is exactly the embedded + design: zero egress, fully offline, and every model in the snapshot priced. + +**This is a new egress surface, and ADR-0064's *"adds no new egress surface"* sentence is hereby superseded.** +It is recorded rather than quietly outgrown. Its posture, and its place in the outbound-path inventory, is §8. + +> **Amendment (2026-07-13, implementation — §4.2 + §9 reconciled).** The runtime refresh is **purely additive for +> shipped models**: it adds the long tail the snapshot does not carry, and it **never writes a model the snapshot +> already pins**. That is the exact reading §4.2's "the shipped snapshot is the floor" and §9's "a price change on +> an already-shipped model is a human decision" require together — a runtime path cannot silently move a +> human-verified price, so it does not touch one at all. +> +> The first implementation got this wrong in a way worth recording: its "floor" guard was +> `if (shipped === undefined || model.output > 0)`, and the line above had already dropped every `output <= 0`, so +> the second clause was *always true* and the guard was `if (true)`. A refreshed row replaced its shipped row +> wholesale — so a hostile or merely typo'd upstream `output: 0.00000001` on `gpt-5.5` recorded $14.50 of real +> spend as $0.00, and the cost cap stopped tripping. The safety control this ADR exists to build, defeated by the +> code that builds it. The floor is now: a shipped id is skipped; only a NEW, priced id is admitted. + +#### 4a. Both axes get a MANUAL trigger — one command, not two + +`relavium models refresh` already exists and refreshes provider availability +([ADR-0064](0064-live-model-catalog.md)). It now refreshes **both axes**, because "refresh what I know about +models" is one user intent, not two: + +| Invocation | Refreshes | Egress | +|---|---|---| +| `relavium models refresh` | **both** — provider lists **and** the catalog | provider APIs + models.dev | +| `relavium models refresh --providers` | availability only (today's behaviour) | provider APIs | +| `relavium models refresh --catalog` | metadata only | models.dev | + +Every form is **user-initiated**, and an explicit command **is** consent — so the catalog fetch is allowed here +even with `auto_refresh = false` (the default). This is what makes default-OFF a livable default rather than a +dead end: a user who wants current prices types one command and gets them, with no standing background egress. +Each form reports per-source outcomes and honours `--json` ([ADR-0049](0049-cli-machine-output-contract.md)), +so a script (or a later `/models refresh` palette entry) can drive it without a second code path. The +in-REPL slash surface is **not** in scope here — this ADR builds the primitive and its shell entry point; a +palette entry is a curated-surface decision ([ADR-0056](0056-cli-in-app-slash-command-system-and-manifest.md)) +to be made when it is wanted. + +The supply-chain question — *"a wrong upstream price feeds a safety control"* — is real, bounded, and +**directional**: + +- **A too-HIGH price is safe.** The cap over-estimates and refuses early; the user loses a turn, not money. +- **A too-LOW price is the dangerous direction** — the cap under-counts and real overspend follows. Two guards + (§9) exist specifically for it, and the pre-egress estimate uses the **highest** applicable context tier for + the same reason. +- And the honest comparison: for a model **not** in the snapshot, today's alternative is **no price at all** — + cost recorded as `0`, cap skipped entirely. **A wrong price still *engages* the cap; a missing price does + not.** That asymmetry is the whole argument. + +### 5. User pricing wins over the catalog — and the divergence is loud + +[ADR-0065](0065-provider-economics-and-extensibility.md) §2 set the precedence as *"static `MODEL_PRICING` wins +for known canonical ids; the overlay fills unknown ids only — a user cannot **silently** misprice a shipped +model."* That rule has a gap this ADR would otherwise open: a user prices an id the table does not know, the +catalog later *learns* that id, and the generated tier silently takes over — the user's explicit override +vanishes with no warning. + +**This ADR narrows that precedence to `user > catalog`**, and the reason comes from ADR-0065 itself: it +introduced **custom `base_url`** endpoints (OpenRouter, Azure, LiteLLM, enterprise gateways). On those, the +public list price is simply **wrong** — the user's negotiated or marked-up rate is the *correct* one. A rule +that lets a public catalog override a rate the user deliberately typed would misprice exactly the users who +took the trouble to be accurate. + +ADR-0065's actual intent — *not **silently*** — is preserved by making the divergence **visible**: when a user +override disagrees with a catalog price, the model picker, `/cost`, and `models pricing` all say so. The user +gets what they asked for; they simply cannot do it in silence. + +Full precedence: **`user` > `catalog snapshot` > (none)**. The **live** tier remains non-authoritative on price +([ADR-0064](0064-live-model-catalog.md) §6, unchanged) and authoritative on availability. + +### 6. Reasoning: the accepted tiers are COMPUTED, never copied + +models.dev's `reasoning_options[].values` are **provider-wire** values (`none | minimal | low | medium | high | +xhigh | max`). Relavium's `ReasoningEffort` is the normalized `off | low | medium | high | max` +([ADR-0066](0066-normalized-reasoning-effort-control.md)). **They are different vocabularies, and reading one +as the other silently breaks three of our four adapters** — a literal read drops `off` from every Claude model +(where `off` is `thinking:{type:'disabled'}`, not an effort value at all) and drops `off`+`max` from `gpt-5.5`. + +So `@relavium/llm` gains a **pure bridge**: + +```text +acceptedTiers(provider, model) = { t ∈ REASONING_EFFORTS : wire(provider, t) ∈ catalog.values(model) } +``` + +with `off` resolved against each provider's **disable axis** — Anthropic `thinking:{type:'disabled'}`, OpenAI +`'none'`, DeepSeek's toggle, and **Gemini's `thinkingBudget: 0`, which `gemini-2.5-pro` does not have** (so +`off` is simply **not in that model's accepted set**). A raw catalog string **never** reaches a picker, a +config, or the wire. + +`ModelPricing.reasoning: boolean` becomes a **control descriptor**: the shape (`effort` / +`budget_tokens{min,max}` / `toggle` / none) and the accepted normalized tiers. The Gemini adapter selects +`thinkingLevel` vs `thinkingBudget` **from that descriptor, per model** — which is what fixes the live bug. + +An out-of-range tier is **never silently promoted** (that would change behaviour *and* raise spend). The picker +offers only accepted tiers; an authored value outside the set fails **pre-flight** with an actionable message; +the run-time last resort is to **withhold the field with a visible notice**. + +### 7. What the retirement breaks, named up front + +`contextWindowForModel(model)` ([pricing.ts](../../packages/llm/src/pricing.ts)) reads +`MODEL_PRICING[model].contextWindowTokens` and is consumed by +[ADR-0062](0062-context-compaction-and-cli-history-commands.md)'s compaction path (the context-fullness +indicator and the auto-compaction threshold). Retiring the table **breaks it**, and it is named here rather +than discovered mid-implementation: it is re-sourced from the catalog, which *widens* it — compaction currently +degrades to "no window known" for the 85 models outside the table, and will stop doing so. + +### 8. Outbound posture — the fifth path + +[security-review.md](../standards/security-review.md) frames the outbound surface as a **closed** inventory: +*"There are **four** outbound-URL paths … and they share one vetted SSRF range-primitive"*. The refresh of §4 +is a **fifth**, and that document is updated in the same change — an ADR that adds an egress path without +amending the inventory would defeat the inventory's purpose. + +Its posture, which is **not** the SSRF posture, and the difference matters: + +- The destination is a **fixed, compile-time constant host** (`models.dev`) — **not** user-supplied and **not** + model-supplied. The SSRF primitive exists to defend paths where an *attacker chooses the URL*; here nobody + does. It is the same category as ADR-0064's provider fetch, not the same as `http_request`. +- **HTTPS only; no cross-host redirect is followed** (a redirect off `models.dev` is an error, not a hop). +- The request carries **no user data, no key, no telemetry** — an unauthenticated `GET` of a public file. +- It lives **host-side** (`apps/cli/src/engine/`), never in `packages/core` or the pure part of `packages/llm` + — engine purity (rule 5) forbids a platform import there. The pure merge keeps taking data as an argument. +- The response is **Zod-validated at the boundary** and normalized before it reaches any Relavium type — the + same discipline [ADR-0064](0064-live-model-catalog.md) §1 applies to a provider's `ModelListing`. + +### 9. Two guards, because a third party now influences a safety-relevant number + +- **A price change on an ALREADY-SHIPPED model fails the sync.** New rows merge automatically; a *moved* price + on a model we already ship is a **human** decision, surfaced as a red CI check, never a silent bot commit. +- **A live conformance test pins the effort mapping against the real API.** The key-gated nightly suite + (`packages/llm/src/conformance/*`) gains: *the tiers we claim this model accepts, it actually accepts.* + Without it, a stale catalog entry re-introduces exactly the Gemini bug, silently — and this is the **only** + mechanism that can catch that. A one-off manual probe cannot: it proves a fact once, not continuously. + +> **Amendment (2026-07-13, implementation — §9 wired).** Both guards are live in +> `.github/workflows/models-catalog.yml`: +> +> - **The weekly price-change check** is `pnpm sync:models` (Monday 06:00 UTC): its money guards fail red ONLY on a +> MOVED or VANISHED price on a model we already ship — the human decision — and pass on benign additive drift. +> Going red on every additive change (models.dev adds our-provider models constantly) would train the maintainer to +> ignore the check, eroding the very protection it exists to give (sync.mjs says so in its own comments). The +> *automatic* PR for the additive case ("new rows merge automatically") is deferred: it needs a create-pull-request +> action pinned to a verified SHA. See deferred-tasks.md. +> - **The live effort conformance test** is `packages/llm/src/conformance/effort.conformance.test.ts` (nightly +> 07:00 UTC, key-gated per provider): for a representative EFFORT-shaped reasoning model per provider it sends +> every tier the catalog claims it accepts and asserts the real API does not reject it. The Gemini probe must be +> effort-shaped (`gemini-3-flash-preview`, not the budget-shaped `gemini-2.5-flash`) — a budget-shaped model +> sends `thinkingBudget`, always accepted, and never the `thinkingLevel` that can 400 on tier drift, so it would +> be a tautology on the very provider whose acceptance was the original bug. An offline gate pins the probe ids +> to real shipped models so a rename cannot silently turn the live lane into a no-op. + +### 10. What the catalog does NOT own — named, so it is not discovered mid-build + +| Need | Status | Where it comes from instead | +|---|---|---| +| **Request shape** — the field *names* a provider wants | **Absent.** models.dev describes *models*, not *wire protocols*. | Provider docs, in the adapter. §10a. | +| **Media output rates** ([ADR-0044](0044-media-access-governance-read-media-save-to-cost.md)) | **Absent** — `cost: null` on every image model. | No shipped model carries one today, so nothing is lost. The field stays, filled from a Relavium overlay if ever needed. | +| **Deprecation dates** | A `status` flag, not an ISO date. | *When to warn a user* is a Relavium **editorial** call, not a data fact — it stays in a small Relavium-owned overlay. Adopting the flag would **lose** information we already have. | +| **`cache_read` when absent** | Absent on 19 of 97 models. | It is **`undefined`, never `0`**. `0` means *"no discount"* in `ModelPricing` and would **price cached input at zero** — a silent undercharge in the mechanism this ADR is hardening. Absent ⇒ fall back to the full input rate. | +| **Reachability with *your* key** | Absent by design. | `listModels` (§1). A catalog cannot know your account. | + +> **Amendment (2026-07-13, implementation — §10 and §11).** Three corrections, recorded rather than rewritten. +> +> 1. **`cache_read` absent ⇒ the full input rate** — §10's rule, which the first implementation of the swap +> violated. `catalogPricing` wrote `?? 0`, and `cost()` bills `cacheReadTokens × rate`, so **0 charges nothing**: +> eleven catalog models publish no cache rate, OpenAI auto-caches, and the cached fraction of every prompt on +> `o1-pro` ($150/MTok in) billed at $0.00. The clause existed precisely to prevent this; the code was written +> against the clause and the test that "checked" it restated the buggy expression. Corrected, and now asserted as +> behaviour (a 100k-token cached read must cost the input rate, not zero). +> +> 2. **The deprecation overlay is REAL, and was briefly deleted.** §10 says the retirement date "stays in a small +> Relavium-owned overlay". The first cut of the swap removed the two DeepSeek dates on the theory that the live +> provider list would supply them. It does not: **no adapter populates `ModelListing.deprecatedAt`** (the +> OpenAI-compatible list is id-only), so `deprecated` became permanently `false` for every model in the product +> and `deepseek-chat` was set to stop working in eleven days with nothing to say so. The overlay +> (`packages/llm/src/catalog/deprecations.ts`) is what §10 always specified: one date per model, from a published +> announcement — not a second price table. +> +> 3. **§11's context tiers are now CONSUMED.** They were parsed, guarded and exported, and read by no pricing code — +> so a >200k `gemini-2.5-pro` turn billed at the cheap tier. That was a tolerable gap while those models threw +> `UnknownModelError`; it became a silent 2× under-bill the moment the catalog started pricing them. The realized +> fold prices the tier the prompt landed in; the pre-egress estimate takes the HIGHEST, because a cap that +> over-estimates refuses a turn the user could have afforded while one that under-estimates lets money escape. + +#### 10a. `max_tokens` vs `max_completion_tokens` — a rule, not a probe + +Our OpenAI adapter calls **Chat Completions** and sends `max_tokens`. OpenAI's reasoning models (o-series, +GPT-5) **reject** it and require `max_completion_tokens` — which is the second half of the maintainer's +"max tokens errors" report (the first half being the absence of any clamp against `limit.output`, §7). The +same adapter also serves **every custom OpenAI-compatible `base_url`** +([ADR-0065](0065-provider-economics-and-extensibility.md)) — LM Studio, Ollama, vLLM, LiteLLM, enterprise +gateways — most of which implement only the legacy `max_tokens`. Switching the field globally would trade one +broken population for another. + +**The rule:** the **official OpenAI endpoint** gets `max_completion_tokens` (the current field; `max_tokens` +is the deprecated one). **Everything else** — DeepSeek, and any custom `base_url` — keeps `max_tokens`, the +field every OpenAI-compatible server implements. A per-provider config key overrides it for an exotic endpoint +that wants the modern field. + +> **Amendment (2026-07-13, implementation).** Two corrections, recorded rather than rewritten (ADRs are +> append-only). +> +> 1. **"Official" is decided by HOST, not by "did a caller pass a base URL".** The first implementation read +> `deps.baseURL !== undefined` as *custom*, and the CLI stores a `--base-url` **verbatim** — so a user who +> registers OpenAI with its own endpoint spelled out (`https://api.openai.com/v1/`, trailing slash) was +> classified custom, and got the deprecated field **and no clamp** on the official API. That is this very bug, +> restored by a typo. The adapter now compares the resolved `hostname` against the provider's own host. +> +> 2. **The "per-provider config key" does not exist, and is not needed.** The override an exotic gateway actually +> has is the **`providerOptions` escape hatch** ([ADR-0011](0011-internal-llm-abstraction.md) §1.D): a request +> that carries no `maxTokens` but sets `providerOptions.max_completion_tokens` sends exactly that. What the +> first implementation got *wrong* was the collision — the mapped cap and an escape-hatch cap under a +> **different key** both reached the wire, and OpenAI rejects a request carrying both. The adapter now +> guarantees **exactly one** cap field: whichever we map wins outright and the other key is dropped; an +> un-mapped escape-hatch cap stands untouched. A config key can be added if a real endpoint ever needs one — +> none has. + +*Considered discovering the dialect at runtime (send the modern field; on a 400 that names the parameter, retry +once with the legacy field and cache the result per endpoint) — **rejected**.* It burns a real turn to learn +what a constant already tells us; it depends on **string-matching a provider's error message**, which is not a +contract; and it introduces mutable per-endpoint state that has to be persisted, invalidated, and made +deterministic for tests. A rule with a documented escape hatch is knowable, testable, and costs the user +nothing. + +### 11. The seam holds, so the source is replaceable + +models.dev's raw shape (`reasoning_options`, `cost.tiers`, `limit.input`) is Zod-parsed and normalized at the +boundary and **never appears in `@relavium/llm`'s public types** — the pattern +[ADR-0064](0064-live-model-catalog.md) §1 already established for `ModelListing` (and the same instinct +[ADR-0011](0011-internal-llm-abstraction.md) applies to vendor SDKs, though models.dev is an aggregator, not a +provider SDK). models.dev is therefore an **implementation detail of the snapshot generator**, not an +architectural commitment: when Relavium builds its own metadata source, it swaps behind the same normalized +types — one file, not a refactor. + +### 12. Per-model REQUEST capabilities — the same "a table cannot say what is true" fix, one more axis + +> **Amendment (2026-07-14, implementation).** The same class this ADR already fixed for **pricing** (a hand-typed +> table → a per-model generated row) and **reasoning shape** (§6, a provider-wide boolean → a per-model control +> descriptor) has a third instance: **which request PARAMETERS a model accepts.** `temperature`, `tool_call`, +> `structured_output` and `attachment` all vary **per model within a single provider** — models.dev publishes each +> as a per-model boolean, and (checked against the live payload) all four range over `[false, true]` for the +> providers Relavium serves. The pre-existing `CapabilityFlags` (`packages/llm/src/types.ts`) is a **per-PROVIDER** +> struct, so it cannot express that `gpt-5.6-luna` rejects `temperature` while its siblings accept it; sending the +> parameter to a model that rejects it is a live 400 (the reported bug). +> +> - **The data lives on the model, not the seam.** A per-model `requestCapabilities` descriptor is carried on +> `CatalogModel` (parsed from models.dev, normalized like `reasoning`), NOT forced into the provider-level +> `CapabilityFlags` — exactly as §6 put the reasoning shape on the model. It is **enrichment**, parsed leniently +> (a shape change never evicts a priced model — the §11/M7 rule), and stores **only a `false`**: absent ⇒ +> **accepted**, the safe default, so a model with no capability data (a custom `base_url`, a brand-new id) is +> never denied a parameter it takes on the strength of missing metadata. +> - **The adapters WITHHOLD, they do not send-and-fail.** `modelAccepts(model, param)` gates the wire parameter in +> all three OpenAI-compatible/Anthropic/Gemini adapters. A rejected `temperature`/`structured_output` is dropped, +> turning a guaranteed 400 into a served turn at the model's default — the same "withhold, never guess" posture +> §6 established for reasoning. +> - **Anthropic thinking pins `temperature` to 1.** Independently, enabling extended thinking (`thinking:{enabled| +> adaptive}`) forbids any other temperature on Anthropic. The adapter now withholds a caller temperature whenever +> it enabled thinking for the turn — the reasoning the author selected wins, and the 400 is removed. +> - **Scope, honestly.** `temperature` and `structured_output` are safe **silent** param-withholds (a preference/ +> optimization the wire can omit, and the turn still runs). `tool_call` and `attachment` are NOT: dropping tools +> silently would leave the agent unable to act, and dropping an image input would change what the model sees. +> Their catalog data is carried (the mechanism is complete), but gating them is deferred to a **louder** signal +> (config-time validation, or a gate-level notice like §6's effort gate), not a silent drop. A user-facing notice +> for the two silent withholds (so a dropped `temperature` is *said out loud*, §6's own standard) is likewise a +> follow-up — the correctness fix (no 400) ships first. +> - **The data populates via the sync, not by hand.** The snapshot is GENERATED (§3); these fields are parsed by +> the schema and land on the next `pnpm sync:models`. Until then the mechanism is inert-but-correct (absent ⇒ +> accepted), and the shipped snapshot is **not hand-edited** — the generated-file discipline holds. + +### K7. `strict_cost_cap` — refuse a turn on a model we cannot price + +> **Amendment (2026-07-14, recorded — append-only).** The `ADR-0071 §K7` citations across the config spec, the +> workflow spec, and the engine all refer to THIS decision, which had no home section: `K7` was an implementation +> milestone label that leaked into the citations before the rule was written up. Recorded now so the citations +> resolve, and numbered `K7` to match them rather than renumbering forty-five references. +> +> A cost cap cannot bound a turn on a model it cannot PRICE — the pre-egress governor has no estimate, so it +> degrades to `allow` and says so ONCE (`onUnpriced`): a cap that silently does not apply is a false sense of +> safety, and one notice per model is the honest middle between silence and per-turn noise. For a self-hosted model +> with ~no metered cost that is the right trade. But a user who set the cap SPECIFICALLY to bound an untrusted model +> wants the opposite, so **`strict_cost_cap`** (default `false`; `[chat].strict_cost_cap` / +> `budget.strict_cost_cap`) makes the governor **REFUSE** the turn instead — *"if we cannot price it, we do not run +> it."* It is INERT without a positive `max_cost_microcents` (no cap ⇒ nothing to enforce), and `models pricing +> ` closes the hole either way (once priced, the cap applies normally, pre-egress and realized). + +## Consequences + +### Positive + +- **The cost cap becomes a real control.** Priced models go from 12 to 97 — at install time, offline, by + default. That is the property a *live-only* fetch could never have guaranteed. +- **The Gemini live bug is fixed**, and its whole class becomes unrepresentable: the control's shape is data, + not an adapter-wide assumption. +- `gpt-5.4-pro` — the model that produced the bug report — is priced, its effort tiers are correct, and its + output ceiling is known. One row, three findings. +- **Compaction gets wider**, not narrower: `contextWindowForModel` stops returning `undefined` for the 85 + models outside the old table (§7). +- Two silent drifts corrected (`claude-sonnet-4-6` 64k→128k, `gpt-5.5` context 1M→1.05M). +- **Adding a provider stays cheap**: adapter + enum entry + one line in `CATALOG_PROVIDER_KEYS` (§2). + +### Negative + +- **A new (fifth) outbound path**, superseding ADR-0064's "no new egress" clause. Mitigated by being + **default-OFF** (§4), a fixed host with no user data (§8), additive-only, and a no-op offline — but it is + new, it is recorded here, and [security-review.md](../standards/security-review.md) is amended in the same + change rather than left stale. +- **A third party can now influence a safety-relevant number** for models outside the shipped snapshot. + Bounded by the additive rule and the two guards (§9), and directional: a too-low price is the dangerous one, + which is why the pre-egress estimate takes the **highest** applicable tier. Compared honestly against the + status quo — *no price at all* — a wrong price still engages the cap. +- **`contextWindowForModel` and every other `MODEL_PRICING` consumer must be re-sourced** (§7). Named, not + discovered. +- **[ADR-0065](0065-provider-economics-and-extensibility.md)'s precedence is narrowed** (§5): `user` now beats + the catalog. This is a deliberate reversal of "static wins for known canonical ids", justified by ADR-0065's + own custom-`base_url` feature, and it keeps that rule's actual intent ("not *silently*") by making any + divergence visible. +- **Freshness is bounded** by the release cadence with the refresh off (the default). A brand-new model is + unpriced until the next release or an explicit `models refresh`. The day-0 escape hatch already exists: + `relavium models pricing `. +- Tiered (context-size) pricing must now be modelled: the cap is a safety control, and a flat rate understates + long-context spend by up to 2×. `ModelPricing` gains an optional tier field. + +### Neutral + +- `ProviderId` stays **closed**; the pure merge stays in `@relavium/llm`; the live tier stays non-authoritative + on price. ADR-0064's architecture is preserved — only the static tier's *source* and its *egress* sentence + change, and its Related pointer to `pricing.ts` as the registry's canonical home is amended in place. +- [ADR-0066](0066-normalized-reasoning-effort-control.md)'s normalized five-tier vocabulary is **unchanged**. + What changes is that a model now *declares* which of those tiers it accepts, and what native shape carries + them, instead of the adapter assuming both. +- **No new runtime dependency** (§3) — native `fetch`, existing Zod, `node:crypto`. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index f582f5ca..0376ec78 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -114,6 +114,7 @@ flowchart TD | 0068 | [Full-screen TUI renderer, `ink` 7, and the CLI component test harness (refines ADR-0047)](0068-full-screen-tui-renderer-ink7-harness.md) | Accepted | 2026-07-09 | | 0069 | [`string-width` for the CLI renderer's display-width measurement](0069-string-width-for-the-cli-renderer.md) | Accepted | 2026-07-10 | | 0070 | [Durable per-model session cost attribution — the `session_costs` aggregate, single-owner cost writes, and the reconciliation invariant](0070-durable-per-model-session-cost-attribution.md) | Accepted | 2026-07-12 | +| 0071 | [models.dev as the model-metadata source; the hand-maintained registry is retired (supersedes two ADR-0064 clauses, corrects an ADR-0066 premise)](0071-models-dev-as-the-model-metadata-source.md) | Accepted | 2026-07-13 | ## Creating a new ADR diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index a2f25037..fee5083d 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -128,7 +128,9 @@ The command set below is the confirmed surface. Commands ship **per workstream** | `relavium init` _(planned)_ | Initialize a `.relavium/` directory in the current project. | | `relavium agent ` _(planned)_ | Manage agents (list / create / test). | | `relavium models` | List the cached model catalog (refreshes on first run if the cache is empty). See [`relavium models`](#relavium-models). | -| `relavium models refresh` | Force a live re-fetch of each connected provider's model list into the local cache, reporting per-provider outcomes. | +| `relavium models refresh` | Re-fetch **both axes**: each connected provider's live model list (availability) **and** the models.dev catalog (price, ceilings, reasoning tiers). Reports per-source outcomes. | +| `relavium models refresh --providers` | Availability only — the provider lists. ADR-0064's original behaviour. | +| `relavium models refresh --catalog` | Metadata only — the models.dev catalog. **Needs no provider key.** | | `relavium models pricing ` | Hand-enter a user price for a model the registry does not know, so the cost cap enforces it. See [`relavium models`](#relavium-models). | | `relavium provider ` | Manage providers and API keys in the OS keychain (`list` / `add` / `set-key` / `remove-key` / `test`). | @@ -224,12 +226,18 @@ relavium models pricing my-custom-model --provider openai --input 3 --output 9 ``` - **`relavium models`** (no subcommand) lists the cached catalog (read-only). On the **very first run** — when the cache is empty — it does one minimal **blocking** refresh, then lists; an empty result stays a clean exit `0` (an empty catalog is not a fault, like `relavium list`). Human output is one line per model (` ctx= []`). -- **`relavium models refresh`** forces a live re-fetch of **each connected provider** (a provider whose key resolves via the OS keychain → `RELAVIUM__API_KEY` env var) and prints a per-provider outcome. The refresh is **per-provider isolated**: one provider's failure (bad key, network, endpoint drift) or a provider without a list endpoint **never** fails the whole command — that provider is reported `failed` / `skipped` and the others still refresh. A per-provider failure is therefore **not** a command failure (exit `0` with the report). The **one** hard fault is an explicit `refresh` with **zero** providers connected (no key at all): that is a clean exit `2` naming how to add a key, because nothing could be fetched. -- **`relavium models pricing --provider --input --output [--cached ]`** hand-enters the per-million-token price of a model the static registry does **not** know — a custom-endpoint model, or a new provider model not yet in the shipped [pricing.ts](../../../packages/llm/src/pricing.ts) (2.5.G S10, [ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) §1–2). Prices are **USD per million tokens** (`--input` prompt, `--output` completion, `--cached` cache-read; stored as integer micro-cents, `usd × 1e8`, never a float). The row is written as `source='user'` and a live `models refresh` **never** clobbers it. This **closes the cost-cap gap** ([ADR-0064](../../decisions/0064-live-model-catalog.md) §6): before, an unknown model had no price, so `budget.max_cost_microcents` / `[chat].max_cost_microcents` **degraded to allow** for it; once user-priced, the cap is enforced (pre-egress **and** realized) on `run`, a `run` resumed via [`relavium gate`](#relavium-gate), `chat` / `chat-resume` (incl. a `/clear` rebuild, re-read fresh), the Home chat, and one-shot `agent run`. Guards (each a clean exit `2`, nothing written): a **canonical** model id is refused (the shipped price always wins, so an override would be silently ignored); an **unregistered provider** is refused (register it first with `relavium provider add`); a **negative / non-finite / implausibly-large** price is refused; and the **same model id already user-priced under a *different* provider** is refused (the overlay keys by model id, so a second provider's price could not be distinguished — use a distinct id or re-price under that provider). The static registry still wins for a known id, so a user can never misprice a shipped model. +- **`relavium models refresh`** re-fetches **two axes**, because *"refresh what I know about models"* is one intent, not two ([ADR-0071](../../decisions/0071-models-dev-as-the-model-metadata-source.md) §4a): the **provider lists** say which models a key can reach, and the **catalog** says what they cost and which reasoning tiers they take. `--providers` and `--catalog` address one axis each; neither flag (or both) means both. + + The provider half is **per-provider isolated**: one provider's failure (bad key, network, endpoint drift) or a provider without a list endpoint **never** fails the whole command — that provider is reported `failed` / `skipped` and the others still refresh. A per-provider failure is therefore **not** a command failure (exit `0` with the report). An explicit `--providers` refresh with **zero** providers connected (no key at all) is a clean exit `2` naming how to add a key, because nothing could be fetched. + + The catalog half needs **no key at all** — which is what makes `--catalog` the form a user with no provider configured yet can still run to see what things cost. A failed catalog fetch is a **note on stderr, never an error**: the shipped snapshot still answers, so a refresh can never leave Relavium knowing less than it shipped knowing. Under `--json`, each source emits its own record (`{ provider, … }` per provider, then `{ source: 'catalog', status, models, added }`). + +- **The catalog is EMBEDDED and answers offline.** Relavium ships a generated models.dev snapshot — every price, ceiling and reasoning control — so a fresh install with no network prices every model it ships. `[catalog] auto_refresh` (**default `false`**) governs whether a bare `relavium models` refreshes it first; an explicit `models refresh` fetches regardless, because a command the user typed **is** consent. A local-first tool does not contact a third party unless asked. +- **`relavium models pricing --provider --input --output [--cached ]`** hand-enters the per-million-token price of a model. It MAY be a model the generated catalog (`packages/llm/src/catalog/`) already prices — **the USER outranks the catalog** ([ADR-0071](../../decisions/0071-models-dev-as-the-model-metadata-source.md) §5): a negotiated rate, an enterprise discount, or a price the snapshot has not caught up with wins over the shipped one (2.5.G S10, [ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md) §1–2). Prices are **USD per million tokens** (`--input` prompt, `--output` completion, `--cached` cache-read; stored as integer micro-cents, `usd × 1e8`, never a float). The row is written as `source='user'` and a live `models refresh` **never** clobbers it. Overriding a catalog price is **not silent**: the command echoes the catalog price it replaces, so the divergence is stated at set-time (ADR-0071 §5). This also **closes the cost-cap gap** ([ADR-0064](../../decisions/0064-live-model-catalog.md) §6) for a model the catalog does NOT price: before, an unknown model had no price, so `budget.max_cost_microcents` / `[chat].max_cost_microcents` **degraded to allow** for it; once user-priced, the cap is enforced (pre-egress **and** realized) on `run`, a `run` resumed via [`relavium gate`](#relavium-gate), `chat` / `chat-resume` (incl. a `/clear` rebuild, re-read fresh), the Home chat, and one-shot `agent run`. `--clear` retires the override; the model falls back to the catalog price, or to unpriced if the catalog has none. Guards (each a clean exit `2`, nothing written): an **unregistered provider** is refused (register it first with `relavium provider add`); a **negative / non-finite / implausibly-large** price is refused; a price under a provider the catalog does **not** anchor the model to is refused (it would never apply); and the **same model id already user-priced under a *different* provider** is refused (the overlay keys by model id, so a second provider's price could not be distinguished — use a distinct id or re-price under that provider). - **Security.** A provider key is read only to make the live request (over the bounded, abortable, secret-free `listModels` seam) and is **never** logged, persisted (the cache holds no key), or placed in the report / `--json` payload / any error message. A failing provider surfaces only the seam's already-redacted message (or a generic `refresh failed`), never a raw cause. `models pricing` writes only a model id + provider + integer prices — no key, ever. - **`--json`** ([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)) emits **one NDJSON record per line**, stdout-pure, key-free: - `relavium models --json` — one record per model: `{ provider, modelId, displayName, contextWindowTokens, maxOutputTokens, source, lastRefreshedAt, deprecationDate }` (`null` for an absent optional; `source` ∈ `static | live | user`; `lastRefreshedAt` is epoch-ms). - - `relavium models refresh --json` — one record per provider: `{ provider, status, added, updated, deactivated, error }`, where `status` ∈ `refreshed | skipped-no-key | skipped-unsupported | failed`, the three counts are the model ids added / refreshed-in-place / soft-deactivated (`null` unless `status` is `refreshed`), and `error` is a short, secret-free reason (`null` unless `failed`). + - `relavium models refresh --json` — the **two axes** each emit their own record (§4a). A per-**provider** record: `{ provider, status, added, updated, deactivated, error }`, where `status` ∈ `refreshed | skipped-no-key | skipped-unsupported | failed`, the three counts are the model ids added / refreshed-in-place / soft-deactivated (`null` unless `status` is `refreshed`), and `error` is a short, secret-free reason (`null` unless `failed`). And, when the catalog axis ran, exactly one **catalog** record: `{ source, status, models, added, reason? }` — `source` is always the literal `'catalog'`; `status` ∈ `refreshed | failed`; `models` is the total the refreshed catalog describes and `added` the count the shipped snapshot did **not** carry (both `0` on a failure — the snapshot still answers); `reason` is present **only** on `failed` and is a short, secret-free sentence (never a raw fetch error, which can carry a URL). - `relavium models pricing --json` — one record: `{ model, provider, source, inputCostPerMtokMicrocents, outputCostPerMtokMicrocents, cachedInputCostPerMtokMicrocents }` (the stored integer micro-cents; `source` is always `user`). ### Read-command `--json` output diff --git a/docs/reference/contracts/config-spec.md b/docs/reference/contracts/config-spec.md index ccf13506..11469724 100644 --- a/docs/reference/contracts/config-spec.md +++ b/docs/reference/contracts/config-spec.md @@ -64,8 +64,24 @@ MCP server registrations follow the same merge: globally registered servers (`co ```toml update_channel = "stable" # stable | beta +[catalog] +# Refresh the model catalog (price, ceilings, reasoning tiers) from models.dev automatically. +# DEFAULT false — and the default is the design (ADR-0071 §4). +# +# Relavium SHIPS a generated models.dev snapshot: a fresh install with no network prices every model +# it ships, offline, and contacts nobody. A local-first tool that reaches out to a third party by +# default violates its own spirit even when the payload is innocuous, so the standing egress is +# opt-in. `relavium models refresh` fetches regardless — a command the user typed IS consent, and +# that is what makes default-OFF a livable default rather than a dead end. +# +# `true` ⇒ a bare `relavium models` refreshes the catalog first (a failure is silent; the snapshot +# still answers). The refresh is ADDITIVE ONLY: it may add models and enrich thin ones, and it can +# never leave a model less priced than the shipped snapshot did. +auto_refresh = false + [preferences] default_model = "claude-sonnet-4-6" +default_provider = "anthropic" # ADR-0059: the provider serving default_model — anthropic | openai | gemini | deepseek. Persisted by the /models picker + the onboarding wizard at pick time (authoritative there — the live model list is per-provider), so the next chat resolves the provider WITHOUT id inference. Load-bearing for a live-discovered id whose spelling the prefix map cannot place (e.g. chatgpt-4o-latest). Absent ⇒ inference from the id (catalog first, then prefix). reasoning_effort = "medium" # ADR-0066 §6: the GLOBAL default reasoning-effort tier — off | low | medium | high | max; the fallback BELOW any [chat].reasoning_effort. Written by the /models picker's effort sub-step. Absent ⇒ no reasoning control (the provider default). theme = "dark" alt_screen = true # ADR-0068 §e (2.6.F): the full-screen alternate-screen renderer for the bare Home + `relavium chat`. Since Step 4b-3 the DEFAULT is ON — a TTY opens full-screen — so this key is the durable OPT-OUT: `false` keeps the byte-identical INLINE renderer (native scrollback + the emulator's own a11y — the screen-reader fallback), `true` forces it on. The `--no-alt-screen` flag is the per-invocation opt-out and overrides this key; a non-TTY / `--json` / CI path always renders inline regardless. Absent ⇒ the phase default (alt-ON since 4b-3 — ADR-0068 §b). @@ -96,9 +112,10 @@ stdio-only fields (`command`/`args`/`env`) are rejected on a network registratio > **Writing the global config** ([ADR-0063](../../decisions/0063-cli-config-write-contract.md)). Config is > almost entirely **read-only** (hand-edited, git-committed). The one write path is the CLI persisting a chosen -> default: `/models` and the 2.5.G onboarding wizard set **`[preferences].default_model`**, and (ADR-0066 §6) the -> `/models` picker's **effort sub-step** additionally sets **`[preferences].reasoning_effort`** for a reasoning -> model — **only** those two `[preferences]` keys, and only through a **typed setter** (never a generic key/value +> default: `/models` and the 2.5.G onboarding wizard set **`[preferences].default_model`** and (ADR-0059) its +> **`default_provider`** in one atomic write, and (ADR-0066 §6) the `/models` picker's **effort sub-step** +> additionally sets **`[preferences].reasoning_effort`** for a reasoning model — **only** those three `[preferences]` +> keys, and only through a **typed setter** (never a generic key/value > writer), so a secret can never be written by construction (there is no `api_key` field in the schema; keys live > only in the OS keychain, [ADR-0006](../../decisions/0006-os-keychain-for-api-keys.md)). A partial write leaves > the other key unchanged (a model-only pick never clears a prior effort default). @@ -134,6 +151,7 @@ focus_area = "security and type safety" [chat] # agent-session (chat-mode) defaults — see contracts/agent-session-spec.md default_model = "claude-sonnet-4-6" # model for a chat session that names none; absent at every [chat] layer ⇒ falls back to global [preferences].default_model (ADR-0063) +default_provider = "anthropic" # ADR-0059: the provider serving [chat].default_model, resolved from the SAME config layer as the effective default_model (NEVER independently — a fresh model paired with a stale layer's provider dials the wrong adapter → 404). Absent when the model resolves ⇒ inferred from the id (catalog first, then prefix). reasoning_effort = "medium" # ADR-0066: reasoning-effort tier baked onto the DEFAULT chat agent — off | low | medium | high | max; absent ⇒ no reasoning control (the provider default). Ignored on a model without a controllable reasoning tier. fs_scope = "sandboxed" # SAME tier enum as [defaults].fs_scope above (not re-listed here) max_turns = 50 # hard session TURN cap → SessionDeps.maxTurns (DoS fail-safe; absent ⇒ engine default 50; positiveInt — 0 is rejected here) — DISTINCT from max_messages @@ -142,6 +160,11 @@ auto_compact = true # ADR-0062: auto-compact when a turn's real i compact_threshold = 0.8 # ADR-0062: the context-window fraction that triggers auto-compaction; a fraction in (0, 1] (absent ⇒ 0.8) max_cost_microcents = 0 # 0 = unbounded; >0 = per-session pre-egress cost cap (the same governor as a workflow budget — ADR-0028) on_exceed = "pause_for_approval" # fail | pause_for_approval | warn — when a session hits its cap +strict_cost_cap = false # ADR-0071 §K7: refuse a turn on a model we cannot PRICE. Default false — the cap + # degrades to allow with a one-time notice (an unpriced model is a hole in it); + # true blocks the turn. `models pricing ` closes the hole either way. + # INERT without a positive max_cost_microcents: with no cap there is nothing to + # enforce, so strict_cost_cap does nothing until a cap is also set. allowed_commands = [] # !-shell allowlist (ADR-0061): EXACT full-command-string match; EMPTY/absent ⇒ !-shell disabled allowed_command_globs = [] # opt-in glob form of the !-shell allowlist (riskier); empty/absent ⇒ none ``` @@ -166,7 +189,7 @@ allowed_command_globs = [] # opt-in glob form of the !-shell allowlist ( > > `reasoning_effort` ([ADR-0066](../../decisions/0066-normalized-reasoning-effort-control.md)) is the normalized reasoning-effort tier — `off | low | medium | high | max` — baked onto the **built-in default chat agent** only (an explicit `--agent` owns its own `reasoning_effort` in its YAML). It resolves per-field project → workspace, and (ADR-0066 §6) — like `default_model` — additionally falls back to the global **`[preferences].reasoning_effort`** below both `[chat]` layers. Each adapter maps the tier to its provider's **native** control; a model with no controllable reasoning tier ignores it (the engine gates on the model's capability). Absent ⇒ no reasoning control (the provider default). Interactively, the `/effort` command and a **live chat**'s `/models` effort sub-step set the tier as a **per-turn session override** (no reseat) without editing config; the **bare-Home** `/models` effort sub-step instead writes the `[preferences].reasoning_effort` default for the next session. The active tier shows in the footer. > -> The `[chat]` block resolves **per field** (each key independently, last-writer-wins project → workspace) — a project that sets only `max_turns` still inherits `default_model`/`max_messages` from the workspace layer. (Contrast `[defaults].media_cost_estimate`, which resolves **whole-object**: the highest layer present replaces the table outright.) **`default_model` and `reasoning_effort` each have one extra fallback**: absent at both `[chat]` layers, each falls through to its global **`[preferences]`** counterpart (`default_model` per [ADR-0063](../../decisions/0063-cli-config-write-contract.md) §1; `reasoning_effort` per [ADR-0066](../../decisions/0066-normalized-reasoning-effort-control.md) §6) — the write targets of `/models` (model + its effort sub-step) and the wizard — so a user's "preferred model / effort everywhere" governs chat too, exactly as `[preferences].default_model` already governs a workflow's `[defaults].model`. Full precedence for each: `[chat].` (project → workspace) → `[preferences].` (global). No OTHER `[chat]` field reads the global layer. The `!`-shell allowlist is the **one exception**: `allowed_commands` (exact) + `allowed_command_globs` (globs) are a **coupled unit**, so a project that sets **either** array owns the **whole** allowlist and does **not** inherit the other array from the workspace. Otherwise a project narrowing `allowed_commands` would silently keep the workspace's broader globs — lock to `git status`, yet still allow `git push` via an inherited `git *`. Only when a project sets **neither** allowlist array do both fall through to the workspace; a present array otherwise REPLACES (never merges) the lower layer's. This is what guarantees a narrower project can never inherit a broader workspace entry. +> The `[chat]` block resolves **per field** (each key independently, last-writer-wins project → workspace) — a project that sets only `max_turns` still inherits `default_model`/`max_messages` from the workspace layer. (Contrast `[defaults].media_cost_estimate`, which resolves **whole-object**: the highest layer present replaces the table outright.) **`default_model` and `reasoning_effort` each have one extra fallback**: absent at both `[chat]` layers, each falls through to its global **`[preferences]`** counterpart (`default_model` per [ADR-0063](../../decisions/0063-cli-config-write-contract.md) §1; `reasoning_effort` per [ADR-0066](../../decisions/0066-normalized-reasoning-effort-control.md) §6) — the write targets of `/models` (model + its effort sub-step) and the wizard — so a user's "preferred model / effort everywhere" governs chat too, exactly as `[preferences].default_model` already governs a workflow's `[defaults].model`. Full precedence for each: `[chat].` (project → workspace) → `[preferences].` (global). **`default_provider` ([ADR-0059](../../decisions/0059-in-place-model-reseat.md)) also reads the global layer, but COUPLED to `default_model`** — NOT resolved independently: it is taken from the SAME layer that supplied the model (absent there ⇒ id inference), so a lower layer's stale provider can never pair with a higher layer's model (an `openai` provider leaking onto a project's `claude` `default_model` would bind the wrong adapter). No OTHER `[chat]` field reads the global layer. The `!`-shell allowlist is a **second coupled group**: `allowed_commands` (exact) + `allowed_command_globs` (globs) are a **coupled unit**, so a project that sets **either** array owns the **whole** allowlist and does **not** inherit the other array from the workspace. Otherwise a project narrowing `allowed_commands` would silently keep the workspace's broader globs — lock to `git status`, yet still allow `git push` via an inherited `git *`. Only when a project sets **neither** allowlist array do both fall through to the workspace; a present array otherwise REPLACES (never merges) the lower layer's. This is what guarantees a narrower project can never inherit a broader workspace entry. > > `allowed_commands` / `allowed_command_globs` gate the **`!`-shell escape** (2.5.D, [ADR-0061](../../decisions/0061-cli-input-layer-file-injection-and-shell-escape.md)) — a chat user typing `!command` runs it through the **one** `run_command` boundary (they map to the engine's camelCase `allowedCommands` / `allowedCommandGlobs`, the SAME allowlist a workflow `run_command` uses). `allowed_commands` is **exact full-command-string** match (`git status`, `ls -la` — `git` never authorizes `git push --force`); `allowed_command_globs` is the opt-in, riskier pattern form. **Both default to EMPTY ⇒ `!`-shell is disabled** — the `empty ⇒ disabled` symmetry [security-review.md](../../standards/security-review.md) pins, with **no chat-specific relaxation** (there is no curated default: `run_command` has no argument/file confidentiality floor, so even a "read-only" default set — `cat`, `grep` — would reopen `!cat .env` → provider). `!`-shell is first-class via a first-class **opt-in** (the user lists commands, or the 2.5.G onboarding offers a reviewed seed), and a non-allowlisted `!cmd` gets an **actionable, secret-free deny hint** naming the exact line to add. `enforcePolicy(allowedCommands)` runs **before** the mode-aware `confirmAction`, so even `auto` mode never runs a command absent from the allowlist. Editing chat `allowed_commands` is a [security-review.md](../../standards/security-review.md) trigger. diff --git a/docs/reference/contracts/workflow-yaml-spec.md b/docs/reference/contracts/workflow-yaml-spec.md index 6fc15271..5b7440fb 100644 --- a/docs/reference/contracts/workflow-yaml-spec.md +++ b/docs/reference/contracts/workflow-yaml-spec.md @@ -327,11 +327,12 @@ workflow: budget: max_cost_microcents: 5000000 # ~$0.05 cap (integer micro-cents) on_exceed: pause_for_approval # fail | pause_for_approval | warn + strict_cost_cap: false # ADR-0071 §K7: refuse a turn on a model with no price (default false) timeout_ms: 300000 # whole-run wall-clock cap max_parallel: 4 # max concurrent in-flight LLM calls (bounds a wide fan-out) ``` -The cost cap is **pre-egress**: before each LLM call the engine checks `cumulative + worstCaseNextEstimate(maxTokens)` against `max_cost_microcents` and applies `on_exceed` — `fail` stops the run, `pause_for_approval` suspends it like a human gate (resumed via `resume_budget`), `warn` proceeds after a `budget:warning` event. This is a **BYOK-local** safety rail, distinct from Phase-2 managed-mode billing ([ADR-0014](../../decisions/0014-managed-metering-quota-and-billing.md)). The `budget:warning` / `budget:paused` / `run:timeout` events are defined in [sse-event-schema.md](sse-event-schema.md). +The cost cap is **pre-egress**: before each LLM call the engine checks `cumulative + worstCaseNextEstimate(maxTokens)` against `max_cost_microcents` and applies `on_exceed` — `fail` stops the run, `pause_for_approval` suspends it like a human gate (resumed via `resume_budget`), `warn` proceeds after a `budget:warning` event. An **unpriced model** (a custom `base_url`, or one the catalog does not carry) is a hole in the cap: the governor cannot estimate a turn's cost, so it **degrades to `allow` with a one-time notice** (ADR-0071 §K7) rather than hard-failing an otherwise-valid run. `strict_cost_cap: true` inverts that — an unpriced model is refused pre-egress, for a run that must not proceed on a model it cannot bound. This is a **BYOK-local** safety rail, distinct from Phase-2 managed-mode billing ([ADR-0014](../../decisions/0014-managed-metering-quota-and-billing.md)). The `budget:warning` / `budget:paused` / `run:timeout` events are defined in [sse-event-schema.md](sse-event-schema.md). ## Edges diff --git a/docs/reference/desktop/database-schema.md b/docs/reference/desktop/database-schema.md index cbbac6ba..854e1123 100644 --- a/docs/reference/desktop/database-schema.md +++ b/docs/reference/desktop/database-schema.md @@ -86,7 +86,7 @@ CREATE UNIQUE INDEX idx_llm_providers_name ON llm_providers (name) WHERE deleted Models offered by each provider, including pricing used for local cost tracking. The `*_per_mtok_microcents` columns are price **per million tokens, in integer micro-cents** (one micro-cent = 1e-8 USD = cents x 1,000,000; see the [money/cost convention](#sqlite-type-conventions)). The three `media_*_cost_microcents` columns are the projection of `ModelPricing.mediaOutputRates` (1.AF/D17, [ADR-0044](../../decisions/0044-media-access-governance-read-media-save-to-cost.md) §3) — integer micro-cents **per billed media-output unit** (per image, per audio-second, per video-second); **NULL** when the model has no metered media rate (the realized fold + the pre-egress estimate degrade to 0 for it — H4). `document`/PDF is excluded (it bills as tokens). No shipped model carries a media rate yet, so these are NULL across the seeded catalog. -**Live-discovery cache role ([ADR-0064](../../decisions/0064-live-model-catalog.md) §4/§5).** As of 2.5.G this table doubles as the **live-discovery cache** — "which model ids a given key can reach" — filled by a bulk refresh over the seam's `listModels`, with the static `MODEL_PRICING` registry enriching **at read time** (the registry is **never** seeded into the DB — that would create a second, drift-prone pricing home). The `source` discriminant records provenance: **`static`** (a hardcoded capability/media seed — the media-routing `upsert` path's default), **`live`** (discovered via `listModels` — the refresh writes it), **`user`** (user-supplied pricing, [ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md)). `last_refreshed_at` is the freshness stamp backing the 24h TTL. The bulk refresh (`replaceProviderModels`) **soft-deactivates** (`is_active = 0`, `deleted_at` left NULL) every currently-active `source='live'` row of a provider whose model id vanishes from the new list, and **reactivates** a reappearing one by reusing the same row — it **never hard-DELETEs** (`model_catalog.id` is an FK target from five tables) and **never touches a `source='user'` or `source='static'` row** (a refresh must not clobber user pricing or regress the media-routing seed). The existing narrow media-routing projection (`resolveMediaSurface` / the D15 capability load-check) is untouched by the widening. Because SQLite `ALTER TABLE ADD` cannot carry a CHECK, the closed `source` value set is validated at the store read boundary (`coerceModelCatalogSource`, degrading a foreign value to `static`), like `media_surface`. +**Live-discovery cache role ([ADR-0064](../../decisions/0064-live-model-catalog.md) §4/§5).** As of 2.5.G this table doubles as the **live-discovery cache** — "which model ids a given key can reach" — filled by a bulk refresh over the seam's `listModels`, with the generated model catalog (ADR-0071) enriching **at read time** (it is **never** seeded into the DB — that would create a second, drift-prone pricing home). The `source` discriminant records provenance: **`static`** (a hardcoded capability/media seed — the media-routing `upsert` path's default), **`live`** (discovered via `listModels` — the refresh writes it), **`user`** (user-supplied pricing, [ADR-0065](../../decisions/0065-provider-economics-and-extensibility.md)). `last_refreshed_at` is the freshness stamp backing the 24h TTL. The bulk refresh (`replaceProviderModels`) **soft-deactivates** (`is_active = 0`, `deleted_at` left NULL) every currently-active `source='live'` row of a provider whose model id vanishes from the new list, and **reactivates** a reappearing one by reusing the same row — it **never hard-DELETEs** (`model_catalog.id` is an FK target from five tables) and **never touches a `source='user'` or `source='static'` row** (a refresh must not clobber user pricing or regress the media-routing seed). The existing narrow media-routing projection (`resolveMediaSurface` / the D15 capability load-check) is untouched by the widening. Because SQLite `ALTER TABLE ADD` cannot carry a CHECK, the closed `source` value set is validated at the store read boundary (`coerceModelCatalogSource`, degrading a foreign value to `static`), like `media_surface`. | Column | Type | Constraints | |--------|------|-------------| @@ -99,6 +99,7 @@ Models offered by each provider, including pricing used for local cost tracking. | `input_cost_per_mtok_microcents` | INTEGER | NOT NULL DEFAULT 0 | | `output_cost_per_mtok_microcents` | INTEGER | NOT NULL DEFAULT 0 | | `cached_input_cost_per_mtok_microcents` | INTEGER | NOT NULL DEFAULT 0 | +| `cached_input_stated` | INTEGER (bool) | NOT NULL DEFAULT 0 — did the USER state the cache rate (migration 0011, ADR-0071 §10)? `0` ⇒ derive it from the catalog discount; `1` ⇒ the number above is the user's own | | `media_image_cost_microcents` | INTEGER | NULL — µ¢ per output image (1.AF/D17) | | `media_audio_cost_microcents` | INTEGER | NULL — µ¢ per output audio-second | | `media_video_cost_microcents` | INTEGER | NULL — µ¢ per output video-second | diff --git a/docs/reference/shared-core/llm-provider-seam.md b/docs/reference/shared-core/llm-provider-seam.md index 982aae80..5aa45a23 100644 --- a/docs/reference/shared-core/llm-provider-seam.md +++ b/docs/reference/shared-core/llm-provider-seam.md @@ -37,7 +37,11 @@ interface LlmRequest { tools?: ToolDef[]; // JSON-Schema params, normalized toolChoice?: 'auto' | 'none' | 'required' | { name: string }; temperature?: number; - maxTokens?: number; // REQUIRED downstream for Anthropic; we default it + maxTokens?: number; // REQUIRED downstream for Anthropic; we default it. CLAMPED to the model's published + // `maxOutputTokens` (ADR-0071 §7) — down, never up; a cap below the ceiling is the author's + // budget. NOT clamped on a custom `base_url` (we cannot describe what it serves). The + // OpenAI-compatible adapter's WIRE FIELD is endpoint-dependent (§10a): OpenAI's own API + // takes `max_completion_tokens`; DeepSeek's API and every custom base URL take `max_tokens`. reasoningEffort?: ReasoningEffort; // normalized tier off|low|medium|high|max (ADR-0066); each adapter maps to its provider's native tier — CANONICAL, wins over a colliding providerOptions key; absent ⇒ provider default stopSequences?: string[]; responseFormat?: ResponseFormat; // structured-output request (ADR-0030) @@ -500,7 +504,7 @@ Per-provider list-models endpoint contracts: | --- | --- | --- | --- | | Anthropic (`anthropic`) | `/v1/models` (SDK `models.list()`, auto-paginating `has_more`/`last_id`) | **Rich** | `id`, `display_name`→`displayName`, `max_input_tokens`→`contextWindowTokens` (omit if 0), `max_tokens`→`maxOutputTokens` (omit if 0). The list is clean — no filter (the rich `capabilities` object is ignored). | | Gemini (`gemini`) | `/v1beta/models` (SDK `models.list()`) | **Rich** | `name` (strip `models/`)→`id`, `displayName`, `inputTokenLimit`→`contextWindowTokens`, `outputTokenLimit`→`maxOutputTokens`. **Filter:** keep only rows whose `supportedActions` (the SDK's projection of REST `supportedGenerationMethods`) includes `generateContent`. Key sent as the `x-goog-api-key` header, never a `?key=` query param (ADR-0064 §9). | -| OpenAI / DeepSeek (`openai-compatible`) | `/v1/models` (SDK `models.list()`) | **Id-only** | `id`→`id`, no context/price. **Filter:** keep the `gpt` / `o` / `*chat*` / `deepseek` families; DENY `embedding`/`tts`/`whisper`/`image`/`moderation`/`realtime`/`audio`/`dall-e`/`transcribe`/`search`/`instruct`/`ocr`/`davinci`/`babbage`/`ft:` — each matched on a `-`/`_` **segment boundary** (so `search` denies `gpt-4o-search-preview` but NOT `o3-deep-research`); **union-in** any id present in `MODEL_PRICING` for that provider (cost-eligibility always wins). | +| OpenAI / DeepSeek (`openai-compatible`) | `/v1/models` (SDK `models.list()`) | **Id-only** | `id`→`id`, no context/price. **Filter:** keep the `gpt` / `o` / `*chat*` / `deepseek` families; DENY `embedding`/`tts`/`whisper`/`image`/`moderation`/`realtime`/`audio`/`dall-e`/`transcribe`/`search`/`instruct`/`ocr`/`davinci`/`babbage`/`ft:` — each matched on a `-`/`_` **segment boundary** (so `search` denies `gpt-4o-search-preview` but NOT `o3-deep-research`); **union-in** any id the generated catalog prices for that provider (ADR-0071 — cost-eligibility always wins; the deny-list still applies, so a priced embedding stays out). | ## What must be normalized diff --git a/docs/roadmap/current.md b/docs/roadmap/current.md index 381a1bc6..6ad27322 100644 --- a/docs/roadmap/current.md +++ b/docs/roadmap/current.md @@ -15,8 +15,9 @@ breakdown, now historical, is in 2026-07-08); its plan is in [phases/phase-2.6-conversational-authoring.md](phases/phase-2.6-conversational-authoring.md). Workstream **2.6.F (platform floor + the full-screen TUI renderer)** is **merged to `main`** -(PR #74, 2026-07-11). **2.6.C** (the reseat transcript-carry + the `/cost` per-model breakdown) is **complete on -`development`** (2026-07-12, PR pending) — see [Active now](#what-is-active-now). +(PR #74, 2026-07-11), and so is **2.6.C** (the reseat transcript-carry + the `/cost` per-model breakdown, +PR #75, 2026-07-13) — see [Active now](#what-is-active-now). **2.6.Q** is next and is **blocked**: it carries +seven decisions the maintainer must answer before any code is written. ## Where we are @@ -72,7 +73,7 @@ it; that `displayWidth` under-counted 8 539 code points against `ink`'s own widt Ctrl-C during a Home hatch stranded mouse reporting on the user's shell. All folded, each with a break-verified regression test. -### Phase 2.6.C — reseat transcript-carry + the `/cost` per-model breakdown (complete on `development`, 2026-07-12) +### Phase 2.6.C — reseat transcript-carry + the `/cost` per-model breakdown (merged to `main`, PR #75, 2026-07-13) Driven by two maintainer manual-test findings. **F1:** a mid-session `/models` reseat **blanked the alt-screen viewport** — the reseat builds a fresh view store, and on the full-screen renderer (the TTY default since 2.6.F, with @@ -104,6 +105,12 @@ step's central change — had **zero test coverage** at any layer. All folded, e - **Reported spend is a systematic under-estimate of the provider's invoice** — an egress that streamed content but ended without a usage chunk, and a mid-stream failure, are recorded as 0 on *both* sides of the invariant. That is a usage-capture gap in the seam, filed against **2.6.Q** (ADR-0070 §3). +- **Cross-turn tool-call memory** — a "the reseat forgets tool calls" report turned out to be an *engine* property, + not a reseat one: tool pairs never cross **any** turn boundary (ADR-0062 §6). Deferred behind a default-off toggle, + but **three already-live bugs gate it** (no pre-egress context-window check; overflow is a fatal, non-retryable + `bad_request`; the budget gate prices only output tokens) — see [deferred-tasks.md](deferred-tasks.md). +- **The CLI e2e suite migrates the developer's real `~/.relavium/history.db`** — verified, and it is what let a + broken migration reach real data while staying invisible to CI. Tracked in [deferred-tasks.md](deferred-tasks.md). **Open obligations carried out of 2.6.F:** diff --git a/docs/roadmap/deferred-tasks.md b/docs/roadmap/deferred-tasks.md index 678f8089..18a4b677 100644 --- a/docs/roadmap/deferred-tasks.md +++ b/docs/roadmap/deferred-tasks.md @@ -551,6 +551,48 @@ ships. **Home:** Phase 3 (or a later Phase-2.6 workstream) — natural sibling to the AgentSession (1.V) "Faithful cross-turn transcript" item just below; whichever lands first should absorb the other. +## Home's chat has no way to view or pick the bound agent (found 2026-07-14) + +> Raised while answering a question about editing the agent used by a Home-opened chat. The headline gap is +> **already tracked** at [phase-2.6-conversational-authoring.md §2.6.G](phases/phase-2.6-conversational-authoring.md) +> — recorded here anyway because the investigation surfaced a narrower residual that 2.6.G's current task +> description does not clearly cover, so it doesn't get lost when that workstream starts. + +**Confirmed today:** `apps/cli/src/home/drive-home.tsx:510` launches every Home chat with +`agentRef: undefined` — i.e. always the built-in `buildDefaultChatAgent` (`apps/cli/src/chat/default-agent.ts`: +fixed system prompt, a 3-tool read-only grant, no `temperature`). There is no picker, no `--agent`-equivalent +affordance, and none of the registered chat slash commands +(`apps/cli/src/commands/repl-commands.ts` — help/exit/cancel/export/workflows/cost/doctor/mode/effort/ +thinking/compact/trim/clear/models/scrollback/edit/copy) shows or edits the bound agent's full config. The +only way to bind a custom `.agent.yaml` today is `relavium chat --agent ` from a shell — a path Home +never takes. + +**The "pick a different agent" half is already scheduled — do not re-file it.** 2.6.G's task list says, almost +verbatim: *"`/agents` browser (Home + chat): tabs Defined | Sessions. Defined: the agent catalog with +'start a chat with this agent' (closing the Home's built-in-agent-only gap)."* That is this exact gap, already +named and owned. + +**The narrower residual 2.6.G's current description does not clearly cover:** a **read-only** view of the +*full* config of the agent already bound to the *current* session — system prompt, tool grant, `temperature`, +`reasoning_effort`, provider — not just picking a different agent for a *new* chat. 2.6.G's *Sessions* tab is +described only as *"recent + in-progress sessions... a detail view (transcript summary, cost, model +attribution)"* — model attribution (which model/provider) is there, but system prompt / tools / temperature / +reasoning_effort are not mentioned. Today there is nowhere at all — Home or the terminal `chat` command — to +inspect that for a session already in flight. + +**This should stay read-only, not become an edit surface.** `resolveChatAgent`'s doc comment is explicit that +one agent binds a session for its whole lifetime by design — *"ADR-0024 — one agent per session, no +mid-session switching."* So the fix here is a **view**, not a live-mutate command: "show me what I'm actually +talking to" (worth having even by itself, e.g. after a `/clear` reseat or a `--agent` invocation where the +resolved config isn't obvious), not "let me tweak the running agent." Changing agents mid-session is already +correctly out of scope by design, and 2.6.G's "start a chat with this agent" (a *new* session) is the +sanctioned way to switch. + +**Home:** 2.6.G — when that workstream is scoped, confirm the Sessions-tab detail view (or a lightweight +`/agent` / `/whoami`-style command available in both Home and the terminal `chat` command) explicitly +includes system prompt, tool grant, `temperature`, and `reasoning_effort` for the *current* session, not only +model/provider/cost. If it's deliberately left out, that should be a stated decision, not a silent gap. + ## AgentSession (1.V) follow-ups > **2026-06-16 — 1.V `AgentSession` (ADR-0024) + 1.AC budget governor (ADR-0028) merged in PR #26** (after two @@ -960,6 +1002,40 @@ cross-turn transcript" item just below; whichever lands first should absorb the expand the thinking to inform the decision; every other key (mode cycle, edits, the most-permissive approve/reject chord) stays swallowed. *(apps/cli/src/render/tui/chat-input.ts)* +## The CLI e2e suite opens and MIGRATES the developer's real `~/.relavium/history.db` (2.6.C spin-off, 2026-07-13) + +> **DONE 2026-07-13 (PR #76 review fold).** `regression.e2e.test.ts` now redirects `HOME`/`USERPROFILE` to a +> fresh `mkdtempSync` dir in a suite-wide `beforeEach` (restored + removed in `afterEach`), so the full-shell +> `run(argv)` case migrates a throwaway db, never the developer's real history. A guard in the same hook refuses +> to proceed if the redirect fails to take (`resolveHomeDir({}) === realHome`), so a future test cannot silently +> re-acquire the default path. The evidence and rationale below are kept for the record. + +> Found while diagnosing a red CI run during 2.6.C (PR #75). Verified, not inferred — see the evidence below. + +`apps/cli/src/harness/regression.e2e.test.ts` drives the real CLI shell (`run(argv('run', …, '--json'), io)`) +without pointing it at a database, so the run resolves the **default** path and opens +`~/.relavium/history.db` — the developer's actual chat history. It does not merely read it: it **runs +migrations against it**. + +**Evidence.** Executing that one test file with `HOME` pointed at an empty directory creates +`$HOME/.relavium/history.db` with all 11 migrations applied. Under a real `HOME` those migrations land on +real data. (The sibling test at `:315` does this correctly — `mkdtempSync` + an explicit `dbPath` — so the +isolation exists; this path just does not use it.) + +**Why this is worth fixing rather than tolerating.** It is not a hypothetical: during 2.6.C the coupling +actively **hid a bug from CI and converted it into damage to real data instead**. A migration was re-cut +while in development, which changes its journal timestamp; drizzle replays such a migration, so +`CREATE TABLE session_costs` ran a second time against the table it had itself created. On CI this is +invisible — a fresh runner has no `~/.relavium/history.db`, so nothing had been applied and nothing could +conflict. The failure surfaced only on the maintainer's machine, against a 3.3 MB database of real +sessions. A test suite that writes to real user data both damages it and blinds CI to the damage. + +**Fix:** give the failing path the same isolation the sibling already has — a temp dir + an explicit db +path — and, as a floor, make the e2e harness refuse to run against the default history path at all, so a +future test cannot silently re-acquire it. + +**Home:** a `chore` pass, or whichever workstream next touches the CLI harness. + ## Sonar code-quality backlog > **2026-06-14 (PR #18 review).** Verified Sonar findings in **already-merged** code (1.L/1.L2/1.T/0.x), @@ -1101,3 +1177,52 @@ cross-turn transcript" item just below; whichever lands first should absorb the `home-input` `DISABLE_BRACKETED_PASTE` export + the drive-home test assertion) so both surfaces rely on ink's unmount cleanup uniformly. Low; deferred to ride Step 4's `writeControl` rework rather than churn it twice. *(low · apps/cli/src/home/drive-home.tsx + render/tui/home-input.ts; Step 2.6.F-4)* + +- **The Anthropic adapter's non-streaming `generate()` cannot carry a large `max_tokens`.** The SDK refuses a + non-streaming request whose cap implies a >10-minute generation (empirically: `claude-opus-4-5` is accepted at + ~21 000 and refused at ~24 000), and **every** Anthropic row in the catalog has a `maxOutputTokens` of 32 000 or + more. So the ADR-0071 §7 clamp — which holds the cap AT the ceiling — cannot bring a large authored cap back + under that threshold: `generate()` still fails, just with the SDK's message instead of a provider 400. Reachable + only through the public seam (`LlmProvider.generate`, the conformance harness, `validateProviderKey`), because + the engine's own agent turn takes `stream()`; the seam's other callers all pass a tiny cap. Resolve by either + capping the non-streaming Anthropic request under the SDK threshold, or failing with a typed, actionable + `bad_request` naming `stream()` rather than surfacing an SDK string. + *(medium · packages/llm/src/adapters/anthropic.ts; found by the 2.6.Q Step-5 Opus review)* + +- **Cache WRITES are billed at the flat rate, never a context tier's.** `ratesFor` (`packages/llm/src/cost-tracker.ts`) + moves input, output and cache-read onto the tier a prompt lands in (ADR-0071 §11), but cache-write stays on the flat + `cacheWritePerMtokMicrocents` — because models.dev's tier schema publishes `input`, `output` and `cache_read` and + **no `cache_write`**, so a per-tier write rate is not a number we have. Scaling one from the input tier's multiple + would be a guess on a money path, which is the thing this ADR exists to stop. The exposure is a cache-write-heavy + prompt above the 272k threshold on the four `gpt-5.6` variants (the only shipped models with both a cache-write rate + and a tier). Resolve by extending `CatalogPriceTier` + the upstream schema **if models.dev starts publishing it**, or + by asking upstream to. + *(medium · packages/llm/src/cost-tracker.ts + catalog/models-dev-schema.ts; found by the 2.6.Q Step-6 Sonnet review)* + +- **Auto-open a PR for additive catalog drift.** ADR-0071 §9 wants new models to "merge automatically" while a + moved shipped-model price stays a red human-reviewed check. The red check ships (`.github/workflows/models-catalog.yml` + `weekly-catalog-check` runs `pnpm sync:models`, red ONLY on a moved/vanished shipped-model price; additive drift is green). The *automatic* + half — a bot PR that runs `pnpm sync:models` and commits the additive diff — is deferred because it needs a + third-party `create-pull-request` action pinned to a verified commit SHA (the repo pins every action by SHA; + inventing one unseen is the supply-chain risk rule 3 forbids). Add it as a second job once the SHA is verified. + Until then a maintainer runs `pnpm sync:models` locally when the weekly check goes red — which also forces the + `--accept-price-changes` human decision on a moved price, exactly as §9 intends. + *(low · .github/workflows/models-catalog.yml; ADR-0071 §9)* + + +- **Per-model capability gating: `tool_call`/`attachment` + a withheld-parameter notice.** The 2.6.Q capability + gating (ADR-0071 §12) carries all four per-model `requestCapabilities` in the catalog and WITHHOLDS the two safe + silent param-drops — `temperature` and `structured_output` — at the adapters. `tool_call` and `attachment` change + request MEANING (dropping tools leaves the agent unable to act; dropping an image input changes what the model + sees), so silently withholding them is wrong: they need a LOUDER signal — a config/authoring-time validation, or a + gate-level notice like the effort gate (ADR-0066 §6). The catalog data is already parsed and carried, so this is + the wiring only. Likewise, a user-facing notice for the two silent withholds (so a dropped `temperature` is *said + out loud*, §6's standard) is a follow-up — the correctness fix (no 400) shipped first. + *(low · packages/llm/src/adapters/* + a gate-level check; ADR-0071 §12; found by the 2.6.Q capability review)* + +- **Surface a user↔catalog price DIVERGENCE on the ongoing surfaces, not only at set-time.** ADR-0071 §5 promises a + user price that diverges from the catalog is surfaced in the `/models` picker and `/cost`, not just echoed by + `models pricing` when it is set. Today only the set-time echo is wired: the picker and `/cost` show the effective + (user) price without flagging that it OVERRIDES a different catalog price. Add a divergence marker to both so a + stale/wrong override is visible on the surfaces a user actually reads, per §5's intent. + *(low · apps/cli/src/render/tui/model-picker-view.tsx + the /cost breakdown; ADR-0071 §5; found by the 2.6.Q review)* diff --git a/docs/roadmap/phases/phase-2.6-conversational-authoring.md b/docs/roadmap/phases/phase-2.6-conversational-authoring.md index 794f9803..254e3214 100644 --- a/docs/roadmap/phases/phase-2.6-conversational-authoring.md +++ b/docs/roadmap/phases/phase-2.6-conversational-authoring.md @@ -213,13 +213,14 @@ approval; `/create` works from the Home; a malformed agent YAML surfaces its fie diagnostic on every surface; the untrusted-import consent gate holds; a security review of the write surface + the secret-taint gate + the import gate passes. **Required ADR:** shared with 2.6.A (ADR-0058). -### 2.6.C — Mid-session model reseat — residual + transcript-carry fix — ✅ **Done (PR pending, 2026-07-12)** +### 2.6.C — Mid-session model reseat — residual + transcript-carry fix — ✅ **Done (PR #75, merged 2026-07-13)** > **Shipped early in 2.5.G** (ADR-0059, PR #66, 2026-07-07): the `/models` mid-chat reseat, per-message > `modelId` attribution, and the context-loss notice. Retained for the residual below and as the > cross-reference home. > -> **Status:** ✅ **Done (2026-07-12)** — both tasks shipped, plus the residual that turned out to need its own ADR. +> **Status:** ✅ **Done — merged to `main` 2026-07-13 (PR #75)**. Both tasks shipped, plus the residual that turned +> out to need its own ADR. > The reseat now carries the rendered conversation (F1), with the switch marker landing beneath it; and `/cost` shows > a per-model breakdown backed by a new durable `session_costs` table > ([ADR-0070](../../decisions/0070-durable-per-model-session-cost-attribution.md) — the per-message `model_id` this @@ -659,8 +660,8 @@ and the first localized agentic CLI (a genuine differentiator — competitor i18 (`[v]`/`[x]`/`[||]`) without Unicode. - **`/settings`** (Home + chat): a sectioned screen (appearance / language / chat defaults / models in use / update channel) over the **extended** [ADR-0063](../../decisions/0063-cli-config-write-contract.md) - typed-setter (new keys: `theme`, `language`, `models_in_use`; still global-preferences-only, atomic, - secret-incapable by construction — project files stay hand-authored). + typed-setter (new keys: `theme`, `language`, `models_in_use`, `show_pricing`; still + global-preferences-only, atomic, secret-incapable by construction — project files stay hand-authored). - **Curated "models in use" list** *(manual-test finding 2026-07-11):* a `/settings` section where the user picks which catalog models are "in use", so **`/models` shows only the curated set** instead of the whole merged catalog — today `/models` lists **every** reachable model from **every** provider and only a live @@ -670,6 +671,25 @@ and the first localized agentic CLI (a genuine differentiator — competitor i18 shows everything** (back-compatible, so nothing hard-blocks). The **default for a newly-connected provider** (seed all / none / a starter subset) is owned by **2.6.I** (see there). No new ADR beyond the ADR-0063 key addition. +- **Price-visibility toggle** *(picker display preference, same shape as "models in use"):* a `/settings` + switch for whether the `/models` picker renders each model's `$/MTok` badge + (`formatModelPrice`, `apps/cli/src/render/tui/model-picker.ts:357`). **Default OFF** — a user picking a + default model has not spent anything yet, so a wall of per-token prices up front reads as sticker shock + rather than useful information; showing it is a deliberate ask, not the first thing a new user sees. + Persisted as `[preferences].show_pricing` (boolean, default `false`) via the same extended ADR-0063 + setter as `models_in_use`/`theme`. Scope precisely, so this stays cosmetic and never safety-relevant: + - Gates the **picker's** price column only — never `/cost` (a session's *already-incurred* spend) or + `models pricing` (an explicit, user-requested price lookup): those are post-commitment or + explicitly-asked-for moments, not the "before I've tried this" moment the toggle protects. + - A model with **no known price at all** (`priceKnown: false` — the ADR-0028 cost cap will not apply to + it) keeps that warning visible regardless of the toggle: it is a safety disclosure, not a price + display, and hiding it would let a user select an unprotected model without knowing. + - Interactive-only, mirroring the i18n stance below: `--json`/machine output is unaffected either way + ([ADR-0049](../../decisions/0049-cli-machine-output-contract.md)) — a script always gets full data. + - The onboarding wizard's first model pick (2.6.J) reuses this same picker, so default-OFF applies there + too with no extra wiring — the exact moment the sticker-shock concern is about. + + No new ADR beyond the ADR-0063 key addition. - **i18n foundation**: an in-house string catalog (data ≠ code — zero conditional logic in translation data; no runtime dependency expected, else ADR); `[preferences].language`; locales **`en`, `es`, `tr`, `fr`, `de`** in-phase; a CI **key-parity test** (fails on missing/extra keys across all locales) @@ -685,7 +705,9 @@ and the first localized agentic CLI (a genuine differentiator — competitor i18 interactive surfaces. **Acceptance:** `/settings` edits persist atomically and round-trip; the curated `models_in_use` set -filters `/models` to the chosen models (unset ⇒ all, back-compatible); themes switch live incl. the +filters `/models` to the chosen models (unset ⇒ all, back-compatible); `show_pricing` defaults off and +hides only the picker's `$/MTok` badge (never `/cost`, never `models pricing`, never an unpriced-model +safety warning, never `--json` output); themes switch live incl. the accessibility pair; the color-free path stays legible; interactive surfaces run fully in all five locales with CI-enforced key parity; diagnostics and `--json` remain English-stable per [ADR-0049](../../decisions/0049-cli-machine-output-contract.md); the machine-output contract is diff --git a/docs/runbooks/add-a-provider.md b/docs/runbooks/add-a-provider.md index 8482a234..a8f2f780 100644 --- a/docs/runbooks/add-a-provider.md +++ b/docs/runbooks/add-a-provider.md @@ -96,22 +96,37 @@ reported `no key` and never probed. For a machine-readable result use `relavium ## 4–5. Discover and list the models ```bash -relavium models refresh # force a live re-fetch of each connected provider's model list -relavium models # list the cached catalog (auto-refreshes once on an empty cache) +relavium models refresh # refresh BOTH axes: provider lists + the model catalog +relavium models refresh --providers # availability only (which ids your key can reach) +relavium models refresh --catalog # metadata only (price, ceilings, reasoning tiers) +relavium models # list the cached catalog (auto-refreshes once on an empty cache) ``` -`models refresh` is **per-provider isolated** — one provider's failure never fails the whole -command (it is reported `failed` / `skipped`, the others still refresh). The catalog is a -local cache of *which model ids each key can reach*; the shipped `MODEL_PRICING` registry is -the pricing authority for a known model. (The interactive Home's `/models` picker +**There are two axes, and they come from different places** ([ADR-0071](../decisions/0071-models-dev-as-the-model-metadata-source.md)): + +| Axis | Source | Answers | +|---|---|---| +| **Availability** | the provider's own API, with **your key** | *Can I call this model?* | +| **Metadata** | the **shipped catalog snapshot** (generated, reviewed, offline) | *What does it cost, what is its ceiling, which reasoning tiers does it accept?* | + +A provider's `/models` endpoint returns availability, **not economics** — no price, no +reasoning tiers — which is why the second axis exists. The metadata snapshot ships **inside the +binary**: it is correct offline, on first run, with no network at all. The optional background +refresh is **OFF by default** (`[catalog] auto_refresh`); `relavium models refresh` above is a +**user-initiated** fetch and always allowed. + +`models refresh` is **per-source isolated** — one provider's failure never fails the whole +command (it is reported `failed` / `skipped`, the others still refresh), and a failed catalog +fetch is a **no-op**: it can never leave you *less* priced than the snapshot you installed with. +(The interactive Home's `/models` picker additionally **dims** a model not available on your key and **flags** a deprecated one; the plain `relavium models` list is ` ctx= []` — [ADR-0064](../decisions/0064-live-model-catalog.md) §7/§10.) -## 6. Price a model the registry does not know +## 6. Price a model the catalog does not know — or override one it does -A **custom-endpoint model**, or a brand-new vendor model not yet in the shipped registry, -has **no price** — so the cost cap (`budget.max_cost_microcents` for a workflow, +A **custom-endpoint model**, or a brand-new vendor model not yet in the shipped catalog, has +**no price** — so the cost cap (`budget.max_cost_microcents` for a workflow, `[chat].max_cost_microcents` for chat) would **degrade to "allow"** for it. Hand-enter its price so the cap is enforced ([ADR-0065](../decisions/0065-provider-economics-and-extensibility.md) §1–2): @@ -122,15 +137,23 @@ relavium models pricing my-gateway-llama --provider openai --input 3 --output 9 ``` Prices are **USD per million tokens** (stored as integer micro-cents). The price is written -as a user row and a live `models refresh` **never** clobbers it; once set, the model is -enforced by the cost cap on `run`, a `run` resumed via `relavium gate`, `chat` / -`chat-resume`, the interactive Home, and one-shot `agent run`. Guards (each exit 2, nothing -written): a **built-in-priced** model is refused (the shipped price always wins); an -**unregistered provider** is refused (do step 1 first); the **same model id already priced -under a different provider** is refused (the cost cap keys by model id, so it could not tell -them apart); and a **negative / non-finite / implausibly-large** price is refused. Look up the -real price at the provider's pricing page — the one `--pricing-url` recorded in step 1 (the -`provider add` confirmation echoes it). +as a user row and **no refresh — provider or catalog — ever clobbers it**; once set, the model +is enforced by the cost cap on `run`, a `run` resumed via `relavium gate`, `chat` / +`chat-resume`, the interactive Home, and one-shot `agent run`. + +**Your price wins over the catalog's** ([ADR-0071](../decisions/0071-models-dev-as-the-model-metadata-source.md) §5). +This is deliberate, and it is why: on a custom `base_url` (OpenRouter, Azure, LiteLLM, an +enterprise gateway) the **public list price is simply wrong** — your negotiated or marked-up +rate is the correct one. A catalog that could overrule you would misprice exactly the users who +took the trouble to be accurate. What you *cannot* do is diverge **silently**: pricing a model +the catalog also knows is allowed, and the disagreement is then surfaced in the model picker, +in `/cost`, and at `models pricing` time. Nothing is hidden; you are simply trusted. + +Guards (each exit 2, nothing written): an **unregistered provider** is refused (do step 1 +first); the **same model id already priced under a different provider** is refused (the cost cap +keys by model id, so it could not tell them apart); and a **negative / non-finite / +implausibly-large** price is refused. Look up the real price at the provider's pricing page — +the one `--pricing-url` recorded in step 1 (the `provider add` confirmation echoes it). ## Rotate or remove a key diff --git a/docs/standards/security-review.md b/docs/standards/security-review.md index 0a49aad4..d7902596 100644 --- a/docs/standards/security-review.md +++ b/docs/standards/security-review.md @@ -133,7 +133,38 @@ A chat-only relaxation of any rule here is a security violation, not a feature. bytes as **UNTRUSTED, nonce-fenced, bounded** context. **Editing the chat `allowed_commands` set, the input-layer boundary reuse, or the read-side confidentiality floor is a mandatory-review trigger** (below). -## Network and outbound URLs (SSRF — four egress paths, all on one primitive) +## Network and outbound URLs (SSRF — four attacker-influenceable egress paths, all on one primitive; plus one fixed-host path) + +> **Amended 2026-07-13 ([ADR-0071](../decisions/0071-models-dev-as-the-model-metadata-source.md)) — a FIFTH +> outbound path exists: the model-catalog refresh.** It is listed apart from the four below **on purpose**, and +> the distinction is the whole point of this section: the four paths are dangerous because **something other +> than Relavium chooses the URL** — a user config, an agent's YAML, a *model's* tool call. The SSRF primitive +> exists to defend that. The catalog refresh's destination is a **compile-time constant** (`models.dev`): no +> user, no agent, and no model can influence it, so there is no SSRF vector to range-block. It is the same +> category as the provider fetch of [ADR-0064](../decisions/0064-live-model-catalog.md) §9, not the same as +> `http_request`. +> +> It is nonetheless **egress**, and this inventory is a closed list, so it is recorded here rather than left to +> be discovered. Its posture: +> +> - **HTTPS only**, to a fixed hostname; certificate validation never disabled. A **cross-host redirect is an +> error, not a hop** — a 30x off `models.dev` fails the refresh rather than following. +> - **No user data, no key, no telemetry** — an unauthenticated `GET` of a public file. Nothing about the user's +> session, keys, prompts, or machine leaves. +> - **Default OFF** (`[catalog] auto_refresh = false`), following [ADR-0068](../decisions/0068-full-screen-tui-renderer-ink7-harness.md)'s +> opt-in-then-flip convention for a new surface. A **user-initiated** `relavium models refresh` may always +> fetch — an explicit command is consent. +> - **Host-side only** (`apps/cli/src/engine/`). `packages/core` and the pure part of `packages/llm` keep **zero +> platform imports** (CLAUDE.md rule 5); the merge stays a pure function taking data as an argument. +> - The response is **Zod-validated at the boundary** before it becomes a Relavium type — untrusted input, in +> the same class as a provider's model list. +> - **Additive only:** a refresh can never leave a model *less* priced than the shipped snapshot did, and a +> failed refresh is a **no-op**. This matters because pricing feeds a **safety control** (the ADR-0028 cost +> cap): the guarantee is that a bad or hostile upstream cannot *lower* the floor the binary shipped with. +> +> A change that gives the catalog fetch a **non-constant** destination (a configurable mirror, a proxy setting) +> moves it into the four below and **requires the SSRF primitive** — that is a mandatory-review change, not a +> config tweak. There are **four** outbound-URL paths (the fourth — the multimodal media `url` carrier — is now wired host-side via [ADR-0043](../decisions/0043-media-egress-failover-rematerialization-ssrf.md)'s `fetchMediaBytes`; diff --git a/eslint.config.mjs b/eslint.config.mjs index 3ae9c7b8..13fd9c7c 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -224,7 +224,16 @@ export default tseslint.config( // flat config ignores `/* eslint-env */` comments, so the grant must live here. files: ['tools/**/*.{js,mjs,cjs}'], languageOptions: { - globals: { console: 'readonly', process: 'readonly' }, + // `URL` / `fetch` / `AbortSignal` are for `sync-models-dev` (ADR-0071): it resolves the snapshot path + // relative to its own module, fetches the upstream catalog, and bounds that fetch with a timeout — a hung + // sync in CI is a silent one. All three are Node-22 globals (ADR-0067's floor), so no polyfill is implied. + globals: { + console: 'readonly', + process: 'readonly', + URL: 'readonly', + fetch: 'readonly', + AbortSignal: 'readonly', + }, }, }, { diff --git a/package.json b/package.json index 1c9077be..66234394 100644 --- a/package.json +++ b/package.json @@ -17,12 +17,15 @@ "typecheck:tools": "tsc -p tsconfig.tools.json", "test": "turbo run test", "coverage": "vitest run --coverage", - "ci": "turbo run lint typecheck test && pnpm typecheck:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure", + "ci": "turbo run lint typecheck test && pnpm typecheck:tools && pnpm lint:tools && turbo run build format:check && pnpm lint:fence-check && pnpm lint:engine-deps && pnpm lint:bundle-closure", "lint:fence-check": "node tools/lint-fixtures/assert-fence.mjs", "lint:engine-deps": "node tools/engine-deps/check.mjs", "lint:bundle-closure": "node tools/bundle-closure/check.mjs", "format": "prettier --write .", - "format:check": "prettier --check ." + "format:check": "prettier --check .", + "sync:models": "pnpm turbo run build --filter=@relavium/llm && node tools/sync-models-dev/sync.mjs", + "sync:models:check": "pnpm turbo run build --filter=@relavium/llm && node tools/sync-models-dev/sync.mjs --check", + "lint:tools": "eslint tools --ignore-pattern 'tools/lint-fixtures/**'" }, "devDependencies": { "@eslint/js": "catalog:", diff --git a/packages/core/src/engine/agent-runner.e2e.test.ts b/packages/core/src/engine/agent-runner.e2e.test.ts index 2acd7a35..8e034515 100644 --- a/packages/core/src/engine/agent-runner.e2e.test.ts +++ b/packages/core/src/engine/agent-runner.e2e.test.ts @@ -395,6 +395,30 @@ function agentExecutor( }); } +/** A budget workflow whose model is UNPRICED — a custom id neither the catalog nor a user prices. */ +function unpricedBudgetWorkflow(strict: boolean): ReturnType { + return parseWorkflow( + `schema_version: '1.0' +workflow: + id: e2e-budget-unpriced + budget: + max_cost_microcents: 1000000 + on_exceed: warn${strict ? '\n strict_cost_cap: true' : ''} + agents: + - id: a + model: my-self-hosted-model + provider: openai + system_prompt: hi + nodes: + - id: n + type: agent + agent_ref: a + prompt_template: 'go' + edges: [] +`, + ); +} + function budgetWorkflow(onExceed: string): ReturnType { return parseWorkflow( `schema_version: '1.0' @@ -441,6 +465,54 @@ describe('AgentRunner resource governance end-to-end (ADR-0028, 1.AC)', () => { expect(warning?.type === 'budget:warning' && warning.thresholdPct).toBe(0); }); + it('an UNPRICED model reaches the onUnpriced sink THROUGH WorkflowEngine (ADR-0071 §K7)', async () => { + // THE bug two commits missed: WorkflowEngine took `onUnpriced` in its deps and then never stored or forwarded + // it, so the notice was DEAD on `relavium run` / `relavium gate` — the batch surfaces, where an unattended run + // is the highest-risk place for a cap to silently not apply. The sink is injected here exactly as the CLI + // injects it, and the model is unpriced, so a working forward is the only way this array is non-empty. + const unpriced: string[] = []; + const engine = new WorkflowEngine({ + host: createInMemoryHost(), + executor: agentExecutor(() => + provider( + [ + { type: 'text_delta', text: 'ok' }, + { type: 'stop', stopReason: 'stop', usage: { inputTokens: 1, outputTokens: 1 } }, + ], + 'openai', + ), + ), + onUnpriced: (model) => unpriced.push(model), + }); + const events = await drain( + engine.start({ workflow: unpricedBudgetWorkflow(false), inputs: { text: 'x' } }), + ); + expect(events.at(-1)?.type).toBe('run:completed'); // unpriced degrades to allow — the run still finishes + expect(unpriced).toEqual(['my-self-hosted-model']); // …but the notice fired, through the engine boundary + }); + + it('strict_cost_cap BLOCKS an unpriced model through WorkflowEngine, regardless of on_exceed', async () => { + const engine = new WorkflowEngine({ + host: createInMemoryHost(), + executor: agentExecutor(() => + provider( + [ + { type: 'text_delta', text: 'ok' }, + { type: 'stop', stopReason: 'stop', usage: { inputTokens: 1, outputTokens: 1 } }, + ], + 'openai', + ), + ), + }); + const events = await drain( + engine.start({ workflow: unpricedBudgetWorkflow(true), inputs: { text: 'x' } }), + ); + // on_exceed is `warn`, but strict overrides it: an unpriceable model is a HARD pre-egress fail. + expect(events.at(-1)?.type).toBe('run:failed'); + const terminal = events.at(-1); + expect(terminal?.type === 'run:failed' && terminal.error.code).toBe('budget_exceeded'); + }); + it('fails the run when on_exceed is fail', async () => { const engine = new WorkflowEngine({ host: createInMemoryHost(), diff --git a/packages/core/src/engine/agent-runner.test.ts b/packages/core/src/engine/agent-runner.test.ts index 501cee68..4062d44f 100644 --- a/packages/core/src/engine/agent-runner.test.ts +++ b/packages/core/src/engine/agent-runner.test.ts @@ -1,4 +1,4 @@ -import type { Agent, ContentPart, OutputModality } from '@relavium/shared'; +import type { Agent, ContentPart, OutputModality, ReasoningEffort } from '@relavium/shared'; import { LlmProviderError, UnsupportedCapabilityError, makeLlmError } from '@relavium/llm'; import type { CapabilityFlags, @@ -1001,12 +1001,20 @@ describe('createAgentNodeExecutor — reasoning-effort gate (ADR-0066, the workf } it('SENDS the authored tier when the model is reasoning-capable', async () => { - const req = await capturedReq({ resolveReasoning: () => true }, reasoningAgent); + const req = await capturedReq( + { + resolveEffortTiers: () => new Set(['off', 'low', 'medium', 'high', 'max']), + }, + reasoningAgent, + ); expect(req?.reasoningEffort).toBe('high'); }); it('WITHHOLDS the tier when the model is NOT reasoning-capable (a non-reasoning model would reject it)', async () => { - const req = await capturedReq({ resolveReasoning: () => false }, reasoningAgent); + const req = await capturedReq( + { resolveEffortTiers: () => new Set() }, + reasoningAgent, + ); expect(req?.reasoningEffort).toBeUndefined(); }); @@ -1015,8 +1023,59 @@ describe('createAgentNodeExecutor — reasoning-effort gate (ADR-0066, the workf expect(req?.reasoningEffort).toBeUndefined(); }); + it('WITHHOLDS a tier the model REJECTS — a NON-EMPTY accepted set that omits the authored one', async () => { + // THE case this gate exists for, and the one the suite was missing: the two tests above use "everything" and + // "nothing", and both pass against a gate that merely checks whether the set is empty. The real bug is a model + // that reasons, offers a ladder, and is simply not offering THIS rung — `gpt-5.4-pro` with `low`. A review + // pointed out that `effortToSend` could be changed to promote a rejected tier to `accepted[0]` and every + // integration test would still be green. + const req = await capturedReq( + { resolveEffortTiers: () => new Set(['medium', 'high', 'max']) }, + reasoningAgent, // authors `high`… which IS accepted — so re-author to the rejected one below. + ); + expect(req?.reasoningEffort).toBe('high'); + + const rejected = await capturedReq( + { resolveEffortTiers: () => new Set(['medium', 'high', 'max']) }, + { ...reasoningAgent, reasoning_effort: 'low' as const }, + ); + expect(rejected?.reasoningEffort).toBeUndefined(); // withheld — NEVER promoted to a neighbouring tier + }); + + it('TELLS the host when it withholds — a silently-dropped tier is billed at the provider default', async () => { + // The gate replaced a loud 400 with a quiet no-op. On an authored workflow the quiet no-op is the worse of the + // two: the run succeeds, the knob does nothing, and nothing in the output admits it. + const withheld: Array<{ kind: string; model: string }> = []; + await capturedReq( + { + resolveEffortTiers: () => new Set(['medium', 'high', 'max']), + onEffortWithheld: (result, model) => withheld.push({ kind: result.kind, model }), + }, + { ...reasoningAgent, reasoning_effort: 'low' as const }, + ); + expect(withheld).toHaveLength(1); + expect(withheld[0]?.kind).toBe('rejected'); + expect(withheld[0]?.model).toBe(reasoningAgent.model); + + // …and it stays quiet when the tier rides, so the channel is a signal and not noise on every turn. + const quiet: string[] = []; + await capturedReq( + { + resolveEffortTiers: () => new Set(['medium', 'high', 'max']), + onEffortWithheld: (result) => quiet.push(result.kind), + }, + reasoningAgent, + ); + expect(quiet).toEqual([]); + }); + it('sends nothing when the agent authored NO tier — even on a capable model', async () => { - const req = await capturedReq({ resolveReasoning: () => true }, AGENT); + const req = await capturedReq( + { + resolveEffortTiers: () => new Set(['off', 'low', 'medium', 'high', 'max']), + }, + AGENT, + ); expect(req?.reasoningEffort).toBeUndefined(); }); }); diff --git a/packages/core/src/engine/agent-runner.ts b/packages/core/src/engine/agent-runner.ts index 35ff9f8d..9ab6ddd4 100644 --- a/packages/core/src/engine/agent-runner.ts +++ b/packages/core/src/engine/agent-runner.ts @@ -65,7 +65,12 @@ import { type PreEgressHook, } from './agent-turn.js'; import { BudgetExceededError, BudgetPauseError } from './budget-governor.js'; -import { gateReasoningEffort } from './reasoning-effort.js'; +import { effortToSend, gateReasoningEffort } from './reasoning-effort.js'; +import type { + EffortGateResult, + ReasoningCapCheck, + ResolveEffortTiers, +} from './reasoning-effort.js'; import type { MediaJobSubmission, NodeExecContext, @@ -93,13 +98,32 @@ export interface AgentRunnerDeps { */ readonly resolveMediaSurface?: (model: string) => MediaSurface | undefined; /** - * Whether a model supports reasoning ([ADR-0066](../../../../docs/decisions/0066-normalized-reasoning-effort-control.md)) — - * the host-injected per-model catalog projection (`model_catalog.capabilities.reasoning`), mirroring - * {@link resolveMediaSurface}. The engine is platform-pure (no DB), so the host injects it. Gates the - * `reasoningEffort` send: a non-reasoning model would reject the field, so the tier is sent only when this - * returns `true`. Absent or `undefined` ⇒ treated as NOT reasoning (safe — the field is not sent). + * WHICH reasoning tiers a model accepts ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6) + * — the host-injected per-model catalog projection, mirroring {@link AgentRunnerDeps.resolveMediaSurface}. The + * engine is platform-pure (no catalog, no DB), so the host injects it. + * + * It replaced a boolean, and the upgrade is the point: *"does this model reason"* is not the question the wire + * asks. `gpt-5.4-pro` reasons and rejects `low`. Absent / `undefined` / an empty set ⇒ the field is withheld — + * safe, and never a guess. + */ + readonly resolveEffortTiers?: ResolveEffortTiers; + /** + * Does a budget-shaped model withhold the accepted tier because this node's `max_tokens` leaves no room for its + * minimum thinking budget (review M6)? Host-injected (reads the catalog's budget range); absent ⇒ unchecked, and + * a tight `max_tokens` drops thinking silently, as before. Surfaces as a `capped` verdict through onEffortWithheld. + */ + readonly withheldByCap?: ReasoningCapCheck; + /** + * Called when an authored `reasoning_effort` is WITHHELD — the tier the agent asked for is not one the bound + * model takes ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * Withholding is right; withholding **silently** is not, and that is the trade this gate would otherwise make. + * It replaced a loud provider 400 with a turn that runs, drops the field, and bills at the provider's default + * tier — so an author who wrote `reasoning_effort: max` on a model whose ladder stops at `high` would have no + * way to learn that the knob they set does nothing. The engine cannot print, so the host is handed the gate's + * own verdict (which carries what the model WOULD accept) and decides where it goes. */ - readonly resolveReasoning?: (model: string) => boolean | undefined; + readonly onEffortWithheld?: (result: EffortGateResult, model: string) => void; /** The shared tool registry (1.T) the agent dispatches through (ADR-0037). */ readonly registry: ToolRegistry; /** The registry's tool defs — the source of the LLM-visible schema + descriptions for granted tools. */ @@ -802,15 +826,27 @@ function resolveGenKnobs( ): { temperature?: number; maxTokens?: number; reasoningEffort?: ReasoningEffort } { const temperature = node.temperature ?? agent.temperature; const maxTokens = node.max_tokens ?? agent.max_tokens; - // ADR-0066: send the reasoning-effort tier ONLY when the agent authored one AND the primary model is - // reasoning-capable (the shared {@link gateReasoningEffort} rule — a non-reasoning model would reject the field). - // The per-fallback-entry re-gate lives in the chain (a non-reasoning fallback entry strips the tier), so a failover - // to a different-capability model never carries an unsupported field. - const reasoningEffort = gateReasoningEffort( + // ADR-0066/0071: send the tier ONLY when the model is on record as ACCEPTING it — not merely as reasoning. + // `gpt-5.4-pro` reasons and rejects `low`; a boolean capability said `true` and let that straight through. + // A rejected tier is WITHHELD, never promoted to a neighbour (that would change behaviour and raise spend + // silently). The per-fallback-entry re-gate lives in the chain, so a failover to a different-capability model + // never carries a field it does not take. + const gate = gateReasoningEffort( agent.reasoning_effort, agent.model, - deps.resolveReasoning, + deps.resolveEffortTiers, + deps.withheldByCap !== undefined && maxTokens !== undefined + ? { maxTokens, withheldByCap: deps.withheldByCap } + : undefined, ); + // …and TELL the host when it withheld. The gate turned a loud 400 into a quiet no-op, and a quiet no-op on a + // knob the author deliberately set is the worse of the two: the run succeeds, the field is gone, and the bill + // lands at the provider's default tier with nothing in the output to explain why. `capped` — a budget model whose + // tier this node's `max_tokens` withholds (review M6) — is the same silent no-op and rides the same channel. + if (gate.kind === 'rejected' || gate.kind === 'uncontrollable' || gate.kind === 'capped') { + deps.onEffortWithheld?.(gate, agent.model); + } + const reasoningEffort = effortToSend(gate); return { ...(temperature === undefined ? {} : { temperature }), ...(maxTokens === undefined ? {} : { maxTokens }), diff --git a/packages/core/src/engine/agent-session.test.ts b/packages/core/src/engine/agent-session.test.ts index 93b7643d..9086bb03 100644 --- a/packages/core/src/engine/agent-session.test.ts +++ b/packages/core/src/engine/agent-session.test.ts @@ -7,6 +7,7 @@ import type { ProviderId, StreamChunk, } from '@relavium/llm'; +import type { ReasoningEffort } from '@relavium/shared'; import { AgentSchema, RunEventSchema, @@ -796,7 +797,7 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' const onSession = session( harness([textTurn('ok')], { resolveProvider: () => on.provider, - resolveReasoning: () => true, + resolveEffortTiers: () => new Set(['off', 'low', 'medium', 'high', 'max']), }).deps, reader, ); @@ -808,13 +809,32 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' const offSession = session( harness([textTurn('ok')], { resolveProvider: () => off.provider, - resolveReasoning: () => false, + resolveEffortTiers: () => new Set(), }).deps, reader, ); offSession.start(); await offSession.sendMessage('go'); expect(off.effort()).toBeUndefined(); + + // …and the case BOTH of the above miss: a model that reasons, publishes a ladder, and simply does not carry + // THIS rung. "Everything" and "nothing" are both satisfied by a gate that only asks whether the set is empty; + // only a non-empty set that omits the requested tier tests the gate that is actually shipping. + const partial = capturing(); + const withheld: string[] = []; + const partialSession = session( + harness([textTurn('ok')], { + resolveProvider: () => partial.provider, + resolveEffortTiers: () => new Set(['medium', 'high', 'max']), + onEffortWithheld: (result) => withheld.push(result.kind), + }).deps, + reader, + ); + partialSession.setReasoningEffort('low'); // a REAL tier — just not one this model takes + partialSession.start(); + await partialSession.sendMessage('go'); + expect(partial.effort()).toBeUndefined(); // withheld, never promoted to the nearest acceptable tier + expect(withheld).toEqual(['rejected']); // …and the surface is TOLD, so the user is not billed in the dark }); it('setReasoningEffort override wins over the authored tier on the NEXT turn — no reseat (ADR-0066 §5)', async () => { @@ -838,8 +858,10 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' }, }; const s = session( - harness([textTurn('ok')], { resolveProvider: () => provider, resolveReasoning: () => true }) - .deps, + harness([textTurn('ok')], { + resolveProvider: () => provider, + resolveEffortTiers: () => new Set(['off', 'low', 'medium', 'high', 'max']), + }).deps, reader, ); s.start(); @@ -877,8 +899,10 @@ describe('AgentSession — reseat-less modes + mid-turn abort (ADR-0057 Step 2)' }, }; const s = session( - harness([textTurn('ok')], { resolveProvider: () => provider, resolveReasoning: () => false }) - .deps, + harness([textTurn('ok')], { + resolveProvider: () => provider, + resolveEffortTiers: () => new Set(), + }).deps, reader, ); s.start(); diff --git a/packages/core/src/engine/agent-session.ts b/packages/core/src/engine/agent-session.ts index 50014cd3..858b2b51 100644 --- a/packages/core/src/engine/agent-session.ts +++ b/packages/core/src/engine/agent-session.ts @@ -70,7 +70,12 @@ import { } from './agent-turn.js'; import { BudgetPauseError } from './budget-governor.js'; import type { AbortControllerLike } from './execution-host.js'; -import { gateReasoningEffort } from './reasoning-effort.js'; +import { effortToSend, gateReasoningEffort } from './reasoning-effort.js'; +import type { + EffortGateResult, + ReasoningCapCheck, + ResolveEffortTiers, +} from './reasoning-effort.js'; import type { NodeStreamEvent } from './node-executor.js'; import type { SessionResumeState } from './session-resume.js'; @@ -242,13 +247,29 @@ export interface SessionDeps { */ readonly resolvePrice?: PricingOverlay; /** - * Whether the bound model supports reasoning ([ADR-0066](../../../../docs/decisions/0066-normalized-reasoning-effort-control.md)) - * — the host-injected per-model `model_catalog.capabilities.reasoning` projection (mirrors the `AgentRunner`'s - * `resolveReasoning`). Gates the `reasoningEffort` send: the authored `agent.reasoning_effort` is passed to a turn - * only when this returns `true` (a non-reasoning model rejects the field). Absent/`undefined` ⇒ not reasoning - * ⇒ withheld. `@relavium/core` never imports `@relavium/db`, so the host injects the catalog lookup. + * WHICH reasoning tiers the bound model accepts ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6) + * — the host-injected per-model catalog projection, the same one {@link AgentRunnerDeps.resolveEffortTiers} + * takes. Gates the `reasoningEffort` send: the effective tier (session override → the agent's authored tier) is + * passed to a turn only when the model is on record as taking it. + * + * It replaced a boolean. `gpt-5.4-pro` reasons **and rejects `low`** — a boolean answered `true` for it and the + * tier went to the provider unexamined. Absent / `undefined` / an empty set ⇒ withheld, never guessed. + * `@relavium/core` never imports the catalog, so the host injects the lookup. + */ + readonly resolveEffortTiers?: ResolveEffortTiers; + /** + * Does a budget-shaped model withhold the accepted tier because THIS turn's `max_tokens` leaves no room for its + * minimum thinking budget (review M6)? Host-injected (it reads the catalog's budget range); absent ⇒ the cap is + * not checked and a small `max_tokens` withholds thinking silently, as before. Paired with `max_tokens` at the + * gate so the withhold becomes a `capped` verdict the host can voice. + */ + readonly withheldByCap?: ReasoningCapCheck; + /** + * Called when the effective tier is WITHHELD because the bound model does not take it — the session mirror of + * {@link AgentRunnerDeps.onEffortWithheld}. The surface decides where the sentence goes (the CLI puts it in the + * transcript's notice channel); the engine only reports that it happened, and what the model WOULD accept. */ - readonly resolveReasoning?: (model: string) => boolean | undefined; + readonly onEffortWithheld?: (result: EffortGateResult, model: string) => void; /** * Feed the running session cost to a budget governor so a host that wires {@link preEgress} to * `BudgetGovernor.checkPreEgress` also keeps the governor's cumulative total current (ADR-0028, 1.AC). @@ -483,8 +504,9 @@ export class AgentSession { * pricing, nor the memoized `#plan`) and applies on the **next** turn (each `sendMessage` reads the override at * turn start). Pass `undefined` to clear it (fall back to the agent's authored `reasoning_effort`). Callable in * **any** state incl. mid-turn (takes effect next turn); **inert once `cancelled`** (no further turn reads it). - * The tier is still gated per turn by the host's per-model capability ({@link AgentSessionDeps.resolveReasoning}), - * so setting it on a non-reasoning model is a harmless no-op at send time. + * The tier is still gated per turn against the tiers the bound model accepts ({@link SessionDeps.resolveEffortTiers}), + * so a tier the model does not take is withheld at send — and {@link SessionDeps.onEffortWithheld} says so, rather + * than leaving the user to wonder why the knob did nothing. */ setReasoningEffort(effort: ReasoningEffort | undefined): void { this.#reasoningEffort = effort; @@ -1084,14 +1106,28 @@ export class AgentSession { // Advertise-filter (ADR-0057): narrow the model-visible tool set per the host's mode (best-effort; the // confirm floor stays authoritative). No policy / no filter ⇒ advertise every granted tool. const llmTools = buildLlmTools(this.#deps.tools, grantedToolIds, turnPolicy?.advertise); - // ADR-0066: resolve the effective reasoning-effort tier (session override → agent's authored tier) and gate it - // on the bound model's per-model capability (a non-reasoning model would reject it). Read at turn start so a - // mid-session setReasoningEffort applies to the NEXT turn — the no-reseat per-turn semantics (§5). - const reasoningEffort = gateReasoningEffort( + // ADR-0066/0071: resolve the effective reasoning-effort tier (session override → agent's authored tier) and + // gate it on WHICH TIERS the bound model accepts. Read at turn start so a mid-session setReasoningEffort + // applies to the NEXT turn — the no-reseat per-turn semantics (§5). + const effortGate = gateReasoningEffort( this.#reasoningEffort ?? this.#agent.reasoning_effort, this.#agent.model, - this.#deps.resolveReasoning, + this.#deps.resolveEffortTiers, + this.#deps.withheldByCap !== undefined && this.#agent.max_tokens !== undefined + ? { maxTokens: this.#agent.max_tokens, withheldByCap: this.#deps.withheldByCap } + : undefined, ); + // Withholding is right; withholding silently is not. The host gets the verdict and says so — otherwise a tier + // the user set is dropped, the turn runs at the provider's default, and nothing anywhere admits it. `capped` + // (a budget model whose tier the request's cap withholds, review M6) is voiced through the same channel. + if ( + effortGate.kind === 'rejected' || + effortGate.kind === 'uncontrollable' || + effortGate.kind === 'capped' + ) { + this.#deps.onEffortWithheld?.(effortGate, this.#agent.model); + } + const reasoningEffort = effortToSend(effortGate); return runAgentTurn({ system: this.#systemPrompt(), messages: this.#messages, diff --git a/packages/core/src/engine/agent-turn.ts b/packages/core/src/engine/agent-turn.ts index c24f6295..6ce1c630 100644 --- a/packages/core/src/engine/agent-turn.ts +++ b/packages/core/src/engine/agent-turn.ts @@ -47,6 +47,7 @@ import { type LlmRequest, type MediaUnitsEstimate, type PricingOverlay, + type ProviderId, type ResponseFormat, type StreamChunk, type ToolDef as LlmToolDef, @@ -106,6 +107,10 @@ export const DEFAULT_AGENT_TURN_LIMITS: AgentTurnLimits = { export type PreEgressHook = (info: { readonly model: string; readonly maxTokens?: number; + /** The routing provider for this call — forwarded to the budget governor's endpoint estimate so it keys on the + * ACTUAL provider (custom base_url ⇒ `custom`, no clamp), not the model's catalog provider (review M2). Optional: + * a media-only gate (`maxTokens: 0`) omits it harmlessly, since the token estimate is 0 regardless of endpoint. */ + readonly provider?: ProviderId; readonly outputModalities?: readonly OutputModality[]; readonly mediaUnitsEstimate?: readonly MediaUnitsEstimate[]; }) => void | Promise; @@ -612,10 +617,15 @@ async function dispatchToolCalls( * a {@link BudgetPauseError} (`pause_for_approval`) and any other error propagate as-is (the run path maps * the pause to a `paused` node outcome). Extracted from the turn loop to keep its complexity in budget. */ -async function awaitPreEgress(params: AgentTurnParams, activeModel: string): Promise { +async function awaitPreEgress( + params: AgentTurnParams, + activeModel: string, + activeProvider: ProviderId | undefined, +): Promise { try { await params.preEgress?.({ model: activeModel, + ...(activeProvider === undefined ? {} : { provider: activeProvider }), ...(params.maxTokens === undefined ? {} : { maxTokens: params.maxTokens }), ...(params.outputModalities === undefined ? {} @@ -737,6 +747,10 @@ async function driveAgentTurn( // overlay (2.5.G S10) lets the tracker price a user-priced model the static registry lacks. const costTracker = new CostTracker(params.resolvePrice); let activeModel = primaryModel; + // The provider that pairs with `activeModel`, updated together in `onAttempt`. A failover moves BOTH, so the + // pre-egress endpoint estimate keys on the routing provider actually in play, not the model's catalog provider + // (review M2). Starts on the primary entry's provider. + let activeProvider: ProviderId | undefined = params.planEntries[0]?.provider.id; let nonSkippedAttempts = 0; const onAttempt = (record: AttemptRecord): void => { @@ -744,6 +758,7 @@ async function driveAgentTurn( // the next entry's streamed tokens would be mis-attributed to a provider that never ran. if (record.outcome === 'skipped') return; activeModel = record.model; + activeProvider = record.provider; nonSkippedAttempts += 1; usage.engaged = true; // a non-skipped attempt RAN — mark engaged even if it then errored at zero usage if (record.usage === undefined) return; @@ -774,13 +789,18 @@ async function driveAgentTurn( costTracker, onAttempt, // The pre-egress budget hook runs before EVERY provider attempt, not just the first turn, so a failover - // to a more expensive model is also gated (1.AC). The chain's PreAttemptHook only supplies `{ model, - // maxTokens }`, so wrap the hook to also carry the turn-static media estimate (1.AF/D17) — otherwise the - // failover-attempt budget check would silently drop the media addend (ADR-0044 §3). + // to a more expensive model is also gated (1.AC). The chain's PreAttemptHook supplies `{ model, provider, + // maxTokens }` — `provider` is THIS attempt's routing provider (review M2) — so wrap the hook to also carry + // the turn-static media estimate (1.AF/D17); otherwise the failover-attempt check would silently drop the + // media addend (ADR-0044 §3). `...info` forwards `provider` to the governor's endpoint estimate unchanged. ...(preEgress === undefined ? {} : { - preAttempt: (info: { readonly model: string; readonly maxTokens?: number }) => + preAttempt: (info: { + readonly model: string; + readonly provider: ProviderId; + readonly maxTokens?: number; + }) => preEgress({ ...info, ...(params.outputModalities === undefined @@ -803,7 +823,7 @@ async function driveAgentTurn( // agent's final artifact and `generate()` is one round-trip). The two budget gates below mirror the text // path: `awaitPreEgress` (primary-model, zero-egress-on-cancel) then the chain's per-attempt `preAttempt`. if (requestsMediaOutput(params)) { - await awaitPreEgress(params, activeModel); + await awaitPreEgress(params, activeModel, activeProvider); throwIfAborted(params.signal); const turn = await generateOneTurn(chain, messages, params); throwIfAborted(params.signal); // cancel-wins independent of adapter cooperation (mirrors the stream path) @@ -845,7 +865,7 @@ async function driveAgentTurn( // • `FallbackChain.preAttempt` then runs again per chain attempt against the ACTUAL (possibly // failed-over) model, so a failover to a pricier model is still enforced. `streamOneTurn` maps a // chain-path Budget*Error back into this taxonomy via `chunk.error.cause`. - await awaitPreEgress(params, activeModel); + await awaitPreEgress(params, activeModel, activeProvider); // The preEgress hook is awaited (its budget check may be async), so the signal can fire during // that await. Re-check before engaging the provider so a cancel there costs no egress — symmetric // with the post-stream re-check below. diff --git a/packages/core/src/engine/budget-governor.test.ts b/packages/core/src/engine/budget-governor.test.ts index 1e8833bf..38ddf896 100644 --- a/packages/core/src/engine/budget-governor.test.ts +++ b/packages/core/src/engine/budget-governor.test.ts @@ -1,5 +1,10 @@ import { describe, expect, it } from 'vitest'; -import type { PricingOverlay } from '@relavium/llm'; +import { + estimateMaxNextCost, + type EndpointKind, + type PricingOverlay, + type ProviderId, +} from '@relavium/llm'; import type { Budget } from '@relavium/shared'; import { BudgetExceededError, BudgetGovernor, BudgetPauseError } from './budget-governor.js'; @@ -13,24 +18,31 @@ describe('BudgetGovernor', () => { budget?: Budget; defaultMaxTokensEstimate?: number; resolvePrice?: PricingOverlay; + resolveEndpoint?: (provider: ProviderId) => EndpointKind; } = {}, ): { governor: BudgetGovernor; warnings: Omit, 'runId'>[]; + unpriced: string[]; } { const warnings: Omit, 'runId'>[] = []; + const unpriced: string[] = []; const governor = new BudgetGovernor({ budget: overrides.budget ?? budget, ...(overrides.defaultMaxTokensEstimate === undefined ? {} : { defaultMaxTokensEstimate: overrides.defaultMaxTokensEstimate }), ...(overrides.resolvePrice === undefined ? {} : { resolvePrice: overrides.resolvePrice }), + ...(overrides.resolveEndpoint === undefined + ? {} + : { resolveEndpoint: overrides.resolveEndpoint }), + onUnpriced: (model) => unpriced.push(model), emit: (event) => { warnings.push(event); return Promise.resolve(); }, }); - return { governor, warnings }; + return { governor, warnings, unpriced }; } it('allows a call whose estimate stays within the cap', async () => { @@ -119,10 +131,58 @@ describe('BudgetGovernor', () => { // UnknownModelError. The pre-egress governor must NOT hard-fail an otherwise-valid run on it; it // degrades to `allow` (mirrors the FallbackChain's unpriced⇒no-cost policy). Even with on_exceed: fail // and the run already over a notional cap, an unpriced model resolves rather than throwing. - const { governor, warnings } = makeGovernor({ budget: { ...budget, on_exceed: 'fail' } }); + const { governor, warnings, unpriced } = makeGovernor({ + budget: { ...budget, on_exceed: 'fail' }, + }); governor.updateCost(900_000); await expect(governor.checkPreEgress('my-self-hosted-model', 10_000)).resolves.toBeUndefined(); - expect(warnings).toHaveLength(0); + expect(warnings).toHaveLength(0); // it did not exceed — nothing WAS billed + // …but it is UNPRICED, so the cap could not apply, and that is said once (ADR-0071 §K7): a cap that silently + // does not apply is a false sense of safety. + expect(unpriced).toEqual(['my-self-hosted-model']); + }); + + describe('strict_cost_cap (ADR-0071 §K7)', () => { + it('BLOCKS an unpriced model when on — "if you cannot price it, do not run it"', () => { + // The opt-in for a user who set a cap SPECIFICALLY to bound an untrusted model. The silent degrade-to-allow + // is the wrong trade for them: an unpriced model is a hole in the cap, and they would rather refuse the turn. + const { governor } = makeGovernor({ + budget: { ...budget, on_exceed: 'fail', strict_cost_cap: true }, + }); + const result = governor.evaluatePreEgress('my-self-hosted-model', 10_000); + expect(result.kind).toBe('fail'); + if (result.kind === 'fail') { + expect(result.error.message).toContain('no price'); + expect(result.error.message).toContain('strict_cost_cap'); + } + }); + + it('does NOT block a PRICED model — strict only bites when we genuinely cannot price', () => { + const { governor } = makeGovernor({ + budget: { ...budget, on_exceed: 'fail', strict_cost_cap: true }, + }); + governor.updateCost(0); + expect(governor.evaluatePreEgress('claude-haiku-4-5', 1000).kind).toBe('allow'); + }); + + it('OFF (the default) degrades to allow with a notice, not a block', async () => { + const { governor, unpriced } = makeGovernor({ budget: { ...budget, on_exceed: 'fail' } }); + // `evaluatePreEgress` classifies; `checkPreEgress` is what APPLIES the result and fires the sink. Drive the + // applying path, so the "with a notice" in this test's name is actually asserted. + expect(governor.evaluatePreEgress('my-self-hosted-model', 10_000).kind).toBe('unpriced'); + await expect( + governor.checkPreEgress('my-self-hosted-model', 10_000), + ).resolves.toBeUndefined(); + expect(unpriced).toEqual(['my-self-hosted-model']); + }); + }); + + it('notifies UNPRICED only once per model — a loop must not repeat it every turn', async () => { + const { governor, unpriced } = makeGovernor(); + await governor.checkPreEgress('my-self-hosted-model', 1000); + await governor.checkPreEgress('my-self-hosted-model', 1000); + await governor.checkPreEgress('another-unpriced-one', 1000); + expect(unpriced).toEqual(['my-self-hosted-model', 'another-unpriced-one']); // deduped per model }); it('accepts a media-unit estimate and folds it as a disjoint addend (1.AF/D17)', () => { @@ -196,4 +256,48 @@ describe('BudgetGovernor', () => { expect(warnings).toHaveLength(0); }); }); + + describe('endpoint keys on the ROUTING provider, not the model catalog (review M2)', () => { + // A custom `openai` gateway (OpenRouter/LiteLLM) serving `deepseek-v4-flash`: the wire is UNCLAMPED (a gateway + // may serve anything under a familiar id), so the estimate must reflect the FULL request — keyed on the routing + // provider ('openai' = custom here), never the model's catalog provider ('deepseek' = official) which clamps to + // the ceiling and under-authorizes. Before M2, resolveEndpoint(model)→catalog provider→official→clamp→allow. + const HUGE = 10_000_000; + const resolveEndpoint = (provider: ProviderId): EndpointKind => + provider === 'openai' ? 'custom' : 'official'; + + it('routes the endpoint by the provider argument — same model, opposite clamp', () => { + const official = estimateMaxNextCost('deepseek-v4-flash', HUGE, undefined, 'official'); + const custom = estimateMaxNextCost('deepseek-v4-flash', HUGE, undefined, 'custom'); + expect(custom).toBeGreaterThan(official); // the catalog ceiling clamp is real for this model + + // A cap between the clamped and unclamped cost: official passes, custom must not. + const cap = official + Math.round((custom - official) / 2); + const { governor } = makeGovernor({ + budget: { max_cost_microcents: cap, on_exceed: 'fail' }, + resolveEndpoint, + }); + governor.updateCost(0); + + // On its own API (official) → clamped to the ceiling → under the cap → allow. + expect( + governor.evaluatePreEgress('deepseek-v4-flash', HUGE, undefined, 'deepseek').kind, + ).toBe('allow'); + // Through the custom 'openai' gateway → unclamped → over the cap → fail. Keying on the catalog provider + // ('deepseek') would have wrongly clamped THIS path and waved the overspend through (the M2 defect). + expect(governor.evaluatePreEgress('deepseek-v4-flash', HUGE, undefined, 'openai').kind).toBe( + 'fail', + ); + }); + + it('omitting the provider (a media-only gate) defaults to official — a harmless no-op at maxTokens 0', () => { + const { governor } = makeGovernor({ + budget: { ...budget, on_exceed: 'fail' }, + resolveEndpoint, + }); + governor.updateCost(0); + // maxTokens 0 → token estimate 0 regardless of endpoint, so the absent provider cannot mis-authorize. + expect(governor.evaluatePreEgress('deepseek-v4-flash', 0).kind).toBe('allow'); + }); + }); }); diff --git a/packages/core/src/engine/budget-governor.ts b/packages/core/src/engine/budget-governor.ts index 8b168356..70402653 100644 --- a/packages/core/src/engine/budget-governor.ts +++ b/packages/core/src/engine/budget-governor.ts @@ -2,8 +2,10 @@ import { estimateMaxNextCost, estimateMediaCost, UnknownModelError, + type EndpointKind, type MediaUnitsEstimate, type PricingOverlay, + type ProviderId, } from '@relavium/llm'; import type { Budget } from '@relavium/shared'; @@ -27,10 +29,14 @@ export class BudgetExceededError extends Error { readonly spentMicrocents: number, readonly limitMicrocents: number, readonly projectedMicrocents: number, + // A caller-supplied message for the case the cap fails NOT because spend exceeded it, but because it could not + // be enforced at all — an unpriced model under `strict_cost_cap` (ADR-0071 §K7). Absent ⇒ the projection line. + message?: string, ) { super( - `pre-egress budget check failed: projected ${projectedMicrocents} micro-cents exceeds ` + - `the cap of ${limitMicrocents} micro-cents (spent ${spentMicrocents})`, + message ?? + `pre-egress budget check failed: projected ${projectedMicrocents} micro-cents exceeds ` + + `the cap of ${limitMicrocents} micro-cents (spent ${spentMicrocents})`, ); } } @@ -80,6 +86,12 @@ export type BudgetCheckResult = readonly limitMicrocents: number; readonly thresholdPct: number; } + /** + * The turn is ALLOWED, but the model has no price, so the cap could not be applied to it (ADR-0071 §K7). Carried + * — not swallowed — so the surface can say so once: a cost cap that silently does not apply is a false sense of + * safety, and the user who set one deserves to know which model slipped past it. + */ + | { readonly kind: 'unpriced'; readonly model: string } | { readonly kind: 'fail'; readonly error: BudgetExceededError } | { readonly kind: 'pause'; readonly error: BudgetPauseError }; @@ -95,8 +107,11 @@ export class BudgetGovernor { event: Omit, 'runId'>, ) => Promise; readonly #overlay: PricingOverlay | undefined; + readonly #resolveEndpoint: ((provider: ProviderId) => EndpointKind) | undefined; #cumulativeCostMicrocents = 0; #warningEmitted = false; + readonly #onUnpriced: ((model: string, capMicrocents: number) => void) | undefined; + readonly #unpricedNotified = new Set(); // once per model — a standing condition, not a per-turn event constructor(params: { readonly budget: Budget; @@ -107,11 +122,36 @@ export class BudgetGovernor { /** The user-pricing overlay (2.5.G S10) — makes the PRE-EGRESS estimate price a user-priced model that the * static registry lacks, so `max_cost_microcents` enforces it (the cap-gap fix). Absent ⇒ static-only. */ readonly resolvePrice?: PricingOverlay; + /** + * Is this model's provider on its OWN API, or behind a custom `base_url` + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §7)? + * + * The adapter clamps an authored `max_tokens` to the model's published ceiling on an official endpoint, and + * deliberately does NOT on a custom one (a gateway may serve anything under a familiar id). The estimate has to + * make the SAME call, or it stops describing the request: assume `official` on a gateway and the estimate lands + * BELOW what the wire can spend, so the governor under-authorizes and waves through a call it should have + * stopped. The engine cannot know a base URL — the host injects the answer, exactly as it injects the price. + * + * Absent ⇒ every model is treated as official, which is the adapter's own default for an un-overridden endpoint. + * + * Keyed on the ROUTING PROVIDER, not the model: a custom gateway serving another provider's model id is + * `custom` at the wire yet `official` by the model's catalog provider, and estimating from the catalog + * provider under-authorizes the turn (review M2). The provider rides the pre-egress info per attempt. + */ + readonly resolveEndpoint?: (provider: ProviderId) => EndpointKind; + /** + * Called when a turn runs on a model we cannot PRICE, so the cap could not apply to it (ADR-0071 §K7). Fired + * once per model. The engine cannot print; the host routes the notice (chat → the transcript, `run` → stderr). + * Absent ⇒ silent, and `strict_cost_cap` (which BLOCKS instead) is the loud alternative for anyone who wants it. + */ + readonly onUnpriced?: (model: string, capMicrocents: number) => void; }) { this.#budget = params.budget; this.#defaultMaxTokensEstimate = params.defaultMaxTokensEstimate ?? DEFAULT_MAX_TOKENS_ESTIMATE; this.#emit = params.emit; this.#overlay = params.resolvePrice; + this.#onUnpriced = params.onUnpriced; + this.#resolveEndpoint = params.resolveEndpoint; } /** Update the governor with the engine's authoritative running cumulative cost. */ @@ -128,6 +168,7 @@ export class BudgetGovernor { model: string, maxTokens: number | undefined, mediaUnitsEstimate?: readonly MediaUnitsEstimate[], + provider?: ProviderId, ): BudgetCheckResult { // A cap of 0 means UNBOUNDED (`[chat].max_cost_microcents`: "0 = unbounded"): never block, and never // reach the `thresholdPct` division below (which would be `/0` → NaN). A workflow `BudgetSchema` forbids @@ -142,7 +183,14 @@ export class BudgetGovernor { // modalities the model rates (a missing rate degrades to 0); both share the UnknownModelError // degrade-to-allow below, so an unpriced model never hard-fails the run. estimate = - estimateMaxNextCost(model, maxTokens ?? this.#defaultMaxTokensEstimate, this.#overlay) + + estimateMaxNextCost( + model, + maxTokens ?? this.#defaultMaxTokensEstimate, + this.#overlay, + // Key the endpoint on the routing provider (review M2). A media-only gate omits it (`maxTokens: 0` + // makes the token estimate 0 regardless), so `official` is a harmless default there. + (provider === undefined ? undefined : this.#resolveEndpoint?.(provider)) ?? 'official', + ) + (mediaUnitsEstimate === undefined ? 0 : estimateMediaCost(model, mediaUnitsEstimate, this.#overlay)); @@ -152,7 +200,22 @@ export class BudgetGovernor { // degrade to `allow`, mirroring the FallbackChain's "unpriced ⇒ no-cost" policy (H4). A self-hosted // model has ~no metered cost, so the cap simply does not constrain it. Any other error is a real bug. if (err instanceof UnknownModelError) { - return { kind: 'allow' }; + // A model with no price. The cap CANNOT bound it — we do not know what a turn costs. Two ways to treat that: + if (this.#budget.strict_cost_cap === true) { + // The user asked for a hard cap. If we cannot price it, we do not run it — that is what "strict" means. + return { + kind: 'fail', + error: new BudgetExceededError( + this.#cumulativeCostMicrocents, + this.#budget.max_cost_microcents, + this.#cumulativeCostMicrocents, + `model '${model}' has no price, so the ${this.#budget.max_cost_microcents}-micro-cent cap cannot be enforced on it (strict_cost_cap is on). Price it with \`relavium models pricing ${model}\`, or turn strict_cost_cap off.`, + ), + }; + } + // The ordinary trade (ADR-0028 H4): a self-hosted model has ~no metered cost, and refusing an otherwise + // valid run over a missing price is worse than the small risk. Allow — but SAY it is unpriced, once. + return { kind: 'unpriced', model }; } throw err; } @@ -203,9 +266,19 @@ export class BudgetGovernor { model: string, maxTokens: number | undefined, mediaUnitsEstimate?: readonly MediaUnitsEstimate[], + provider?: ProviderId, ): Promise { - const result = this.evaluatePreEgress(model, maxTokens, mediaUnitsEstimate); + const result = this.evaluatePreEgress(model, maxTokens, mediaUnitsEstimate, provider); if (result.kind === 'allow') return; + if (result.kind === 'unpriced') { + // Once per model — a standing condition, not an event (a `loop` over an unpriced model must not repeat it + // every iteration). The engine cannot print; the host is told and decides where the sentence goes. + if (!this.#unpricedNotified.has(result.model)) { + this.#unpricedNotified.add(result.model); + this.#onUnpriced?.(result.model, this.#budget.max_cost_microcents); + } + return; + } if (result.kind === 'warn') { if (!this.#warningEmitted) { this.#warningEmitted = true; diff --git a/packages/core/src/engine/engine.ts b/packages/core/src/engine/engine.ts index bcd80bd3..e630d0ad 100644 --- a/packages/core/src/engine/engine.ts +++ b/packages/core/src/engine/engine.ts @@ -50,7 +50,7 @@ import { type RunStatus, type TokensUsed, } from '@relavium/shared'; -import type { MediaJobStatus, PricingOverlay } from '@relavium/llm'; +import type { EndpointKind, MediaJobStatus, PricingOverlay, ProviderId } from '@relavium/llm'; import { buildRunPlan, type BuildRunPlanOptions } from '../dag.js'; import { InterpolationError } from '../errors.js'; @@ -243,11 +243,23 @@ export interface WorkflowEngineDeps { * The user-pricing overlay (2.5.G S10, [ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md) * §2) — a `ReadonlyMap` the host projects from the `model_catalog` `source='user'` rows. * It feeds the workflow PRE-EGRESS budget governor so a model with no static price, once user-priced, is enforced - * by `budget.max_cost_microcents`. Static `MODEL_PRICING` still wins for a known id (fills an UNKNOWN id only). + * by `budget.max_cost_microcents`. The USER tier outranks the catalog (ADR-0071 §1). * Injected exactly like the realized path's overlay, which the node executor's runner already carries; omit ⇒ * an unknown model degrades cost governance to `allow` loudly, unchanged. */ readonly resolvePrice?: PricingOverlay; + /** + * Is a model's provider on its own API, or behind a custom `base_url` (ADR-0071 §7)? Forwarded to the pre-egress + * {@link BudgetGovernor}: the adapter clamps an authored `max_tokens` to the model's ceiling only on an official + * endpoint, and an estimate that assumes otherwise stops describing the request the wire will carry. Absent ⇒ + * official (the adapter's own default). + */ + readonly resolveEndpoint?: (provider: ProviderId) => EndpointKind; + /** + * Called once per model when a turn runs UNPRICED, so the cost cap could not apply to it (ADR-0071 §K7). The + * engine cannot print; the host routes it (`run` → stderr). Absent ⇒ silent (`strict_cost_cap` is the block). + */ + readonly onUnpriced?: (model: string, capMicrocents: number) => void; } function maskInputs( @@ -343,6 +355,8 @@ class RunExecution { /** The user-pricing overlay (2.5.G S10, ADR-0065 §2) — into the workflow PRE-EGRESS governor so a user-priced * model is enforced by `budget`. Host-injected; the realized path rides the runner's own `resolvePrice`. */ resolvePrice?: PricingOverlay; + resolveEndpoint?: (provider: ProviderId) => EndpointKind; + onUnpriced?: (model: string, capMicrocents: number) => void; /** When present, the run is REHYDRATED from this checkpoint (resume) rather than started fresh (1.R). */ checkpoint?: CheckpointState; }) { @@ -372,6 +386,10 @@ class RunExecution { defaultMaxTokensEstimate: this.#maxTokensEstimate, emit: (draft) => this.#emitDurable({ ...draft, runId: this.runId }), ...(params.resolvePrice === undefined ? {} : { resolvePrice: params.resolvePrice }), + ...(params.resolveEndpoint === undefined + ? {} + : { resolveEndpoint: params.resolveEndpoint }), + ...(params.onUnpriced === undefined ? {} : { onUnpriced: params.onUnpriced }), }); } @@ -1074,7 +1092,8 @@ class RunExecution { // Pass the media-unit estimate (1.AF/D17) so the governor folds a per-modality media addend into the // projection; `outputModalities` rides the hook info for request-lowering/observability but the cost // calc needs only the units. - return (info) => governor.checkPreEgress(info.model, info.maxTokens, info.mediaUnitsEstimate); + return (info) => + governor.checkPreEgress(info.model, info.maxTokens, info.mediaUnitsEstimate, info.provider); } /** Run one attempt of a vertex; returns its outcome (an uncaught handler throw → a single `internal`). */ @@ -2192,6 +2211,12 @@ export class WorkflowEngine { readonly #resolverCapabilities: ResolverCapabilities; readonly #maxTokensEstimate: number; readonly #resolvePrice: PricingOverlay | undefined; + // Stored + forwarded to every RunExecution, exactly like #resolvePrice. They reached this class through + // WorkflowEngineDeps and then died here — the constructor never read them, so `start()`/`resumeFromCheckpoint()` + // built a governor without an endpoint resolver (ADR-0071 §7 — the estimate assumed `official` and under- + // authorized a custom-base_url turn) and without an unpriced sink (§K7 — the notice was dead on `run`/`gate`). + readonly #resolveEndpoint: ((provider: ProviderId) => EndpointKind) | undefined; + readonly #onUnpriced: ((model: string, capMicrocents: number) => void) | undefined; readonly #runs = new Map(); constructor(deps: WorkflowEngineDeps) { @@ -2202,6 +2227,8 @@ export class WorkflowEngine { this.#resolverCapabilities = deps.resolverCapabilities ?? {}; this.#maxTokensEstimate = deps.maxTokensEstimate ?? DEFAULT_MAX_TOKENS_ESTIMATE; this.#resolvePrice = deps.resolvePrice; + this.#resolveEndpoint = deps.resolveEndpoint; + this.#onUnpriced = deps.onUnpriced; } /** @@ -2231,6 +2258,8 @@ export class WorkflowEngine { resolverCapabilities: this.#resolverCapabilities, maxTokensEstimate: this.#maxTokensEstimate, ...(this.#resolvePrice === undefined ? {} : { resolvePrice: this.#resolvePrice }), + ...(this.#resolveEndpoint === undefined ? {} : { resolveEndpoint: this.#resolveEndpoint }), + ...(this.#onUnpriced === undefined ? {} : { onUnpriced: this.#onUnpriced }), }); this.#runs.set(runId, execution); void execution.begin(); @@ -2327,6 +2356,8 @@ export class WorkflowEngine { resolverCapabilities: this.#resolverCapabilities, maxTokensEstimate: this.#maxTokensEstimate, ...(this.#resolvePrice === undefined ? {} : { resolvePrice: this.#resolvePrice }), + ...(this.#resolveEndpoint === undefined ? {} : { resolveEndpoint: this.#resolveEndpoint }), + ...(this.#onUnpriced === undefined ? {} : { onUnpriced: this.#onUnpriced }), checkpoint, }); this.#runs.set(input.runId, execution); diff --git a/packages/core/src/engine/reasoning-effort.test.ts b/packages/core/src/engine/reasoning-effort.test.ts new file mode 100644 index 00000000..09b87ed6 --- /dev/null +++ b/packages/core/src/engine/reasoning-effort.test.ts @@ -0,0 +1,138 @@ +import type { ReasoningEffort } from '@relavium/shared'; +import { describe, expect, it } from 'vitest'; + +import { effortToSend, gateReasoningEffort } from './reasoning-effort.js'; + +/** + * THE EFFORT GATE — and it had **no test at all** before this, which is how F3 shipped. + * + * The old gate was pass-through-or-withhold: *"does this model reason?"* → yes → send whatever tier the user + * picked, unexamined. But reasoning is not the question the wire asks. `gpt-5.4-pro` reasons **and rejects + * `low`**. `gemini-2.5-pro` reasons and **cannot be turned off**. A boolean answers `true` to both, and the + * rejected value goes straight to the provider — which is the 400 the maintainer reported. + */ + +const tiers = (...list: ReasoningEffort[]): ReadonlySet => new Set(list); + +describe('gateReasoningEffort — it CLAMPS now; it used to just pass through', () => { + it('THE BUG: a tier the model rejects is WITHHELD, not sent', () => { + // gpt-5.4-pro accepts {medium, high, max}. The old gate saw "reasons: true" and sent `low` anyway. + const result = gateReasoningEffort('low', 'gpt-5.4-pro', () => tiers('medium', 'high', 'max')); + expect(result).toEqual({ + kind: 'rejected', + requested: 'low', + accepted: ['medium', 'high', 'max'], + }); + expect(effortToSend(result)).toBeUndefined(); // nothing reaches the wire + }); + + it('a rejected tier is NEVER promoted to a neighbour — that would change behaviour AND raise spend, silently', () => { + // The tempting "fix" is to round `low` up to `medium` so the call succeeds. It must not: the user gets more + // thinking than they asked for, pays more for it, and is told nothing. Withholding is honest; promotion is not. + const result = gateReasoningEffort('low', 'm', () => tiers('high')); + expect(result.kind).toBe('rejected'); + expect(effortToSend(result)).toBeUndefined(); + }); + + it('an ACCEPTED tier is sent unchanged', () => { + const result = gateReasoningEffort('high', 'm', () => tiers('medium', 'high', 'max')); + expect(result).toEqual({ kind: 'send', effort: 'high' }); + expect(effortToSend(result)).toBe('high'); + }); + + it('`off` is gated like any other tier — gemini-2.5-pro cannot be turned off, so `off` is withheld', () => { + // Google: "N/A: Cannot disable thinking". The old code mapped off→MINIMAL and sent it, which neither disabled + // thinking nor was a value the model takes — so a user who switched reasoning OFF was billed for reasoning. + const proTiers = tiers('low', 'medium', 'high', 'max'); // no `off` + expect(gateReasoningEffort('off', 'gemini-2.5-pro', () => proTiers).kind).toBe('rejected'); + // …while a model that CAN be disabled sends it. + expect( + effortToSend(gateReasoningEffort('off', 'gemini-2.5-flash', () => tiers('off', 'low'))), + ).toBe('off'); + }); +}); + +describe('the CAP gate — an accepted tier a small max_tokens withholds is `capped`, not silent (review M6)', () => { + const accepts = () => tiers('off', 'low', 'medium', 'high', 'max'); + + it('an accepted tier the cap withholds returns `capped` with the offending max_tokens', () => { + // The model accepts `medium`, but the injected check (a budget-shaped model under a tight cap) says the adapter + // would drop it. The gate surfaces `capped` — not `send` (silent drop) and not `rejected` (the model is fine). + const result = gateReasoningEffort('medium', 'claude-haiku-4-5', accepts, { + maxTokens: 500, + withheldByCap: () => true, + }); + expect(result).toEqual({ kind: 'capped', requested: 'medium', maxTokens: 500 }); + expect(effortToSend(result)).toBeUndefined(); // the field is withheld, exactly as the adapter would + }); + + it('when the cap check passes, the accepted tier sends unchanged', () => { + const result = gateReasoningEffort('medium', 'claude-haiku-4-5', accepts, { + maxTokens: 8000, + withheldByCap: () => false, + }); + expect(result).toEqual({ kind: 'send', effort: 'medium' }); + }); + + it('`off` is never cap-checked — it carries no budget, so the check is not even consulted', () => { + let consulted = false; + const result = gateReasoningEffort('off', 'claude-haiku-4-5', accepts, { + maxTokens: 1, + withheldByCap: () => { + consulted = true; + return true; + }, + }); + expect(result).toEqual({ kind: 'send', effort: 'off' }); + expect(consulted).toBe(false); + }); + + it('with no cap wired, an accepted tier still sends (unchanged behaviour, the adapter default is large)', () => { + expect(gateReasoningEffort('medium', 'claude-haiku-4-5', accepts)).toEqual({ + kind: 'send', + effort: 'medium', + }); + }); + + it('a REJECTED tier short-circuits before the cap check — the model, not the cap, is the blocker', () => { + let consulted = false; + const result = gateReasoningEffort('low', 'gpt-5.4-pro', () => tiers('medium', 'high', 'max'), { + maxTokens: 1, + withheldByCap: () => { + consulted = true; + return true; + }, + }); + expect(result.kind).toBe('rejected'); + expect(consulted).toBe(false); + }); +}); + +describe('the withhold cases — every one of them omits the field', () => { + it('no tier requested ⇒ `unset`, and nothing is sent (the common path)', () => { + expect(gateReasoningEffort(undefined, 'm', () => tiers('high'))).toEqual({ kind: 'unset' }); + }); + + it('an EMPTY accepted set ⇒ `uncontrollable` — the model reasons but exposes no tier', () => { + // `deepseek-reasoner`. Distinct from "does not reason", and the distinction matters: it tells the picker to + // offer NOTHING rather than to offer everything, which is precisely the old behaviour. + expect(gateReasoningEffort('high', 'deepseek-reasoner', () => new Set())).toEqual({ + kind: 'uncontrollable', + }); + }); + + it('an UNKNOWN model (no resolver, or a custom endpoint) ⇒ `uncontrollable` — the SAFE default', () => { + // Guessing is what put a rejected value on the wire in the first place. A model we cannot describe gets no + // reasoning field at all. + expect(gateReasoningEffort('high', 'some-custom-model', () => undefined).kind).toBe( + 'uncontrollable', + ); + expect(gateReasoningEffort('high', 'm', undefined).kind).toBe('uncontrollable'); + }); + + it('`accepted` is carried on a rejection so a surface can say something ACTIONABLE', () => { + // "gpt-5.4-pro does not accept `low` — it takes medium, high or max" beats "the request failed". + const result = gateReasoningEffort('off', 'gpt-5.4-pro', () => tiers('medium', 'high', 'max')); + expect(result.kind === 'rejected' && result.accepted).toEqual(['medium', 'high', 'max']); + }); +}); diff --git a/packages/core/src/engine/reasoning-effort.ts b/packages/core/src/engine/reasoning-effort.ts index 8151a4cf..c7228daf 100644 --- a/packages/core/src/engine/reasoning-effort.ts +++ b/packages/core/src/engine/reasoning-effort.ts @@ -1,20 +1,95 @@ import type { ReasoningEffort } from '@relavium/shared'; /** - * Gate the normalized reasoning-effort tier ([ADR-0066](../../../../docs/decisions/0066-normalized-reasoning-effort-control.md)) - * against a model's per-model capability. The tier is sent to the provider ONLY when BOTH hold: an effort was - * resolved (authored on the agent, a `[chat]` config default baked onto the agent, or a session-level override) AND - * the host's per-model capability projection (`resolveReasoning`, the ADR-0064 catalog capability) says THIS model - * reasons. A non-reasoning model would reject the field, so an unknown/absent resolver (`undefined`) is treated as - * NOT reasoning — the field is withheld (the safe default; §4). + * The host's per-model projection of **which reasoning tiers this model will actually accept** + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). * - * The one home for the gate (used by both the workflow `AgentRunner` path and the `AgentSession` per-turn build) so - * the rule cannot drift between them. Pure — `packages/core` stays platform-free; the host injects `resolveReasoning`. + * It replaces a `boolean`, and the upgrade is the whole point: *"does this model reason"* is not the question the + * wire asks. `gpt-5.4-pro` reasons **and rejects `low`**; `gpt-5-pro` reasons and accepts only `high`; + * `gemini-2.5-pro` reasons and **cannot be turned off at all**. A boolean answers `true` to every one of them and + * the tier the user picked goes straight to the provider — which is the 400 the maintainer reported. + * + * `undefined` ⇒ the host cannot describe this model (no resolver wired, or a custom endpoint the catalog does not + * carry). An **empty set** ⇒ the model reasons but exposes no controllable tier (`deepseek-reasoner`). Both mean + * the same thing at the wire: **withhold the field**. + */ +export type ResolveEffortTiers = (model: string) => ReadonlySet | undefined; + +/** What the gate decided, and — when it withheld — enough to tell the user WHY without guessing. */ +export type EffortGateResult = + | { readonly kind: 'send'; readonly effort: ReasoningEffort } + /** No tier was resolved at all (nothing authored, no config default, no session override). Not a problem. */ + | { readonly kind: 'unset' } + /** The model does not reason, or exposes no controllable tier. The field is withheld. */ + | { readonly kind: 'uncontrollable' } + /** + * The model reasons, but **rejects the requested tier**. Withheld — never silently promoted to a neighbouring + * one, which would change behaviour *and* raise spend without the user asking. `accepted` is what it WOULD take, + * so a surface can say something actionable instead of "it failed". + */ + | { + readonly kind: 'rejected'; + readonly requested: ReasoningEffort; + readonly accepted: readonly ReasoningEffort[]; + } + /** + * The model ACCEPTS the tier, but a budget-shaped model withholds it because the request's `max_tokens` leaves no + * room for its minimum thinking budget (review M6). The blocker is the CAP, not the model — so the message points + * the user at `max_tokens`, not at the tier. `maxTokens` is the cap that was too small. + */ + | { readonly kind: 'capped'; readonly requested: ReasoningEffort; readonly maxTokens: number }; + +/** Does a budget-shaped model withhold this tier under this output cap? Host-injected (it reads the catalog for the + * model's budget range); the pure gate stays platform-free. Called only for a NON-`off` tier the model accepts. */ +export type ReasoningCapCheck = ( + model: string, + tier: ReasoningEffort, + maxTokens: number, +) => boolean; + +/** + * Gate — and now **CLAMP** — the normalized reasoning-effort tier against what the model actually accepts + * ([ADR-0066](../../../../docs/decisions/0066-normalized-reasoning-effort-control.md), + * [ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * The previous version was **pass-through-or-withhold**, not a clamp: if the model reasoned *at all*, whatever + * tier the user picked went to the wire unexamined. That is the F3 bug. It now sends a tier only if the model is + * on record as taking it. + * + * A rejected tier is **withheld, never promoted**. Substituting the nearest acceptable tier would silently change + * what the model does and silently raise what it costs — and the user would have no way to know either happened. + * + * The one home for the rule (both the workflow `AgentRunner` path and the `AgentSession` per-turn build use it), + * so it cannot drift between them. Pure — `packages/core` stays platform-free; the host injects the resolver. */ export function gateReasoningEffort( effort: ReasoningEffort | undefined, model: string, - resolveReasoning: ((model: string) => boolean | undefined) | undefined, -): ReasoningEffort | undefined { - return effort !== undefined && resolveReasoning?.(model) === true ? effort : undefined; + resolveEffortTiers: ResolveEffortTiers | undefined, + cap?: { readonly maxTokens: number; readonly withheldByCap: ReasoningCapCheck }, +): EffortGateResult { + if (effort === undefined) return { kind: 'unset' }; + + const accepted = resolveEffortTiers?.(model); + // No resolver, an unknown model, or a model with no controllable tier — all withhold. An unknown model is the + // safe default on purpose: guessing is what put a rejected value on the wire in the first place. + if (accepted === undefined || accepted.size === 0) return { kind: 'uncontrollable' }; + + if (!accepted.has(effort)) + return { kind: 'rejected', requested: effort, accepted: [...accepted] }; + + // The tier is accepted — but a budget-shaped model still withholds it at send when the request's cap leaves no + // room for the budget floor (review M6). `off` is never budgeted, so it is exempt. Surface it as `capped` so the + // adapter's silent drop becomes a message that names `max_tokens` as the fix. A cap is only checked when the host + // both wired the check and set a `max_tokens`; an absent cap uses the adapter's (large) default, which never + // withholds. The gate and the adapter agree by construction: both read `reasoningBudgetFor`. + if (effort !== 'off' && cap?.withheldByCap(model, effort, cap.maxTokens) === true) { + return { kind: 'capped', requested: effort, maxTokens: cap.maxTokens }; + } + return { kind: 'send', effort }; +} + +/** The tier to send, or `undefined` to omit the field — the shorthand a request-builder wants. */ +export function effortToSend(result: EffortGateResult): ReasoningEffort | undefined { + return result.kind === 'send' ? result.effort : undefined; } diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 56fdde9f..79c59941 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -161,6 +161,14 @@ export type { RunLoopInvariantCode } from './engine/invariant-error.js'; // to the package — AgentSession (1.V) imports it intra-package, not from this curated surface. export { createAgentNodeExecutor } from './engine/agent-runner.js'; export type { AgentRunnerDeps } from './engine/agent-runner.js'; +// The per-model reasoning-effort gate (ADR-0071 §6). The engine decides WHETHER a tier is sent; the host injects +// which tiers a model takes and receives the verdict when one is withheld, so it can say so in its own voice. +export { effortToSend, gateReasoningEffort } from './engine/reasoning-effort.js'; +export type { + EffortGateResult, + ReasoningCapCheck, + ResolveEffortTiers, +} from './engine/reasoning-effort.js'; export { DEFAULT_AGENT_TURN_LIMITS } from './engine/agent-turn.js'; export type { AgentTurnLimits, PreEgressHook } from './engine/agent-turn.js'; diff --git a/packages/db/drizzle/0011_odd_thunderbolt.sql b/packages/db/drizzle/0011_odd_thunderbolt.sql new file mode 100644 index 00000000..5fff0425 --- /dev/null +++ b/packages/db/drizzle/0011_odd_thunderbolt.sql @@ -0,0 +1,17 @@ +ALTER TABLE `model_catalog` ADD `cached_input_stated` integer DEFAULT 0 NOT NULL;--> statement-breakpoint +-- WHY A FLAG AND NOT A NULLABLE COLUMN (ADR-0071 §10). +-- +-- `cached_input_cost_per_mtok_microcents` is `NOT NULL DEFAULT 0`, and SQLite cannot drop a NOT NULL constraint with +-- ALTER — the only route is a table rebuild, and `model_catalog.id` is an FK target from five tables. So the FACT of +-- the statement gets its own column instead, additively, and every existing row is correct at the default: +-- +-- • A `source='user'` row written before this migration and carrying a NON-ZERO cache rate was, necessarily, one +-- the user stated — the old command only ever wrote the column when `--cached` was passed. The backfill below +-- records that. +-- • A row carrying `0` is genuinely ambiguous in the old schema: it may be "never mentioned" (overwhelmingly the +-- common case — the flag was optional and rarely used) or an explicit `--cached 0`. It stays `not stated`, which +-- resolves to the catalog's discount applied to the user's own input rate rather than to free tokens. Erring +-- toward billing is the only safe direction on a money column: the user can restate `--cached 0` and be believed. +UPDATE `model_catalog` + SET `cached_input_stated` = 1 + WHERE `source` = 'user' AND `cached_input_cost_per_mtok_microcents` > 0; diff --git a/packages/db/drizzle/meta/0011_snapshot.json b/packages/db/drizzle/meta/0011_snapshot.json new file mode 100644 index 00000000..9fea6e51 --- /dev/null +++ b/packages/db/drizzle/meta/0011_snapshot.json @@ -0,0 +1,2225 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "99ed22cf-fd27-45e7-8afa-3900a627c037", + "prevId": "2e420300-8768-49c2-b459-8e366540f315", + "tables": { + "agent_sessions": { + "name": "agent_sessions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_slug": { + "name": "agent_slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "agent_snapshot": { + "name": "agent_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "working_dir": { + "name": "working_dir", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "git_ref": { + "name": "git_ref", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fs_scope_tier": { + "name": "fs_scope_tier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'sandboxed'" + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'active'" + }, + "context_json": { + "name": "context_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "total_input_tokens": { + "name": "total_input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_output_tokens": { + "name": "total_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_cost_microcents": { + "name": "total_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "exported_workflow_path": { + "name": "exported_workflow_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_agent_sessions_status": { + "name": "idx_agent_sessions_status", + "columns": [ + "status", + "\"updated_at\" desc" + ], + "isUnique": false, + "where": "\"agent_sessions\".\"deleted_at\" is null" + }, + "idx_agent_sessions_agent": { + "name": "idx_agent_sessions_agent", + "columns": [ + "agent_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"agent_sessions\".\"agent_id\" is not null" + }, + "idx_agent_sessions_updated": { + "name": "idx_agent_sessions_updated", + "columns": [ + "\"updated_at\" desc", + "\"id\" desc" + ], + "isUnique": false, + "where": "\"agent_sessions\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "agent_sessions_agent_id_agents_id_fk": { + "name": "agent_sessions_agent_id_agents_id_fk", + "tableFrom": "agent_sessions", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "agent_sessions_model_id_model_catalog_id_fk": { + "name": "agent_sessions_model_id_model_catalog_id_fk", + "tableFrom": "agent_sessions", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "agent_sessions_fs_scope_tier_check": { + "name": "agent_sessions_fs_scope_tier_check", + "value": "\"agent_sessions\".\"fs_scope_tier\" in ('sandboxed', 'project', 'full')" + }, + "agent_sessions_status_check": { + "name": "agent_sessions_status_check", + "value": "\"agent_sessions\".\"status\" in ('active', 'idle', 'exported', 'ended')" + } + } + }, + "agents": { + "name": "agents", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "system_prompt": { + "name": "system_prompt", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "''" + }, + "tools": { + "name": "tools", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "input_schema": { + "name": "input_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "output_schema": { + "name": "output_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "source_path": { + "name": "source_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_agents_slug": { + "name": "idx_agents_slug", + "columns": [ + "slug" + ], + "isUnique": true, + "where": "\"agents\".\"deleted_at\" is null" + }, + "idx_agents_model": { + "name": "idx_agents_model", + "columns": [ + "model_id" + ], + "isUnique": false + }, + "idx_agents_active": { + "name": "idx_agents_active", + "columns": [ + "is_active", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"agents\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "agents_model_id_model_catalog_id_fk": { + "name": "agents_model_id_model_catalog_id_fk", + "tableFrom": "agents", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "llm_providers": { + "name": "llm_providers", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "base_url": { + "name": "base_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "api_key_keychain_ref": { + "name": "api_key_keychain_ref", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "default_headers": { + "name": "default_headers", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "kind": { + "name": "kind", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "pricing_reference_url": { + "name": "pricing_reference_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_llm_providers_name": { + "name": "idx_llm_providers_name", + "columns": [ + "name" + ], + "isUnique": true, + "where": "\"llm_providers\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "media_objects": { + "name": "media_objects", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "handle": { + "name": "handle", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "mime_type": { + "name": "mime_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "modality": { + "name": "modality", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "byte_length": { + "name": "byte_length", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_referenced_at": { + "name": "last_referenced_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "media_objects_handle_unique": { + "name": "media_objects_handle_unique", + "columns": [ + "handle" + ], + "isUnique": true + }, + "idx_media_objects_gc": { + "name": "idx_media_objects_gc", + "columns": [ + "last_referenced_at" + ], + "isUnique": false, + "where": "\"media_objects\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "media_objects_modality_check": { + "name": "media_objects_modality_check", + "value": "\"media_objects\".\"modality\" in ('image', 'audio', 'video', 'document')" + } + } + }, + "media_references": { + "name": "media_references", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "handle": { + "name": "handle", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "scope_kind": { + "name": "scope_kind", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "scope_id": { + "name": "scope_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_media_references_unique": { + "name": "idx_media_references_unique", + "columns": [ + "handle", + "scope_kind", + "scope_id" + ], + "isUnique": true + }, + "idx_media_references_scope": { + "name": "idx_media_references_scope", + "columns": [ + "scope_kind", + "scope_id" + ], + "isUnique": false + }, + "idx_media_references_handle": { + "name": "idx_media_references_handle", + "columns": [ + "handle" + ], + "isUnique": false + } + }, + "foreignKeys": { + "media_references_handle_media_objects_handle_fk": { + "name": "media_references_handle_media_objects_handle_fk", + "tableFrom": "media_references", + "tableTo": "media_objects", + "columnsFrom": [ + "handle" + ], + "columnsTo": [ + "handle" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "media_references_scope_kind_check": { + "name": "media_references_scope_kind_check", + "value": "\"media_references\".\"scope_kind\" in ('run', 'node', 'session', 'workspace')" + } + } + }, + "messages": { + "name": "messages", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "step_execution_id": { + "name": "step_execution_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "sequence_number": { + "name": "sequence_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "content_parts": { + "name": "content_parts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_calls": { + "name": "tool_calls", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_call_id": { + "name": "tool_call_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finish_reason": { + "name": "finish_reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_messages_step": { + "name": "idx_messages_step", + "columns": [ + "step_execution_id", + "sequence_number" + ], + "isUnique": false + }, + "idx_messages_run": { + "name": "idx_messages_run", + "columns": [ + "run_id", + "created_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "messages_step_execution_id_step_executions_id_fk": { + "name": "messages_step_execution_id_step_executions_id_fk", + "tableFrom": "messages", + "tableTo": "step_executions", + "columnsFrom": [ + "step_execution_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "model_catalog": { + "name": "model_catalog", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "context_window_tokens": { + "name": "context_window_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "max_output_tokens": { + "name": "max_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "input_cost_per_mtok_microcents": { + "name": "input_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_cost_per_mtok_microcents": { + "name": "output_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cached_input_cost_per_mtok_microcents": { + "name": "cached_input_cost_per_mtok_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cached_input_stated": { + "name": "cached_input_stated", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "media_image_cost_microcents": { + "name": "media_image_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_audio_cost_microcents": { + "name": "media_audio_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_video_cost_microcents": { + "name": "media_video_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "media_surface": { + "name": "media_surface", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'chat'" + }, + "supports_tool_calling": { + "name": "supports_tool_calling", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "supports_vision": { + "name": "supports_vision", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "supports_streaming": { + "name": "supports_streaming", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "supports_json_mode": { + "name": "supports_json_mode", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "0" + }, + "capabilities": { + "name": "capabilities", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "deprecation_date": { + "name": "deprecation_date", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'static'" + }, + "last_refreshed_at": { + "name": "last_refreshed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_model_catalog_provider_model": { + "name": "idx_model_catalog_provider_model", + "columns": [ + "provider_id", + "model_id" + ], + "isUnique": true, + "where": "\"model_catalog\".\"deleted_at\" is null" + }, + "idx_model_catalog_provider": { + "name": "idx_model_catalog_provider", + "columns": [ + "provider_id" + ], + "isUnique": false + }, + "idx_model_catalog_active": { + "name": "idx_model_catalog_active", + "columns": [ + "is_active" + ], + "isUnique": false, + "where": "\"model_catalog\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "model_catalog_provider_id_llm_providers_id_fk": { + "name": "model_catalog_provider_id_llm_providers_id_fk", + "tableFrom": "model_catalog", + "tableTo": "llm_providers", + "columnsFrom": [ + "provider_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "run_costs": { + "name": "run_costs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_run_costs_run": { + "name": "idx_run_costs_run", + "columns": [ + "run_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "run_costs_run_id_runs_id_fk": { + "name": "run_costs_run_id_runs_id_fk", + "tableFrom": "run_costs", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "run_costs_model_id_model_catalog_id_fk": { + "name": "run_costs_model_id_model_catalog_id_fk", + "tableFrom": "run_costs", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "run_events": { + "name": "run_events", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "step_execution_id": { + "name": "step_execution_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "seq": { + "name": "seq", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "event_type": { + "name": "event_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "level": { + "name": "level", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'info'" + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "payload_json": { + "name": "payload_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "ts": { + "name": "ts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_run_events_run_seq": { + "name": "idx_run_events_run_seq", + "columns": [ + "run_id", + "seq" + ], + "isUnique": true + }, + "idx_run_events_step": { + "name": "idx_run_events_step", + "columns": [ + "step_execution_id", + "ts" + ], + "isUnique": false, + "where": "\"run_events\".\"step_execution_id\" is not null" + }, + "idx_run_events_run_type": { + "name": "idx_run_events_run_type", + "columns": [ + "run_id", + "event_type", + "ts" + ], + "isUnique": false + } + }, + "foreignKeys": { + "run_events_run_id_runs_id_fk": { + "name": "run_events_run_id_runs_id_fk", + "tableFrom": "run_events", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "runs": { + "name": "runs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "workflow_id": { + "name": "workflow_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "workflow_path": { + "name": "workflow_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "project_root": { + "name": "project_root", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "workflow_definition_snapshot": { + "name": "workflow_definition_snapshot", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "execution_mode": { + "name": "execution_mode", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'local'" + }, + "trigger_type": { + "name": "trigger_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'manual'" + }, + "trigger_metadata": { + "name": "trigger_metadata", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "input_json": { + "name": "input_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "output_json": { + "name": "output_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_json": { + "name": "error_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "total_input_tokens": { + "name": "total_input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_output_tokens": { + "name": "total_output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "total_cost_microcents": { + "name": "total_cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_runs_workflow": { + "name": "idx_runs_workflow", + "columns": [ + "workflow_id", + "\"created_at\" desc" + ], + "isUnique": false + }, + "idx_runs_status": { + "name": "idx_runs_status", + "columns": [ + "status", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"runs\".\"deleted_at\" is null" + }, + "idx_runs_cost": { + "name": "idx_runs_cost", + "columns": [ + "workflow_id", + "created_at", + "total_cost_microcents" + ], + "isUnique": false, + "where": "\"runs\".\"deleted_at\" is null" + }, + "idx_runs_created": { + "name": "idx_runs_created", + "columns": [ + "\"created_at\" desc", + "\"id\" desc" + ], + "isUnique": false, + "where": "\"runs\".\"deleted_at\" is null" + } + }, + "foreignKeys": { + "runs_workflow_id_workflows_id_fk": { + "name": "runs_workflow_id_workflows_id_fk", + "tableFrom": "runs", + "tableTo": "workflows", + "columnsFrom": [ + "workflow_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "runs_status_check": { + "name": "runs_status_check", + "value": "\"runs\".\"status\" in ('pending', 'running', 'paused', 'completed', 'failed', 'cancelled')" + }, + "runs_execution_mode_check": { + "name": "runs_execution_mode_check", + "value": "\"runs\".\"execution_mode\" in ('local', 'cloud', 'managed')" + } + } + }, + "session_costs": { + "name": "session_costs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model": { + "name": "model", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "model_catalog_id": { + "name": "model_catalog_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "call_count": { + "name": "call_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "unpriced_calls": { + "name": "unpriced_calls", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "is_legacy": { + "name": "is_legacy", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_session_costs_session_model": { + "name": "idx_session_costs_session_model", + "columns": [ + "session_id", + "model", + "is_legacy" + ], + "isUnique": true + }, + "idx_session_costs_session": { + "name": "idx_session_costs_session", + "columns": [ + "session_id", + "cost_microcents" + ], + "isUnique": false + } + }, + "foreignKeys": { + "session_costs_session_id_agent_sessions_id_fk": { + "name": "session_costs_session_id_agent_sessions_id_fk", + "tableFrom": "session_costs", + "tableTo": "agent_sessions", + "columnsFrom": [ + "session_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "session_costs_model_catalog_id_model_catalog_id_fk": { + "name": "session_costs_model_catalog_id_model_catalog_id_fk", + "tableFrom": "session_costs", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_catalog_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "session_costs_model_nonempty": { + "name": "session_costs_model_nonempty", + "value": "\"session_costs\".\"model\" <> ''" + } + } + }, + "session_messages": { + "name": "session_messages", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "sequence_number": { + "name": "sequence_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "content_parts": { + "name": "content_parts", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_calls": { + "name": "tool_calls", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tool_call_id": { + "name": "tool_call_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "finish_reason": { + "name": "finish_reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "compaction_dropped_through_sequence": { + "name": "compaction_dropped_through_sequence", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_session_messages_seq": { + "name": "idx_session_messages_seq", + "columns": [ + "session_id", + "sequence_number" + ], + "isUnique": true + }, + "idx_session_messages_session": { + "name": "idx_session_messages_session", + "columns": [ + "session_id", + "created_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "session_messages_session_id_agent_sessions_id_fk": { + "name": "session_messages_session_id_agent_sessions_id_fk", + "tableFrom": "session_messages", + "tableTo": "agent_sessions", + "columnsFrom": [ + "session_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "session_messages_model_id_model_catalog_id_fk": { + "name": "session_messages_model_id_model_catalog_id_fk", + "tableFrom": "session_messages", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "step_executions": { + "name": "step_executions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_id": { + "name": "node_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "node_type": { + "name": "node_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "agent_id": { + "name": "agent_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "agent_snapshot": { + "name": "agent_snapshot", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "model_id": { + "name": "model_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "attempt_number": { + "name": "attempt_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "input_json": { + "name": "input_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "output_json": { + "name": "output_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_json": { + "name": "error_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "duration_ms": { + "name": "duration_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "input_tokens": { + "name": "input_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "output_tokens": { + "name": "output_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cached_tokens": { + "name": "cached_tokens", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "cost_microcents": { + "name": "cost_microcents", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_step_exec_run": { + "name": "idx_step_exec_run", + "columns": [ + "run_id", + "created_at" + ], + "isUnique": false + }, + "idx_step_exec_run_node": { + "name": "idx_step_exec_run_node", + "columns": [ + "run_id", + "node_id", + "attempt_number" + ], + "isUnique": false + }, + "idx_step_exec_agent": { + "name": "idx_step_exec_agent", + "columns": [ + "agent_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"step_executions\".\"agent_id\" is not null" + }, + "idx_step_exec_model": { + "name": "idx_step_exec_model", + "columns": [ + "model_id", + "\"created_at\" desc" + ], + "isUnique": false, + "where": "\"step_executions\".\"model_id\" is not null" + }, + "idx_step_exec_cost": { + "name": "idx_step_exec_cost", + "columns": [ + "model_id", + "created_at", + "cost_microcents" + ], + "isUnique": false, + "where": "\"step_executions\".\"model_id\" is not null" + } + }, + "foreignKeys": { + "step_executions_run_id_runs_id_fk": { + "name": "step_executions_run_id_runs_id_fk", + "tableFrom": "step_executions", + "tableTo": "runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "step_executions_agent_id_agents_id_fk": { + "name": "step_executions_agent_id_agents_id_fk", + "tableFrom": "step_executions", + "tableTo": "agents", + "columnsFrom": [ + "agent_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + }, + "step_executions_model_id_model_catalog_id_fk": { + "name": "step_executions_model_id_model_catalog_id_fk", + "tableFrom": "step_executions", + "tableTo": "model_catalog", + "columnsFrom": [ + "model_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "no action", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "step_executions_status_check": { + "name": "step_executions_status_check", + "value": "\"step_executions\".\"status\" in ('pending', 'running', 'completed', 'failed', 'skipped')" + } + } + }, + "workflows": { + "name": "workflows", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "definition": { + "name": "definition", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "input_schema": { + "name": "input_schema", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "source_path": { + "name": "source_path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "1" + }, + "deleted_at": { + "name": "deleted_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "idx_workflows_slug": { + "name": "idx_workflows_slug", + "columns": [ + "slug" + ], + "isUnique": true, + "where": "\"workflows\".\"deleted_at\" is null" + }, + "idx_workflows_active": { + "name": "idx_workflows_active", + "columns": [ + "is_active", + "\"updated_at\" desc" + ], + "isUnique": false, + "where": "\"workflows\".\"deleted_at\" is null" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": { + "idx_agent_sessions_status": { + "columns": { + "\"updated_at\" desc": { + "isExpression": true + } + } + }, + "idx_agent_sessions_agent": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_agent_sessions_updated": { + "columns": { + "\"updated_at\" desc": { + "isExpression": true + }, + "\"id\" desc": { + "isExpression": true + } + } + }, + "idx_agents_active": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_runs_workflow": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_runs_status": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_runs_created": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + }, + "\"id\" desc": { + "isExpression": true + } + } + }, + "idx_step_exec_agent": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_step_exec_model": { + "columns": { + "\"created_at\" desc": { + "isExpression": true + } + } + }, + "idx_workflows_active": { + "columns": { + "\"updated_at\" desc": { + "isExpression": true + } + } + } + } + } +} \ No newline at end of file diff --git a/packages/db/drizzle/meta/_journal.json b/packages/db/drizzle/meta/_journal.json index d1f8108c..f80c451b 100644 --- a/packages/db/drizzle/meta/_journal.json +++ b/packages/db/drizzle/meta/_journal.json @@ -78,6 +78,13 @@ "when": 1783849225312, "tag": "0010_clumsy_jocasta", "breakpoints": true + }, + { + "idx": 11, + "version": "6", + "when": 1783935625312, + "tag": "0011_odd_thunderbolt", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/model-catalog-store.test.ts b/packages/db/src/model-catalog-store.test.ts index 2a09da96..30b9ca1f 100644 --- a/packages/db/src/model-catalog-store.test.ts +++ b/packages/db/src/model-catalog-store.test.ts @@ -5,6 +5,7 @@ import { createClient, runMigrations, type DbClient } from './client.js'; import { createModelCatalogStore, ModelCatalogCapabilitiesError, + type ModelCatalogListing, type ModelCatalogStore, } from './model-catalog-store.js'; import { createProviderStore, type ProviderStore } from './provider-store.js'; @@ -866,6 +867,50 @@ describe('createModelCatalogStore (2.5.G / ADR-0064 — live-discovery cache)', expect(listing?.cachedInputCostPerMtokMicrocents).toBe(42); }); + it('clearUserPricing resets the pricing columns, so a later partial re-price does not resurrect the cleared cache rate (review M3)', () => { + const read = (): ModelCatalogListing | undefined => + store.listByProvider(providerId).find((m) => m.modelId === 'cache-clear-model'); + + // 1) Price it WITH a stated cache rate. + store.upsert({ + providerId, + modelId: 'cache-clear-model', + displayName: 'Cache Clear', + source: 'user', + inputCostPerMtokMicrocents: 1_000_000, + outputCostPerMtokMicrocents: 2_000_000, + cachedInputCostPerMtokMicrocents: 5_000_000, + cachedInputStated: true, + }); + expect(read()).toMatchObject({ + cachedInputCostPerMtokMicrocents: 5_000_000, + cachedInputStated: true, + }); + + // 2) Clear it. + expect(store.clearUserPricing('cache-clear-model', providerId)).toBe(true); + expect(read()).toBeUndefined(); // deactivated ⇒ off every active-only reader + + // 3) Re-price PARTIALLY — input/output only, NO cache rate. The upsert reuses the FK-stable (now inactive) + // row. Before M3 the cleared row still carried cached=5_000_000/stated=true, and the omitted cache column + // was preserved — silently billing cache tokens at the cleared rate as if the user had just stated it. + store.upsert({ + providerId, + modelId: 'cache-clear-model', + source: 'user', + inputCostPerMtokMicrocents: 3_000_000, + outputCostPerMtokMicrocents: 4_000_000, + }); + + const repriced = read(); + expect(repriced?.inputCostPerMtokMicrocents).toBe(3_000_000); + expect(repriced?.outputCostPerMtokMicrocents).toBe(4_000_000); + // THE FIX: the cleared cache rate does not come back, and it is no longer marked user-stated — so a reader + // derives the cache rate from the catalog discount instead of billing the stale $5. + expect(repriced?.cachedInputCostPerMtokMicrocents).toBe(0); + expect(repriced?.cachedInputStated).toBe(false); + }); + it('providerRefreshedAt isolates by provider and ignores non-live rows', () => { const providerB = providerStore.upsert({ name: 'anthropic', diff --git a/packages/db/src/model-catalog-store.ts b/packages/db/src/model-catalog-store.ts index ee5401cc..b6fd3b69 100644 --- a/packages/db/src/model-catalog-store.ts +++ b/packages/db/src/model-catalog-store.ts @@ -74,6 +74,8 @@ export interface ModelCatalogUpsert { readonly inputCostPerMtokMicrocents?: number; readonly outputCostPerMtokMicrocents?: number; readonly cachedInputCostPerMtokMicrocents?: number; + /** Did the USER state the cache rate (ADR-0071 §10)? Absent ⇒ preserve whatever the row already says. */ + readonly cachedInputStated?: boolean; /** The provenance discriminant ([ADR-0064] §4). OMITTED ⇒ `'static'` (a hardcoded seed), so every existing * media-routing caller is unchanged; the live refresh writes `'live'`; user pricing writes `'user'`. */ readonly source?: ModelCatalogSource; @@ -101,6 +103,8 @@ export interface ModelCatalogListing { readonly inputCostPerMtokMicrocents: number; readonly outputCostPerMtokMicrocents: number; readonly cachedInputCostPerMtokMicrocents: number; + /** `true` ⇒ the number above is the user's own; `false` ⇒ they never said, so a reader derives it (ADR-0071 §10). */ + readonly cachedInputStated: boolean; /** Live-discovered deprecation epoch-ms (ADR-0064 §7); `undefined` when none. */ readonly deprecationDate?: number; /** Provenance, validated at the read boundary (a foreign value degrades to `'static'`). */ @@ -167,6 +171,16 @@ export interface ModelCatalogStore { listByProvider: (providerId: string) => ModelCatalogListing[]; /** Active, non-deleted rows across every provider, ordered by model id — the cross-provider `/models` catalog. */ listAll: () => ModelCatalogListing[]; + /** + * Retire a `source='user'` pricing row (ADR-0071 §5) — `relavium models pricing --clear`. + * + * SOFT-deactivates (`is_active = 0`), like the live refresh does: `model_catalog.id` is an FK target from five + * tables, so a hard DELETE would orphan history. The active-only readers stop seeing it, so the overlay drops the + * price and the model falls back to the catalog's — which is exactly what "clear" means. + * + * Returns `false` when there was no active user row to clear (an honest no-op, never a lie). + */ + clearUserPricing: (modelId: string, providerId: string) => boolean; /** * Bulk live-upsert for one provider's discovered models ([ADR-0064] §5), in ONE transaction: each `rows` entry * is upserted as `source='live'` with `lastRefreshedAt=now` (reusing the existing (provider, model) row id so @@ -237,6 +251,7 @@ function toListing(row: ModelCatalogRow): ModelCatalogListing { inputCostPerMtokMicrocents: row.inputCostPerMtokMicrocents, outputCostPerMtokMicrocents: row.outputCostPerMtokMicrocents, cachedInputCostPerMtokMicrocents: row.cachedInputCostPerMtokMicrocents, + cachedInputStated: row.cachedInputStated, ...(row.deprecationDate === null ? {} : { deprecationDate: row.deprecationDate }), source: coerceModelCatalogSource(row.source), ...(row.lastRefreshedAt === null ? {} : { lastRefreshedAt: row.lastRefreshedAt }), @@ -514,6 +529,9 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo input.inputCostPerMtokMicrocents ?? existing?.inputCostPerMtokMicrocents ?? 0, outputCostPerMtokMicrocents: input.outputCostPerMtokMicrocents ?? existing?.outputCostPerMtokMicrocents ?? 0, + ...(input.cachedInputStated === undefined + ? {} + : { cachedInputStated: input.cachedInputStated }), cachedInputCostPerMtokMicrocents: input.cachedInputCostPerMtokMicrocents ?? existing?.cachedInputCostPerMtokMicrocents ?? @@ -583,6 +601,43 @@ export function createModelCatalogStore(db: Db, deps: ModelCatalogStoreDeps): Mo .all() .map(toListing), + clearUserPricing: (modelId, providerId) => + // `withBusyRetry` for residual cross-process contention, matching `upsert` / `replaceProviderModels`: a + // `--clear` can race a concurrent `models refresh` on the same row and hit SQLITE_BUSY. + withBusyRetry(() => { + // SOFT-deactivate, never DELETE — `model_catalog.id` is an FK target from five tables, so removing the row + // would orphan the history that references it. Deactivating is enough: every reader is active-only, so the + // overlay stops carrying the price and the model falls back to the catalog's, which is what "clear" means. + // + // RESET the user pricing columns while deactivating (review M3): `upsert` looks the row up by + // `(provider, model)` with `deletedAt IS NULL` — it ignores `isActive` — so a later partial re-price + // (`--input`/`--output`, no `--cached`) REUSES this FK-stable row and PRESERVES any column it omits. Left + // intact, a cleared `cachedInputCostPerMtokMicrocents`/`cachedInputStated` would resurrect as if the user + // had just stated it, silently billing cache-read tokens at the cleared rate. Zeroing them here means a + // reused row starts from a clean baseline, so an omitted cache rate derives from the catalog discount. + const result = db + .update(modelCatalog) + .set({ + isActive: false, + inputCostPerMtokMicrocents: 0, + outputCostPerMtokMicrocents: 0, + cachedInputCostPerMtokMicrocents: 0, + cachedInputStated: false, + updatedAt: deps.now(), + }) + .where( + and( + eq(modelCatalog.modelId, modelId), + eq(modelCatalog.providerId, providerId), + eq(modelCatalog.source, 'user'), // never touch a live/static row — those are not the user's to clear + eq(modelCatalog.isActive, true), + isNull(modelCatalog.deletedAt), + ), + ) + .run(); + return result.changes > 0; + }), + replaceProviderModels: (providerId, rows, now) => // `BEGIN IMMEDIATE` — this reads existing rows then writes, so a DEFERRED begin would hit the read→write // lock-upgrade race — plus `withBusyRetry` for residual cross-process contention (ADR-0064 amendment note). diff --git a/packages/db/src/schema.ts b/packages/db/src/schema.ts index 3bd436df..2d57a237 100644 --- a/packages/db/src/schema.ts +++ b/packages/db/src/schema.ts @@ -114,6 +114,19 @@ export const modelCatalog = sqliteTable( inputCostPerMtokMicrocents: microcents('input_cost_per_mtok_microcents'), outputCostPerMtokMicrocents: microcents('output_cost_per_mtok_microcents'), cachedInputCostPerMtokMicrocents: microcents('cached_input_cost_per_mtok_microcents'), + /** + * Did the USER actually state a cache-read rate (ADR-0071 §10)? + * + * The money column is `NOT NULL DEFAULT 0`, so it cannot tell "the user typed `--cached 0`" — a genuinely free + * cache on a self-hosted endpoint — apart from "the user never mentioned cache reads". Reading a stored `0` as + * "free" bills a whole class of tokens at nothing; reading it as "unset" discards an explicit instruction. One + * money column cannot hold both meanings, so the FACT of the statement gets its own flag. + * + * `0` ⇒ not stated: the cache rate is derived at read time from the catalog's cache DISCOUNT applied to the + * user's own input rate (never the catalog's absolute rate — a user who negotiated $0.10/MTok on a model whose + * catalog cache rate is $0.50 would otherwise pay 5× MORE for a cache hit than for a miss). + */ + cachedInputStated: boolFlag('cached_input_stated', false), // Per-modality media-OUTPUT rates (1.AF/D17, ADR-0044 §3) — integer µ¢ per billed unit (per image, // per audio-second, per video-second). NULLABLE: a model with no metered media rate degrades to 0 // (H4). The projection of `ModelPricing.mediaOutputRates`; no shipped model carries one yet. diff --git a/packages/llm/src/adapters/anthropic.test.ts b/packages/llm/src/adapters/anthropic.test.ts index 4a49bb7a..a63cb7e8 100644 --- a/packages/llm/src/adapters/anthropic.test.ts +++ b/packages/llm/src/adapters/anthropic.test.ts @@ -1,8 +1,11 @@ import Anthropic from '@anthropic-ai/sdk'; -import { describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it } from 'vitest'; +import { clearCatalogRefresh, installCatalogRefresh } from '../catalog/lookup.js'; +import { catalogModelFixture as catModel } from '../conformance/fixtures/catalog.js'; import { UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; +import { canDisableReasoning } from '../reasoning-wire.js'; import type { LlmMessage, StreamChunk } from '../types.js'; import { anthropicErrorToLlmError, @@ -63,6 +66,89 @@ function captureBody(messages: LlmMessage[]): Promise { .then(() => body); } +describe('AnthropicAdapter — request-capability + thinking/temperature gating (ADR-0071 amendment · review M4)', () => { + afterEach(clearCatalogRefresh); + const anthropicOk = (): Response => + new Response( + JSON.stringify({ + id: 'm', + type: 'message', + role: 'assistant', + model: 'x', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + stop_sequence: null, + usage: { input_tokens: 1, output_tokens: 1 }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ); + const capture = (): { + adapter: ReturnType; + sent: () => Record; + } => { + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(anthropicOk()); + }, + maxRetries: 0, + }); + return { adapter, sent: () => sent }; + }; + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + it('WITHHOLDS temperature when thinking is ENABLED — Anthropic requires temperature=1 with extended thinking (M4)', async () => { + // haiku-4-5 is budget-shaped: reasoningEffort `medium` ⇒ thinking:{type:'enabled'}. A caller temperature next to + // it is a guaranteed 400. Withhold it — the reasoning the user asked for wins; the temperature is dropped. + const { adapter, sent } = capture(); + await adapter.generate( + { + model: 'claude-haiku-4-5', + maxTokens: 8192, + messages, + reasoningEffort: 'medium', + temperature: 0.7, + }, + 'k', + ); + expect(sent()['thinking']).toMatchObject({ type: 'enabled' }); + expect(sent()).not.toHaveProperty('temperature'); + }); + + it('SENDS temperature when thinking is OFF — the reconcile is scoped to enabled thinking only', async () => { + const { adapter, sent } = capture(); + await adapter.generate( + { + model: 'claude-haiku-4-5', + maxTokens: 8192, + messages, + reasoningEffort: 'off', + temperature: 0.7, + }, + 'k', + ); + expect(sent()['thinking']).toEqual({ type: 'disabled' }); + expect(sent()['temperature']).toBe(0.7); + }); + + it('WITHHOLDS temperature for a model whose catalog rejects it (per-model capability), no thinking involved', async () => { + installCatalogRefresh({ + 'cap-claude': catModel({ + modelId: 'cap-claude', + provider: 'anthropic', + requestCapabilities: { temperature: false }, + }), + }); + const { adapter, sent } = capture(); + await adapter.generate( + { model: 'cap-claude', maxTokens: 8192, messages, temperature: 0.5 }, + 'k', + ); + expect(sent()).not.toHaveProperty('temperature'); + }); +}); + describe('AnthropicAdapter', () => { it('exposes the anthropic id and the full capability surface', () => { expect(anthropicAdapter.id).toBe('anthropic'); @@ -159,6 +245,161 @@ describe('AnthropicAdapter', () => { }); }); + /** + * THE OTHER HALF OF THE LIVE BUG (ADR-0071 §6). The reasoning field is chosen PER MODEL. + * + * `claude-haiku-4-5` publishes a token BUDGET and **no effort axis at all** — the maintainer confirmed this + * independently against Anthropic. ADR-0066 filed the budget shape as "legacy", and it is, for the industry; + * it is not legacy for one of the four Claude models we ship. The adapter sent `output_config.effort` to it + * anyway, which is a parameter that model does not take. + */ + it('a BUDGET-shaped Claude (haiku-4-5) gets thinking.budget_tokens — NOT output_config.effort', async () => { + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response( + JSON.stringify({ + id: 'm', + type: 'message', + role: 'assistant', + model: 'claude-haiku-4-5', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + stop_sequence: null, + usage: { input_tokens: 1, output_tokens: 1 }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + maxRetries: 0, + }); + const base = { + model: 'claude-haiku-4-5', // catalog: { budgetTokens: { min: 1024 } } — no `max`, no effort values + maxTokens: 8192, + messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }], + }; + + await adapter.generate({ ...base, reasoningEffort: 'high' }, 'k'); + // The ceiling is 80% of the request's own cap — NOT the whole cap. Thinking that eats every token leaves no + // ANSWER: `budget_tokens: max_tokens - 1` is accepted by Anthropic and returns one token of reply, which is + // a turn the user pays for in full and gets nothing from. floor(8192 * 0.8) = 6553. + expect(sent['thinking']).toEqual({ type: 'enabled', budget_tokens: 5171 }); // 1024 + 75% of [1024, 6553] + expect('output_config' in sent).toBe(false); // the field this model does not take is NEVER sent + + await adapter.generate({ ...base, reasoningEffort: 'low' }, 'k'); + expect(sent['thinking']).toEqual({ type: 'enabled', budget_tokens: 2406 }); // 25% of [1024, 6553] + + // `off` is the independent disable switch on BOTH shapes. + await adapter.generate({ ...base, reasoningEffort: 'off' }, 'k'); + expect(sent['thinking']).toEqual({ type: 'disabled' }); + }); + + it('NEVER sends budget_tokens >= max_tokens — Anthropic rejects it, and we were doing it', async () => { + // Found by an adversarial review, and it was real: `reasoningBudgetFor` used to return the model's FLOOR when + // the range was degenerate ("the least thinking it can do"). With `max_tokens: 256` on a model whose minimum + // budget is 1024, that put `budget_tokens: 1024` on the wire — a guaranteed 400. + // + // The honest answer is that reasoning cannot be enabled under that cap at all, so the field is WITHHELD. The + // tempting alternative — quietly raising `max_tokens` to make room — would change what the user asked for AND + // what they pay, without telling them. + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response( + JSON.stringify({ + id: 'm', + type: 'message', + role: 'assistant', + model: 'claude-haiku-4-5', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + stop_sequence: null, + usage: { input_tokens: 1, output_tokens: 1 }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + maxRetries: 0, + }); + + // The full matrix the review swept — extended, because the answer-headroom raises the floor: a cap of 1024 can + // no longer afford 1024 tokens of thinking AND a reply. The first viable cap is 1280 (floor(1280*0.8) = 1024). + for (const maxTokens of [1, 256, 512, 1000, 1024, 1279]) { + for (const tier of ['low', 'medium', 'high', 'max'] as const) { + await adapter.generate( + { + model: 'claude-haiku-4-5', + maxTokens, + messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }], + reasoningEffort: tier, + }, + 'k', + ); + expect('thinking' in sent, `maxTokens=${maxTokens} ${tier}: no valid budget exists`).toBe( + false, + ); + } + } + + // …and the first cap that CAN hold the floor AND leave room for an answer enables it. + await adapter.generate( + { + model: 'claude-haiku-4-5', + maxTokens: 1280, + messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }], + reasoningEffort: 'max', + }, + 'k', + ); + const thinking = sent['thinking'] as { type: string; budget_tokens: number }; + expect(thinking.type).toBe('enabled'); + expect(thinking.budget_tokens).toBe(1024); // the model's floor — all the headroom allows + expect(thinking.budget_tokens).toBeLessThan(1280); // the invariant Anthropic enforces + expect(1280 - thinking.budget_tokens).toBeGreaterThanOrEqual(256); // …and the ANSWER still has room + }); + + it('a model the catalog does not know gets NO reasoning field — a guess is what broke this', async () => { + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response( + JSON.stringify({ + id: 'm', + type: 'message', + role: 'assistant', + model: 'x', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + stop_sequence: null, + usage: { input_tokens: 1, output_tokens: 1 }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + maxRetries: 0, + }); + await adapter.generate( + { + model: 'some-custom-endpoint-model', + maxTokens: 1024, + messages: [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }], + reasoningEffort: 'high', + }, + 'k', + ); + expect('thinking' in sent).toBe(false); + expect('output_config' in sent).toBe(false); + }); + it('maps the reasoning-effort tier to output_config.effort + adaptive thinking; off disables; unset omits (ADR-0066)', async () => { let sent: Record = {}; const adapter = createAnthropicAdapter({ @@ -203,6 +444,101 @@ describe('AnthropicAdapter', () => { expect('output_config' in sent).toBe(false); // off never sets output_config await adapter.generate({ ...base }, 'k'); // unset ⇒ no thinking, no output_config (provider default) expect('thinking' in sent).toBe(false); + + // A tier the LADDER omits is served from the model's OTHER axis, not dropped. `claude-opus-4-5` publishes + // ['low','medium','high'] AND `budgetTokens: {min: 1024}`; `max` is not an effort level it takes, so it goes + // out as a budget. Reading the two axes as mutually exclusive would have withheld reasoning entirely from a + // model that serves the tier perfectly well — including on the failover path, where the rescue turn would + // then run with no reasoning at all. + await adapter.generate( + { ...base, model: 'claude-opus-4-5', maxTokens: 8192, reasoningEffort: 'max' }, + 'k', + ); + expect('output_config' in sent).toBe(false); + expect(sent['thinking']).toEqual({ type: 'enabled', budget_tokens: 6553 }); // 80% of 8192 + }); + + it('CLAMPS max_tokens to the model ceiling — and the thinking budget is carved out of the CLAMPED cap', async () => { + // ADR-0071 §7. Two facts have to hold TOGETHER, and holding only one of them is a 400: + // 1. `max_tokens: 200000` on claude-opus-4-5 (ceiling 64_000) is rejected outright. + // 2. Anthropic ALSO requires `budget_tokens < max_tokens`. The thinking ceiling is derived from the cap, so + // clamping the cap and NOT the derivation would put the budget ABOVE the max_tokens we actually send — + // trading one 400 for another. The cap is computed once, and both uses read that one value. + // + // Driven through STREAM because the Anthropic SDK refuses a non-streaming request whose `max_tokens` implies a + // >10-minute generation — which a 64 000-token cap does. Same `buildCommonBody`, same body on the wire. + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response('event: message_stop\ndata: {"type":"message_stop"}\n\n', { + status: 200, + headers: { 'content-type': 'text/event-stream' }, + }), + ); + }, + maxRetries: 0, + }); + + await collect( + adapter.stream( + { + model: 'claude-opus-4-5', + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + maxTokens: 200_000, + reasoningEffort: 'max', // NOT on opus-4-5's ladder ⇒ served from its budget axis + }, + 'k', + ), + ); + + expect(sent['max_tokens']).toBe(64_000); // clamped to the model's real ceiling + const thinking = sent['thinking'] as { type: string; budget_tokens: number }; + expect(thinking.type).toBe('enabled'); + expect(thinking.budget_tokens).toBe(51_200); // 80% of the CLAMPED cap — not of the 200 000 that was asked for + expect(thinking.budget_tokens).toBeLessThan(sent['max_tokens'] as number); // Anthropic's own invariant + }); + + it('an EMPTY descriptor is NOT disable-able — `thinking: {disabled}` is still a field, and still a 400', async () => { + // The `off` branch answered a question about the PROVIDER ("Anthropic can always disable") instead of about the + // MODEL. A model whose descriptor is `{}` reasons but publishes no knob at all; sending it a disable is the same + // guess, in the opposite direction, as sending it an effort level. The picker offers `off` for no such model, + // and now neither does the wire. + let sent: Record = {}; + const adapter = createAnthropicAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve( + new Response( + JSON.stringify({ + id: 'm', + type: 'message', + role: 'assistant', + model: 'x', + content: [{ type: 'text', text: 'ok' }], + stop_reason: 'end_turn', + stop_sequence: null, + usage: { input_tokens: 1, output_tokens: 1 }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ); + }, + maxRetries: 0, + }); + expect(canDisableReasoning('anthropic', {})).toBe(false); // the predicate the branch now asks + await adapter.generate( + { + model: 'claude-opus-4-8', + maxTokens: 1024, + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + reasoningEffort: 'off', + }, + 'k', + ); + // …and the real catalog model, which DOES publish a knob, still disables — the guard is a filter, not a mute. + expect(sent['thinking']).toEqual({ type: 'disabled' }); }); it('rejects handle and url media sources with an explicit bad_request error (1.AF)', async () => { diff --git a/packages/llm/src/adapters/anthropic.ts b/packages/llm/src/adapters/anthropic.ts index 6f78eec4..6baac389 100644 --- a/packages/llm/src/adapters/anthropic.ts +++ b/packages/llm/src/adapters/anthropic.ts @@ -1,10 +1,19 @@ import Anthropic from '@anthropic-ai/sdk'; import { mediaModalityOf } from '@relavium/shared'; -import type { AbortSignalLike, ContentPart, ReasoningEffort, StopReason } from '@relavium/shared'; +import type { AbortSignalLike, ContentPart, StopReason } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; +import { catalogModel, modelAccepts } from '../catalog/lookup.js'; +import { cappedMaxTokens } from '../output-cap.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; +import { + ANTHROPIC_WIRE, + acceptedWireValue, + canDisableReasoning, + reasoningBudgetFor, + thinkingCeiling, +} from '../reasoning-wire.js'; import { normalizeToolCall, toWire } from '../tool-normalizer.js'; import type { CapabilityFlags, @@ -46,12 +55,8 @@ const PROVIDER = 'anthropic'; const DEFAULT_MAX_TOKENS = 4096; /** Anthropic's API caps `temperature` at 1 (the shared contract's envelope is the wider [0, 2]). */ const MAX_TEMPERATURE = 1; -/** ADR-0066: the normalized reasoning-effort tier → Anthropic's native `output_config.effort` levels. Anthropic has - * a native `max`, so all four non-`off` tiers map 1:1; `off` is handled separately (thinking disabled). */ -const ANTHROPIC_REASONING_EFFORT: Record< - Exclude, - 'low' | 'medium' | 'high' | 'max' -> = { low: 'low', medium: 'medium', high: 'high', max: 'max' }; +// The tier → wire map moved to `reasoning-wire.ts` (ADR-0071 §6): `acceptedTiers` must compose it with the +// catalog's per-model values, and two copies of "what we send a provider" are two chances to disagree. /** * Anthropic supports the full common-path surface; provider-specific features go via @@ -460,13 +465,87 @@ function toAnthropicTool(toolDef: ToolDef): Anthropic.Tool { return tool; } +/** + * ADR-0066/0071: map the normalized reasoning-effort tier onto Anthropic's PER-MODEL control — it has BOTH shapes + * in play, and the shipped adapter sent `output_config.effort` to both. + * + * • effort-shaped (`claude-opus-4-8`, …) → `output_config.effort` + ADAPTIVE thinking. + * • budget-shaped (`claude-haiku-4-5` — NO effort axis at all) → the legacy `thinking.budget_tokens`. Legacy for + * the industry, not for haiku: one of the four Claude models we ship, and it publishes a budget and no ladder. + * + * `off` is neither shape: `thinking: {type:'disabled'}`, an independent switch that works on both. A model the + * catalog cannot describe (a custom endpoint) gets NO reasoning field, `off` included — `thinking:{disabled}` is + * still a field, and still a 400 on a model with no reasoning surface. A guess is what put a rejected value on the + * wire in the first place. `maxTokens` is the CLAMPED cap already on `body.max_tokens`, so a derived budget stays + * under it (Anthropic rejects `budget_tokens >= max_tokens`). + */ +function applyAnthropicReasoning( + body: Omit, + req: LlmRequest, + maxTokens: number, +): void { + if (req.reasoningEffort === undefined) return; + const controls = catalogModel(req.model)?.reasoning; + if (controls === undefined) return; // unknown/custom model — withhold every reasoning field, `off` included + + if (req.reasoningEffort === 'off') { + // `canDisableReasoning`, not a bare `true`. An EMPTY descriptor (`{}`) means the model reasons but publishes no + // knob at all — `thinking:{disabled}` is still a field, and still a 400 on a model with no reasoning surface to + // switch. Asking the same predicate the picker asks is what keeps the two in step. + if (canDisableReasoning('anthropic', controls)) { + body.thinking = { type: 'disabled' }; + } + return; + } + + if (acceptedWireValue('anthropic', req.reasoningEffort, controls) !== undefined) { + // MEMBERSHIP, not presence. `claude-opus-4-5` publishes ['low','medium','high'] — no `max` — and the old branch + // tested only that an effort axis EXISTED, then sent `effort: 'max'` anyway. It reaches the wire on a FAILOVER, + // where the chain re-points a request at a weaker model. + body.thinking = { type: 'adaptive' }; + body.output_config = { + ...body.output_config, + // The effort level MERGES alongside any structured-output `format` already on output_config. + effort: ANTHROPIC_WIRE[req.reasoningEffort], + }; + return; + } + + if (controls.budgetTokens !== undefined) { + // The BUDGET shape — and also the fallback when a model's effort axis does not contain THIS tier. + // `claude-opus-4-5` publishes both, so a tier it cannot express as an effort level is still expressible as a + // budget. Anthropic requires `budget_tokens < max_tokens`, and a budget that eats the whole cap leaves no + // answer — so the ceiling reserves room for the reply (see THINKING_BUDGET_SHARE). + const budget = reasoningBudgetFor( + req.reasoningEffort, + controls.budgetTokens, + thinkingCeiling(maxTokens), // the CLAMPED cap — the one actually on the wire + ); + // `undefined` ⇒ the model's MINIMUM budget does not fit under this request's `max_tokens` (haiku's floor is + // 1024; a request capped at 256 has none). Withhold rather than send a value the API rejects — and rather than + // quietly raising `max_tokens` to make room, which would change both what the user asked for and what they pay. + if (budget !== undefined) { + body.thinking = { type: 'enabled', budget_tokens: budget }; + } + } + // A model that publishes NO usable control gets the reasoning field WITHHELD. +} + /** The shared request body (everything except the `stream` discriminant each method sets). */ function buildCommonBody( req: LlmRequest, ): Omit { + // The output cap, held at or below the model's own ceiling (ADR-0071 §7). Anthropic REQUIRES `max_tokens`, so an + // absent one defaults — and the default is clamped too, in case a model's ceiling is ever below it. + // + // This value is also the ceiling the thinking budget is derived from, a few lines down. Clamping here and not + // there would put `budget_tokens` above the `max_tokens` we actually send, which Anthropic rejects outright — + // so it is computed ONCE and both uses read it. + const maxTokens = + cappedMaxTokens(req.maxTokens ?? DEFAULT_MAX_TOKENS, req.model) ?? DEFAULT_MAX_TOKENS; const body: Omit = { model: req.model, - max_tokens: req.maxTokens ?? DEFAULT_MAX_TOKENS, + max_tokens: maxTokens, messages: mergeAdjacentSameRole(req.messages.map(toAnthropicMessage)), }; if (req.system !== undefined) { @@ -478,28 +557,24 @@ function buildCommonBody( if (req.toolChoice !== undefined) { body.tool_choice = toAnthropicToolChoice(req.toolChoice); } - if (req.responseFormat?.type === 'json') { + // `structured_output` is gated on the MODEL's per-model capability (ADR-0071 amendment): a model can reject a + // response-format request its provider supports, and sending it is a 400. Withhold, never send-and-fail. + if (req.responseFormat?.type === 'json' && modelAccepts(req.model, 'structuredOutput')) { // Native structured output via output_config (ADR-0030); the canonical JSON-Schema bridges here. body.output_config = { format: { type: 'json_schema', schema: req.responseFormat.schema as Record }, }; } - if (req.reasoningEffort !== undefined) { - // ADR-0066: Anthropic's tier-native reasoning control — `output_config.effort` (the level) + ADAPTIVE thinking - // to enable it (no token budget: the tier-native path avoids the legacy budget_tokens constraint). `off` DISABLES - // thinking. The effort level MERGES alongside any structured-output `format` already on output_config. Anthropic - // has a native `max` tier, so all five normalized tiers map 1:1 (no coarsening here). - if (req.reasoningEffort === 'off') { - body.thinking = { type: 'disabled' }; - } else { - body.thinking = { type: 'adaptive' }; - body.output_config = { - ...body.output_config, - effort: ANTHROPIC_REASONING_EFFORT[req.reasoningEffort], - }; - } - } - if (req.temperature !== undefined) { + applyAnthropicReasoning(body, req, maxTokens); + // Extended thinking pins `temperature` to 1 on Anthropic — `thinking:{enabled|adaptive}` alongside any other + // temperature is a guaranteed 400 (review M4). `applyAnthropicReasoning` (above) just set `body.thinking`, so read + // it here: a non-`disabled` thinking block means reasoning is ON, and the caller's temperature must be WITHHELD. + const thinkingEnabled = body.thinking !== undefined && body.thinking.type !== 'disabled'; + if ( + req.temperature !== undefined && + modelAccepts(req.model, 'temperature') && // the per-model capability (gpt-class parity; ADR-0071 amendment) + !thinkingEnabled + ) { // The shared contract is the provider-agnostic [0, 2] envelope (common.ts); Anthropic's API // accepts temperature in [0, 1]. Fail fast (the adapter's "never silently drop" posture) rather // than forward a value the provider will 400 on — the guard stays provider-local, contract diff --git a/packages/llm/src/adapters/gemini.test.ts b/packages/llm/src/adapters/gemini.test.ts index 48c9ceda..4198a683 100644 --- a/packages/llm/src/adapters/gemini.test.ts +++ b/packages/llm/src/adapters/gemini.test.ts @@ -1,6 +1,9 @@ -import { describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it } from 'vitest'; -import type { AbortSignalLike } from '@relavium/shared'; +import { clearCatalogRefresh, installCatalogRefresh } from '../catalog/lookup.js'; +import { catalogModelFixture } from '../conformance/fixtures/catalog.js'; + +import type { AbortSignalLike, ReasoningEffort } from '@relavium/shared'; import { UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; @@ -417,6 +420,48 @@ describe('geminiErrorToLlmError — classification', () => { }); }); +describe('Gemini adapter — per-model request-capability gating (ADR-0071 amendment)', () => { + afterEach(clearCatalogRefresh); + const catModel = catalogModelFixture; // the shared fixture; each row below pins `provider: 'gemini'` + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'hi' }] }]; + + it('WITHHOLDS temperature for a model that rejects it, SENDS it when accepted', () => { + installCatalogRefresh({ + 'cap-gemini': catModel({ + modelId: 'cap-gemini', + provider: 'gemini', + requestCapabilities: { temperature: false }, + }), + 'cap-gemini-ok': catModel({ modelId: 'cap-gemini-ok', provider: 'gemini' }), + }); + expect( + buildGeminiRequest({ model: 'cap-gemini', temperature: 0.4, messages }).config, + ).not.toHaveProperty('temperature'); + expect( + buildGeminiRequest({ model: 'cap-gemini-ok', temperature: 0.4, messages }).config[ + 'temperature' + ], + ).toBe(0.4); + }); + + it('WITHHOLDS structured output (responseJsonSchema) for a model that rejects it', () => { + installCatalogRefresh({ + 'cap-gemini-so': catModel({ + modelId: 'cap-gemini-so', + provider: 'gemini', + requestCapabilities: { structuredOutput: false }, + }), + }); + const config = buildGeminiRequest({ + model: 'cap-gemini-so', + messages, + responseFormat: { type: 'json', schema: { type: 'object' } }, + }).config; + expect(config).not.toHaveProperty('responseJsonSchema'); + expect(config).not.toHaveProperty('responseMimeType'); + }); +}); + describe('Gemini adapter — request building (buildGeminiRequest)', () => { it('routes system → systemInstruction, tools → functionDeclarations, and tool choice modes', () => { const request = buildGeminiRequest({ @@ -477,56 +522,133 @@ describe('Gemini adapter — request building (buildGeminiRequest)', () => { }); }); - it('maps the reasoning-effort tier to thinkingConfig.thinkingLevel + includeThoughts on a thinking tier (ADR-0066)', () => { - // A non-off tier also sets includeThoughts:true so raising effort SURFACES the reasoning it bills for (the only - // switch that returns Gemini thought parts). All five tiers assert thinkingLevel; medium is the picker default. - expect( - buildGeminiRequest({ ...REQ, reasoningEffort: 'high' }).config['thinkingConfig'], - ).toEqual({ - thinkingLevel: 'HIGH', - includeThoughts: true, + /** + * THE LIVE BUG, and its fix (ADR-0071 §6). The reasoning field is chosen PER MODEL, not per adapter. + * + * The test these replace asserted `thinkingLevel` for every tier on `gemini-2.5-flash` — and its own comment + * said "a Pro model rejects budget 0", so the author already suspected the shape was not universal. It is not: + * Google's docs for the `generateContent` API this adapter calls state that **Gemini 2.5 does not support + * `thinkingLevel` at all** and takes `thinkingBudget` instead. `gemini-2.5-flash` and `gemini-2.5-pro` are the + * only two Gemini rows we ship, so `/effort` on Gemini has been sending a parameter the models do not take. + */ + it('a BUDGET-shaped model (gemini-2.5-*) gets thinkingBudget — NOT thinkingLevel', () => { + // Catalog: gemini-2.5-flash → { toggle, budgetTokens: { min: 0, max: 24576 } }. No effort axis. + const built = (effort: ReasoningEffort): unknown => + buildGeminiRequest({ ...REQ, reasoningEffort: effort, maxTokens: 8192 }).config[ + 'thinkingConfig' + ]; + + // The tiers scale across the range under a ceiling that RESERVES ROOM FOR THE ANSWER: floor(8192 * 0.8) = 6553, + // not the whole 8192. `max` used to hand the entire output cap to the thoughts, so the model thought to the + // limit and then had nothing left to reply with — a request the API happily accepts and that returns no answer. + expect(built('low')).toEqual({ thinkingBudget: 1638, includeThoughts: true }); // 25% of [0, 6553] + expect(built('high')).toEqual({ thinkingBudget: 4915, includeThoughts: true }); // 75% + expect(built('max')).toEqual({ thinkingBudget: 6553, includeThoughts: true }); // the ceiling, not maxTokens + // `off` on Gemini is `thinkingBudget: 0` — the real disable — never MINIMAL, which still thinks and still bills. + expect(built('off')).toEqual({ thinkingBudget: 0 }); + expect('thinkingConfig' in buildGeminiRequest(REQ).config).toBe(false); // unset ⇒ omitted + }); + + it('CLAMPS maxOutputTokens to the model ceiling — and the thinking budget follows the CLAMPED cap', () => { + // ADR-0071 §7. `gemini-2.5-pro`'s ceiling is 65_536; an authored 200_000 is a 400 on every turn. + // + // The coupling is the sharp edge: the thinking budget is carved OUT of the output cap, so clamping the cap and + // deriving the budget from the RAW one would hand the model a budget larger than the cap we actually send. + const request = buildGeminiRequest({ + ...REQ, + model: 'gemini-2.5-pro', + maxTokens: 200_000, + reasoningEffort: 'max', }); - // Gemini tops out at HIGH — `max` coarsens to it (no separate xhigh/max tier). - expect(buildGeminiRequest({ ...REQ, reasoningEffort: 'max' }).config['thinkingConfig']).toEqual( - { - thinkingLevel: 'HIGH', - includeThoughts: true, - }, - ); + expect(request.config['maxOutputTokens']).toBe(65_536); // clamped to the model's real ceiling + const thinking = request.config['thinkingConfig'] as { thinkingBudget: number }; + // 80% of the CLAMPED cap (52_428), not of the 200 000 asked for — and capped by the model's own budget max. + expect(thinking.thinkingBudget).toBeLessThanOrEqual(52_429); + expect(thinking.thinkingBudget).toBeLessThan(65_536); // the answer keeps room, which is the whole point + }); + + it("leaves a cap BELOW the ceiling alone — the author's budget is not a mistake to correct", () => { + const request = buildGeminiRequest({ ...REQ, model: 'gemini-2.5-pro', maxTokens: 4_096 }); + expect(request.config['maxOutputTokens']).toBe(4_096); + }); + + it('a TOGGLE model with a non-zero floor can still be turned OFF — picker and wire agree', () => { + // gemini-2.5-flash-lite publishes BOTH a toggle and `budgetTokens: { min: 512, … }`. The picker offers `off` + // (a toggle IS a disable switch); the adapter used to test only `min === 0` and silently withhold the field — + // so the user turned reasoning off, was billed for it anyway, and nothing told them. Both sides now ask the + // one predicate, {@link canDisableReasoning}, so they cannot drift apart again. expect( - buildGeminiRequest({ ...REQ, reasoningEffort: 'medium' }).config['thinkingConfig'], - ).toEqual({ - thinkingLevel: 'MEDIUM', - includeThoughts: true, + buildGeminiRequest({ + ...REQ, + model: 'gemini-2.5-flash-lite', + reasoningEffort: 'off', + maxTokens: 8192, + }).config['thinkingConfig'], + ).toEqual({ thinkingBudget: 0 }); + }); + + it('withholds the budget when even the model floor will not fit under the cap', () => { + // gemini-2.5-pro's floor is 128 thought tokens. A 64-token answer cap leaves a ceiling of floor(64 * 0.8) = 51, + // under the floor — no budget in the range is sendable, so the field is omitted and the model uses its default, + // rather than us putting a value on the wire that the API will reject outright. + const request = buildGeminiRequest({ + ...REQ, + model: 'gemini-2.5-pro', + reasoningEffort: 'low', + maxTokens: 64, }); - expect(buildGeminiRequest({ ...REQ, reasoningEffort: 'low' }).config['thinkingConfig']).toEqual( + expect('thinkingConfig' in request.config).toBe(false); + }); + + it('an EFFORT-shaped model (gemini-3.x) gets thinkingLevel — the shape follows the model', () => { + const req: LlmRequest = { ...REQ, model: 'gemini-3.5-flash' }; // catalog: effortValues + expect( + buildGeminiRequest({ ...req, reasoningEffort: 'high' }).config['thinkingConfig'], + ).toEqual({ thinkingLevel: 'HIGH', includeThoughts: true }); + // Gemini's ladder stops at HIGH — `max` coarsens onto it, honestly. + expect(buildGeminiRequest({ ...req, reasoningEffort: 'max' }).config['thinkingConfig']).toEqual( { - thinkingLevel: 'LOW', + thinkingLevel: 'HIGH', includeThoughts: true, }, ); - // Gemini has no universal disable (a Pro model rejects budget 0) — `off` degrades to the lowest tier and does - // NOT force thought output on (minimal thinking). - expect(buildGeminiRequest({ ...REQ, reasoningEffort: 'off' }).config['thinkingConfig']).toEqual( - { - thinkingLevel: 'MINIMAL', - }, - ); - expect('thinkingConfig' in buildGeminiRequest(REQ).config).toBe(false); // unset ⇒ omitted (provider default) }); - it('deep-merges the tier onto a caller providerOptions.thinkingConfig — sibling keys survive (ADR-0066)', () => { - // A caller who enabled thought output + a budget must not lose them when effort is also set: the canonical - // thinkingLevel wins on its one key, includeThoughts:false is respected, and thinkingBudget survives. + it('gemini-2.5-pro CANNOT be turned off — the field is WITHHELD, never downgraded to MINIMAL', () => { + // Google: "N/A: Cannot disable thinking". Catalog: budgetTokens.min = 128. `acceptedTiers` never offers `off` + // for it, and if one arrives anyway the adapter withholds rather than substituting a value that neither + // disables thinking nor is one the model takes. Silently billing a user for reasoning they switched OFF is + // the worst reading of this bug, and it is the one the old code shipped. + const built = buildGeminiRequest({ + ...REQ, + model: 'gemini-2.5-pro', + reasoningEffort: 'off', + }).config; + expect('thinkingConfig' in built).toBe(false); + }); + + it('a model the catalog does not know gets NO reasoning field — a guess is what broke this', () => { + const built = buildGeminiRequest({ + ...REQ, + model: 'some-custom-endpoint-model', + reasoningEffort: 'high', + }).config; + expect('thinkingConfig' in built).toBe(false); + }); + + it('deep-merges onto a caller providerOptions.thinkingConfig — sibling keys survive (ADR-0066)', () => { + // A caller who set thought output must not lose it when effort is also set: the canonical key wins on ITS + // key, and a non-colliding sibling survives. const built = buildGeminiRequest({ ...REQ, + model: 'gemini-3.5-flash', // effort-shaped, so `thinkingLevel` is the canonical key here reasoningEffort: 'high', - providerOptions: { thinkingConfig: { includeThoughts: false, thinkingBudget: 2048 } }, + providerOptions: { thinkingConfig: { includeThoughts: false, topK: 5 } }, }); expect(built.config['thinkingConfig']).toEqual({ thinkingLevel: 'HIGH', // canonical wins on this key includeThoughts: false, // the caller's explicit choice is NOT overridden - thinkingBudget: 2048, // a non-colliding sibling survives + topK: 5, // a non-colliding sibling survives }); }); diff --git a/packages/llm/src/adapters/gemini.ts b/packages/llm/src/adapters/gemini.ts index 4424818c..a8444769 100644 --- a/packages/llm/src/adapters/gemini.ts +++ b/packages/llm/src/adapters/gemini.ts @@ -10,7 +10,17 @@ import type { } from '@relavium/shared'; import { assertStreamable, assertSupported } from '../capabilities.js'; +import { catalogModel, modelAccepts } from '../catalog/lookup.js'; +import { cappedMaxTokens } from '../output-cap.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; +import { + GEMINI_WIRE, + acceptedWireValue, + canDisableReasoning, + reasoningBudgetFor, + thinkingCeiling, + toGeminiThinkingLevel, +} from '../reasoning-wire.js'; import { GeminiToolCallIds, normalizeToolCall, toWire } from '../tool-normalizer.js'; import { UnsupportedCapabilityError } from '../errors.js'; import type { @@ -57,16 +67,6 @@ import { */ const PROVIDER = 'gemini'; -/** ADR-0066: the normalized reasoning-effort tier → Gemini's native `thinkingConfig.thinkingLevel` enum values. - * Gemini tops out at HIGH, so `max`→HIGH (a coarsening); it has no universal disable (a Pro model rejects budget - * 0), so `off` degrades to the lowest tier MINIMAL. The loose config Record takes the enum's string value directly. */ -const GEMINI_THINKING_LEVEL: Record = { - off: 'MINIMAL', - low: 'LOW', - medium: 'MEDIUM', - high: 'HIGH', - max: 'HIGH', -}; /** * Gemini's common-path capability surface (restricted tool schema; ids synthesized). 1.AE wires @@ -569,18 +569,91 @@ const GEMINI_RESPONSE_MODALITY: Record = { */ function buildThinkingConfig( reasoningEffort: ReasoningEffort, + model: string, + maxTokens: number | undefined, providerOptions: LlmRequest['providerOptions'], -): Record { +): Record | undefined { const poThinking: Record = providerOptions !== undefined && isRecord(providerOptions['thinkingConfig']) ? providerOptions['thinkingConfig'] : {}; const surfaceThoughts = reasoningEffort !== 'off' && poThinking['includeThoughts'] === undefined; - return { + const withThoughts = (config: Record): Record => ({ ...poThinking, - thinkingLevel: GEMINI_THINKING_LEVEL[reasoningEffort], + ...config, ...(surfaceThoughts ? { includeThoughts: true } : {}), - }; + }); + + const entry = catalogModel(model); + const controls = entry?.reasoning; + + // THE MODEL DECIDES THE FIELD. `gemini-2.5-*` take `thinkingBudget`; `gemini-3.x` take `thinkingLevel`. The + // shipped adapter sent `thinkingLevel` to every reasoning model, and our only two shipped Gemini rows are 2.5 — + // so `/effort` on Gemini has been sending a parameter Google's docs say those models do not support. + if ( + controls !== undefined && + reasoningEffort !== 'off' && + acceptedWireValue('gemini', reasoningEffort, controls) !== undefined + ) { + // MEMBERSHIP, not presence. `gemini-3-pro-preview` publishes ['low','high'] — no `medium` — and the old branch + // tested only that an effort axis EXISTED, then sent `thinkingLevel: 'MEDIUM'` anyway. + return withThoughts({ thinkingLevel: toGeminiThinkingLevel(GEMINI_WIRE[reasoningEffort]) }); + } + + if (reasoningEffort === 'off') { + // `off` is NOT a thinkingLevel — MINIMAL is the *lowest* level, not an off switch, and a model set to it still + // thinks, and still bills for it. Disabling on Gemini is `thinkingBudget: 0`, available exactly when + // {@link canDisableReasoning} says it is: a published `toggle`, or a budget range whose floor IS zero. + // + // Asking the one predicate is what keeps the picker and the wire in agreement. They used to disagree, and the + // divergence cost real money: `gemini-2.5-flash-lite` publishes a toggle AND `min: 512`, so `acceptedTiers` + // OFFERED `off` while this branch — which looked only at `min === 0` — silently withheld it. The user switched + // reasoning off, was billed for it anyway, and nothing said so. `gemini-2.5-pro` (floor 128, no toggle) truly + // cannot be disabled: neither the picker nor this branch will claim otherwise. + return controls !== undefined && canDisableReasoning('gemini', controls) + ? withThoughts({ thinkingBudget: 0 }) + : undefined; + } + + if (entry !== undefined && controls?.budgetTokens !== undefined) { + // The BUDGET shape — also where an effort-shaped model lands when its ladder does not contain THIS tier. + // + // The ceiling reserves room for the ANSWER. Spending the whole output cap on thoughts is accepted by the API + // and useless: the model thinks to the limit, then has nothing left to reply with. The request's own cap wins + // when it has one; otherwise the model's published output ceiling stands in for it. + const cap = maxTokens ?? entry.maxOutputTokens; + const budget = reasoningBudgetFor(reasoningEffort, controls.budgetTokens, thinkingCeiling(cap)); + // `undefined` ⇒ even the model's minimum budget does not fit under this cap. Withhold rather than send a value + // the API will reject. + return budget === undefined ? undefined : withThoughts({ thinkingBudget: budget }); + } + + // The model reasons but publishes no control (or is not in the catalog at all — a custom endpoint). Withhold + // the field: a guess here is exactly what put a rejected value on the wire in the first place. + return undefined; +} + +/** + * Apply the per-model thinking control onto the config (ADR-0066/0071) — `thinkingLevel` for Gemini 3, + * `thinkingBudget` for Gemini 2.5, chosen PER MODEL in {@link buildThinkingConfig}. `undefined` there means the + * model exposes no control we can prove, so the field is WITHHELD entirely rather than guessed at (and any caller + * `thinkingConfig` still stands). Extracted from {@link buildGeminiRequest} to keep that lowering flat. + */ +function applyThinkingConfig( + config: Record, + req: LlmRequest, + maxOutputTokens: number | undefined, +): void { + if (req.reasoningEffort === undefined) return; + const thinkingConfig = buildThinkingConfig( + req.reasoningEffort, + req.model, + maxOutputTokens, // the CLAMPED cap — the thinking budget must be carved out of what we actually send + req.providerOptions, + ); + if (thinkingConfig !== undefined) { + config['thinkingConfig'] = thinkingConfig; + } } /** Lower a canonical request into the Gemini request shape (system → `systemInstruction`, etc.). */ @@ -595,21 +668,23 @@ export function buildGeminiRequest(req: LlmRequest): GeminiRequest { if (req.toolChoice !== undefined) { config['toolConfig'] = toGeminiToolChoice(req.toolChoice); } - if (req.responseFormat?.type === 'json') { + // Gate `structured_output`/`temperature` on the MODEL's per-model capability (ADR-0071 amendment) — a model can + // reject a parameter its provider supports, and sending it is a 400. Withhold, don't send-and-fail; absent ⇒ ok. + if (req.responseFormat?.type === 'json' && modelAccepts(req.model, 'structuredOutput')) { // Native structured output (ADR-0030): JSON mime type + the canonical schema as responseJsonSchema. config['responseMimeType'] = 'application/json'; config['responseJsonSchema'] = req.responseFormat.schema; } - if (req.temperature !== undefined) { + if (req.temperature !== undefined && modelAccepts(req.model, 'temperature')) { config['temperature'] = req.temperature; } - if (req.maxTokens !== undefined) { - config['maxOutputTokens'] = req.maxTokens; - } - if (req.reasoningEffort !== undefined) { - // ADR-0066: Gemini's tier-native thinking control — see {@link buildThinkingConfig} for the deep-merge rationale. - config['thinkingConfig'] = buildThinkingConfig(req.reasoningEffort, req.providerOptions); + // The output cap, held at or below the model's own ceiling (ADR-0071 §7) — down, never up: a cap BELOW the + // ceiling is the author's deliberate budget, and raising it would spend their money for them. + const maxOutputTokens = cappedMaxTokens(req.maxTokens, req.model); + if (maxOutputTokens !== undefined) { + config['maxOutputTokens'] = maxOutputTokens; } + applyThinkingConfig(config, req, maxOutputTokens); if (req.outputModalities !== undefined && req.outputModalities.some((m) => m !== 'text')) { // Lower the node's non-text output_modalities to Gemini `responseModalities` (inline media-out, // 1.AG/ADR-0046). The per-modality capability gate (assertMediaCapabilities) has already rejected an diff --git a/packages/llm/src/adapters/openai.test.ts b/packages/llm/src/adapters/openai.test.ts index 72bed473..a0ffa64e 100644 --- a/packages/llm/src/adapters/openai.test.ts +++ b/packages/llm/src/adapters/openai.test.ts @@ -1,8 +1,10 @@ import { APIConnectionError, APIConnectionTimeoutError, APIError, APIUserAbortError } from 'openai'; -import { describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it } from 'vitest'; import type { AbortSignalLike, ReasoningEffort } from '@relavium/shared'; +import { clearCatalogRefresh, installCatalogRefresh } from '../catalog/lookup.js'; +import { catalogModelFixture as catModel } from '../conformance/fixtures/catalog.js'; import { InvalidBaseUrlError, UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError } from '../llm-error.js'; import { @@ -89,6 +91,52 @@ const okResponse = (): Response => headers: { 'content-type': 'application/json' }, }); +describe('per-model request-capability gating (ADR-0071 amendment)', () => { + afterEach(clearCatalogRefresh); + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + const capture = (): { oai: LlmProvider; sent: () => Record } => { + let sent: Record = {}; + const oai = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + return { oai, sent: () => sent }; + }; + + it('WITHHOLDS temperature for a model that rejects it — the 400 becomes a dropped field', async () => { + installCatalogRefresh({ + 'cap-openai': catModel({ + modelId: 'cap-openai', + requestCapabilities: { temperature: false }, + }), + }); + const { oai, sent } = capture(); + await oai.generate({ model: 'cap-openai', messages, temperature: 0.7 }, 'k'); + expect(sent()).not.toHaveProperty('temperature'); + }); + + it('SENDS temperature for a model with no capability data (absent ⇒ accepted, unchanged behaviour)', async () => { + installCatalogRefresh({ 'cap-ok': catModel({ modelId: 'cap-ok' }) }); + const { oai, sent } = capture(); + await oai.generate({ model: 'cap-ok', messages, temperature: 0.7 }, 'k'); + expect(sent()['temperature']).toBe(0.7); + }); + + it('WITHHOLDS response_format for a model that rejects structured_output', async () => { + installCatalogRefresh({ + 'cap-so': catModel({ modelId: 'cap-so', requestCapabilities: { structuredOutput: false } }), + }); + const { oai, sent } = capture(); + await oai.generate( + { model: 'cap-so', messages, responseFormat: { type: 'json', schema: { type: 'object' } } }, + 'k', + ); + expect(sent()).not.toHaveProperty('response_format'); + }); +}); + /** Build an SSE Response from a list of chunk objects (shared across the streaming describes). */ const sse = (chunks: readonly unknown[]): Response => new Response(chunks.map((c) => `data: ${JSON.stringify(c)}\n\n`).join('') + 'data: [DONE]\n\n', { @@ -231,7 +279,266 @@ describe('OpenAI-compatible adapter', () => { }); }); - it('maps the reasoning-effort tier per provider: OpenAI reasoning_effort (max→xhigh, off→none) + DeepSeek thinking (ADR-0066)', async () => { + it('an UNKNOWN model gets NO reasoning field — on the OpenAI arm AND the DeepSeek arm', async () => { + // Found by an adversarial review. Fixing Gemini and Anthropic left these two sending the field unconditionally: + // an unknown model (a custom `base_url`, or one so new we have no metadata) was handed `reasoning_effort` / + // `thinking` regardless. The host's gate already withholds — but `@relavium/llm` is a public SEAM, and it must + // not depend on a caller having run the gate. Guessing at a model we cannot describe is the whole bug class. + let sent: Record = {}; + const capture = (): Response => { + return okResponse(); + }; + const oai = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(capture()); + }, + }); + const deepseek = createOpenAiAdapter({ + providerId: 'deepseek', + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(capture()); + }, + }); + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + for (const tier of ['off', 'low', 'high', 'max'] as const) { + await oai.generate( + { model: 'some-custom-endpoint-model', messages, reasoningEffort: tier }, + 'k', + ); + expect('reasoning_effort' in sent, `openai ${tier}`).toBe(false); + + await deepseek.generate( + { model: 'some-custom-endpoint-model', messages, reasoningEffort: tier }, + 'k', + ); + expect('thinking' in sent, `deepseek ${tier}`).toBe(false); + } + + // …and a model the catalog DOES know still gets it, so the guard is a filter and not a mute button. + await oai.generate({ model: 'gpt-5.5', messages, reasoningEffort: 'high' }, 'k'); + expect(sent['reasoning_effort']).toBe('high'); + }); + + it('THE DIALECT: official OpenAI gets `max_completion_tokens`; DeepSeek and a custom base_url keep `max_tokens`', async () => { + // ADR-0071 §10a. OpenAI's official Chat Completions deprecated `max_tokens`, and its REASONING models reject it + // outright — the second half of the maintainer's "max tokens errors". But this same adapter serves every custom + // OpenAI-compatible endpoint (LM Studio, Ollama, vLLM, LiteLLM, a gateway), most of which implement only the + // legacy field. Switching globally would trade one broken population for another, so the rule is by ENDPOINT. + let sent: Record = {}; + const capture = (init: RequestInit | undefined): Response => { + sent = parseJsonBody(init); + return okResponse(); + }; + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + const official = createOpenAiAdapter({ + fetch: (_i, init) => Promise.resolve(capture(init)), + }); + await official.generate({ model: 'gpt-5.5', messages, maxTokens: 100 }, 'k'); + expect(sent['max_completion_tokens']).toBe(100); + expect('max_tokens' in sent).toBe(false); // the deprecated field must NOT ride alongside it + + // DeepSeek's own API is OFFICIAL — it is our default base URL, not a caller's override — and it takes the + // legacy field. `official` is not a synonym for `openai`. + const deepseek = createOpenAiAdapter({ + providerId: 'deepseek', + fetch: (_i, init) => Promise.resolve(capture(init)), + }); + await deepseek.generate({ model: 'deepseek-chat', messages, maxTokens: 100 }, 'k'); + expect(sent['max_tokens']).toBe(100); + expect('max_completion_tokens' in sent).toBe(false); + + // A custom `base_url` under the `openai` provider id keeps the legacy field too — this is the D3 population + // that a global switch would have broken. + const custom = createOpenAiAdapter({ + baseURL: 'https://gateway.example.com/v1', + fetch: (_i, init) => Promise.resolve(capture(init)), + }); + await custom.generate({ model: 'gpt-5.5', messages, maxTokens: 100 }, 'k'); + expect(sent['max_tokens']).toBe(100); + expect('max_completion_tokens' in sent).toBe(false); + }); + + it('CLAMPS an over-ceiling cap on an official endpoint — and leaves a custom endpoint alone', async () => { + let sent: Record = {}; + const capture = (init: RequestInit | undefined): Response => { + sent = parseJsonBody(init); + return okResponse(); + }; + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + // gpt-5.4-pro's ceiling is 128_000. An authored 200_000 is a 400 on every turn, not an ambitious request. + const official = createOpenAiAdapter({ + fetch: (_i, init) => Promise.resolve(capture(init)), + }); + await official.generate({ model: 'gpt-5.4-pro', messages, maxTokens: 200_000 }, 'k'); + expect(sent['max_completion_tokens']).toBe(128_000); + + // …but a custom endpoint may serve a different model under that id, with its own limits. We do not silently + // lower a number the user typed on a model we cannot describe. + const custom = createOpenAiAdapter({ + baseURL: 'https://gateway.example.com/v1', + fetch: (_i, init) => Promise.resolve(capture(init)), + }); + await custom.generate({ model: 'gpt-5.4-pro', messages, maxTokens: 200_000 }, 'k'); + expect(sent['max_tokens']).toBe(200_000); + }); + + it('NEVER sends BOTH cap fields — a providerOptions `max_tokens` cannot ride alongside the mapped one', async () => { + // A REGRESSION the dialect rule introduced, caught by review. The escape-hatch merge is `{...providerOptions, + // ...body}`, and `body` winning was automatic only while the mapped key and the escape-hatch key were the SAME + // STRING. Renaming the mapped key on the official endpoint meant the old one no longer shadowed anything — so + // both went out, and OpenAI rejects a request carrying both. A 400 on exactly the population §10a rescues. + let sent: Record = {}; + const oai = createOpenAiAdapter({ + fetch: (_i, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + await oai.generate( + { + model: 'gpt-5.5', + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + maxTokens: 100, + providerOptions: { max_tokens: 999 }, // the stale key the caller might still be carrying + }, + 'k', + ); + expect(sent['max_completion_tokens']).toBe(100); // the mapped cap wins outright… + expect('max_tokens' in sent).toBe(false); // …and the colliding key is GONE, not merely outranked + }); + + it('an escape-hatch cap with NO mapped cap still stands — the §10a override for an exotic gateway', async () => { + // The gateway that speaks OpenAI's protocol but wants the modern field has no config key to ask with. It asks + // through `providerOptions`, and that only works if we leave an un-mapped cap alone. + let sent: Record = {}; + const custom = createOpenAiAdapter({ + baseURL: 'https://gateway.example.com/v1', + fetch: (_i, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + await custom.generate( + { + model: 'gpt-5.5', + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + providerOptions: { max_completion_tokens: 256 }, // no req.maxTokens — theirs is the only cap + }, + 'k', + ); + expect(sent['max_completion_tokens']).toBe(256); + expect('max_tokens' in sent).toBe(false); + }); + + it("OpenAI's OWN url spelled out by hand is OFFICIAL — a trailing slash must not restore the bug", async () => { + // The CLI stores a `--base-url` VERBATIM, so `https://api.openai.com/v1/` and `https://api.openai.com/v1` are + // different strings. Classifying by "was a string passed" made the first one CUSTOM: deprecated field, no + // clamp — the original bug, restored on the official endpoint by a typo. We classify by HOST. + let sent: Record = {}; + for (const url of [ + 'https://api.openai.com/v1/', // a trailing slash + 'https://api.openai.com/v1', + 'https://api.openai.com', // no /v1 at all + 'https://API.OpenAI.com/v1', // host case is not significant + 'https://api.openai.com./v1', // a trailing-dot FQDN — DNS says this is the same host, and so do we + ]) { + const oai = createOpenAiAdapter({ + baseURL: url, + fetch: (_i, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + await oai.generate( + { + model: 'gpt-5.4-pro', + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + maxTokens: 200_000, + }, + 'k', + ); + expect(sent['max_completion_tokens'], url).toBe(128_000); // the modern field AND the clamp + expect('max_tokens' in sent, url).toBe(false); + } + }); + + it('a LOOKALIKE host is NOT official — the classification must never slide the other way', async () => { + // The dangerous direction. Misreading a gateway as official would send it the modern field AND clamp its cap + // against a catalog that does not describe what it serves. + let sent: Record = {}; + for (const url of [ + 'https://evil.api.openai.com.attacker.net/v1', // the official host as a SUBDOMAIN of someone else's + 'https://api.openai.com.evil.com/v1', // …and as a prefix + 'https://api-openai.com/v1', // a hyphen away + ]) { + const impostor = createOpenAiAdapter({ + baseURL: url, + fetch: (_i, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + await impostor.generate( + { + model: 'gpt-5.4-pro', + messages: [{ role: 'user', content: [{ type: 'text', text: 'go' }] }], + maxTokens: 200_000, + }, + 'k', + ); + expect(sent['max_tokens'], url).toBe(200_000); // legacy field, NOT clamped — it is not OpenAI + expect('max_completion_tokens' in sent, url).toBe(false); + } + }); + + it('an EMPTY descriptor gets NO reasoning field either — `deepseek-reasoner` reasons, but publishes no knob', async () => { + // The second adversarial review found this: the DeepSeek arm gated on `catalogModel(m)?.reasoning !== undefined` + // and then sent `thinking` UNCONDITIONALLY. `deepseek-reasoner`'s descriptor is `{}` — not `undefined` — so the + // gate opened, and every tier (including `off` → `thinking: {type:'disabled'}`) went on the wire for a model + // whose controllable tiers upstream declined to describe. `acceptedTiers` returns the empty set for `{}`, which + // is exactly what the picker already showed the user; the wire now agrees with it. + let sent: Record = {}; + const deepseek = createOpenAiAdapter({ + providerId: 'deepseek', + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + for (const tier of ['off', 'low', 'medium', 'high', 'max'] as const) { + await deepseek.generate({ model: 'deepseek-reasoner', messages, reasoningEffort: tier }, 'k'); + expect('thinking' in sent, `deepseek-reasoner ${tier}`).toBe(false); + } + }); + + it('a tier OUTSIDE the published ladder is withheld — `gpt-5.4-pro` rejects both `low` and `off`', async () => { + // MEMBERSHIP, not presence. The arm used to test that an effort axis existed and then send its own tier name. + let sent: Record = {}; + const oai = createOpenAiAdapter({ + fetch: (_input, init) => { + sent = parseJsonBody(init); + return Promise.resolve(okResponse()); + }, + }); + const messages = [{ role: 'user' as const, content: [{ type: 'text' as const, text: 'go' }] }]; + + for (const tier of ['low', 'off'] as const) { + await oai.generate({ model: 'gpt-5.4-pro', messages, reasoningEffort: tier }, 'k'); + expect('reasoning_effort' in sent, `gpt-5.4-pro ${tier}`).toBe(false); + } + // …but a tier it DOES publish rides. + await oai.generate({ model: 'gpt-5.4-pro', messages, reasoningEffort: 'high' }, 'k'); + expect(sent['reasoning_effort']).toBe('high'); + }); + + it('maps the reasoning-effort tier per provider: OpenAI reasoning_effort (max→the model’s top, off→none) + DeepSeek thinking (ADR-0066)', async () => { let sent: Record = {}; const oai = createOpenAiAdapter({ fetch: (_input, init) => { @@ -257,6 +564,14 @@ describe('OpenAI-compatible adapter', () => { await oai.generate({ ...base }, 'k'); // unset ⇒ omitted (provider default, unchanged behavior) expect('reasoning_effort' in sent).toBe(false); + // A model that publishes a DISTINCT `'max'` above `'xhigh'` (the gpt-5.6 family) reaches it: `max` must send + // the model's OWN top, not stop one rung short at `'xhigh'` (review M1). gpt-5.5 above tops at `'xhigh'` and + // still coarsens `max → 'xhigh'`, so the two together pin the per-model branch. + await oai.generate({ model: 'gpt-5.6', messages: base.messages, reasoningEffort: 'max' }, 'k'); + expect(sent['reasoning_effort']).toBe('max'); + await oai.generate({ model: 'gpt-5.6', messages: base.messages, reasoningEffort: 'high' }, 'k'); + expect(sent['reasoning_effort']).toBe('high'); // the intermediate tiers are unchanged + // DeepSeek (the other id this shared adapter serves) controls thinking via a `thinking` OBJECT, not the OpenAI // `reasoning_effort` key (ADR-0066): off→disabled; DeepSeek has only two graded levels, so low/medium/high→high // and max→max; unset ⇒ omitted. @@ -1015,7 +1330,9 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => }); await adapter.generate( { - model: 'gpt-5.5', + // gpt-4o ACCEPTS temperature (its catalog requestCapabilities does not forbid it) — the per-model + // capability gate (ADR-0071 §12) would rightly withhold it on a `temperature: false` model like gpt-5.5. + model: 'gpt-4o', temperature: 0.5, stopSequences: ['STOP'], providerOptions: { seed: 42, model: 'attacker-override' }, @@ -1026,7 +1343,7 @@ describe('OpenAI-compatible adapter — request building + secret safety', () => expect(sent['temperature']).toBe(0.5); expect(sent['stop']).toEqual(['STOP']); expect(sent['seed']).toBe(42); // escape-hatch field reached the wire - expect(sent['model']).toBe('gpt-5.5'); // mapped field wins over providerOptions + expect(sent['model']).toBe('gpt-4o'); // mapped field wins over providerOptions }); it('maps tool_choice {name} to a named function choice', async () => { diff --git a/packages/llm/src/adapters/openai.ts b/packages/llm/src/adapters/openai.ts index f9fe06cb..719ee740 100644 --- a/packages/llm/src/adapters/openai.ts +++ b/packages/llm/src/adapters/openai.ts @@ -18,7 +18,10 @@ import { import { assertStreamable, assertSupported } from '../capabilities.js'; import { InvalidBaseUrlError, UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError, kindFromHttpStatus, makeLlmError } from '../llm-error.js'; -import { MODEL_PRICING } from '../pricing.js'; +import { catalogModel, catalogModelIds, modelAccepts } from '../catalog/lookup.js'; +import { isNonChatModelId } from '../model-kind.js'; +import { cappedMaxTokens, type EndpointKind } from '../output-cap.js'; +import { DEEPSEEK_WIRE, acceptedTiers, openAiWireValue } from '../reasoning-wire.js'; import { normalizeToolCall, toWire } from '../tool-normalizer.js'; import type { CapabilityFlags, @@ -67,6 +70,41 @@ import { const DEEPSEEK_BASE_URL = 'https://api.deepseek.com'; +/** + * The HOSTS whose APIs are the providers' own — the ones this adapter's dialect rules are written against. + * + * Classified by host, not by "did a caller pass a base URL string". A user who registers OpenAI with its own + * endpoint spelled out (`relavium provider add openai --base-url https://api.openai.com/v1/`) IS on the official + * API, and the CLI stores the URL VERBATIM — so a trailing slash, or `/v1` versus no `/v1`, is enough to miss a + * string comparison. Missing it would send the deprecated `max_tokens` to a reasoning model that rejects it, and + * skip the output clamp: the exact bug this work removes, restored on the official endpoint by a typo. + */ +const OFFICIAL_HOSTS: Readonly> = { + openai: 'api.openai.com', + deepseek: 'api.deepseek.com', +}; + +/** + * Is this base URL the provider's OWN API? + * + * The trailing dot is not pedantry: `api.openai.com.` is a legitimate fully-qualified spelling that DNS resolves + * identically, the CLI stores a `--base-url` VERBATIM, and the adjacent SSRF check already normalizes it away. Left + * unnormalized here it would classify the REAL official endpoint as custom — deprecated field, no clamp — which is + * the bug this work removes, reinstated by a dot. + * + * An unparseable URL is treated as custom: the conservative side, and unreachable in practice (`assertHttpsBaseUrl` + * has already parsed it by the time we get here). Belt only. + */ +function endpointKindFor(providerId: OpenAiProviderId, baseURL: string | undefined): EndpointKind { + if (baseURL === undefined) return 'official'; // no override at all — our own default + try { + const host = new URL(baseURL).hostname.toLowerCase().replace(/\.$/, ''); + return host === OFFICIAL_HOSTS[providerId] ? 'official' : 'custom'; + } catch { + return 'custom'; + } +} + /** * OpenAI's common-path capability surface. 1.AE wires the real media input matrix (image, audio, * document in) and sets `vision` to the derived alias of `media.input.image`. DeepSeek remains @@ -403,65 +441,26 @@ export function openaiErrorToLlmError(err: unknown, provider: ProviderId, key?: // --- Live model discovery: the id-only list filter (ADR-0064 §3) ------------------------------ /** - * Id SEGMENTS that are NOT chat-completions text models — DENIED from the OpenAI/DeepSeek live list - * (ADR-0064 §3). The OpenAI `/v1/models` list is id-only (no capability metadata), so the filter is an - * id-family heuristic: deny wins over allow, so `gpt-image-1` / `gpt-4o-audio-preview` / `omni-moderation` - * are dropped even though they match a `gpt`/`o` allow-family. Each token is matched on a `-`/`_` SEGMENT - * boundary (not a bare substring), so `search` denies `gpt-4o-search-preview` but NOT `o3-deep-research` - * (re**search**), and `dall-e`'s internal `-` is a literal segment. The tail entries - * (`instruct`/`ocr`/`davinci`/`babbage`) drop non-chat completion families that otherwise pass the - * gpt/deepseek allow-family; all are priced-rescue-safe (the `pricedIds.has(id)` short-circuit wins first). - */ -const OPENAI_DENY_SUBSTRINGS = [ - 'embedding', - 'tts', - 'whisper', - 'image', - 'moderation', - 'realtime', - 'audio', - 'dall-e', - 'transcribe', - 'search', - 'instruct', - 'ocr', - 'davinci', - 'babbage', -] as const; - -/** Escape a literal string for embedding inside a `RegExp`. The deny tokens carry no metacharacters today - * (`dall-e`'s `-` is literal outside a character class), but this keeps the boundary match safe if one is - * ever added. */ -function escapeRegExp(text: string): string { - // `String.raw` avoids the doubled backslash of `'\\$&'` — the replacement is a literal `\` + the `$&` match ref. - return text.replace(/[.*+?^${}()|[\]\\]/g, String.raw`\$&`); -} - -/** True when a deny `token` occurs on a `-`/`_` segment boundary in `lower` — a word-boundary match, so - * `search` fires on `gpt-4o-search-preview` (`-search-`) but not `o3-deep-research` (`re`+`search`). */ -function denyTokenMatches(lower: string, token: string): boolean { - return new RegExp(`(^|[-_])${escapeRegExp(token)}([-_]|$)`).test(lower); -} - -/** - * The MODEL_PRICING native ids + canonical keys for one OpenAI-compatible provider — unioned into the live - * list so a **cost-eligible** id ALWAYS survives the id-family heuristic (ADR-0064 §3), even if a future - * priced id doesn't match a `gpt`/`o`/`chat`/`deepseek` family. + * The CATALOG's ids for one OpenAI-compatible provider — unioned into the live list so a **cost-eligible** id + * ALWAYS survives the id-family heuristic (ADR-0064 §3), even if a priced id does not match a + * `gpt`/`o`/`chat`/`deepseek` family. + * + * Its source is now the generated catalog snapshot (ADR-0071), not the hand-typed pricing table it replaced — + * eighty-odd models wider, which is the point. That breadth is NOT a licence to smuggle a non-chat model past the + * deny-list, so `isNonChatModelId` still filters it (an embedding is priced, and is still not something you can + * chat with; see `keepOpenAiModelId`). */ export function pricedModelIdsFor(provider: ProviderId): ReadonlySet { const ids = new Set(); - for (const [canonicalId, pricing] of Object.entries(MODEL_PRICING)) { - if (pricing.provider === provider) { - ids.add(canonicalId); - ids.add(pricing.nativeId); - } + for (const id of catalogModelIds()) { + if (catalogModel(id)?.provider === provider) ids.add(id); } return ids; } /** * Keep an OpenAI/DeepSeek model id iff it is a chat-capable text model (ADR-0064 §3). A priced id is kept - * unconditionally (cost-eligibility wins); otherwise `ft:` fine-tunes and every {@link OPENAI_DENY_SUBSTRINGS} + * unconditionally (cost-eligibility wins); otherwise `ft:` fine-tunes and every non-chat family ({@link isNonChatModelId}) * family are denied, and only the `gpt` / `o` / `deepseek` / `*chat*` families are kept. Pure + * unit-tested. */ @@ -473,7 +472,7 @@ export function keepOpenAiModelId(id: string, pricedIds: ReadonlySet): b if (lower.startsWith('ft:')) { return false; } - if (OPENAI_DENY_SUBSTRINGS.some((deny) => denyTokenMatches(lower, deny))) { + if (isNonChatModelId(id)) { return false; } return ( @@ -650,20 +649,6 @@ function toOpenAiTool(toolDef: ToolDef, provider: ProviderId): OpenAI.ChatComple return { type: 'function', function: fn }; } -/** ADR-0066: the normalized reasoning-effort tier → OpenAI's native `reasoning_effort` values. `off`→'none', - * `max`→'xhigh' (its highest); low/medium/high are 1:1. A SUBSET of the SDK's `ReasoningEffort` union, so the - * assignment to `body.reasoning_effort` needs no cast. */ -const OPENAI_REASONING_EFFORT: Record< - ReasoningEffort, - 'none' | 'low' | 'medium' | 'high' | 'xhigh' -> = { - off: 'none', - low: 'low', - medium: 'medium', - high: 'high', - max: 'xhigh', -}; - /** DeepSeek's native reasoning control (a Relavium-local shape — NOT a vendor SDK type — so nothing crosses the * seam): the create-chat-completion `thinking` object (verified 2026-07-07, api-docs.deepseek.com). */ interface DeepSeekThinking { @@ -676,10 +661,10 @@ interface DeepSeekThinking { * documented coarsening onto v4's actual capability). */ const DEEPSEEK_THINKING: Record = { off: { type: 'disabled' }, - low: { type: 'enabled', reasoning_effort: 'high' }, - medium: { type: 'enabled', reasoning_effort: 'high' }, - high: { type: 'enabled', reasoning_effort: 'high' }, - max: { type: 'enabled', reasoning_effort: 'max' }, + low: { type: 'enabled', reasoning_effort: DEEPSEEK_WIRE.low }, + medium: { type: 'enabled', reasoning_effort: DEEPSEEK_WIRE.medium }, + high: { type: 'enabled', reasoning_effort: DEEPSEEK_WIRE.high }, + max: { type: 'enabled', reasoning_effort: DEEPSEEK_WIRE.max }, }; function toOpenAiToolChoice(choice: ToolChoice): OpenAI.ChatCompletionToolChoiceOption { @@ -711,7 +696,11 @@ type OpenAiCompatibleBody = Omit; + } else if (provider === 'deepseek') { + // `deepseek-reasoner`'s descriptor is EMPTY (`{}`): it reasons, but publishes no controllable tier. + // `acceptedTiers` returns the empty set for `{}`, so nothing goes on the wire and the picker offers nothing: + // they agree by construction rather than by two people remembering the same rule. + body.thinking = DEEPSEEK_THINKING[req.reasoningEffort]; + } +} + +/** The output-cap field this endpoint takes (ADR-0071 §10a). ONE place decides it, so no caller can send both. */ +function outputCapField( + provider: ProviderId, + endpoint: EndpointKind, +): 'max_tokens' | 'max_completion_tokens' { + // OpenAI's own Chat Completions deprecated `max_tokens`, and its reasoning models reject it outright. Every other + // OpenAI-compatible server — DeepSeek's API, LM Studio, Ollama, vLLM, LiteLLM, an enterprise gateway — implements + // the legacy field, and most implement only that. + return provider === 'openai' && endpoint === 'official' ? 'max_completion_tokens' : 'max_tokens'; } /** Lower a canonical `responseFormat: json` to OpenAI's `response_format`: DeepSeek supports only @@ -984,6 +1067,7 @@ async function* streamChunks( client: OpenAI, req: LlmRequest, provider: ProviderId, + endpoint: EndpointKind, key: string, ): AsyncIterable { const state: OpenAiStreamState = { @@ -997,7 +1081,11 @@ async function* streamChunks( let sdkStream: AsyncIterable; try { sdkStream = await client.chat.completions.create( - { ...buildCommonBody(req, provider), stream: true, stream_options: { include_usage: true } }, + { + ...buildCommonBody(req, provider, endpoint), + stream: true, + stream_options: { include_usage: true }, + }, buildRequestOptions(req), ); } catch (err) { @@ -1059,6 +1147,17 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { if (deps.baseURL !== undefined) { assertHttpsBaseUrl(deps.baseURL); } + // OFFICIAL vs CUSTOM — decided ONCE, here, where the base URL is actually known (ADR-0071 §7/§10a). + // + // A CUSTOM endpoint is another server speaking OpenAI's protocol: LM Studio, Ollama, vLLM, an enterprise + // gateway. It governs two things downstream — the output-cap FIELD NAME (only OpenAI's own API takes + // `max_completion_tokens`) and whether we CLAMP that cap against the catalog at all (a proxy may serve something + // quite different under a familiar model id, and silently lowering a number the user typed is a behaviour change + // we have no right to make on a model we cannot describe). + // + // By HOST, not by "was a string passed": DeepSeek's own `api.deepseek.com` is official (it is our default), and + // so is `https://api.openai.com/v1/` typed out by hand. See {@link endpointKindFor}. + const endpoint: EndpointKind = endpointKindFor(providerId, deps.baseURL); const createClient = (key: string): OpenAI => new OpenAI({ apiKey: key, @@ -1076,7 +1175,7 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { const client = createClient(key); try { const completion = await client.chat.completions.create( - { ...buildCommonBody(req, providerId), stream: false }, + { ...buildCommonBody(req, providerId, endpoint), stream: false }, buildRequestOptions(req), ); const choice = completion.choices[0]; @@ -1104,13 +1203,13 @@ export function createOpenAiAdapter(deps: OpenAiAdapterDeps = {}): LlmProvider { assertStreamable(providerId, supports); assertMediaCapabilities(providerId, supports, req); // per-modality input/output gate (ADR-0031, 1.AE) assertNoStreamingMediaOutput(providerId, req); // media-out is generate()-only; streaming triad deferred (ADR-0046 §4) - return streamChunks(createClient(key), req, providerId, key); + return streamChunks(createClient(key), req, providerId, endpoint, key); }, /** * Live model discovery (ADR-0064 §1) over the SDK's `models.list()`. The OpenAI/DeepSeek list is * ID-ONLY (no context/price metadata), so each row maps to a bare `{ id }` `ModelListing` and is - * filtered to chat-capable text families via `keepOpenAiModelId` (unioned with `MODEL_PRICING` for - * cost-eligibility). The provider id (`openai` | `deepseek`) selects the priced-id union set. Bounded + + * filtered to chat-capable text families via `keepOpenAiModelId` (unioned with the generated catalog's + * priced ids for cost-eligibility). The provider id (`openai` | `deepseek`) selects the priced-id union set. Bounded + * abortable + secret-free via `boundedListModels`; a per-row parse failure drops only that row. */ async listModels(key: string, signal?: AbortSignalLike): Promise { diff --git a/packages/llm/src/adapters/shared.ts b/packages/llm/src/adapters/shared.ts index d5ed85c7..6d2236a2 100644 --- a/packages/llm/src/adapters/shared.ts +++ b/packages/llm/src/adapters/shared.ts @@ -3,7 +3,7 @@ import type { AbortSignalLike } from '@relavium/shared'; import { mediaSupportReason } from '../capabilities.js'; import { UnsupportedCapabilityError } from '../errors.js'; import { LlmProviderError, makeLlmError } from '../llm-error.js'; -import { MODEL_PRICING, isCanonicalModelId } from '../pricing.js'; +import { catalogModel } from '../catalog/lookup.js'; import { LlmMessageSchema, ModelListingSchema } from '../types.js'; import type { CapabilityFlags, @@ -52,12 +52,12 @@ export function estimateRequestTokens(input: EstimateTokensInput): number { } /** - * The model's context window in tokens from the shared pricing catalog (ADR-0062) — the shared default behind - * each adapter's `contextLimit`. `undefined` for an unrated / custom-base-URL model (the engine then skips - * auto-compaction rather than guess a window). No cast: the canonical-id guard narrows the index. + * The model's context window in tokens, from the generated catalog (ADR-0062/0071) — the shared default behind + * each adapter's `contextLimit`. `undefined` for a model the catalog does not carry (a custom base URL, or one + * newer than the snapshot); the engine then skips auto-compaction rather than guess a window. */ export function contextLimitFor(model: string): number | undefined { - return isCanonicalModelId(model) ? MODEL_PRICING[model].contextWindowTokens : undefined; + return catalogModel(model)?.contextWindowTokens; } /** diff --git a/packages/llm/src/budget-estimator.test.ts b/packages/llm/src/budget-estimator.test.ts index a43f159c..4e3bef6e 100644 --- a/packages/llm/src/budget-estimator.test.ts +++ b/packages/llm/src/budget-estimator.test.ts @@ -1,11 +1,13 @@ import { describe, expect, it } from 'vitest'; import { estimateMaxNextCost, estimateMediaCost } from './budget-estimator.js'; -import { MODEL_PRICING, type ModelPricing } from './pricing.js'; +import { catalogPricing } from './catalog/pricing.js'; +import type { ModelPricing } from './pricing.js'; describe('estimateMaxNextCost', () => { it('estimates output-only worst case at maxTokens', () => { - const model = MODEL_PRICING['claude-sonnet-4-6']; + const model = catalogPricing('claude-sonnet-4-6'); + if (model === undefined) throw new Error('claude-sonnet-4-6 is not priced'); // 10_000 output tokens @ $15/MTok = 150_000 micro-cents expect(estimateMaxNextCost('claude-sonnet-4-6', 10_000)).toBe( Math.round((10_000 * model.outputPerMtokMicrocents) / 1_000_000), @@ -22,13 +24,48 @@ describe('estimateMaxNextCost', () => { }); it('uses the cheaper mini rate for a mini model', () => { - const model = MODEL_PRICING['gpt-5.4-mini']; + const model = catalogPricing('gpt-5.4-mini'); + if (model === undefined) throw new Error('gpt-5.4-mini is not priced'); expect(estimateMaxNextCost('gpt-5.4-mini', 100_000)).toBe( Math.round((100_000 * model.outputPerMtokMicrocents) / 1_000_000), ); }); }); +describe('estimateMaxNextCost — the estimate must price the request we ACTUALLY send (ADR-0071 §7)', () => { + it('prices the CLAMPED cap, not the authored one — the governor was killing runs over phantom money', () => { + // `gemini-2.5-pro`'s output ceiling is 65_536. An agent authored with `max_tokens: 200000` used to be + // pre-authorized for THREE TIMES the spend the model is physically capable of producing. With `on_exceed: fail` + // that killed the run over money that could never be spent; with `pause_for_approval` it was a human gate for + // the same phantom. Before the clamp landed the request simply 400'd, so the gap was invisible — now the + // request is valid, and an estimate of an unsendable request is just wrong. + const authored = estimateMaxNextCost('gemini-2.5-pro', 200_000); + const ceiling = estimateMaxNextCost('gemini-2.5-pro', 65_536); + expect(authored).toBe(ceiling); // the over-ceiling ask is priced at the ceiling — the only spend that can occur + expect(authored).toBeLessThan(estimateMaxNextCost('gemini-2.5-pro', 65_536) * 3); + }); + + it('still prices a cap BELOW the ceiling at what was asked for — the clamp is one-directional', () => { + const small = estimateMaxNextCost('gemini-2.5-pro', 1_000); + expect(small).toBeGreaterThan(0); + expect(small).toBeLessThan(estimateMaxNextCost('gemini-2.5-pro', 65_536)); + }); + + it('does NOT clamp a CUSTOM endpoint — the same bug pointing the other way, and the dangerous direction', () => { + // The adapter deliberately does not clamp a custom `base_url` (it may serve anything under a familiar id). An + // estimate that clamps ANYWAY lands BELOW what the wire can spend — so the governor under-authorizes and waves + // through the very call it exists to stop. `on_exceed: fail` then fails to fail. + // + // Over-estimating kills a valid run; UNDER-estimating spends the user's money. The estimate has to make the + // same call the adapter makes, and the host is the only one who knows which endpoint this is. + const asOfficial = estimateMaxNextCost('gpt-5.5', 500_000, undefined, 'official'); + const asCustom = estimateMaxNextCost('gpt-5.5', 500_000, undefined, 'custom'); + expect(asOfficial).toBe(estimateMaxNextCost('gpt-5.5', 128_000)); // clamped to the ceiling + expect(asCustom).toBeGreaterThan(asOfficial); // …and the gateway is priced for what it can actually emit + expect(asCustom).toBe(estimateMaxNextCost('gpt-5.5', 500_000, undefined, 'custom')); + }); +}); + describe('estimateMediaCost (1.AF/D17 — pre-egress per-modality media estimate)', () => { it('degrades to 0 for a real model (no row carries a media rate in 1.AF)', () => { // Every shipped row leaves mediaOutputRates undefined, so a media-output turn adds no estimate — the diff --git a/packages/llm/src/budget-estimator.ts b/packages/llm/src/budget-estimator.ts index bbbd2c70..aa75e591 100644 --- a/packages/llm/src/budget-estimator.ts +++ b/packages/llm/src/budget-estimator.ts @@ -1,6 +1,7 @@ import type { MediaBilledModality } from '@relavium/shared'; -import { priceModel, type PricingOverlay } from './cost-tracker.js'; +import { priceModel, worstCaseRates, type PricingOverlay } from './cost-tracker.js'; +import { cappedMaxTokens, type EndpointKind } from './output-cap.js'; const TOKENS_PER_MTOK = 1_000_000; @@ -11,18 +12,38 @@ const TOKENS_PER_MTOK = 1_000_000; * (or a configured default), because the engine does not tokenize the prompt locally. * This is intentionally conservative: it may block slightly early rather than overshoot. * + * **The estimate must price the request the WIRE will carry** + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §7). The adapter holds an + * authored cap to the model's own output ceiling, so the cap the estimate reasons about is the clamped one — + * otherwise `gemini-2.5-pro` with `max_tokens: 200000` (ceiling 65 536) is pre-authorized for three times the + * spend the model can physically produce, and `on_exceed: fail` kills the run over money that could never leave + * the account. + * + * `endpoint` is not decoration, and getting it wrong is the same bug pointing the other way. The adapter does NOT + * clamp a custom `base_url` (it may serve anything under a familiar id), so estimating a custom endpoint AS IF it + * clamped produces an estimate BELOW what the wire can spend — and a governor that under-authorizes waves through + * a call it should have stopped. Absent ⇒ `'official'`, matching the adapter's own default for an un-overridden + * endpoint; a host that registers a custom `base_url` must say so. + * * All figures are integer micro-cents. */ export function estimateMaxNextCost( modelId: string, maxOutputTokens: number, overlay?: PricingOverlay, + endpoint: EndpointKind = 'official', ): number { const p = priceModel(modelId, overlay); - if (maxOutputTokens <= 0) { + // A model the catalog cannot describe passes through unclamped — the same rule the adapter follows, so the + // estimate stays a faithful prediction of the request rather than a second, disagreeing opinion about it. + const capped = cappedMaxTokens(maxOutputTokens, modelId, endpoint) ?? maxOutputTokens; + if (capped <= 0) { return 0; } - return Math.round((maxOutputTokens * p.outputPerMtokMicrocents) / TOKENS_PER_MTOK); + // The HIGHEST tier the model has (ADR-0071 §11). The engine does not tokenize the prompt locally, so it cannot + // know which side of a 200k/272k threshold this turn will land on — and on a SAFETY control, guessing the cheap + // side is the guess that lets money escape. + return Math.round((capped * worstCaseRates(p).output) / TOKENS_PER_MTOK); } /** One element of the pre-egress media estimate: a billed modality + its assumed unit count (a count for diff --git a/packages/llm/src/catalog/catalog-model.ts b/packages/llm/src/catalog/catalog-model.ts new file mode 100644 index 00000000..f23bd151 --- /dev/null +++ b/packages/llm/src/catalog/catalog-model.ts @@ -0,0 +1,113 @@ +import type { ProviderId } from '../types.js'; + +/** + * A context-size pricing tier — "above N context tokens, the rate changes" + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §11). + * + * `gemini-2.5-pro` is 1.25/10 below 200k and **2.5/15 above**. A flat rate understates long-context spend by up + * to 2× — tolerable when the cost cap was advisory, not when it is a safety control. The **pre-egress estimate + * takes the HIGHEST applicable tier**: a cap that over-estimates refuses a turn the user could have afforded; a + * cap that under-estimates lets real money escape. Only one of those is recoverable. + */ +export interface CatalogPriceTier { + /** The context-token threshold above which these rates apply. */ + readonly aboveContextTokens: number; + readonly inputPerMtokMicrocents: number; + readonly outputPerMtokMicrocents: number; + readonly cachedInputPerMtokMicrocents?: number; +} + +/** + * How a model exposes its reasoning control — **per MODEL, not per provider**, which is the whole point. + * + * [ADR-0066](../../../../docs/decisions/0066-normalized-reasoning-effort-control.md) made the native shape a + * property of the **adapter** and the capability a per-model `boolean`. A boolean cannot say *"this model takes + * a token budget in [128, 32768] and has no off switch"* — and that inexpressibility shipped as a live bug: our + * Gemini adapter sends `thinkingLevel` to `gemini-2.5-*`, which Google's docs say **do not support it**. This + * type is the correction. + * + * The three axes are **not** mutually exclusive — `gemini-2.5-flash` has both `toggle` and `budgetTokens`; + * `claude-sonnet-4-6` has both `effortValues` and `budgetTokens`. An adapter picks the shape it can lower. + * + * An **empty** descriptor (`reasoning: {}`) is a real and distinct state: the model *reasons* but exposes **no + * control** (`deepseek-reasoner`). That is not the same as no reasoning at all — it tells the picker to offer + * nothing, rather than to offer everything. + */ +export interface ReasoningControls { + /** + * The **PROVIDER-WIRE** effort values this model accepts (`none|minimal|low|medium|high|xhigh|max`) — NOT + * Relavium's normalized {@link ReasoningEffort}. The two vocabularies overlap enough to be dangerous: reading + * these as our tiers drops `off` from **every** Claude model (where `off` is `thinking:{disabled}`, not an + * effort value) and drops `off`+`max` from `gpt-5.5`. They are only ever *composed* with an adapter's wire map + * by `acceptedTiers` — never copied. + */ + readonly effortValues?: readonly string[]; + /** + * The token-budget axis, when the model takes one. **`min` is load-bearing**: `gemini-2.5-flash` has `min: 0` + * (so thinking CAN be disabled) while `gemini-2.5-pro` has `min: 128` (so it **cannot** — Google's docs say + * "N/A: Cannot disable thinking"). One field, and the `off` tier's availability falls straight out of it. A + * hand-maintained boolean could never have carried that. + */ + readonly budgetTokens?: { readonly min: number; readonly max?: number }; + /** The model exposes a plain on/off switch for thinking. */ + readonly toggle?: true; +} + +/** + * One model's metadata, normalized from the upstream catalog — the **generated** replacement for the + * hand-maintained `MODEL_PRICING` row ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md)). + * + * Money is **integer micro-cents per million tokens** (1 micro-cent = 1e-8 USD) — no float, ever, on any path + * that reaches the cost cap. + */ +export interface CatalogModel { + readonly provider: ProviderId; + readonly modelId: string; + readonly displayName: string; + readonly contextWindowTokens: number; + /** The model's own output ceiling. Nothing clamps against it today — half of "max tokens errors". */ + readonly maxOutputTokens: number; + readonly inputPerMtokMicrocents: number; + readonly outputPerMtokMicrocents: number; + /** + * **Absent ≠ 0.** 19 of the ~97 imported models carry no cache-read rate (`gpt-5.4-pro` among them). `0` means + * *"no discount"* — writing it for an absent rate would bill cached input at **zero**, a silent undercharge in + * the mechanism this work exists to harden. Absent ⇒ the cost path falls back to the full input rate. + */ + readonly cachedInputPerMtokMicrocents?: number; + /** Cache-WRITE, where a provider charges one (Anthropic does). */ + readonly cacheWritePerMtokMicrocents?: number; + /** Context-size pricing tiers, when the model has them. Absent ⇒ the flat rate applies at every length. */ + readonly contextTiers?: readonly CatalogPriceTier[]; + /** Absent ⇒ the model does not reason. Present-but-empty ⇒ it reasons with **no controllable tier**. */ + readonly reasoning?: ReasoningControls; + /** Absent ⇒ the model accepts every request parameter (the safe default). Present ⇒ carries only the parameters + * the model does NOT accept. See {@link RequestCapabilities}. */ + readonly requestCapabilities?: RequestCapabilities; +} + +/** + * Per-model REQUEST capabilities (ADR-0071 amendment) — whether the model accepts a given request PARAMETER, + * sourced per-model from models.dev the same way `reasoning` is. `temperature`, `tool_call`, `structured_output` + * and `attachment` all vary per model WITHIN a single provider (e.g. `gpt-5.6-luna` rejects `temperature` while + * its siblings accept it), so a provider-wide {@link CapabilityFlags} boolean cannot say what is true — the same + * shape of problem ADR-0071 fixed for pricing and reasoning. This is DISTINCT from `CapabilityFlags`, and lives + * here (not on the seam struct) exactly as `reasoning` does. + * + * A field is present ONLY when upstream says the model does NOT accept the parameter (`false`); **absent ⇒ + * accepted** — the safe default, so a model we have no data for is never denied a parameter it takes. The adapters + * WITHHOLD the wire parameter when the flag is `false`, turning a provider 400 into a dropped-and-noted field. + */ +export interface RequestCapabilities { + /** `false` ⇒ the model rejects a `temperature` parameter (models.dev). Absent ⇒ accepted. */ + readonly temperature?: boolean; + /** `false` ⇒ the model rejects tool/function-call definitions. Absent ⇒ accepted. */ + readonly toolCall?: boolean; + /** `false` ⇒ the model rejects a structured-output / response-format request. Absent ⇒ accepted. */ + readonly structuredOutput?: boolean; + /** `false` ⇒ the model rejects non-text (image/file) input attachments. Absent ⇒ accepted. */ + readonly attachment?: boolean; +} + +/** The generated snapshot: canonical model id → its metadata. Keyed by id alone, matching the merge's key. */ +export type CatalogSnapshot = Readonly>; diff --git a/packages/llm/src/catalog/catalog-providers.ts b/packages/llm/src/catalog/catalog-providers.ts new file mode 100644 index 00000000..a4e7ee78 --- /dev/null +++ b/packages/llm/src/catalog/catalog-providers.ts @@ -0,0 +1,42 @@ +import { LLM_PROVIDERS } from '@relavium/shared'; + +import type { ProviderId } from '../types.js'; + +/** + * Relavium's {@link ProviderId} → the provider's key in the upstream metadata catalog + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §2). + * + * **The ONLY place the two vocabularies meet.** Everything downstream — the sync tool, the merge, the picker — + * iterates {@link LLM_PROVIDERS} rather than a literal list, so **adding a provider is one line here**, not a + * rewrite (see the `add-llm-adapter` skill, step 5). + * + * Two traps this table closes, both of which cost real money if missed: + * + * • **`gemini` ≠ `google`.** Upstream keys Gemini as `google`. A hand-written `provider === 'gemini'` lookup + * against the raw payload silently finds nothing — every Gemini model would arrive UNPRICED, and an unpriced + * model skips the ADR-0028 cost cap entirely. The typed `Record` makes the mapping + * exhaustive: a new `ProviderId` is a **compile error** here until it is mapped. + * • **`google-vertex` is NOT imported.** It republishes the same Gemini ids at *different* prices. A naive + * flatten over every upstream provider would register each Gemini model twice, and the second write would + * win — pricing the user's Gemini traffic at Vertex rates. Only the mapped key is ever read; the other + * ~162 upstream providers are dropped, because Relavium cannot call them at all. + */ +export const CATALOG_PROVIDER_KEYS: Record = { + anthropic: 'anthropic', + openai: 'openai', + gemini: 'google', + deepseek: 'deepseek', +}; + +/** + * The upstream keys we import, in {@link LLM_PROVIDERS} order — what the sync tool iterates. + * Derived, never hand-listed, so it cannot drift from the enum. + */ +export const CATALOG_UPSTREAM_KEYS: readonly string[] = LLM_PROVIDERS.map( + (id) => CATALOG_PROVIDER_KEYS[id], +); + +/** The {@link ProviderId} an upstream key maps back to, or `undefined` for a provider we have no adapter for. */ +export function providerIdForCatalogKey(upstreamKey: string): ProviderId | undefined { + return LLM_PROVIDERS.find((id) => CATALOG_PROVIDER_KEYS[id] === upstreamKey); +} diff --git a/packages/llm/src/catalog/deprecations.ts b/packages/llm/src/catalog/deprecations.ts new file mode 100644 index 00000000..fbffa4c8 --- /dev/null +++ b/packages/llm/src/catalog/deprecations.ts @@ -0,0 +1,34 @@ +/** + * **When to warn a user** — a Relavium-owned overlay, and deliberately the ONE thing the generated catalog does not + * decide ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §10). + * + * models.dev publishes a `status` flag, not a retirement date, and a flag cannot tell a user *"this stops working + * in eleven days"*. That sentence is an editorial call about our own users, made from the provider's own + * announcement — which is why it survives the deletion of the hand-typed price table rather than dying with it. + * + * The first implementation of the swap dropped these dates on the argument that "the provider is the only one who + * knows when the provider is retiring something, so it should come from the live list". The argument is right and + * the conclusion was wrong: **no adapter populates `ModelListing.deprecatedAt`** — the OpenAI-compatible list is + * id-only, and Anthropic's and Gemini's mappers carry limits and names, nothing else. So `deprecated` became + * permanently `false` for every model in the product, and `deepseek-chat` was set to stop working on 2026-07-24 + * with nothing anywhere to say so. Information we already had, thrown away. + * + * This is NOT a second pricing home. It carries no price, no ceiling, no capability — one date per model, from a + * published retirement notice, and the merge unions it with whatever the live list and the user say (earliest + * wins). A model absent from it is simply not announced as retiring. + * + * **Adding one:** cite the provider's announcement in the comment. Removing one once its date has passed is not + * urgent — the entry keeps flagging a model that genuinely no longer works. + */ +export const MODEL_DEPRECATIONS: Readonly> = { + // DeepSeek's legacy aliases (api-docs.deepseek.com/quick_start/pricing, verified 2026-07-03): both retire at + // 2026-07-24 15:59 UTC, superseded by `deepseek-v4-flash` / `deepseek-v4-pro`, which serve non-thinking and + // thinking on one id each. + 'deepseek-chat': '2026-07-24T15:59:00Z', + 'deepseek-reasoner': '2026-07-24T15:59:00Z', +}; + +/** The announced retirement date for a model id, or `undefined` if none is announced. */ +export function deprecationFor(modelId: string): string | undefined { + return MODEL_DEPRECATIONS[modelId]; +} diff --git a/packages/llm/src/catalog/lookup.test.ts b/packages/llm/src/catalog/lookup.test.ts new file mode 100644 index 00000000..e9e50328 --- /dev/null +++ b/packages/llm/src/catalog/lookup.test.ts @@ -0,0 +1,125 @@ +import { afterEach, describe, expect, it } from 'vitest'; + +import { cost } from '../cost-tracker.js'; + +import type { CatalogModel } from './catalog-model.js'; +import { + catalogModel, + catalogModelIds, + clearCatalogRefresh, + installCatalogRefresh, + modelAccepts, +} from './lookup.js'; +import { CATALOG_SNAPSHOT } from './snapshot.js'; + +/** + * The runtime catalog overlay (ADR-0071 §4). The refresh is installed as module state; these tests own the floor + * that keeps it from making the product worse than the shipped snapshot. + * + * They live HERE, in `@relavium/llm`, and not in the CLI's `catalog-refresh.test.ts`, on purpose: the CLI test + * imports `installCatalogRefresh` across the package boundary (from the built `dist`), so a break in this source + * would not fail it until a rebuild — the exact stale-artifact trap that makes a break-verification lie. Tested at + * the source, a broken floor fails immediately. + */ + +afterEach(clearCatalogRefresh); // module state — a leaked refresh would poison every later test in the process + +/** A well-formed refreshed row — override what a case cares about. */ +function model(partial: Partial & Pick): CatalogModel { + return { + provider: 'openai', + displayName: partial.modelId, + contextWindowTokens: 100_000, + maxOutputTokens: 10_000, + inputPerMtokMicrocents: 1_000_000, + outputPerMtokMicrocents: 2_000_000, + ...partial, + }; +} + +describe('modelAccepts — per-model request-capability, default accepted (ADR-0071 amendment)', () => { + it('returns false ONLY for a parameter the catalog explicitly marks unsupported', () => { + installCatalogRefresh({ + 'cap-tail': model({ + modelId: 'cap-tail', + requestCapabilities: { temperature: false, structuredOutput: false }, + }), + }); + expect(modelAccepts('cap-tail', 'temperature')).toBe(false); + expect(modelAccepts('cap-tail', 'structuredOutput')).toBe(false); + expect(modelAccepts('cap-tail', 'toolCall')).toBe(true); // not marked ⇒ accepted + expect(modelAccepts('cap-tail', 'attachment')).toBe(true); + }); + + it('a model with no capability data, and an UNCATALOGUED id, both accept everything (the safe default)', () => { + installCatalogRefresh({ 'plain-tail': model({ modelId: 'plain-tail' }) }); + expect(modelAccepts('plain-tail', 'temperature')).toBe(true); + expect(modelAccepts('some-custom-base-url-model', 'temperature')).toBe(true); // never withhold on missing data + }); +}); + +describe('installCatalogRefresh — additive only, and the shipped snapshot is the FLOOR', () => { + it('NEVER touches a shipped model — a priced-but-lower refresh is IGNORED (§9)', () => { + // The money bug the first cut shipped. The "floor" guard was `if (shipped === undefined || model.output > 0)`, + // and the line above had already dropped every `output <= 0`, so the second clause was ALWAYS true and the whole + // thing was `if (true)`: a refreshed row replaced its shipped row wholesale. A hostile — or typo'd — upstream + // `output: 1 microcent` on `gpt-5.5` then recorded real spend as ~nothing, and the cost cap stopped tripping. + const shipped = CATALOG_SNAPSHOT['gpt-5.5']; + expect(shipped).toBeDefined(); + if (shipped === undefined) return; + + installCatalogRefresh({ + 'gpt-5.5': model({ + modelId: 'gpt-5.5', + outputPerMtokMicrocents: 1, // ~free — the cap-defeating price + inputPerMtokMicrocents: 1, + contextWindowTokens: 8_000, // a hundredth of the real window + reasoning: { effortValues: ['high'] }, // one tier, not five + }), + }); + + // Every field is the SHIPPED one — the refresh did not write it. + expect(catalogModel('gpt-5.5')).toEqual(shipped); + // …so a $14.50 turn is still $14.50, not $0.00. + const billed = cost('gpt-5.5', { inputTokens: 1_000_000, outputTokens: 100_000 }); + expect(billed).toBeGreaterThan(1_000_000_000); // > $10, not the ~$0 the downgrade would have billed + }); + + it('ADDS a model the snapshot never carried — the long tail, which is the point of refreshing', () => { + expect(catalogModel('gpt-7-tail')).toBeUndefined(); // the premise + installCatalogRefresh({ 'gpt-7-tail': model({ modelId: 'gpt-7-tail' }) }); + expect(catalogModel('gpt-7-tail')?.outputPerMtokMicrocents).toBe(2_000_000); + expect(catalogModelIds()).toContain('gpt-7-tail'); + expect(catalogModelIds()).toContain('gpt-5.5'); // …and the snapshot is still all there + }); + + it('does NOT admit a NEW model with no output price — we price a model or we do not carry it', () => { + installCatalogRefresh({ + 'gpt-7-free': model({ modelId: 'gpt-7-free', outputPerMtokMicrocents: 0 }), + }); + expect(catalogModel('gpt-7-free')).toBeUndefined(); + }); + + it('does NOT admit a NEW model with no INPUT price either — input 0 bills input (and cached input) as FREE', () => { + // The floor guarded only OUTPUT, so an `input: 0` new model slipped in and billed the input side at zero — a + // silent undercharge in the exact mechanism the floor hardens. + installCatalogRefresh({ + 'gpt-7-input-free': model({ modelId: 'gpt-7-input-free', inputPerMtokMicrocents: 0 }), + }); + expect(catalogModel('gpt-7-input-free')).toBeUndefined(); + }); + + it('a later refresh REPLACES the earlier one — the overlay is the last install, not a union', () => { + installCatalogRefresh({ 'gpt-7-a': model({ modelId: 'gpt-7-a' }) }); + installCatalogRefresh({ 'gpt-7-b': model({ modelId: 'gpt-7-b' }) }); + expect(catalogModel('gpt-7-a')).toBeUndefined(); // gone with the previous install + expect(catalogModel('gpt-7-b')).toBeDefined(); + }); + + it('clear drops the refresh — the snapshot answers alone again', () => { + installCatalogRefresh({ 'gpt-7-tail': model({ modelId: 'gpt-7-tail' }) }); + clearCatalogRefresh(); + expect(catalogModel('gpt-7-tail')).toBeUndefined(); + expect(catalogModel('gpt-5.5')).toEqual(CATALOG_SNAPSHOT['gpt-5.5']); + }); +}); diff --git a/packages/llm/src/catalog/lookup.ts b/packages/llm/src/catalog/lookup.ts new file mode 100644 index 00000000..f3e7f026 --- /dev/null +++ b/packages/llm/src/catalog/lookup.ts @@ -0,0 +1,120 @@ +import type { ReasoningEffort } from '@relavium/shared'; + +import { acceptedTiers } from '../reasoning-wire.js'; + +import type { CatalogModel, RequestCapabilities } from './catalog-model.js'; +import { CATALOG_SNAPSHOT } from './snapshot.js'; + +/** + * The REFRESHED catalog, if the host has one — models.dev as of the last `relavium models refresh --catalog`. + * + * Module state, and deliberately so. Reading a file is platform work and `@relavium/llm` does none; the host does + * it and hands the result in as plain data, which is the same seam `keyFor` and the pricing overlay already use. + * Threading a catalog through every adapter call instead would put a parameter on `contextLimitFor`, + * `cappedMaxTokens`, `effortTiersFor` and four adapters, for a value that is a process-wide constant. + * + * Empty until installed. Nothing must depend on it being installed: with no refresh — which is the DEFAULT + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §4, `auto_refresh = false`) + * — the shipped snapshot answers everything, offline, exactly as it was designed to. + */ +let refreshed: Readonly> = {}; + +/** + * Install a refreshed catalog (ADR-0071 §4). **Additive only, and the shipped snapshot is the FLOOR.** + * + * A refresh only ADDS models the snapshot never carried — it **never** touches a shipped model, not even to enrich + * it (ADR-0071 §9: a change to an already-shipped row is a human decision, surfaced as a red CI check, never a + * silent runtime write). So a row is taken ONLY when the snapshot does not carry the model at all AND the refreshed + * row is priced on both sides: a malformed, truncated, or half-fetched payload degrades to the snapshot rather than + * to a blank catalog, and a model that was priced yesterday cannot become unpriced today because a third-party + * aggregator had a bad deploy. The cost cap is a safety control; it does not get to lapse because someone else's + * JSON changed. + */ +export function installCatalogRefresh(models: Readonly>): number { + const kept: Record = {}; + for (const [id, model] of Object.entries(models)) { + // A model the SHIPPED snapshot pins is NEVER touched — the snapshot's price is human-verified, and a runtime + // refresh does not get to move it (ADR-0071 §9: a price change on an already-shipped model is a human decision, + // surfaced as a red CI check, never a silent bot commit). THIS is the floor §4.2 promises, and it is airtight + // for the reason that matters: the refresh cannot make a known model cheaper, because it does not write one. + // + // The first version of this was a bug wearing a floor's comment. It read `if (shipped === undefined || + // model.output > 0)` — but the line above had already dropped every `output <= 0`, so the second clause was + // ALWAYS true and the whole guard was `if (true)`. A refreshed row replaced its shipped row wholesale: a moved + // (lower) price, a dropped context tier, fewer reasoning tiers all sailed through. A hostile — or simply + // typo'd — upstream `output: 0.00000001` on `gpt-5.5` recorded $14.50 of real spend as $0.00, and a cost cap of + // any value never tripped. The safety control the ADR exists to build, defeated by the code that builds it. + if (CATALOG_SNAPSHOT[id] !== undefined) continue; + // A NEW model — the long tail the 80-row snapshot does not carry, which is the whole reason to refresh. Admit it + // only if it carries a real price on BOTH sides: an unpriceable row is not an enrichment, and a priced-maybe- + // wrong new model is still strictly better than an unknown one (which degrades the cap to `allow` entirely). + // Guarding only OUTPUT let a `input: 0` model in with input (and cached input, which derives from it) billed + // FREE — a silent undercharge in the very mechanism this floor hardens. + if (model.outputPerMtokMicrocents <= 0 || model.inputPerMtokMicrocents <= 0) continue; + kept[id] = model; + } + refreshed = kept; + // The count of models ACTUALLY admitted — new, priced, and not shadowing a shipped id. The host reports this as + // `added`, and returning it here is what keeps that number honest: computing it host-side by re-applying the + // floor's predicates by hand is exactly how a report drifts from what got installed (a payload with one priced + // and one unpriced new model reported `added: 2` while only one landed). + return Object.keys(kept).length; +} + +/** Drop the refreshed catalog — the shipped snapshot answers alone again. For tests, and for `--catalog` failures. */ +export function clearCatalogRefresh(): void { + refreshed = {}; +} + +/** + * Look up a model's metadata: the refreshed catalog if the host installed one, else the shipped snapshot + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md)). + * + * `undefined` for an id neither carries — a brand-new model, or one behind a custom `base_url`. That is a + * **supported** state, not an error: such a model is simply unpriced and un-described, and every consumer degrades + * the same way (the cost cap flags it; the reasoning field is withheld rather than guessed at). + * + * The adapters read this directly, which is not a new coupling: they already imported `MODEL_PRICING` for the same + * purpose. What changes is only that the answer is generated and per-model, instead of hand-typed and per-provider. + */ +export function catalogModel(modelId: string): CatalogModel | undefined { + return refreshed[modelId] ?? CATALOG_SNAPSHOT[modelId]; +} + +/** Every model id we can describe — the snapshot, plus whatever a refresh added. */ +export function catalogModelIds(): readonly string[] { + return [...new Set([...Object.keys(CATALOG_SNAPSHOT), ...Object.keys(refreshed)])]; +} + +/** + * Does **this model** accept a given request PARAMETER (ADR-0071 amendment)? `temperature`, `tool_call`, + * `structured_output` and `attachment` vary per model within a provider, so an adapter must ask the catalog before + * it puts the parameter on the wire — sending one a model rejects (e.g. `temperature` on `gpt-5.6-luna`) is a 400. + * + * `true` unless the catalog explicitly says `false`: an un-described model (custom `base_url`, brand-new id) or one + * with no capability data is assumed to ACCEPT the parameter — the same degrade-to-supported default the rest of + * the catalog uses, so we never withhold a parameter a model actually takes on the strength of missing metadata. + */ +export function modelAccepts(modelId: string, param: keyof RequestCapabilities): boolean { + return catalogModel(modelId)?.requestCapabilities?.[param] !== false; +} + +/** + * The reasoning-effort tiers **this model id** accepts — the ONE predicate every surface must ask. + * + * There is exactly one right answer to "can the user set effort on this model, and to what", and it has to be the + * same answer for the picker, the `/effort` command, the engine's gate, the footer, and the wire. It has not been: + * the CLI was carrying three separately-written copies of `catalogModel(m)` + `acceptedTiers(...)` plus a fourth, + * older boolean (`modelSupportsReasoning`, an id heuristic over the hand-typed pricing table) that disagreed with + * them on sixteen shipped models. Agreement by convention is not agreement; this is the construction that makes + * divergence impossible. + * + * The empty set means "no controllable tier", and it has two distinct causes the caller may want to tell apart: + * the model is not in the catalog at all (a custom `base_url`, or one newer than our snapshot), or it is and + * publishes no knob (`deepseek-reasoner`). Both withhold the field; only the first is fixed by a catalog refresh. + * Use {@link catalogModel} directly to distinguish them. + */ +export function effortTiersFor(modelId: string): ReadonlySet { + const entry = catalogModel(modelId); + return entry === undefined ? new Set() : acceptedTiers(entry.provider, entry.reasoning); +} diff --git a/packages/llm/src/catalog/models-dev-schema.test.ts b/packages/llm/src/catalog/models-dev-schema.test.ts new file mode 100644 index 00000000..d5e1a114 --- /dev/null +++ b/packages/llm/src/catalog/models-dev-schema.test.ts @@ -0,0 +1,336 @@ +import { describe, expect, it } from 'vitest'; + +import { + CATALOG_PROVIDER_KEYS, + CATALOG_UPSTREAM_KEYS, + providerIdForCatalogKey, +} from './catalog-providers.js'; +import { ModelsDevPayloadSchema, normalizeCatalog } from './models-dev-schema.js'; + +/** + * THE BOUNDARY (ADR-0071 §11) — where a third-party payload becomes a Relavium type, and where a money surface + * and a wire parameter get decided. Everything here is about what must NOT happen: a wrong price, a `0` that + * means "free" when it means "unknown", a model silently missing, or one provider's rates written under another + * provider's id. + */ + +/** A minimal upstream model. Overrides are shallow-merged so a test can express exactly one deviation. */ +const upstreamModel = (over: Record = {}): Record => ({ + id: 'm1', + name: 'Model One', + limit: { context: 200_000, output: 64_000 }, + cost: { input: 5, output: 25 }, + ...over, +}); + +const payload = (providers: Record>): Record => + Object.fromEntries(Object.entries(providers).map(([key, models]) => [key, { id: key, models }])); + +describe('CATALOG_PROVIDER_KEYS — the one place the two vocabularies meet', () => { + it('maps `gemini` to `google` — the upstream key is NOT our ProviderId', () => { + // Miss this and every Gemini model arrives UNPRICED — and an unpriced model silently skips the cost cap. + expect(CATALOG_PROVIDER_KEYS.gemini).toBe('google'); + expect(providerIdForCatalogKey('google')).toBe('gemini'); + }); + + it('is EXHAUSTIVE over ProviderId — a new provider is a compile error until it is mapped', () => { + // The type is `Record`, so this is really a compile-time claim; the runtime check pins + // that no entry was left as an empty string, which would silently match nothing. + for (const key of CATALOG_UPSTREAM_KEYS) expect(key).not.toBe(''); + expect(CATALOG_UPSTREAM_KEYS).toHaveLength(Object.keys(CATALOG_PROVIDER_KEYS).length); + }); + + it('does NOT map an upstream provider we have no adapter for', () => { + expect(providerIdForCatalogKey('google-vertex')).toBeUndefined(); + expect(providerIdForCatalogKey('requesty')).toBeUndefined(); + expect(providerIdForCatalogKey('azure')).toBeUndefined(); + }); +}); + +describe('normalizeCatalog — what we import, and what we refuse to', () => { + it('imports ONLY our four providers — the other ~162 are not callable and must not enter the catalog', () => { + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { m1: upstreamModel({ id: 'gpt-x' }) }, + // Not ours: no adapter, no key, no way to call it. A row here would be an uncallable model in the picker. + 'some-aggregator': { m1: upstreamModel({ id: 'aggregated-x' }) }, + }), + ); + const { catalog } = normalizeCatalog(raw); + expect(Object.keys(catalog)).toEqual(['gpt-x']); + }); + + it('IGNORES `google-vertex` — it republishes the same Gemini ids at DIFFERENT prices', () => { + // A naive flatten registers every Gemini model twice and the second write wins — pricing the user's Gemini + // traffic at Vertex rates. The mapping is a lookup, never an iteration over the payload's own keys. + const raw = ModelsDevPayloadSchema.parse( + payload({ + google: { m: upstreamModel({ id: 'gemini-x', cost: { input: 1.25, output: 10 } }) }, + 'google-vertex': { m: upstreamModel({ id: 'gemini-x', cost: { input: 99, output: 99 } }) }, + }), + ); + const { catalog } = normalizeCatalog(raw); + expect(catalog['gemini-x']?.provider).toBe('gemini'); + expect(catalog['gemini-x']?.inputPerMtokMicrocents).toBe(125_000_000); // $1.25, NOT $99 + }); + + it('REGRESSION: a malformed model under a provider we do NOT import cannot kill the sync', () => { + // This is not hypothetical — it broke the first run. `requesty` publishes `budget_tokens` with no `min`, + // and a single strict schema over the whole 166-provider payload died on it. We can never call `requesty`; + // being hostage to its data quality for data we never read is the bug. Validate what we consume. + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { good: upstreamModel({ id: 'gpt-x' }) }, + requesty: { + bad: upstreamModel({ + id: 'whatever', + reasoning: true, + reasoning_options: [{ type: 'budget_tokens' }], // no `min` — invalid for us, irrelevant to us + }), + }, + }), + ); + const { catalog, dropped } = normalizeCatalog(raw); + expect(catalog['gpt-x']).toBeDefined(); // the sync SURVIVES + expect(dropped).toEqual([]); // and never even looked at `requesty` + }); + + it('drops a malformed model of OUR OWN provider — one bad row, not a dead sync — and REPORTS it', () => { + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { + 'gpt-x': upstreamModel({ id: 'gpt-x' }), + 'gpt-bad': upstreamModel({ id: 'gpt-bad', cost: { input: -1, output: 5 } }), // a NEGATIVE rate + }, + }), + ); + const { catalog, dropped } = normalizeCatalog(raw); + expect(catalog['gpt-x']).toBeDefined(); + expect(catalog['gpt-bad']).toBeUndefined(); // a negative rate would corrupt the cap — never written + expect(dropped).toHaveLength(1); // …and never silent: the sync prints it + expect(dropped[0]?.modelId).toBe('gpt-bad'); + }); + + it('a dropped model is reported by its OWN id, never the record key — the shipped-model guard depends on it', () => { + // The sync FAILS if a model we already ship would be dropped (losing its price silently un-caps it). That + // guard compares the dropped id against the committed snapshot, which is keyed by MODEL ID. Upstream keys + // its record by id today — but if that ever diverges, reporting the key would make the guard look at the + // wrong name and wave the regression straight through. So the id is read from the model, not the key. + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { + 'some-record-key': upstreamModel({ id: 'gpt-real-id', cost: null }), // key ≠ id, and unpriceable + }, + }), + ); + const { dropped } = normalizeCatalog(raw); + expect(dropped[0]?.modelId).toBe('gpt-real-id'); // NOT 'some-record-key' + }); + + it('…and falls back to the record key only when the failed row has no usable id of its own', () => { + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { 'fallback-key': { name: 'no id at all', cost: { input: 1, output: 1 } } }, + }), + ); + const { dropped } = normalizeCatalog(raw); + expect(dropped[0]?.modelId).toBe('fallback-key'); + }); + + it('drops an UNPRICEABLE model rather than writing it at $0 — a $0 row PASSES the cost cap', () => { + // Upstream carries `cost: null` on image models (they bill per image, an axis we do not model). Importing + // one at zero is worse than its absence: an unpriced model is FLAGGED, a $0 model sails through the cap. + const raw = ModelsDevPayloadSchema.parse( + payload({ openai: { img: upstreamModel({ id: 'gpt-image', cost: null }) } }), + ); + const { catalog, dropped } = normalizeCatalog(raw); + expect(catalog['gpt-image']).toBeUndefined(); + expect(dropped[0]?.reason).toContain('unpriceable'); + }); + + it('THROWS on a real cross-provider id collision rather than letting one price silently win', () => { + const raw = ModelsDevPayloadSchema.parse( + payload({ + openai: { a: upstreamModel({ id: 'shared-id' }) }, + anthropic: { b: upstreamModel({ id: 'shared-id' }) }, + }), + ); + // The catalog is keyed by model id (matching the merge), so this is a genuine ambiguity — and a generator + // that quietly halves its own output is exactly the failure mode this whole workstream exists to end. + expect(() => normalizeCatalog(raw)).toThrow(/appears under BOTH/); + }); +}); + +describe('ENRICHMENT is decoupled from the money gate — a priced model is never evicted by a field we enrich with (review M7)', () => { + const one = (over: Record): ReturnType => + normalizeCatalog( + ModelsDevPayloadSchema.parse(payload({ openai: { m: upstreamModel({ id: 'm', ...over }) } })), + ); + + it('an UNKNOWN reasoning_options shape is SKIPPED, not fatal — the priced model survives with a thinner descriptor', () => { + // Upstream adds a control type we do not recognize alongside one we do. A whole-array discriminatedUnion would + // fail the row and DROP a fully-priced model — which reads to the §9 guard as a vanished price. It must stay. + const { catalog, dropped } = one({ + reasoning: true, + reasoning_options: [ + { type: 'verbosity', values: ['terse', 'verbose'] }, // unknown — skip it + { type: 'effort', values: ['low', 'high'] }, // known — keep it + ], + }); + expect(dropped).toHaveLength(0); + expect(catalog['m']?.inputPerMtokMicrocents).toBe(500_000_000); // still priced + expect(catalog['m']?.reasoning).toEqual({ effortValues: ['low', 'high'] }); // only the recognized control + }); + + it('a MALFORMED known option is skipped too (an `effort` with no values), model kept', () => { + const { catalog, dropped } = one({ + reasoning: true, + reasoning_options: [{ type: 'effort' }, { type: 'budget_tokens', min: 1024 }], + }); + expect(dropped).toHaveLength(0); + expect(catalog['m']?.reasoning).toEqual({ budgetTokens: { min: 1024 } }); + }); + + it('`reasoning: null` is treated as "no reasoning", not a parse error — the model is admitted, priced, unreasoning', () => { + const { catalog, dropped } = one({ reasoning: null }); + expect(dropped).toHaveLength(0); + expect(catalog['m']?.outputPerMtokMicrocents).toBe(2_500_000_000); + expect(catalog['m']).not.toHaveProperty('reasoning'); + }); + + it('`limit: null` drops the model CLEANLY (no ceiling to clamp), not via a fatal parse of the whole row', () => { + const { catalog, dropped } = one({ limit: null }); + expect(catalog['m']).toBeUndefined(); // dropped — but as an unpriceable-shape drop, reported, not a crash + expect(dropped.some((d) => d.modelId === 'm')).toBe(true); + }); + + it('carries per-model REQUEST capabilities, storing ONLY the rejected ones (ADR-0071 amendment)', () => { + // Upstream marks this model as rejecting temperature + structured_output, accepting tool_call + attachment. + const { catalog } = one({ + temperature: false, + structured_output: false, + tool_call: true, + attachment: true, + }); + // Only the `false` (rejected) parameters are stored; the accepted ones add nothing (absent ⇒ accepted). + expect(catalog['m']?.requestCapabilities).toEqual({ + temperature: false, + structuredOutput: false, + }); + }); + + it('a model that accepts everything carries NO requestCapabilities (the common case adds nothing)', () => { + const { catalog } = one({ temperature: true, tool_call: true }); + expect(catalog['m']).not.toHaveProperty('requestCapabilities'); + }); + + it('a missing/odd capability value degrades to accepted (never a parse failure)', () => { + const { catalog, dropped } = one({ temperature: null }); // null ⇒ accepted, model still priced + expect(dropped).toHaveLength(0); + expect(catalog['m']).not.toHaveProperty('requestCapabilities'); + }); + + it('a `reasoning_options` CONTAINER shape change (null / non-array) thins the descriptor, never EVICTS the model', () => { + // The element-level leniency defended a bad OPTION; the container itself was still strict, so `reasoning_options: + // null` (models.dev already emits explicit null for reasoning/cost/limit) or an array→object change failed the + // whole ModelSchema.safeParse and dropped a fully-priced model — the §9 vanished-price red M7 exists to prevent. + for (const bad of [null, {}, 'nope', 3]) { + const { catalog, dropped } = one({ reasoning: true, reasoning_options: bad }); + expect(dropped, JSON.stringify(bad)).toHaveLength(0); + expect(catalog['m']?.inputPerMtokMicrocents).toBe(500_000_000); // still priced + // `reasoning: true` with no usable options ⇒ the empty descriptor (reasons, but no controllable tier), NOT a + // dropped row and NOT a spurious control. + expect(catalog['m']?.reasoning).toEqual({}); + } + }); +}); + +describe('the money boundary — integer micro-cents, and absent ≠ zero', () => { + const one = (over: Record): ReturnType => + normalizeCatalog( + ModelsDevPayloadSchema.parse(payload({ openai: { m: upstreamModel({ id: 'm', ...over }) } })), + ); + + it('converts USD-per-Mtok to INTEGER micro-cents — no float ever reaches the cost cap', () => { + const { catalog } = one({ cost: { input: 0.435, output: 0.87 } }); + expect(catalog['m']?.inputPerMtokMicrocents).toBe(43_500_000); + expect(catalog['m']?.outputPerMtokMicrocents).toBe(87_000_000); + expect(Number.isInteger(catalog['m']?.inputPerMtokMicrocents)).toBe(true); + }); + + it('an ABSENT cache-read rate stays UNDEFINED — writing 0 would bill cached input as FREE', () => { + // `0` means "no discount" in ModelPricing. 19 of our ~97 models have no cache-read rate (gpt-5.4-pro among + // them). Coercing absent → 0 is a silent undercharge in the exact mechanism this work is hardening. + const { catalog } = one({ cost: { input: 5, output: 25 } }); + expect(catalog['m']).not.toHaveProperty('cachedInputPerMtokMicrocents'); + }); + + it('a PRESENT cache-read rate of 0 is kept as 0 — that is a real "no discount", not a missing value', () => { + const { catalog } = one({ cost: { input: 5, output: 25, cache_read: 0 } }); + expect(catalog['m']?.cachedInputPerMtokMicrocents).toBe(0); + }); + + it('carries CONTEXT TIERS — a flat rate understates long-context spend by up to 2x', () => { + const { catalog } = one({ + cost: { + input: 1.25, + output: 10, + tiers: [{ input: 2.5, output: 15, tier: { type: 'context', size: 200_000 } }], + }, + }); + expect(catalog['m']?.contextTiers).toEqual([ + { + aboveContextTokens: 200_000, + inputPerMtokMicrocents: 250_000_000, + outputPerMtokMicrocents: 1_500_000_000, + }, + ]); + }); +}); + +describe('the reasoning descriptor — the shape is PER MODEL, which a boolean could never say', () => { + const reasoningOf = (over: Record): unknown => + normalizeCatalog( + ModelsDevPayloadSchema.parse(payload({ openai: { m: upstreamModel({ id: 'm', ...over }) } })), + ).catalog['m']?.reasoning; + + it('a non-reasoning model has NO descriptor', () => { + expect(reasoningOf({})).toBeUndefined(); + }); + + it('an EFFORT model carries its PROVIDER-WIRE values verbatim — never re-read as our tiers', () => { + // `gpt-5.4-pro` accepts {medium, high, xhigh} and REJECTS `low` — the maintainer's bug report, in one row. + expect( + reasoningOf({ + reasoning: true, + reasoning_options: [{ type: 'effort', values: ['medium', 'high', 'xhigh'] }], + }), + ).toEqual({ effortValues: ['medium', 'high', 'xhigh'] }); + }); + + it('a BUDGET model carries min/max — and `min` is what says whether `off` is even possible', () => { + // gemini-2.5-pro: min 128 ⇒ thinking CANNOT be disabled (Google: "N/A: Cannot disable thinking"). + // gemini-2.5-flash: min 0 ⇒ it can. One field, and the `off` tier's availability falls straight out of it. + expect( + reasoningOf({ + reasoning: true, + reasoning_options: [{ type: 'budget_tokens', min: 128, max: 32_768 }], + }), + ).toEqual({ budgetTokens: { min: 128, max: 32_768 } }); + }); + + it('carries BOTH axes when a model has both (toggle + budget, or effort + budget)', () => { + expect( + reasoningOf({ + reasoning: true, + reasoning_options: [{ type: 'toggle' }, { type: 'budget_tokens', min: 0, max: 24_576 }], + }), + ).toEqual({ toggle: true, budgetTokens: { min: 0, max: 24_576 } }); + }); + + it('an EMPTY descriptor is a distinct, real state: the model reasons but has NO controllable tier', () => { + // `deepseek-reasoner`. This is NOT the same as "does not reason" — it tells the picker to offer NOTHING, + // rather than to offer everything (which is precisely today's bug). + expect(reasoningOf({ reasoning: true })).toEqual({}); + }); +}); diff --git a/packages/llm/src/catalog/models-dev-schema.ts b/packages/llm/src/catalog/models-dev-schema.ts new file mode 100644 index 00000000..149e06eb --- /dev/null +++ b/packages/llm/src/catalog/models-dev-schema.ts @@ -0,0 +1,375 @@ +import { z } from 'zod'; + +import type { ProviderId } from '../types.js'; +import { isNonChatModelId } from '../model-kind.js'; +import { providerIdForCatalogKey } from './catalog-providers.js'; +import type { + CatalogModel, + CatalogPriceTier, + ReasoningControls, + RequestCapabilities, +} from './catalog-model.js'; + +/** + * The BOUNDARY between the upstream metadata catalog and Relavium + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §11). + * + * Everything in this file describes a **third-party payload**, and nothing in it escapes: the raw shapes below + * (`reasoning_options`, `cost.tiers`, `limit.input`, …) are Zod-parsed and normalized into + * {@link CatalogModel} before any other Relavium code sees them. That is what keeps the upstream source a + * *replaceable implementation detail* rather than an architectural commitment — the same discipline + * [ADR-0064](../../../../docs/decisions/0064-live-model-catalog.md) §1 applies to a provider's `ModelListing`. + * + * The schema is **deliberately lenient about fields we do not consume** (`.passthrough()` is NOT used — unknown + * keys are simply dropped by Zod's default strip). Upstream adds fields regularly; a sync must not fail because + * a field we never read appeared. It is **strict about the MONEY fields** (`cost`, `limit`) — those bound the cost + * cap and the output ceiling, and a bad value there must fail loudly. The ENRICHMENT fields (`reasoning`, + * `reasoning_options`) sit in between: consumed, but parsed leniently (review M7) — a shape change there must never + * EVICT a fully-priced model (which would read as a vanished price and fire the §9 guard), only thin its descriptor. + */ + +/** USD-per-million-tokens → integer micro-cents-per-million-tokens. 1 USD = 1e8 micro-cents; no float, ever. */ +const USD_PER_MTOK_TO_MICROCENTS = 100_000_000; +const toMicrocents = (usdPerMtok: number): number => + Math.round(usdPerMtok * USD_PER_MTOK_TO_MICROCENTS); + +/** A price must be a non-negative, finite USD figure. A negative or NaN rate would corrupt the cost cap. */ +const UsdRate = z.number().finite().nonnegative(); + +/** + * One context-size pricing tier. Upstream expresses "above N context tokens, the rate changes" — which our flat + * `ModelPricing` cannot say, and which understates long-context spend by up to 2× if ignored (`gemini-2.5-pro` + * is 1.25/10 below 200k and **2.5/15 above**). + */ +const CostTierSchema = z.object({ + input: UsdRate, + output: UsdRate, + cache_read: UsdRate.optional(), + tier: z.object({ type: z.string(), size: z.number().finite().positive() }), +}); + +const CostSchema = z.object({ + input: UsdRate, + output: UsdRate, + /** + * ABSENT ≠ 0. 19 of the ~97 models we import carry no cache-read rate (including `gpt-5.4-pro`). Coercing an + * absent rate to `0` would price cached input at **zero** — a silent undercharge in the very mechanism this + * work is hardening. It stays `undefined` and the cost path falls back to the full input rate (§11). + */ + cache_read: UsdRate.optional(), + cache_write: UsdRate.optional(), + tiers: z.array(CostTierSchema).optional(), +}); + +const LimitSchema = z.object({ + context: z.number().finite().positive(), + /** The model's max OUTPUT tokens — the ceiling nothing clamps against today, which is half of "max tokens errors". */ + output: z.number().finite().positive(), +}); + +/** + * The reasoning control, as upstream describes it — and the single most valuable field in the payload. + * + * Its **`type` is the shape of the control**, and the shape is **per model, not per provider**: `gemini-2.5-*` + * take `budget_tokens`, `gemini-3.x` take `effort`, `gemma` takes `toggle`. Assuming one shape for a whole + * adapter is precisely the false premise that shipped as a live bug (ADR-0066's dated correction note). + * + * `values` are **PROVIDER-WIRE** strings (`none|minimal|low|medium|high|xhigh|max`), NOT Relavium's normalized + * `ReasoningEffort`. They are carried through as-is and are only ever *composed* with an adapter's wire map by + * `acceptedTiers` — reading them as our tiers would silently drop `off` from every Claude model. + */ +const ReasoningOptionSchema = z.discriminatedUnion('type', [ + z.object({ type: z.literal('effort'), values: z.array(z.string().min(1)).min(1) }), + z.object({ + type: z.literal('budget_tokens'), + min: z.number().finite().nonnegative(), + max: z.number().finite().positive().optional(), + }), + z.object({ type: z.literal('toggle') }), +]); + +const ModelSchema = z.object({ + id: z.string().min(1), + name: z.string().min(1), + // ENRICHMENT fields (`reasoning`, `reasoning_options`) are parsed LENIENTLY, decoupled from the money gate + // (review M7). A fully-priced model must never be EVICTED because a field we merely enrich with changed shape: + // upstream adding a new `reasoning_options.type` we don't recognize, or emitting `reasoning: null`, would make a + // strict schema drop the model — and a dropped ALREADY-SHIPPED model reads to `diffCatalog` as a VANISHED price, + // firing the §9 money guard red for a non-money reason. So `reasoning` tolerates `null`, and each + // `reasoning_options` entry is validated ONE AT A TIME in {@link toReasoningControls} (an unrecognized shape is + // skipped, yielding a thinner reasoning descriptor), never as a whole-array `discriminatedUnion` that one bad + // element fails. The CONTAINER tolerates `null` AND a wrong TYPE (`.catch` → undefined): a `reasoning_options: + // null` or an array→object change must thin the descriptor, not evict the priced model — element-only leniency + // left that container-shape gap. `cost`/`limit` stay authoritative — they ARE the money surface — tolerating only + // `null` (absent). + reasoning: z.boolean().nullish(), + reasoning_options: z + .array(z.unknown()) + .nullish() + .catch(() => undefined), + // Per-model REQUEST capabilities (ADR-0071 amendment). Upstream carries these as top-level per-model booleans; + // they vary per model within a provider (`gpt-5.6-luna` has `temperature: false`). ENRICHMENT, so `nullish` and + // never fatal — a missing/odd value degrades to "accepted", the safe default. + temperature: z.boolean().nullish(), + tool_call: z.boolean().nullish(), + structured_output: z.boolean().nullish(), + attachment: z.boolean().nullish(), + limit: LimitSchema.nullish(), + /** `null` on every image model upstream — priced per image, an axis we do not model (§11). */ + cost: CostSchema.nullish(), +}); + +export type ModelsDevModel = z.infer; + +/** + * The payload's OUTER structure only — deliberately shallow. + * + * The first draft of this validated the whole 166-provider payload with one strict schema, and it **failed on + * the very first run**: `requesty` — a provider Relavium has no adapter for and can never call — publishes a + * `budget_tokens` option with no `min`, and that one malformed entry killed the entire sync. Validating a + * third-party payload wholesale makes us hostage to 162 providers' data quality for data we never read. + * + * So the rule is: **validate what we consume; do not parse what we do not.** The models of our four providers + * are validated ONE AT A TIME below (`parseModels`), so a single bad row is a dropped model, not a dead sync. + */ +const ProviderSchema = z.object({ + models: z.record(z.string(), z.unknown()), +}); + +/** + * The payload is a bag of providers, and it is parsed as `unknown` per provider for the same reason the models + * are: **one malformed entry among 166 must not kill a sync that reads four of them.** `id` is not even required + * here — we key providers by the record key (which `CATALOG_PROVIDER_KEYS` maps), so a provider stub that omits + * its own `id` field is no reason to fail. + */ +export const ModelsDevPayloadSchema = z.record(z.string(), z.unknown()); +export type ModelsDevPayload = z.infer; + +/** A model the upstream payload carried but we could not use, and why. Surfaced by the sync, never swallowed. */ +export interface DroppedModel { + readonly provider: ProviderId; + readonly modelId: string; + readonly reason: string; +} + +/** Normalize the upstream reasoning options into Relavium's control descriptor. Absent/empty ⇒ `undefined`. + * Each raw option is validated INDEPENDENTLY (review M7): an unrecognized shape is skipped, so a new upstream + * control type thins the descriptor rather than evicting the whole priced model via a failed array parse. */ +function toReasoningControls(raw: z.infer): ReasoningControls | undefined { + if (raw.reasoning !== true) return undefined; + const controls: { + effortValues?: readonly string[]; + budgetTokens?: { readonly min: number; readonly max?: number }; + toggle?: true; + } = {}; + for (const rawOption of raw.reasoning_options ?? []) { + const parsed = ReasoningOptionSchema.safeParse(rawOption); + if (!parsed.success) continue; // an unknown/ malformed control shape — skip it, keep the priced model + const option = parsed.data; + if (option.type === 'effort') controls.effortValues = option.values; + else if (option.type === 'budget_tokens') { + controls.budgetTokens = + option.max === undefined ? { min: option.min } : { min: option.min, max: option.max }; + } else controls.toggle = true; + } + // `reasoning: true` with NO options is a real upstream shape (e.g. `deepseek-reasoner`): the model reasons, + // but exposes no control. That is NOT the same as "no reasoning" — the descriptor exists and is empty, which + // is what tells the picker to offer nothing rather than to offer everything. + return controls; +} + +/** + * Normalize the upstream per-model request-capability booleans into Relavium's descriptor (ADR-0071 amendment). + * Only a `false` (the model does NOT accept the parameter) is carried — absent/true/null all mean "accepted", the + * safe default the adapters treat as "send it". Returns `undefined` when the model accepts everything, so the + * common case adds NOTHING to the row. + */ +function toRequestCapabilities(raw: z.infer): RequestCapabilities | undefined { + const caps: { + temperature?: boolean; + toolCall?: boolean; + structuredOutput?: boolean; + attachment?: boolean; + } = {}; + if (raw.temperature === false) caps.temperature = false; + if (raw.tool_call === false) caps.toolCall = false; + if (raw.structured_output === false) caps.structuredOutput = false; + if (raw.attachment === false) caps.attachment = false; + return Object.keys(caps).length === 0 ? undefined : caps; +} + +function toPriceTiers( + cost: NonNullable>, +): readonly CatalogPriceTier[] | undefined { + if (cost.tiers === undefined || cost.tiers.length === 0) return undefined; + return cost.tiers.map((tier) => ({ + aboveContextTokens: tier.tier.size, + inputPerMtokMicrocents: toMicrocents(tier.input), + outputPerMtokMicrocents: toMicrocents(tier.output), + ...(tier.cache_read === undefined + ? {} + : { cachedInputPerMtokMicrocents: toMicrocents(tier.cache_read) }), + })); +} + +/** + * Normalize ONE upstream model into a {@link CatalogModel}, or `undefined` if it is not usable as a chat model. + * + * A model is **dropped** when it has no `cost` or no `limit` — upstream's image models carry `cost: null`, and + * they are priced per image, an axis Relavium does not model. Importing them would put rows in the catalog that + * claim a price of zero, which is worse than their absence: a $0 row *engages* the cost cap and passes it. + */ +export function normalizeCatalogModel( + provider: ProviderId, + raw: ModelsDevModel, +): CatalogModel | undefined { + // A null/absent `limit` is treated as absent, not a parse error (review M7): the model is dropped cleanly (no + // output ceiling to clamp against) rather than failing the whole row and reading as a VANISHED priced model. + if ( + raw.cost === null || + raw.cost === undefined || + raw.limit === null || + raw.limit === undefined + ) { + return undefined; + } + const { cost, limit } = raw; + const reasoning = toReasoningControls(raw); + const requestCapabilities = toRequestCapabilities(raw); + const tiers = toPriceTiers(cost); + return { + provider, + modelId: raw.id, + displayName: raw.name, + contextWindowTokens: limit.context, + maxOutputTokens: limit.output, + inputPerMtokMicrocents: toMicrocents(cost.input), + outputPerMtokMicrocents: toMicrocents(cost.output), + ...(cost.cache_read === undefined + ? {} + : { cachedInputPerMtokMicrocents: toMicrocents(cost.cache_read) }), + ...(cost.cache_write === undefined + ? {} + : { cacheWritePerMtokMicrocents: toMicrocents(cost.cache_write) }), + ...(tiers === undefined ? {} : { contextTiers: tiers }), + ...(reasoning === undefined ? {} : { reasoning }), + ...(requestCapabilities === undefined ? {} : { requestCapabilities }), + }; +} + +/** Best-effort read of a FAILED row's `id`, for the drop report. `unknown` in, `string | undefined` out. */ +function idOfUnvalidated(raw: unknown): string | undefined { + if (typeof raw !== 'object' || raw === null) return undefined; + const id: unknown = (raw as { id?: unknown }).id; + return typeof id === 'string' && id.length > 0 ? id : undefined; +} + +/** + * Normalize ONE upstream model row into `catalog`, or record why it was DROPPED — a malformed, non-chat, or + * unpriceable row is a dropped model, never a dead sync. A model id that clashes ACROSS providers is a real + * ambiguity and THROWS; a clash WITHIN one provider (two record keys that normalize to the same `id`) keeps the + * first and drops the duplicate VISIBLY, rather than letting one arbitrarily overwrite a row already admitted. + */ +function processModel( + providerId: ProviderId, + recordKey: string, + rawModel: unknown, + catalog: Record, + dropped: DroppedModel[], +): void { + // NOT A CHAT MODEL — dropped before anything else, and this is load-bearing, not tidiness. + // + // `keepOpenAiModelId` short-circuits on `pricedIds.has(id)`: a PRICED id bypasses the live list's deny-list + // entirely, so a cost-eligible model can never be filtered out. Once the catalog becomes the priced set, any + // non-chat model in it would be *rescued* by that short-circuit and land in the user's model picker as something + // to chat with. `text-embedding-3-large` is priced upstream and arrived in the very first snapshot exactly that + // way. Sharing ONE filter with the live list (`isNonChatModelId`) is what makes the cascade impossible; two + // filters that can disagree is what makes it inevitable. + if (isNonChatModelId(recordKey)) return; + + const parsed = ModelSchema.safeParse(rawModel); + if (!parsed.success) { + // The reported id must be the MODEL'S OWN `id`, not the record key. The sync's shipped-model guard compares a + // dropped id against the committed snapshot, which is keyed by model id — so a key that differs from the id + // would let a model we already ship fall out of the catalog SILENTLY, taking its price (and therefore the cost + // cap) with it. Upstream happens to key by id today; the guard must not depend on that. Best-effort here, + // because the row failed validation and its `id` may be junk too. + const issues = parsed.error.issues.map((i) => `${i.path.join('.')} ${i.message}`).join('; '); + dropped.push({ + provider: providerId, + modelId: idOfUnvalidated(rawModel) ?? recordKey, + reason: `schema: ${issues}`, + }); + return; + } + + const model = normalizeCatalogModel(providerId, parsed.data); + if (model === undefined) { + // Almost always an image model: upstream carries `cost: null` because it bills per IMAGE, an axis we do not + // model. Importing it would write a $0 row — worse than absence, because a $0 row *passes* the cost cap instead + // of flagging the model as unpriced. + dropped.push({ + provider: providerId, + modelId: parsed.data.id, + reason: 'no cost or no limit (unpriceable)', + }); + return; + } + + const clash = catalog[model.modelId]; + if (clash !== undefined) { + if (clash.provider !== providerId) { + throw new Error( + `catalog: model id '${model.modelId}' appears under BOTH '${clash.provider}' and '${providerId}'. ` + + `The catalog is keyed by model id, so this is a real ambiguity — resolve it in CATALOG_PROVIDER_KEYS ` + + `(is one of them a mirror, like google-vertex?) rather than letting one price silently win.`, + ); + } + // SAME provider, same id from a second record key (two aliases can normalize onto one `id`). Keep the first and + // drop this duplicate visibly — an unconditional overwrite would let a later row silently replace an admitted + // one's price, and a price change is exactly what this transform refuses to make in silence. + dropped.push({ + provider: providerId, + modelId: model.modelId, + reason: `duplicate model id within '${providerId}' (record key '${recordKey}') — kept the first`, + }); + return; + } + catalog[model.modelId] = model; +} + +/** + * Normalize a validated payload into the catalog snapshot — the whole build-time transform. + * + * Keyed by **model id alone**, matching the merge's existing key ({@link mergeModelCatalog} drops a + * cross-provider id collision rather than letting one provider's price corrupt another's). A collision *within* + * our four providers would be a real ambiguity, so it **throws** rather than silently picking a winner — a + * generator that quietly halves the catalog is exactly the failure this whole workstream exists to end. + */ +export function normalizeCatalog(payload: ModelsDevPayload): { + readonly catalog: Record; + readonly dropped: readonly DroppedModel[]; +} { + const catalog: Record = {}; + const dropped: DroppedModel[] = []; + + for (const [upstreamKey, rawProvider] of Object.entries(payload)) { + const providerId = providerIdForCatalogKey(upstreamKey); + if (providerId === undefined) continue; // 162 upstream providers we have no adapter for — not callable. + + const provider = ProviderSchema.safeParse(rawProvider); + if (!provider.success) { + throw new Error( + `catalog: provider '${upstreamKey}' — which we DO import — has no usable \`models\` map. Refusing to ` + + `continue: silently importing zero models for a provider we can call would leave every one of its ` + + `models unpriced, and an unpriced model skips the cost cap.`, + ); + } + + for (const [recordKey, rawModel] of Object.entries(provider.data.models)) { + // ONE MODEL AT A TIME. A malformed row is a dropped model, not a dead sync (see `ProviderSchema`). + processModel(providerId, recordKey, rawModel, catalog, dropped); + } + } + return { catalog, dropped }; +} diff --git a/packages/llm/src/catalog/pricing.ts b/packages/llm/src/catalog/pricing.ts new file mode 100644 index 00000000..c867ece3 --- /dev/null +++ b/packages/llm/src/catalog/pricing.ts @@ -0,0 +1,77 @@ +import type { ModelPricing } from '../pricing.js'; + +import type { CatalogModel } from './catalog-model.js'; +import { catalogModel, catalogModelIds } from './lookup.js'; + +/** + * The catalog, read as a **price** ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §1). + * + * `ModelPricing` survives the swap because it is a **contract**, not a table: the `CostTracker` bills against it, + * the pre-egress governor estimates against it, and a user's `models pricing` row IS one (`PricingOverlay`). What + * dies is the hand-typed `MODEL_PRICING` object that used to be its only source — twelve rows, maintained by hand, + * that had already drifted from reality on two numbers before anything thought to compare them. + * + * Three fields the catalog cannot supply, and what happens to each: + * + * - **`nativeId`** — the provider-native id. Every row in the retired table set it equal to its own key, because + * the canonical id IS the provider's id; there was never a second name. The catalog key is that id. + * - **`mediaOutputRates`** — per-modality media rates. The retired table declared the field and **no row ever set + * it**: media folds at 0 until a verified rate lands, and a fabricated rate is worse than none (ADR-0044 §3). + * A USER row can still carry one, which is the only way one has ever been carried. + * - **`deprecatedAt`** — a provider's retirement announcement. models.dev publishes a `status` FLAG, not a date, and + * a flag cannot tell a user *"this stops working in eleven days"*. It lives in Relavium's own small overlay + * ({@link MODEL_DEPRECATIONS}), which ADR-0071 §10 always specified and the first cut of this swap wrongly deleted: + * the theory was that the live provider list would carry it, but **no adapter populates `ModelListing.deprecatedAt`** + * (the OpenAI list is id-only), so `deprecated` went permanently `false` for every model in the product. + */ +export function catalogPricing(modelId: string): ModelPricing | undefined { + const entry = catalogModel(modelId); + return entry === undefined ? undefined : toPricing(entry); +} + +/** Project one catalog row onto the pricing contract. Pure — no lookup, so a caller with the row in hand can reuse it. */ +export function toPricing(entry: CatalogModel): ModelPricing { + return { + provider: entry.provider, + nativeId: entry.modelId, // the canonical id IS the provider's id — there was never a second name + displayName: entry.displayName, + contextWindowTokens: entry.contextWindowTokens, + maxOutputTokens: entry.maxOutputTokens, + inputPerMtokMicrocents: entry.inputPerMtokMicrocents, + outputPerMtokMicrocents: entry.outputPerMtokMicrocents, + // ABSENT ⇒ THE FULL INPUT RATE. Never 0 (ADR-0071 §10, which exists because of exactly this line). + // + // The first version wrote `?? 0` with a comment claiming that 0 "bills a cached read at the full input price". + // It does not. `cost()` computes `cacheReadTokens × rate / 1e6`, so 0 bills the cached fraction of every prompt + // at NOTHING — and eleven catalog models publish no cache rate, including `gpt-5.4-pro` and `o1-pro`. OpenAI + // auto-caches, so on a 1M-token prompt that is 90% cached, `o1-pro` billed $0.00 for 900 000 tokens that cost + // $135. Worse than a wrong invoice: `max_cost_microcents` is a SAFETY control, and it cannot trip on money it + // never counts. + // + // "No published cache rate" means the provider does not DISCOUNT cache reads, not that it gives them away. The + // full input rate is the honest reading, and it is the one the ADR wrote down before the code got it wrong. + cachedInputPerMtokMicrocents: + entry.cachedInputPerMtokMicrocents ?? entry.inputPerMtokMicrocents, + ...(entry.cacheWritePerMtokMicrocents === undefined + ? {} + : { cacheWritePerMtokMicrocents: entry.cacheWritePerMtokMicrocents }), + // The context tiers ride onto the contract (ADR-0071 §11). They were parsed, guarded and exported — and read by + // nothing, so a >200k `gemini-2.5-pro` turn billed at the CHEAP rate. Tolerable while those models threw + // `UnknownModelError` (a loud gap); a silent 2× under-bill the moment the catalog started pricing them. + ...(entry.contextTiers === undefined ? {} : { contextTiers: entry.contextTiers }), + // `reasoning` was a BOOLEAN on the retired contract, and answering it was the bug (ADR-0071 §6): "does this + // model reason" is not the question the wire asks. The catalog carries the CONTROL — which tiers, in which + // shape — and `effortTiersFor` is what every surface asks now. The boolean is not projected, because nothing + // should ever ask it again. + }; +} + +/** + * Every model id we can price — the diagnostic list an `UnknownModelError` names. + * + * A FUNCTION, not a const: a `models refresh --catalog` adds models, and a constant captured at import time would go + * on naming yesterday's list in the error that tells a user which models exist. + */ +export function pricedModelIds(): readonly string[] { + return catalogModelIds(); +} diff --git a/packages/llm/src/catalog/snapshot-guard.test.ts b/packages/llm/src/catalog/snapshot-guard.test.ts new file mode 100644 index 00000000..e0a179fb --- /dev/null +++ b/packages/llm/src/catalog/snapshot-guard.test.ts @@ -0,0 +1,163 @@ +import { describe, expect, it } from 'vitest'; + +import { CATALOG_SNAPSHOT } from './snapshot.js'; +import type { CatalogModel, CatalogSnapshot } from './catalog-model.js'; +import { diffCatalog, moneyFingerprint } from './snapshot-guard.js'; + +/** + * THE MONEY GUARDS (ADR-0071 §9) — the only thing standing between a third-party data file and a silently + * weakened cost cap. These tests exist because the FIRST version of this guard was blind in two ways at once, + * and neither would ever have announced itself. + */ + +const model = (over: Partial = {}): CatalogModel => ({ + provider: 'openai', + modelId: 'm', + displayName: 'M', + contextWindowTokens: 200_000, + maxOutputTokens: 64_000, + inputPerMtokMicrocents: 500_000_000, + outputPerMtokMicrocents: 2_500_000_000, + ...over, +}); + +const snapshot = (...models: CatalogModel[]): CatalogSnapshot => + Object.fromEntries(models.map((m) => [m.modelId, m])); + +describe('moneyFingerprint — EVERY money field, not just the flat pair', () => { + it('a moved CONTEXT-TIER rate changes the fingerprint even when the flat rate does not', () => { + // The first guard compared `input/output` only. But the pre-egress estimate takes the HIGHEST applicable + // tier, so on a long-context turn the TIER rate — not the flat rate — is the number that sizes the cap. + // Halving only gemini-2.5-pro's >200k tier moved no flat rate, tripped nothing, and would have capped every + // long-context turn against half its true cost. + const flat = { inputPerMtokMicrocents: 125_000_000, outputPerMtokMicrocents: 1_000_000_000 }; + const before = model({ + ...flat, + contextTiers: [ + { + aboveContextTokens: 200_000, + inputPerMtokMicrocents: 250_000_000, + outputPerMtokMicrocents: 1_500_000_000, + }, + ], + }); + const after = model({ + ...flat, // the flat rate is IDENTICAL — this is the whole trap + contextTiers: [ + { + aboveContextTokens: 200_000, + inputPerMtokMicrocents: 125_000_000, // …but the >200k rate was halved + outputPerMtokMicrocents: 1_500_000_000, + }, + ], + }); + expect(moneyFingerprint(before)).not.toBe(moneyFingerprint(after)); + }); + + it('a moved CACHE-READ or CACHE-WRITE rate changes the fingerprint', () => { + const base = model({ cachedInputPerMtokMicrocents: 50_000_000 }); + expect(moneyFingerprint(base)).not.toBe( + moneyFingerprint(model({ cachedInputPerMtokMicrocents: 25_000_000 })), + ); + expect(moneyFingerprint(model({ cacheWritePerMtokMicrocents: 625_000_000 }))).not.toBe( + moneyFingerprint(model({ cacheWritePerMtokMicrocents: 312_500_000 })), + ); + }); + + it('distinguishes an ABSENT cache-read rate from a rate of ZERO', () => { + // `0` means "no discount"; absent means "no rate, fall back to the full input price". Collapsing them + // would let upstream flip one into the other — billing cached input as FREE — without tripping the guard. + expect(moneyFingerprint(model({}))).not.toBe( + moneyFingerprint(model({ cachedInputPerMtokMicrocents: 0 })), + ); + }); + + it('does NOT change for a non-money edit (a renamed display name)', () => { + expect(moneyFingerprint(model({ displayName: 'A' }))).toBe( + moneyFingerprint(model({ displayName: 'B' })), + ); + }); +}); + +describe('diffCatalog — the two ways a sync can quietly weaken a safety control', () => { + it('REGRESSION: catches a moved price on a model whose id is a BARE JS IDENTIFIER (`o1`, `o3`)', () => { + // THE bug this file exists for. The first guard regex-matched the generated snapshot's TEXT and required a + // single-quoted key — but prettier's default `quoteProps: 'as-needed'` emits `o1: {` unquoted. The regex saw + // 88 of 90 models; `o1` and `o3` had NO baseline at all, so a halved `o1` price passed in silence and the + // fatal "a shipped model vanished" check could not fire for them either. Diffing DATA makes the key's byte + // shape irrelevant — which is the only way this stays fixed. + const before = snapshot(model({ modelId: 'o1', inputPerMtokMicrocents: 1_500_000_000 })); + const after = { o1: model({ modelId: 'o1', inputPerMtokMicrocents: 750_000_000 }) }; + const { moved } = diffCatalog(before, after); + expect(moved).toHaveLength(1); + expect(moved[0]?.modelId).toBe('o1'); + }); + + it('catches a shipped model that VANISHED — for any reason, including reasons no drop-list would show', () => { + // The first guard could only see models the normalizer explicitly DROPPED. A model that simply disappeared + // from the payload — upstream deleted it, a CATALOG_PROVIDER_KEYS edit erased its whole provider — appeared + // in no list at all and was removed in silence. An absent model is an unpriced model, and an unpriced model + // skips the cost cap entirely. + const { vanished } = diffCatalog(snapshot(model({ modelId: 'gone' })), {}); + expect(vanished).toEqual(['gone']); + }); + + it('a NEW model is `added`, never `moved` — pricing a model can only INCREASE what the cap covers', () => { + const { added, moved, vanished } = diffCatalog({}, { fresh: model({ modelId: 'fresh' }) }); + expect(added).toEqual(['fresh']); + expect(moved).toEqual([]); + expect(vanished).toEqual([]); + }); + + it('an unchanged catalog diffs to NOTHING — the guard must not cry wolf', () => { + const same = snapshot(model({ modelId: 'a' }), model({ modelId: 'b' })); + const { moved, vanished, added } = diffCatalog(same, { ...same }); + expect([moved, vanished, added]).toEqual([[], [], []]); + }); +}); + +describe('the SHIPPED snapshot itself — 80 chat models, all priced, none of them non-chat', () => { + const models = Object.values(CATALOG_SNAPSHOT); + + it('is not empty, and every model carries a positive input AND output price', () => { + // A `0` price is not "free" — it is a model that would sail straight through the cost cap. + expect(models.length).toBeGreaterThan(50); + for (const m of models) { + expect(m.inputPerMtokMicrocents, `${m.modelId} input`).toBeGreaterThan(0); + expect(m.outputPerMtokMicrocents, `${m.modelId} output`).toBeGreaterThan(0); + expect(Number.isInteger(m.inputPerMtokMicrocents), `${m.modelId} is an integer`).toBe(true); + } + }); + + it('contains ZERO non-chat models — an embedding in the catalog becomes an embedding in the picker', () => { + // `keepOpenAiModelId` short-circuits on `pricedIds.has(id)`, so once the catalog is the priced set, any + // non-chat model in it is RESCUED past the live list's deny-list and offered to the user as a chat model. + // `text-embedding-3-large` is priced upstream and arrived in the very first snapshot exactly that way. + const nonChat = models.filter((m) => + /(^|[-_])(embedding|tts|image|realtime|audio)([-_]|$)/.test(m.modelId), + ); + expect(nonChat.map((m) => m.modelId)).toEqual([]); + }); + + it('still carries the four models this whole workstream exists for', () => { + // The bug report, and the three shapes a `reasoning: boolean` could never express. + expect(CATALOG_SNAPSHOT['gpt-5.4-pro']?.reasoning).toEqual({ + effortValues: ['medium', 'high', 'xhigh'], // REJECTS `low`, which the picker offers today + }); + expect(CATALOG_SNAPSHOT['gemini-2.5-pro']?.reasoning).toEqual({ + budgetTokens: { min: 128, max: 32_768 }, // no effort axis; min:128 ⇒ `off` is IMPOSSIBLE + }); + expect(CATALOG_SNAPSHOT['claude-haiku-4-5']?.reasoning?.effortValues).toBeUndefined(); // budget, not effort + expect(CATALOG_SNAPSHOT['deepseek-reasoner']?.reasoning).toEqual({}); // reasons, but no controllable tier + }); + + it('preserves the max-output ceiling and context window as DISTINCT fields (they are not interchangeable)', () => { + // A swap here is invisible to a shallow test and would clamp max_tokens against the CONTEXT window — a + // number ~3x too large on Opus, which is exactly the "max tokens error" this work is meant to end. + const opus = CATALOG_SNAPSHOT['claude-opus-4-8']; + expect(opus?.contextWindowTokens).toBe(1_000_000); + expect(opus?.maxOutputTokens).toBe(128_000); + // …and the drift the hand-typed table carried, now corrected from source: + expect(CATALOG_SNAPSHOT['claude-sonnet-4-6']?.maxOutputTokens).toBe(128_000); // the table said 64_000 + }); +}); diff --git a/packages/llm/src/catalog/snapshot-guard.ts b/packages/llm/src/catalog/snapshot-guard.ts new file mode 100644 index 00000000..4b718cd3 --- /dev/null +++ b/packages/llm/src/catalog/snapshot-guard.ts @@ -0,0 +1,98 @@ +import type { CatalogModel, CatalogSnapshot } from './catalog-model.js'; + +/** + * The catalog's MONEY GUARDS — what must never change silently + * ([ADR-0071](../../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §9). + * + * These live here, in typed and unit-tested code, rather than in the sync tool — and that is a correction, not a + * preference. The first version compared prices by **regex-matching the generated snapshot's text**, and it was + * blind in a way nobody would have noticed: + * + * Prettier's default `quoteProps: 'as-needed'` strips quotes from any key that is a valid JS identifier. Two + * of our ninety models — `o1` and `o3` — are therefore emitted **unquoted**, the regex required a quote, and + * so those two models had **no baseline at all**. A halved `o1` price passed the guard in silence, and the + * fatal "a shipped model was dropped" check could not fire for them either. + * + * The lesson is structural: the generated file's exact bytes are *prettier's* decision, not the tool's, so any + * guard that reads them as text is one formatting default away from going quietly blind. Diff the **data**. + */ + +/** + * Every money field of a model, as one comparable string. + * + * ALL of them, not just the flat pair. The first version compared `input/output` only — leaving `cache_read`, + * `cache_write`, and **every context tier** unguarded. That is the worst possible field to miss: the pre-egress + * estimate takes the **highest applicable tier**, so for any long-context turn the *tier* rate — not the flat + * rate — is the number that sizes the ADR-0028 cap. Twelve of our ninety models are tiered. Halving only + * `gemini-2.5-pro`'s >200k tier moved no flat rate, tripped nothing, and would have capped every long-context + * turn against half its true cost. + */ +export function moneyFingerprint(model: CatalogModel): string { + const tiers = (model.contextTiers ?? []) + .map( + (t) => + `${t.aboveContextTokens}:${t.inputPerMtokMicrocents}/${t.outputPerMtokMicrocents}/${t.cachedInputPerMtokMicrocents ?? '-'}`, + ) + .join(','); + return [ + model.inputPerMtokMicrocents, + model.outputPerMtokMicrocents, + model.cachedInputPerMtokMicrocents ?? '-', // `-` ≠ `0`: absent means "no rate", 0 means "no discount". + model.cacheWritePerMtokMicrocents ?? '-', + tiers, + ].join('|'); +} + +/** A shipped model whose price moved. */ +export interface MovedPrice { + readonly modelId: string; + readonly before: string; + readonly after: string; +} + +export interface CatalogDiff { + /** Models we already ship whose money changed. A human decision — never a silent bot commit. */ + readonly moved: readonly MovedPrice[]; + /** + * Models we already ship that are **GONE** from the new catalog — for ANY reason: upstream deleted them, + * upstream stopped pricing them, a provider-key edit erased a whole provider, the deny-list started matching + * them. The first version could only see models the normalizer explicitly *dropped*; a model that simply + * VANISHED from the payload appeared in no list at all and was removed in silence. An absent model is an + * UNPRICED model, and an unpriced model skips the cost cap entirely — so this is fatal by design. + */ + readonly vanished: readonly string[]; + /** New models. These merge freely: adding a price can only ever *increase* what the cap covers. */ + readonly added: readonly string[]; +} + +/** + * Diff the committed snapshot against a freshly-normalized catalog. + * + * `added` is informational. `moved` and `vanished` are the two ways a sync can quietly weaken a **safety + * control**, and the sync refuses both unless a human says otherwise. + */ +export function diffCatalog( + before: CatalogSnapshot, + after: Readonly>, +): CatalogDiff { + const moved: MovedPrice[] = []; + const vanished: string[] = []; + const added: string[] = []; + + for (const [modelId, shipped] of Object.entries(before)) { + const fresh = after[modelId]; + if (fresh === undefined) { + vanished.push(modelId); + continue; + } + const wasFingerprint = moneyFingerprint(shipped); + const nowFingerprint = moneyFingerprint(fresh); + if (wasFingerprint !== nowFingerprint) { + moved.push({ modelId, before: wasFingerprint, after: nowFingerprint }); + } + } + for (const modelId of Object.keys(after)) { + if (before[modelId] === undefined) added.push(modelId); + } + return { moved, vanished, added }; +} diff --git a/packages/llm/src/catalog/snapshot.ts b/packages/llm/src/catalog/snapshot.ts new file mode 100644 index 00000000..dd15c018 --- /dev/null +++ b/packages/llm/src/catalog/snapshot.ts @@ -0,0 +1,1032 @@ +// GENERATED FILE — DO NOT EDIT BY HAND. Run `pnpm sync:models`. +// +// The model-catalog snapshot (ADR-0071). Source: https://models.dev/api.json +// Catalog SHA-256: 4db066a3de7bf27d6f54955958a5cb8f60185826dfe9c77aef7aa0e1a370fdc0 +// Models: 80 +// +// This SHIPS IN THE BINARY on purpose. The cost cap (ADR-0028) is a safety control, and a safety control that +// only works when a third-party host is reachable is not one — so every model here is priced offline, on first +// run, with no network at all. The optional refresh (ADR-0071 §4) is default-OFF and can only ADD to this floor. +// +// Reviewing a diff here is reviewing a MONEY change. A price that moves on a model we already ship fails the +// sync deliberately (`--accept-price-changes` to take it) — because a rate that silently halves also silently +// halves the cap's protection. + +import type { CatalogSnapshot } from './catalog-model.js'; + +export const CATALOG_SNAPSHOT: CatalogSnapshot = { + 'claude-fable-5': { + provider: 'anthropic', + modelId: 'claude-fable-5', + displayName: 'Claude Fable 5', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 5000000000, + cachedInputPerMtokMicrocents: 100000000, + cacheWritePerMtokMicrocents: 1250000000, + reasoning: { effortValues: ['low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'claude-haiku-4-5': { + provider: 'anthropic', + modelId: 'claude-haiku-4-5', + displayName: 'Claude Haiku 4.5 (latest)', + contextWindowTokens: 200000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 100000000, + outputPerMtokMicrocents: 500000000, + cachedInputPerMtokMicrocents: 10000000, + cacheWritePerMtokMicrocents: 125000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-haiku-4-5-20251001': { + provider: 'anthropic', + modelId: 'claude-haiku-4-5-20251001', + displayName: 'Claude Haiku 4.5', + contextWindowTokens: 200000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 100000000, + outputPerMtokMicrocents: 500000000, + cachedInputPerMtokMicrocents: 10000000, + cacheWritePerMtokMicrocents: 125000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-1': { + provider: 'anthropic', + modelId: 'claude-opus-4-1', + displayName: 'Claude Opus 4.1 (latest)', + contextWindowTokens: 200000, + maxOutputTokens: 32000, + inputPerMtokMicrocents: 1500000000, + outputPerMtokMicrocents: 7500000000, + cachedInputPerMtokMicrocents: 150000000, + cacheWritePerMtokMicrocents: 1875000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-1-20250805': { + provider: 'anthropic', + modelId: 'claude-opus-4-1-20250805', + displayName: 'Claude Opus 4.1', + contextWindowTokens: 200000, + maxOutputTokens: 32000, + inputPerMtokMicrocents: 1500000000, + outputPerMtokMicrocents: 7500000000, + cachedInputPerMtokMicrocents: 150000000, + cacheWritePerMtokMicrocents: 1875000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-5': { + provider: 'anthropic', + modelId: 'claude-opus-4-5', + displayName: 'Claude Opus 4.5 (latest)', + contextWindowTokens: 200000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2500000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + reasoning: { effortValues: ['low', 'medium', 'high'], budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-5-20251101': { + provider: 'anthropic', + modelId: 'claude-opus-4-5-20251101', + displayName: 'Claude Opus 4.5', + contextWindowTokens: 200000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2500000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + reasoning: { effortValues: ['low', 'medium', 'high'], budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-6': { + provider: 'anthropic', + modelId: 'claude-opus-4-6', + displayName: 'Claude Opus 4.6', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2500000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + reasoning: { effortValues: ['low', 'medium', 'high', 'max'], budgetTokens: { min: 1024 } }, + }, + 'claude-opus-4-7': { + provider: 'anthropic', + modelId: 'claude-opus-4-7', + displayName: 'Claude Opus 4.7', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2500000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + reasoning: { effortValues: ['low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'claude-opus-4-8': { + provider: 'anthropic', + modelId: 'claude-opus-4-8', + displayName: 'Claude Opus 4.8', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2500000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + reasoning: { effortValues: ['low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'claude-sonnet-4-5': { + provider: 'anthropic', + modelId: 'claude-sonnet-4-5', + displayName: 'Claude Sonnet 4.5 (latest)', + contextWindowTokens: 1000000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 300000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 30000000, + cacheWritePerMtokMicrocents: 375000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-sonnet-4-5-20250929': { + provider: 'anthropic', + modelId: 'claude-sonnet-4-5-20250929', + displayName: 'Claude Sonnet 4.5', + contextWindowTokens: 1000000, + maxOutputTokens: 64000, + inputPerMtokMicrocents: 300000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 30000000, + cacheWritePerMtokMicrocents: 375000000, + reasoning: { budgetTokens: { min: 1024 } }, + }, + 'claude-sonnet-4-6': { + provider: 'anthropic', + modelId: 'claude-sonnet-4-6', + displayName: 'Claude Sonnet 4.6', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 300000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 30000000, + cacheWritePerMtokMicrocents: 375000000, + reasoning: { effortValues: ['low', 'medium', 'high', 'max'], budgetTokens: { min: 1024 } }, + }, + 'claude-sonnet-5': { + provider: 'anthropic', + modelId: 'claude-sonnet-5', + displayName: 'Claude Sonnet 5', + contextWindowTokens: 1000000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 20000000, + cacheWritePerMtokMicrocents: 250000000, + reasoning: { toggle: true, effortValues: ['low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'deepseek-chat': { + provider: 'deepseek', + modelId: 'deepseek-chat', + displayName: 'DeepSeek Chat', + contextWindowTokens: 1000000, + maxOutputTokens: 384000, + inputPerMtokMicrocents: 14000000, + outputPerMtokMicrocents: 28000000, + cachedInputPerMtokMicrocents: 280000, + }, + 'deepseek-reasoner': { + provider: 'deepseek', + modelId: 'deepseek-reasoner', + displayName: 'DeepSeek Reasoner', + contextWindowTokens: 1000000, + maxOutputTokens: 384000, + inputPerMtokMicrocents: 14000000, + outputPerMtokMicrocents: 28000000, + cachedInputPerMtokMicrocents: 280000, + reasoning: {}, + }, + 'deepseek-v4-flash': { + provider: 'deepseek', + modelId: 'deepseek-v4-flash', + displayName: 'DeepSeek V4 Flash', + contextWindowTokens: 1000000, + maxOutputTokens: 384000, + inputPerMtokMicrocents: 14000000, + outputPerMtokMicrocents: 28000000, + cachedInputPerMtokMicrocents: 280000, + reasoning: { toggle: true, effortValues: ['high', 'max'] }, + requestCapabilities: { attachment: false }, + }, + 'deepseek-v4-pro': { + provider: 'deepseek', + modelId: 'deepseek-v4-pro', + displayName: 'DeepSeek V4 Pro', + contextWindowTokens: 1000000, + maxOutputTokens: 384000, + inputPerMtokMicrocents: 43500000, + outputPerMtokMicrocents: 87000000, + cachedInputPerMtokMicrocents: 362500, + reasoning: { toggle: true, effortValues: ['high', 'max'] }, + requestCapabilities: { attachment: false }, + }, + 'gemini-2.0-flash': { + provider: 'gemini', + modelId: 'gemini-2.0-flash', + displayName: 'Gemini 2.0 Flash', + contextWindowTokens: 1048576, + maxOutputTokens: 8192, + inputPerMtokMicrocents: 10000000, + outputPerMtokMicrocents: 40000000, + cachedInputPerMtokMicrocents: 2500000, + }, + 'gemini-2.0-flash-lite': { + provider: 'gemini', + modelId: 'gemini-2.0-flash-lite', + displayName: 'Gemini 2.0 Flash-Lite', + contextWindowTokens: 1048576, + maxOutputTokens: 8192, + inputPerMtokMicrocents: 7500000, + outputPerMtokMicrocents: 30000000, + }, + 'gemini-2.5-flash': { + provider: 'gemini', + modelId: 'gemini-2.5-flash', + displayName: 'Gemini 2.5 Flash', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 30000000, + outputPerMtokMicrocents: 250000000, + cachedInputPerMtokMicrocents: 3000000, + reasoning: { toggle: true, budgetTokens: { min: 0, max: 24576 } }, + }, + 'gemini-2.5-flash-lite': { + provider: 'gemini', + modelId: 'gemini-2.5-flash-lite', + displayName: 'Gemini 2.5 Flash-Lite', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 10000000, + outputPerMtokMicrocents: 40000000, + cachedInputPerMtokMicrocents: 1000000, + reasoning: { toggle: true, budgetTokens: { min: 512, max: 24576 } }, + }, + 'gemini-2.5-pro': { + provider: 'gemini', + modelId: 'gemini-2.5-pro', + displayName: 'Gemini 2.5 Pro', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + contextTiers: [ + { + aboveContextTokens: 200000, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 25000000, + }, + ], + reasoning: { budgetTokens: { min: 128, max: 32768 } }, + }, + 'gemini-3-flash-preview': { + provider: 'gemini', + modelId: 'gemini-3-flash-preview', + displayName: 'Gemini 3 Flash Preview', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 50000000, + outputPerMtokMicrocents: 300000000, + cachedInputPerMtokMicrocents: 5000000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + }, + 'gemini-3-pro-preview': { + provider: 'gemini', + modelId: 'gemini-3-pro-preview', + displayName: 'Gemini 3 Pro Preview', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 1200000000, + cachedInputPerMtokMicrocents: 20000000, + contextTiers: [ + { + aboveContextTokens: 200000, + inputPerMtokMicrocents: 400000000, + outputPerMtokMicrocents: 1800000000, + cachedInputPerMtokMicrocents: 40000000, + }, + ], + reasoning: { effortValues: ['low', 'high'] }, + }, + 'gemini-3.1-flash-lite': { + provider: 'gemini', + modelId: 'gemini-3.1-flash-lite', + displayName: 'Gemini 3.1 Flash Lite', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 25000000, + outputPerMtokMicrocents: 150000000, + cachedInputPerMtokMicrocents: 2500000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + }, + 'gemini-3.1-flash-lite-preview': { + provider: 'gemini', + modelId: 'gemini-3.1-flash-lite-preview', + displayName: 'Gemini 3.1 Flash Lite Preview', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 25000000, + outputPerMtokMicrocents: 150000000, + cachedInputPerMtokMicrocents: 2500000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + }, + 'gemini-3.1-pro-preview': { + provider: 'gemini', + modelId: 'gemini-3.1-pro-preview', + displayName: 'Gemini 3.1 Pro Preview', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 1200000000, + cachedInputPerMtokMicrocents: 20000000, + contextTiers: [ + { + aboveContextTokens: 200000, + inputPerMtokMicrocents: 400000000, + outputPerMtokMicrocents: 1800000000, + cachedInputPerMtokMicrocents: 40000000, + }, + ], + reasoning: { effortValues: ['low', 'medium', 'high'] }, + }, + 'gemini-3.1-pro-preview-customtools': { + provider: 'gemini', + modelId: 'gemini-3.1-pro-preview-customtools', + displayName: 'Gemini 3.1 Pro Preview Custom Tools', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 1200000000, + cachedInputPerMtokMicrocents: 20000000, + contextTiers: [ + { + aboveContextTokens: 200000, + inputPerMtokMicrocents: 400000000, + outputPerMtokMicrocents: 1800000000, + cachedInputPerMtokMicrocents: 40000000, + }, + ], + reasoning: { effortValues: ['low', 'medium', 'high'] }, + }, + 'gemini-3.5-flash': { + provider: 'gemini', + modelId: 'gemini-3.5-flash', + displayName: 'Gemini 3.5 Flash', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 150000000, + outputPerMtokMicrocents: 900000000, + cachedInputPerMtokMicrocents: 15000000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + }, + 'gemini-flash-latest': { + provider: 'gemini', + modelId: 'gemini-flash-latest', + displayName: 'Gemini Flash Latest', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 30000000, + outputPerMtokMicrocents: 250000000, + cachedInputPerMtokMicrocents: 7500000, + reasoning: { toggle: true, budgetTokens: { min: 0, max: 24576 } }, + }, + 'gemini-flash-lite-latest': { + provider: 'gemini', + modelId: 'gemini-flash-lite-latest', + displayName: 'Gemini Flash-Lite Latest', + contextWindowTokens: 1048576, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 10000000, + outputPerMtokMicrocents: 40000000, + cachedInputPerMtokMicrocents: 2500000, + reasoning: { toggle: true, budgetTokens: { min: 512, max: 24576 } }, + }, + 'gemini-omni-flash-preview': { + provider: 'gemini', + modelId: 'gemini-omni-flash-preview', + displayName: 'Gemini Omni Flash Preview', + contextWindowTokens: 131072, + maxOutputTokens: 65536, + inputPerMtokMicrocents: 150000000, + outputPerMtokMicrocents: 1750000000, + reasoning: {}, + requestCapabilities: { toolCall: false }, + }, + 'gpt-3.5-turbo': { + provider: 'openai', + modelId: 'gpt-3.5-turbo', + displayName: 'GPT-3.5-turbo', + contextWindowTokens: 16385, + maxOutputTokens: 4096, + inputPerMtokMicrocents: 50000000, + outputPerMtokMicrocents: 150000000, + cachedInputPerMtokMicrocents: 0, + requestCapabilities: { toolCall: false, structuredOutput: false, attachment: false }, + }, + 'gpt-4': { + provider: 'openai', + modelId: 'gpt-4', + displayName: 'GPT-4', + contextWindowTokens: 8192, + maxOutputTokens: 8192, + inputPerMtokMicrocents: 3000000000, + outputPerMtokMicrocents: 6000000000, + requestCapabilities: { structuredOutput: false }, + }, + 'gpt-4-turbo': { + provider: 'openai', + modelId: 'gpt-4-turbo', + displayName: 'GPT-4 Turbo', + contextWindowTokens: 128000, + maxOutputTokens: 4096, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 3000000000, + requestCapabilities: { structuredOutput: false }, + }, + 'gpt-4.1': { + provider: 'openai', + modelId: 'gpt-4.1', + displayName: 'GPT-4.1', + contextWindowTokens: 1047576, + maxOutputTokens: 32768, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 800000000, + cachedInputPerMtokMicrocents: 50000000, + }, + 'gpt-4.1-mini': { + provider: 'openai', + modelId: 'gpt-4.1-mini', + displayName: 'GPT-4.1 mini', + contextWindowTokens: 1047576, + maxOutputTokens: 32768, + inputPerMtokMicrocents: 40000000, + outputPerMtokMicrocents: 160000000, + cachedInputPerMtokMicrocents: 10000000, + }, + 'gpt-4.1-nano': { + provider: 'openai', + modelId: 'gpt-4.1-nano', + displayName: 'GPT-4.1 nano', + contextWindowTokens: 1047576, + maxOutputTokens: 32768, + inputPerMtokMicrocents: 10000000, + outputPerMtokMicrocents: 40000000, + cachedInputPerMtokMicrocents: 2500000, + }, + 'gpt-4o': { + provider: 'openai', + modelId: 'gpt-4o', + displayName: 'GPT-4o', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 125000000, + }, + 'gpt-4o-2024-05-13': { + provider: 'openai', + modelId: 'gpt-4o-2024-05-13', + displayName: 'GPT-4o (2024-05-13)', + contextWindowTokens: 128000, + maxOutputTokens: 4096, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 1500000000, + }, + 'gpt-4o-2024-08-06': { + provider: 'openai', + modelId: 'gpt-4o-2024-08-06', + displayName: 'GPT-4o (2024-08-06)', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 125000000, + }, + 'gpt-4o-2024-11-20': { + provider: 'openai', + modelId: 'gpt-4o-2024-11-20', + displayName: 'GPT-4o (2024-11-20)', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 125000000, + }, + 'gpt-4o-mini': { + provider: 'openai', + modelId: 'gpt-4o-mini', + displayName: 'GPT-4o mini', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 15000000, + outputPerMtokMicrocents: 60000000, + cachedInputPerMtokMicrocents: 7500000, + }, + 'gpt-5': { + provider: 'openai', + modelId: 'gpt-5', + displayName: 'GPT-5', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5-chat-latest': { + provider: 'openai', + modelId: 'gpt-5-chat-latest', + displayName: 'GPT-5 Chat (latest)', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: {}, + requestCapabilities: { toolCall: false }, + }, + 'gpt-5-codex': { + provider: 'openai', + modelId: 'gpt-5-codex', + displayName: 'GPT-5-Codex', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false, attachment: false }, + }, + 'gpt-5-mini': { + provider: 'openai', + modelId: 'gpt-5-mini', + displayName: 'GPT-5 Mini', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 25000000, + outputPerMtokMicrocents: 200000000, + cachedInputPerMtokMicrocents: 2500000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5-nano': { + provider: 'openai', + modelId: 'gpt-5-nano', + displayName: 'GPT-5 Nano', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 5000000, + outputPerMtokMicrocents: 40000000, + cachedInputPerMtokMicrocents: 500000, + reasoning: { effortValues: ['minimal', 'low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5-pro': { + provider: 'openai', + modelId: 'gpt-5-pro', + displayName: 'GPT-5 Pro', + contextWindowTokens: 400000, + maxOutputTokens: 272000, + inputPerMtokMicrocents: 1500000000, + outputPerMtokMicrocents: 12000000000, + reasoning: { effortValues: ['high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.1': { + provider: 'openai', + modelId: 'gpt-5.1', + displayName: 'GPT-5.1', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.1-chat-latest': { + provider: 'openai', + modelId: 'gpt-5.1-chat-latest', + displayName: 'GPT-5.1 Chat', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['medium'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.1-codex': { + provider: 'openai', + modelId: 'gpt-5.1-codex', + displayName: 'GPT-5.1 Codex', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.1-codex-max': { + provider: 'openai', + modelId: 'gpt-5.1-codex-max', + displayName: 'GPT-5.1 Codex Max', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 125000000, + outputPerMtokMicrocents: 1000000000, + cachedInputPerMtokMicrocents: 12500000, + reasoning: { effortValues: ['low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.1-codex-mini': { + provider: 'openai', + modelId: 'gpt-5.1-codex-mini', + displayName: 'GPT-5.1 Codex mini', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 25000000, + outputPerMtokMicrocents: 200000000, + cachedInputPerMtokMicrocents: 2500000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.2': { + provider: 'openai', + modelId: 'gpt-5.2', + displayName: 'GPT-5.2', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.2-chat-latest': { + provider: 'openai', + modelId: 'gpt-5.2-chat-latest', + displayName: 'GPT-5.2 Chat', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + reasoning: { effortValues: ['medium'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.2-codex': { + provider: 'openai', + modelId: 'gpt-5.2-codex', + displayName: 'GPT-5.2 Codex', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + reasoning: { effortValues: ['low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.2-pro': { + provider: 'openai', + modelId: 'gpt-5.2-pro', + displayName: 'GPT-5.2 Pro', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 2100000000, + outputPerMtokMicrocents: 16800000000, + reasoning: { effortValues: ['medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false, structuredOutput: false }, + }, + 'gpt-5.3-chat-latest': { + provider: 'openai', + modelId: 'gpt-5.3-chat-latest', + displayName: 'GPT-5.3 Chat (latest)', + contextWindowTokens: 128000, + maxOutputTokens: 16384, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + }, + 'gpt-5.3-codex': { + provider: 'openai', + modelId: 'gpt-5.3-codex', + displayName: 'GPT-5.3 Codex', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.3-codex-spark': { + provider: 'openai', + modelId: 'gpt-5.3-codex-spark', + displayName: 'GPT-5.3 Codex Spark', + contextWindowTokens: 128000, + maxOutputTokens: 32000, + inputPerMtokMicrocents: 175000000, + outputPerMtokMicrocents: 1400000000, + cachedInputPerMtokMicrocents: 17500000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.4': { + provider: 'openai', + modelId: 'gpt-5.4', + displayName: 'GPT-5.4', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 25000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2250000000, + cachedInputPerMtokMicrocents: 50000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.4-mini': { + provider: 'openai', + modelId: 'gpt-5.4-mini', + displayName: 'GPT-5.4 mini', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 75000000, + outputPerMtokMicrocents: 450000000, + cachedInputPerMtokMicrocents: 7500000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.4-nano': { + provider: 'openai', + modelId: 'gpt-5.4-nano', + displayName: 'GPT-5.4 nano', + contextWindowTokens: 400000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 20000000, + outputPerMtokMicrocents: 125000000, + cachedInputPerMtokMicrocents: 2000000, + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.4-pro': { + provider: 'openai', + modelId: 'gpt-5.4-pro', + displayName: 'GPT-5.4 Pro', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 3000000000, + outputPerMtokMicrocents: 18000000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 6000000000, + outputPerMtokMicrocents: 27000000000, + }, + ], + reasoning: { effortValues: ['medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false, structuredOutput: false }, + }, + 'gpt-5.5': { + provider: 'openai', + modelId: 'gpt-5.5', + displayName: 'GPT-5.5', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 3000000000, + cachedInputPerMtokMicrocents: 50000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 4500000000, + cachedInputPerMtokMicrocents: 100000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.5-pro': { + provider: 'openai', + modelId: 'gpt-5.5-pro', + displayName: 'GPT-5.5 Pro', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 3000000000, + outputPerMtokMicrocents: 18000000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 6000000000, + outputPerMtokMicrocents: 27000000000, + }, + ], + reasoning: { effortValues: ['medium', 'high', 'xhigh'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.6': { + provider: 'openai', + modelId: 'gpt-5.6', + displayName: 'GPT-5.6', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 3000000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 4500000000, + cachedInputPerMtokMicrocents: 100000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.6-luna': { + provider: 'openai', + modelId: 'gpt-5.6-luna', + displayName: 'GPT-5.6 Luna', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 100000000, + outputPerMtokMicrocents: 600000000, + cachedInputPerMtokMicrocents: 10000000, + cacheWritePerMtokMicrocents: 125000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 900000000, + cachedInputPerMtokMicrocents: 20000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.6-sol': { + provider: 'openai', + modelId: 'gpt-5.6-sol', + displayName: 'GPT-5.6 Sol', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 3000000000, + cachedInputPerMtokMicrocents: 50000000, + cacheWritePerMtokMicrocents: 625000000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 4500000000, + cachedInputPerMtokMicrocents: 100000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + 'gpt-5.6-terra': { + provider: 'openai', + modelId: 'gpt-5.6-terra', + displayName: 'GPT-5.6 Terra', + contextWindowTokens: 1050000, + maxOutputTokens: 128000, + inputPerMtokMicrocents: 250000000, + outputPerMtokMicrocents: 1500000000, + cachedInputPerMtokMicrocents: 25000000, + cacheWritePerMtokMicrocents: 312500000, + contextTiers: [ + { + aboveContextTokens: 272000, + inputPerMtokMicrocents: 500000000, + outputPerMtokMicrocents: 2250000000, + cachedInputPerMtokMicrocents: 50000000, + }, + ], + reasoning: { effortValues: ['none', 'low', 'medium', 'high', 'xhigh', 'max'] }, + requestCapabilities: { temperature: false }, + }, + o1: { + provider: 'openai', + modelId: 'o1', + displayName: 'o1', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 1500000000, + outputPerMtokMicrocents: 6000000000, + cachedInputPerMtokMicrocents: 750000000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'o1-pro': { + provider: 'openai', + modelId: 'o1-pro', + displayName: 'o1-pro', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 15000000000, + outputPerMtokMicrocents: 60000000000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + o3: { + provider: 'openai', + modelId: 'o3', + displayName: 'o3', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 800000000, + cachedInputPerMtokMicrocents: 50000000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'o3-deep-research': { + provider: 'openai', + modelId: 'o3-deep-research', + displayName: 'o3-deep-research', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 1000000000, + outputPerMtokMicrocents: 4000000000, + cachedInputPerMtokMicrocents: 250000000, + reasoning: { effortValues: ['medium'] }, + requestCapabilities: { temperature: false }, + }, + 'o3-mini': { + provider: 'openai', + modelId: 'o3-mini', + displayName: 'o3-mini', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 110000000, + outputPerMtokMicrocents: 440000000, + cachedInputPerMtokMicrocents: 55000000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false, attachment: false }, + }, + 'o3-pro': { + provider: 'openai', + modelId: 'o3-pro', + displayName: 'o3-pro', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 2000000000, + outputPerMtokMicrocents: 8000000000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'o4-mini': { + provider: 'openai', + modelId: 'o4-mini', + displayName: 'o4-mini', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 110000000, + outputPerMtokMicrocents: 440000000, + cachedInputPerMtokMicrocents: 27500000, + reasoning: { effortValues: ['low', 'medium', 'high'] }, + requestCapabilities: { temperature: false }, + }, + 'o4-mini-deep-research': { + provider: 'openai', + modelId: 'o4-mini-deep-research', + displayName: 'o4-mini-deep-research', + contextWindowTokens: 200000, + maxOutputTokens: 100000, + inputPerMtokMicrocents: 200000000, + outputPerMtokMicrocents: 800000000, + cachedInputPerMtokMicrocents: 50000000, + reasoning: { effortValues: ['medium'] }, + requestCapabilities: { temperature: false }, + }, +}; + +/** The SHA-256 of this catalog's own data — changes when, and only when, what we ship changes. */ +export const CATALOG_SHA256 = '4db066a3de7bf27d6f54955958a5cb8f60185826dfe9c77aef7aa0e1a370fdc0'; diff --git a/packages/llm/src/conformance/effort.conformance.test.ts b/packages/llm/src/conformance/effort.conformance.test.ts new file mode 100644 index 00000000..a49465fe --- /dev/null +++ b/packages/llm/src/conformance/effort.conformance.test.ts @@ -0,0 +1,184 @@ +import { describe, expect, it } from 'vitest'; + +import { createAnthropicAdapter } from '../adapters/anthropic.js'; +import { createGeminiAdapter } from '../adapters/gemini.js'; +import { createOpenAiAdapter } from '../adapters/openai.js'; +import { catalogModel } from '../catalog/lookup.js'; +import { CATALOG_SNAPSHOT } from '../catalog/snapshot.js'; +import { acceptedTiers, wireValueFor } from '../reasoning-wire.js'; +import type { ReasoningEffort } from '@relavium/shared'; + +import type { LlmProvider } from '../types.js'; + +/** + * THE ONLY MECHANISM THAT CATCHES A STALE CATALOG RE-INTRODUCING THE REASONING BUG (ADR-0071 §9). + * + * The whole point of this work is that a model DECLARES which effort tiers it accepts, and the adapters send only + * those. But the catalog is a snapshot: models.dev could change a model's `reasoning_options`, or a provider could + * tighten what it accepts, and our shipped snapshot would go on claiming a tier the model now rejects — silently, + * the exact Gemini bug this ADR fixed, back from the dead. Nothing offline can see it: the fixtures replay what we + * recorded, and a unit test proves the mapping is internally consistent, not that the PROVIDER agrees. + * + * So this asks the real API. For each shipped reasoning model, it sends EVERY tier the catalog says the model + * accepts and asserts the provider does not reject it. A 400 here means the catalog is lying about that model — + * which is precisely what a stale snapshot looks like, and precisely what a one-off manual probe cannot keep + * catching (it proves a fact once; the catalog drifts continuously). + * + * Key-gated and nightly (testing.md): skipped with no key, so it never gates a PR. `off` is excluded — disabling + * reasoning is a different wire shape whose acceptance the adapters already gate, and sending it proves nothing + * about the effort ladder this test exists to pin. + */ + +/** A representative reasoning model per provider — one with a non-trivial effort ladder to actually exercise. */ +const PROBE: Record<'anthropic' | 'openai' | 'gemini' | 'deepseek', string> = { + anthropic: 'claude-opus-4-8', + openai: 'gpt-5.5', + // EFFORT-shaped, deliberately. `gemini-2.5-flash` is budget-shaped, so the adapter sends `thinkingBudget` + // (always accepted) and NEVER `thinkingLevel` — the one wire value that can 400 on tier drift. A budget probe + // is a tautology, and Gemini is the provider whose `thinkingLevel` acceptance was the ORIGINAL bug (§Context/3). + gemini: 'gemini-3.5-flash', + deepseek: 'deepseek-v4-pro', +}; + +/** The graded tiers the catalog claims a model accepts (never `off` — see the file header). */ +function gradedTiersFor(modelId: string): Exclude[] { + const entry = catalogModel(modelId); + if (entry === undefined) return []; + return [...acceptedTiers(entry.provider, entry.reasoning)].filter( + (t): t is Exclude => t !== 'off', + ); +} + +/** + * The graded tiers to actually SEND — one per distinct WIRE value. Several normalized tiers collapse onto one + * provider value (DeepSeek's `low`/`medium`/`high` all map to `reasoning_effort: 'high'`, Gemini's `max` onto + * `HIGH`), so probing every tier would fire real, billed calls proving the identical fact. Probing each distinct + * wire value once tests everything the ladder can express, and nothing twice. + */ +function distinctWireTiers(modelId: string): Exclude[] { + const entry = catalogModel(modelId); + if (entry === undefined) return []; + const seen = new Set(); + const tiers: Exclude[] = []; + for (const tier of gradedTiersFor(modelId)) { + const wire = wireValueFor(entry.provider, tier); + if (wire !== undefined && !seen.has(wire)) { + seen.add(wire); + tiers.push(tier); + } + } + return tiers; +} + +/** Send one tier at the real API and assert it is not rejected — a returned result, non-empty output. */ +async function assertTierAccepted( + adapter: LlmProvider, + model: string, + tier: ReasoningEffort, + key: string, +): Promise { + const result = await adapter.generate( + { + model, + maxTokens: 64, + reasoningEffort: tier, + messages: [{ role: 'user', content: [{ type: 'text', text: 'Reply with one word.' }] }], + }, + key, + ); + // The assertion is "the provider did not 400 on the tier" — `generate` throwing an LlmError is the failure. A + // returned result with output tokens is proof the tier rode. + expect(result.usage.outputTokens, `${model} rejected tier '${tier}'`).toBeGreaterThan(0); +} + +const anthropicKey = process.env['ANTHROPIC_API_KEY'] ?? ''; +describe('anthropic — effort conformance (live, nightly): the catalog does not lie about accepted tiers', () => { + const model = PROBE.anthropic; + const tiers = gradedTiersFor(model); // the full claimed ladder — named in the test title + const probeTiers = distinctWireTiers(model); // deduped by wire value — what we actually send + it.skipIf(anthropicKey === '' || tiers.length === 0)( + `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, + async () => { + const adapter = createAnthropicAdapter({ maxRetries: 0 }); + for (const tier of probeTiers) { + await assertTierAccepted(adapter, model, tier, anthropicKey); + } + }, + ); +}); + +const openaiKey = process.env['OPENAI_API_KEY'] ?? ''; +describe('openai — effort conformance (live, nightly): the catalog does not lie about accepted tiers', () => { + const model = PROBE.openai; + const tiers = gradedTiersFor(model); // the full claimed ladder — named in the test title + const probeTiers = distinctWireTiers(model); // deduped by wire value — what we actually send + it.skipIf(openaiKey === '' || tiers.length === 0)( + `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, + async () => { + const adapter = createOpenAiAdapter({ providerId: 'openai', maxRetries: 0 }); + for (const tier of probeTiers) { + await assertTierAccepted(adapter, model, tier, openaiKey); + } + }, + ); +}); + +const geminiKey = process.env['GEMINI_API_KEY'] ?? ''; +describe('gemini — effort conformance (live, nightly): the catalog does not lie about accepted tiers', () => { + const model = PROBE.gemini; + const tiers = gradedTiersFor(model); // the full claimed ladder — named in the test title + const probeTiers = distinctWireTiers(model); // deduped by wire value — what we actually send + it.skipIf(geminiKey === '' || tiers.length === 0)( + `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, + async () => { + const adapter = createGeminiAdapter(); + for (const tier of probeTiers) { + await assertTierAccepted(adapter, model, tier, geminiKey); + } + }, + ); +}); + +const deepseekKey = process.env['DEEPSEEK_API_KEY'] ?? ''; +describe('deepseek — effort conformance (live, nightly): the catalog does not lie about accepted tiers', () => { + const model = PROBE.deepseek; + const tiers = gradedTiersFor(model); // the full claimed ladder — named in the test title + const probeTiers = distinctWireTiers(model); // deduped by wire value — what we actually send + it.skipIf(deepseekKey === '' || tiers.length === 0)( + `${model} accepts every tier the catalog claims (${tiers.join(', ')})`, + async () => { + const adapter = createOpenAiAdapter({ providerId: 'deepseek', maxRetries: 0 }); + for (const tier of probeTiers) { + await assertTierAccepted(adapter, model, tier, deepseekKey); + } + }, + ); +}); + +/** + * The OFFLINE half — this DOES gate a PR (no key needed). The live probes above can only run for a model the + * catalog carries; if a PROBE id ever drifts out of the snapshot, the live test silently skips (empty tier list) + * and the drift goes unnoticed. This pins the probes to real shipped models so a rename can't quietly disable the + * one mechanism §9 relies on. + */ +describe('effort conformance — the probe models are real, so the live lane cannot silently no-op', () => { + it('every probe id is a shipped reasoning model with a non-empty effort ladder', () => { + for (const [provider, modelId] of Object.entries(PROBE)) { + const entry = CATALOG_SNAPSHOT[modelId]; + expect(entry, `${provider} probe '${modelId}' is not in the catalog`).toBeDefined(); + expect(entry?.provider).toBe(provider); + // EFFORT-shaped, not merely "has a ladder". A budget-shaped model ALSO reports a full graded ladder (the + // budget axis fills every tier), so `length > 0` would have passed the tautological `gemini-2.5-flash` probe + // the fold just removed. The live guard only bites when the adapter sends an effort WIRE VALUE the provider + // can reject — which requires `effortValues`. Pinning it here is what stops a future edit from silently + // reverting to a budget probe on `gemini`/`anthropic` (the two providers a budget axis covers). + expect( + entry?.reasoning?.effortValues, + `${modelId} is budget-shaped — a budget probe can never 400 on tier drift, so it is a tautology`, + ).toBeDefined(); + expect(distinctWireTiers(modelId).length, `${modelId} has no tiers to probe`).toBeGreaterThan( + 0, + ); + } + }); +}); diff --git a/packages/llm/src/conformance/fixtures/anthropic.ts b/packages/llm/src/conformance/fixtures/anthropic.ts index 8f71c872..a78fec64 100644 --- a/packages/llm/src/conformance/fixtures/anthropic.ts +++ b/packages/llm/src/conformance/fixtures/anthropic.ts @@ -264,7 +264,10 @@ const modelsList = JSON.stringify({ capabilities: null, }, { - id: 'claude-haiku-4-5', + // The DATED PIN, not the rolling alias — real Anthropic `models.list()` returns the pinned snapshot + // (`claude-haiku-4-5-20251001`), and the rolling alias (`claude-haiku-4-5`) is the id the merge's + // alias↔dated-pin equivalence rescues from a wrong `not-on-key` dim (model-catalog.ts, ADR-0064 §6 amendment). + id: 'claude-haiku-4-5-20251001', type: 'model', display_name: 'Claude Haiku 4.5', created_at: '2025-11-01T00:00:00Z', @@ -275,7 +278,7 @@ const modelsList = JSON.stringify({ ], has_more: false, first_id: 'claude-opus-4-8', - last_id: 'claude-haiku-4-5', + last_id: 'claude-haiku-4-5-20251001', }); // The drift fixture (ADR-0064 §8): one row carries an unknown future field (ignored), one row has NO id @@ -332,7 +335,7 @@ export const ANTHROPIC_FIXTURES: ConformanceFixtures = { reasoningStream: { text: 'let me think', reasoningTokens: 4 }, structuredOutput: { text: '{"ok":true}' }, listModels: { - ids: ['claude-opus-4-8', 'claude-haiku-4-5'], + ids: ['claude-opus-4-8', 'claude-haiku-4-5-20251001'], // The rich row: display name + context (max_input_tokens) + output (max_tokens) all mapped. sample: { id: 'claude-opus-4-8', diff --git a/packages/llm/src/conformance/fixtures/catalog.ts b/packages/llm/src/conformance/fixtures/catalog.ts new file mode 100644 index 00000000..2114f8d7 --- /dev/null +++ b/packages/llm/src/conformance/fixtures/catalog.ts @@ -0,0 +1,28 @@ +import type { CatalogModel } from '../../catalog/catalog-model.js'; + +/** + * A minimal, well-formed {@link CatalogModel} row for a test that needs to INSTALL one (via + * `installCatalogRefresh`) rather than lean on the shipped snapshot — the per-model capability gates, the + * alias↔dated-pin scoping tests, and the adapter request-building tests all do. + * + * The defaults are only there to be valid: a positive price on BOTH sides (the refresh floor rejects a `0` on + * either — an unpriced row is not an enrichment), a real context/output window, and a display name. No test asserts + * on them; every one that cares overrides the field it cares about. `provider` defaults to `openai` and is the + * override each adapter's suite passes. + * + * Lives under `src/conformance/` because that path is EXCLUDED from `tsconfig.build.json` — a test-only helper must + * not ship in the published `dist`. + */ +export function catalogModelFixture( + over: Partial & Pick, +): CatalogModel { + return { + provider: 'openai', + displayName: over.modelId, + contextWindowTokens: 100_000, + maxOutputTokens: 10_000, + inputPerMtokMicrocents: 1_000_000, + outputPerMtokMicrocents: 2_000_000, + ...over, + }; +} diff --git a/packages/llm/src/cost-tracker.test.ts b/packages/llm/src/cost-tracker.test.ts index 8ca02cfb..f532e044 100644 --- a/packages/llm/src/cost-tracker.test.ts +++ b/packages/llm/src/cost-tracker.test.ts @@ -2,7 +2,8 @@ import { describe, expect, it } from 'vitest'; import { CostTracker, cost, mediaCost, priceModel } from './cost-tracker.js'; import { UnknownModelError } from './errors.js'; -import { KNOWN_MODEL_IDS, MODEL_PRICING, type ModelPricing } from './pricing.js'; +import { catalogPricing, pricedModelIds } from './catalog/pricing.js'; +import type { ModelPricing } from './pricing.js'; /** A throwaway priced model carrying media-output rates — no 1.AF table row has them, so tests construct one. */ const PRICED_MEDIA: ModelPricing = { @@ -49,7 +50,7 @@ describe('priceModel', () => { if (err instanceof UnknownModelError) { expect(err.code).toBe('unknown_model'); expect(err.modelId).toBe('gpt-9-ultra'); - expect(err.knownModels).toEqual(KNOWN_MODEL_IDS); + expect(err.knownModels).toEqual(pricedModelIds()); } } }); @@ -221,10 +222,18 @@ describe('user-pricing overlay (2.5.G S10, ADR-0065 §2)', () => { expect(p.provider).toBe('openai'); }); - it('priceModel keeps the STATIC registry authoritative for a known id even when the overlay collides', () => { + it('THE FLIP: the USER outranks the catalog, even for a model the catalog knows', () => { + // This test asserted the OPPOSITE until ADR-0071 §1, and the reversal is deliberate. + // + // Registry-first made sense while the registry was our own hand-verified table: a user could not misprice a + // shipped model. The catalog is a snapshot of a third-party aggregator, regenerated from models.dev — and the + // user is the one holding the invoice. Their negotiated rate, their enterprise discount, or simply a price our + // snapshot has not caught up with is not a hint for a generated file to overrule. const p = priceModel('claude-opus-4-8', OVERLAY); - expect(p.displayName).toBe('Claude Opus 4.8'); // the static row, not the tampered overlay - expect(p.inputPerMtokMicrocents).toBe(500_000_000); // $5/MTok, not the overlay's 1µ¢ + expect(p.displayName).toBe('Tampered Opus'); // the user's row, not the catalog's + expect(p.inputPerMtokMicrocents).toBe(1); // …and their price is what we bill at + // …while the catalog still answers for a model they have NOT priced. + expect(priceModel('claude-opus-4-8').inputPerMtokMicrocents).toBeGreaterThan(1); }); it('priceModel still throws UnknownModelError for an id absent from BOTH tiers', () => { @@ -257,11 +266,17 @@ describe('user-pricing overlay (2.5.G S10, ADR-0065 §2)', () => { }); }); -describe('MODEL_PRICING table invariants (the values seeded into model_catalog)', () => { - it('keys match KNOWN_MODEL_IDS and every catalog-projection field is complete + integer', () => { - const byLocale = (a: string, b: string): number => a.localeCompare(b); - expect(Object.keys(MODEL_PRICING).sort(byLocale)).toEqual([...KNOWN_MODEL_IDS].sort(byLocale)); - const rows: Array<[string, ModelPricing]> = Object.entries(MODEL_PRICING); +describe("the priced catalog's invariants (the values seeded into model_catalog)", () => { + it('every priced model is complete, and every price is an integer micro-cent', () => { + // The same invariant the hand-typed table was held to, now applied to eighty generated rows instead of twelve. + // It matters MORE, not less: a generated file is only as good as its generator's guards, and a float price or a + // negative rate would silently corrupt every cost figure downstream of it. + const rows: Array<[string, ModelPricing]> = pricedModelIds().map((id) => { + const priced = catalogPricing(id); + if (priced === undefined) throw new Error(`${id} is in pricedModelIds but has no price`); + return [id, priced]; + }); + expect(rows.length).toBeGreaterThan(50); for (const [id, row] of rows) { expect(row.nativeId.length, id).toBeGreaterThan(0); expect(row.displayName.length, id).toBeGreaterThan(0); diff --git a/packages/llm/src/cost-tracker.ts b/packages/llm/src/cost-tracker.ts index a69062a7..dd924374 100644 --- a/packages/llm/src/cost-tracker.ts +++ b/packages/llm/src/cost-tracker.ts @@ -1,12 +1,8 @@ import type { MediaBilledModality } from '@relavium/shared'; +import { catalogPricing, pricedModelIds } from './catalog/pricing.js'; import { UnknownModelError } from './errors.js'; -import { - isCanonicalModelId, - KNOWN_MODEL_IDS, - MODEL_PRICING, - type ModelPricing, -} from './pricing.js'; +import type { ModelPricing } from './pricing.js'; import type { MediaUnitsEntry, Usage } from './types.js'; /** @@ -19,30 +15,101 @@ import type { MediaUnitsEntry, Usage } from './types.js'; * An optional **user-pricing overlay** (2.5.G S10, [ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md) §2) * — canonical model id → {@link ModelPricing}, host-built from the `model_catalog` `source='user'` rows and * injected **exactly like `keyFor`** (a plain Relavium map; `@relavium/core`/`@relavium/llm` never import - * `@relavium/db`). It fills the price of an id ABSENT from the static registry so `max_cost_microcents` can - * enforce it; the static registry ALWAYS wins for a known id (a user can never misprice a shipped model). + * `@relavium/db`). It OUTRANKS the catalog (ADR-0071 §1): the catalog is a generated snapshot of a third-party + * aggregator, and the user is the one holding the invoice. A partial override stays partial — a dimension they did + * not state inherits the catalog's, and a cache read is never free. `models pricing` says out loud which catalog + * price it replaced, because the user gets what they asked for; they simply cannot do it in silence (§5). */ export type PricingOverlay = ReadonlyMap; /** - * Look up pricing for a model id with precedence **static → overlay → throw** (ADR-0065 §2): the static - * {@link MODEL_PRICING} wins for a known canonical id; the optional user `overlay` fills an UNKNOWN id; a truly - * unknown id throws `UnknownModelError` (never a silent zero — the caller degrades cost governance to `allow` - * with a loud, visible notice). + * Look up pricing for a model id: **user → catalog → throw** + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §1). + * + * **The precedence flipped, deliberately.** It used to be static-first: the hand-typed registry won for a known + * id, and a user's `models pricing` row could only fill an id the registry LACKED. That made sense while the + * registry was our own verified table — and it is exactly wrong now. The catalog is a snapshot of a third-party + * aggregator; the user is the one holding the invoice. When they tell us their negotiated rate, their enterprise + * discount, or a price the snapshot has not caught up with, that is not a hint to be overruled by a file we + * generated last Tuesday. The user's number wins. + * + * A truly unknown id throws `UnknownModelError` — never a silent zero, which would bill a run at nothing and + * enforce a cost cap that can never trip. The caller degrades cost governance to `allow`, loudly. */ export function priceModel(modelId: string, overlay?: PricingOverlay): ModelPricing { - if (isCanonicalModelId(modelId)) { - return MODEL_PRICING[modelId]; // the static registry is the pricing authority for a known id + const fromUser = overlay?.get(modelId); + if (fromUser !== undefined) { + return fromUser; // the user holds the invoice; we hold a snapshot of someone else's table } - const fromOverlay = overlay?.get(modelId); - if (fromOverlay !== undefined) { - return fromOverlay; // the user tier fills an id the static registry does not carry + const fromCatalog = catalogPricing(modelId); + if (fromCatalog !== undefined) { + return fromCatalog; } - throw new UnknownModelError(modelId, KNOWN_MODEL_IDS); + throw new UnknownModelError(modelId, pricedModelIds()); } const TOKENS_PER_MTOK = 1_000_000; +/** The three input-side rates that a context tier can move. Output moves too; it rides alongside. */ +export interface Rates { + readonly input: number; + readonly output: number; + readonly cachedInput: number; +} + +/** + * The rates for a prompt of `contextTokens` — the tier it actually landed in (ADR-0071 §11). + * + * A model with no tiers is flat, which is every model the retired table carried and every price a user states. For + * a tiered one, the HIGHEST threshold at or below the prompt's size wins: `gemini-2.5-pro` is $1.25/$10 up to 200k + * and $2.50/$15 above it, so a 500k-token prompt billed at the cheap rate under-states its own cost by 2×. + * + * `contextTokens` is the whole input side — the prompt, cached or not, plus what is being written INTO the cache. + * All of it is context the model has to hold this turn, and it is a long conversation's cached history that pushes + * it over the threshold in the first place. + * + * One gap, deliberately not papered over: **cache WRITES are billed at the flat rate**, never a tier's. models.dev's + * tier schema publishes `input`, `output` and `cache_read` — and no `cache_write` — so a per-tier write rate is not a + * number we have. Inventing one by scaling would be a guess on a money path. Filed in deferred-tasks; the exposure is + * a cache-write-heavy prompt above 272k on the four `gpt-5.6` variants. + */ +function ratesFor(p: ModelPricing, contextTokens: number): Rates { + const flat: Rates = { + input: p.inputPerMtokMicrocents, + output: p.outputPerMtokMicrocents, + cachedInput: p.cachedInputPerMtokMicrocents, + }; + if (p.contextTiers === undefined || p.contextTiers.length === 0) return flat; + let best: Rates = flat; + let bestThreshold = -1; + for (const tier of p.contextTiers) { + if (contextTokens > tier.aboveContextTokens && tier.aboveContextTokens > bestThreshold) { + bestThreshold = tier.aboveContextTokens; + best = { + input: tier.inputPerMtokMicrocents, + output: tier.outputPerMtokMicrocents, + // A tier that states no cache rate of its own falls back to THAT TIER's input rate — the same rule the base + // level follows (ADR-0071 §10), and for the same reason: a cache read is never free just because nobody said + // what it costs. It does NOT carry the base tier's discount forward, and it does not need to: every shipped + // model with a real base cache discount states one at the tier level too. + cachedInput: tier.cachedInputPerMtokMicrocents ?? tier.inputPerMtokMicrocents, + }; + } + } + return best; +} + +/** + * The rates for the WORST case — the highest tier the model has (ADR-0071 §11). + * + * The pre-egress estimate does not know how long the prompt will be (the engine does not tokenize locally), so it + * assumes the expensive end. A cap that over-estimates refuses a turn the user could have afforded; a cap that + * under-estimates lets real money escape. Only one of those is recoverable. + */ +export function worstCaseRates(p: ModelPricing): Rates { + return ratesFor(p, Number.MAX_SAFE_INTEGER); +} + /** * The integer micro-cent cost of one usage record. Each token class is `tokens × * pricePerMtokMicrocents / 1e6`, rounded once (the only float in the path; everything downstream @@ -57,12 +124,15 @@ export function cost(modelId: string, usage: Usage, overlay?: PricingOverlay): n const p = priceModel(modelId, overlay); const cacheReadTokens = usage.cacheReadTokens ?? 0; const cacheWriteTokens = usage.cacheWriteTokens ?? 0; + // The tier the prompt ACTUALLY landed in (ADR-0071 §11) — the whole input side, cached or not, is context the + // model had to hold. A flat-priced model (and every user-stated price) resolves to its one set of rates. + const rates = ratesFor(p, usage.inputTokens + cacheReadTokens + cacheWriteTokens); const perClass = (tokens: number, ratePerMtok: number): number => Math.round((tokens * ratePerMtok) / TOKENS_PER_MTOK); return ( - perClass(usage.inputTokens, p.inputPerMtokMicrocents) + - perClass(usage.outputTokens, p.outputPerMtokMicrocents) + - perClass(cacheReadTokens, p.cachedInputPerMtokMicrocents) + + perClass(usage.inputTokens, rates.input) + + perClass(usage.outputTokens, rates.output) + + perClass(cacheReadTokens, rates.cachedInput) + perClass(cacheWriteTokens, p.cacheWritePerMtokMicrocents ?? 0) + // Media is a DISJOINT addend (1.AF/D17, ADR-0044 §3) — priced per image / audio-second / video-second, // never mixed into the token cost path, so the cumulative figure folds realized media spend. @@ -126,8 +196,8 @@ export class CostTracker { #cumulativeMicrocents = 0; readonly #overlay: PricingOverlay | undefined; - /** `overlay` (2.5.G S10) is the host-injected user-pricing tier — consulted after the static registry for an - * id it does not carry, so a user-priced model's realized spend is folded into the running total. */ + /** `overlay` (2.5.G S10) is the host-injected user-pricing tier — consulted FIRST (ADR-0071 §1), so a model the + * user has priced is billed at their number, and everything else at the catalog's. */ constructor(overlay?: PricingOverlay) { this.#overlay = overlay; } diff --git a/packages/llm/src/fallback-chain.test.ts b/packages/llm/src/fallback-chain.test.ts index 9b42d476..40078ed7 100644 --- a/packages/llm/src/fallback-chain.test.ts +++ b/packages/llm/src/fallback-chain.test.ts @@ -411,6 +411,30 @@ describe('FallbackChain.generate — per-attempt cost across a failover', () => expect(trace[1]?.cost?.cumulativeCostMicrocents).toBe(2_050_000); // 1_750_000 + 300_000 expect(tracker.cumulativeCostMicrocents).toBe(2_050_000); }); + + it('preAttempt carries THIS attempt’s routing provider — the primary, then the failover (review M2)', async () => { + // The pre-egress endpoint estimate keys on the routing provider, which MOVES on a failover. Each preAttempt + // must therefore report the provider of the entry actually being dialed, not a fixed primary. + const primary = makeProvider({ id: 'anthropic', generate: rejects('anthropic', 'overloaded') }); + const fallback = makeProvider({ id: 'openai', generate: resolves('recovered') }); + const seen: { model: string; provider: ProviderId }[] = []; + const { options } = makeOptions({ + preAttempt: (info) => { + seen.push({ model: info.model, provider: info.provider }); + }, + }); + const chain = new FallbackChain( + [entry(primary, 'claude-opus-4-8'), entry(fallback, 'gpt-5.4-mini')], + options, + ); + + await chain.generate(userReq); + + expect(seen).toEqual([ + { model: 'claude-opus-4-8', provider: 'anthropic' }, + { model: 'gpt-5.4-mini', provider: 'openai' }, + ]); + }); }); // --- ADR-0030 reasoning strip on cross-provider failover ------------------------------------- @@ -581,6 +605,77 @@ describe('withEntryModel (ADR-0066 §4 — per-fallback-entry reasoning gate)', expect(out.reasoningEffort).toBe('max'); }); + /** + * THE FAILOVER HOLE (ADR-0071 §6) — found by an adversarial review, and the sharpest finding in it. + * + * The re-gate used to ask `modelSupportsReasoning(model)`: a BOOLEAN. `gpt-5.5` accepts every tier; `gpt-5.4-pro` + * rejects `low` outright (it publishes ['medium','high','xhigh']). Both "support reasoning", so the boolean kept + * the tier — and the fallback, whose entire job is to RESCUE a failing turn, put a rejected value on the wire. + * + * A 400 on an unsupported parameter is fatal and non-retryable. So the rescue does not merely fail: it ABORTS + * the rest of the chain, killing the turn it exists to save. The gate now re-checks the tiers the entry model + * actually accepts. + */ + it('STRIPS a tier the fallback model REJECTS — even though that model reasons', () => { + const req: LlmRequest = { + model: 'gpt-5.5', // accepts low + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], + reasoningEffort: 'low', + }; + // gpt-5.4-pro's ladder starts at `medium`, and OpenAI has no budget field to express `low` some other way. + const out = withEntryModel(req, 'gpt-5.4-pro'); + expect(out.model).toBe('gpt-5.4-pro'); + expect('reasoningEffort' in out).toBe(false); + }); + + /** + * …but STRIPPING IS NOT FREE, so it must not happen when the model can serve the tier by another route. + * + * `claude-opus-4-5` publishes an effort ladder WITHOUT `max` — and, alongside it, a token budget. Its adapter + * already falls back to `thinking.budget_tokens` for exactly this case, so `max` is perfectly serviceable. An + * earlier version of `acceptedTiers` treated the two axes as an either/or and reported the ladder only: the + * rescue turn then ran with NO reasoning at all, discarding what the user asked for more aggressively than the + * model required. The axes are a union. + */ + it('KEEPS a tier the fallback model serves through its BUDGET axis, not its ladder', () => { + const req: LlmRequest = { + model: 'claude-opus-4-8', + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], + reasoningEffort: 'max', + }; + const out = withEntryModel(req, 'claude-opus-4-5'); + expect(out.model).toBe('claude-opus-4-5'); + expect(out.reasoningEffort).toBe('max'); + }); + + it("STRIPS `low` on a failover to gpt-5.4-pro — the maintainer's bug, reached through the rescue path", () => { + const req: LlmRequest = { + model: 'gpt-5.5', // accepts all five + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], + reasoningEffort: 'low', + }; + const out = withEntryModel(req, 'gpt-5.4-pro'); // publishes ['medium','high','xhigh'] — no `low` + expect('reasoningEffort' in out).toBe(false); + }); + + it('STRIPS the tier for a model the catalog does not know (a custom endpoint)', () => { + const req: LlmRequest = { + model: 'gpt-5.5', + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], + reasoningEffort: 'high', + }; + expect('reasoningEffort' in withEntryModel(req, 'some-custom-endpoint-model')).toBe(false); + }); + + it('KEEPS a tier the fallback model DOES accept', () => { + const req: LlmRequest = { + model: 'gpt-5.5', + messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }], + reasoningEffort: 'high', + }; + expect(withEntryModel(req, 'gpt-5.4-pro').reasoningEffort).toBe('high'); // `high` IS in its ladder + }); + it('is a plain model swap when no tier is set', () => { const req: LlmRequest = { model: 'gpt-4o', diff --git a/packages/llm/src/fallback-chain.ts b/packages/llm/src/fallback-chain.ts index 637a72b1..67db274a 100644 --- a/packages/llm/src/fallback-chain.ts +++ b/packages/llm/src/fallback-chain.ts @@ -13,7 +13,8 @@ import type { Usage, } from './types.js'; import { requestSupportReason } from './capabilities.js'; -import { modelSupportsReasoning } from './pricing.js'; +import { catalogModel } from './catalog/lookup.js'; +import { acceptedTiers } from './reasoning-wire.js'; export type { BackoffStrategy }; @@ -112,6 +113,11 @@ export interface AttemptRecord { export type PreAttemptHook = (info: { readonly model: string; readonly maxTokens?: number; + /** The provider THIS attempt targets — the routing provider, which on a failover differs from the primary. + * The pre-egress endpoint estimate must key on it (not the model's catalog provider): a custom gateway + * serving another provider's model id is `custom` at the wire yet `official` by catalog, and the mismatch + * under-authorizes real spend (review M2). */ + readonly provider: ProviderId; }) => void | Promise; /** Dependencies injected into a {@link FallbackChain} — all timing is injectable so tests are deterministic. */ @@ -212,16 +218,30 @@ export function stripReasoningParts(req: LlmRequest): LlmRequest { * ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md) §4). A failover to a * non-reasoning model must not carry the primary's tier: the provider would reject the unsupported parameter, and a * `400` on an unsupported param is fatal + non-retryable — so the whole remaining chain would abort rather than the - * failover rescuing the turn. The per-model capability is the SAME {@link modelSupportsReasoning} the host projects + * failover rescuing the turn. The per-model capability is the SAME {@link acceptedTiers} the host projects * to the engine gate, so the primary is gated at the engine and each fallback entry is re-gated here. Exported for * a focused unit test (like {@link stripReasoningParts}). */ export function withEntryModel(req: LlmRequest, model: string): LlmRequest { - const next = { ...req, model }; - if (next.reasoningEffort !== undefined && !modelSupportsReasoning(model)) { - delete next.reasoningEffort; - } - return next; + const next: LlmRequest = { ...req, model }; + if (next.reasoningEffort === undefined) return next; + + // ADR-0071 §6: re-gate on the tiers the ENTRY MODEL ACCEPTS — not on whether it reasons. + // + // This used to ask `modelSupportsReasoning(model)`, a boolean, and that is how the primary bug reached the wire + // through a failover: `claude-opus-4-8` accepts `max`, `claude-opus-4-5` does NOT (it publishes + // ['low','medium','high']). Both "support reasoning", so the boolean kept the tier — and the fallback, whose + // whole job is to RESCUE a failing turn, sent a value the rescue model rejects. A 400 on an unsupported param is + // fatal and non-retryable, so the chain aborts instead: the failover kills the turn it exists to save. + const entry = catalogModel(model); + const accepted = acceptedTiers(entry?.provider ?? 'openai', entry?.reasoning); + if (accepted.has(next.reasoningEffort)) return next; + + // The entry model rejects this tier — return it stripped. A fresh object (not a mutate-then-return of `next`) + // so the two exit paths are demonstrably distinct values. + const stripped: LlmRequest = { ...next }; + delete stripped.reasoningEffort; + return stripped; } /** The backoff delay before the `retryIndex`-th retry of an entry (0 = before the 2nd attempt). */ @@ -461,6 +481,7 @@ export class FallbackChain { const maxTokens = entryReq.maxTokens; await this.#options.preAttempt?.({ model: entry.model, + provider: entry.provider.id, ...(maxTokens === undefined ? {} : { maxTokens }), }); const key = await this.#resolveKey(entry.provider.id); @@ -495,6 +516,7 @@ export class FallbackChain { const maxTokens = entryReq.maxTokens; await this.#options.preAttempt?.({ model: entry.model, + provider: entry.provider.id, ...(maxTokens === undefined ? {} : { maxTokens }), }); const key = await this.#resolveKey(entry.provider.id); diff --git a/packages/llm/src/index.ts b/packages/llm/src/index.ts index 9d3944ab..eeb50b4a 100644 --- a/packages/llm/src/index.ts +++ b/packages/llm/src/index.ts @@ -129,16 +129,19 @@ export { } from './llm-error.js'; // CostTracker + the canonical model-pricing table (1.B). -export { - MODEL_PRICING, - KNOWN_MODEL_IDS, - isCanonicalModelId, - contextWindowForModel, - modelSupportsReasoning, -} from './pricing.js'; -export type { ModelPricing, CanonicalModelId } from './pricing.js'; +// The hand-typed price table is GONE from this surface (ADR-0071 §1), and so are the id guards it fed: +// `MODEL_PRICING`, `KNOWN_MODEL_IDS`, `isCanonicalModelId`, `CanonicalModelId`, and `modelSupportsReasoning`. +// +// `isCanonicalModelId` asked "is this one of OUR models". The honest question is "can we price this one", and a +// user-priced model is perfectly billable while never having been canonical — `priceModel`'s own throw answers it. +// `modelSupportsReasoning` asked "does this model reason", which is not the question the wire asks at all +// (`gpt-5.4-pro` reasons AND rejects `low`); `effortTiersFor` is the answer. +// +// `ModelPricing` survives as the CONTRACT — the CostTracker bills against it and a `models pricing` row IS one. +export { contextWindowForModel } from './pricing.js'; +export type { ModelPricing } from './pricing.js'; // The pure live/static/user merge helper (ADR-0064 §6) — reused by every surface's model catalog / picker. -export { mergeModelCatalog } from './model-catalog.js'; +export { collapseAliasDatedPinPairs, datedPinBase, mergeModelCatalog } from './model-catalog.js'; export type { ModelCatalogEntry, MergeModelCatalogInput, PricingSource } from './model-catalog.js'; export { priceModel, cost, mediaCost, CostTracker } from './cost-tracker.js'; export type { CostUpdate, PricingOverlay } from './cost-tracker.js'; @@ -175,3 +178,36 @@ export type { // `providerKind` derives the ADR-0064 protocol `kind` from a provider id (used by the later merge/refresh steps). // `createCustomOpenAiProvider` builds a per-provider OpenAI-compatible adapter for a custom base_url (ADR-0065 §3, S9). export { createCustomOpenAiProvider, defaultProviders, providerKind } from './providers.js'; + +// --- The generated model catalog (ADR-0071) ------------------------------------------------ +// The reasoning CONTROL is per-model data now, not a per-adapter assumption. The host projects a model's +// ACCEPTED TIERS from it (`resolveEffortTiers`), so a tier the model would reject never reaches the wire. +export { catalogModel, effortTiersFor, modelAccepts } from './catalog/lookup.js'; +// The generated snapshot itself + its pricing projection (ADR-0071 §1) — what `MODEL_PRICING` used to be. +export { CATALOG_SNAPSHOT } from './catalog/snapshot.js'; +export { catalogPricing, toPricing, pricedModelIds } from './catalog/pricing.js'; +// The refresh seam (ADR-0071 §4): the HOST fetches models.dev and installs the result; `@relavium/llm` does no I/O. +// Additive only — the shipped snapshot is the floor, so a bad payload degrades to it rather than to a blank catalog. +export { installCatalogRefresh, clearCatalogRefresh, catalogModelIds } from './catalog/lookup.js'; +export { CATALOG_PROVIDER_KEYS } from './catalog/catalog-providers.js'; +export { normalizeCatalog, ModelsDevPayloadSchema } from './catalog/models-dev-schema.js'; +export type { + CatalogModel, + CatalogPriceTier, + ReasoningControls, + RequestCapabilities, +} from './catalog/catalog-model.js'; +export { + acceptedTiers, + canDisableReasoning, + openAiWireValue, + reasoningControlShape, + reasoningWithheldByCap, + wireValueFor, + CANONICAL_ON_TIER, +} from './reasoning-wire.js'; +// The output cap (ADR-0071 §7) — an authored `max_tokens` held at or below the model's real ceiling. Exported +// because the PRE-EGRESS ESTIMATE must be computed from the same number the wire will carry: a governor that +// pre-authorizes spend on tokens the model is physically incapable of producing kills runs over phantom money. +export { cappedMaxTokens } from './output-cap.js'; +export type { EndpointKind } from './output-cap.js'; diff --git a/packages/llm/src/model-catalog.test.ts b/packages/llm/src/model-catalog.test.ts index 665c5284..67d1b8cf 100644 --- a/packages/llm/src/model-catalog.test.ts +++ b/packages/llm/src/model-catalog.test.ts @@ -1,9 +1,27 @@ -import { describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it } from 'vitest'; -import { mergeModelCatalog, type ModelCatalogEntry } from './model-catalog.js'; -import { MODEL_PRICING, type ModelPricing } from './pricing.js'; +import { + collapseAliasDatedPinPairs, + mergeModelCatalog, + type ModelCatalogEntry, +} from './model-catalog.js'; +import type { CatalogModel } from './catalog/catalog-model.js'; +import { clearCatalogRefresh, installCatalogRefresh } from './catalog/lookup.js'; +import { catalogModelFixture } from './conformance/fixtures/catalog.js'; +import { catalogPricing } from './catalog/pricing.js'; +import { CATALOG_SNAPSHOT } from './catalog/snapshot.js'; +import type { ModelPricing } from './pricing.js'; import type { ModelListing, ProviderId } from './types.js'; +// The alias↔pin negative tests below install SYNTHETIC catalog rows (additive overlay) — clear it after each so a +// leaked refresh never poisons a later test in the process. +afterEach(clearCatalogRefresh); + +/** A minimal well-formed refreshed catalog row for the synthetic alias/pin pairs the scoping tests need — the ONE + * shared fixture, keeping this suite's positional call shape. */ +const catModel = (modelId: string, provider: ProviderId): CatalogModel => + catalogModelFixture({ modelId, provider }); + // A fixed clock so the deprecation check is deterministic. deepseek-chat/-reasoner deprecate 2026-07-24 15:59Z. const BEFORE_DEEPSEEK_DEPRECATION = Date.parse('2026-07-05T00:00:00Z'); const AFTER_DEEPSEEK_DEPRECATION = Date.parse('2026-08-01T00:00:00Z'); @@ -15,7 +33,7 @@ const liveMap = ( rows: ReadonlyArray, ): ReadonlyMap => new Map(rows); -/** A minimal user-supplied ModelPricing for an id absent from MODEL_PRICING (ADR-0065 USER tier). */ +/** A minimal user-supplied ModelPricing (ADR-0065 USER tier — which now OUTRANKS the catalog, ADR-0071 §1). */ const userPricing = (provider: ProviderId): ModelPricing => ({ provider, nativeId: 'x', @@ -27,48 +45,65 @@ const userPricing = (provider: ProviderId): ModelPricing => ({ cachedInputPerMtokMicrocents: 10, }); +describe('collapseAliasDatedPinPairs — one row per model in the picker (ADR-0064 amendment)', () => { + const e = (modelId: string, provider: ProviderId = 'anthropic'): ModelCatalogEntry => ({ + modelId, + provider, + displayName: modelId, + pricingSource: 'catalog', + priceKnown: true, + available: true, + deprecated: false, + }); + + it('drops the dated pin when its rolling alias is ALSO present, keeping the alias', () => { + const out = collapseAliasDatedPinPairs([e('claude-opus-4-1'), e('claude-opus-4-1-20250805')]); + expect(out.map((x) => x.modelId)).toEqual(['claude-opus-4-1']); + }); + + it('KEEPS a lone dated pin with no alias sibling — hiding it would leave the model unpickable', () => { + const out = collapseAliasDatedPinPairs([e('claude-opus-4-1-20250805')]); + expect(out.map((x) => x.modelId)).toEqual(['claude-opus-4-1-20250805']); + }); + + it('KEEPS a non-anthropic dated variant — OpenAI YYYY-MM-DD is a different shape, out of scope', () => { + const out = collapseAliasDatedPinPairs([ + e('gpt-4o', 'openai'), + e('gpt-4o-2024-05-13', 'openai'), + ]); + expect(out.map((x) => x.modelId).sort()).toEqual(['gpt-4o', 'gpt-4o-2024-05-13']); + }); + + it('does NOT drop an anthropic dated pin whose base id belongs to a DIFFERENT provider (identity, not string match)', () => { + const out = collapseAliasDatedPinPairs([ + e('shared-base', 'openai'), + e('shared-base-20250101', 'anthropic'), + ]); + expect(out).toHaveLength(2); + }); +}); + describe('mergeModelCatalog (ADR-0064 §6)', () => { - it('with no live/user data, surfaces every static model as registry-priced and available (static presence)', () => { + it('with no live/user data, surfaces every CATALOG model as catalog-priced and available', () => { const entries = mergeModelCatalog({ now: BEFORE_DEEPSEEK_DEPRECATION }); - expect(entries).toHaveLength(Object.keys(MODEL_PRICING).length); + expect(entries).toHaveLength(Object.keys(CATALOG_SNAPSHOT).length); const opus = byId(entries, 'claude-opus-4-8'); expect(opus).toMatchObject({ provider: 'anthropic', displayName: 'Claude Opus 4.8', - pricingSource: 'registry', + pricingSource: 'catalog', priceKnown: true, - available: true, // no live data for anthropic -> static presence + available: true, // no live data for anthropic -> catalog presence deprecated: false, }); - expect(opus?.pricing).toBe(MODEL_PRICING['claude-opus-4-8']); + expect(opus?.pricing).toEqual(catalogPricing('claude-opus-4-8')); expect(opus?.contextWindowTokens).toBe(1_000_000); }); - it('surfaces supportsReasoning from the STATIC registry tier (ADR-0066) — true for a reasoning model, false otherwise', () => { - const entries = mergeModelCatalog({ now: BEFORE_DEEPSEEK_DEPRECATION }); - // A registry model tagged `reasoning: true` exposes the effort-controllable capability (incl. DeepSeek v4)… - expect(byId(entries, 'claude-opus-4-8')?.supportsReasoning).toBe(true); - expect(byId(entries, 'deepseek-v4-flash')?.supportsReasoning).toBe(true); - // …a registry model NOT so tagged (the legacy non-thinking `deepseek-chat`) is false — no effort sub-step. - expect(byId(entries, 'deepseek-chat')?.supportsReasoning).toBe(false); - }); - - it('a LIVE-only model gates via the §4 id heuristic — a known reasoning family ON, an ambiguous id OFF (ADR-0066)', () => { - const entries = mergeModelCatalog({ - live: liveMap([ - [ - 'openai', - [ - { id: 'o5-mini', displayName: 'o5 mini' }, // a future o-series id (whole family reasons) ⇒ ON - { id: 'gpt-4o-2026', displayName: 'GPT-4o' }, // not a reasoning family ⇒ OFF (over-match would 400) - ], - ], - ]), - now: BEFORE_DEEPSEEK_DEPRECATION, - }); - expect(byId(entries, 'o5-mini')?.supportsReasoning).toBe(true); // heuristic covers a new reasoning-family member - expect(byId(entries, 'gpt-4o-2026')?.supportsReasoning).toBe(false); // conservative — no false positive - }); + // The two tests that lived here asserted `ModelCatalogEntry.supportsReasoning`, a field that is GONE (ADR-0071 + // §6) along with the id heuristic behind it. They were asserting the wrong question — "does this model reason" — + // and the answer to the right one ("which tiers does it accept") is now catalog data, tested in + // `reasoning-wire.test.ts` and `catalog/lookup`'s `effortTiersFor` against every one of the 80 shipped models. it('availability: a static model NOT in a CONNECTED provider live list is dimmed, one present is available', () => { const entries = mergeModelCatalog({ @@ -82,6 +117,68 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(byId(entries, 'gpt-5.5')?.available).toBe(true); }); + it('ALIAS↔DATED-PIN equivalence: an alias is available when its dated pin is live, and vice versa (Anthropic, ADR-0064 §6 amendment)', () => { + // Anthropic's models.list() returns ONE of the pair; the catalog ships BOTH (`claude-haiku-4-5` + + // `claude-haiku-4-5-20251001`). The id the list omits is still callable on the same key, so it must not dim. + // Real Anthropic returns the DATED PIN, so the alias is the one that used to wrongly dim: + const pinLive = mergeModelCatalog({ + live: liveMap([['anthropic', [{ id: 'claude-haiku-4-5-20251001' }]]]), + now: BEFORE_DEEPSEEK_DEPRECATION, + }); + expect(byId(pinLive, 'claude-haiku-4-5-20251001')?.available).toBe(true); // in the live list + expect(byId(pinLive, 'claude-haiku-4-5')?.available).toBe(true); // rescued by its live dated-pin sibling + // A non-sibling model absent from the live list still dims — the rescue is scoped to the pair. + expect(byId(pinLive, 'claude-opus-4-8')?.available).toBe(false); + + // Symmetric — if the live list returns only the ALIAS, the dated pin is rescued instead. + const aliasLive = mergeModelCatalog({ + live: liveMap([['anthropic', [{ id: 'claude-haiku-4-5' }]]]), + now: BEFORE_DEEPSEEK_DEPRECATION, + }); + expect(byId(aliasLive, 'claude-haiku-4-5')?.available).toBe(true); + expect(byId(aliasLive, 'claude-haiku-4-5-20251001')?.available).toBe(true); + }); + + it('the alias rescue CANNOT fabricate availability for a non-catalog sibling id', () => { + // A live dated-pin id that is NOT a shipped catalog row cannot rescue the alias — the catalog-row gate is what + // stops an arbitrary id conjuring availability for a priced model the key cannot actually reach. + const entries = mergeModelCatalog({ + live: liveMap([['anthropic', [{ id: 'claude-haiku-4-5-99999999' }]]]), // a dated id NOT in the catalog + now: BEFORE_DEEPSEEK_DEPRECATION, + }); + expect(byId(entries, 'claude-haiku-4-5')?.available).toBe(false); // NOT rescued — the sibling isn't a catalog row + }); + + it('the alias rescue is ANTHROPIC-ONLY — a non-Anthropic dated pair is NOT rescued (the scoping decision)', () => { + // A synthetic OpenAI alias+dated-pin pair, both shipped catalog rows, with only the dated pin live. The rescue + // is scoped to Anthropic this round, so the openai alias must STILL dim — pinning the deliberate `provider !== + // 'anthropic'` gate so a future broadening (or a new non-Anthropic dated family) cannot sail through silently. + installCatalogRefresh({ + 'zzz-probe': catModel('zzz-probe', 'openai'), + 'zzz-probe-20250101': catModel('zzz-probe-20250101', 'openai'), + }); + const entries = mergeModelCatalog({ + live: liveMap([['openai', [{ id: 'zzz-probe-20250101' }]]]), + now: BEFORE_DEEPSEEK_DEPRECATION, + }); + expect(byId(entries, 'zzz-probe-20250101')?.available).toBe(true); // in the live list + expect(byId(entries, 'zzz-probe')?.available).toBe(false); // NOT rescued — equivalence is anthropic-only + }); + + it('the DATED-PIN rescue also requires the live base to be a CATALOG row (the IF-branch fabrication guard)', () => { + // A synthetic anthropic dated pin whose BASE alias is live but is NOT a shipped catalog row. The dated pin must + // still dim — exercising the line-146 `catalogModel(base)?.provider === provider` conjunct in isolation (the + // symmetric guard to the alias-branch `99999999` case above). + installCatalogRefresh({ + 'claude-probe-4-0-20250101': catModel('claude-probe-4-0-20250101', 'anthropic'), + }); + const entries = mergeModelCatalog({ + live: liveMap([['anthropic', [{ id: 'claude-probe-4-0' }]]]), // the base — but NOT a catalog row + now: BEFORE_DEEPSEEK_DEPRECATION, + }); + expect(byId(entries, 'claude-probe-4-0-20250101')?.available).toBe(false); // NOT rescued — base isn't a catalog row + }); + it('an EMPTY live list for a provider dims all that provider’s static models', () => { const entries = mergeModelCatalog({ live: liveMap([['anthropic', []]]), @@ -117,14 +214,17 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(byId(entries, 'gpt-6-preview')?.displayName).toBe('gpt-6-preview'); }); - it('price precedence: the static registry WINS for a known id even when user pricing is supplied', () => { + it('price precedence: the USER WINS over the catalog, even for a model the catalog knows (THE FLIP)', () => { + // Asserted the opposite until ADR-0071 §1. Registry-first protected the user from mispricing a model WE had + // verified; the catalog is a generated snapshot of a third party, and the user is holding the invoice. + const mine = userPricing('anthropic'); const entries = mergeModelCatalog({ - userPricing: new Map([['claude-opus-4-8', userPricing('anthropic')]]), + userPricing: new Map([['claude-opus-4-8', mine]]), now: BEFORE_DEEPSEEK_DEPRECATION, }); const opus = byId(entries, 'claude-opus-4-8'); - expect(opus?.pricingSource).toBe('registry'); - expect(opus?.pricing).toBe(MODEL_PRICING['claude-opus-4-8']); // NOT the user object + expect(opus?.pricingSource).toBe('user'); // …and the badge names the price we would actually bill at + expect(opus?.pricing).toBe(mine); }); it('price precedence: the USER tier fills an UNKNOWN id', () => { @@ -141,22 +241,25 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(entry?.maxOutputTokens).toBe(custom.maxOutputTokens); }); - it('three tiers on one known id: registry wins price, live wins context, available by live membership', () => { - // claude-opus-4-8 present in ALL THREE tiers at once — the full ADR-0064 §6 per-field split must hold. + it('three tiers on one id: USER wins price, LIVE wins context, availability by live membership', () => { + // claude-opus-4-8 present in ALL THREE tiers at once — the full per-field split (ADR-0064 §6, ADR-0071 §1). + // The two authorities are deliberately different: the PROVIDER is freshest about its own model's limits, and + // the USER is authoritative about what they are being charged. + const mine = userPricing('anthropic'); const entries = mergeModelCatalog({ live: liveMap([['anthropic', [{ id: 'claude-opus-4-8', contextWindowTokens: 500_000 }]]]), - userPricing: new Map([['claude-opus-4-8', userPricing('anthropic')]]), + userPricing: new Map([['claude-opus-4-8', mine]]), now: BEFORE_DEEPSEEK_DEPRECATION, }); const opus = byId(entries, 'claude-opus-4-8'); - expect(opus?.pricingSource).toBe('registry'); // registry wins even with a live AND user tier present - expect(opus?.pricing).toBe(MODEL_PRICING['claude-opus-4-8']); // NOT the user object + expect(opus?.pricingSource).toBe('user'); + expect(opus?.pricing).toBe(mine); expect(opus?.contextWindowTokens).toBe(500_000); // live wins for context expect(opus?.priceKnown).toBe(true); expect(opus?.available).toBe(true); // in the live list }); - it('context/output: the LIVE value wins over the static one for BOTH fields when present, else static', () => { + it('context/output: the LIVE value wins over the catalog for BOTH fields when present, else the catalog', () => { const entries = mergeModelCatalog({ live: liveMap([ [ @@ -172,25 +275,32 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { // a model with no live entry of its own falls back to the static values const sonnet = byId(entries, 'claude-sonnet-4-6'); expect(sonnet?.contextWindowTokens).toBe( - MODEL_PRICING['claude-sonnet-4-6'].contextWindowTokens, + CATALOG_SNAPSHOT['claude-sonnet-4-6']?.contextWindowTokens, ); - expect(sonnet?.maxOutputTokens).toBe(MODEL_PRICING['claude-sonnet-4-6'].maxOutputTokens); + expect(sonnet?.maxOutputTokens).toBe(CATALOG_SNAPSHOT['claude-sonnet-4-6']?.maxOutputTokens); }); - it('deprecation: a static deprecatedAt flags the model only once now >= the date', () => { + it("DEPRECATION survives the swap — it is Relavium's editorial call, not a price (ADR-0071 §10)", () => { + // The first cut of the big swap DELETED these two dates, on the argument that "the provider is the only one who + // knows when the provider is retiring something, so it should come from the live list". The argument is right and + // the conclusion was wrong: NO adapter populates `ModelListing.deprecatedAt` — the OpenAI-compatible list is + // id-only, and the Anthropic/Gemini mappers carry limits and names and nothing else. So `deprecated` went + // permanently `false` for every model in the product, and `deepseek-chat` was set to stop working on 2026-07-24 + // with nothing anywhere to say so. Information we already had, thrown away. + // + // models.dev publishes a `status` FLAG, and a flag cannot say "this stops working in eleven days". The date lives + // in a Relavium-owned overlay — one date per model, from a published announcement, and not a second price table. const before = mergeModelCatalog({ now: BEFORE_DEEPSEEK_DEPRECATION }); expect(byId(before, 'deepseek-chat')?.deprecated).toBe(false); - expect(byId(before, 'deepseek-chat')?.deprecatedAt).toBe('2026-07-24T15:59:00Z'); + expect(byId(before, 'deepseek-chat')?.deprecatedAt).toBe('2026-07-24T15:59:00Z'); // announced, not yet past const after = mergeModelCatalog({ now: AFTER_DEEPSEEK_DEPRECATION }); expect(byId(after, 'deepseek-chat')?.deprecated).toBe(true); - // a non-deprecated model stays clear - expect(byId(after, 'deepseek-v4-flash')?.deprecated).toBe(false); + expect(byId(after, 'deepseek-v4-flash')?.deprecated).toBe(false); // its replacement stays clear }); - it('deprecation is a UNION: the EARLIER of the static and live dates is effective', () => { + it('deprecation is still a UNION of live and user — the EARLIER date is effective', () => { const entries = mergeModelCatalog({ - // deepseek-v4-flash has no static deprecation; a live list marks it deprecated earlier live: liveMap([ ['deepseek', [{ id: 'deepseek-v4-flash', deprecatedAt: '2026-07-01T00:00:00Z' }]], ]), @@ -200,14 +310,17 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(flash?.deprecatedAt).toBe('2026-07-01T00:00:00Z'); expect(flash?.deprecated).toBe(true); // now (07-05) >= live date (07-01) - // when both are present, the earlier wins + // A user who knows a retirement date the provider's list has not published yet still gets the earlier one. const both = mergeModelCatalog({ live: liveMap([ ['deepseek', [{ id: 'deepseek-chat', deprecatedAt: '2027-01-01T00:00:00Z' }]], ]), + userPricing: new Map([ + ['deepseek-chat', { ...userPricing('deepseek'), deprecatedAt: '2026-07-24T15:59:00Z' }], + ]), now: BEFORE_DEEPSEEK_DEPRECATION, }); - expect(byId(both, 'deepseek-chat')?.deprecatedAt).toBe('2026-07-24T15:59:00Z'); // static earlier than live + expect(byId(both, 'deepseek-chat')?.deprecatedAt).toBe('2026-07-24T15:59:00Z'); }); it('an unparseable deprecatedAt is treated as not-deprecated (never throws)', () => { @@ -220,8 +333,8 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(weird?.deprecatedAt).toBeUndefined(); }); - it('ignores a live listing whose id collides with a DIFFERENT provider’s static model (no field corruption)', () => { - // a rogue / mis-keyed 'deepseek' live list claims 'gpt-5.5' (a real OpenAI static id) with junk fields + it('ignores a live listing whose id collides with a DIFFERENT provider’s catalog model (no field corruption)', () => { + // a rogue / mis-keyed 'deepseek' live list claims 'gpt-5.5' (a real OpenAI catalog id) with junk fields const entries = mergeModelCatalog({ live: liveMap([ [ @@ -241,9 +354,9 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { }); const gpt = byId(entries, 'gpt-5.5'); expect(gpt?.provider).toBe('openai'); // stays openai - expect(gpt?.displayName).toBe(MODEL_PRICING['gpt-5.5'].displayName); // registry, NOT 'HIJACKED' - expect(gpt?.contextWindowTokens).toBe(MODEL_PRICING['gpt-5.5'].contextWindowTokens); // registry, not 1 - expect(gpt?.maxOutputTokens).toBe(MODEL_PRICING['gpt-5.5'].maxOutputTokens); + expect(gpt?.displayName).toBe(CATALOG_SNAPSHOT['gpt-5.5']?.displayName); // the catalog, NOT 'HIJACKED' + expect(gpt?.contextWindowTokens).toBe(CATALOG_SNAPSHOT['gpt-5.5']?.contextWindowTokens); // the catalog, not 1 + expect(gpt?.maxOutputTokens).toBe(CATALOG_SNAPSHOT['gpt-5.5']?.maxOutputTokens); expect(gpt?.deprecated).toBe(false); // the rogue deprecatedAt is dropped expect(gpt?.deprecatedAt).toBeUndefined(); // openai has NO live list (only the mis-keyed deepseek one) -> gpt-5.5 falls back to static presence @@ -314,14 +427,14 @@ describe('mergeModelCatalog (ADR-0064 §6)', () => { expect(reversed.map((e) => e.modelId)).toEqual(forward.map((e) => e.modelId)); // insertion-order independent }); - it('does not mutate MODEL_PRICING', () => { - const snapshot = JSON.stringify(MODEL_PRICING); + it('does not mutate the CATALOG SNAPSHOT', () => { + const snapshot = JSON.stringify(CATALOG_SNAPSHOT); mergeModelCatalog({ live: liveMap([['anthropic', [{ id: 'claude-opus-4-8', contextWindowTokens: 1 }]]]), userPricing: new Map([['x', userPricing('openai')]]), now: AFTER_DEEPSEEK_DEPRECATION, }); - expect(JSON.stringify(MODEL_PRICING)).toBe(snapshot); + expect(JSON.stringify(CATALOG_SNAPSHOT)).toBe(snapshot); }); }); diff --git a/packages/llm/src/model-catalog.ts b/packages/llm/src/model-catalog.ts index 9baa47de..abbe5da1 100644 --- a/packages/llm/src/model-catalog.ts +++ b/packages/llm/src/model-catalog.ts @@ -1,31 +1,37 @@ -import { MODEL_PRICING, modelSupportsReasoning, type ModelPricing } from './pricing.js'; +import { deprecationFor } from './catalog/deprecations.js'; +import { toPricing } from './catalog/pricing.js'; +import { catalogModel, catalogModelIds } from './catalog/lookup.js'; +import type { ModelPricing } from './pricing.js'; import type { ModelListing, ProviderId } from './types.js'; /** - * The merged model catalog (ADR-0064 §6) — the pure reconciliation of LIVE discovery (which model ids a key - * can reach, from `LlmProvider.listModels`), the STATIC registry ({@link MODEL_PRICING}), and the optional - * USER-pricing tier (ADR-0065, filled additively from the `model_catalog` `source='user'` rows). It lives in - * `@relavium/llm` and is **pure / I/O-free** (the host does the keychain/db/network work and passes plain data - * in), so every surface — the CLI `/models` picker, the desktop, the VS Code extension — reuses the same - * precedence. Selection (availability) and pricing are kept cleanly separate: the live list decides - * **availability**, the static registry stays the **pricing authority**. + * The merged model catalog (ADR-0064 §6, ADR-0071 §1) — the pure reconciliation of LIVE discovery (which model + * ids a key can actually reach, from `LlmProvider.listModels`), the generated CATALOG (`catalog/snapshot.ts`, + * synced from models.dev — what the hand-typed registry used to be), and the USER-pricing tier (ADR-0065, from + * the `model_catalog` `source='user'` rows). It lives in `@relavium/llm` and is **pure / I/O-free** (the host does + * the keychain/db/network work and passes plain data in), so every surface — the CLI `/models` picker, the + * desktop, the VS Code extension — reuses the same precedence. + * + * Selection and pricing stay cleanly separate: the **live list decides availability**, and pricing resolves + * **user → catalog**, the same order `priceModel` bills at. The live tier is never a pricing authority (providers + * rarely return a price, and a refresh must never overwrite a known one). */ /** Where an entry's effective pricing came from. `none` ⇒ the cost cap will not apply (ADR-0064 §6 / ADR-0065). */ -export type PricingSource = 'registry' | 'user' | 'none'; +export type PricingSource = 'catalog' | 'user' | 'none'; /** One reconciled model in the merged catalog (ADR-0064 §6). */ export interface ModelCatalogEntry { readonly modelId: string; readonly provider: ProviderId; readonly displayName: string; - /** From live ?? static ?? user (live is fresher when present, e.g. Anthropic's `max_input_tokens`). */ + /** From live ?? user ?? catalog (live is fresher when present, e.g. Anthropic's `max_input_tokens`). */ readonly contextWindowTokens?: number; readonly maxOutputTokens?: number; /** - * The **effective** pricing: static ({@link MODEL_PRICING}) for a known id, else the USER tier for an - * unknown id, else undefined. The live tier is **never** a pricing authority (ADR-0064 §6) — providers - * rarely return a price and a refresh must never overwrite a known one. + * The **effective** pricing: the USER's row if they set one, else the generated catalog's, else undefined. + * The live tier is **never** a pricing authority (ADR-0064 §6) — providers rarely return a price, and a + * refresh must never overwrite a known one. */ readonly pricing?: ModelPricing; readonly pricingSource: PricingSource; @@ -52,14 +58,6 @@ export interface ModelCatalogEntry { readonly deprecated: boolean; /** The effective ISO deprecation date — the earlier of the static and live dates (their union). */ readonly deprecatedAt?: string; - /** - * Whether the model exposes a controllable reasoning-effort tier ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md)). - * Sourced from the STATIC registry only (`MODEL_PRICING[id].reasoning === true`) — the same authority as - * `modelSupportsReasoning`, so the picker's effort sub-step and the engine's `resolveReasoning` gate agree. A - * live-only / user-only id (no registry tier) is `false` (reasoning capability is a shipped-registry fact, never - * inferred from a discovery listing or a user price row). - */ - readonly supportsReasoning: boolean; } /** Input to {@link mergeModelCatalog} — all plain data the host resolves and passes in (keeps the merge pure). */ @@ -67,12 +65,14 @@ export interface MergeModelCatalogInput { /** * Per-provider LIVE listings from `listModels`. A provider **present** in the map (even with `[]`) has live * data, so availability is decided by list membership (an empty `[]` dims all that provider's static models). - * A provider **absent** from the map has **no** live data — its registry models fall back to static presence. + * A provider **absent** from the map has **no** live data — its catalog models fall back to catalog presence. */ readonly live?: ReadonlyMap; /** - * ADR-0065 USER tier: user-supplied pricing by model id. Fills an **unknown** id only — the static registry - * always wins for a known id (ADR-0064 §6 / ADR-0065 §2), so a user cannot silently misprice a shipped model. + * ADR-0065 USER tier: user-supplied pricing by model id — and it **OUTRANKS the catalog** (ADR-0071 §1). It used + * to fill an unknown id only, on the reasoning that a user should not be able to misprice a shipped model. That + * reasoning belonged to a table WE verified. The catalog is a snapshot of a third-party aggregator, and the user + * is the one holding the invoice: their negotiated rate is not a hint for a generated file to overrule. */ readonly userPricing?: ReadonlyMap; /** @@ -92,7 +92,8 @@ export interface MergeModelCatalogInput { interface Tiers { provider: ProviderId; live?: ModelListing; - registry?: ModelPricing; + /** The generated catalog's row (ADR-0071) — what the hand-typed `MODEL_PRICING` registry used to be. */ + catalog?: ModelPricing; user?: ModelPricing; } @@ -105,13 +106,79 @@ function earlierIsoDate(a: string | undefined, b: string | undefined): string | return pa <= pb ? a : b; } -/** The pricing provenance for a merged entry: the registry wins, then the user tier, else none (ADR-0064 §6). */ +/** The pricing provenance for a merged entry: the USER wins, then the catalog, else none (ADR-0064 §6). */ function pricingSourceOf(t: Tiers): PricingSource { - if (t.registry) return 'registry'; + // USER first — the same precedence `priceModel` applies (ADR-0071 §1). The badge must name the price we would + // actually BILL at, or it is a lie the user reads while being charged something else. if (t.user) return 'user'; + if (t.catalog) return 'catalog'; return 'none'; } +/** A model id shaped `base-YYYYMMDD` → its rolling-alias base; else `undefined`. Anthropic pins a snapshot with an + * 8-digit date suffix (`claude-haiku-4-5-20251001`), and the base (`claude-haiku-4-5`) is the rolling alias. */ +const DATED_PIN = /^(.+)-\d{8}$/; +export function datedPinBase(id: string): string | undefined { + return DATED_PIN.exec(id)?.[1]; +} + +/** + * Collapse an alias↔dated-pin PAIR to the rolling alias for the display catalog the picker renders (ADR-0064 + * amendment). Anthropic ships BOTH `claude-opus-4-1` (the rolling alias) and `claude-opus-4-1-20250805` (its dated + * pin) as byte-identical priced rows, so the picker offers two selectable rows for one model. Drop the dated pin — + * but ONLY when its alias base is ALSO present as an entry (both are real rows): a lone dated pin with no alias + * sibling stays visible, or it would disappear from the picker with no substitute, which is worse than a duplicate + * (the same "both must be present" gate {@link hasLiveSibling} applies). + * + * This is IDENTITY dedup, NOT an availability judgement — deliberately distinct from the picker's "never HIDE a + * dimmed/deprecated model" rule (model-picker.ts): that rule is about a model you cannot currently USE; this removes + * a second copy of one model you can. ANTHROPIC-only by construction — `datedPinBase` matches the `-YYYYMMDD` shape, + * and OpenAI's dated variants (`gpt-4o-2024-05-13`, a `YYYY-MM-DD` shape) do not match, so they are naturally out of + * scope (the maintainer's existing scoping decision, mirroring `hasLiveSibling`). + */ +export function collapseAliasDatedPinPairs( + entries: readonly ModelCatalogEntry[], +): ModelCatalogEntry[] { + const byId = new Map(entries.map((e) => [e.modelId, e])); + return entries.filter((e) => { + if (e.provider !== 'anthropic') return true; + const base = datedPinBase(e.modelId); + if (base === undefined) return true; // not a dated pin — keep + // Drop the dated pin ONLY when its rolling-alias base is itself a present anthropic row (the pair is real). + // An absent alias short-circuits to `undefined !== 'anthropic'` ⇒ keep, exactly as the explicit guard did. + return byId.get(base)?.provider !== 'anthropic'; + }); +} + +/** + * Does a keyed provider's live list carry this model's alias↔dated-pin SIBLING? (ADR-0064 §6 amendment, ADR-0071.) + * + * Anthropic's `models.list()` returns only ONE of a rolling alias (`claude-haiku-4-5`) and its dated pin + * (`claude-haiku-4-5-20251001`), yet the catalog SHIPS BOTH as priced rows — so the id the list omits dimmed as + * `not-on-key` even though the SAME key calls it (server-side both resolve to the same model). The current id is + * available when its sibling IS in the live list AND is itself a shipped catalog row of the same provider. The + * catalog-row gate is what stops this fabricating availability for an arbitrary unpriced id. ANTHROPIC-ONLY, by the + * maintainer's scoping decision — the OpenAI `gpt-4o` dated family is deliberately out of scope for this round. + */ +function hasLiveSibling( + modelId: string, + provider: ProviderId, + live: ReadonlyMap, +): boolean { + if (provider !== 'anthropic') return false; + const listings = live.get(provider); + if (listings === undefined) return false; + const base = datedPinBase(modelId); + if (base !== undefined) { + // `modelId` is the dated PIN → the live rolling alias `base` is its sibling. + return listings.some((l) => l.id === base) && catalogModel(base)?.provider === provider; + } + // `modelId` is the rolling ALIAS → a live dated pin whose base equals it is its sibling. + return listings.some( + (l) => datedPinBase(l.id) === modelId && catalogModel(l.id)?.provider === provider, + ); +} + /** * Availability + its reason (2.5.G key-awareness). Key gate FIRST: a provider absent from `keyedProviders` has no * resolvable key, so its model is genuinely uncallable → unavailable with an actionable `'no-key'` reason, @@ -120,8 +187,12 @@ function pricingSourceOf(t: Tiers): PricingSource { * §6 "never everything unavailable" safe default — PRESERVED, but now only for a KEYED provider). `keyedProviders` * ABSENT ⇒ not key-gated (every provider treated as keyed): the `available` BOOLEAN is unchanged from pre-change; * the only new output is the additive `'not-on-key'` reason on a live-omitted static model — informational. + * + * ADR-0064 §6 amendment (ADR-0071): before dimming a live-omitted model, {@link hasLiveSibling} rescues an + * alias↔dated-pin pair — the id the provider's list left out is still callable on the same key. */ function resolveAvailability( + modelId: string, t: Tiers, live: ReadonlyMap, keyedProviders: ReadonlySet | undefined, @@ -129,9 +200,9 @@ function resolveAvailability( const providerKeyed = keyedProviders === undefined || keyedProviders.has(t.provider); if (!providerKeyed) return { available: false, unavailableReason: 'no-key' }; if (live.has(t.provider)) { - return t.live !== undefined - ? { available: true } - : { available: false, unavailableReason: 'not-on-key' }; + if (t.live !== undefined) return { available: true }; + if (hasLiveSibling(modelId, t.provider, live)) return { available: true }; + return { available: false, unavailableReason: 'not-on-key' }; } return { available: true }; } @@ -144,8 +215,12 @@ function buildTiers( userPricing: ReadonlyMap, ): Map { const tiers = new Map(); - for (const [id, registry] of Object.entries(MODEL_PRICING) as [string, ModelPricing][]) { - tiers.set(id, { provider: registry.provider, registry }); + // The refreshed catalog when the host installed one, else the shipped snapshot — `catalogModel` decides, so the + // picker shows a model a `models refresh --catalog` just added without a second merge path. + for (const id of catalogModelIds()) { + const entry = catalogModel(id); + if (entry === undefined) continue; + tiers.set(entry.modelId, { provider: entry.provider, catalog: toPricing(entry) }); } for (const [provider, listings] of live) { for (const listing of listings) { @@ -169,15 +244,34 @@ function buildEntry( input: MergeModelCatalogInput, live: ReadonlyMap, ): ModelCatalogEntry { - const pricing = t.registry ?? t.user; // registry wins for a known id; user fills an unknown one. + // USER OUTRANKS THE CATALOG (ADR-0071 §1) — the flip. The old rule was registry-first, and it made sense while + // the registry was our own verified table: a user could not misprice a shipped model. The catalog is a snapshot + // of a third-party aggregator, and the user is the one holding the invoice. Their negotiated rate, their + // enterprise discount, or simply a price we have not re-synced is not a hint to be overruled by a generated file. + const pricing = t.user ?? t.catalog; const pricingSource = pricingSourceOf(t); + // The LIMITS stay live-first: a provider's own list is the freshest word on its own model's window, and a user + // pricing row rarely carries one. Below it, the same user > catalog order as the price. const contextWindowTokens = - t.live?.contextWindowTokens ?? t.registry?.contextWindowTokens ?? t.user?.contextWindowTokens; + t.live?.contextWindowTokens ?? t.user?.contextWindowTokens ?? t.catalog?.contextWindowTokens; const maxOutputTokens = - t.live?.maxOutputTokens ?? t.registry?.maxOutputTokens ?? t.user?.maxOutputTokens; - const { available, unavailableReason } = resolveAvailability(t, live, input.keyedProviders); + t.live?.maxOutputTokens ?? t.user?.maxOutputTokens ?? t.catalog?.maxOutputTokens; + const { available, unavailableReason } = resolveAvailability( + modelId, + t, + live, + input.keyedProviders, + ); + // Deprecation is a UNION of three sources, and the EARLIEST wins — a warning is only useful before the date. + // + // models.dev publishes a `status` flag, not a date, so the retirement date lives in Relavium's own small overlay + // (ADR-0071 §10 — an editorial call about our users, not a data fact). It is NOT a second pricing home: one date + // per model, from a published announcement. The first version of the swap dropped it on the theory that the live + // list would carry it — but no adapter populates `ModelListing.deprecatedAt` (the OpenAI list is id-only), so + // `deprecated` was permanently `false` for every model in the product and `deepseek-chat` was set to stop working + // in eleven days with nothing to say so. const deprecatedAt = earlierIsoDate( - earlierIsoDate(t.registry?.deprecatedAt, t.live?.deprecatedAt), + earlierIsoDate(deprecationFor(modelId), t.live?.deprecatedAt), t.user?.deprecatedAt, ); const parsedDeprecation = deprecatedAt === undefined ? Number.NaN : Date.parse(deprecatedAt); @@ -185,7 +279,9 @@ function buildEntry( return { modelId, provider: t.provider, - displayName: t.registry?.displayName ?? t.live?.displayName ?? t.user?.displayName ?? modelId, + // The catalog's name first: models.dev carries a curated one ("GPT-5.4 Pro"), while a provider's live list + // often echoes the raw id back. A user row's name is the last resort — they set a PRICE, not a label. + displayName: t.catalog?.displayName ?? t.live?.displayName ?? t.user?.displayName ?? modelId, ...(contextWindowTokens !== undefined ? { contextWindowTokens } : {}), ...(maxOutputTokens !== undefined ? { maxOutputTokens } : {}), ...(pricing !== undefined ? { pricing } : {}), @@ -195,20 +291,15 @@ function buildEntry( ...(unavailableReason !== undefined ? { unavailableReason } : {}), deprecated, ...(deprecatedAt !== undefined ? { deprecatedAt } : {}), - // Reasoning capability via the SAME authority as the engine gate (ADR-0066 §4): the registry flag for a known - // id (authoritative — true or false), else the conservative id heuristic for a live-discovered id. So the - // picker's effort sub-step lights up exactly for the models the engine will actually honor — including a newly - // released reasoning family member absent from the registry. - supportsReasoning: modelSupportsReasoning(modelId), }; } /** - * Reconcile live discovery ⋈ the static registry ⋈ the user tier into one deterministically-ordered catalog + * Reconcile live discovery ⋈ the generated catalog ⋈ the user tier into one deterministically-ordered catalog * (ADR-0064 §6). Pure: no I/O, no `Date.now()` (the caller passes `now`). Per-field precedence — - * availability ← live (else static presence); price ← registry ?? user (never live); context/output ← live ?? - * static ?? user; deprecation ← the earliest of the static, live, and user dates; priceKnown ← a static or - * user price exists. + * availability ← live (else static presence); price ← **user ?? catalog** (the USER outranks the catalog, + * ADR-0071 §1; never live); context/output ← live ?? catalog ?? user; deprecation ← the earliest of the catalog, + * live, and user dates; priceKnown ← a user or catalog price exists. */ export function mergeModelCatalog(input: MergeModelCatalogInput): ModelCatalogEntry[] { const live = input.live ?? new Map(); diff --git a/packages/llm/src/model-kind.ts b/packages/llm/src/model-kind.ts new file mode 100644 index 00000000..d0a86121 --- /dev/null +++ b/packages/llm/src/model-kind.ts @@ -0,0 +1,65 @@ +/** + * Is this model id a CHAT model? — the one canonical home of that question (CLAUDE.md rule 8). + * + * It began life inside the OpenAI adapter as the live-list filter ([ADR-0064](../../../docs/decisions/0064-live-model-catalog.md) §3): + * a provider's `/v1/models` is id-only, with no capability metadata, so the filter is an id-family heuristic. + * It moved here when a **second** consumer appeared — the generated catalog + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md)) — and the two must not + * drift, for a reason that is not cosmetic: + * + * `keepOpenAiModelId` short-circuits on `pricedIds.has(id)` — **a priced id bypasses this deny-list entirely**, + * so that a cost-eligible model can never be filtered out of the live list. Once the catalog becomes the priced + * set, any non-chat model the catalog imports would be *rescued* by that short-circuit and appear in the user's + * model picker as something they can chat with. `text-embedding-3-large` is priced upstream and would have + * arrived exactly that way. Two filters that disagree is a cascade; one filter cannot. + */ + +/** + * Id SEGMENTS that are NOT chat text models. **Deny wins over allow**, so `gpt-image-1` / + * `gpt-4o-audio-preview` / `omni-moderation` are dropped even though they match a `gpt`/`o` allow-family. + */ +const NON_CHAT_SEGMENTS = [ + 'embedding', + 'tts', + 'whisper', + 'image', + 'moderation', + 'realtime', + 'audio', + 'dall-e', + 'transcribe', + 'search', + 'instruct', + 'ocr', + 'davinci', + 'babbage', +] as const; + +/** Escape a literal string for embedding inside a `RegExp`. The tokens carry no metacharacters today + * (`dall-e`'s `-` is literal outside a character class), but this keeps the boundary match safe if one is added. */ +function escapeRegExp(text: string): string { + // `String.raw` avoids the doubled backslash of `'\\$&'` — the replacement is a literal `\` + the `$&` match ref. + return text.replace(/[.*+?^${}()|[\]\\]/g, String.raw`\$&`); +} + +/** + * ONE combined matcher, compiled ONCE at module load — every segment as an alternation, still anchored on a + * `-`/`_` boundary. The previous form built a fresh `RegExp` per segment on EVERY call: fourteen compiles per + * model id, and a bulk `sync:models` runs this over thousands of ids. No `g` flag — `.test` on a stateful regex + * would carry `lastIndex` between calls. + */ +const NON_CHAT_SEGMENT_RE = new RegExp( + `(^|[-_])(?:${NON_CHAT_SEGMENTS.map(escapeRegExp).join('|')})([-_]|$)`, +); + +/** + * True when `id` names something other than a chat text model. + * + * Every token is matched on a `-`/`_` **SEGMENT boundary**, never as a bare substring — which is load-bearing: + * `search` must fire on `gpt-4o-search-preview` (`-search-`) and must NOT fire on `o3-deep-research` + * (re**search**), a real reasoning model that a substring match would silently delete from the catalog and the + * picker alike. + */ +export function isNonChatModelId(id: string): boolean { + return NON_CHAT_SEGMENT_RE.test(id.toLowerCase()); +} diff --git a/packages/llm/src/output-cap.test.ts b/packages/llm/src/output-cap.test.ts new file mode 100644 index 00000000..888a5498 --- /dev/null +++ b/packages/llm/src/output-cap.test.ts @@ -0,0 +1,55 @@ +import { describe, expect, it } from 'vitest'; + +import { CATALOG_SNAPSHOT } from './catalog/snapshot.js'; +import { cappedMaxTokens } from './output-cap.js'; + +/** + * The output cap (ADR-0071 §7) — the other half of the maintainer's "max tokens errors". + * + * Nothing in the shipped code compared an authored `max_tokens` against the model's real output limit, because + * nothing KNEW the limit: `MODEL_PRICING` carried a context window and no output ceiling at all. So an agent + * authored with `max_tokens: 200000` on a 64 000-token model 400'd on every single turn, and the workflow it sat + * in never ran. + */ +describe('cappedMaxTokens — down to the model ceiling, never up', () => { + it('CLAMPS a cap above the model ceiling — the 400 that had no fix', () => { + // gpt-5.4-pro publishes maxOutputTokens: 128_000. + expect(CATALOG_SNAPSHOT['gpt-5.4-pro']?.maxOutputTokens).toBe(128_000); // the premise + expect(cappedMaxTokens(200_000, 'gpt-5.4-pro')).toBe(128_000); + }); + + it("LEAVES a cap below the ceiling ALONE — it is the author's budget, not a mistake to correct", () => { + // The tempting "helpful" move is to raise a small cap to the model's maximum. That spends the user's money on + // their behalf: a low cap is a cost control, a latency budget, a hard bound on a summary's length. + expect(cappedMaxTokens(500, 'gpt-5.4-pro')).toBe(500); + }); + + it('passes an ABSENT cap through — the provider default stands, we do not invent one', () => { + expect(cappedMaxTokens(undefined, 'gpt-5.4-pro')).toBeUndefined(); + }); + + it('does NOT clamp a model the catalog cannot describe — there is no ceiling to clamp against', () => { + expect(cappedMaxTokens(999_999, 'some-model-we-have-never-heard-of')).toBe(999_999); + }); + + it('does NOT clamp a CUSTOM endpoint, even for an id the catalog knows', () => { + // A `base_url` pointing at LM Studio / vLLM / a gateway may serve something entirely different under a familiar + // id, with its own limits. Silently LOWERING a number the user typed, on a model we are only guessing at, is a + // behaviour change we have no right to make — the asymmetry with WITHHOLDING the reasoning field there (which + // is safe, and which we do) is deliberate. + expect(cappedMaxTokens(200_000, 'gpt-5.4-pro', 'custom')).toBe(200_000); + }); + + it('a cap EXACTLY at the ceiling is untouched — the boundary is inclusive', () => { + expect(cappedMaxTokens(128_000, 'gpt-5.4-pro')).toBe(128_000); + }); + + it('every shipped model has a ceiling to clamp against — the invariant the clamp rests on', () => { + // The clamp is only as good as the data behind it: a model whose row carried no output ceiling would pass an + // unbounded cap straight through and 400 exactly as before. This is the one assertion here that is NOT a + // restatement of `Math.min` — it checks the CATALOG, which is generated and can regress upstream. + for (const [id, model] of Object.entries(CATALOG_SNAPSHOT)) { + expect(model.maxOutputTokens, `${id} has no output ceiling`).toBeGreaterThan(0); + } + }); +}); diff --git a/packages/llm/src/output-cap.ts b/packages/llm/src/output-cap.ts new file mode 100644 index 00000000..28d1321f --- /dev/null +++ b/packages/llm/src/output-cap.ts @@ -0,0 +1,46 @@ +import { catalogModel } from './catalog/lookup.js'; + +/** + * The request's output cap, held **at or below the model's own ceiling** + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §7). + * + * The other half of the maintainer's "max tokens errors". An authored `max_tokens: 200000` on a model whose + * `limit.output` is 64 000 is not an ambitious request — it is a 400, every turn, and the workflow it sits in + * never runs. Nothing in the shipped code compared the two: `MODEL_PRICING` carried a context window and no + * output limit at all, so there was nothing to compare against. The catalog carries both. + * + * **Down, never up.** A cap BELOW the model's ceiling is the author's deliberate choice — a cost control, a + * latency budget, a hard bound on a summary's length — and raising it to the ceiling would spend the user's money + * on their behalf. Only the impossible half is corrected. + */ + +/** + * Can we describe the endpoint this request is going to? + * + * The catalog describes MODELS as their providers serve them. A custom `base_url` ([ADR-0065](../../../docs/decisions/0065-provider-economics-and-extensibility.md)) + * — LM Studio, Ollama, vLLM, an enterprise gateway — may serve something entirely different under a familiar id, + * with its own limits. Clamping there would silently lower a cap the user set on a model we are only guessing at, + * and a silent lowering is a behaviour change we have no right to make. + * + * This is the same reasoning that WITHHOLDS the reasoning field on an unknown model, pointed at a different + * decision — and it lands the other way, because the two failures are not symmetric. Withholding a field we + * cannot justify is safe; lowering a number the user typed is not. + */ +export type EndpointKind = 'official' | 'custom'; + +/** + * The `max_tokens` to send: the caller's, capped at the model's published output ceiling. + * + * `undefined` in ⇒ `undefined` out — no cap authored, so the provider's own default stands. An id the catalog + * does not carry, or a custom endpoint, passes through untouched: we clamp only against a limit we actually know. + */ +export function cappedMaxTokens( + requested: number | undefined, + model: string, + endpoint: EndpointKind = 'official', +): number | undefined { + if (requested === undefined || endpoint === 'custom') return requested; + const ceiling = catalogModel(model)?.maxOutputTokens; + if (ceiling === undefined) return requested; // not in the catalog — nothing to clamp against + return Math.min(requested, ceiling); +} diff --git a/packages/llm/src/pricing.test.ts b/packages/llm/src/pricing.test.ts index c56b43ac..25f4d756 100644 --- a/packages/llm/src/pricing.test.ts +++ b/packages/llm/src/pricing.test.ts @@ -1,78 +1,137 @@ import { describe, expect, it } from 'vitest'; -import { - contextWindowForModel, - KNOWN_MODEL_IDS, - MODEL_PRICING, - modelSupportsReasoning, -} from './pricing.js'; +import { catalogPricing, pricedModelIds, toPricing } from './catalog/pricing.js'; +import { CATALOG_SNAPSHOT } from './catalog/snapshot.js'; +import { cost } from './cost-tracker.js'; +import { estimateMaxNextCost } from './budget-estimator.js'; +import { contextWindowForModel } from './pricing.js'; /** - * `contextWindowForModel` (ADR-0062 §7) — the pure catalog lookup the CLI footer context-fullness indicator uses - * without going through the provider seam. It must return the SAME `contextWindowTokens` the adapters' `contextLimit` - * returns for a known model, and `undefined` for a custom base-URL model absent from the catalog (which degrades the - * indicator + auto-compaction to "not applicable", never a crash). + * `contextWindowForModel` (ADR-0062 §7) — the pure lookup the CLI's context-fullness footer uses without going + * through the provider seam. It must return the SAME window the adapters' `contextLimit` returns, and `undefined` + * for a model the catalog does not carry (which degrades the indicator + auto-compaction to "not applicable", + * never a crash). + * + * It read the hand-typed table until ADR-0071 §1, and the table said `gpt-5.5`'s window was 1 000 000 while the + * generated catalog said 1 050 000 — so the percentage every user saw was computed against a window the model does + * not have. Nothing compared the two, which is the whole reason there is now only one. */ -describe('modelSupportsReasoning (ADR-0066)', () => { - it('is true for a tagged reasoning model (incl. DeepSeek v4), false for a non-reasoning + unknown/custom', () => { - expect(modelSupportsReasoning('claude-opus-4-8')).toBe(true); - expect(modelSupportsReasoning('gpt-5.5')).toBe(true); - expect(modelSupportsReasoning('gemini-2.5-pro')).toBe(true); - // DeepSeek v4 exposes the controllable `thinking` param (ADR-0066) — the effort IS controllable, so ON. - expect(modelSupportsReasoning('deepseek-v4-flash')).toBe(true); - expect(modelSupportsReasoning('deepseek-v4-pro')).toBe(true); - // The legacy `deepseek-chat` (non-thinking alias) is NOT reasoning-controllable → OFF. - expect(modelSupportsReasoning('deepseek-chat')).toBe(false); - // `deepseek-reasoner` IS a thinking model, but it's the fixed always-on alias — effort is not adjustable on it - // (sending `thinking:{type:'disabled'}` for `off` would 400), so it stays OFF (no effort picker). ADR-0066 §2. - expect(modelSupportsReasoning('deepseek-reasoner')).toBe(false); - // Unknown / custom base-URL model NOT matching a reasoning family ⇒ the SAFE default. - expect(modelSupportsReasoning('some-custom-base-url-model-xyz')).toBe(false); - expect(modelSupportsReasoning('')).toBe(false); +describe('contextWindowForModel (ADR-0062 §7)', () => { + it('returns the CATALOG window — the one source, never a second opinion about it', () => { + for (const [id, entry] of Object.entries(CATALOG_SNAPSHOT)) { + expect(contextWindowForModel(id), id).toBe(entry.contextWindowTokens); + } }); - it('the §4 id heuristic gates a NON-registry model in a known reasoning family (conservative)', () => { - // A live-discovered id absent from MODEL_PRICING but in a family whose whole set reasons is gated ON… - expect(modelSupportsReasoning('o5-mini')).toBe(true); // a future o-series id - expect(modelSupportsReasoning('gpt-5.9-turbo')).toBe(true); // a future reasoning gpt-5 id - expect(modelSupportsReasoning('claude-opus-5')).toBe(true); // a future Opus - expect(modelSupportsReasoning('deepseek-v4-turbo')).toBe(true); // a future non-registry DeepSeek v4 id - expect(modelSupportsReasoning('deepseek-v5-pro')).toBe(true); // a future DeepSeek v5 - expect(modelSupportsReasoning('gemini-3.0-flash-thinking')).toBe(true); // an explicit "thinking" id - // …while an AMBIGUOUS / non-reasoning id stays OFF (over-matching would earn a provider 400). - expect(modelSupportsReasoning('gpt-4o')).toBe(false); // not a reasoning family - expect(modelSupportsReasoning('gpt-5-chat-latest')).toBe(false); // the non-reasoning gpt-5 conversational variant - expect(modelSupportsReasoning('deepseek-v5-chat')).toBe(false); // a hypothetical future non-thinking DeepSeek `-chat` variant (mirrors the gpt-5-chat guard) - expect(modelSupportsReasoning('claude-sonnet-9')).toBe(false); // base Sonnet is version-dependent — registry only - expect(modelSupportsReasoning('gemini-2.0-flash')).toBe(false); // Gemini by version is not heuristic-matched + it('returns a concrete window for a pinned model', () => { + expect(contextWindowForModel('gpt-5.5')).toBe(1_050_000); }); - it('the registry is AUTHORITATIVE for a canonical id — the flag (not the id heuristic) decides', () => { - // `deepseek-chat` is a canonical id (the legacy non-thinking alias) with NO `reasoning` flag → false, decided by - // the registry, not any id match. `deepseek-v4-flash`, also canonical, is true via its registry flag (above). - expect(modelSupportsReasoning('deepseek-chat')).toBe(false); - expect(modelSupportsReasoning('deepseek-v4-flash')).toBe(true); + it('returns undefined for a model the catalog does not carry (a custom base URL)', () => { + expect(contextWindowForModel('some-custom-base-url-model-xyz')).toBeUndefined(); + expect(contextWindowForModel('')).toBeUndefined(); }); }); -describe('contextWindowForModel (ADR-0062 §7)', () => { - it('returns the catalog window for a known canonical model', () => { - // Narrow rather than assert (CLAUDE.md rule 1): under noUncheckedIndexedAccess KNOWN_MODEL_IDS[0] is - // `CanonicalModelId | undefined`; the guard narrows it to `CanonicalModelId` (a string, and a valid - // MODEL_PRICING key) so no `as` cast is needed. - const id = KNOWN_MODEL_IDS[0]; - if (id === undefined) throw new Error('MODEL_PRICING catalog is unexpectedly empty'); - const win = contextWindowForModel(id); - expect(win).toBe(MODEL_PRICING[id].contextWindowTokens); - expect(win).toBeGreaterThan(0); +/** The catalog, read as a price (ADR-0071 §1) — what `MODEL_PRICING` used to be, generated instead of typed. */ +describe('catalogPricing — the projection that replaced the hand-typed table', () => { + it('prices every model the catalog carries, and only those', () => { + expect(pricedModelIds()).toHaveLength(Object.keys(CATALOG_SNAPSHOT).length); + expect(pricedModelIds().length).toBeGreaterThan(50); // the retired table had twelve rows + for (const id of pricedModelIds()) { + const priced = catalogPricing(id); + expect(priced, id).toBeDefined(); + expect(priced?.inputPerMtokMicrocents, id).toBeGreaterThanOrEqual(0); + expect(priced?.outputPerMtokMicrocents, id).toBeGreaterThan(0); + } + expect(catalogPricing('a-model-nobody-has-heard-of')).toBeUndefined(); }); - it('returns a concrete window for a specific pinned model (claude-sonnet-4-6 = 1M)', () => { - expect(contextWindowForModel('claude-sonnet-4-6')).toBe(1_000_000); + it("names the model by the catalog's id — there was never a second, `native` name", () => { + // The retired contract carried a `nativeId` alongside the key, for a provider that called the model something + // else. No row ever used it: every single one set `nativeId` equal to its own key. + for (const [id, entry] of Object.entries(CATALOG_SNAPSHOT)) { + expect(toPricing(entry).nativeId, id).toBe(id); + } }); - it('returns undefined for an unknown (custom base-URL) model', () => { - expect(contextWindowForModel('some-custom-base-url-model-xyz')).toBeUndefined(); - expect(contextWindowForModel('')).toBeUndefined(); + it('NEVER bills a cached read at zero — a model with no published cache rate pays the full input rate', () => { + // The first projection wrote `?? 0`. `cost()` computes `cacheReadTokens × rate / 1e6`, so 0 does not mean + // "charge the normal rate" — it means CHARGE NOTHING, and eleven catalog models publish no cache rate. OpenAI + // auto-caches, so the cached fraction of every prompt on `o1-pro` ($150/MTok in) billed at $0.00. + // + // Asserted as BEHAVIOUR, not as a restatement of the projection: the previous test compared `toPricing`'s output + // to `x ?? 0` — the very expression it was testing — so it passed for any projection of the same bug. + const uncached = Object.entries(CATALOG_SNAPSHOT).filter( + ([, e]) => e.cachedInputPerMtokMicrocents === undefined, + ); + expect(uncached.length).toBeGreaterThan(0); // the premise: such models exist + + // 100k cached tokens — deliberately under every context-tier threshold (the lowest is 200k), so this measures + // the cache-rate fallback and nothing else. A 1M-token read would ALSO cross the tier boundary on the models + // that have one, and would be measuring two things at once. + const CACHED = 100_000; + for (const [id, entry] of uncached) { + const billed = cost(id, { inputTokens: 0, outputTokens: 0, cacheReadTokens: CACHED }); + expect(billed, id).toBe(Math.round((CACHED * entry.inputPerMtokMicrocents) / 1_000_000)); + expect(billed, id).toBeGreaterThan(0); // the point: NOT zero + } + + // …and a model that DOES publish a discount still gets it — the fallback is a floor, not a flattening. + const discounted = Object.entries(CATALOG_SNAPSHOT).find( + ([, e]) => + e.cachedInputPerMtokMicrocents !== undefined && + e.cachedInputPerMtokMicrocents < e.inputPerMtokMicrocents && + e.contextTiers === undefined, + ); + expect(discounted).toBeDefined(); + if (discounted === undefined) return; + const [id, entry] = discounted; + expect(cost(id, { inputTokens: 0, outputTokens: 0, cacheReadTokens: 1_000_000 })).toBe( + entry.cachedInputPerMtokMicrocents, + ); + }); + + it('prices a LONG prompt at the tier it actually landed in — not the cheap one (ADR-0071 §11)', () => { + // `gemini-2.5-pro`: $1.25/$10 up to 200k, $2.50/$15 above. The tiers were parsed, guarded and exported — and + // read by nothing, so every long-context turn billed at half price. That was a tolerable gap while these models + // threw `UnknownModelError`; it became a silent 2× under-bill the moment the catalog started pricing them. + const entry = CATALOG_SNAPSHOT['gemini-2.5-pro']; + expect(entry?.contextTiers?.[0]?.aboveContextTokens).toBe(200_000); // the premise + + const base = entry?.inputPerMtokMicrocents ?? 0; + const dear = entry?.contextTiers?.[0]?.inputPerMtokMicrocents ?? 0; + const short = cost('gemini-2.5-pro', { inputTokens: 100_000, outputTokens: 0 }); + const long = cost('gemini-2.5-pro', { inputTokens: 300_000, outputTokens: 0 }); + expect(short).toBe(Math.round((100_000 * base) / 1_000_000)); // under the threshold: the base rate + expect(long).toBe(Math.round((300_000 * dear) / 1_000_000)); // over it: the ABOVE rate + expect(long / 3).toBeGreaterThan(short); // …strictly dearer PER TOKEN, which is the under-bill that was live + }); + + it('the pre-egress estimate takes the HIGHEST tier — a cap that under-estimates lets money escape', () => { + // The estimate cannot know how long the prompt will be (the engine does not tokenize locally). On a SAFETY + // control, guessing the cheap side is the guess that lets real money out. + const entry = CATALOG_SNAPSHOT['gemini-2.5-pro']; + const dearOutput = entry?.contextTiers?.[0]?.outputPerMtokMicrocents ?? 0; + const baseOutput = entry?.outputPerMtokMicrocents ?? 0; + expect(dearOutput).toBeGreaterThan(baseOutput); // the premise + + // The ask is also CLAMPED to the model's 65 536-token ceiling (ADR-0071 §7) — the two rules compose, and this + // pins BOTH: the cap the wire will carry, priced at the tier the estimate must assume. + const cap = entry?.maxOutputTokens ?? 0; + expect(estimateMaxNextCost('gemini-2.5-pro', 1_000_000)).toBe( + Math.round((cap * dearOutput) / 1_000_000), + ); + // …and it is strictly dearer than the cheap tier would have been — which is the money that used to escape. + expect(estimateMaxNextCost('gemini-2.5-pro', 1_000_000)).toBeGreaterThan( + Math.round((cap * baseOutput) / 1_000_000), + ); + }); + + it('does NOT project a reasoning boolean — nothing should ever ask that question again', () => { + const entry = CATALOG_SNAPSHOT['gpt-5.4-pro']; + expect(entry).toBeDefined(); + if (entry === undefined) return; + expect('reasoning' in toPricing(entry)).toBe(false); }); }); diff --git a/packages/llm/src/pricing.ts b/packages/llm/src/pricing.ts index b4a460be..d25e7852 100644 --- a/packages/llm/src/pricing.ts +++ b/packages/llm/src/pricing.ts @@ -1,28 +1,25 @@ import type { MediaBilledModality } from '@relavium/shared'; +import type { CatalogPriceTier } from './catalog/catalog-model.js'; +import { catalogModel } from './catalog/lookup.js'; import type { ProviderId } from './types.js'; /** - * The canonical model-pricing table — the in-code **source of truth** the adapters (1.C/1.G/1.H) - * and the `CostTracker` (1.B) share, keyed on canonical model id. `model_catalog` - * ([database-schema.md](../../../docs/reference/desktop/database-schema.md)) ships empty and is a - * display projection *seeded from here*. Prices are **integer micro-cents per million tokens** - * (1 micro-cent = 1e-8 USD) — no float, ever. The `input` / `output` / `cachedInput` prices map to - * the DB's `*_per_mtok_microcents` columns; `cacheWrite` (Anthropic-only) is **in-code only** — - * there is no `model_catalog` cache-write column, so it is never seeded or persisted. + * The **pricing contract** — what a price IS, not what any model costs + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §1). * - * USD/MTok → micro-cents/MTok is `usd × 1e8` (e.g. $5.00 → 500_000_000). + * The twelve-row hand-typed `MODEL_PRICING` table that used to live here is GONE. It was our own verified data + * once; by the time it was retired it had drifted from reality on two numbers (`claude-sonnet-4-6`'s output + * ceiling, `gpt-5.5`'s context window) and nobody had noticed, because nothing compared it to anything. That is + * not a maintenance failure — it is what a hand-typed table of someone else's prices always becomes. * - * **Verification (2026-06-11).** Every row verified against the provider's live pricing page: - * Anthropic via the claude-api pricing page (Opus 4.8 / Sonnet 4.6 / Haiku 4.5 unchanged; the - * flagship **Claude Fable 5** added); OpenAI (developers.openai.com), Gemini (ai.google.dev), and - * DeepSeek (api-docs.deepseek.com) re-fetched the same day. The prior - * `gpt-4o` / `gpt-4o-mini` / `gemini-2.0-flash` / `gemini-1.5-pro` rows were retired — shut down or - * removed from the provider catalogs — and replaced with the current flagship/mini and Pro/Flash - * models. Gemini Pro/Flash are context-tiered: the ≤200K (Pro) and text/image/video (Flash) tier is - * used here. **DeepSeek re-verified 2026-07-03** (api-docs.deepseek.com/quick_start/pricing): the current ids - * are `deepseek-v4-flash` (default) and `deepseek-v4-pro` (premium), each serving non-thinking + thinking on one - * id; the legacy `deepseek-chat`/`-reasoner` aliases deprecate 2026-07-24 15:59 UTC (re-verify / remove then). + * The numbers now come from the generated catalog (`catalog/snapshot.ts`, synced from models.dev) via + * {@link catalogPricing}, and from the USER, who outranks it: they hold the invoice, we hold a snapshot of a + * third-party aggregator. `priceModel` resolves **user → catalog → throw**. + * + * The shape survives because it is a contract: the `CostTracker` bills against it, the pre-egress governor + * estimates against it, and a `models pricing` row IS one. Prices are **integer micro-cents per million tokens** + * (1 micro-cent = 1e-8 USD) — no float, ever. USD/MTok → micro-cents/MTok is `usd × 1e8`. */ export interface ModelPricing { @@ -35,10 +32,29 @@ export interface ModelPricing { readonly maxOutputTokens: number; readonly inputPerMtokMicrocents: number; readonly outputPerMtokMicrocents: number; - /** Cache-read (cached-input) price; 0 when the provider does not discount cache reads. */ + /** + * Cache-read (cached-input) price — **never 0 for "no discount"** (ADR-0071 §10). + * + * `cost()` bills `cacheReadTokens × this / 1e6`, so 0 does not mean "charge the normal rate", it means **charge + * nothing**. A provider that publishes no cache-read rate does not discount cache reads; it does not hand them + * out free. The projection from the catalog therefore falls back to {@link ModelPricing.inputPerMtokMicrocents}, + * and a user row that omits the dimension inherits the catalog's rather than zeroing it. + */ readonly cachedInputPerMtokMicrocents: number; /** Cache-write price, where the provider charges one (Anthropic does); undefined otherwise. */ readonly cacheWritePerMtokMicrocents?: number; + /** + * Context-size pricing tiers — "above N context tokens, the rate changes" (ADR-0071 §11). + * + * `gemini-2.5-pro` is $1.25/$10 below 200k and **$2.50/$15 above**; `gpt-5.5` and eleven other models the catalog + * newly prices are tiered at 272k. A flat rate understates long-context spend by up to 2×, and the cost cap is a + * SAFETY control, not an estimate. The realized fold prices the tier the prompt actually landed in; the + * pre-egress estimate takes the HIGHEST applicable one, because a cap that over-estimates refuses a turn the user + * could have afforded while one that under-estimates lets real money escape. Only one of those is recoverable. + * + * A USER row carries none: their stated price is a flat price, which is what they told us. + */ + readonly contextTiers?: readonly CatalogPriceTier[]; /** * ISO-8601 date this model is scheduled to retire, if any (ADR-0064 §7). The pure {@link mergeModelCatalog} * flags an entry `deprecated` once `now >= deprecatedAt` (unioned with a live-list deprecation date, taking @@ -60,245 +76,41 @@ export interface ModelPricing { // Keyed by the canonical `MediaBilledModality` set (image/audio/video) via a mapped type, so the keys // stay in sync with `MEDIA_BILLED_MODALITIES` at compile time — never a hand-maintained literal. readonly mediaOutputRates?: { readonly [K in MediaBilledModality]?: number }; - /** - * Whether this model supports a reasoning-effort control ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md)) — - * the static per-model capability the host projects to `resolveReasoning` (gating whether `reasoningEffort` is - * sent + whether the picker offers the effort selector). **Opt-in**: absent ⇒ `false` (the SAFE default — a - * non-reasoning model must never receive the field). Set `true` only for a model whose adapter maps the tier - * (so DeepSeek stays absent until its adapter mapping lands, even though v4 reasons — the effort is not - * controllable there yet). - */ - readonly reasoning?: boolean; -} - -const USD_PER_MTOK_TO_MICROCENTS = 100_000_000; // 1 USD = 1e8 micro-cents -/** USD-per-million-tokens → integer micro-cents-per-million-tokens. */ -const usd = (perMtok: number): number => Math.round(perMtok * USD_PER_MTOK_TO_MICROCENTS); - -/** Canonical model id → pricing. The canonical id is what an authored agent/workflow names. */ -export const MODEL_PRICING = { - // --- Anthropic (verified 2026-06-11: platform.claude.com pricing page) ---------------------- - 'claude-fable-5': { - provider: 'anthropic', - nativeId: 'claude-fable-5', - displayName: 'Claude Fable 5', - reasoning: true, - contextWindowTokens: 1_000_000, - maxOutputTokens: 128_000, - inputPerMtokMicrocents: usd(10), - outputPerMtokMicrocents: usd(50), - cachedInputPerMtokMicrocents: usd(1), // cache read = 0.1× input - cacheWritePerMtokMicrocents: usd(12.5), // cache write (5-min TTL) = 1.25× input - }, - 'claude-opus-4-8': { - provider: 'anthropic', - nativeId: 'claude-opus-4-8', - displayName: 'Claude Opus 4.8', - reasoning: true, - contextWindowTokens: 1_000_000, - maxOutputTokens: 128_000, - inputPerMtokMicrocents: usd(5), - outputPerMtokMicrocents: usd(25), - cachedInputPerMtokMicrocents: usd(0.5), // cache read = 0.1× input - cacheWritePerMtokMicrocents: usd(6.25), // cache write (5-min TTL) = 1.25× input - }, - 'claude-sonnet-4-6': { - provider: 'anthropic', - nativeId: 'claude-sonnet-4-6', - displayName: 'Claude Sonnet 4.6', - reasoning: true, - contextWindowTokens: 1_000_000, - maxOutputTokens: 64_000, - inputPerMtokMicrocents: usd(3), - outputPerMtokMicrocents: usd(15), - cachedInputPerMtokMicrocents: usd(0.3), - cacheWritePerMtokMicrocents: usd(3.75), - }, - 'claude-haiku-4-5': { - provider: 'anthropic', - nativeId: 'claude-haiku-4-5', - displayName: 'Claude Haiku 4.5', - reasoning: true, - contextWindowTokens: 200_000, - maxOutputTokens: 64_000, - inputPerMtokMicrocents: usd(1), - outputPerMtokMicrocents: usd(5), - cachedInputPerMtokMicrocents: usd(0.1), - cacheWritePerMtokMicrocents: usd(1.25), - }, - - // --- OpenAI (verified 2026-06-11: developers.openai.com/api/docs/pricing) ------------------- - 'gpt-5.5': { - provider: 'openai', - nativeId: 'gpt-5.5', - displayName: 'GPT-5.5', - reasoning: true, - contextWindowTokens: 1_000_000, - maxOutputTokens: 128_000, - inputPerMtokMicrocents: usd(5), - outputPerMtokMicrocents: usd(30), - cachedInputPerMtokMicrocents: usd(0.5), // OpenAI auto-caches; no separate write charge - }, - 'gpt-5.4-mini': { - provider: 'openai', - nativeId: 'gpt-5.4-mini', - displayName: 'GPT-5.4 mini', - reasoning: true, - contextWindowTokens: 400_000, - maxOutputTokens: 128_000, - inputPerMtokMicrocents: usd(0.75), - outputPerMtokMicrocents: usd(4.5), - cachedInputPerMtokMicrocents: usd(0.075), - }, - - // --- Gemini (verified 2026-06-11: ai.google.dev/gemini-api/docs/pricing; context-tiered) ---- - 'gemini-2.5-flash': { - provider: 'gemini', - nativeId: 'gemini-2.5-flash', - displayName: 'Gemini 2.5 Flash', - reasoning: true, - contextWindowTokens: 1_048_576, - maxOutputTokens: 65_536, - inputPerMtokMicrocents: usd(0.3), // text/image/video tier (audio: $1.00/MTok) - outputPerMtokMicrocents: usd(2.5), - cachedInputPerMtokMicrocents: usd(0.03), - }, - 'gemini-2.5-pro': { - provider: 'gemini', - nativeId: 'gemini-2.5-pro', - displayName: 'Gemini 2.5 Pro', - reasoning: true, - contextWindowTokens: 1_048_576, - maxOutputTokens: 65_536, - inputPerMtokMicrocents: usd(1.25), // prompts ≤200K tier (>200K: $2.50 in / $15 out) - outputPerMtokMicrocents: usd(10), - cachedInputPerMtokMicrocents: usd(0.125), - }, - - // --- DeepSeek (verified 2026-07-03: api-docs.deepseek.com/quick_start/pricing; via the OpenAI-compatible - // adapter) — the current ids are `deepseek-v4-flash` (default tier) and `deepseek-v4-pro` (premium tier). Each - // serves BOTH non-thinking and thinking (default) modes on ONE id — the mode is a request param, not a - // separate model — so there is no per-mode row. Reasoning-effort IS controllable (ADR-0066): the create-chat- - // completion API takes a `thinking` object (`type: enabled|disabled` + `reasoning_effort: high|max`), mapped in - // openai.ts (`reasoning: true` below). The legacy `deepseek-chat` (non-thinking) / `deepseek-reasoner` (thinking) - // aliases are kept below until they deprecate on 2026-07-24 15:59 UTC and stay reasoning-uncontrollable. - 'deepseek-v4-flash': { - provider: 'deepseek', - nativeId: 'deepseek-v4-flash', - displayName: 'DeepSeek-V4-Flash', - contextWindowTokens: 1_000_000, - maxOutputTokens: 384_000, - inputPerMtokMicrocents: usd(0.14), - outputPerMtokMicrocents: usd(0.28), - cachedInputPerMtokMicrocents: usd(0.0028), // cache-hit input - reasoning: true, // ADR-0066: v4 exposes a controllable `thinking` param (off / high / max) - }, - 'deepseek-v4-pro': { - provider: 'deepseek', - nativeId: 'deepseek-v4-pro', - displayName: 'DeepSeek-V4-Pro', - contextWindowTokens: 1_000_000, - maxOutputTokens: 384_000, - inputPerMtokMicrocents: usd(0.435), - outputPerMtokMicrocents: usd(0.87), - cachedInputPerMtokMicrocents: usd(0.003625), // cache-hit input - reasoning: true, // ADR-0066: v4 exposes a controllable `thinking` param (off / high / max) - }, - // Legacy aliases — deprecating 2026-07-24 15:59 UTC. Kept so an existing agent/config that still names them - // keeps costing correctly until then; the pricing page no longer lists them, so these hold the last verified - // (2026-06-11) values — re-verify or remove at deprecation. - 'deepseek-chat': { - provider: 'deepseek', - nativeId: 'deepseek-chat', - displayName: 'DeepSeek-V4-Flash (chat, legacy)', - contextWindowTokens: 1_000_000, - maxOutputTokens: 384_000, - inputPerMtokMicrocents: usd(0.14), - outputPerMtokMicrocents: usd(0.28), - cachedInputPerMtokMicrocents: usd(0.0028), // cache-hit input - deprecatedAt: '2026-07-24T15:59:00Z', // legacy alias retires 2026-07-24 15:59 UTC (see header) - }, - 'deepseek-reasoner': { - provider: 'deepseek', - nativeId: 'deepseek-reasoner', - displayName: 'DeepSeek-V4-Flash (reasoner, legacy)', - contextWindowTokens: 1_000_000, - maxOutputTokens: 384_000, - // The thinking-mode alias of v4-flash — thinking is a request PARAM, not a separate model, so it bills at the - // SAME v4-flash rate as `deepseek-chat` (the prior 0.435/0.87 was a stale R1-era carryover from before v4). - inputPerMtokMicrocents: usd(0.14), - outputPerMtokMicrocents: usd(0.28), - cachedInputPerMtokMicrocents: usd(0.0028), - deprecatedAt: '2026-07-24T15:59:00Z', // legacy alias retires 2026-07-24 15:59 UTC (see header) - }, -} as const satisfies Readonly>; - -/** The canonical model ids the pricing table covers. */ -export type CanonicalModelId = keyof typeof MODEL_PRICING; - -const CANONICAL_MODEL_IDS = new Set(Object.keys(MODEL_PRICING)); - -/** - * Type guard: is `value` a canonical model id the pricing table covers? Backed by a `Set` of own - * keys, so it is immune to prototype-chain keys (`'toString'`, `'constructor'`, …) — unlike `in`. - */ -export function isCanonicalModelId(value: string): value is CanonicalModelId { - return CANONICAL_MODEL_IDS.has(value); + // `reasoning?: boolean` lived here, and is GONE (ADR-0071 §6). "Does this model reason" is not the question the + // wire asks — `gpt-5.4-pro` reasons AND rejects `low`, `claude-haiku-4-5` reasons with no effort ladder at all, + // `gemini-2.5-pro` reasons and cannot be turned off. The catalog carries the CONTROL, and `effortTiersFor` reads + // it. A price is a price; it was never the right place to answer a capability question. } -/** Every canonical model id, for diagnostics (e.g. the unknown-model error). */ -export const KNOWN_MODEL_IDS: readonly CanonicalModelId[] = - Object.keys(MODEL_PRICING).filter(isCanonicalModelId); +// `MODEL_PRICING`, `CanonicalModelId`, `isCanonicalModelId`, `KNOWN_MODEL_IDS` lived here — all GONE (ADR-0071 §1). +// +// Twelve models, hand-typed from four pricing pages, re-verified by hand whenever someone remembered. The catalog +// carries eighty, generated, with an output ceiling and a per-model reasoning control the table never had — and it +// is regenerated by `pnpm sync:models` with money-guards that refuse a silent price change. +// +// `isCanonicalModelId` is not replaced by an equivalent. It asked "is this one of OUR models", and the honest +// question is "can we price this one" — `catalogPricing(id) !== undefined`, or `priceModel`'s own throw. The +// difference matters: a user-priced model is perfectly billable and was never "canonical". -/** - * A CONSERVATIVE model-id reasoning heuristic ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md) §4) - * — the second arm of {@link modelSupportsReasoning}, applied ONLY to an id absent from the static registry (a - * live-discovered model whose list endpoint omits a reasoning flag). Each arm is a family/pattern where the WHOLE - * matched set reasons, so a new member of a known reasoning family (e.g. a next o-series id) gates correctly before - * the registry is updated. Deliberately **narrow**: it does NOT prefix-match ambiguous families whose lineup mixes - * reasoning and non-reasoning members by *version* (base Claude Sonnet, Gemini by version), because OVER-matching - * would send the tier to a non-reasoning model and earn a provider rejection — strictly worse than the safe - * under-match (no effort UX until the registry adds the model, the same maintenance shape as pricing). DeepSeek IS - * matched, but only the `deepseek-v[4-9]` prefix (whose whole set serves the `thinking` param); its legacy - * `deepseek-chat`/`-reasoner` aliases are not `v`-prefixed, so they never match. The `-chat` exclusion keeps - * OpenAI's non-reasoning `gpt-5-chat` conversational variant — and any future `deepseek-v_-chat` non-thinking - * variant — out. - */ -export function reasoningModelIdHeuristic(model: string): boolean { - const m = model.toLowerCase(); - if (/^o\d/.test(m)) return true; // OpenAI o-series (o1 / o3 / o4 / o5+) — the entire family reasons - if (m.startsWith('gpt-5') && !m.includes('chat')) return true; // the reasoning gpt-5 line (gpt-5-chat is non-reasoning) - if (m.startsWith('claude-opus')) return true; // Claude Opus reasons (extended thinking) - if (/^deepseek-v[4-9]/.test(m) && !m.includes('chat')) return true; // DeepSeek v4+ serves the `thinking` param; a future `-chat` non-thinking variant stays out (ADR-0066 §4) - if (m.includes('thinking')) return true; // an explicit "thinking" model id (e.g. a Gemini thinking variant) - return false; -} - -/** - * Whether a model supports a reasoning-effort control ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md) - * §4) — the per-model capability the host projects to the engine's `resolveReasoning` gate (and the `/models` - * picker's effort selector). The STATIC registry is authoritative for a canonical id (its `reasoning` flag, `true` - * OR `false` — so an explicit non-reasoning member always wins); a NON-registry id (a live-discovered model) falls - * back to the conservative {@link reasoningModelIdHeuristic}. A pure host-side helper, like {@link contextWindowForModel}. - */ -export function modelSupportsReasoning(model: string): boolean { - if (isCanonicalModelId(model)) { - // Widen the literal-union entry to `ModelPricing` so `.reasoning` (absent on the non-reasoning members) reads as - // `boolean | undefined` — each entry `satisfies ModelPricing`, so this is assignment, not a cast. The registry - // is authoritative for a known id, so a false/absent flag is NOT overridden by the id heuristic. - const entry: ModelPricing = MODEL_PRICING[model]; - return entry.reasoning === true; - } - return reasoningModelIdHeuristic(model); -} +// `reasoningModelIdHeuristic` / `modelSupportsReasoning` lived here too, and are GONE (ADR-0071 §6). +// +// They answered "does this model reason" by pattern-matching its id — `/^o\d/`, `startsWith('gpt-5')`, and so on — +// and that is not the question the wire asks. `gpt-5.4-pro` reasons AND rejects `low`; `claude-haiku-4-5` reasons +// and has no effort ladder at all; `gemini-2.5-pro` reasons and cannot be turned off. A boolean answers `true` to +// every one of them, and the tier the user picked went straight to the provider. `effortTiersFor(id)` is the one +// predicate every surface asks now. /** - * The context window (max tokens) for a canonical model id, or `undefined` for an unknown id (e.g. a custom - * base-URL model absent from the catalog). A light, pure host-side helper — the SAME catalog value the adapters' - * `LlmProvider.contextLimit` returns — for UI that needs the window WITHOUT going through the provider seam: the - * ADR-0062 §7 footer context-fullness indicator (last input tokens ÷ window). A custom-model `undefined` degrades - * the indicator to "not shown", exactly as it degrades auto-compaction (ADR-0062 §5). + * The context window for a model id, or `undefined` for one the catalog does not carry (a custom base-URL model). + * A light, pure host-side helper — the SAME value the adapters' `LlmProvider.contextLimit` returns — for UI that + * needs the window WITHOUT going through the provider seam: the ADR-0062 §7 footer context-fullness indicator + * (last input tokens ÷ window). A custom-model `undefined` degrades the indicator to "not shown", exactly as it + * degrades auto-compaction (ADR-0062 §5). + * + * It read the hand-typed table until 2026-07-13, and the table said `gpt-5.5`'s window was 1 000 000 while the + * generated catalog said 1 050 000 — so the fullness percentage every user saw was computed against a window the + * model does not have. */ export function contextWindowForModel(model: string): number | undefined { - return isCanonicalModelId(model) ? MODEL_PRICING[model].contextWindowTokens : undefined; + return catalogModel(model)?.contextWindowTokens; } diff --git a/packages/llm/src/reasoning-wire.test.ts b/packages/llm/src/reasoning-wire.test.ts new file mode 100644 index 00000000..153f5a42 --- /dev/null +++ b/packages/llm/src/reasoning-wire.test.ts @@ -0,0 +1,363 @@ +import { describe, expect, it } from 'vitest'; + +import { CATALOG_SNAPSHOT } from './catalog/snapshot.js'; +import { + acceptedTiers, + acceptedWireValue, + canDisableReasoning, + openAiWireValue, + reasoningBudgetFor, + reasoningControlShape, + reasoningWithheldByCap, + thinkingCeiling, + wireValueFor, +} from './reasoning-wire.js'; + +/** + * THE EFFORT BRIDGE (ADR-0071 §6) — the fix for the maintainer's bug report, and the one place a literal read of + * the catalog is actively harmful. + * + * The catalog's `effortValues` are **provider-wire** strings; Relavium's tiers are a different vocabulary. Read + * one as the other and you drop `off` from every Claude model (where `off` is `thinking:{disabled}`, not an + * effort value) and drop `off`+`max` from `gpt-5.5`. So the accepted set is COMPUTED — and these tests compute + * it against the REAL shipped catalog, not against fixtures, because a bridge that is right about a fixture and + * wrong about `gemini-2.5-pro` is worth nothing. + */ + +const tiers = (modelId: string): string[] => { + const model = CATALOG_SNAPSHOT[modelId]; + if (model === undefined) throw new Error(`${modelId} is not in the shipped catalog`); + return [...acceptedTiers(model.provider, model.reasoning)].sort(); +}; + +describe('reasoningControlShape — the PRESENTATION shape a picker projects (ADR-0066 amendment)', () => { + it("effortValues ⇒ 'graded', even alongside a co-published budget (claude-opus-4-5 has both)", () => { + expect(reasoningControlShape({ effortValues: ['medium', 'high', 'xhigh'] })).toBe('graded'); + expect( + reasoningControlShape({ + effortValues: ['low', 'medium', 'high'], + budgetTokens: { min: 1024 }, + }), + ).toBe('graded'); + expect(reasoningControlShape({ toggle: true, effortValues: ['high', 'max'] })).toBe('graded'); + }); + + it("a budget with no ladder ⇒ 'budget' (the off/on case)", () => { + expect(reasoningControlShape({ budgetTokens: { min: 1024 } })).toBe('budget'); + expect(reasoningControlShape({ toggle: true, budgetTokens: { min: 0, max: 24576 } })).toBe( + 'budget', + ); + }); + + it("no usable knob ⇒ 'none' (empty descriptor, a lone toggle, or absent)", () => { + expect(reasoningControlShape({})).toBe('none'); // deepseek-reasoner: reasons, no control + expect(reasoningControlShape({ effortValues: [] })).toBe('none'); // an empty ladder is no ladder + expect(reasoningControlShape({ toggle: true })).toBe('none'); // no tier to map "on" to + expect(reasoningControlShape(undefined)).toBe('none'); // does not reason + }); +}); + +describe('the bug report — gpt-5.4-pro REJECTS the tier we offer it today', () => { + it('accepts {medium, high, max} and NOT low, NOT off', () => { + // Catalog: effortValues ['medium','high','xhigh']. Our wire map: max→'xhigh', low→'low', off→'none'. + // So `low` and `off` have no wire value the model takes — and today's picker offers both, which is the 400. + expect(tiers('gpt-5.4-pro')).toEqual(['high', 'max', 'medium']); + }); + + it('gpt-5-pro accepts ONLY high — a single-tier model, which a boolean could never express', () => { + expect(tiers('gpt-5-pro')).toEqual(['high']); + }); + + it('gpt-5.5 keeps ALL FIVE — a literal read of its values would have dropped off and max', () => { + // Its wire values are none/low/medium/high/xhigh. `off`→'none' ✓ and `max`→'xhigh' ✓, both of which a naive + // "is our tier name in the list?" check would have missed entirely. + expect(tiers('gpt-5.5')).toEqual(['high', 'low', 'max', 'medium', 'off']); + }); +}); + +describe('OpenAI top tier is PER MODEL — the gpt-5.6 family publishes a `max` above `xhigh` (review M1)', () => { + // gpt-5.6 effortValues: [...,'xhigh','max']; gpt-5.5 tops at 'xhigh'; gpt-5.4-pro at 'xhigh' too. `max` must + // reach the MODEL's own highest value, not a fixed alias — otherwise the picker's "max" row silently sends the + // second-strongest tier and the flagship's true maximum is unreachable from every surface. + it('openAiWireValue reads the model ladder: max→max when published, else max→xhigh', () => { + const wix6 = CATALOG_SNAPSHOT['gpt-5.6']?.reasoning; + const wix5 = CATALOG_SNAPSHOT['gpt-5.5']?.reasoning; + if (wix6 === undefined || wix5 === undefined) + throw new Error('gpt-5.6/gpt-5.5 not in the catalog'); + expect(openAiWireValue('max', wix6)).toBe('max'); // gpt-5.6 publishes 'max' + expect(openAiWireValue('max', wix5)).toBe('xhigh'); // gpt-5.5 tops at 'xhigh' + expect(openAiWireValue('high', wix6)).toBe('high'); // intermediate tiers unchanged + expect(openAiWireValue('off', wix6)).toBe('none'); // off still maps to 'none' + }); + + it('wireValueFor is catalog-aware WITH controls and defaults to xhigh WITHOUT them (dedup path)', () => { + const wix6 = CATALOG_SNAPSHOT['gpt-5.6']?.reasoning; + if (wix6 === undefined) throw new Error('gpt-5.6 not in the catalog'); + expect(wireValueFor('openai', 'max', wix6)).toBe('max'); + expect(wireValueFor('openai', 'max')).toBe('xhigh'); // no controls ⇒ the static top (never collides on dedup) + }); + + it('acceptedTiers offers max for gpt-5.6 (its wire `max` is a published value)', () => { + expect(tiers('gpt-5.6')).toContain('max'); + // …and acceptedWireValue resolves that tier to the model's own top, not the coarsened xhigh. + const wix6 = CATALOG_SNAPSHOT['gpt-5.6']?.reasoning; + if (wix6 === undefined) throw new Error('gpt-5.6 not in the catalog'); + expect(acceptedWireValue('openai', 'max', wix6)).toBe('max'); + }); +}); + +describe('the live Gemini bug — the catalog says what Google says', () => { + it('gemini-2.5-pro CANNOT be turned off — its budget floor is 128, not 0', () => { + // Google: "N/A: Cannot disable thinking". The catalog: budgetTokens.min = 128. Same fact, one field. + // Today's adapter maps off→MINIMAL and sends it anyway — a value the model does not take. + const model = CATALOG_SNAPSHOT['gemini-2.5-pro']; + expect(model?.reasoning?.budgetTokens?.min).toBe(128); + expect(canDisableReasoning('gemini', model?.reasoning ?? {})).toBe(false); + expect(tiers('gemini-2.5-pro')).not.toContain('off'); + expect(tiers('gemini-2.5-pro')).toEqual(['high', 'low', 'max', 'medium']); // gradable via the budget + }); + + it('gemini-2.5-flash CAN be turned off — its budget floor IS 0', () => { + expect(CATALOG_SNAPSHOT['gemini-2.5-flash']?.reasoning?.budgetTokens?.min).toBe(0); + expect(tiers('gemini-2.5-flash')).toContain('off'); + }); + + it('a Gemini EFFORT model still cannot be turned off — MINIMAL is the floor, not an off switch', () => { + // `gemini-3.5-flash` publishes ['minimal','low','medium','high']. `minimal` is the *lowest* level; a model + // set to it still thinks. Disabling is `thinkingBudget: 0`, a field this model does not take — so `off` is + // not on offer, and mapping off→MINIMAL (as the adapter does today) would bill the user for reasoning they + // asked not to have. + expect(tiers('gemini-3.5-flash')).not.toContain('off'); + }); +}); + +describe('`off` is not an effort value on three of four providers — the asymmetry a literal read destroys', () => { + it('every reasoning ANTHROPIC model can be turned off — the switch is independent of the effort ladder', () => { + // `thinking: {type:'disabled'}`. `off` appears in NO Claude model's effortValues, so a literal read drops it + // from all of them. + for (const [id, model] of Object.entries(CATALOG_SNAPSHOT)) { + if (model.provider !== 'anthropic' || model.reasoning === undefined) continue; + if (Object.keys(model.reasoning).length === 0) continue; // no control at all — see below + expect(tiers(id), `${id} must be able to turn reasoning off`).toContain('off'); + } + }); + + it('claude-haiku-4-5 is BUDGET-shaped — gradable, and off-able, with no effort axis at all', () => { + // The maintainer confirmed this independently. Our adapter sends `output_config.effort` to it today. + expect(CATALOG_SNAPSHOT['claude-haiku-4-5']?.reasoning?.effortValues).toBeUndefined(); + expect(tiers('claude-haiku-4-5')).toEqual(['high', 'low', 'max', 'medium', 'off']); + }); + + it('deepseek-v4-pro keeps all five — low/medium/high all coarsen onto its single `high` wire value', () => { + // Its values are ['high','max']. A literal read would offer only {high, max}; the truth is that `low` and + // `medium` are *expressible* (they coarsen to `high`), and `off` is the independent disable switch. + expect(tiers('deepseek-v4-pro')).toEqual(['high', 'low', 'max', 'medium', 'off']); + }); +}); + +describe('the empty descriptor — reasons, but has NO controllable tier', () => { + it('deepseek-reasoner offers NOTHING — not even off', () => { + // `reasoning: {}` is a real, distinct state: the model thinks, and upstream declines to describe any control + // for it. The safe answer is to withhold the field entirely. Adding `off` on DeepSeek's *general* ability to + // disable would be a guess about a model whose capability nobody documented — and a guess is exactly what + // put a rejected value on the wire in the first place. + expect(CATALOG_SNAPSHOT['deepseek-reasoner']?.reasoning).toEqual({}); + expect(tiers('deepseek-reasoner')).toEqual([]); + }); + + it('a NON-reasoning model offers nothing either', () => { + expect([...acceptedTiers('openai', undefined)]).toEqual([]); + }); +}); + +describe('MEMBERSHIP, not presence — the bug an adversarial review found in the fix itself', () => { + /** + * The adapters branched on `controls.effortValues !== undefined` — the PRESENCE of an effort axis — and then + * sent the mapped wire value UNCHECKED. Presence is not membership, and the gap is a 400: + * + * claude-opus-4-5 publishes ['low','medium','high'] — no `max` → tier `max` sent `effort: 'max'` + * gemini-3-pro-preview publishes ['low','high'] — no `medium` → tier `medium` sent `thinkingLevel: MEDIUM` + * + * Both reach the wire through a FAILOVER, and that is the sting: the fallback chain exists to RESCUE a failing + * turn, and a 400 on an unsupported parameter is fatal and non-retryable — so the rescue kills the turn instead. + */ + it('claude-opus-4-5 does not accept `max` AS AN EFFORT LEVEL — but its budget axis still serves it', () => { + const model = CATALOG_SNAPSHOT['claude-opus-4-5']; + expect(model?.reasoning?.effortValues).toEqual(['low', 'medium', 'high']); // the premise + // No `effort: 'max'` on the wire — that is the 400 this work removes… + expect(acceptedWireValue('anthropic', 'max', model?.reasoning ?? {})).toBeUndefined(); + // …but the model ALSO publishes `budgetTokens`, and the Anthropic adapter falls back to it, so the tier is + // genuinely reachable and the picker must keep offering it. Reading "not on the ladder" as "not accepted" + // would have hidden a tier the model serves perfectly well — the second review caught it. + expect(model?.reasoning?.budgetTokens?.min).toBe(1024); + expect(tiers('claude-opus-4-5')).toContain('max'); + }); + + it('gemini-3-pro-preview does NOT accept `medium` — its ladder skips it', () => { + expect(CATALOG_SNAPSHOT['gemini-3-pro-preview']?.reasoning?.effortValues).toEqual([ + 'low', + 'high', + ]); + expect(tiers('gemini-3-pro-preview')).not.toContain('medium'); + }); + + it('gpt-5.4-pro does NOT accept `low` — the original bug report, at the wire layer', () => { + const model = CATALOG_SNAPSHOT['gpt-5.4-pro']; + expect(acceptedWireValue('openai', 'low', model?.reasoning ?? {})).toBeUndefined(); + expect(acceptedWireValue('openai', 'high', model?.reasoning ?? {})).toBe('high'); + }); +}); + +describe('the Gemini toggle divergence — the picker offered `off` and the adapter dropped it', () => { + it('a toggle-shaped model CAN be turned off, even with a non-zero budget floor', () => { + // `gemini-2.5-flash-lite` has `{ toggle: true, budgetTokens: { min: 512 } }`. `canDisableReasoning` said yes + // (the toggle), so the picker OFFERED `off` — but the adapter's off-branch only looked at `min === 0` and + // silently withheld the field. The user turned reasoning off, was billed for it anyway, and nothing said so. + const model = CATALOG_SNAPSHOT['gemini-2.5-flash-lite']; + expect(model?.reasoning?.toggle).toBe(true); + expect(model?.reasoning?.budgetTokens?.min).toBe(512); // NOT zero — which is why the two disagreed + expect(canDisableReasoning('gemini', model?.reasoning ?? {})).toBe(true); + expect(tiers('gemini-2.5-flash-lite')).toContain('off'); + }); + + it('…and gemini-2.5-pro still cannot — no toggle, and a floor of 128', () => { + expect(canDisableReasoning('gemini', CATALOG_SNAPSHOT['gemini-2.5-pro']?.reasoning ?? {})).toBe( + false, + ); + }); +}); + +describe('the answer must survive the thinking — a budget that eats the cap is not a budget', () => { + it('the `max` tier leaves room to REPLY', () => { + // `max` used to spend 100% of the output cap on thoughts: `budget_tokens: max_tokens - 1` on Anthropic (one + // token of answer), `thinkingBudget == maxOutputTokens` on Gemini (none at all). Both are ACCEPTED by the + // provider — which is what makes it insidious. The user pays for a full turn of reasoning and gets nothing. + const cap = 8192; + const budget = reasoningBudgetFor('max', { min: 1024 }, thinkingCeiling(cap)); + expect(budget).toBeDefined(); + expect(budget).toBeLessThan(cap); + expect(cap - (budget ?? 0)).toBeGreaterThanOrEqual(cap * 0.19); // ~20% reserved for the answer + }); + + it('a cap too small to hold the model floor AND an answer yields NO budget at all', () => { + // Withhold, never squeeze. haiku's floor is 1024; a 1024-token cap cannot carry both. + expect(reasoningBudgetFor('low', { min: 1024 }, thinkingCeiling(1024))).toBeUndefined(); + expect(reasoningBudgetFor('low', { min: 1024 }, thinkingCeiling(1280))).toBe(1024); // the first that can + }); +}); + +describe('reasoningWithheldByCap — the gate can SAY what the adapter used to drop in silence (review M6)', () => { + const haiku = CATALOG_SNAPSHOT['claude-haiku-4-5']?.reasoning; // budget-shaped: { min: 1024 }, no ladder + const opus45 = CATALOG_SNAPSHOT['claude-opus-4-5']?.reasoning; // effort ladder + a budget + + it('a budget-only model withholds under a tight cap, and does NOT under a roomy one', () => { + if (haiku === undefined) throw new Error('claude-haiku-4-5 not in the catalog'); + expect(reasoningWithheldByCap('anthropic', haiku, 'medium', 500)).toBe(true); // 500 < floor+answer → dropped + expect(reasoningWithheldByCap('anthropic', haiku, 'medium', 8000)).toBe(false); // room for the budget + }); + + it('a tier reachable via the EFFORT LADDER is never cap-withheld — it carries no budget', () => { + if (opus45 === undefined) throw new Error('claude-opus-4-5 not in the catalog'); + // `high` is on opus-4-5's ladder → sent as an effort level, so even a tiny cap does not withhold it. + expect(reasoningWithheldByCap('anthropic', opus45, 'high', 100)).toBe(false); + }); + + it('OpenAI/DeepSeek have no budget field, so nothing is ever cap-withheld there', () => { + if (haiku === undefined) throw new Error('claude-haiku-4-5 not in the catalog'); + // Even fed a budget descriptor, an effort-only provider has no budget to exceed. + expect(reasoningWithheldByCap('openai', haiku, 'medium', 1)).toBe(false); + expect(reasoningWithheldByCap('deepseek', haiku, 'medium', 1)).toBe(false); + }); +}); + +describe('an UNKNOWN model gets NO reasoning field — on ALL FOUR arms, not just two', () => { + /** + * Found by an adversarial review, and it was real. Fixing Gemini and Anthropic's *effort* paths left OpenAI and + * DeepSeek sending the field unconditionally — and Anthropic's `off` branch sat in FRONT of the catalog check, + * so an unknown model was sent `thinking: {type:'disabled'}`, which is still a field and still a 400 on a model + * with no reasoning surface. + * + * The host's gate already withholds for an unknown model. The adapter must not depend on a caller having run it: + * `@relavium/llm` is a public seam, and the whole point of this change is that we never guess at a model we + * cannot describe. + */ + const unknown = 'some-custom-endpoint-model'; + + it('the catalog genuinely does not know it — the premise of every case below', () => { + expect(CATALOG_SNAPSHOT[unknown]).toBeUndefined(); + expect(acceptedTiers('openai', undefined).size).toBe(0); + }); + + it('acceptedTiers returns EMPTY for it on every provider, so the gate withholds', () => { + for (const provider of ['openai', 'anthropic', 'gemini', 'deepseek'] as const) { + expect(acceptedTiers(provider, undefined).size, provider).toBe(0); + } + }); +}); + +describe('the whole shipped catalog — no model is offered a tier it would reject', () => { + /** The wire value each provider would send for a tier — the same table `wireValueFor` implements. */ + const WIRE = { + openai: { low: 'low', medium: 'medium', high: 'high', max: 'xhigh' }, + anthropic: { low: 'low', medium: 'medium', high: 'high', max: 'max' }, + gemini: { low: 'low', medium: 'medium', high: 'high', max: 'high' }, + deepseek: { low: 'high', medium: 'high', high: 'high', max: 'max' }, + } as const; + + it('every offered tier is one the ADAPTER can actually put on the wire — by ladder or by budget', () => { + // The invariant the picker rests on. If it holds for all 80 models, no interactive path can produce a 400. + // + // "Publishes the wire value" is NOT the whole invariant, and reading it that way is what an adversarial review + // caught: `claude-opus-4-5`'s ladder omits `max`, but it ALSO publishes a token budget, and its adapter serves + // `max` from that. A tier is legitimate if EITHER route exists — and only if the provider's adapter actually + // has a budget field (OpenAI's `reasoning_effort` and DeepSeek's `thinking` do not, so a budget in the catalog + // buys those models nothing). + const BUDGET_CAPABLE = new Set(['anthropic', 'gemini']); + for (const [id, model] of Object.entries(CATALOG_SNAPSHOT)) { + if (model.reasoning === undefined) continue; + const published = new Set(model.reasoning.effortValues ?? []); + const viaBudget = + model.reasoning.budgetTokens !== undefined && BUDGET_CAPABLE.has(model.provider); + for (const tier of acceptedTiers(model.provider, model.reasoning)) { + if (tier === 'off') continue; // `off` rides the provider's disable axis, checked below. + const servable = published.has(WIRE[model.provider][tier]) || viaBudget; + expect(servable, `${id}: tier '${tier}' is offered but no adapter route can send it`).toBe( + true, + ); + } + } + }); + + it('a BUDGET on a provider whose adapter has no budget field buys the model nothing', () => { + // The mirror of the case above, and the reason `acceptedTiers` cannot blindly union the two axes. If OpenAI + // ever ships a model with `budgetTokens`, the picker must NOT start offering every tier for it: `reasoning_effort` + // is the only reasoning knob that adapter has, and a tier outside the published ladder would be a 400. + const openaiWithBudget = { effortValues: ['high'], budgetTokens: { min: 1024 } } as const; + expect([...acceptedTiers('openai', openaiWithBudget)]).toEqual(['high']); + + const deepseekWithBudget = { budgetTokens: { min: 1024 } } as const; + // A budget alone gives DeepSeek no gradation — but `thinking: {disabled}` still exists, so `off` survives. + expect([...acceptedTiers('deepseek', deepseekWithBudget)]).toEqual(['off']); + }); + + it('no model that cannot be disabled is ever offered `off`', () => { + for (const [id, model] of Object.entries(CATALOG_SNAPSHOT)) { + if (model.reasoning === undefined) continue; + const offered = acceptedTiers(model.provider, model.reasoning).has('off'); + if (!offered) continue; + expect(canDisableReasoning(model.provider, model.reasoning), `${id} offers off`).toBe(true); + } + }); + + it('an EMPTY descriptor cannot be disabled either — not even on a provider with a disable switch', () => { + // `deepseek-reasoner` ships `reasoning: {}`: it reasons, and upstream describes no knob. `canDisableReasoning` + // used to answer `true` for anthropic/deepseek on the provider's general ability, regardless of the model — and + // that answer is what let the adapters send `thinking: {type:'disabled'}` to a model that never said it takes + // one. An empty descriptor means there is nothing to turn. + expect(canDisableReasoning('deepseek', {})).toBe(false); + expect(canDisableReasoning('anthropic', {})).toBe(false); + expect(acceptedTiers('deepseek', CATALOG_SNAPSHOT['deepseek-reasoner']?.reasoning).size).toBe( + 0, + ); + }); +}); diff --git a/packages/llm/src/reasoning-wire.ts b/packages/llm/src/reasoning-wire.ts new file mode 100644 index 00000000..755adbef --- /dev/null +++ b/packages/llm/src/reasoning-wire.ts @@ -0,0 +1,373 @@ +import { REASONING_EFFORTS, type ReasoningEffort } from '@relavium/shared'; + +import type { ReasoningControls } from './catalog/catalog-model.js'; +import type { ProviderId } from './types.js'; + +/** + * How Relavium's normalized reasoning tier becomes a PROVIDER-WIRE value — the one canonical home of that + * mapping ([ADR-0066](../../../docs/decisions/0066-normalized-reasoning-effort-control.md), + * [ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * It used to live as four private consts inside three adapters. It moved here when a second consumer appeared: + * `acceptedTiers`, which must compose these maps with the **catalog's** per-model wire values to decide which + * tiers a model will actually accept. The composition is the whole fix for the maintainer's bug — `gpt-5.4-pro` + * accepts `{medium, high, xhigh}` and rejects `low` — and it is only possible if the map and the catalog are + * read together. Two copies of this map would be two chances to disagree about what we send to a provider. + * + * **The `off` tier is NOT an effort value on three of the four providers**, and that asymmetry is the reason a + * literal read of the catalog's `effortValues` is actively wrong (it would drop `off` from every Claude model): + * + * | provider | `off` is expressed as | so `off` is available when… | + * |-----------|-------------------------------------------|------------------------------------------------------| + * | anthropic | `thinking: { type: 'disabled' }` | always — an independent switch, not an effort value | + * | deepseek | `thinking: { type: 'disabled' }` | always — likewise | + * | openai | `reasoning_effort: 'none'` | **only if `'none'` is in the model's effort values** | + * | gemini | `thinkingConfig.thinkingBudget: 0` | **only if the model takes a budget whose min is 0** | + * + * That last row is what Google's docs say in prose and what the catalog says in one field: `gemini-2.5-pro` has + * `budgetTokens.min = 128`, so it **cannot disable thinking at all** — and `off` must therefore not be offered + * for it. `gemini-2.5-flash` has `min = 0`, so it can. + */ + +/** + * OpenAI's `reasoning_effort`. `off` maps to `'none'` — here it IS an effort value, unlike the other three. `max` + * is the DEFAULT top (`'xhigh'`), but a model may publish a HIGHER `'max'` above it — see {@link openAiWireValue}, + * which reads the model's own ladder so the top tier is per-model, never a fixed alias. + */ +export const OPENAI_WIRE: Record = { + off: 'none', + low: 'low', + medium: 'medium', + high: 'high', + max: 'xhigh', // the DEFAULT top; a model that publishes `'max'` overrides this — see openAiWireValue. +}; + +/** OpenAI's per-model wire value for a tier (review M1). The `gpt-5.6` family publishes a distinct `'max'` ABOVE + * `'xhigh'`, so the top normalized tier must reach the model's OWN highest value, not stop one rung short at the + * static `'xhigh'`; a model with no published `'max'` (`gpt-5.4-pro`, top `'xhigh'`) keeps `OPENAI_WIRE.max`. + * ONE home for this — {@link wireValueFor}, {@link acceptedTiers}, {@link acceptedWireValue} and the adapter all + * read it, so they can never disagree about what "max" sends. */ +export function openAiWireValue( + tier: ReasoningEffort, + controls: ReasoningControls, +): 'none' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' { + if (tier === 'max' && controls.effortValues?.includes('max') === true) return 'max'; + return OPENAI_WIRE[tier]; +} + +/** Anthropic's `output_config.effort`. `off` is absent on purpose — it is `thinking: {type:'disabled'}` instead. */ +export const ANTHROPIC_WIRE: Record< + Exclude, + 'low' | 'medium' | 'high' | 'max' +> = { + low: 'low', + medium: 'medium', + high: 'high', + max: 'max', // Anthropic has a native `max`, so all four non-`off` tiers map 1:1. +}; + +/** + * Gemini's `thinkingConfig.thinkingLevel`. `off` is absent: `MINIMAL` is the *lowest* level, **not** off — a + * model set to MINIMAL still thinks. Disabling is `thinkingBudget: 0`, a different field entirely, which is why + * mapping `off → MINIMAL` (as the shipped adapter does today) both fails to disable thinking and bills the user + * for reasoning they asked not to have. + */ +/** Gemini's thinking ladder, in the CATALOG's (lowercase) vocabulary — the one name for the three levels it takes. */ +export type GeminiWireLevel = 'low' | 'medium' | 'high'; + +export const GEMINI_WIRE: Record, GeminiWireLevel> = { + low: 'low', + medium: 'medium', + high: 'high', + max: 'high', // Gemini's ladder stops at HIGH; `max` coarsens onto it, honestly and documentedly. +}; + +/** + * Every map above is spelled in the CATALOG's vocabulary — lowercase — because its first job is to be compared + * against `ReasoningControls.effortValues`, which is what upstream publishes. Gemini's own `ThinkingLevel` enum + * is the **uppercase** form of the same tokens (`HIGH`, not `high`), so its adapter upper-cases at the wire and + * nowhere else. One map, two spellings of the same token — never two maps that can drift. + */ +const GEMINI_THINKING_LEVEL: Record = { + low: 'LOW', + medium: 'MEDIUM', + high: 'HIGH', +}; +export const toGeminiThinkingLevel = (wire: GeminiWireLevel): 'LOW' | 'MEDIUM' | 'HIGH' => + GEMINI_THINKING_LEVEL[wire]; + +/** + * DeepSeek's `thinking.reasoning_effort`. v4 exposes only two graded levels, so `low`/`medium`/`high` all + * coarsen to `high` and `max` → `max`. `off` is `thinking: {type:'disabled'}`, an independent switch. + */ +export const DEEPSEEK_WIRE: Record, 'high' | 'max'> = { + low: 'high', + medium: 'high', + high: 'high', + max: 'max', +}; + +/** + * The wire value a provider would send for a NON-`off` tier, or `undefined` if the provider has no such map. + * + * `controls` is consulted only for OpenAI's top tier (the model's own `'max'` vs the static `'xhigh'`, review M1); + * omit it (the picker's dedup/projection path) and OpenAI's `max` falls to the default `'xhigh'` — which never + * collides with another tier, so a dedup is unaffected. The gate/adapter path passes it for wire-exact truth. + */ +export function wireValueFor( + provider: ProviderId, + tier: Exclude, + controls?: ReasoningControls, +): string | undefined { + switch (provider) { + case 'openai': + return openAiWireValue(tier, controls ?? {}); + case 'anthropic': + return ANTHROPIC_WIRE[tier]; + case 'gemini': + return GEMINI_WIRE[tier]; + case 'deepseek': + return DEEPSEEK_WIRE[tier]; + } +} + +/** + * Does this model publish **any** control at all? + * + * An EMPTY descriptor (`{}` — `deepseek-reasoner`) is not the same as no descriptor. The model reasons; upstream + * simply declined to describe a knob for it. The honest reading is that there is nothing to turn, so every + * predicate below answers `false` for it and the field is withheld entirely. + */ +function hasAnyControl(controls: ReasoningControls): boolean { + return ( + controls.effortValues !== undefined || + controls.budgetTokens !== undefined || + controls.toggle === true + ); +} + +/** + * Can this adapter express a tier the model's effort ladder does NOT contain, as a **token budget** instead? + * + * Only two of the four can. Anthropic takes `thinking.budget_tokens` and Gemini takes `thinkingBudget`, so for + * those a budget axis genuinely widens what the model accepts. OpenAI's `reasoning_effort` and DeepSeek's + * `thinking.reasoning_effort` are effort-shaped and nothing else: a budget in the catalog is a fact about the + * MODEL that our request for it cannot use. + * + * This is why {@link acceptedTiers} cannot simply union the two axes. A blind union would have the picker offer + * every tier for an OpenAI model that publishes a budget, and the adapter — which has no field to put it in — + * would withhold. That is the picker/wire divergence in a new costume. + */ +function honorsBudget(provider: ProviderId): boolean { + return provider === 'anthropic' || provider === 'gemini'; +} + +/** + * Can this model turn reasoning **off** at all? + * + * Not a preference — a capability, and the four providers answer it in three different places (see the table + * above). `gemini-2.5-pro` genuinely cannot: Google's documentation says *"N/A: Cannot disable thinking"*, and + * the catalog says the same thing in one field (`budgetTokens.min = 128`). Offering `off` for it would send a + * value the API rejects — which is the entire class of bug this work exists to close. + */ +export function canDisableReasoning(provider: ProviderId, controls: ReasoningControls): boolean { + // A model that publishes NO knob cannot be proven to have an off switch either. Asking here — rather than + // relying on every caller to pre-gate — is what stops the next adapter from sending `thinking: {disabled}` to + // `deepseek-reasoner` on the strength of "well, the PROVIDER can usually disable it". + if (!hasAnyControl(controls)) return false; + switch (provider) { + case 'anthropic': + case 'deepseek': + // An independent `thinking: {type:'disabled'}` switch — available on any model that has a knob at all. + return true; + case 'openai': + // `off` IS an effort value here, so the model must actually accept `'none'`. + return controls.effortValues?.includes(OPENAI_WIRE.off) === true; + case 'gemini': + // Disabling is `thinkingBudget: 0`. A model whose budget floor is above zero cannot express it; a `toggle` + // is an explicit on/off and can. + return controls.toggle === true || controls.budgetTokens?.min === 0; + } +} + +/** + * The tiers a model will ACTUALLY accept — computed from the catalog's per-model control, never copied from it + * ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §6). + * + * `undefined` controls ⇒ the model does not reason ⇒ the empty set. An **empty descriptor** (`{}`) is different + * and real (`deepseek-reasoner`): the model reasons, but exposes **no controllable tier**, so the set is empty + * too — and that is what tells the picker to offer *nothing* rather than to offer *everything*, which is exactly + * today's bug. + * + * The axes UNION — a model may publish both, and several do: + * - **Effort values** → a tier is accepted iff its wire value is one of them. This is where `gpt-5.4-pro` + * rejects `low` and `gpt-5-pro` accepts only `high`. + * - **A token budget**, on a provider whose adapter has a budget field ({@link honorsBudget}) → every non-`off` + * tier is reachable, because the adapter maps the tier onto a point inside `[min, max]`. This is how + * `claude-haiku-4-5` (a budget and no ladder) is controllable at all, and how `claude-opus-4-5` serves `max` + * — a tier its ladder omits — without a 400. + * - **Neither** → no gradation exists; only `off` survives, and only if the model can be disabled at all. + */ +export function acceptedTiers( + provider: ProviderId, + controls: ReasoningControls | undefined, +): ReadonlySet { + const accepted = new Set(); + if (controls === undefined) return accepted; // The model does not reason at all. + + // An EMPTY descriptor (`deepseek-reasoner`) means the model reasons but exposes NO control — not even an off + // switch we can prove exists. The safe answer is the empty set: the field is withheld entirely and the picker + // offers nothing. Adding `off` here on the provider's general ability to disable would be a guess about a + // model whose capability upstream declined to describe, and a guess is what put a rejected value on the wire + // in the first place. + if (!hasAnyControl(controls)) return accepted; + + const gradable = REASONING_EFFORTS.filter( + (tier): tier is Exclude => tier !== 'off', + ); + + // The two axes are a UNION, not an either/or. `claude-opus-4-5` publishes BOTH — `effortValues: [low, medium, + // high]` and `budgetTokens: {min: 1024}` — and its adapter already falls back to the budget for a tier the + // ladder does not carry. An exclusive `else if` here would have hidden `max` from the picker for a model that + // serves it perfectly well: the acceptance set must describe what the ADAPTER can send, not one axis of it. + if (controls.effortValues !== undefined) { + const values = new Set(controls.effortValues); + for (const tier of gradable) { + const wire = wireValueFor(provider, tier, controls); + if (wire !== undefined && values.has(wire)) accepted.add(tier); + } + } + if (controls.budgetTokens !== undefined && honorsBudget(provider)) { + // A budget is continuous — every tier maps onto a point inside [min, max], so all of them are reachable. + // Only for a provider whose adapter HAS a budget field, though; see {@link honorsBudget}. + for (const tier of gradable) accepted.add(tier); + } + + if (canDisableReasoning(provider, controls)) accepted.add('off'); + return accepted; +} + +/** The single normalized tier that represents "reasoning ON" for a budget-shaped model (ADR-0066 amendment). It is + * `medium` — a mid budget, always a member of a budget model's accepted set (the budget axis fills every tier) — + * so mapping "on" to it needs ZERO adapter change and the engine gate accepts it verbatim. */ +export const CANONICAL_ON_TIER: ReasoningEffort = 'medium'; + +/** + * The PRESENTATION shape of a model's reasoning control (ADR-0066 amendment) — how a picker should OFFER it, as + * distinct from the wire-accurate {@link acceptedTiers}: + * - **`'graded'`** — a discrete effort ladder (`effortValues`). A picker shows its rungs, deduped by distinct wire + * value (several normalized tiers can collapse onto one provider value — DeepSeek's low/medium/high all map to + * `high`; Gemini's `max` onto `high`). + * - **`'budget'`** — a continuous token budget and no ladder (`budgetTokens`, e.g. `claude-haiku-4-5`). A budget + * has no meaningful discrete rungs, so a picker offers a simple off/on (Claude-Code parity), "on" = + * {@link CANONICAL_ON_TIER}. + * - **`'none'`** — nothing controllable (`{}` like `deepseek-reasoner`, or a lone toggle with no tier to map "on" + * to). No overlay. + * + * `effortValues` WINS over a co-published budget (`claude-opus-4-5` publishes both): a real ladder is shown as one. + * This is PRESENTATION metadata; {@link acceptedTiers} (the wire truth the gate/failover/adapters read) is unchanged. + */ +export function reasoningControlShape( + controls: ReasoningControls | undefined, +): 'graded' | 'budget' | 'none' { + if (controls === undefined) return 'none'; + if (controls.effortValues !== undefined && controls.effortValues.length > 0) return 'graded'; + if (controls.budgetTokens !== undefined) return 'budget'; + return 'none'; +} + +/** + * The wire value to send for a tier — **only if the model actually publishes it**. `undefined` otherwise. + * + * The adapters used to branch on `controls.effortValues !== undefined` — the PRESENCE of the effort axis — and + * then send the mapped value unchecked. Presence is not membership, and the difference is a 400: + * + * `claude-opus-4-5` publishes ['low','medium','high'] — **no 'max'**. Tier `max` → `output_config.effort: 'max'`. + * `gemini-3-pro-preview` publishes ['low','high'] — **no 'medium'**. Tier `medium` → `thinkingLevel: 'MEDIUM'`. + * + * Both reach the wire through a FAILOVER, where the chain re-points a request at a weaker model. `acceptedTiers` + * already encodes the correct rule; this is the same rule, exposed so an adapter can never re-derive a weaker one. + */ +export function acceptedWireValue( + provider: ProviderId, + tier: Exclude, + controls: ReasoningControls, +): string | undefined { + const wire = wireValueFor(provider, tier, controls); + if (wire === undefined || controls.effortValues === undefined) return undefined; + return controls.effortValues.includes(wire) ? wire : undefined; +} + +/** How much of a model's thinking-budget range each tier spends. `max` means "all of it". */ +const BUDGET_FRACTION: Record, number> = { + low: 0.25, + medium: 0.5, + high: 0.75, + max: 1, +}; + +/** + * The share of a request's output cap that thinking may consume. The rest is the ANSWER's. + * + * Without it, `max` spends **100% of the cap on thoughts** — `budget_tokens: max_tokens - 1` on Anthropic leaves + * exactly one token for the reply, and on Gemini `thinkingBudget == maxOutputTokens` leaves none at all. Both are + * accepted by the provider and both are useless: the user pays for a full turn of reasoning and gets no answer. + * A ceiling that reserves nothing for the output is not a ceiling. + */ +export const THINKING_BUDGET_SHARE = 0.8; + +/** The thinking ceiling for a request whose output cap is `maxTokens` — reserving room for the answer itself. */ +export function thinkingCeiling(maxTokens: number): number { + return Math.floor(maxTokens * THINKING_BUDGET_SHARE); +} + +/** + * Map a normalized tier onto a **token budget** for a budget-shaped model — `claude-haiku-4-5`, `gemini-2.5-pro`, + * and every other model whose tier the effort ladder cannot express. + * + * `ceiling` is the caller's hard upper bound, and it is not optional theatre: Anthropic requires + * `budget_tokens < max_tokens`, so a budget derived from the catalog alone can exceed the request's own output + * cap and be rejected. `claude-haiku-4-5` publishes `{ min: 1024 }` with **no max**, which is precisely the case + * where the range has to come from the request. The adapter passes what it can honour; this stays pure. + * + * **Returns `undefined` when no budget in the range fits under the ceiling** — see the comment on the guard + * below. The caller must then WITHHOLD the field: there is no legal value to send. + */ +/** + * Would the adapter WITHHOLD reasoning for this tier purely because the request's output cap is too small (review + * M6)? True only on the budget-shaped path — an effort-ladder tier carries no budget, and `off` is a switch — and + * only when the model's minimum thinking budget cannot fit under `thinkingCeiling(maxTokens)`. This is the SAME + * predicate the adapters act on (they call {@link reasoningBudgetFor} directly), lifted so the gate can SAY the tier + * was dropped rather than the adapter dropping it in silence. `budget-only` models (`claude-haiku-4-5`) route every + * tier here; a model with a matching effort rung (`claude-opus-4-5` for `high`) does not. + */ +export function reasoningWithheldByCap( + provider: ProviderId, + controls: ReasoningControls, + tier: Exclude, + maxTokens: number, +): boolean { + if (!honorsBudget(provider)) return false; // OpenAI/DeepSeek have no budget field — nothing cap-sensitive + if (acceptedWireValue(provider, tier, controls) !== undefined) return false; // reachable via the effort ladder + if (controls.budgetTokens === undefined) return false; // not budget-shaped → the tier is simply unaccepted, not capped + return reasoningBudgetFor(tier, controls.budgetTokens, thinkingCeiling(maxTokens)) === undefined; +} + +export function reasoningBudgetFor( + tier: Exclude, + range: { readonly min: number; readonly max?: number }, + ceiling: number, +): number | undefined { + const hi = Math.min(range.max ?? ceiling, ceiling); + // THE RANGE DOES NOT EXIST. Anthropic requires `budget_tokens < max_tokens`, so a request whose own output cap + // is at or below the model's MINIMUM thinking budget (haiku's floor is 1024) has no valid budget to send at all. + // + // The first version returned `range.min` here — "the least thinking the model can do" — and that is a 400: with + // `max_tokens: 256` it put `budget_tokens: 1024` on the wire. The honest answer is that reasoning cannot be + // enabled under this cap, so the caller WITHHOLDS the field. The tempting alternative — quietly raising + // `max_tokens` to make room — would change what the user asked for and what they pay, without telling them. + // `hi === range.min` is NOT degenerate: the floor itself is a valid budget, and it still sits strictly below the + // caller's cap (the caller passes `maxTokens - 1` as the ceiling). Only `hi < min` means no budget exists at all. + if (hi < range.min) return undefined; + return Math.round(range.min + (hi - range.min) * BUDGET_FRACTION[tier]); +} diff --git a/packages/shared/src/config.ts b/packages/shared/src/config.ts index a30c5e98..eb92baa8 100644 --- a/packages/shared/src/config.ts +++ b/packages/shared/src/config.ts @@ -1,7 +1,12 @@ import { z } from 'zod'; import { URL_HAS_CREDENTIALS, nonEmptyString, nonNegativeInt, positiveInt } from './common.js'; -import { FS_SCOPE_TIERS, ON_EXCEED_ACTIONS, REASONING_EFFORTS } from './constants.js'; +import { + FS_SCOPE_TIERS, + LLM_PROVIDERS, + ON_EXCEED_ACTIONS, + REASONING_EFFORTS, +} from './constants.js'; /** * Configuration schemas (config-spec.md). Validation only — no file IO. The global @@ -133,9 +138,38 @@ export type McpServerRegistration = z.infer; export const GlobalConfigSchema = z .object({ update_channel: UpdateChannelSchema.optional(), + /** + * The model-metadata catalog ([ADR-0071](../../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §4). + * + * Relavium ships a generated snapshot of models.dev — every price, ceiling and reasoning control, embedded in the + * binary and answering **offline**. That is the whole design, and it is what makes the default here `false`. + */ + catalog: z + .object({ + /** + * Refresh the catalog from models.dev automatically. **DEFAULT `false`, and deliberately.** + * + * A local-first tool that contacts a third party BY DEFAULT violates its own spirit even when the payload is + * innocuous (CLAUDE.md rule 6) — so the standing background egress is opt-in, and the shipped snapshot is a + * complete answer without it. [ADR-0068](../../../docs/decisions/0068-full-screen-tui-renderer-ink7-harness.md) + * established the convention this follows: ship a new risk-bearing surface default-OFF, validate it, then + * decide about flipping. + * + * `relavium models refresh` fetches regardless — an explicit command IS consent. That is what makes + * default-OFF a livable default rather than a dead end: a user who wants current prices types one command. + */ + auto_refresh: z.boolean().optional(), + }) + .strict() + .optional(), preferences: z .object({ default_model: z.string().optional(), + // The provider that serves `default_model`, PERSISTED at pick time (ADR-0059 — the provider is authoritative, + // never re-inferred from the id). Written alongside `default_model` by the `/models` picker + the onboarding + // wizard so a next-session chat over a live-discovered id whose spelling the prefix map cannot place (e.g. + // `chatgpt-4o-latest`) still resolves its provider. Absent ⇒ fall back to prefix/catalog inference. + default_provider: z.enum(LLM_PROVIDERS).optional(), theme: z.string().optional(), // The GLOBAL default reasoning-effort tier (ADR-0066 §6) — the effort counterpart of `default_model`, the // write target of the `/models` picker's effort sub-step. Resolved BELOW project/workspace @@ -183,6 +217,9 @@ export type GlobalConfig = z.infer; export const ChatConfigSchema = z .object({ default_model: z.string().optional(), + // The provider serving `[chat].default_model`, persisted at pick time (ADR-0059). See the identical + // `[preferences].default_provider` note above; the project layer overrides the global one, like `default_model`. + default_provider: z.enum(LLM_PROVIDERS).optional(), fs_scope: FsScopeSchema.optional(), // `!`-shell allowlist (ADR-0061): exact full-command-string match (`allowed_commands`) + opt-in glob patterns // (`allowed_command_globs`, riskier). Both empty/absent ⇒ `!` denied (secure-by-default; the user opts in per @@ -205,6 +242,10 @@ export const ChatConfigSchema = z compact_threshold: z.number().gt(0).lte(1).optional(), max_cost_microcents: nonNegativeInt.optional(), // 0/absent = unbounded; >0 = per-session cap on_exceed: z.enum(ON_EXCEED_ACTIONS).optional(), + // Refuse a turn on a model we cannot PRICE (ADR-0071 §K7). Default false. The chat counterpart of a workflow + // `budget.strict_cost_cap`: an unpriced model is a hole in the cap, and a user who set `max_cost_microcents` + // to bound an untrusted model may prefer "if you can't price it, don't run it" over the silent degrade. + strict_cost_cap: z.boolean().optional(), // The default reasoning-effort tier for a chat whose bound agent authors none (ADR-0066) — off/low/medium/high/ // max. Applied to the built-in default chat agent + surfaced as the picker's starting effort; only sent to a // reasoning-capable model. Absent ⇒ the provider default (no reasoning control sent). diff --git a/packages/shared/src/workflow.ts b/packages/shared/src/workflow.ts index 887fcf47..614e98ec 100644 --- a/packages/shared/src/workflow.ts +++ b/packages/shared/src/workflow.ts @@ -179,6 +179,16 @@ export const BudgetSchema = z // differs from `[chat].max_cost_microcents`, an always-present default where 0 = unbounded.) max_cost_microcents: positiveInt, on_exceed: z.enum(ON_EXCEED_ACTIONS), + /** + * Refuse a turn on a model we cannot PRICE (ADR-0071 §K7). **Default `false`.** + * + * The cost cap is a safety control, and a model with no price is a hole in it: the governor cannot know what a + * turn will cost, so it degrades to `allow` and the cap silently does not apply. For most users that is the + * right trade — a self-hosted model has ~no metered cost, and refusing to run it would be worse than the small + * risk. But a user who set a cap SPECIFICALLY to bound spend on an untrusted model may want the opposite: if + * you cannot price it, do not run it. `true` turns the silent degrade into a hard pre-egress refusal. + */ + strict_cost_cap: z.boolean().optional(), }) .strict(); export type Budget = z.infer; diff --git a/tools/sync-models-dev/sync.mjs b/tools/sync-models-dev/sync.mjs new file mode 100644 index 00000000..7aa656df --- /dev/null +++ b/tools/sync-models-dev/sync.mjs @@ -0,0 +1,292 @@ +#!/usr/bin/env node +/** + * Regenerate the model-catalog snapshot from models.dev + * ([ADR-0071](../../docs/decisions/0071-models-dev-as-the-model-metadata-source.md) §3). + * + * pnpm sync:models # fetch, validate, normalize, write, and diff-check + * pnpm sync:models --check # CI: fail if the committed snapshot is stale (writes nothing) + * + * This tool is DELIBERATELY thin. The Zod boundary and the whole normalization live in + * `packages/llm/src/catalog/` — typed, unit-tested, and part of the package — so the transform that decides + * what a model COSTS is covered by the same test suite as everything else, not by an untested build script. + * All this file does is fetch bytes, call that transform, and write the result. + * + * THE GUARD (§9). A price change on a model we ALREADY SHIP fails the sync. New models merge automatically; + * a *moved* price on a shipped model is a human decision — pricing feeds a safety control (the ADR-0028 cost + * cap), and a silent bot commit that halves a rate would silently halve the cap's protection. Re-run with + * `--accept-price-changes` to take them, deliberately, in a reviewable diff. + */ +import { createHash } from 'node:crypto'; +import { readFileSync, writeFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; + +import { format, resolveConfig } from 'prettier'; + +import { diffCatalog } from '../../packages/llm/dist/catalog/snapshot-guard.js'; +import { + ModelsDevPayloadSchema, + normalizeCatalog, +} from '../../packages/llm/dist/catalog/models-dev-schema.js'; + +const SOURCE_URL = 'https://models.dev/api.json'; +const FETCH_TIMEOUT_MS = 30_000; +const SNAPSHOT = fileURLToPath( + new URL('../../packages/llm/src/catalog/snapshot.ts', import.meta.url), +); + +const argv = new Set(process.argv.slice(2)); +const CHECK_ONLY = argv.has('--check'); +const ACCEPT_PRICE_CHANGES = argv.has('--accept-price-changes'); +const ACCEPT_REMOVALS = argv.has('--accept-removals'); + +/** + * The committed snapshot, as DATA — imported from the built module, never regex-parsed from the source text. + * + * The first version read the generated file with a regex that required a single-quoted key. Prettier's default + * `quoteProps: 'as-needed'` unquotes any key that is a valid JS identifier, so `o1` and `o3` were emitted bare — + * the regex matched 88 of 90 models, those two had NO baseline, and a halved `o1` price passed both money guards + * in silence. The generated file's exact bytes are prettier's decision, not this tool's; any guard that reads + * them as TEXT is one formatting default away from going quietly blind. So we diff the data. + */ +async function committedSnapshot() { + try { + // A LITERAL specifier, deliberately. The repo's seam fence (ADR-0011) forbids a computed `import()` outside + // the adapters — a dynamic specifier is exactly how a provider SDK could be smuggled past `@relavium/llm`. + // The rule is right, and the literal costs nothing. `try` because the very first run has no snapshot yet. + const { CATALOG_SNAPSHOT } = await import('../../packages/llm/dist/catalog/snapshot.js'); + return CATALOG_SNAPSHOT ?? {}; + } catch { + return {}; // First run: no baseline exists, so nothing can have "changed". + } +} + +/** + * A DELIBERATELY locale-independent id comparator. The generated snapshot is byte-compared by CI (`--check`) and + * feeds a SHA, so its row order must be identical on every machine. `String.prototype.localeCompare` is the + * opposite of what that needs — it is locale-sensitive (under `tr_TR`, `I`/`i` collate differently than under + * `en_US`), so a dev and CI would sort the same ids differently and the guard would go red with nothing changed. + * Bare `<`/`>` compare by UTF-16 code unit: deterministic, locale-free, and exact for ASCII model ids. + */ +function byCodeUnit(a, b) { + if (a < b) return -1; + if (a > b) return 1; + return 0; +} + +/** + * The SHA of OUR NORMALIZED CATALOG — deliberately not of the upstream body. + * + * The first version pinned the 3.17 MB upstream payload's hash. But we discard ~97% of it, so ANY byte moving in + * any of the 162 providers we never import changed the hash, changed this file, and made `--check` report the + * snapshot STALE when nothing we ship had moved at all. A weekly guard that is red no matter what is not a + * guard. This hash covers exactly what we ship, so it changes when — and only when — our catalog does. + */ +function catalogSha256(catalog) { + const canonical = Object.keys(catalog) + .sort(byCodeUnit) + .map((id) => `${id}=${JSON.stringify(catalog[id])}`) + .join('\n'); + return createHash('sha256').update(canonical).digest('hex'); +} + +function renderSnapshot(catalog, sha256) { + const ids = Object.keys(catalog).sort(byCodeUnit); + const rows = ids + .map((id) => ` ${JSON.stringify(id)}: ${JSON.stringify(catalog[id])},`) + .join('\n'); + return `// GENERATED FILE — DO NOT EDIT BY HAND. Run \`pnpm sync:models\`. +// +// The model-catalog snapshot (ADR-0071). Source: ${SOURCE_URL} +// Catalog SHA-256: ${sha256} +// Models: ${ids.length} +// +// This SHIPS IN THE BINARY on purpose. The cost cap (ADR-0028) is a safety control, and a safety control that +// only works when a third-party host is reachable is not one — so every model here is priced offline, on first +// run, with no network at all. The optional refresh (ADR-0071 §4) is default-OFF and can only ADD to this floor. +// +// Reviewing a diff here is reviewing a MONEY change. A price that moves on a model we already ship fails the +// sync deliberately (\`--accept-price-changes\` to take it) — because a rate that silently halves also silently +// halves the cap's protection. + +import type { CatalogSnapshot } from './catalog-model.js'; + +export const CATALOG_SNAPSHOT: CatalogSnapshot = { +${rows} +}; + +/** The SHA-256 of this catalog's own data — changes when, and only when, what we ship changes. */ +export const CATALOG_SHA256 = ${JSON.stringify(sha256)}; +`; +} + +/** + * The dropped-model report and the two money guards, factored out of {@link main} so its control flow stays + * readable. Writes the additive/removal notes to stdout and THROWS on the one thing that must be a human decision: + * a shipped model whose price MOVED or that VANISHED (each moves how much the ADR-0028 cost cap protects). Reads + * the module-level `--accept-*` flags. + */ +function enforceMoneyGuards({ dropped, moved, vanished, added }) { + if (dropped.length > 0) { + // Never silent: a dropped model is a model whose spend we cannot cap. Say which, and why. + process.stdout.write( + `sync-models-dev: dropped ${dropped.length} unpriceable/malformed model(s):\n` + + dropped.map((d) => ` ${d.provider}/${d.modelId} — ${d.reason}`).join('\n') + + '\n', + ); + } + + // VANISHED: a model we already ship is GONE from the new catalog, for ANY reason — upstream deleted it, + // stopped pricing it, a provider-key edit erased a whole provider, the deny-list started matching it. The + // previous version could only see models the normalizer explicitly DROPPED; one that simply disappeared from + // the payload appeared in no list at all and was removed in silence. An absent model is an UNPRICED model, and + // an unpriced model skips the cost cap entirely. A DELIBERATE removal is real (a provider retires a model), so + // it is expressible — `--accept-removals` — but never the default: the three ways a model can vanish look + // identical from here, and only one of them is intended. + if (vanished.length > 0 && !ACCEPT_REMOVALS) { + throw new Error( + `sync-models-dev: ${vanished.length} model(s) we already SHIP are GONE from the new catalog:\n` + + vanished.map((id) => ` ${id}`).join('\n') + + '\n\nRemoving a model removes its price, and an unpriced model silently skips the ADR-0028 cost cap for ' + + 'users running it TODAY. Find out WHY first — an upstream retirement, a CATALOG_PROVIDER_KEYS edit, and ' + + 'the non-chat filter all look identical from here, and only one of them is intended. Then re-run with ' + + '--accept-removals to take it in a reviewable diff.', + ); + } + if (vanished.length > 0) { + process.stdout.write( + `sync-models-dev: took ${vanished.length} accepted removal(s): ${vanished.join(', ')}\n`, + ); + } + + // MOVED: every money field, not just the flat pair — cache-read, cache-write, and every context tier. The + // pre-egress estimate takes the HIGHEST applicable tier, so on a long-context turn the TIER rate is the number + // that sizes the cap. Halving only gemini-2.5-pro's >200k tier moves no flat rate and would have tripped + // nothing, while capping every long-context turn against half its true cost. + if (moved.length > 0 && !ACCEPT_PRICE_CHANGES) { + throw new Error( + `sync-models-dev: ${moved.length} ALREADY-SHIPPED model(s) changed price:\n` + + moved.map((m) => ` ${m.modelId}:\n was ${m.before}\n now ${m.after}`).join('\n') + + '\n (fields: input|output|cacheRead|cacheWrite|tiers, in µ¢/Mtok; `-` = no rate)\n\n' + + 'This is a human decision, not a bot commit — a price feeds the ADR-0028 cost cap, so a rate that ' + + 'silently moves also silently moves how much protection the cap gives. Verify against the provider, ' + + 'then re-run with --accept-price-changes to take them in a reviewable diff.', + ); + } + if (added.length > 0) { + // Additive and safe: pricing a model can only ever INCREASE what the cap covers. + process.stdout.write(`sync-models-dev: ${added.length} new model(s): ${added.join(', ')}\n`); + } +} + +/** The upstream body cap, mirroring the runtime refresh's (catalog-refresh.ts): a build tool must not OOM — or + * write a nonsense catalog — on a misbehaving or hostile host. */ +const MAX_BODY_BYTES = 16 * 1024 * 1024; + +/** + * Fetch models.dev and return its raw body, BOUNDED. Every failure mode (transport, status, size) throws a usable + * error rather than a shrug. Extracted from {@link main} so the sync's orchestration stays flat. + */ +async function fetchUpstreamBody() { + let response; + try { + // `redirect: 'error'` — a redirect off models.dev is an ERROR, not a hop (ADR-0071 §8): the destination is a + // compile-time constant, and a 30x that quietly moved it elsewhere would be the one way this fixed-host path + // could turn into an attacker-chosen one. A timeout, because a hung sync in CI is a silent one. + response = await fetch(SOURCE_URL, { + redirect: 'error', + signal: AbortSignal.timeout(FETCH_TIMEOUT_MS), + }); + } catch (error) { + // Node's fetch reports every transport failure as the bare string "fetch failed" and hides the real reason in + // `cause`. Surfacing it is the difference between a usable error and a shrug. + const cause = + error instanceof Error && error.cause instanceof Error ? `: ${error.cause.message}` : ''; + const what = error instanceof Error ? error.message : String(error); + throw new Error(`sync-models-dev: could not fetch ${SOURCE_URL} — ${what}${cause}`); + } + if (!response.ok) { + throw new Error( + `sync-models-dev: ${SOURCE_URL} returned ${response.status} ${response.statusText}`, + ); + } + // Pre-check the DECLARED length, then the actual — a host can lie about the former, so both are checked. + const declared = Number(response.headers.get('content-length') ?? '0'); + if (Number.isFinite(declared) && declared > MAX_BODY_BYTES) { + throw new Error( + `sync-models-dev: ${SOURCE_URL} declares ${declared} bytes, over the ${MAX_BODY_BYTES}-byte cap — refusing`, + ); + } + const body = await response.text(); + if (body.length > MAX_BODY_BYTES) { + throw new Error( + `sync-models-dev: ${SOURCE_URL} returned ${body.length} bytes, over the ${MAX_BODY_BYTES}-byte cap — refusing`, + ); + } + return body; +} + +async function main() { + process.stdout.write(`sync-models-dev: fetching ${SOURCE_URL}\n`); + const body = await fetchUpstreamBody(); + + // The Zod boundary: a third-party payload becomes Relavium types HERE, and its raw shape goes no further. + const payload = ModelsDevPayloadSchema.parse(JSON.parse(body)); + const { catalog, dropped } = normalizeCatalog(payload); + const count = Object.keys(catalog).length; + if (count === 0) { + throw new Error( + 'sync-models-dev: the upstream payload yielded ZERO models. Refusing to write an empty catalog — that ' + + 'would leave every model unpriced and silently disable the cost cap. Check CATALOG_PROVIDER_KEYS.', + ); + } + + // THE TWO MONEY GUARDS — a structural diff of the DATA (see `committedSnapshot`), not a scan of the text. + const { moved, vanished, added } = diffCatalog(await committedSnapshot(), catalog); + enforceMoneyGuards({ dropped, moved, vanished, added }); + + // FORMAT WITH PRETTIER before comparing or writing. The generated file lives in the repo and is subject to + // `format:check` like any other source, so the tool must emit byte-for-byte what prettier would. The first + // version did not — it wrote `JSON.stringify`'s double quotes, prettier rewrote them to single ones, and + // `--check` then reported the snapshot STALE **even when it was current**. A weekly CI guard that is red no + // matter what is not a guard: everyone learns to ignore it, and the price-change protection it exists to give + // quietly evaporates. Formatting here makes the comparison apples-to-apples. + // `resolveConfig` returns `null` when there is no prettier config; spreading `null` is a legal no-op, so no + // `?? {}` fallback is needed (and an empty-object literal would be dead weight). + const rendered = await format(renderSnapshot(catalog, catalogSha256(catalog)), { + ...(await resolveConfig(SNAPSHOT)), + filepath: SNAPSHOT, + }); + const current = (() => { + try { + return readFileSync(SNAPSHOT, 'utf8'); + } catch { + return ''; + } + })(); + + if (CHECK_ONLY) { + if (rendered !== current) { + throw new Error( + 'sync-models-dev: the committed snapshot is STALE (upstream has moved). Run `pnpm sync:models`.', + ); + } + process.stdout.write( + `sync-models-dev: snapshot is current (${count} models, sha ${catalogSha256(catalog).slice(0, 12)})\n`, + ); + return; + } + + writeFileSync(SNAPSHOT, rendered); + const verb = rendered === current ? 'unchanged' : 'UPDATED'; + process.stdout.write( + `sync-models-dev: ${verb} — ${count} models across ${new Set(Object.values(catalog).map((m) => m.provider)).size} providers\n` + + (moved.length > 0 ? `sync-models-dev: took ${moved.length} accepted price change(s)\n` : ''), + ); +} + +try { + await main(); +} catch (error) { + process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`); + process.exitCode = 1; +}