[CI validation] Dynamic model catalog - #2
Draft
kargnas wants to merge 45 commits into
Draft
Conversation
모델별 effort, service tier, tool capability를 표현하고 지원하지 않는 선택을 실행 전에 거부하도록 만들었습니다. Constraint: 정적 모델 ID를 계약에서 제거했습니다. Confidence: high Scope-risk: broad Tested: pnpm test
인스턴스 설명을 요청할 때마다 CLI 카탈로그를 다시 읽고 조회 실패를 엔진 상태와 분리해 노출했습니다. Rejected: 마지막 성공 카탈로그 유지 | Refresh가 실제 상태와 달라질 수 있었습니다. Confidence: high Scope-risk: moderate Tested: pnpm test
ACP initialize 메타데이터와 Kimi provider JSON에서 모델과 effort를 읽고 선택값을 각 CLI 실행에 전달했습니다. Constraint: ACP provider마다 모델 메타데이터 표면이 달랐습니다. Rejected: 정적 카탈로그 fallback | CLI 업데이트를 반영하지 못했습니다. Confidence: high Scope-risk: broad Tested: pnpm test
Antigravity의 JSON 모델 목록을 읽고 선택한 모델과 effort를 실행 인자에 전달했습니다. Rejected: 정적 모델 목록 | CLI catalog와 계속 어긋났습니다. Confidence: high Scope-risk: moderate Tested: pnpm test
Claude CLI가 광고하는 alias와 로컬 설정의 model, effort를 카탈로그와 실행 인자에 반영했습니다. Constraint: Claude CLI에는 별도 model list 명령이 없었습니다. Confidence: medium Scope-risk: moderate Tested: pnpm test
Codex app-server에서 현재 모델, effort, service tier를 읽고 새 turn과 resume에 그대로 전달했습니다. Rejected: 세 모델 정적 목록 | 최신 CLI catalog와 일치하지 않았습니다. Confidence: high Scope-risk: broad Tested: pnpm test
xAI 계정의 models endpoint를 카탈로그로 사용하고 모델이 없는 실행의 자동 대체를 제거했습니다. Confidence: high Scope-risk: narrow Tested: pnpm test
Box 전송에는 카탈로그가 제공한 provider와 model을 명시적으로 요구하도록 바꿨습니다. Rejected: 모델 ID prefix 추론 | 새 이름과 alias에서 잘못된 provider를 선택했습니다. Confidence: high Scope-risk: moderate Tested: pnpm test
봇 저장, 일반 turn, 그룹 turn에서 현재 카탈로그를 검증하고 effort와 처리 등급을 driver까지 전달했습니다. Directive: 사라진 모델을 다른 모델로 자동 대체하지 않습니다. Confidence: high Scope-risk: broad Tested: pnpm test
서버 경계 검증을 통과하도록 통신 E2E fixture를 fake ACP의 실제 광고 모델과 effort로 맞췄습니다.
E2E가 정적 모델 순회와 Box 자체 카탈로그 대신 provider가 보고한 기본 선택을 사용하도록 바꿨습니다. Rejected: 실패 시 다음 모델 자동 시도 | 잘못된 선택을 숨겼습니다. Confidence: high Scope-risk: moderate Tested: node --check scripts/e2e-server.mjs
프런트 상태와 Refresh 오류 흐름이 동적 effort, service tier, capability 메타데이터를 보존하도록 바꿨습니다. Confidence: high Scope-risk: moderate Tested: pnpm build
Model Picker가 새 카탈로그를 로딩한 뒤 모델별 effort와 Codex Fast 같은 처리 등급을 선택하도록 만들었습니다. Confidence: high Scope-risk: moderate Tested: pnpm build
NoEngines의 재조회가 공유 상태에 오류를 표시한 뒤 unhandled rejection을 남기지 않도록 처리했습니다.
README에 CLI 기반 모델 조회와 effort, 처리 등급, tool capability 동작을 반영했습니다.
Codex가 같은 속도 모드를 priority와 fast로 함께 보고해 선택 메뉴에 Fast가 중복 표시됐습니다. 현재 설정이 사용하는 fast ID를 남기고 priority 별칭을 제거했습니다. Confidence: high Scope-risk: narrow Tested: pnpm vitest run server/drivers/codex.test.ts
선택창을 열 때마다 CLI 카탈로그를 다시 읽던 동작을 제거했습니다. 캐시된 목록을 즉시 유지하고 5분 주기 및 수동 버튼으로 백그라운드 갱신하도록 변경했습니다. Confidence: high Scope-risk: narrow Tested: pnpm typecheck Tested: packaged macOS UI refresh and spinner
동적 catalog 검증이 활성화된 뒤에도 통신 시나리오가 실제 ACP fixture 모델을 선택하도록 갱신했습니다. Confidence: high Scope-risk: narrow Tested: pnpm exec vitest run server/comms.test.ts --reporter=dot
Droid의 정적 모델 목록을 CLI help catalog와 로컬 Factory 설정 조합으로 교체했습니다. 선택한 reasoning effort도 ACP 세션 설정으로 전달했습니다. Constraint: Droid ACP initialize는 모델 catalog를 노출하지 않습니다. Rejected: 정적 내장 목록 유지 | CLI 업데이트와 계정별 설정을 반영할 수 없었습니다. Confidence: high Scope-risk: narrow Tested: Droid 0.197.0 catalog smoke, pnpm exec vitest run server/drivers/acp/acp.test.ts --reporter=dot, pnpm typecheck
모델 선택 요청에 기본 timeout을 적용하고 Droid가 로컬 기본 모델을 사용할 때 불필요한 CLI 실행을 피했습니다. Catalog probe의 UTF-8 스트림 처리도 함께 보완했습니다. Confidence: high Scope-risk: moderate Tested: pnpm exec vitest run server/drivers/acp/acp.test.ts
빈 모델 ID가 Claude CLI 내부 기본값을 뜻하는 경우 provider metadata를 제거해 클라우드 실행 대상으로 오인되지 않도록 했습니다.
model/list의 모든 cursor 페이지를 합치고 숨김 모델이 기본값으로 보고되어도 첫 표시 모델을 사용하도록 했습니다. Catalog JSON 스트림에는 UTF-8 decoding을 명시했습니다. Constraint: Codex model/list는 cursor 기반으로 여러 페이지를 반환할 수 있음 Confidence: high Scope-risk: moderate Tested: pnpm exec vitest run server/drivers/codex.test.ts
저장값이 없을 때는 모델 기본 처리 등급을 표시하고 명시적인 null은 Standard로 유지하도록 했습니다.
잘못된 모델 선택 요청이 실패 응답만 반환하고 기존 bot 선택값은 보존하는지 각 거부 경로에서 확인했습니다.
모델 목록이 설치된 CLI뿐 아니라 지원되는 provider API에서도 올 수 있음을 현재 동작에 맞춰 설명했습니다.
model/list case의 지역 변수가 다른 switch 절에서 보이지 않도록 case 본문을 block scope로 감쌌습니다.
* Stop handing clients the provider session cursors `resumeCursors` is the harness's own bookkeeping — the native session id to resume, per instance, per task. It goes out on every bot payload and every `bot` SSE frame, and no client has ever read it. It is harmless noise only for as long as every client is this machine. It is still worth not sending: it is internal provider state on the wire, it makes `GET /api/bots` bigger for no reason, and anything that ever consumes this API from somewhere else inherits it by default. Stripped at one chokepoint rather than at each call site, because there are nine of them and a new broadcast should not have to remember. The test asserts on the SSE bytes as well as the HTTP bodies. That is deliberate: this was found by capturing real wire output rather than by reading the code, and the wire is where it has to stay fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Strip the cursors from the standalone task responses too The bot payload went through wireBot, but POST /tasks and PATCH on a task also answer with the task record on its own, straight from the store. A renamed task carries live resume cursors, so that response leaked exactly what the rest of this change stopped sending. wireTask is now the one place a task is trimmed: wireBot's nested tasks, both botWithThread lists, and the two standalone responses. The test asserts on task.body.task and on the rename response, and deletes the bot from an outer finally so a failed assertion earlier in the test doesn't leave it in the store. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* A webhook turn does not inherit auto mode Auto mode is something a person switches on for turns they are present for. A webhook turn starts with nobody at the keyboard, on a payload somebody else wrote — so inheriting that switch silently converts "I trust this bot while I'm watching" into "I trust this bot at 3am on whatever GitHub posts". Everything else about webhook triggers defends against a FORGED request, and that part is solid: hashed secret, constant-time compare, untrusted data framing, loopback by default, rate limit. This is about an authentic one. The guard standing behind auto mode is a regex list its own comment calls not a security boundary, and it must not stand in for a human. So the rule lives with the other policy in auto-approve.ts rather than as a condition at the call site: autoDecision refuses when the turn is unattended, before any allow-list is consulted, so an "always allow" grant can't widen into it either. The approval still appears in the chat and can be answered if someone is around. The mark goes on the DETACHED task's thread, since a webhook runs in its own task — marking the bot's active thread would gate the wrong conversation — and is cleared when the turn settles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Carry the unattended gate across peer-comms hops Review caught that the gate held on the bot that read the webhook payload and released on the bot that acted on it. A webhook turn starts at depth 0, so the agents tools ARE mounted and the bot can hand work to a teammate — whose turn then ran with full auto mode and every always-allow grant, nobody at the keyboard. One hop was all it took, and the depth cap does not help. The mark is now keyed by BOT rather than thread. A bot runs one turn at a time so the identity is exact, and the comms paths know who is asking but not always from which thread — ask_bot had no source thread at all. Both ask_bot and delegate_bot now pass the caller's state to the turn they start. It expires by time instead of being cleared on turn.completed. Bus subscribers fire in registration order and the delegation drain runs AFTER the main fold, so clearing there blanked the flag before the hop that needed to read it — the obvious fix, and wrong. A stale mark only ever means "ask a human", so the failure direction is safe, and the TTL stops the map growing without bound. Two wiring tests, because the existing ones exercise the rule and would all still pass if the mark were never set or never read. Both were confirmed to FAIL with the wiring removed: one deletes the gate in the fold, the other breaks only the hop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pin the ask_bot hop too Review mutated the other comms path — index.ts:124, the ask_bot one — and both existing tests stayed green. The propagation was written correctly, but nothing held it there, so a refactor could have silently reopened the hole this branch exists to close, with the suite passing. It is also the likelier path in practice: a webhook-triggered bot pulling a teammate in for an answer mid-turn is more ordinary than handing the work off asynchronously. Third test drives FAKE_ACP_MODE=ask-peer. The fake asks whichever peer list_bots returns first, so the other bots are hidden to make the target deterministic. Verified to fail with that one line mutated: "the asked teammate auto-approved — ask_bot did not carry the gate". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep unattended approvals guarded --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(effort): declare per-driver reasoning-effort levels Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * feat(effort): pass the level to Claude Code as --effort Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code * feat(effort): send the level on Codex turn/start Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * feat(effort): pass the level to Grok as --reasoning-effort * feat(effort): persist the level per bot and pass it into the turn PATCH /api/bots/:id now validates modelSelection.effort against the target instance's declared capabilities.effortLevels before it's copied through the allowlist, and startTurn hands bot.modelSelection.effort to sendTurn (cleared for cloud routines, which already borrow the instance's default model). Neither Codex nor Grok rejects an unknown level at their own protocol boundary, so this PATCH-time check is the only real gate — it's what keeps "none" (reserved for future engines) from ever reaching Claude or Codex, whose declared lists exclude it. An unavailable target instance offers an empty allowed list, so any effort is rejected: an engine that isn't there cannot promise to honour a level. The happy-path round-trip lives in server/store.test.ts instead of the HTTP suite: server/index.test.ts's fixture pins the harness to a single unknown-driver shadow instance (deliberately, for CI determinism), so no bot in that suite ever has a live target to validate an accepted level against. The API suite therefore covers the reject and no-op branches only; the accept branch is covered at the store layer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): forward effortLevels through the instance registry descriptor Task 1 added capabilities.effortLevels to ProviderAdapter, but ProviderRegistry.describe() — the method behind GET /api/instances, which is what the client actually reads — builds its own capabilities object from just computerMcp and agentsMcp and dropped the new field. Every engine's effort levels therefore came back undefined over HTTP, so Task 6's control (gated on capabilities.effortLevels.length) could never render for any bot regardless of what the client did. Forward the field the same way its two neighbors already are, with registry-level coverage via a new FakeDriverOptions.effortLevels knob. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * feat(effort): add the per-bot effort control to settings Widens the client InstanceInfo.capabilities and ModelSelection types to carry effortLevels/effort, and adds effort levels to the modelSelection patch allowlist (both the local SettingsPanel patch() helper and the updateBot action) so the new segmented-button row — copied from the Computer block's pattern — can clear the field back to the engine default by sending no effort key at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): gate the level only when the target engine resolves PATCH /api/bots/:id is the app's general-purpose bot endpoint, and its callers send merged multi-field bodies: duplicateBot re-sends the source bot's whole modelSelection beside name, title and description, and updateBot debounces unrelated edits into one request. Rejecting an effort the registry could not verify therefore failed the entire request — duplicating a bot whose engine was offline lost its name, title and description with it. The gate now fires only when registry.get() actually resolves the target. An instance that isn't there promises nothing either way, and startTurn already refuses to run a turn on an unavailable instance, so an unverified level never reaches a CLI. A genuinely bad level against a live engine is still a 400. Also drops the `as EffortLevel` cast: this is the boundary that decides whether the string is a level, so it must not assert that it already is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * test(effort): replace the vacuous gate assertion with real coverage The API suite's only effort test asserted that "banana" was rejected, against a fixture whose sole instance is an unknown-driver shadow. Every level was rejected there, so the assertion passed whether the gate worked or rejected unconditionally — it could not fail. What this fixture can honestly prove is the pass-through and the store's replace semantics: a level round-trips through PATCH and GET, and re-sending the selection with the effort key dropped clears it, which is exactly the shape the panel's "Default" button sends. A comment records which branch of the gate this does not reach, so the next reader does not mistake it for full coverage; the comparison against a live engine's declared list needs a resolvable instance, which this fixture deliberately does not have. Also retitles the store's persistence test, which said "defaulting to none" while asserting undefined — the exact confusion between the explicit "none" level and no override that the level type warns about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): keep the level when the model changes within one engine pick() dispatched a bare { instanceId, model }, and both the reducer and the server's patchBot replace modelSelection wholesale, so switching Sonnet to Opus silently reset a configured xhigh back to the engine default with nothing said. The effort row just changed under the user. The selection now carries the effort across when the instance is unchanged, and drops it when the instance changes: effort vocabularies are declared per driver, so a level that survived an engine switch could easily be one the new engine never offered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): stop the panel promising a reversal Codex cannot make The comment on codex.ts's turn/start claimed null would clear a level. That was an assumption, and it is wrong. Probed against the real codex-cli 0.146.0 app-server: with the config default at "low" and the thread overridden to "high", sending effort: null emitted no thread/settings/updated and thread/resume still read back "high" — byte-identical to omitting the key. Setting a level on that same path does commit, so the null legs are genuinely inert, not merely unapplied. There is no clearing mechanism at all: "" is rejected with "reasoning_effort must not be empty", and thread/start carries no effort field. So the omit stays, and the copy changes instead. "(currently: engine default)" asserted a state we cannot deliver on a Codex thread that had already been sent a level; "(Default: no level is sent)" describes what the app actually does, which is true on all three drivers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): label X-High correctly and mark the pressed segment The row leans on `capitalize`, which renders "xhigh" as "Xhigh". Only that one level needs spelling out; the rest capitalize cleanly. The segmented buttons also carried no pressed state for assistive technology — selection was conveyed by background colour alone, while the panel's other controls already set role="switch"/aria-checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * fix(effort): type the client's effort fields with EffortLevel The app declared `effort?: string` and `effortLevels?: readonly string[]`, so a typo in a dispatch reached the server and came back as a 400 rather than failing to compile. Importing a type across the boundary is already how `src/lib/notify.ts` consumes `server/notify.ts`, so this follows the existing seam rather than opening a new one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A52XvU63rgJypGD8U9cmNP * Validate reasoning effort at runtime --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: milind-soni <milindsoni201@gmail.com>
* Run a new webhook first event * Present webhook command as a real task
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Suspend idle Local VMs safely * Harden CUA computer integrations * Skip POSIX shell validation on Windows * Refresh server build for CUA release
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the legacy Connect-key flow with Composio project keys and reusable Sessions, with OS-backed credential storage for packaged apps.
Mirror successful replies and all failure terminal states for delegated turns into the bot-to-bot channel. Includes empty-reply, dispatch-failure, and provider-reload cleanup coverage.
main이 milind-soni#144(정적 effortLevels)와 milind-soni#164(위임 터미널 상태 미러링)을 받으면서 충돌이 발생해 병합을 해결했다. - contracts/index/store/ModelPicker의 effort·serviceTier는 PR의 동적 catalog 계약을 유지하고 main의 정적 EffortLevel 시스템은 제거했다. catalog가 모델별 efforts/serviceTiers를 제공하므로 정적 목록은 중복이다. - PATCH 검증은 catalog 조회 실패 시(엔진 오프라인) 통과시키고 startTurn이 이미지 못 미친 선택을 거부하도록 했다. - comms/unattended e2e는 실제 catalog 모델명을 쓰도록 바꿨고, initialize 후 크래시(crash-on-prompt)와 catalog 실패(helperNoCatalog) fixture를 추가해 main의 크래시/시작불가 미러 테스트를 유지했다. Tested: pnpm typecheck, pnpm vitest run (62 files, 490 passed, 8 skipped) Confidence: high Scope-risk: moderate Reversibility: moderate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporary validation PR for upstream milind-soni#143. It uses the current upstream main as its base so the same GitHub Actions matrix can run without the upstream first-contributor approval gate.