docs(ultragoal): document that shared registries always force the computer suite - #3767
Conversation
…puter suite The SKILL says the computer-use red-team suite is "conditional, not universal" and tells the agent to pick the surface that matches what the change actually ships. The runtime is stricter than that: since #3543 it decides applicability from the computed change set and fails closed, so any edit to a shared behavior registry demands the suite even when the diff contains nothing computer-related. Following the doc as written leads an agent to conclude the suite is skippable, then hit COMPUTER_REDTEAM_CASE_MISSING at `checkpoint --status complete` with no explanation of why -- and the tempting way out is to invent the seven mandatory cases, which is exactly what the gate exists to prevent. Record the real rule instead: the suite is required for computer source, the computer tool, the three shared behavior registries (`config/settings-schema.ts`, `tools/index.ts`, `tools/renderers.ts`), and any incomplete change-set capture; generated bindings, prompt/skill docs and everything else do not trigger it on their own. Also state the sanctioned way out -- supply a genuine suite or escalate for an authorized override -- so the failure mode has a documented exit that is not fabrication. Path claims verified against `categorizeComputerChangePath` and `isComputerControlSurfaceCategory` rather than transcribed by hand. Docs only; no runtime behavior change.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5e7425f6c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - API/package surfaces require a real artifact file or typed receipt whose artifact `kind` contains one of `api`, `package`, `consumer`, `black-box`, or `test-report`; examples: `api-package-test-report`, `package-consumer-report`, `black-box-api-receipt`. Algorithm/math surfaces require a real artifact file or typed receipt whose artifact `kind` contains one of `property`, `boundary`, `edge`, `adversarial`, `failure`, `math`, `algorithm`, or `test-report`; examples: `property-test-report`, `algorithm-boundary-report`. Bare `inlineEvidence` text alone is not sufficient for any surface. | ||
| - The mandatory **computer-use** red-team suite (`kill-switch-bypass`, `suspended-enforcement`, `permission-revoked`, …) is conditional, not universal: require it only when computer/desktop control is genuinely part of the product surface being dogfooded. For every other product type, prove the change through the matching live surface instead — browser-use automation for web/GUI, bash/CLI live invocation or argv replay for CLI, and real artifacts or typed receipts for API/package/algorithm/math. Editing docs, prompts, or skills that merely mention computer-use does not by itself make the computer-use suite applicable; pick the red-team surface that matches what the change actually ships. | ||
| - **The runtime decides applicability from the change set, and it fails closed.** Judgement about "what the change actually ships" does not override it, so check the paths before assuming the suite is skippable. `gjc ultragoal checkpoint --status complete` requires the suite whenever the computed change set touches computer source (`crates/pi-natives/src/computer/**`), the computer tool (`packages/coding-agent/src/tools/computer.ts`, `packages/coding-agent/src/tools/computer/**`), or a **shared behavior registry** — `packages/coding-agent/src/config/settings-schema.ts`, `packages/coding-agent/src/tools/index.ts`, `packages/coding-agent/src/tools/renderers.ts`. The registries are deliberately unconditional: they mix computer and non-computer entries, and a path-only or uninspectable change cannot prove computer controls were untouched, so *any* edit to them demands the suite even when the diff contains nothing computer-related. The suite is also required whenever change-set capture was incomplete. Generated bindings (`packages/natives/native/index.{d.ts,js}`), prompt/skill/doc files, and every other path do not trigger it on their own. | ||
| - Practical consequence: a change that is not about computer-use at all — say a new settings key in `settings-schema.ts` — will still be gated on the seven mandatory cases. Do **not** fabricate them to get past the gate, and do not weaken the gate. Either supply a genuine suite, or treat it as a blocker and escalate to the operator (`gjc ultragoal record-critic-gate-override` exists for an authorized override). |
There was a problem hiding this comment.
Remove the unusable computer-suite override path
For a registry-only change that hits this gate, the suggested override cannot clear the blocker: recordUltragoalCriticGateOverride rejects unless the ledger already contains a terminal-critic hard stop, and its event is consulted only by the separate terminal-critic ceiling check; mandatory computer adversarial-case validation still runs unchanged. The command's --help also describes it specifically as authorization for the terminal-critic ceiling. Agents following this instruction will therefore reach another error—or remain blocked on the same seven cases—even after an operator attempts the documented escalation, so this should point to a real computer-suite override mechanism or state that none exists.
AGENTS.md reference: AGENTS.md:L97-L97
Useful? React with 👍 / 👎.
GJC adversarial exact-head reviewVerdict: REQUEST_CHANGES Blocking — documented override cannot perform the stated action. The new text at The same sentence calls this an authorized operator override, but the command accepts only free-form evidence and session access; it carries no operator/leader authorization context. Either document the real escalation path, or implement and test a dedicated authorized override for the computer-suite applicability gate. Signature: GJC adversarial review | PR #3767 | exact-head VERDICT: REQUEST_CHANGES |
GPT-heavy hostile current-head re-reviewVERDICT: REQUEST_CHANGES Exact snapshot
Blocking bundled-documentation contract errorThe new practical-consequence bullet at
An operator following the added guidance therefore gets rejected or remains blocked on the same seven cases. Remove the false escape from the shipped skill and PR narrative, and state that no current computer-suite waiver exists. Do not weaken the fail-closed runtime gate to fit the text. Hostile review found no second blocker: the listed path categories, unconditional shared registries, No local tests, formatters, CI controls, code changes, push, merge, or formal GitHub review-state mutation were performed; this is an ordinary signed exact-head verdict comment. Signature: GJC GPT-heavy hostile review | PR #3767 | head VERDICT: REQUEST_CHANGES |
glm-deepseek independent exact-head re-reviewVERDICT: REQUEST_CHANGES Prior GPT/Qwen lane retired by owner directive; this is a fresh independent verification. No code mutation, push, merge, CI control, or review-state mutation was performed. Read-only. Exact snapshot
Blocking — documented override cannot perform the stated actionIndependently verified against the exact-head runtime ( The new practical-consequence bullet at
An operator following the added guidance is therefore rejected (no terminal-critic hard stop recorded) or, in the unlikely case one exists, remains blocked on the same seven cases. The shipped skill and the PR narrative both present a non-functional escape. Remediation: remove the false escape from both the shipped skill and the PR description. State that no current computer-suite applicability waiver exists. Do not weaken the fail-closed runtime gate ( Non-blocking verification (passed)The remainder of the added text was checked independently against
No second blocker found. Signature: GJC glm-deepseek independent review | PR #3767 | exact head |
Docs only. Makes the ultragoal SKILL match what the runtime actually enforces.
Problem
The SKILL currently says:
The runtime is stricter.
trustedChangeSetRequiresComputerSuitedecides from the computed change set and fails closed:isComputerControlSurfaceCategoryacceptscode | tool | settings-registry, andsettings-registrycovers three shared behavior registries:config/settings-schema.ts,tools/index.ts,tools/renderers.ts. So any edit to those demands the suite even when the diff contains nothing computer-related. That's deliberate — #3543 addedfails closed for any settings-schema edit/fails closed for any tools-index editand the code comment explains why ("a path-only or uninspectable change cannot prove that computer controls were untouched"). The doc just never caught up.Why it matters
Following the doc as written, an agent reasons "this change has no computer surface, so the suite doesn't apply", finishes the work, then hits
COMPUTER_REDTEAM_CASE_MISSINGatcheckpoint --status completewith no explanation. The tempting way out is to invent the seven mandatory cases — which is exactly what the gate exists to prevent.I hit this on an unrelated feature branch whose only trigger was adding two autorouting keys to
settings-schema.ts.Change
Two bullets after the existing "conditional, not universal" line:
captureIncompletecase. Also lists what does not trigger it (generated bindings, prompt/skill/doc files, everything else).gjc ultragoal record-critic-gate-override. Explicitly says not to fabricate cases and not to weaken the gate.The original "conditional, not universal" guidance is kept — it's still right for choosing an evidence surface; it just isn't what decides gate applicability.
Verification
Every path claim checked by calling
categorizeComputerChangePathdirectly rather than transcribing from source:crates/pi-natives/src/computer/executor.rscodepackages/coding-agent/src/tools/computer.tstoolpackages/coding-agent/src/tools/computer/session.tstoolpackages/coding-agent/src/config/settings-schema.tssettings-registrypackages/coding-agent/src/tools/index.tssettings-registrypackages/coding-agent/src/tools/renderers.tssettings-registrypackages/natives/native/index.d.tsgenerated-binding.../skills/ultragoal/SKILL.mdprompt-doc-behaviordocs/tools/computer.mddocs-staticpackages/coding-agent/src/task/executor.tsotherbun run format-promptsandbun run generate-docs-indexproduce no further changes to this file, and the docs index is unchanged.gjc-skill-state-hooks: 48 pass / 0 fail.computer-red-team-fixtures: 16 pass / 1 fail — the single failure (requires mandatory QA when invalid untracked bytes make inventory incomplete) is pre-existing on cleandev, verified by stashing this diff and re-running to the identical result. Unrelated to a markdown edit.Diff is 2 added lines, no runtime change.