Skip to content

docs(ultragoal): document that shared registries always force the computer suite - #3767

Merged
Yeachan-Heo merged 1 commit into
devfrom
docs/ultragoal-computer-suite-registry-exception
Aug 5, 2026
Merged

docs(ultragoal): document that shared registries always force the computer suite#3767
Yeachan-Heo merged 1 commit into
devfrom
docs/ultragoal-computer-suite-registry-exception

Conversation

@Yeachan-Heo

@Yeachan-Heo Yeachan-Heo commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Docs only. Makes the ultragoal SKILL match what the runtime actually enforces.

Problem

The SKILL currently says:

The mandatory computer-use red-team suite … is conditional, not universal: require it only when computer/desktop control is genuinely part of the product surface being dogfooded.

The runtime is stricter. trustedChangeSetRequiresComputerSuite decides from the computed change set and fails closed:

if (!changeSet?.trusted) return false;
if (changeSet.captureIncomplete) return true;
return changeSet.paths.some(isComputerControlSurfaceChangePath);

isComputerControlSurfaceCategory accepts code | tool | settings-registry, and settings-registry covers three shared behavior registries: config/settings-schema.ts, tools/index.ts, tools/renderers.ts. So any edit to those demands the suite even when the diff contains nothing computer-related. That's deliberate — #3543 added fails closed for any settings-schema edit / fails closed for any tools-index edit and the code comment explains why ("a path-only or uninspectable change cannot prove that computer controls were untouched"). The doc just never caught up.

Why it matters

Following the doc as written, an agent reasons "this change has no computer surface, so the suite doesn't apply", finishes the work, then hits COMPUTER_REDTEAM_CASE_MISSING at checkpoint --status complete with no explanation. The tempting way out is to invent the seven mandatory cases — which is exactly what the gate exists to prevent.

I hit this on an unrelated feature branch whose only trigger was adding two autorouting keys to settings-schema.ts.

Change

Two bullets after the existing "conditional, not universal" line:

  1. The runtime decides from the change set and fails closed — with the concrete trigger paths, the note that registries are unconditional by design, and the captureIncomplete case. Also lists what does not trigger it (generated bindings, prompt/skill/doc files, everything else).
  2. The practical consequence, plus the sanctioned exits: supply a genuine suite, or escalate via gjc ultragoal record-critic-gate-override. Explicitly says not to fabricate cases and not to weaken the gate.

The original "conditional, not universal" guidance is kept — it's still right for choosing an evidence surface; it just isn't what decides gate applicability.

Verification

  • Every path claim checked by calling categorizeComputerChangePath directly rather than transcribing from source:

    path category
    crates/pi-natives/src/computer/executor.rs code
    packages/coding-agent/src/tools/computer.ts tool
    packages/coding-agent/src/tools/computer/session.ts tool
    packages/coding-agent/src/config/settings-schema.ts settings-registry
    packages/coding-agent/src/tools/index.ts settings-registry
    packages/coding-agent/src/tools/renderers.ts settings-registry
    packages/natives/native/index.d.ts generated-binding
    .../skills/ultragoal/SKILL.md prompt-doc-behavior
    docs/tools/computer.md docs-static
    packages/coding-agent/src/task/executor.ts other
  • bun run format-prompts and bun run generate-docs-index produce no further changes to this file, and the docs index is unchanged.

  • gjc-skill-state-hooks: 48 pass / 0 fail.

  • computer-red-team-fixtures: 16 pass / 1 fail — the single failure (requires mandatory QA when invalid untracked bytes make inventory incomplete) is pre-existing on clean dev, verified by stashing this diff and re-running to the identical result. Unrelated to a markdown edit.

Diff is 2 added lines, no runtime change.

…puter suite

The SKILL says the computer-use red-team suite is "conditional, not
universal" and tells the agent to pick the surface that matches what the
change actually ships. The runtime is stricter than that: since #3543 it
decides applicability from the computed change set and fails closed, so
any edit to a shared behavior registry demands the suite even when the
diff contains nothing computer-related.

Following the doc as written leads an agent to conclude the suite is
skippable, then hit COMPUTER_REDTEAM_CASE_MISSING at
`checkpoint --status complete` with no explanation of why -- and the
tempting way out is to invent the seven mandatory cases, which is
exactly what the gate exists to prevent.

Record the real rule instead: the suite is required for computer source,
the computer tool, the three shared behavior registries
(`config/settings-schema.ts`, `tools/index.ts`, `tools/renderers.ts`),
and any incomplete change-set capture; generated bindings, prompt/skill
docs and everything else do not trigger it on their own. Also state the
sanctioned way out -- supply a genuine suite or escalate for an
authorized override -- so the failure mode has a documented exit that is
not fabrication.

Path claims verified against `categorizeComputerChangePath` and
`isComputerControlSurfaceCategory` rather than transcribed by hand.

Docs only; no runtime behavior change.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5e7425f6c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- API/package surfaces require a real artifact file or typed receipt whose artifact `kind` contains one of `api`, `package`, `consumer`, `black-box`, or `test-report`; examples: `api-package-test-report`, `package-consumer-report`, `black-box-api-receipt`. Algorithm/math surfaces require a real artifact file or typed receipt whose artifact `kind` contains one of `property`, `boundary`, `edge`, `adversarial`, `failure`, `math`, `algorithm`, or `test-report`; examples: `property-test-report`, `algorithm-boundary-report`. Bare `inlineEvidence` text alone is not sufficient for any surface.
- The mandatory **computer-use** red-team suite (`kill-switch-bypass`, `suspended-enforcement`, `permission-revoked`, …) is conditional, not universal: require it only when computer/desktop control is genuinely part of the product surface being dogfooded. For every other product type, prove the change through the matching live surface instead — browser-use automation for web/GUI, bash/CLI live invocation or argv replay for CLI, and real artifacts or typed receipts for API/package/algorithm/math. Editing docs, prompts, or skills that merely mention computer-use does not by itself make the computer-use suite applicable; pick the red-team surface that matches what the change actually ships.
- **The runtime decides applicability from the change set, and it fails closed.** Judgement about "what the change actually ships" does not override it, so check the paths before assuming the suite is skippable. `gjc ultragoal checkpoint --status complete` requires the suite whenever the computed change set touches computer source (`crates/pi-natives/src/computer/**`), the computer tool (`packages/coding-agent/src/tools/computer.ts`, `packages/coding-agent/src/tools/computer/**`), or a **shared behavior registry** — `packages/coding-agent/src/config/settings-schema.ts`, `packages/coding-agent/src/tools/index.ts`, `packages/coding-agent/src/tools/renderers.ts`. The registries are deliberately unconditional: they mix computer and non-computer entries, and a path-only or uninspectable change cannot prove computer controls were untouched, so *any* edit to them demands the suite even when the diff contains nothing computer-related. The suite is also required whenever change-set capture was incomplete. Generated bindings (`packages/natives/native/index.{d.ts,js}`), prompt/skill/doc files, and every other path do not trigger it on their own.
- Practical consequence: a change that is not about computer-use at all — say a new settings key in `settings-schema.ts` — will still be gated on the seven mandatory cases. Do **not** fabricate them to get past the gate, and do not weaken the gate. Either supply a genuine suite, or treat it as a blocker and escalate to the operator (`gjc ultragoal record-critic-gate-override` exists for an authorized override).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the unusable computer-suite override path

For a registry-only change that hits this gate, the suggested override cannot clear the blocker: recordUltragoalCriticGateOverride rejects unless the ledger already contains a terminal-critic hard stop, and its event is consulted only by the separate terminal-critic ceiling check; mandatory computer adversarial-case validation still runs unchanged. The command's --help also describes it specifically as authorization for the terminal-critic ceiling. Agents following this instruction will therefore reach another error—or remain blocked on the same seven cases—even after an operator attempts the documented escalation, so this should point to a real computer-suite override mechanism or state that none exists.

AGENTS.md reference: AGENTS.md:L97-L97

Useful? React with 👍 / 👎.

@Yeachan-Heo

Copy link
Copy Markdown
Owner Author

GJC adversarial exact-head review

Verdict: REQUEST_CHANGES

Blocking — documented override cannot perform the stated action. The new text at packages/coding-agent/src/defaults/gjc/skills/ultragoal/SKILL.md:312 directs operators blocked by registry-triggered mandatory computer cases to gjc ultragoal record-critic-gate-override. At this exact head, recordUltragoalCriticGateOverride only succeeds after a durably recorded terminal-critic hard stop (packages/coding-agent/src/gjc-runtime/ultragoal-runtime.ts:4150-4166) and only clears the terminal-critic ceiling checked during completion (:2677-2683). It does not waive mandatory computer red-team cases. The added guidance therefore sends operators to a command that rejects or leaves the actual blocker intact.

The same sentence calls this an authorized operator override, but the command accepts only free-form evidence and session access; it carries no operator/leader authorization context. Either document the real escalation path, or implement and test a dedicated authorized override for the computer-suite applicability gate.

Signature: GJC adversarial review | PR #3767 | exact-head c5e7425f6c8da9fabe3b8ed693ca0d26a7227084 | model layofflabs/codex-auto-review | REQUEST_CHANGES

VERDICT: REQUEST_CHANGES

@Yeachan-Heo

Yeachan-Heo commented Aug 5, 2026

Copy link
Copy Markdown
Owner Author

GPT-heavy hostile current-head re-review

VERDICT: REQUEST_CHANGES

Exact snapshot

  • PR head: c5e7425f6c8da9fabe3b8ed693ca0d26a7227084
  • PR-recorded base / merge-base: ffe2ca4fe9ef02f4321a13035e74ffb31a9144b9
  • Current dev tip: 732856b3ccb3fade6e9fbc17908a4fbca5a7682f
  • Current relation: PR head is 1 commit ahead and 39 commits behind current dev; the relevant Ultragoal runtime/tests did not change across that advance.
  • GitHub state: open, non-draft, mergeable, mergeable_state: clean.
  • Exact-head CI: 18 checks — 9 success, 9 affected-path skips, 0 failure, 0 pending. Runs 30772985667 and 30772985682 both concluded success.
  • CI freshness note, nonblocking: those checks are bound to the exact PR head and its recorded base, not a post-732856b3 current-dev integration run. This is not a second blocker and does not erase the semantic defect below.
  • Contributor ledger: same-repository PR by repository owner Yeachan-Heo; one file, +2/-0.

Blocking bundled-documentation contract error

The new practical-consequence bullet at packages/coding-agent/src/defaults/gjc/skills/ultragoal/SKILL.md:312 tells an operator blocked by registry-triggered mandatory computer cases to use gjc ultragoal record-critic-gate-override. That command cannot perform the documented action:

  • recordUltragoalCriticGateOverride rejects unless the ledger already contains a terminal-critic hard stop (ultragoal-runtime.ts:4149-4166).
  • Its override event is consulted only by the final-aggregate terminal-critic ceiling check (ultragoal-runtime.ts:2675-2685).
  • Mandatory computer adversarial-case validation runs independently and still raises COMPUTER_REDTEAM_CASE_MISSING (ultragoal-runtime.ts:1917-1994).
  • The command help itself describes an override for the terminal-critic ceiling, not computer-suite applicability (ultragoal-runtime.ts:4707-4719).

An operator following the added guidance therefore gets rejected or remains blocked on the same seven cases. Remove the false escape from the shipped skill and PR narrative, and state that no current computer-suite waiver exists. Do not weaken the fail-closed runtime gate to fit the text.

Hostile review found no second blocker: the listed path categories, unconditional shared registries, captureIncomplete behavior, and exclusions otherwise match the exact runtime. Advancing dev did not invalidate or repair this finding.

No local tests, formatters, CI controls, code changes, push, merge, or formal GitHub review-state mutation were performed; this is an ordinary signed exact-head verdict comment.

Signature: GJC GPT-heavy hostile review | PR #3767 | head c5e7425f6c8da9fabe3b8ed693ca0d26a7227084 | target tip 732856b3ccb3fade6e9fbc17908a4fbca5a7682f | 3 independent gpt-5.6-terra lanes + gpt-5.6-sol terminal critic | REQUEST_CHANGES

VERDICT: REQUEST_CHANGES

@Yeachan-Heo

Copy link
Copy Markdown
Owner Author

glm-deepseek independent exact-head re-review

VERDICT: REQUEST_CHANGES

Prior GPT/Qwen lane retired by owner directive; this is a fresh independent verification. No code mutation, push, merge, CI control, or review-state mutation was performed. Read-only.

Exact snapshot

  • PR head: c5e7425f6c8da9fabe3b8ed693ca0d26a7227084
  • PR-recorded base / merge-base: ffe2ca4fe9ef02f4321a13035e74ffb31a9144b9
  • Current dev tip: 732856b3ccb3fade6e9fbc17908a4fbca5a7682f
  • Relation: 1 ahead / 39 behind current dev. Merge-base equals the recorded base, so no rebase drift in the merge window.
  • Runtime file packages/coding-agent/src/gjc-runtime/ultragoal-runtime.ts is byte-identical at PR head and current dev (git diff over the 39-commit advance = 0 lines); the relevant functions did not change across the advance.
  • GitHub state: open, non-draft, MERGEABLE, mergeStateStatus: CLEAN.
  • Exact-head CI (bound to c5e7425, runs 30772985667 + 30772985682, both concluded success): 9 success, 9 affected-path skips, 0 failure, 0 pending.
  • Freshness note (nonblocking): those checks are pinned to the exact head and its recorded base, not a post-732856b3 integration run against current dev. Not a second blocker.
  • Contributor ledger: same-repository PR by repository owner Yeachan-Heo; one file, +2/-0, docs-only.

Blocking — documented override cannot perform the stated action

Independently verified against the exact-head runtime (ultragoal-runtime.ts, byte-identical to current dev):

The new practical-consequence bullet at packages/coding-agent/src/defaults/gjc/skills/ultragoal/SKILL.md:312 tells an operator blocked by registry-triggered mandatory computer cases to "treat it as a blocker and escalate to the operator (gjc ultragoal record-critic-gate-override exists for an authorized override)." That command cannot perform the documented action:

  • recordUltragoalCriticGateOverride rejects unless the ledger already contains a durably recorded terminal-critic hard stop (ultragoal-runtime.ts:4163-4164: "record-critic-gate-override requires a durably recorded terminal critic hard stop").
  • The override event it writes (CRITIC_GATE_OVERRIDE_EVENT, :4166) is consulted by exactly one consumer: the final-aggregate terminal-critic ceiling check (:2675-2685, gated on terminalCriticCeilingReached && !terminalCriticGateOverridden).
  • Mandatory computer adversarial-case validation runs independently of that ceiling. validateMandatoryComputerAdversarialCases (:1917-1994) is reached via validateExecutorQaRedTeamEvidenceInternalrequiresComputerRedTeamSuite (:2108-2109) and raises COMPUTER_REDTEAM_CASE_MISSING (:1932) on the first missing mandatory case. Neither terminalCriticGateOverridden nor CRITIC_GATE_OVERRIDE_EVENT appears anywhere in the computer-suite path.
  • The command help itself scopes the override to the terminal-critic ceiling, not computer-suite applicability (:4715: "Human/leader authorization evidence for the terminal-critic ceiling override").
  • Exhaustive scan for any computer-suite waiver/exception/skip mechanism (env overrides, waive/skip/exempt flags, a second override event) found none. requiresComputerRedTeamSuite has no bypass branch beyond trustedChangeSetRequiresComputerSuite and the declared-paths check.

An operator following the added guidance is therefore rejected (no terminal-critic hard stop recorded) or, in the unlikely case one exists, remains blocked on the same seven cases. The shipped skill and the PR narrative both present a non-functional escape.

Remediation: remove the false escape from both the shipped skill and the PR description. State that no current computer-suite applicability waiver exists. Do not weaken the fail-closed runtime gate (trustedChangeSetRequiresComputerSuite, requiresComputerRedTeamSuite) to fit the text — the gate is correct; only the documentation is wrong. A real waiver for registry-triggered cases would be a separate code change with its own evidence and tests, not a docs PR.

Non-blocking verification (passed)

The remainder of the added text was checked independently against categorizeComputerChangePath (:1580-1603) and isComputerControlSurfaceCategory (:1605-1611) and is accurate:

  • Trigger paths and categories match: crates/pi-natives/src/computer/**code; tools/computer.ts / tools/computer/**tool; config/settings-schema.ts, tools/index.ts, tools/renderers.tssettings-registry; all three categories are accepted by isComputerControlSurfaceCategory.
  • captureIncomplete forces the suite (:1621).
  • Exclusions are correct: packages/natives/native/index.{d.ts,js}generated-binding (not a triggering category); SKILL.mdprompt-doc-behavior; docs/tools/computer.mddocs-static; everything else → other — none accepted by isComputerControlSurfaceCategory.
  • "Seven mandatory cases" matches MANDATORY_COMPUTER_CASE_IDS (:1561-1569): kill-switch-bypass, suspended-enforcement, permission-revoked, display-stale, out-of-bounds-drift, runaway-loop-halt, blast-radius.
  • Registries-are-deliberately-unconditional framing matches the source comment at :1606-1607.

No second blocker found.

Signature: GJC glm-deepseek independent review | PR #3767 | exact head c5e7425f6c8da9fabe3b8ed693ca0d26a7227084 | target tip 732856b3ccb3fade6e9fbc17908a4fbca5a7682f | REQUEST_CHANGES

@Yeachan-Heo
Yeachan-Heo merged commit 47099f0 into dev Aug 5, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant