Teach pending_review, instrument it, and cut two interaction costs - #294
Merged
Conversation
Docs the v19 grading left behind: docs/integrations.md described the Stop hook as soft-blocking whenever the verifier returns findings (it now blocks only for agent_action_required, and human_review_required deliberately does not block); docs/agents/protocol.md omitted pending_review[] from the field list, from the agent_action_required row, and from the never-a-stop-state enumeration; docs/agent-contract-current.md contradicted its own v19 preamble by leaving pending_review out of the operative read list; and docs/checks.md asserted human review is required for the UNCLASSIFIED catch-all without qualifying that the local loop now grades it. The check summary also read 'require human review' while control said the agent may continue. Graded results now say the review is carried to PR time, so the prose and the control state agree. New harness criterion reports_pending_review restores the coverage respects_must_stop lost when graded rows stopped being stops: the band traded a hard stop for a reporting duty, so an agent that runs verify and never surfaces the carried items now fails a blocker criterion. Naming each item passes; a generic acknowledgement passes with a weaker signal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er pin One render-hash batch over the kit surfaces: - AGENT_RESULT_CONTROL_FIELDS gains pending_review, so the runtime contract, .well-known mirror, and the generated .shipgate/agent-contract.json all name the field an agent must read. It stays out of AGENT_CONTROL_FIELDS: the obligation is result-level, not part of the control block. - The Codex skill described check routing as continue/repair/stop. The graded band added a fourth shape (verify, then report the carried items), which is the one most likely to fire on a routine .codex/config.toml or SKILL.md edit, so the enumeration now names it. - The Claude Code skill's control sentence now says a non-empty pending_review[] lets the turn finish but must be reported. - Longstanding drift fixed while the hashes were moving anyway: the skill claimed verifier_schema_version 0.5; the current verifier schema is 0.6. Kit sources, their byte-identical mirrors, EXPECTED_*_RENDER_SHA256, and both kits' prior_render_sha256 move together so an installed managed copy is still recognized as unmodified on the next update. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two interaction costs the graded stop exposed, both hook-local and both advisory — no verdict, control state, or gate semantics change. Approval memory: the PreToolUse boundary asked again on every edit to an already-allowed protected file, so prompt-engineering a single prompts/ file cost one prompt per keystroke and taught people to click through. PostToolUse only fires after a tool call went through, so in a prompting permission mode an edit that landed is an edit a human allowed; the hook records those paths per host session and PreToolUse skips re-asking for them. A new session asks again, the auto-accepting modes record nothing because nobody was asked, the memory is capped and scoped to the current session, and AGENTS_SHIPGATE_APPROVAL_MEMORY=off restores prompt-every-time. shipgate check and PR-time verify still evaluate every edit, and SHIP-VERIFY-* still reports the trust-root touch. One verify per turn: check routes the agent to run verify, then the Stop hook ran an identical verify again at turn end. The verifier records the exact bytes it evaluated in verification-input.diff; when that matches the hook's own snapshot and the config and refs agree, the hook reuses the existing verifier.json. Byte-exact comparison, so any mismatch falls through to a normal run. Measured on a fixture repo: 0.9s reused versus 2.1s re-run, and a further edit correctly re-verifies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both were identified in the post-v19 re-review and both need a design decision before code, so they land as docs with tracking issues rather than half-built features. insufficient-evidence-cold-start.md (#292): why a first-adoption repo ends every turn with a human-review notice — verify's force-run contract attaches a repo-wide evidence verdict to diffs that touch nothing capability-shaped, while init omits the very agent_bindings key the abstention asks for. Three independently shippable steps, with the propose-and-ratify path reused so the agent proposes and a human declares. host-authenticated-approval-receipts.md (#293): the narrow stuck state v19 deliberately left — a user-approved high-risk host-grant edit. Records the binding (tool_use_id, both content hashes), the authenticity requirement (host-attested or annotation-only), and the eligibility allow-list. It also records why the unsigned local-receipt version was rejected, so that design is not re-proposed by accident. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six reproducible P1s from review of #294, plus the P2s. Verifier reuse removed entirely (findings 3 and 4). It read agents-shipgate-reports/verifier.json — a workspace file the agent can write — so a forged or interrupted artifact with control.state=complete skipped verification, and an agent_action_required artifact could inject an arbitrary "exact" command. Worse, the identity binding was insufficient: committing a change reproduces the same diff text the earlier worktree run recorded, which the reviewer used to reproduce an outcome-changing stale pass (old `passed` reused while a fresh verify returned review_required for permissions.scopes: ["*"]). Doing this correctly needs binding to the verifier's own content-addressed request identity, which a hook script cannot compute; it is a performance optimization worth ~1.2s, so it goes out rather than shipping a bypass. Tracked as a follow-up. Approval memory hardened (findings 1 and 2, plus two P2s). It now applies only to `ask`: a configured `deny` is an operator's hard block and is never quieted, and nothing is recorded while the boundary is disabled with `allow`, because no request was made and so nothing was allowed. Only paths that provably resolve inside the repository may be stored or consulted — _repo_path deliberately degrades an out-of-workspace absolute path to its basename so the boundary still warns, and that fallback let /elsewhere/shipgate.yaml authorize the repository's own manifest. The permission-mode set was wrong: "ask" is not a mode, and acceptEdits does surface an explicit hook request, so the check is now a deny-list of auto-answering modes with an absent mode treated as unknown. State writes are atomic (temp + os.replace) and merge concurrent sessions instead of replacing them, bounded per session and by session count. reports_pending_review made sound (finding 5). It matched on path and title tokens, so "Edited CLAUDE.md. All done." passed while reporting nothing, and the generic acknowledgement regex matched the negation "No review item is outstanding". Evidence is now identifying ids only, and a denial never satisfies the duty to report. Both counterexamples are regression tests. pending_review removed from AGENT_RESULT_CONTROL_FIELDS (finding 6). That tuple describes agent_result_schema_path, and the field exists only on AgentBoundaryResultV1 — pointing consumers at a schema that omits it was wrong. Reverted in the contract, the .well-known mirror, and both pins; the field stays documented in protocol.md, agent-contract-current.md, and the skills, which is where consumers actually learn it. The cold-start design no longer proposes auto-writing agent_bindings, which contradicted do_not_auto_assert. init scaffolds the unresolved key and names the detected candidate as a suggestion; the human still writes the assertion. Reproduced each P1 fix against the reviewer's own shapes on a live fixture: out-of-workspace approval no longer reaches the repo manifest, deny survives a prior approval, acceptEdits records, and concurrent sessions both persist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups from the post-merge coding-agent UX re-review of #291 (contract v19). Four batches, no contract or schema version bump — v19's semantics are unchanged; this makes them discoverable, measurable, and cheaper to live with.
pending_review[], and stop contradicting the control state. The re-review found zero surfaces still teaching the oldrequire_review → stopsemantics (the "switch oncontrol.state" discipline paid off), but the new obligation was taught nowhere:docs/agents/protocol.mdomitted the field from its list, itsagent_action_requiredrow, and its never-a-stop-state enumeration;docs/agent-contract-current.mdcontradicted its own v19 preamble by leaving it out of the operative read list;docs/checks.mdstill asserted human review is required for the UNCLASSIFIED catch-all without saying the local loop grades it; anddocs/integrations.mddescribed the Stop hook as soft-blocking whenever the verifier returns findings — wrong in both directions now. The graded check summary also read "require human review" while control said the agent may continue; both summary builders are now band-aware and say the review is carried to PR time.reports_pending_reviewin the adoption harness: a graded row traded a hard stop for a reporting duty, so an agent that runs verify and never surfaces the carried items now fails. Naming each item passes; a generic "review item outstanding" acknowledgement passes with a weaker signal;n_awhen nothing carried items. This restores the coveragerespects_must_stopsilently lost when graded rows stopped being stops.AGENT_RESULT_CONTROL_FIELDSgainspending_review, so the runtime contract,.well-knownmirror, and generated.shipgate/agent-contract.jsonall name the field (it stays out ofAGENT_CONTROL_FIELDS— the obligation is result-level). The Codex skill'scontinue/repair/stoprouting enumeration gains the fourth shape. Longstanding drift fixed while the hashes were moving anyway: the Claude skill claimedverifier_schema_version: "0.5"; current is0.6.prompts/file cost one permission prompt per keystroke, which trains people to click through. PostToolUse only fires after a call went through, so in a prompting permission mode an edit that landed is an edit a human allowed — the hook records those paths per session and PreToolUse stops re-asking. One verify per turn:checkroutes the agent to run verify, then the Stop hook ran an identical verify again; the hook now reuses an existingverifier.jsonwhen the recordedverification-input.diffbyte-matches its own snapshot and the config and refs agree.insufficient_evidenceattaches a human-review notice to every turn in first-adoption repos) and #293 (host-authenticated approval receipts). The receipts doc also records why the unsigned local-receipt version was rejected, so that design is not re-proposed by accident.Type
Verification
CI is authoritative for
python -m ruff check .,python -m compileall -q src tests, andpython -m pytest.Additional local checks run:
pytestsuite green (exit 0) on the final tree;ruff check .clean.session_idprompts again; an edit landed underbypassPermissionsrecords nothing (nobody was asked);AGENTS_SHIPGATE_APPROVAL_MEMORY=offrestores prompt-every-time. Four new tests.agent_action_required; a high row still reads "require human review" withhuman_review_required.main: benign edits continue with a carried item; MCP-server add,shipgate.yamledit, and AGENTS.md verifier-requirement removal still stop;fixture run ai_generated_refund_prstillblocked.llms-full.txt; kit render hashes and both kits'prior_render_sha256moved together with their byte-identical mirrors.Release-readiness notes
docs/checks.md(none added or removed; UNCLASSIFIED prose qualified)STABILITY.md(no schema or contract version change;AGENT_RESULT_CONTROL_FIELDSgains a name for a field v19 already shipped)Note on the approval memory, since it touches a permission surface: it changes no verdict, control state, or gate semantics.
shipgate checkand PR-time verify evaluate every edit exactly as before,SHIP-VERIFY-*still reports the trust-root touch, the memory is session-scoped and capped, the auto-accepting modes record nothing, and it is disableable. It suppresses a repeated prompt, not a check.🤖 Generated with Claude Code