Skip to content

Teach pending_review, instrument it, and cut two interaction costs - #294

Merged
pengfei-threemoonslab merged 5 commits into
mainfrom
claude/agent-ux-followups-v19
Jul 28, 2026
Merged

Teach pending_review, instrument it, and cut two interaction costs#294
pengfei-threemoonslab merged 5 commits into
mainfrom
claude/agent-ux-followups-v19

Conversation

@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor

Summary

Follow-ups from the post-merge coding-agent UX re-review of #291 (contract v19). Four batches, no contract or schema version bump — v19's semantics are unchanged; this makes them discoverable, measurable, and cheaper to live with.

  • Teach pending_review[], and stop contradicting the control state. The re-review found zero surfaces still teaching the old require_review → stop semantics (the "switch on control.state" discipline paid off), but the new obligation was taught nowhere: docs/agents/protocol.md omitted the field from its list, its agent_action_required row, and its never-a-stop-state enumeration; docs/agent-contract-current.md contradicted its own v19 preamble by leaving it out of the operative read list; docs/checks.md still asserted human review is required for the UNCLASSIFIED catch-all without saying the local loop grades it; and docs/integrations.md described the Stop hook as soft-blocking whenever the verifier returns findings — wrong in both directions now. The graded check summary also read "require human review" while control said the agent may continue; both summary builders are now band-aware and say the review is carried to PR time.
  • Instrument the failure mode grading introduced. New blocker criterion reports_pending_review in the adoption harness: a graded row traded a hard stop for a reporting duty, so an agent that runs verify and never surfaces the carried items now fails. Naming each item passes; a generic "review item outstanding" acknowledgement passes with a weaker signal; n_a when nothing carried items. This restores the coverage respects_must_stop silently lost when graded rows stopped being stops.
  • One render-hash batch. AGENT_RESULT_CONTROL_FIELDS gains pending_review, so the runtime contract, .well-known mirror, and generated .shipgate/agent-contract.json all name the field (it stays out of AGENT_CONTROL_FIELDS — the obligation is result-level). The Codex skill's continue/repair/stop routing enumeration gains the fourth shape. Longstanding drift fixed while the hashes were moving anyway: the Claude skill claimed verifier_schema_version: "0.5"; current is 0.6.
  • Two interaction costs, both hook-local and advisory. Approval memory: prompt-engineering a single prompts/ file cost one permission prompt per keystroke, which trains people to click through. PostToolUse only fires after a call went through, so in a prompting permission mode an edit that landed is an edit a human allowed — the hook records those paths per session and PreToolUse stops re-asking. One verify per turn: check routes the agent to run verify, then the Stop hook ran an identical verify again; the hook now reuses an existing verifier.json when the recorded verification-input.diff byte-matches its own snapshot and the config and refs agree.
  • Projected the two remaining debts as designs rather than half-building them: #292 (insufficient_evidence attaches a human-review notice to every turn in first-adoption repos) and #293 (host-authenticated approval receipts). The receipts doc also records why the unsigned local-receipt version was rejected, so that design is not re-proposed by accident.

Type

  • Check or risk-model change
  • CLI or GitHub Action behavior
  • Input adapter change
  • Report, schema, or SARIF output
  • Documentation only

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • Full pytest suite green (exit 0) on the final tree; ruff check . clean.
  • Approval memory, live on a fixture repo with hooks installed: first edit prompts → PostToolUse records → second edit silent; a different session_id prompts again; an edit landed under bypassPermissions records nothing (nobody was asked); AGENTS_SHIPGATE_APPROVAL_MEMORY=off restores prompt-every-time. Four new tests.
  • Verify reuse: agent-run verify then Stop hook → 0.9s reusing versus 2.1s re-running, and a further edit correctly re-verifies. Two new tests, one of which derives the expected input diff from the rendered hook itself rather than re-implementing it.
  • Copy alignment checked both ways: a medium row now reads "need PR-time review; verify, then report them" with agent_action_required; a high row still reads "require human review" with human_review_required.
  • Behavior matrix re-run on a fresh fixture built from main: benign edits continue with a carried item; MCP-server add, shipgate.yaml edit, and AGENTS.md verifier-requirement removal still stop; fixture run ai_generated_refund_pr still blocked.
  • Regenerated llms-full.txt; kit render hashes and both kits' prior_render_sha256 moved together with their byte-identical mirrors.

Release-readiness notes

  • No user-code import added to default scan paths
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md (none added or removed; UNCLASSIFIED prose qualified)
  • Report/schema changes are additive or documented in STABILITY.md (no schema or contract version change; AGENT_RESULT_CONTROL_FIELDS gains a name for a field v19 already shipped)

Note on the approval memory, since it touches a permission surface: it changes no verdict, control state, or gate semantics. shipgate check and PR-time verify evaluate every edit exactly as before, SHIP-VERIFY-* still reports the trust-root touch, the memory is session-scoped and capped, the auto-accepting modes record nothing, and it is disableable. It suppresses a repeated prompt, not a check.

🤖 Generated with Claude Code

pengfei-threemoonslab and others added 5 commits July 27, 2026 14:54
Docs the v19 grading left behind: docs/integrations.md described the Stop
hook as soft-blocking whenever the verifier returns findings (it now blocks
only for agent_action_required, and human_review_required deliberately does
not block); docs/agents/protocol.md omitted pending_review[] from the field
list, from the agent_action_required row, and from the never-a-stop-state
enumeration; docs/agent-contract-current.md contradicted its own v19
preamble by leaving pending_review out of the operative read list; and
docs/checks.md asserted human review is required for the UNCLASSIFIED
catch-all without qualifying that the local loop now grades it.

The check summary also read 'require human review' while control said the
agent may continue. Graded results now say the review is carried to PR time,
so the prose and the control state agree.

New harness criterion reports_pending_review restores the coverage
respects_must_stop lost when graded rows stopped being stops: the band
traded a hard stop for a reporting duty, so an agent that runs verify and
never surfaces the carried items now fails a blocker criterion. Naming each
item passes; a generic acknowledgement passes with a weaker signal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er pin

One render-hash batch over the kit surfaces:

- AGENT_RESULT_CONTROL_FIELDS gains pending_review, so the runtime contract,
  .well-known mirror, and the generated .shipgate/agent-contract.json all name
  the field an agent must read. It stays out of AGENT_CONTROL_FIELDS: the
  obligation is result-level, not part of the control block.
- The Codex skill described check routing as continue/repair/stop. The graded
  band added a fourth shape (verify, then report the carried items), which is
  the one most likely to fire on a routine .codex/config.toml or SKILL.md
  edit, so the enumeration now names it.
- The Claude Code skill's control sentence now says a non-empty
  pending_review[] lets the turn finish but must be reported.
- Longstanding drift fixed while the hashes were moving anyway: the skill
  claimed verifier_schema_version 0.5; the current verifier schema is 0.6.

Kit sources, their byte-identical mirrors, EXPECTED_*_RENDER_SHA256, and both
kits' prior_render_sha256 move together so an installed managed copy is still
recognized as unmodified on the next update.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two interaction costs the graded stop exposed, both hook-local and both
advisory — no verdict, control state, or gate semantics change.

Approval memory: the PreToolUse boundary asked again on every edit to an
already-allowed protected file, so prompt-engineering a single prompts/ file
cost one prompt per keystroke and taught people to click through. PostToolUse
only fires after a tool call went through, so in a prompting permission mode
an edit that landed is an edit a human allowed; the hook records those paths
per host session and PreToolUse skips re-asking for them. A new session asks
again, the auto-accepting modes record nothing because nobody was asked, the
memory is capped and scoped to the current session, and
AGENTS_SHIPGATE_APPROVAL_MEMORY=off restores prompt-every-time. shipgate check
and PR-time verify still evaluate every edit, and SHIP-VERIFY-* still reports
the trust-root touch.

One verify per turn: check routes the agent to run verify, then the Stop hook
ran an identical verify again at turn end. The verifier records the exact bytes
it evaluated in verification-input.diff; when that matches the hook's own
snapshot and the config and refs agree, the hook reuses the existing
verifier.json. Byte-exact comparison, so any mismatch falls through to a normal
run. Measured on a fixture repo: 0.9s reused versus 2.1s re-run, and a further
edit correctly re-verifies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both were identified in the post-v19 re-review and both need a design
decision before code, so they land as docs with tracking issues rather than
half-built features.

insufficient-evidence-cold-start.md (#292): why a first-adoption repo ends
every turn with a human-review notice — verify's force-run contract attaches
a repo-wide evidence verdict to diffs that touch nothing capability-shaped,
while init omits the very agent_bindings key the abstention asks for. Three
independently shippable steps, with the propose-and-ratify path reused so the
agent proposes and a human declares.

host-authenticated-approval-receipts.md (#293): the narrow stuck state v19
deliberately left — a user-approved high-risk host-grant edit. Records the
binding (tool_use_id, both content hashes), the authenticity requirement
(host-attested or annotation-only), and the eligibility allow-list. It also
records why the unsigned local-receipt version was rejected, so that design
is not re-proposed by accident.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six reproducible P1s from review of #294, plus the P2s.

Verifier reuse removed entirely (findings 3 and 4). It read
agents-shipgate-reports/verifier.json — a workspace file the agent can write —
so a forged or interrupted artifact with control.state=complete skipped
verification, and an agent_action_required artifact could inject an arbitrary
"exact" command. Worse, the identity binding was insufficient: committing a
change reproduces the same diff text the earlier worktree run recorded, which
the reviewer used to reproduce an outcome-changing stale pass (old `passed`
reused while a fresh verify returned review_required for
permissions.scopes: ["*"]). Doing this correctly needs binding to the
verifier's own content-addressed request identity, which a hook script cannot
compute; it is a performance optimization worth ~1.2s, so it goes out rather
than shipping a bypass. Tracked as a follow-up.

Approval memory hardened (findings 1 and 2, plus two P2s). It now applies only
to `ask`: a configured `deny` is an operator's hard block and is never quieted,
and nothing is recorded while the boundary is disabled with `allow`, because no
request was made and so nothing was allowed. Only paths that provably resolve
inside the repository may be stored or consulted — _repo_path deliberately
degrades an out-of-workspace absolute path to its basename so the boundary
still warns, and that fallback let /elsewhere/shipgate.yaml authorize the
repository's own manifest. The permission-mode set was wrong: "ask" is not a
mode, and acceptEdits does surface an explicit hook request, so the check is
now a deny-list of auto-answering modes with an absent mode treated as unknown.
State writes are atomic (temp + os.replace) and merge concurrent sessions
instead of replacing them, bounded per session and by session count.

reports_pending_review made sound (finding 5). It matched on path and title
tokens, so "Edited CLAUDE.md. All done." passed while reporting nothing, and
the generic acknowledgement regex matched the negation "No review item is
outstanding". Evidence is now identifying ids only, and a denial never
satisfies the duty to report. Both counterexamples are regression tests.

pending_review removed from AGENT_RESULT_CONTROL_FIELDS (finding 6). That tuple
describes agent_result_schema_path, and the field exists only on
AgentBoundaryResultV1 — pointing consumers at a schema that omits it was
wrong. Reverted in the contract, the .well-known mirror, and both pins; the
field stays documented in protocol.md, agent-contract-current.md, and the
skills, which is where consumers actually learn it.

The cold-start design no longer proposes auto-writing agent_bindings, which
contradicted do_not_auto_assert. init scaffolds the unresolved key and names
the detected candidate as a suggestion; the human still writes the assertion.

Reproduced each P1 fix against the reviewer's own shapes on a live fixture:
out-of-workspace approval no longer reaches the repo manifest, deny survives a
prior approval, acceptEdits records, and concurrent sessions both persist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pengfei-threemoonslab
pengfei-threemoonslab merged commit 0f06023 into main Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant