insufficient_evidence attaches a human-review notice to every turn in first-adoption repos

[pengfei-threemoonslab]

Symptom

On a freshly adopted repo (framework detected, small tool surface), every turn that runs the local loop ends with "A human must review this change before it can merge" — including a docs-only or README-only edit. Reproduced 2026-07-27 on a fresh fixture built from main at contract v19.

Two mechanics combine:

1. verify passes user_requested=True (cli/verify/orchestrator.py:155), so the capability scan runs on every invocation regardless of what the diff touched — the adopted-repo force-run contract.
2. A repo whose tool surface is not statically enumerable trips evidence_below_ie_threshold, returning insufficient_evidence → merge_verdict: human_review_required.

The abstention is correct as a release judgement. Attached to a docs-only turn it reads as a non sequitur, and because insufficient_evidence is immune to baselines, suppressions, severity overrides, and human_ack by explicit contract — with every generated remedy actor: human — there is no action the agent can take.

Why now

Contract v19 (#291) graded the local boundary stop, so benign edits no longer hard-stop at check. That moved this to the top of the remaining coding-agent UX friction: it is now the main reason a routine turn still ends with a review notice.

Aggravating detail (disproved — see Revision)

init detects the framework with high confidence and writes a manifest that does not declare agent_bindings — so the routed onboarding path manufactures the very gap verify then abstains on.

Measured 2026-07-28: a repository with one literal Agent(...) resolves its root through the single-top-level-agent fallback and reports binding_coverage.gap_count == 0 without any agent_bindings key. init does not manufacture the gap.

Revision (2026-07-28, after PR #296)

Measuring the two real cold-start shapes retired most of the original direction, and found the actual root cause elsewhere. Kept open with the criteria below.

The root cause was discoverability, not the verdict. The decision engine already generated the exact manifest snippet each gap wants (EvidenceGapAction.declaration_template), but the only way to reach one was to walk report.json down to release_decision.evidence_coverage.evidence_gaps[].next_action.declaration_template. A three-line, one-time task looked like schema archaeology, so repositories stayed abstained indefinitely.

Retired — 1. init scaffolds agent_bindings. Wrong premise (above), and actively harmful: every CHANGE_ME line becomes a source warning, and four warnings crosses _MAX_TOLERATED_SOURCE_WARNINGS and trips the abstention on its own. The value init has is also the wrong selector — agent_bindings.root.object matches the assignment target for the OpenAI Agents SDK, not the detected name= literal.

Retired — 2. Scope the verdict to the diff. Circular: insufficient_evidence means the static view of the agent surface is unreliable, so a rule reasoning from that view cannot be trusted. A config-bound or dynamically constructed toolkit presents as a diff touching no recognized capability surface while changing what the agent can do — precisely the case the verdict exists for would escape it. The delta variant ("no new gaps versus base") fails identically, because an unseeable addition yields an identical gap set. The verdict stays.

Retired — 3. Agent-proposable declaration. Effect, authority, and binding declarations are human-owned by the do_not_auto_assert contract. An agent proposing a tool's effect is the invented evidence that contract forbids, so there is no agent-executable path to build here. (Structural tool-source proposals under #282 remain fine; those are not claims about behavior.)

Shipped in #296

• suggested-declarations.yaml: the templates assembled next to the report, merged per target, deleted when nothing is owed.
• Three template defects that surfacing them exposed — the binding template asserted complete: true / tools: [] on the human's behalf; the authority template was unfillable for every mode; the selector template pointed at a schema that could not accept it. Plus a guard test enumerating every shipped template.
• Gap attribution: verify reports whether this diff introduced a gap or inherited it.
• Framework-correct low-confidence remedy (tool_inventories does not exist for openai_agents_sdk).

Acceptance (revised)

The original two criteria — a docs-only turn with no human-review notice, and an agent-executable remediation — are withdrawn: the first requires the retired verdict change, the second is forbidden by do_not_auto_assert. What remains for this issue:

• A weak-evidence repository's verify names a concrete, human-executable next step, and following it moves the verdict off abstention. (Verified: filling the scaffold cleared inferred_effect_only and missing_authority_evidence; the fixture went to blocked because the declared financial_write tool has no approval policy.)
• No shipped declaration template asserts a value the human owns, and every one validates when filled for each accepted value.
• A docs-only turn states that it introduced no new evidence gap rather than implying the current change is under suspicion.
• fixture run ai_generated_refund_pr unchanged; the abstention still fires where the diff is a capability change.
• Open: re-run the labeled corpus and report the movement. 2026-W27 attributed most of benign_escalation_rate 0.286 to a cold-start whole-repo-surface artifact; that attribution now needs re-testing, since the artifact was not what this work turned out to be about. must_block_caught / needs_human_caught must hold at 1.0.
• Open: decide whether the residual — a weak-evidence repository still ending each turn with a human-review notice until someone makes the declarations — is acceptable as designed, or belongs with the host-authenticated approval work in Host-authenticated in-session approval receipts (unblock user-approved high-risk host-grant edits) #293.

Non-goals (unchanged): weakening insufficient_evidence where the diff is a capability change; letting an agent author binding, effect, or authority declarations.