From 0558056ce7713556f37163819ad904c93566a081 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 16 Jul 2026 21:52:25 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20agent=20failure=20modes=20=E2=80=94=20v?= =?UTF-8?q?alidated=20catalogue=20+=20structural=20mitigations=20(#130)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design deliverable for the build-chain campaign (PyAutoBuild#155 Phase 5): the 2026-07-15 A-E catalogue validated by measurement and enlarged out-of-sample with this campaign's own errors (F1-F5, incl. the shared-index sweep repeated x3 with a same-morning memory note about it); taxonomy compressed to two root causes; the refusals-vs-informs hypothesis tested and sharpened (informs include the agent's own live self-awareness) with a measured refusal false-positive cost; six ranked mitigations led by pipefail-as-default and a Mind commit guard; memory verdict (consult-on-purpose yes, tripwire no); first estimate of the unnoticed-error set. Docs only; each mechanism is its own future PR. Co-Authored-By: Claude Fable 5 --- docs/agent_failure_modes.md | 190 ++++++++++++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 docs/agent_failure_modes.md diff --git a/docs/agent_failure_modes.md b/docs/agent_failure_modes.md new file mode 100644 index 0000000..e6f6264 --- /dev/null +++ b/docs/agent_failure_modes.md @@ -0,0 +1,190 @@ +# Agent failure modes — the validated catalogue and structural mitigations + +**Status:** design deliverable for PyAutoBrain#130 (build-chain campaign +PyAutoBuild#155, Phase 5 — deliberately last, so every earlier phase's errors +enlarge the evidence). **Method:** the original 2026-07-15 catalogue's +checkable claims were re-measured, and the 2026-07-16 campaign session is +mined as an out-of-sample test of the central hypothesis — by an agent that +was *reading these briefs while making the new errors*, which is itself the +strongest datum here. + +## 1. The original catalogue, validated + +Checked 2026-07-16: git **2.34.1** (D1's `merge-tree` needs ≥2.38 — +confirmed); `PyAutoMind/repos.yaml` now carries **33** repos (was 30 — the +hand-list gap C1 measures keeps growing); `smoke_tests.txt` is **11** entries +(A2's false claim confirmed falsifiable in seconds); the `use_jax` defaults +(A4) were re-verified during Phase 3. Nothing in A–E was refuted. The A–E +taxonomy survives, with one compression (§3). + +## 2. The out-of-sample additions (2026-07-16, one campaign session) + +Same discipline — claim, truth, cost, caught by: + +| # | error | caught by | +|---|---|---| +| F1 | `git commit` in the shared PyAutoMind checkout swept a concurrent session's staged rename (×1), then a *directory-scoped* commit swept two more (×2), then a concurrent session swept THIS session's staged records (×3, reverse direction) — **E1 repeated three times in one day, with a fresh memory note about E1 written that same morning** | git log reading, after push, each time | +| F2 | `cmd \| head; $?` read the pager's exit, not the command's — **twice** — and `black --check -q`'s silence was read as cleanliness once | self, on re-reading output (luck) | +| F3 | A checkpoint claim ("no DISABLE_JAX pins remain — question may be moot") built on a `head -4` grep was **published to a GitHub issue** before being falsified | a deliberate falsifying probe (D2's habit, applied on purpose this time) | +| F4 | Two *stale claims of this session's own completed tasks* blocked new work | `worktree_check_conflict` **refused** — forced the completion recording that should already have happened | +| F5 | The PyAuto API gate **refused a pure file-inspection command** (pattern-matched path text as API symbols) | n/a — a refusal false positive; cost: one bypassed re-run | + +F1 is the headline: the trap was documented in the morning, in a memory note +written specifically about it, by the agent that then walked into it three +times before dinner. **B1 is not an anecdote; it is the normal case.** + +## 3. The compressed taxonomy + +The A–E grouping compresses to two root causes plus a hygiene class: + +1. **Verification aimed at the believed path** (A1–A5, D1, D3, F2, F3): real + work producing real evidence for a proposition nobody questioned. The + agent reliably verifies — and reliably verifies the wrong thing. +2. **State consulted from memory instead of from the world** (B1, B2, C1, + C2, E1, F1, F4): notes, recollections and stale registrations standing in + for `git status`, `repos.yaml`, and the claims ledger at the decisive + moment. +3. **Convention/shared-state hygiene** (E2–E4, F5's cost column): real but + secondary; the guards that exist here already work. + +## 4. The hypothesis, tested on the new data + +**"Refusals catch, reminders don't" survives — with a sharper statement and +a measured cost column.** + +- Refusals that fired correctly this session: `worktree_check_conflict` ×2 + (F4), the `test_pre_build_skill` drift guard (caught a real doc/script + drift on clean main during a ship), `lifecycle.py check` (flagged a + half-done completion), and the brand-new `validate_env_profiles` — which on + first contact found 3 dead config patterns and 15 vacuous-JAX scripts that + a day of human-authored enumeration had missed. Hit rate on things they + guard: still effectively 100%. +- Informs that failed this session: the E1 memory note (three more sweeps); + the agent's *own live awareness* of the A5 error shape (it re-committed + that exact shape in F3 while mid-campaign about it). **Awareness is an + inform. Even self-awareness.** That is the sharper statement: the dividing + line is not documentation-vs-tooling, it is whether anything *interrupts + the action*, because at the decisive moment the agent is not consulting + anything — not notes, not even its own recent conclusions. +- The cost column (previously unmeasured): F5 shows refusals false-positive; + the API gate misfired on path text and cost one bypass. One misfire against + dozens of correct refusals across two days is a good ratio, but every new + refusal proposed below must state its expected false-positive surface — + a refusal that misfires often trains bypass-by-default, which is worse + than no guard (the brief's own routing-around warning). + +## 5. Ranked mitigations + +Each: catalogue entries caught → why it fires at the decisive moment → cost +→ how it fails. + +1. **`set -o pipefail` (+ never `-q` on check-mode tools) as the agent-shell + default.** Catches F2's entire class (three instances in one day) by + making the *instrument* loud instead of the agent careful — the purest + delete-the-trap available, since the mistake becomes unmakeable rather + than detectable. Cost: zero per task; failure mode: legitimately-failing + pipe heads (grep-no-match) now need explicit handling — small, visible, + and itself error-revealing. **This is a one-line hook/profile change; do + it first.** +2. **A Mind commit guard (refusal).** In the shared PyAutoMind checkout, + refuse any `git commit` that (a) lacks explicit `--` *file* pathspecs, or + (b) would include index entries outside those paths (another session's + staged work). Catches E1 + F1 (four incidents across two days, the + highest-frequency error in the whole catalogue). Fires at the decisive + moment by construction — it *is* the action. Cost: a wrapper or PreToolUse + check scoped to one repo; false-positive surface: deliberate multi-file + registry commits, handled by listing the files (which is the desired + behaviour anyway). +3. **Stale-claim auto-expiry (refusal-adjacent).** F4 shows completed tasks + linger in `active.md` and block; the conflict guard catches it late but + correctly. Cheaper at the source: `worktree_remove` (already the cleanup + choke-point) warns-then-offers the `active.md` release + completion record + when the branch it removes is merged. Catches F4's class before it costs a + conflict roundtrip. Cost: one prompt in an existing tool. +4. **Blessed instruments for the repeated questions** (C2, D1, D2): the + campaign's own `validate_env_profiles` is the proof the pattern works — + the question "is this config sane?" got one tested implementation and + immediately out-performed recall. Remaining candidates, each a small task: + "does this branch contribute to main?" (D1/D2's hand-rolled disaster — + `pyauto-gut`'s remit), and "which repos exist?" → `repos.yaml` is already + canonical; the missing piece is that nothing *routes* to it, which is + mitigation 5. +5. **Make the router consult the body map, not the agent.** C2's lesson + ("nothing routes the agent to the artifact while deciding") generalises: + wherever a conductor needs a repo list, it should *call* the sizing + faculty / `repos.yaml` loader rather than accept a hand-list argument. + Grep the conductors for hand-list parameters; each is a small refactor. + Fires at the decisive moment because the enumeration happens inside the + tool rather than in the prompt. +6. **Adversarial pass as a checkpoint stage, not a virtue.** It caught the + three worst 07-15 errors and F3 — every time it actually ran. It cannot be + a hook (it is judgment), but it can be a *structural stage*: the ship + skills' checkpoint format gains a required "falsified-by" line per + load-bearing claim (what would make this false; what was run to check). + This is dangerously close to the banned checklist — the honest difference + is that it is enforced by the checkpoint *reader* (the human sees a + missing falsified-by line), not remembered by the agent. Cost: minutes per + ship; failure mode: rote compliance — mitigate by keeping it to + load-bearing claims only. **Trial it; measure whether it goes rote.** + +## 6. The memory system, attacked honestly + +The day's evidence cuts both ways. Failures: B1, B2, and F1's note that did +not fire — memory as *tripwire* is refuted twice over. Successes: the +intake-handfix note correctly guided a live intake (consulted deliberately at +task start), and the campaign-state note is what makes cross-session +continuation possible at all. **Verdict: memory earns its keep as +consult-on-purpose context and fails reliably as guard-at-the-moment.** The +policy that follows: guard-shaped memories ("NEVER do X in Y") are candidates +for conversion into refusals (mitigation 2 is exactly the E1 note, +converted); a memory that cannot be converted should state *when it will be +consulted*, or it is a hope, not a mechanism. + +## 7. Estimating the unnoticed set + +Method: count claims that crossed an external boundary (issue comment, commit +message, PR body, user-facing summary) and were *later* found wrong, as a +lower bound on the rate at which unverified boundary-crossing claims are +wrong. This campaign: 3 such catches (A2 on 07-15; F3's published checkpoint; +the Phase 1 brief's "origin READMEs correctly pin" claim, half-refuted by +measurement) across roughly 40 boundary-crossing checkpoint claims — **~7% +caught-late rate among claims that WERE eventually re-derived.** The honest +estimate is that claims never re-derived are wrong at a comparable rate: +in a session producing dozens of boundary-crossing claims, **expect 1–3 +undetected false statements to be live in the record right now.** The +mitigation is not zero-defect (impossible) but boundary discipline: +mitigation 6 targets exactly the boundary-crossing moment. + +## 8. Rejected + +- **Documentation as primary mechanism** — refuted a third time by F1, now + with the strongest possible variant (same-day self-authored note). +- **"Try harder" / awareness** — refuted by F3: the error was made *during* + a campaign about that error, by the agent writing about it. +- **A universal pre-action checklist** — banned by the brief, and F1 shows + why: the decisive moment has no checklist-consulting step to hook into. +- **Rejecting the refusals hypothesis** — attempted per the brief; the new + data strengthened it instead. Its honest weaknesses: refusals only exist + where someone built one (coverage is reactive), and F5's false-positive + cost is real. Neither overturns the hit-rate asymmetry. + +## 9. Open decisions (human) + migration + +- Mitigation 1 (`pipefail` default): approve as a harness/profile change — + one line, highest ratio. 2 (Mind commit guard) and 3 (claim expiry): small + tasks, file on approval. 4–5: mine the conductors, file individually. + 6 (falsified-by lines): trial on the next ship series, review whether it + went rote after ~10 ships. +- Whether F5's API-gate false-positive class warrants a scope fix (only scan + arguments that are Python code, not path strings) — small, filed + separately. +- Each mechanism lands as its own PR behind its own plan; nothing here + changes behaviour by merging this document. + +## Trust nothing here + +Written by the agent that made F1–F3 *while writing about A–E*. The §7 +estimate is constructed from this document's own author's error rate and +should be treated as optimistic. Every §1 number is re-checkable in seconds; +§2's incidents are all in the campaign's issue trail (PyAutoBuild#155/#156/ +#161, PyAutoHeart#83) with commands and commit SHAs.