diff --git a/decisions/0031-run-evaluation-split.md b/decisions/0031-run-evaluation-split.md new file mode 100644 index 0000000..16a9056 --- /dev/null +++ b/decisions/0031-run-evaluation-split.md @@ -0,0 +1,119 @@ +--- +id: adr-0031 +title: Run evaluation is a deterministic collector plus a rubric-ratified judge, reporting to a roll-up issue +kind: adr +status: proposed +supersedes: +superseded-by: +--- + +# ADR 0031: Run evaluation is a deterministic collector plus a rubric-ratified judge, reporting to a roll-up issue + +- Status: Proposed +- Date: 2026-07-02 + +## Context + +A retroactive evaluation of a completed full-pipeline run on a consumer +repository (spec `docs/specs/03-spec-run-evaluation/`) established that the +suite's failures were overwhelmingly detectable during the run from API state +alone — label timelines, run shapes, diff properties, artifact greps — yet +were detected by a human watching, or not at all. The pipeline reached +`sdd:done` on a feature that was not demoable; recovery, review-loop closure, +and integrity restoration were all manual; the one surface built to fix +operator legibility (ADR 0023) had never run on any consumer because its +default-on predicate (`vars.SDD_STATUS != '0'`) evaluates false when the +variable is unset (GitHub expressions coerce `null` and `'0'` to the same +number). Nothing in the suite aggregates phase latency, dwell, loop counts, +requirement coverage, or cost. + +The evaluation capability must therefore be trustworthy (its own signals +cannot lie or die silently), cheap enough to run on every boundary, and +comparable across runs and suite versions. + +## Decision + +1. **Two parts.** A deterministic collector (`sdd-eval-collect`: thin + wrapper + composite action, no `engine:`, OTLP-exempt per ADR 0020 §5) + computes every hard metric; a separate engine-bearing judge + (`sdd-eval-judge`: gh-aw source + hosted lock + wrapper, full ADR 0020 + burden) scores only what requires inference. +2. **Four trigger tiers** on the collector: `workflow_run` completion of + `sdd-*` agents; lifecycle label transitions; `sdd:done` feature + post-mortem; weekly scheduled aggregation. The judge fires at post-mortem + and on schedule; per-run judging is anomaly-sampled only. +3. **Delivery is a roll-up issue upsert** (sentinel ``, + history in comments — the weekly-roll-up pattern), plus a versioned + `eval_snapshot.json` artifact as the machine surface. No committed + scorecards, no tracking-issue body edits, no lifecycle label writes. +4. **App identity, default-on** — with the enablement predicate written + unset-safe and lint-asserted (the ADR 0023 surface's coercion failure is + the motivating incident). +5. **Rubric ratification.** Qualitative scoring runs only under a committed, + versioned rubric in which every item carries an anchored scale and a + machine-check class (`deterministic`/`assisted`/`inference`); a rubric + version takes effect only after an explicit operator ratification recorded + on the roll-up issue. The judge stamps rubric version, pinned model, + suite ref, and snapshot schema version into every score. +6. **Budget is anchored to the run being judged**: per-feature judge budget + defaults to 5% of the feature's own recorded pipeline usage (floor/cap + via repository variables), seeded from the evaluated run's baseline. +7. **The retro is a workflow**: `/retro-eval ` re-runs the entire + retroactive analysis (collector backfill + judge post-mortem) for any + tracking issue; the original retro is its first manual execution. +8. **Eval reports; a separate actor acts.** Any mutation driven by findings + (labels, issues, nudges, escalation) belongs to the actor agent + (`docs/specs/04-spec-eval-actor/`), which is opt-in and separately + trusted. + +## Reasoning + +- The repo's standing posture — no LLM where API state suffices (ADR 0023) — + fits the evidence exactly: the motivating retro computed all of its + headline findings deterministically and needed inference only for R-ID + semantics, revert intent, and quality judgments. +- Splitting reporter from actor keeps the reporting surface side-effect-free + and its trust posture minimal (one issue write), mirroring the + status/monitor split that already exists. +- Ratified, versioned rubrics are what make scores comparable; an unratified + rubric silently changing between runs would reintroduce the drift the + judge exists to measure. +- Anchoring the judge budget to the evaluated feature's own cost keeps the + evaluator strictly cheaper than the thing it evaluates, at any scale. + +## Verification + +- `wrappers/sdd-eval-collect.yml` exists with no `engine:` and no compiled + lock; its enablement predicate passes the unset-safe lint. +- `.github/workflows/sdd-eval-judge.md` declares the ADR 0020 block and a + pinned model; its wrapper maps the OTLP secret. +- A seeded run produces: snapshot artifact validating against its schema, + scorecard upsert on the roll-up issue, post-mortem comment with all four + reproducibility stamps. +- `scripts/test-eval-rubric.py` gates rubric changes in `lint.yml`; a judge + dry-run refuses an unratified rubric version. +- The lifecycle state-machine validator shows no new lifecycle-label + writers. + +## Consequences + +- Operators get boundary-time detection of the failure classes the evaluated + run surfaced (stranding, false signals, integrity tripwires, proof + weakening, cost anomalies) and a comparable post-mortem per feature. +- The suite gains its first feedback instrument: suite-version regressions + become visible as scorecard trends. +- Two new wrappers per consumer (collector, judge) plus roll-up issue noise + bounded to one issue per repository. +- The `sdd:done ≠ demoable` gap becomes measurable; closing it (demoability + gates, integration-task planning, recovery fixes) remains pipeline work + tracked outside this ADR. + +## Cross-links + +- ADR 0020 (observability mandate; §5 exemption), ADR 0023 (deterministic + status surface — pattern and the coercion lesson), ADR 0019 (App + identity), ADR 0015 (wrapper logic in composite actions), ADR 0004 + (distribution), ADR 0001 (`needs-human` stays the human hand-off; eval + never applies it). +- `docs/specs/03-spec-run-evaluation/` (this capability's spec), + `docs/specs/04-spec-eval-actor/` (the actor). diff --git a/docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md b/docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md new file mode 100644 index 0000000..ae83636 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/03-spec-run-evaluation.md @@ -0,0 +1,521 @@ +--- +id: spec-run-evaluation +title: Run evaluation — deterministic collector and rubric-ratified judge for SDD pipeline runs +kind: spec +status: planned +tracking-issue: +supersedes: +--- + +# 03-spec-run-evaluation + +> The repository is public. This file, and everything committed to the repo, +> carries no employer name, no private org slug, no internal repository name, +> no internal URL, no cost figure, and no contributor personal data. The run +> that motivated this spec is referred to only as "the evaluated run." + +## Context + +A full feature was run through the SDD pipeline on a consumer repository (the +evaluated run: full-pipeline path, suite pinned to v0.3.0, June 2026). The +pipeline reached `sdd:done`; the feature was not user-demoable, and two +human-written remediation PRs followed. A retroactive evaluation reconstructed +the run from its own exhaust and found that every one of its headline failures +was **detectable during the run from API state alone**: + +- the tracking issue's status surface had zero successful runs ever (an + enablement-predicate bug: `vars.X != '0'` is false when the variable is + unset, because GitHub expressions coerce `null` and `'0'` to the same + number); +- 17 `needs-human` episodes totalling ~134 h of dwell, 11 of them in the back + half of the run, every recovery manual (23 human `/execute` comments); +- the automated stranding recovery recovered 0 of 6 escalations; +- review loops re-raised identical findings across passes and across sibling + PRs, closed only by ~80 human inline dispositions; +- two live silent-revert incidents (a merged spike doc deleted by a green + agent PR and never restored; a five-way stale-base revert bundle that sat + warning-only for 13 h), while the dedicated revert guard answered + INCONCLUSIVE on every invocation; +- proof artifacts declared in the spec were delivered `#[ignore]`-gated, + env-gated, or behind cargo features no binary compiled — `sdd:done` was + reachable with no executed end-to-end demonstration. + +During the evaluated run, exactly one failure class was caught by automation +the framework owned; everything else was caught by red consumer CI +(post-damage) or a human watching. Nothing in the suite aggregates phase +latency, dwell time, loop counts, cost per feature, or requirement coverage. + +This spec adds a **run-evaluation capability**: a deterministic collector +that computes hard metrics from API state at every pipeline boundary, and a +separate engine-bearing judge that scores what requires inference, under a +version-ratified rubric. It also makes the retroactive evaluation itself a +durable, re-runnable workflow so any tracking issue can be evaluated after +the fact with one command. + +## Introduction / Overview + +Two cooperating components, following the repo's "no LLM where API state +suffices" posture: + +1. **`sdd-eval-collect`** — a deterministic utility workflow (thin wrapper + + composite action, no `engine:`, OTLP-exempt per ADR 0020 §5 — the + sdd-status/sdd-monitor pattern). It fires on all four trigger tiers, + computes the metric catalogue from GitHub API state and run artifacts, + writes a machine-readable snapshot artifact, and upserts a human-readable + scorecard onto a roll-up issue (history preserved in comments — the + weekly-roll-up pattern `sdd-derive` already uses). + +2. **`sdd-eval-judge`** — an engine-bearing agent (gh-aw source + compiled + lock + thin wrapper, full ADR 0020 observability burden). It consumes the + collector's snapshots plus the underlying diffs/threads and scores the + dimensions that need judgment (spec fidelity of merged diffs, proof + deliverability, revert suspicion, review-loop convergence quality, ledger + evidence quality), each score stamped with the pinned model and the + ratified rubric version. + +The **evaluation agent reports; it never acts**. Labels, issues, nudges, and +escalations driven by eval findings belong to a separate actor agent +(`docs/specs/04-spec-eval-actor/`). + +## Goals + +1. Replace "a human watching" as the primary failure detector: every + detection-gap-matrix row whose signal is deterministic is computed by the + collector during the run, not reconstructed weeks later. +2. Make `sdd:done ≠ demoable` visible at the moment it happens: proof-delivery + and demoability metrics are part of the per-feature post-mortem. +3. Make runs comparable: one metric catalogue, version-stamped rubric, + pinned judge model, so scores trend across runs and suite versions. +4. Make the retro re-runnable: `/retro-eval ` reproduces the full + retroactive evaluation for any tracking issue. +5. Keep the deterministic path free: the collector runs with no engine, no + OTLP block, and negligible latency, like sdd-status. + +## User Stories + +- As an operator, I want a scorecard that updates at each phase boundary so + that I see stranding, dwell, loop counts, and cost while I can still act. +- As an operator, I want a post-mortem comment when a feature reaches + `sdd:done` that tells me which requirement IDs are delivered / partial / + missing and which declared proofs actually executed, so that "done" is a + claim I can trust or challenge immediately. +- As a suite maintainer, I want recurrence fingerprints (the same finding or + failure signature re-appearing after a fix) surfaced automatically, so a + point fix that needed a guard is caught by the second incident, not the + third. +- As a suite maintainer, I want cost per feature and per phase trended across + runs so sizing and prompt changes show up as slope changes. +- As an operator, I want the judge's scores reproducible — same rubric + version, same pinned model — so a score delta means the run changed, not + the yardstick. + +## Demoable Units of Work + +### Unit 1: `sdd-eval-collect` — deterministic collector and scorecard delivery + +**Purpose:** Compute the hard-metric catalogue from API state and run +artifacts on every trigger tier; persist a machine-readable snapshot; upsert +the human-readable roll-up. Demoable: label events on a seeded test tracker +produce a scorecard comment and a snapshot artifact with the documented +schema. +**Depends on:** none (foundation) +**Affected areas:** `wrappers/sdd-eval-collect.yml` (new), +`.github/actions/sdd-eval-collect/action.yml` (new), +`.github/actions/shared/parent-walk.js` (reuse), `templates/.github/labels.yml` +(no new labels; read-only), `scripts/quick-setup.sh` (installer entry), +`shared/sdd-interaction.md` (command table), `docs/sdd/` (surface doc). + +**Functional Requirements:** + +- **R1.1**: `sdd-eval-collect` shall be a deterministic utility workflow: a + thin wrapper plus composite action with no `engine:` and no compiled + `.lock.yml`, exempt from the OTLP mandate per ADR 0020 §5. +- **R1.2**: The wrapper shall fire on all four trigger tiers: (a) + `workflow_run` completion of every `sdd-*` agent workflow; (b) lifecycle + label transitions on tracking issues (`issues: labeled/unlabeled` for + `sdd:*` and `needs-human`); (c) feature post-mortem on the `sdd:done` + transition; (d) a weekly `schedule` for cross-run aggregation. A token-strict + `/eval` comment command (write-access-gated, `eyes` ack) forces a refresh. +- **R1.3**: The enablement predicate shall be **default-on with a correct + unset case**: the opt-out gate must evaluate to enabled when the repository + variable is unset. It shall be written as an explicit disable comparison + (`vars.SDD_EVAL == '0'` negated at the job level via + `vars.SDD_EVAL != '0' || vars.SDD_EVAL == ''` or an equivalent + `format()`-normalized form) and covered by a lint assertion, because the + evaluated run demonstrated that `vars.X != '0'` alone is false for unset + variables (GitHub expression coercion) and silently disabled a default-on + surface fleet-wide. +- **R1.4**: On tiers (a)–(c) the collector shall resolve the affected + tracking issue (branch-prefix join keys per ADR 0023's reasoning, shared + parent walk) and compute the **per-feature metric set**: + - phase wall-clock per lifecycle transition, and human-gate share of + elapsed wall-clock; + - `needs-human` episodes: applier, clearer, dwell, and position in the run + (fraction of elapsed run time), flagging late episodes (position > 0.5); + - stranding: any task `sdd:in-progress` with no open PR and no queued or + in-flight execute run beyond the execute timeout plus slack; monitor + re-dispatch attempts observed vs. recoveries achieved; + - run-shape anomalies on the completed run that triggered tier (a): + conclusion success with empty safe-output items on an implementation + directive, `num_turns == 1` with zero tool calls, `startup_failure`, + activation-cancelled after successful route/ack, fan-wide + `startup_failure` across agentic wrappers while pure-actions wrappers + pass; + - review-loop counters per open `sdd/` PR: auto-revise / check-revise / + conflict-resolve marker counts vs. caps, review-comment and reaction + tallies, thumbs-down reactions attached to runs whose conclusion was + `cancelled` (false-signal detection), finding fingerprints + (path + rule/quote hash) re-raised after a thread was resolved, both + within a PR and across sibling PRs of the same tracker; + - thread-resolution audit via GraphQL: bot-resolved threads whose last + comment is not a bot reply containing a commit sha or a rebuttal; + - integrity: for every open `sdd/`, `spec/`, `arch/` PR, the net diff + against current origin/main — deletions or reversions of files/hunks that + main has independently advanced, docs-only assertion for spec-agent PRs, + size/file-count tripwires on "docs" PRs; recurrence-fingerprint match of + new incidents against closed "fixed" framework issues; a spec-edit + tripwire flagging any implementation PR that modifies a merged + `docs/specs/**` spec file (the evaluated run's spec drift was spread by + exactly such an edit) so the judge's fidelity item re-runs on the edited + spec; + - sizing: net-diff distribution of merged `sdd/` PRs vs. the ADR 0026 + floor and this spec's advisory ceiling (§ Design Considerations); + single-task-Unit count; tasks vs. plan-comment tree (ADR 0010 guarantees + exact materialization); + - proof delivery: the spec's declared proof artifacts joined against the + merged diffs — `#[ignore]`/environment-gate/`required-features` guards + detected on the tests that implement declared proofs, and proofs with no + implementing artifact at all; + - gate/CI parity: gate-passed-then-first-consumer-CI-fail joins per PR; + bot commit subjects pre-checked against discovered commitlint config; + - requirement coverage: R-ID presence joins spec → task bodies → PR + diffs/titles/bodies; + - dispatch health: cascade fan-outs vs. run starts they produced; manual + `/execute`/`/dispatch` counts (the human-touch inventory); + - status-surface health: exactly one status comment per tracker, fresher + than the last pipeline event; + - cost: per-run token usage parsed from run artifacts — both artifact + layouts (`agent_usage.json` where present; the engine result record in + the agent stdio log otherwise) — aggregated per phase, per model tier, + and per feature, with cost-per-net-line computed at post-mortem. +- **R1.5**: Every tier-(a)–(c) invocation shall upload an + `eval_snapshot.json` artifact conforming to a documented, versioned schema + (`schema_version` field; documented in `docs/sdd/sdd-eval.md`). The + snapshot is the machine surface the judge and the actor agent consume. +- **R1.6**: The collector shall upsert one roll-up issue per repository, + located by the `` sentinel (title prefix `[sdd-eval]`), + editing the issue body in place with the latest per-feature scorecards and + appending one comment per feature post-mortem (tier c) so history is + preserved in comments — the established weekly-roll-up pattern. It shall + never edit tracking-issue bodies and never write lifecycle labels; its only + writes are the roll-up issue, its own comments/reactions, and artifacts. +- **R1.7**: On tier (d), the collector shall aggregate across features: + trend tables for phase latency, needs-human rate/dwell/position, loop + counts, recurrence fingerprints, cost per feature and per net line, and + gate-latency share — appended to the roll-up issue as a dated comment. +- **R1.8**: The collector shall run under the suite's App identity (ADR + 0019) so scorecard writes attribute to the bot, with `issues: write`, + `pull-requests: read`, `actions: read`, `checks: read`, `contents: read`. +- **R1.9**: All wrapper logic beyond trigger routing shall live in the + composite action (ADR 0015); shared join code (parent walk, branch-prefix + resolution) shall be extracted and reused, not duplicated. +- **R1.10**: `/eval` shall join the command vocabulary in + `shared/sdd-interaction.md` and pass the command-table consistency gate; + the wrapper-lock contract, lifecycle state-machine (no lifecycle writes by + this workflow), and leak-scan lint gates shall pass unchanged. + +**Proof Artifacts:** + +- Test: on a seeded test repository, applying `sdd:in-progress` to a task + with no PR and waiting past the timeout threshold produces a scorecard + entry flagging one stranded task; the `eval_snapshot.json` artifact + validates against the documented schema. Fails before this unit (no + workflow exists). +- CLI: `gh api` for the roll-up issue after a seeded `sdd:done` transition + shows a post-mortem comment containing phase wall-clock, needs-human, and + proof-delivery sections. Fails before this unit. +- File: `wrappers/sdd-eval-collect.yml` exists, carries no `engine:`, and + the repository compiles no `sdd-eval-collect.lock.yml`; the enablement + predicate matches the R1.3 unset-safe form and a lint test asserts it. + +### Unit 2: Scoring rubric, machine-check plan, and ratification gate + +**Purpose:** Commit the qualitative scoring rubric as a versioned document +with a machine-check plan per item, and gate rubric versions behind operator +ratification so judge scores are comparable and auditable. Demoable: the +rubric file passes its lint, and an unratified rubric version is refused by +the judge. +**Depends on:** none (parallel to Unit 1) +**Affected areas:** `docs/specs/03-spec-run-evaluation/rubric.md` (new), +`scripts/test-eval-rubric.py` (new lint), `.github/workflows/lint.yml` +(gate registration), `shared/sdd-interaction.md` (`/ratify-rubric`). + +**Functional Requirements:** + +- **R2.1**: `rubric.md` shall enumerate every judged dimension as an item + with: a stable id (`RB--`), the question being scored, an anchored + scale (each point defined by observable evidence, not adjectives), the + evidence sources (which snapshot fields, diffs, threads), and a + **machine-check plan** classifying the item as `deterministic` (collector + computes it; the judge must not re-score it), `assisted` (collector + computes inputs, judge scores), or `inference` (judge-only), with the + concrete check named for the first two classes. +- **R2.2**: The rubric shall carry a `version:` in frontmatter. Any change to + item ids, scales, or machine-check classes requires a version bump; the + lint (`test-eval-rubric.py`) asserts id stability, scale anchoring + (every scale point cites an evidence source), and version-bump-on-change + (hash recorded in the file). +- **R2.3**: A rubric version shall take effect only after **ratification**: + a write-access `/ratify-rubric ` comment on the roll-up issue (or + the spectacles-side PR approval when the rubric changes in-repo), recorded + by the collector as a dated ledger comment. The judge shall refuse to score + with an unratified version and fall back to the last ratified one, saying + so in its output. +- **R2.4**: The launch rubric (v1) shall cover at minimum: spec fidelity of + merged diffs per R-ID (delivered / partial / missing, judged on wired-in + behavior, not text presence), proof-artifact deliverability (executed vs. + gated vs. missing), demoability at `sdd:done` (does an executed end-to-end + artifact exist), silent-revert suspicion on PR diffs vs. origin/main, + revise-diff-addresses-threads quality, assumption-ledger evidence quality + (settled-by-circular-citation detection), source fidelity of the + generated spec (contradictions between generated normative requirements + and the source document they were derived from — every downstream stage + validates against the generated spec and is blind to this class by + construction, so it is judged at the spec boundary), plan quality, and + contributor-facing tone of bot copy. Each anchored to the failure classes + the evaluated run exhibited. + +**Proof Artifacts:** + +- File: `rubric.md` exists with ≥8 items, every item carrying id, anchored + scale, evidence sources, and machine-check class; `scripts/test-eval-rubric.py` + passes in `lint.yml` and fails when an item's scale point lacks an evidence + citation (demonstrated by a fixture). +- Test: the ratification ledger flow on a test roll-up issue — `/ratify-rubric v1` + recorded, a judge dry-run with `v2-unratified` refuses and names the + fallback. Fails before this unit. + +### Unit 3: `sdd-eval-judge` — engine-bearing inference scoring + +**Purpose:** Score the rubric's assisted and inference items on real run +evidence, reproducibly and within budget. Demoable: a post-mortem judge run +on a completed feature produces a rubric-versioned, model-stamped score +comment on the roll-up issue. +**Depends on:** Unit 1 (snapshots), Unit 2 (ratified rubric) +**Affected areas:** `.github/workflows/sdd-eval-judge.md` (new gh-aw +source + compiled lock), `wrappers/sdd-eval-judge.yml` (new), +`scripts/quick-setup.sh`, `docs/sdd/sdd-eval.md`. + +**Functional Requirements:** + +- **R3.1**: `sdd-eval-judge` shall be a gh-aw agent distributed per ADR 0004 + (hosted lock + thin wrapper) carrying the full ADR 0020 observability + block, importing `shared/rigor.md` (evidence standards) and the eval + interaction contract. +- **R3.2**: The judge shall fire on tier (c) (`sdd:done` post-mortem) and + tier (d) (scheduled aggregation), and on demand via a write-access + `/eval judge` comment. Per-run (tier a) judging is **sampled, not + universal**: only runs the collector flags anomalous (run-shape anomalies, + integrity tripwires) are judged, to hold budget. +- **R3.3**: Every score the judge emits shall be stamped with: rubric + version, engine model id (pinned in the workflow source, not floating), + prompt/source revision (the suite ref), and the snapshot `schema_version` + it consumed. Scores without all four stamps are invalid by definition. +- **R3.4**: The judge shall score only `assisted` and `inference` rubric + items; for `deterministic` items it shall quote the collector's value and + may annotate but not override it. +- **R3.5**: Per-feature judge budget shall default to **5% of the feature's + own recorded pipeline token usage** (from the collector's cost metric), + with an absolute floor and cap set as repository variables + (`SDD_EVAL_JUDGE_MIN`/`_MAX`); the baseline figure is seeded from the + evaluated run's recorded usage. On budget exhaustion the judge emits a + partial scorecard that names the unscored items — never a silent + truncation. +- **R3.6**: The judge's output is one comment per invocation on the roll-up + issue (post-mortem scores under the feature's section), plus its standard + artifacts. It shall write no labels, no issues, and no PR comments. +- **R3.7**: The judge's remediation-delta mode: when invoked on a feature + whose tracking issue gained post-`sdd:done` human PRs referencing it, it + shall classify the remediation diff hunks into + specced-but-not-delivered / needed-but-unspecced / quality-refactor and + report the split — the evaluated run's B3 analysis, productized. + +**Proof Artifacts:** + +- Test: a post-mortem judge run on a seeded completed feature emits a score + comment carrying all four reproducibility stamps and per-R-ID verdicts; + re-running with the same inputs and pins yields the same rubric version and + verdict set. Fails before this unit. +- CLI: with the budget variable set below the run's needs, the judge output + names the unscored items explicitly. Fails before this unit. + +### Unit 4: Durable retro-eval entry point + +**Purpose:** Make the one-off retroactive evaluation re-runnable: given any +tracking-issue number, reproduce the full Phase-1 analysis (fetch, compute, +classify, report). The retro of the evaluated run is this workflow's first +(manual) execution. Demoable: `/retro-eval ` on a completed tracker +yields the full retro scorecard as a roll-up comment. +**Depends on:** Units 1–3 +**Affected areas:** `wrappers/sdd-eval-collect.yml` (backfill mode), +`.github/workflows/sdd-eval-judge.md` (retro directive), +`shared/sdd-interaction.md` (`/retro-eval`), `docs/sdd/sdd-eval.md`. + +**Functional Requirements:** + +- **R4.1**: A write-access `/retro-eval ` comment (and a + `workflow_dispatch` input equivalent) shall run the collector in + **backfill mode**: reconstructing phase timelines, episodes, loop counts, + integrity scans, proof delivery, coverage, and cost for the named tracking + issue from historical API state and any still-retained artifacts, tolerant + of expired artifacts (cost fields degrade to "unavailable", never fail the + run). +- **R4.2**: Backfill shall then invoke the judge post-mortem (R3.7 included + when remediation PRs exist), producing the complete retro scorecard — + executive metrics, yardstick table, per-R-ID verdicts, remediation split — + as one roll-up comment thread for the feature. +- **R4.3**: The entry point shall be idempotent per issue: re-running + `/retro-eval` on the same issue upserts the feature's section rather than + duplicating it, and records each execution in the ledger. +- **R4.4**: `docs/sdd/sdd-eval.md` shall document the retro procedure, + including the artifact-retention caveat (run artifacts expire; retro-eval + within the retention window preserves cost evidence, after it the retro is + timeline-only). + +**Proof Artifacts:** + +- Browser/CLI: `/retro-eval` on a seeded historical tracker produces the + full scorecard comment; a second invocation updates rather than duplicates + it. Fails before this unit. +- File: `docs/sdd/sdd-eval.md` documents backfill mode and the retention + caveat. + +## Non-Goals + +- **Acting on findings.** No labels applied or cleared, no issues opened on + findings, no nudges, no blocking. That is the actor agent + (`04-spec-eval-actor`), deliberately separated so the reporting surface + stays trustworthy and side-effect-free. +- **Replacing sdd-validate's gates.** The collector observes and reports + gate outcomes; it does not become a merge gate itself in v1. (The retro's + recommendation to make the net-diff integrity signal a required check is a + pipeline change, tracked separately.) +- **Dashboards.** OTLP spans from the judge flow to the existing collector + infrastructure; standing up dashboards/alerting is outside this spec. +- **Fixing the failures it measures.** Stranding recovery, review-loop + convergence, status-surface enablement are pipeline issues with their own + tracks; this spec only guarantees they are *seen*. + +## Design Considerations + +- **Two parts, not one.** Every headline detection in the motivating retro + was deterministic (timeline joins, run-shape checks, diff properties, + greps). An engine-bearing evaluator for those would add cost, latency, + drift, and the ADR 0020 burden to signals that must be trustworthy and + cheap — the ADR 0023 reasoning, reapplied. Inference is reserved for + judgment calls (R-ID semantics, revert intent, tone). +- **Roll-up issue, not committed scorecards.** Scorecards are operational + telemetry about a run, not knowledge about the system; committing them + would churn the repo and drag distillery-sync into ingesting run noise. + The issue surface keeps history in comments, supports the ratification + ledger, and is already the pattern operators know. +- **Default-on, correctly.** The status surface was default-on by intent and + default-off in fact for its entire life because of one expression-coercion + subtlety. R1.3 encodes the lesson as a requirement plus a lint, not a code + comment. +- **Sizing ceiling (advisory).** The evaluated run's floors worked and + overshot: median merged net diff ~1.7× the 400-line target, with the three + largest PRs carrying the heaviest review loops. The collector reports an + advisory ceiling breach at 2× the floor (800 net lines); ratifying a hard + ceiling is a rubric/process decision left to the operator. +- **Judge sampling.** Universal per-run judging at the evaluated run's scale + (hundreds of agentic runs per feature) would dwarf the pipeline's own + cost. Anomaly-gated sampling plus post-mortem depth keeps the default + budget inside R3.5's 5% envelope. +- **Cross-layout cost parsing.** The engine migration mid-June changed the + artifact layout; the collector must parse both (`agent_usage.json` and the + engine result record in the agent stdio log) or cost silently + under-reports — the retro hit exactly this. + +## Repository Standards + +Distribution per ADR 0004 (judge: source + self-contained lock + thin +wrapper; collector: wrapper + composite action, ADR 0015). All ten `lint.yml` +gates apply, most relevantly: wrapper-lock contract (secrets ⊆ declared; +caller ≥ callee permissions), lifecycle state-machine validator (this suite +writes no lifecycle labels — the validator must show no writer additions), +command-table consistency (`/eval`, `/ratify-rubric`, `/retro-eval`), +requirement-ID cross-references, and leak-scan. Conventional commits; +`needs-human` contract per ADR 0001 (the eval surfaces never apply it — that +is the actor's decision under its own spec). + +## Technical Considerations + +- Tracking-issue resolution reuses `.github/actions/shared/parent-walk.js` + and the branch-prefix join keys; no `closingIssuesReferences` (ADR 0023). +- Finding fingerprints: hash of (file path, rule id or normalized quote) + per review thread; recurrence = same fingerprint after a resolved thread + or across sibling `sdd/` PRs of one tracker. Persisted in the snapshot so + recurrence detection is stateless per invocation. +- The integrity scan shells `git diff --name-status origin/main...HEAD` + against a **full-history fetch with credentials** — the evaluated run's + revert guard failed precisely because it ran on a shallow, credential-less + checkout; the collector must not repeat that (assert non-shallow before + scanning, else report the scan itself as failed). +- GraphQL is required for thread-resolution state and minimized-comment + flags; REST alone cannot compute the resolution audit. +- Snapshot schema is versioned independently of the rubric; the judge + records both. +- The weekly tier reuses the collector's own snapshots as input — it never + recomputes history it already recorded. + +## Security Considerations + +- The collector runs on an App token with write access only to issues; it + edits only content carrying its own sentinel. Prompt-injection surface: + issue/PR text flows into scorecards — the collector treats all body text + as data (no command interpretation beyond the token-strict `/eval` gate, + which checks actor write access via the repository-permission API, not + `author_association`). +- The judge reads diffs and threads from the repository it evaluates; + its safe-outputs are restricted to comments on the roll-up issue. It + carries the standard firewall/egress allowlist; OTLP endpoint handling per + ADR 0020 §3 (secret URL, masked, artifact-scrubbed). +- Snapshots and scorecards live on the consumer repository and may quote + consumer code; nothing from them is ever committed to this public + repository. + +## Open Questions + +- Whether the advisory 2×-floor sizing ceiling should become a blocking + triage constraint after a few scored runs (informed by ADR 0026 and the + evaluated run's oversize/review-loop correlation). +- Whether tier-(a) collector runs on very busy repositories need their own + concurrency shaping — the status surface's cancel-in-progress storm + produced misleading `cancelled` conclusions during the evaluated run + (informed by #310-class false signals). +- Whether the judge's remediation-delta mode should also fire on *pre*-done + human PRs that reference the tracker (mid-run human interventions were a + material fraction of the evaluated run). +- Where the cross-consumer aggregation lives once more than one consumer + runs the suite (the roll-up issue is per-repo by design; a fleet view is + out of scope here). + +## Gap Analysis + +Empty — forward-authored spec. + +## Verification + +- On a seeded test repository with the suite installed: drive a miniature + feature through spec → done with one deliberate stranding, one oversized + PR, one `#[ignore]`-gated proof, and one stale-base deletion; assert the + scorecard flags all four, the post-mortem carries per-R-ID verdicts and + the proof-delivery table, and `eval_snapshot.json` validates against its + schema. +- Ratify rubric v1, run the judge post-mortem twice with identical pins; + assert identical rubric/model stamps and verdict sets. +- Run `/retro-eval` against the seeded feature after deleting its artifacts; + assert cost fields degrade to "unavailable" and everything else completes. +- All ten `lint.yml` gates green, including the new rubric lint and the + R1.3 enablement-predicate assertion. diff --git a/docs/specs/03-spec-run-evaluation/_handoff.md b/docs/specs/03-spec-run-evaluation/_handoff.md new file mode 100644 index 0000000..64e7150 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/_handoff.md @@ -0,0 +1,213 @@ +# Handoff brief — SDD run evaluation (retro + eval-agent spec) + +> Working document; the leading underscore keeps it out of distillery-sync. +> This file names no consumer identities (the repository is public and +> leak-scanned): `` and `` are placeholders the +> operator maps at session start. Do not commit the real names to this +> repository — including in commit messages. + +## Mission (original intent) + +A full spec was run through the SDD pipeline on ``, tracked +by its issue **#478** (the networking spec, +`docs/specs/03-spec-networking/networking-with-diagrams.md` on the +consumer). After the pipeline completed, **two human-written PRs (#581 and +`#589`, >3000 lines total)** were required before the feature was +user-demoable. Deliverables, in order: + +1. **Retroactive evaluation** of that run: detailed *what worked, what + didn't, what needs to improve* in the spectacles agent factory. + Delivered as an issue on the private consumer repo. +2. **A spec for an evaluation agent** that monitors future runs so + performance can be measured qualitatively and quantitatively — + deterministic where API state suffices, inference-based where judgment + is required. The durable, re-runnable retro workflow is folded into + this spec (interview answer E1-c). Includes a plan for machine-checking + the scoring rubric plus a ratification gate (C2), and a companion + sketch for a separate **actor agent** that acts on eval findings (D5). + +Sequencing: retro first, spec second (E2). Nothing in the factory changes +until after the post-mortem (E3). + +## Binding decisions from the operator interview + +Full Q&A: `_interview.md` (same directory). The load-bearing ones: + +- **Success ranking (B1)**: functional correctness > spec adherence > + wall-clock > cost. `needs-human` is acceptable but must occur *early*. +- **Run shape (B2/B3/B5/B6)**: full pipeline path; all `SDD_*` variables at + defaults; wrappers pinned to tag 0.3.0; CodeRabbit installed; the two + remediation PRs are human-written integration plumbing + refactor — + i.e. the retro evaluates a **demoability shortfall**, not just the + pipeline's own output. +- **Operator priors to verify (B7)**: pain — output proven only in CI + behind a feature flag (not end-to-end); risks not spiked up front caused + late `needs-human`. Surprises — spike workflow, PR scope/size, plan + quality. +- **Spec quality is in scope (C1)**: if the input spec could have been + better, the framework should have exposed that early. +- **Eval agent (D1-D7)**: deterministic collector + separate engine judge; + all trigger tiers; results to a roll-up issue on the private repo; App + identity, default-on; all metric dimensions matter; judge budget + baseline = the networking run's own usage; build directly (no + dogfooding). + +## State of work (what a fresh session inherits) + +All on branch `claude/spectacles-eval-workflow-e66d2n`: + +- `_interview.md` — questions + operator answers. +- `evidence/_run-window-evidence.md` — **the evidence backbone**: a + structured harvest of the 39 spectacles issues around the run window + (June 15–24, 2026). Contains: failure timeline, failure taxonomy, + fix-cost accounting, the three safety-critical silent-revert incidents, + a detection-gap matrix (feeds the eval-agent spec directly), and the + pre-run baseline yardsticks (§6) the run must be scored against. +- `evidence/_run-window-issue-records.json` — the per-issue structured + records behind the synthesis. +- `evidence/_export-run-evidence.sh` — one-command `gh` export of a run's + full forensic surface (issue tree, timelines, comments, minimized flags, + PRs + diffs, workflow runs, optional artifacts). Useful even with direct + access, as a bulk fetcher. + +Established facts to reuse (verified, do not re-derive): + +- **Suite version**: tag v0.3.0 == current `main` (d928c5f, tagged + 2026-06-24). The run's tail ran exactly this tree; its start rode the + v0.2.0→v0.3.0 fix storm (81 commits, most of them fixes prompted by the + run itself). +- **Machine-readable exhaust** an evaluator can parse: lifecycle labels + with single authorized writers (`scripts/lifecycle-states.yml`), comment + sentinels (``, ``, + `` retry markers, monitor audit lines), + the `## Task` block grammar with `blocked by #N` edges, branch-prefix + join keys (`sdd/-`, `spec/`, `arch/` — ADR 0023 says branch regex, + not closingIssuesReferences), per-run Actions artifacts + (`agent_usage.json` = tokens, `prompt.txt` = governing prompt, + `safe-output-items.jsonl` = every requested mutation), and opt-in OTLP + spans (ADR 0020; the consumer had it enabled — read path per the + operator's observability runbook in `/observability`). +- **Nothing aggregates any of this today** — no phase-latency, dwell-time, + loop-count, cost-per-feature, or R-ID-coverage computation exists. +- **Artifact retention clock**: run artifacts expire ~90 days after the + June run — extract token/transcript data before ~mid-September. + +## Status (2026-07-02): both phases delivered + +- Phase 1 retro delivered as a new issue on the consumer repo (its #616); + genericized copy at `evidence/_retro-report.md`. Headlines: 119.7 h + end-to-end, ~56% human-gate share, 17 needs-human episodes (11 late), + 8/30 R-IDs delivered working at `sdd:done` (19 partial, 3 missing), + remediation split a/b/c = 45/30/25% of 5,671 lines, monitor recovery 0/6, + status surface 0 successes ever (unset-coercion bug in the wrapper's + `vars.SDD_STATUS != '0'` gate — new framework bug found by the retro), + two live #326-class silent-revert incidents (one merged doc deletion never + restored), ≈81M effective tokens (lower bound). +- Phase 2 delivered on this branch: `03-spec-run-evaluation.md` (collector + + judge, four trigger tiers, roll-up delivery, `/retro-eval` backfill), + `rubric.md` (v1-draft, unratified), ADR 0031, actor sketch + `docs/specs/04-spec-eval-actor/`. +- Not done (deliberately — E3): no factory changes, no spectacles issues + filed. First candidates for filing: the `SDD_STATUS` unset-coercion bug; + the revert guard's always-INCONCLUSIVE environment (shallow clone). + +## Phase 1 — retroactive evaluation (do this first) + +Reconstruct the consumer-side run, compute the metrics, score against the +yardsticks, verify the operator's priors, write the report. + +1. **Fetch** (direct `gh`/API access, or run the export script): tracking + issue #478 + full sub-issue tree (bodies, comments, label timelines + with actors/timestamps), all `spec/`/`arch/`/`sdd/` PRs (reviews, + review threads incl. minimized/resolved state, diffs, check runs), + remediation PRs #581/#589, workflow-run index for June, artifacts for + the agent runs (at minimum `agent_usage.json`, `aw_info.json`, + `prompt.txt`), OTel data per the runbook, and the pilot baselines + (runs behind spectacles issues #252/#255/#272). +2. **Compute the deterministic metrics** (the yardstick set is + `evidence/_run-window-evidence.md` §6): + - end-to-end wall-clock, split per phase from label events; human-gate + latency share (target <50%) + - `needs-human` count, cause, dwell time, and *position* in the run + (operator standard: early OK, late is a failure — B1/B7) + - task tree shape vs the plan comment (ADR 0010 guarantees the plan + was materialized exactly); units/tasks counts; single-task-Unit + fraction; median + distribution of net-diff per `sdd/` PR (yardstick + ≈400 lines) + - revise-loop counts per PR (auto-revise / check-revise / + conflict-resolve markers) vs caps; finding-recurrence + (re-raised-after-resolved fingerprints, issue #327 class) + - dispatch health: monitor audit lines, stranded re-dispatches, manual + `/dispatch`/`/execute` count (human-touch inventory, B4) + - gate-pass-then-consumer-CI-fail count per PR (parity class: #312, + #318) + - integrity audit: net-diff-vs-main scan of every `sdd/` PR for the + silent-revert class (#287/#317/#326) + - cost: tokens per run/phase/tier from `agent_usage.json` + OTel; + cost-per-net-line ("aic", #272); include the ~25M-token dead triage + run (#271) + - R-ID coverage: spec requirement IDs → task bodies → PR diffs +3. **Analyze the remediation delta (the B3 question, most important + single analysis)**: classify every hunk of PRs #581/#589 as + (a) specced-but-not-delivered, (b) needed-but-unspecced (spec gap the + framework should have exposed early — C1), or (c) quality/refactor. + This quantifies the demoability shortfall and apportions it between + spec quality and factory execution. +4. **Audit the operator's priors (B7)** against evidence: proof-artifacts + proven only in CI behind a feature flag (e2e gap); which risks were + spiked vs which caused late `needs-human`. +5. **Write the retro report**: exec summary → scorecard vs yardsticks → + what worked (with evidence) → what didn't (timeline + taxonomy, both + framework-side from the evidence backbone and consumer-side from the + reconstruction) → what to improve (prioritized; map each item to the + detection-gap matrix row and open spectacles issues) → metric appendix. + Deliver as a **new issue on the private consumer repo** (C4). A + genericized copy may be committed here. + +## Phase 2 — evaluation-agent spec (after the retro) + +Spec at `docs/specs/03-spec-run-evaluation/` (this directory becomes the +spec dir; follow `docs/specs/TEMPLATE.md` frontmatter and demoable-unit +format), plus an ADR (`decisions/`, next free number) per house style. +Requirements fixed by the interview: + +- **Two-part posture (D1)**: deterministic collector (composite action, + no engine, OTLP-exempt per ADR 0020 §5 — the sdd-status/sdd-monitor + pattern) + engine-bearing judge (full ADR 0020 observability burden). +- **All trigger tiers (D2)**: per agent run (`workflow_run` completion), + per phase boundary (label transitions), per feature (`sdd:done` + post-mortem), scheduled cross-run aggregation. +- **Delivery (D3)**: roll-up issue upsert (the `sdd-unspecced-scan` + pattern — history preserved in comments) on the private repo. +- **Identity/rollout (D4)**: App token; default-on (like sdd-status). +- **Metric catalogue (D5)**: start from the detection-gap matrix + (evidence doc §5) — its "deterministic signal" column is the + collector's backlog; its "inference signal" column is the judge's. The + retro's computed metrics become the launch set; all dimensions in + scope. +- **Rubric (C2)**: committed rubric doc, machine-checkable plan per item, + version-stamped scores, and a ratification gate (operator sign-off + workflow) before a rubric version takes effect. +- **Judge reproducibility (D6)**: pinned model + rubric version stamped + into every score; budget baseline = the networking run's usage. +- **Durable retro workflow (E1-c)**: the spec includes a re-runnable + retro-eval entry point (give it a tracking-issue number, get the full + Phase-1 analysis) — the one-off retro is its first manual execution. +- **Actor-agent sketch (D5)**: separate short spec for the agent that + consumes eval findings and acts (labels, issues, escalation) — eval + agent reports, actor agent acts. +- Distribution and CI conventions are mapped in the evidence doc and + `workflows/README.md`: gh-aw source + lock + thin wrapper + installer + entry + the ten `lint.yml` gates (incl. wrapper-lock contract, + lifecycle state machine, command table). + +## Constraints + +- This repository is **public** with a leak-scan CI gate: no consumer org + or repo names, no internal URLs, no cost figures in any committed file + or commit message. Underscore-prefixed files are excluded from + distillery ingestion. +- Branch: all work on `claude/spectacles-eval-workflow-e66d2n`; commit in + conventional-commit style; push with `-u origin`. +- The retro report (with real names) goes to the private repo issue, not + here. diff --git a/docs/specs/03-spec-run-evaluation/_interview.md b/docs/specs/03-spec-run-evaluation/_interview.md new file mode 100644 index 0000000..3e73bf7 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/_interview.md @@ -0,0 +1,276 @@ +# Run-evaluation interview — questions before we begin + +> Working document for the run-evaluation effort. The leading underscore keeps +> it out of distillery-sync ingestion; it is not a spec. Per +> `shared/repo-conventions.md` this file names no consumer repository: the run +> under evaluation is referred to as "the evaluated run" (tracking issue #478, +> implementation PRs #589 and #581 on the consumer repository). + +## Purpose + +Two deliverables hang off this interview: + +1. **A retroactive evaluation** of the completed SDD pipeline run behind + tracking issue #478: what worked, what didn't, and what needs to improve in + the agent factory. +2. **A spec for an evaluation agent** that monitors future runs, measuring + them qualitatively and quantitatively, through both deterministic + computation and inference. + +The questions below cover only what the spectacles repository cannot answer +about itself. Already mined from the tree, and not asked again here: the +machine-readable exhaust of every pipeline stage (lifecycle labels and their +authorized writers in `scripts/lifecycle-states.yml`, comment sentinels such +as `` and the auto-revise retry markers, the `## Task` +block grammar and `blocked by #N` edges, branch-prefix join keys per ADR 0023, +per-run Actions artifacts including `agent_usage.json` and `prompt.txt`, the +opt-in OTLP export of ADR 0020); what `sdd-status`, `sdd-monitor`, +`sdd-validate`, and `sdd-review` already observe; and the conventions a new +distributed workflow must follow (ADR 0004 two-layer distribution, the ten +`lint.yml` gates, the installer wrapper list). + +Answer inline, tersely. "Unknown" and "don't care" are valid answers. +Questions marked **[blocking]** gate the start of the retroactive evaluation; +the rest shape scope and the eval-agent spec. + +--- + +## A. Access to evidence + +**A1. [blocking] Consumer-repo access.** This session's GitHub scope is +limited to the spectacles repository; the run's evidence (issue timelines, +minimized comments, PR review threads, workflow runs, artifacts) lives on the +consumer repository. Which route do you want? + +- (a) widen this session's repository scope so the evaluator reads the + consumer repo directly (needs GraphQL for `isMinimized` comments and + timeline actor/timestamp data); +- (b) run the evaluation *on* the consumer repository via the wrapper model, + the same way the suite itself is distributed; +- (c) you export a forensic bundle (issue + sub-issue timelines with label + events, all comments including minimized ones, PR diffs and review threads, + the workflow-run list, downloaded run artifacts) into a fixtures directory + on this branch. + +**A2. [blocking] Artifact retention.** Are the Actions run artifacts for the +runs behind the spec PR and implementation PRs #589/#581 — +`prompt.txt`, `aw_info.json`, `agent_usage.json`, `otel.jsonl`, +`agent-stdio.log`, `safe-output-items.jsonl` — still retrievable? The run was +roughly two weeks ago, so a default 90-day retention window closes around +mid-September: token/cost and transcript evidence expires with it. Can +someone with access run `gh aw audit ` for the relevant runs? Were +any issues or comments deleted (not minimized) during the run? + +**A3. OTLP telemetry.** Was `GH_AW_OTEL_ENDPOINT` set on the consumer +repository during the run? If yes: where does the collector live, what +backend stores the spans, what is retained, and what is the *read* path (the +in-repo key is write-only by design, ADR 0020)? + +**A4. Distillery store.** Was Distillery configured on the consumer +repository (`DISTILLERY_PROJECT` et al.), and can the evaluator query the +store as a reconstruction surface, or must everything come from GitHub APIs? + +**A5. Baselines.** The ADRs cite observed-run evidence from a pilot +repository (issues #252, #255, #272 — e.g. "~2 days across 4 human gates", +"70–98-line PRs"). Do you consider those the regression baselines for the +evaluated run, and can the evaluator access them? + +--- + +## B. Ground truth about the evaluated run + +**B1. [blocking] Success definition.** What did "success" mean for this +feature? "User demoable" is the phrase used — who was the demo audience, did +the demo happen, and did it pass? Rank these for this run: functional +correctness, spec fidelity, autonomy (zero `needs-human`), wall-clock +latency, cost. + +**B2. [blocking] Path taken.** Did the run take the full pipeline (plan +comment, Unit/task tree, `/dispatch` cascade) or the fastpath/agile single-PR +path? More than 3000 lines across two implementation PRs sits oddly against +the 400-line demoable-unit floor (ADR 0026) and `SDD_AGILE_MAX=800` — was the +two-PR shape the planned task tree, or the result of consolidation, retries, +or rework? + +**B3. [blocking] Provenance of PRs #589 and #581.** The task statement says +that *after completion* the spec "required these two PRs … to be user +demoable." Which is it: + +- (a) the two PRs *are* the pipeline's designed output; +- (b) the pipeline finished but its output was not demoable, and these PRs + are remediation written by humans; +- (c) remediation written by agents re-prompted outside the pipeline; +- (d) something else. + +The answer decides whether the retro evaluates the pipeline's output or the +pipeline's *shortfall*, so precision here matters more than anywhere else. + +**B4. Out-of-band human interventions.** Inventory what humans did that left +no timeline trace: direct pushes to `sdd/` branches, manual merges or +rebases, verbal or chat-channel scope decisions, label removals to pause the +cascade. Roughly how many human touches happened end-to-end, and which do you +regard as legitimate design-intended gates versus framework failures? + +**B5. Consumer configuration during the run window.** Values of +`SDD_AUTO_MERGE`, `SDD_AUTO_DISPATCH`, `SDD_MONITOR`, `SDD_STATUS`, +`SDD_AGILE_MAX`, `SDD_SPEC_MIN_UNIT`, `SDD_TRIAGE_MIN_TASK`, +`SDD_MAX_REVIEW_ITERATIONS`, `SDD_REVISE_ON_CHECKS`, any +`GH_AW_MODEL_*` overrides — and which spectacles ref the wrappers were pinned +to (`@main` or a tag). Also: ADRs 0029/0030 are `status: proposed` — which +behavior was live during the window? + +**B6. Third-party reviewers.** Was CodeRabbit (or any other automated +reviewer) installed on the consumer repository? Its `CHANGES_REQUESTED` +reviews drive the auto-revise loop and its silence drives monitor nudges — +both change the marker counts the retro will score. + +**B7. Your priors.** Before evidence: your top three pain points and top +three pleasant surprises from this run. The retro will verify these against +the record rather than assume them, but they anchor where to dig. + +--- + +## C. Retro-evaluation scope and rubric + +**C1. Is the input spec in scope?** The original networking spec was provided +to the pipeline as-is. Should the retro also evaluate the *spec itself* as a +factory input (testability of its requirements, diagram fidelity, +demoable-unit decomposability), or only the factory's handling of it? + +**C2. Rubric authority.** For qualitative scoring, the candidates overlap and +none is machine-checkable as written: `shared/rigor.md`, +`shared/sdd-proof-artifacts.md`, `shared/sdd-gates.md`, the specs' Success +Metrics prose. Should a ratified scoring rubric be one of the retro's +deliverables, and who ratifies it? + +**C3. Cost economics.** Is cost a target dimension — tokens per feature, per +task, per model tier; Actions minutes? Is there billing-side data outside the +repositories that the evaluator can join against? + +**C4. Report destination and audience.** Who reads the retro report, and +where does it live? Note the constraint: spectacles is public and its +leak-scan forbids committing consumer identities, so a report that names the +consumer repository needs a home outside this tree (consumer repo doc, private +issue, or externally) — or a genericized version lands here. + +--- + +## D. The evaluation agent (spec for future runs) + +**D1. Posture.** The repo's philosophy is "no LLM where API state suffices." +The natural shape is a two-part design: a deterministic collector +(sdd-status/sdd-monitor pattern: composite action, no engine, OTLP-exempt) +computing hard metrics, plus a separate engine-bearing judge for inference +scoring (spec fidelity, artifact quality, code quality). Do you want that +split, or a single engine-bearing agent doing both? + +**D2. Trigger tiers.** When should evaluation fire: per agent run +(`workflow_run` completion), per phase boundary (lifecycle label +transitions), per feature (`sdd:done` post-mortem), scheduled cross-run +aggregation — or several of these as tiers? + +**D3. Results destination.** Sentinel comment edited in place (loses +history), roll-up issue in the `sdd-unspecced-scan` pattern (history in +comments), committed scorecard doc with frontmatter so distillery-sync +ingests it (the spike `budget_hours`/`actual_hours` fields are the repo's +only quantitative-self-report precedent), OTLP spans toward a dashboard — or +a combination? + +**D4. Identity and rollout.** May the eval agent write to consumer +repositories, and under which identity (App token per ADR 0019 vs plain +`GITHUB_TOKEN`)? Opt-in repository variable in the house style +(`SDD_EVAL=1`, like the monitor) or default-on (like the status surface)? + +**D5. Metric priorities and enforcement.** The genuinely unmeasured +dimensions found in the tree: phase latency from label timelines, +`needs-human` rate and dwell time, auto-revise/nudge/re-dispatch loop counts, +R-ID coverage from spec through tasks to PRs, sizing-estimate error against +the 400/800-line floors, gate-finding fire rates, cost per feature, +fastpath-versus-full-path outcomes, proof-artifact quality. Which matter +most? And is the agent report-only, or may it act (apply labels, open +issues, block)? + +**D6. Judge reproducibility and budget.** For the inference part: pinned +model and rubric version stamped into every score (so scores are comparable +across runs)? What per-run inference budget is acceptable? + +**D7. Dogfooding.** Should the eval-agent spec itself be run through the SDD +pipeline (filed as a feature issue, `/spec`, the works) as its own next live +test — or built directly on a branch like this one? + +--- + +## E. Engagement logistics + +**E1. [blocking] What does "build a workflow" mean here?** Two readings: + +- (a) a one-off orchestrated analysis performed now, producing the retro + report and the eval-agent spec; +- (b) a durable, re-runnable retro-evaluation workflow artifact (a slash + command or wrapper that can retro-eval any tracking issue), of which this + run is the first invocation. + +Or (c): do (a) now, and fold the durable version into the eval-agent spec. + +**E2. Sequencing.** Retro report first and eval-agent spec second (the retro +findings feed the spec's metric choices), or in parallel? Any deadline — +noting the artifact-retention clock from A2? + +**E3. Environment changes in flight.** Anything about to change in the +factory (ADR 0029/0030 rollout, model changes, new consumers) that the +evaluation should account for so its findings aren't stale on arrival? + +--- + +## Answers (operator, 2026-07-02) + +Consumer identities are replaced by placeholders in this public file; the +operator supplies the mapping at session start. + +- **A1**: (a) — widen access. *Outcome: session-level allowlists cannot be + widened mid-session; work moved to a local session with direct `gh` + access instead.* +- **A2**: yes — run artifacts are retrievable. +- **A3**: yes — OTLP was set; observability runbook lives in + `/observability`. +- **A4**: yes and yes — Distillery configured and queryable. +- **A5**: yes — the earlier pilot runs are accessible too. +- **B1**: success = user-achievable outcomes. Ranking: functional + correctness, spec adherence, wall-clock, cost (in that order). + `needs-human` is allowed but should occur **early** in the run to + unblock later tests. +- **B2**: full pipeline path. The remediation content was integration + plumbing to expose capabilities to users, plus some refactor. +- **B3**: (b) — the pipeline completed but its output was not user-demoable; + the two remediation PRs are human-written. +- **B4**: yes — out-of-band human touches occurred (inventory to be + reconstructed from evidence plus operator memory). +- **B5**: all repository variables at defaults; wrappers pinned to tag + 0.3.0. +- **B6**: yes — CodeRabbit installed. +- **B7**: pain: (1) not end-to-end — proven only in CI behind a feature + flag; (2) not all risks were spiked up front, causing `needs-human` late + in the run. Surprises: the spike workflow, good PR scope and size, a + well-defined plan. +- **C1**: yes — if the spec could be better, the framework should expose + that early. +- **C2**: the deliverable should include a plan for machine-checking the + rubric plus a ratification gate. +- **C3**: cost is in scope via OTel token usage. +- **C4**: retro report → a private-repo issue; the operator reads it to + produce new specs. +- **D1**: confirmed — deterministic collector + separate engine-bearing + judge. +- **D2**: all trigger tiers (per-run, per-boundary, per-feature, scheduled + aggregation). +- **D3**: roll-up issue in the private repo. +- **D4**: App identity; default-on. +- **D5**: all listed metrics matter. A separate **actor agent** will act on + findings — draft a spec for it too. +- **D6**: the networking run's own usage is the starting budget baseline + for the judge. +- **D7**: build directly on a branch; no dogfooding through the pipeline. +- **E1**: (c) — one-off orchestrated retro now; fold the durable + retro-evaluation workflow into the eval-agent spec. +- **E2**: retro first, spec second. +- **E3**: nothing changes until after the post-mortem. diff --git a/docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh b/docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh new file mode 100755 index 0000000..18b9844 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/evidence/_export-run-evidence.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# Export a forensic bundle of an SDD pipeline run for retroactive evaluation. +# +# Run this from any machine where `gh` is authenticated with read access to +# the target repository. Produces .tar.gz containing the tracking +# issue, its full sub-issue tree, timelines (label events with actors and +# timestamps), all comments, minimized-comment flags (GraphQL), every +# spec/arch/sdd pull request with reviews + review threads + diffs, the +# workflow-run index for the window, and (optionally) per-run jobs and +# agent artifacts. +# +# Usage: +# _export-run-evidence.sh [out-dir] [--artifacts] +# +# EXTRA_PRS="581 589" _export-run-evidence.sh acme/widgets 478 +# also exports PRs whose head branches are not sdd/ spec/ arch/ prefixed +# (e.g. human remediation PRs). +# +# WINDOW_FROM=2026-06-01 overrides the workflow-run window start. +# +# GraphQL comment/thread queries fetch the first 100 nodes per item; for the +# rare item with more, the REST exports still contain every comment body. + +set -euo pipefail + +REPO="${1:?usage: $0 [out-dir] [--artifacts]}" +TRACKER="${2:?usage: $0 [out-dir] [--artifacts]}" +OUT="${3:-run-evidence-${TRACKER}}" +ARTIFACTS="${4:-}" +WINDOW_FROM="${WINDOW_FROM:-2026-06-01}" + +OWNER="${REPO%%/*}" +NAME="${REPO##*/}" + +mkdir -p "$OUT/issues" "$OUT/prs" "$OUT/runs" "$OUT/graphql" + +api() { gh api -H "Accept: application/vnd.github+json" "$@"; } +note() { printf '>> %s\n' "$*" >&2; } + +fetch_json() { # fetch_json ; writes [] on failure + if ! api --paginate "$1" > "$2" 2>/dev/null; then echo '[]' > "$2"; fi +} + +numbers_in() { jq -r '.[].number' "$1" 2>/dev/null || true; } + +# ---- 1. tracking issue + two-level sub-issue tree ------------------------- +note "tracking issue #$TRACKER and sub-issue tree" +api "repos/$REPO/issues/$TRACKER" > "$OUT/issues/$TRACKER.json" +fetch_json "repos/$REPO/issues/$TRACKER/sub_issues" "$OUT/issues/$TRACKER.sub_issues.json" + +: > "$OUT/.issuelist" +echo "$TRACKER" >> "$OUT/.issuelist" +while IFS= read -r unit; do + [ -n "$unit" ] || continue + echo "$unit" >> "$OUT/.issuelist" + fetch_json "repos/$REPO/issues/$unit/sub_issues" "$OUT/issues/$unit.sub_issues.json" + while IFS= read -r task; do + [ -n "$task" ] || continue + echo "$task" >> "$OUT/.issuelist" + done <<< "$(numbers_in "$OUT/issues/$unit.sub_issues.json")" +done <<< "$(numbers_in "$OUT/issues/$TRACKER.sub_issues.json")" +sort -un "$OUT/.issuelist" -o "$OUT/.issuelist" + +while IFS= read -r n; do + [ -n "$n" ] || continue + note "issue #$n: body, comments, timeline" + api "repos/$REPO/issues/$n" > "$OUT/issues/$n.json" + fetch_json "repos/$REPO/issues/$n/comments" "$OUT/issues/$n.comments.json" + fetch_json "repos/$REPO/issues/$n/timeline" "$OUT/issues/$n.timeline.json" +done < "$OUT/.issuelist" + +# ---- 2. pull requests ------------------------------------------------------ +note "pull-request index" +gh pr list --repo "$REPO" --state all --limit 300 \ + --json number,title,headRefName,baseRefName,state,createdAt,closedAt,mergedAt,additions,deletions,changedFiles,author,labels,url \ + > "$OUT/prs/index.json" + +: > "$OUT/.prlist" +jq -r '.[] | select(.headRefName | test("^(sdd|spec|arch)/")) | .number' \ + "$OUT/prs/index.json" >> "$OUT/.prlist" +for x in ${EXTRA_PRS:-}; do echo "$x" >> "$OUT/.prlist"; done +sort -un "$OUT/.prlist" -o "$OUT/.prlist" + +while IFS= read -r n; do + [ -n "$n" ] || continue + note "PR #$n: body, reviews, threads, files, commits, diff" + api "repos/$REPO/pulls/$n" > "$OUT/prs/$n.json" + fetch_json "repos/$REPO/pulls/$n/reviews" "$OUT/prs/$n.reviews.json" + fetch_json "repos/$REPO/pulls/$n/comments" "$OUT/prs/$n.review_comments.json" + fetch_json "repos/$REPO/issues/$n/comments" "$OUT/prs/$n.comments.json" + fetch_json "repos/$REPO/issues/$n/timeline" "$OUT/prs/$n.timeline.json" + fetch_json "repos/$REPO/pulls/$n/files" "$OUT/prs/$n.files.json" + fetch_json "repos/$REPO/pulls/$n/commits" "$OUT/prs/$n.commits.json" + api "repos/$REPO/pulls/$n" -H "Accept: application/vnd.github.v3.diff" \ + > "$OUT/prs/$n.diff" 2>/dev/null || true +done < "$OUT/.prlist" + +# ---- 3. minimized flags + review threads (GraphQL; REST omits both) -------- +# shellcheck disable=SC2016 # $owner/$name/$num are GraphQL variables, not shell +GQL='query($owner:String!,$name:String!,$num:Int!){ + repository(owner:$owner,name:$name){ + issueOrPullRequest(number:$num){ + __typename + ... on Issue { comments(first:100){nodes{ + databaseId isMinimized minimizedReason createdAt author{login}}}} + ... on PullRequest { + comments(first:100){nodes{ + databaseId isMinimized minimizedReason createdAt author{login}}} + reviewThreads(first:100){nodes{ + isResolved isOutdated path + comments(first:50){nodes{databaseId isMinimized author{login}}}}} + } + } + } +}' +sort -un "$OUT/.issuelist" "$OUT/.prlist" | while IFS= read -r n; do + [ -n "$n" ] || continue + note "GraphQL flags for #$n" + gh api graphql -f query="$GQL" -F owner="$OWNER" -F name="$NAME" -F num="$n" \ + > "$OUT/graphql/$n.json" 2>/dev/null || echo '{}' > "$OUT/graphql/$n.json" +done + +# ---- 4. workflow runs in the window ---------------------------------------- +note "workflow runs since $WINDOW_FROM" +api --paginate -X GET "repos/$REPO/actions/runs" \ + -f created=">=$WINDOW_FROM" -F per_page=100 \ + --jq '.workflow_runs[] | {id,name,event,status,conclusion,head_branch,head_sha,run_started_at,updated_at,html_url}' \ + > "$OUT/runs/index.jsonl" || : > "$OUT/runs/index.jsonl" + +# ---- 5. optional: per-run jobs + agent artifacts ---------------------------- +if [ "$ARTIFACTS" = "--artifacts" ]; then + jq -r 'select(.name | test("sdd-|distillery")) | .id' "$OUT/runs/index.jsonl" \ + > "$OUT/.runlist" + while IFS= read -r rid; do + [ -n "$rid" ] || continue + note "run $rid: jobs + artifacts" + fetch_json "repos/$REPO/actions/runs/$rid/jobs" "$OUT/runs/$rid.jobs.json" + fetch_json "repos/$REPO/actions/runs/$rid/artifacts" "$OUT/runs/$rid.artifacts.json" + while IFS=$'\t' read -r aid aname; do + [ -n "$aid" ] || continue + note " artifact $aname" + api "repos/$REPO/actions/artifacts/$aid/zip" \ + > "$OUT/runs/$rid.$aname.zip" 2>/dev/null || true + done <<< "$(jq -r '.artifacts[]? | [.id, .name] | @tsv' "$OUT/runs/$rid.artifacts.json")" + done < "$OUT/.runlist" +fi + +# ---- 6. repo labels (variables need admin; ignore failures) ----------------- +fetch_json "repos/$REPO/labels" "$OUT/labels.json" +gh variable list --repo "$REPO" > "$OUT/variables.txt" 2>/dev/null || true + +tar -czf "$OUT.tar.gz" "$OUT" +note "done: $OUT.tar.gz" diff --git a/docs/specs/03-spec-run-evaluation/evidence/_retro-report.md b/docs/specs/03-spec-run-evaluation/evidence/_retro-report.md new file mode 100644 index 0000000..14c5302 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/evidence/_retro-report.md @@ -0,0 +1,330 @@ +# Retro (genericized): an SDD full-pipeline run, June 2026 + +> Genericized copy of the retro delivered to the consumer repository's +> tracker (its issue #616). Consumer identity, stack-specific names, and +> spend figures are removed per the leak-scan contract; issue/PR numbers +> refer to the consumer repository except where prefixed "spectacles". + +Retroactive evaluation of the full-pipeline SDD run behind consumer tracking issue #478 (June 19-24, 2026; suite pinned v0.3.0; repository variables at defaults; a third-party reviewer installed). Every number is computed from the run's own exhaust. + +**Verdict in one line:** the factory *manufactured* the feature but did not *deliver* it — it shipped 8.3k net lines with a strong plan and honest per-task discipline, then declared `sdd:done` on a feature no user could run, having spent over half its wall-clock waiting on a human, with every unit of review-loop convergence and every stranding recovery performed manually by the operator. + +--- + +## 1. Executive summary + +- **The pipeline completed**: `/spec` (06-19 22:48 UTC) → `sdd:done` (06-24 22:31), **119.7 h** end-to-end. It produced the spec (PR #482), architecture + assumption ledger (#484, #491), 3 spikes (#485/#486/#511), a 12-task tree across 4 units, and **11 merged implementation PRs** (~8.3k net lines) with 30/30 requirement IDs textually traceable spec→task→PR. +- **The output was not user-demoable.** At `sdd:done`, **8 of 30 R-IDs were delivered working end-to-end, 19 partial, 3 missing**; of the spec's declared proof artifacts, **1 was delivered as specified, 9 weakened** (`#[ignore]`-gated, env-gated, CI-lane-only, or behind a cargo feature no binary compiled), **5 missing**. No R-ID, proof artifact, or gate required an *executed user-visible demo*, so `sdd:done` was reachable with none. +- **Remediation cost (PRs #581 + #589): 5,671 changed lines**, classified hunk-by-hunk: + - **45% (2,560 lines) specced-but-not-delivered** — behavior R-IDs required that the pipeline didn't deliver in working form (the VM-side transport transport, a rootless per-sandbox networking mode, proxies built-but-never-served, delivered-but-broken isolated-mode task enforcement); + - **30% (1,702 lines) needed-but-unspecced** — integration the feature needed that no R-ID covered (guest boot/init plumbing: 295 lines in a file **no pipeline PR touched**; daemon egress; demo scaffolding; e2e test plan); + - **25% (1,409 lines) quality/refactor** of pipeline output that worked. +- **Human effort was the pipeline's hidden engine**: 80 human touches — **23 manual `/execute`**, 17 `/revise`, 3 `/dispatch`, 33 substantive decision comments — plus per-finding triage of every review loop and hand-restoration of reverted files. Human-gate share of wall-clock ≈ **56%** (yardstick: <50%). +- **Automated recovery recovered nothing**: sdd-monitor escalated 6 stranding episodes, **0/6 recovered without a human** (spectacles #320 confirmed live). **The status surface never ran**: `sdd-status` has **0 successes in 10,633 runs ever** on this repo — the wrapper's `vars.SDD_STATUS != '0'` gate is false when the variable is unset (GitHub expressions coerce `null` and `'0'` to `0`), so "default-on" is actually default-off. The tracking issue never had a status comment; the operator watched raw Actions streams for 5 days. +- **The silent-revert class (spectacles #326) fired live, twice**: agent arch PR **#491 deleted the 308-line name-resolution spike doc** merged 10 minutes earlier (all checks green, auto-merged; restore PR #543 closed unmerged — **the doc is still absent from main today**), and PR #556's initial diff was a 5-way stale-base revert bundle (deleted the #552 a process-signalling spec, reverted #537's hardening) that sat **warning-only for 13.5 h**. The dedicated merged-change revert guard was **INCONCLUSIVE on every invocation all run** (shallow clone, no credentials). No integrity condition hard-blocked a merge at any point. +- **Cost (lower bound; some artifacts already expired)**: ≈ **81.2M effective tokens** across ≥90 agentic runs attributable to the #478 tree — ≈ **9.8k effective tokens per net line kept** (91% of traffic is cache reads; fresh output was ~0.94M tokens), plus ~15% of it (12.1M) in runs that produced no accepted output (crashes, cancels, quota kills). Detail in appendix C. + +The operator's pre-evidence priors held up well: "proven only in CI behind a feature flag" is **confirmed**; "unspiked risks caused late needs-human" is **half the story** (late episodes split ~5 unspiked-risk vs ~6 framework failures); the three pleasant surprises (spike workflow, PR scope, plan quality) are all **real but overstated** (§6). + +--- + +## 2. Run reconstruction + +### Phase timeline (from tracker label events) + +| Phase | Interval | Wall-clock | +|---|---|---| +| `/spec` → `sdd:spec` → `sdd:triage` (spec PR #482 merged) | 06-19 22:48 → 06-20 02:52 | 3.8 h | +| `sdd:triage` → `sdd:ready` (arch #484/#491, spikes #485/#486, plan, `/approve`, materialize) | → 06-20 06:21 | 3.5 h | +| `sdd:ready` → `sdd:in-progress` (`/dispatch`) | → 06-20 06:27 | 0.1 h | +| `sdd:in-progress` → `sdd:review` | → 06-22 16:13 | 57.8 h | +| `sdd:review` → `sdd:done` | → 06-24 22:31 | 54.3 h | + +The front half (spec through materialized tree: 7.4 h, one human decision gate on the second spike) is the design working as intended. The back 112 h are dominated by stranding dwell, review-loop convergence, and human gates. + +### Tree shape + +4 units; Unit 3 correctly collapsed to a single task (ADR 0028 — zero uncollapsed single-task units). 8 planned tasks (#495–#502) + 1 planned-then-split (the operator `/revise`-split Task 1.2 into three before `/approve`) + **3 operator-filed mid-run tasks** (#526 hardening deferrals, #542 "wire the built component into the live launch path", #550 process-signalling) + 1 human-only hardware-gated task (#535/PR #540). Spike #511 (the network-switch attach protocol) was also mid-run — the protocol was load-bearing and unspiked at plan time. + +### maps task → merged PR + +495→507, 496→525, 497→522, 498→521, 499→556, 500→546, 501→561, 502→554, 526→537, 542→547, 550→555 (fastpath). All 11 merged by the operator (mostly by arming auto-merge). Zero green agent PRs closed unmerged (the pilot-era trust-loss event did not recur); one mis-routed duplicate (#538) was human-closed. + +--- + +## 3. Scorecard vs the pre-run yardsticks + +Yardsticks are the framework's own, from its pre-run design issues (spectacles #247/#252/#254/#255/#257/#272). + +| Yardstick (pre-run standard) | Target | Measured | Verdict | +|---|---|---|---| +| Median merged sdd-PR net diff | ≈400 | **694** (79–1765; three PRs >1300) | ❌ oversize — the floors fixed the pilot's 100-line problem and overshot; no ceiling exists | +| Single-task-Unit count | 0 | 0 (Unit 3 collapsed) | ✅ | +| Over-decomposition (tasks ≫ scope) | 1–2 tasks / ~250 lines | 11 PRs / 8,265 net lines ≈ 751/PR | ✅ class did not recur | +| Human-gate share of wall-clock | <50% | **≈56%** | ❌ | +| `needs-human` position | early OK, late = failure | **17 episodes; 11 after the 50% mark**; median dwell 3.8 h, max 34.2 h, total 134.5 h | ❌ | +| Green agent PRs closed unmerged (trust loss) | 0 | 0 | ✅ | +| One fresh status comment per tracker | 1, newer than last event | **0 — surface never ran (gate bug)** | ❌ hard miss | +| Every external wait bounded | 30-min-class SLA | monitor flags fired, but recovery 0/6; strandings dwelt 12–34 h | ❌ | +| Revise grounded in PR thread | yes | ✅ assembled directive consumed threads (#247 worked) | ✅ | +| Every thread closed with reasoned verdict | yes | ❌ human performed ~80 inline dispositions; blanket resolution + re-raise loops (#311/#327 confirmed) | ❌ | +| Cost ratio trended (`aic`) | trended per feature | computed here (appendix C); nothing trends it automatically | ❌ (no automation) | + +**Score: 4 of 12.** Every miss was *detectable during the run from API state alone* — none required inference (§8). + +--- + +## 4. The demoability shortfall (the B3 question) + +The pipeline hit `sdd:done` with the feature un-runnable by a user. The remediation delta apportions responsibility precisely: + +### 4a. Specced-but-not-delivered (2,560 lines, 45%) — factory execution + +The five big items, all covered by explicit R-IDs: + +1. **the VM-side control transport (R1.4/R1.5, R2.3/R2.4)** — R1.5 explicitly requires the fd-pass attach over the VM-side channel. No pipeline diff contains it; PR #522's title promised it and its body deferred it ("split + merge on green scope," operator-approved under stranding pressure). #581 added the missing transport implementation and its attach gating. +2. **A rootless per-sandbox networking mode (R1.5/R1.6)** — the pipeline's mechanism never worked without root; #589 replaced it with a different userspace-network stack after external reviewer advice (PR still open at time of writing; verified working 07-01). +3. **Proxies built but never served (R3.3/R3.4, R4.4)** — PR #546 built a complete B5 egress proxy (+549 lines, passing tokio tests) and #554 a complete mTLS reverse proxy — **the daemon only bind-checked them; nothing called `serve()` outside tests**. R3.3's own curl proof artifact was unpassable at merge. #581's `start_host_proxies` wired them to the live registry. +4. **Delivered-but-broken enforcement (R1.2)** — a task-launch path hardcoded the permissive network mode for every task, leaking host egress to an isolated session's tasks; caught only at integration. +5. **Registry not routable (R3.3/R3.6)** — hostnames registered but no live routing; rename lifecycle broken. + +### 4b. Needed-but-unspecced (1,702 lines, 30%) — spec/plan gap the framework should have exposed early + +- **Guest boot/init integration: the guest-init source file — 295 lines in a file zero pipeline PRs touched** (mount, pty, root-switch, and daemon-egress plumbing). Every in-guest sandbox died exit-125 without it. No R-ID mentions the guest boot path; the spec's "Affected areas" never named it, so *end-to-end wiring was nobody's task* — the same gap the operator had to file #542 for mid-run. +- Daemon's own egress, demo scaffolding (empty-workspace launch), PTY fixes, and the **e2e test plan (`test-plan.md`/`test-plan.sh`, 415 lines) — the demoability artifact itself arrived in remediation**. + +### 4c. Quality/refactor (1,409 lines, 25%) + +Mostly code-motion of *working* pipeline output (moving working attach code behind a network abstraction and extracting a shared crate). This is the healthy fraction: it means the humans found the pipeline's code worth keeping and restructuring rather than rewriting. + +**Split of blame:** ~45% factory execution (R-IDs not delivered working, several knowably so — a grep for `#[ignore]`/`required-features` at validate time would have flagged 9 of 15 proofs), ~30% spec input (C1: unvalidated implementation hypotheses transcribed as settled R-IDs — see §6; integration wiring structurally unowned), ~25% ordinary engineering iteration. + +--- + +## 5. What worked (with evidence) + +1. **The front half is genuinely good.** Spec in 3.8 h with a real human gate that caught real gaps (operator `/revise` at hour 3.6 added two missing requirement areas → R3.6/R4.9). Triage produced an architecture doc, an assumption ledger, and two spikes by hour 4. +2. **The spike workflow.** #485 (name-resolution) and #486 (implementation-choice) turned around in 2.4 h/1.1 h, and the plan comment demonstrably folded both ("Spike wave drained…"). Mid-run spike #511 produced a 615-line protocol doc that unblocked the run's worst stall. This is the operator's surprise #1, and it is real — with the caveat that #486 contained a math error and wrong protocol name (human-fixed next day, PR #490), and #511 existed only because the risk wasn't spiked up front. +3. **The plan comment** (17.6 kB): units in dependency order, per-task `Depends on` edges, R-ID coverage lines, `ALREADY EXISTS` reuse anchors with `file:line`, model tiers, per-task proofs, ADR-0028 collapse note. Materialization matched it exactly (ADR 0010 held). +4. **Sizing reform held**: no over-decomposition recurrence; 1:1 task→PR; conventional commits; clean branch join keys. +5. **The assembled revise directive (#247) worked**: the surviving revise runs consumed all unresolved threads — feedback never had to be re-typed (what *didn't* work is loop closure, §6). +6. **Fastpath**: #550 → `/fastpath` → spec stub → `/approve` → PR #555 merged in ~5.5 h. The single-PR path works when scoped right. +7. **The code itself**: 75% of remediation was *not* rework of broken pipeline code; the humans kept the substance (the mesh core, policy types, proxy implementations, and the IP allocator) and mostly wired, hardened, or restructured it. +8. **sdd-monitor as a detector** flagged every stranding (6/6) with accurate audit lines and correct attempt accounting — as a *recovery* mechanism it failed completely (§6). + +--- + +## 6. What didn't work + +### 6a. `sdd:done` ≠ demoable — the contract gap +Nothing in gates, proofs, or lifecycle requires an *executed end-to-end demo*. All heavy proofs ran `#[ignore]`+env-gated in a CI lane **the human added** (PRs #508/#510), or sat behind cargo features (`networking-wg`, `networking-proxy`) that no shipped binary compiled. The empty-PR rule was honored per-PR while the *feature* stayed undemonstrable. (Operator pain #1: **confirmed**.) + +### 6b. Integration wiring was structurally nobody's job +The spec's per-unit "Affected areas" never included the launch path or guest boot; the plan planned units, not the seam between them. The operator had to notice (#499's investigation found the new mode handled identically to an existing one) and file #542 mid-run; the guest-init source was never touched at all until remediation. + +### 6c. Evidence standards inside the ledger — spec risk didn't surface early (C1) +The input spec was an unusually strong PRD fusing requirements with an implementation *hypothesis overlay* (transport, filter-API, no-root, and resolver assumptions). The pipeline transcribed hypotheses into R-IDs, and the assumption ledger marked the costliest unknowns **"settled" citing the spec itself** ("settled | spec R1.5") — circular evidence. Consequences: the two spikes went to a cheap library choice while the *actual* top risks (the component attach protocol (34 h stall, mid-run spike #511); the rootless mechanism (#589 pivot, still open); a filter-API assumption (R2.2 dead, its use case never shipped, split to #553)) went unspiked. Spike #485's own Finding A contradicted its recommendation and nothing caught it until the operator's "**Hold — do not merge**" on PR #546 forced the mid-run mechanism re-scope. Answer to C1: **yes — the framework had the right early artifacts (ledger, spikes, spec gate) and they under-verified; the operator's `/revise` was the only substantive spec review.** + +### 6d. Stranding and recovery: every recovery was a human +6 monitor escalations, 4 episode groups, **0/6 auto-recovered** (#320 confirmed): protected-path `.github/` capability gap (#495), a dispatch that produced no run (#497), a ~60 m timeout kill (#498), an engine crash from netns thrashing (#496), and **Anthropic usage-window quota exhaustion** (#499/#500 — operator-confirmed), a failure mode the framework doesn't model at all (monitor's 3 attempts fire within ~24 min against multi-hour quota windows, and only after `needs-human` already excludes the task). Net: **23 manual `/execute`**. Of the cascade's 12 automatic `/execute` fan-outs, only 1 led to a PR without further human action. + +### 6e. The review loop does not converge on its own +sdd-review re-raised identical findings across passes (same path+rule: R1.4 finding ×5 on #522; `_ => "tcp"` ×4 on #556) with no memory of dispositions (#327). sdd-execute pushed fixes but never closed threads with verdicts (#311). The operator closed everything: 5 batch "N findings validated against HEAD remain" `/revise` comments, ~80 inline dispositions (last commenter in 29/35 threads on #556), plus a deferral sink (#526→#537) whose fixes were then **silently reverted by stale-base sibling branches** (#554/#556) and re-raised — a cross-PR recurrence loop. False signals compounded it: **172 thumbs-down, 0 thumbs-up** stamped on consumed findings (#310), and `needs-human` flags on #554/#556/#561 that were stale, mis-graded, or catch-22 by the time a human looked (each hand-cleared on 06-24). CodeRabbit: 0 `CHANGES_REQUESTED` ever; it caught one HIGH security bug and one revise regression as plain comments; it was paused (`@coderabbitai ignore`) an hour before #556 merged. + +### 6f. Integrity: the highest-severity class ran unguarded +Beyond the two live incidents in §1: the **merged-change revert guard (the #287 fix) was inconclusive on every invocation** — shallow clone without credentials, so it could never see origin/main. Gate-2 treats merged-doc deletions as clearable Warnings by design; a spec deletion demotes the boundary to the lighter gate set; validator check runs always conclude success. **At no point in the run did any framework check hard-block a merge for an integrity reason.** Detection remained what it was in the pilot era: red consumer CI or a human reading net-diffs. + +### 6g. Observability inverted +The one surface built to fix "human watching" (#254 → ADR 0023) was dead-on-arrival for every consumer (the `vars.SDD_STATUS != '0'` unset-coercion bug — 10,633 runs, 0 successes, and its own skip-storm produced 1,947 no-op runs *inside this window*). Meanwhile live signals actively lied: thumbs-down on consumed findings, red "agent" checks on PR commits from concurrency-cancelled runs, `needs-human` labels outliving their cause. Gate/consumer-CI parity also failed twice exactly as #312/#318 predicted (commitlint red on #554 human-reworded; `build-macos`/`clippy` and KVM-lane failures reaching already-open PRs #507/#525). + +--- + +## 7. Priors audit (B7) + +| Prior | Verdict | Evidence | +|---|---|---| +| Pain: proven only in CI behind a feature flag | **Confirmed** | every heavy proof `#[ignore]`/env-gated in the human-added netns lane or feature-gated; `--network` CLI + test-plan arrived in #581 | +| Pain: unspiked risks → late `needs-human` | **Partially confirmed** | 11/17 episodes late, but ~5 trace to unspiked risk (the VM-side transport, launch-path wiring, R2.2, component ownership) vs ~6 to framework failures (quota, engine crash, stale labels); the single worst unspiked-risk hit (the attach protocol, 34.2 h) landed at the 25% mark | +| Surprise: spike workflow | **Partially confirmed** | fast + plan-folded; but #486 partial+erroneous (human fix #490) and #511 only existed because the risk was missed up front | +| Surprise: PR scope & size | **Scope confirmed, size refuted** | clean 1:1 task→PR; median 694 net (target ~400), three PRs >1300 carried the heaviest review loops | +| Surprise: well-defined plan | **Partially confirmed** | structurally excellent; needed an immediate human split of Task 1.2, missed integration wiring entirely, two mid-run re-scopes | + +--- + +## 8. What to improve (prioritized; each maps to a detection-gap-matrix row and open spectacles issues) + +**P0 — correctness/integrity (blocks trusting merges):** +1. **Ship the base-agnostic net-diff-vs-main guard as a required, re-run-on-push check** (spectacles #326; subsumes #317). This run added two more recurrences (#491's doc deletion — merged and never restored; #556's revert bundle). Also *fix the existing revert guard's environment* (full fetch + credentials) — a guard that always answers INCONCLUSIVE is absence with extra steps. +2. **Demoability gate**: `sdd:done` requires an *executed* end-to-end proof — a committed test-plan-style artifact whose run is a check, not prose. Add a proof-artifact *delivery* audit at validate time (deterministic: grep merged diffs for `#[ignore]`, env gates, `required-features` vs. the spec's declared proofs). New issue; extends #256/sdd-gates. +3. **Fix `SDD_STATUS` unset-coercion** (one-line wrapper fix; new spectacles issue). Every "found by human watching" failure in this run had a dead status surface behind it. + +**P1 — the two loops that consumed the operator:** +4. **Review-loop convergence contract** (#311/#327): per-thread close requires "Fixed in ``" or a rebuttal; re-review only the diff-since-last-pass; finding fingerprints deduped across pushes *and across sibling PRs of the same tracker* (the #537-revert recurrence). Fix reaction semantics on cancelled runs (#310). +5. **Recovery that recovers** (#320): model quota exhaustion as *retriable-later* (respect the window, don't burn attempts in 24 min), reset labels on timeout kills, allow monitor re-dispatch to fire while `needs-human` is present when the cause is infrastructural, and make `/execute` on a stranded task always produce either a run or a visible refusal reason. +6. **Plan must emit integration**: a mandatory per-feature "wiring/e2e demo" task (or per-unit integration criterion in Affected-areas) so the #542-class seam is owned before dispatch. + +**P2 — inputs and sizing:** +7. **Adversarial evidence standard for the assumption ledger** (C1): an entry citing only the spec cannot be "settled"; unvalidated mechanisms on the critical path must spike before materialization (the spike *workflow* is the proven asset here — point it at the right targets). +8. **Sizing ceiling** to pair with the ADR-0026 floor (median 694, max 1765; the three >1300 PRs carried the worst loops); split-at-PR-time guidance. +9. **Gate/CI parity** (#312/#318): commitlint + consumer lanes (`--locked`, feature-matrix builds) in the Pre-PR gate; both bit again this run. + +**P3 — hygiene:** stale `needs-human` auto-revalidation on each push; route rejections must leave a visible trace on the item (silent route declines cost several of the 23 manual `/execute`); quota/window telemetry on every engine failure. + +Items 1–3 are the eval-agent's first customers: every metric in this report was computed from API state after the fact; the eval agent's job is to compute them *during* the run (spec follows separately, per the engagement plan). + +--- + +## Appendix A — needs-human episodes (17) + +| # | Issue | Applied | Dwell h | Run position | Cause class | +|---|---|---|---|---|---| +| 1 | 478 | 06-20 03:33 | 1.8 | 4% | spike boundary decision (design-intended) | +| 2 | 495 | 06-20 06:34 | 12.8 | 6% | protected-path gap + quota-class engine failures | +| 3 | 478 | 06-20 08:02 | 11.3 | 8% | mirror of #495 | +| 4 | 478 | 06-20 19:43 | 5.5 | 17% | monitor re-escalation | +| 5 | 496 | 06-21 04:55 | 34.2 | 25% | unspiked the network-switch attach protocol (→ spike #511) | +| 6 | 478 | 06-21 05:51 | 33.3 | 26% | mirror (496/497/498 strand group: dispatch gap, 60 m timeout, engine crash) | +| 7 | 478 | 06-23 03:13 | 1.0 | 64% | usage-window quota | +| 8 | 526 | 06-23 03:30 | 0.6 | 64% | hardware-gated proof unverifiable + protected ci yml | +| 9 | 499 | 06-23 06:13 | 15.5 | 66% | quota, then real plan gap (R2.2 decision) | +| 10 | 500 | 06-23 06:22 | 7.8 | 67% | sdd-spec engine crash + mechanism re-scope decision | +| 11 | 478 | 06-23 06:51 | 7.3 | 67% | mirror | +| 12 | 500 | 06-23 14:32 | 0.1 | 73% | residual | +| 13 | 478 | 06-23 14:46 | 0.3 | 74% | sdd-spec failure surface | +| 14 | 542 | 06-23 15:17 | 0.7 | 74% | switch-ownership contradiction (operator decision) | +| 15 | 478 | 06-23 19:32 | 2.2 | 78% | monitor re-escalation of 499 | +| 16 | 499 | 06-23 22:44 | 0.2 | 80% | re-scope ack | +| 17 | 550 | 06-24 04:23 | — (never cleared) | 85% | fastpath residual; task done, label orphaned | + +Total dwell 134.5 h (episodes overlap); median 3.8 h. 4 episodes ≤25% mark, 11 after 50%. + +## Appendix B — per-PR review forensics + +| PR | net | auto-revise | CodeRabbit reviews | review comments | 👎 on comments | human inline comments | +|---|---|---|---|---|---|---| +| 507 | 79 | 0 | 0 | 2 | 0 | 1 | +| 521 | 253 | 1 | 1 | 6 | 5 | 0 | +| 522 | 694 | 3 | 1 | 27 | 23 | 7 | +| 525 | 1351 | 3 | 4 | 42 | 41 | 4 | +| 537 | 196 | 2 | 1 | 19 | 15 | 0 | +| 546 | 901 | 3 | 3 | 23 | 18 | 3 | +| 547 | 423 | 1 | 1 | 17 | 14 | 0 | +| 554 | 944 | 3 | 6 | 24 | 19 | 3 | +| 555 | 137 | 1 | 1 | 1 | 0 | 0 | +| 556 | 1504 | 2 | 1 | 66 | 50 | 2 | +| 561 | 1765 | 2 | 8 | 28 | 21 | 1 | + +Gate-passed-then-CI-failed: #507 (`build-macos`, `clippy`), #525 (KVM e2e lane); commitlint bit #554 (#312 class). Red `sdd-execute-*/agent` checks on commits of 521/522/525/547/555/556 are concurrency-cancel/crash noise — misleading signal (#310 class). + +## Appendix C — cost + +Attributable to the #478 tree, June 19–24 (lower bound — artifacts for a few runs, e.g. task #500's spec-crash era, had already expired): + +| Workflow | Runs (with artifacts) | Effective tokens | +|---|---|---| +| sdd-execute-sonnet | 25 | 24.5M | +| sdd-execute-opus | 16 | 16.4M | +| sdd-review | 14 | 13.0M | +| sdd-validate | 20 | 11.0M | +| sdd-spec | 9 | 10.4M | +| sdd-triage | 5 | 5.9M | +| sdd-execute-haiku | 1 | 0.08M | +| **Total** | **90** | **81.2M** | + +Composition: 73.6M cache-read + 6.6M cache-write + 0.94M output + 44k fresh input. Waste share: 10 non-success runs carrying 12.1M effective (~15%) — crashes, concurrency cancels, quota kills. Per net line kept (8,265): ≈9.8k effective tokens (≈114 output tokens). The evaluation/review half (review+validate = 24.0M) cost more than either execute tier individually — the convergence loop is a first-class cost center, consistent with its role as the run's throughput bottleneck. + +Attribution: agent artifacts (`agent_usage.json` for pre-06-19 runs; the engine result record in `agent-stdio.log` post-migration), joined run→tree via prompt targets. "Effective tokens" = input + output + cache-read + cache-write (raw traffic; the AWF 25M rail weights cache reads lower). Baselines for comparison: the pilot-era dead triage run alone burned ~25M effective (spectacles #271); pilot PRs were 70–98 net lines. + +## Appendix D — method + +Evidence bundle: full issue-tree timelines (label events with actors), all comments incl. minimized flags (GraphQL), all spec/arch/sdd + remediation PRs with reviews/threads/diffs/check-runs, 16k workflow runs June 15–24, and per-run agent artifacts, harvested via `gh` API. Deterministic metrics computed by script; hunk classification of #581/#589 and R-ID delivery audit performed per-file against the spec and every pipeline diff, cross-checked against issue/PR comment streams. Numbers marked ≈ are bounded by artifact retention and run-list pagination limits. + +## Addendum (2026-07-02): spec transcription drift — the generated spec contradicted its source's topology + +Operator-supplied evidence from remediation PR #589 (a spec-amending commit +inside the remediation), strengthening §6c: + +The spec-generation step did not just transcribe unvalidated hypotheses — +it **inverted the source document's topology**. The source is explicit that +the daemon is *not* on the internal switch and that host→sandbox access +goes through published loopback ports; the generated spec wrote, as +normative requirements (R3.1/R3.3/R4.4 and a proof artifact), that +hostnames resolve to switch addresses reached *via the switch* — routes the +daemon could never take under the source's own deployment topology. + +Nothing in the pipeline caught it: the ledger was built *on top of* the +generated spec; validate/review checked implementations against the drifted +R-IDs, so they validated conformance to a contradiction; the human spec-PR +review at hour 3.6 added missing scope but did not cross-check the source's +diagrams. The contradiction surfaced during human remediation ~9 days after +the spec merged, and the fix was an 18/−11 amendment **to the spec itself** +inside the remediation PR. + +Two consequences for the analysis above. First, part of the §4a +"specced-but-not-delivered" ledger shifts: some remediated routing was a +faithful implementation of drifted requirements — a spec-input defect, not +factory execution. Second, the C1 verdict gains a class: **generation-time +source infidelity**, to which every downstream stage is blind by +construction because each treats the generated spec as ground truth. The +eval-agent rubric gains a judge item for it (RB-input-3: generated +requirements cross-checked against the source document at the spec +boundary, where a correction is cheap and everything downstream inherits +it). + +### Root cause of the drift (investigated 2026-07-02) + +Traced through the spec-generation run's full transcript, git pickaxe on the +spec file, and the source document. Four stages: + +1. **Origin — generation wrote *both* models, three lines apart.** The + initial generated spec was internally contradictory from birth: one + requirement resolved local hostnames to an internal-switch address + (wrong), while the adjacent requirement correctly said requests route + through the published loopback port. The transcript proves the agent read + the full source in one pass (retrieval was not the cause), but its + visible reasoning modeled only the *remote* mesh case — where + switch-address reachability is correct — and the name-resolution + *mechanism* question; the local host→sandbox path is never engaged + ("published", "not on the switch" appear zero times in its thinking). + The source set the trap: its requirements-level use-case diagram reads + "DNS resolves → sandbox (own IP via the switch)", while the correction + (daemon off-switch; access via published loopback ports) lives ~200 + lines later in an implementation overlay, with the tension never + flagged. The generator sampled a different topology model per + requirement and ran no reconciliation pass over its own output. + **Root cause at origin: a dual-model source × a single-pass generator + with no cross-requirement or source-topology consistency check.** Not + hallucination — both sentences have direct source antecedents; the + missing piece is the join. +2. **Survival — no stage checks spec semantics.** The contradiction passed + the structural spec gates (units, R-ID grammar, proof presence), the + hour-3.6 human `/revise` (scope-focused), the architecture/ledger stage + (which cites the generated spec as evidence, institutionalizing it as + ground truth), and the third-party reviewer (zero reviews on the + docs-only spec PR). +3. **Spread — execution harmonized toward the wrong pole.** During the + operator-directed Unit-3 mechanism re-scope, the execute agent rewrote + the correct requirement **and deleted the only correct sentence**, + aligning it with the wrong one — predictable, since the wrong + requirement was the normative-looking anchor ("shall resolve to…") and + the correct phrase was welded to the mechanism sentence being replaced. + The operator's directives specified mechanism and naming, not routing + target, so nothing in the loop contested it. The vector — **an + implementation PR editing a merged spec** — has no guard analogous to + the plan-fidelity guarantee of ADR 0010. +4. **Correction — only when routing became load-bearing**, 11 days after + origin, as a spec amendment inside the remediation PR. + +What would have caught it, and when: + +| Check | Stage caught | Cost | +|---|---|---| +| Internal-consistency pass on the generated spec (requirement-vs-requirement contradiction) | origin, pre-PR | needs no source document at all | +| Source-fidelity judge (RB-input-3) over diagrams/topology sections | origin, spec boundary | one judge pass | +| Spec-edit tripwire: implementation PR touching a merged `docs/specs/**` file re-triggers both checks | spread, deterministic | one changed-files join | + +Encoded in the eval-agent spec (collector R1.4 spec-edit tripwire) and +rubric (RB-input-3, extended). The general lesson: the pipeline has a +fidelity guarantee plan→tree (ADR 0010) but none source→spec or +spec→spec-edit, and every downstream stage compounds whichever model the +spec asserts. diff --git a/docs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.md b/docs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.md new file mode 100644 index 0000000..ff25d8e --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/evidence/_run-window-evidence.md @@ -0,0 +1,201 @@ +# Retro Evidence Backbone — SDD Agent-Factory Run (June 2026, consumer tracking issue #478, networking feature, full-pipeline path, v0.3.0-era) + +All issue numbers refer to the spectacles repository unless noted. The pilot target is referred to only as "the consumer repo." Data reflects state as of 2026-07-01. + +--- + +## 1. Run-window failure timeline (June 15–24) + +Chronological by `created_at`. One line each: date, issue, stage, what happened, consumer impact. + +- **2026-06-15 · #271 · triage** — sdd-triage phase-A run died on the AWF hard rail: "429 Maximum effective tokens exceeded (25096706.10 / 25000000)"; 2.5M input tokens over 33 calls, per-call context 33K→122K, mid-run cache break, zero safe output (`{"items":[]}`). Impact: hard stall of the feature's triage plus ~25M effective tokens wasted. +- **2026-06-15 · #272 · spec/triage** — over-decomposition persisted a layer up from #252: "Demoable units land at ~100 net lines when an engineer's natural PR is ~500"; "Sub-issue count = 2 + N + M," each task paying the full per-task pipeline cost. Impact: structurally inflated token/CI/review cost per net line shipped. Still open (Lever 4). +- **2026-06-18 · #283 · installer/infra** — after the Copilot→Claude engine migration, all 10 agentic wrappers still passed `COPILOT_GITHUB_TOKEN`; undeclared secret to a reusable workflow is a hard `startup_failure` ("`sdd-spec`, `sdd-triage`, `sdd-validate` all `startup_failure` in ~1s"). Impact: total pipeline stall on every main-pinned install; pure-actions wrappers masked the breadth. +- **2026-06-18 · #285 · spec/MCP infra** — unconfigured `DISTILLERY_MCP_URL` resolved to empty string; gateway schema validation exited 1 before the agent ran ("the whole pipeline is dead at `sdd-spec`"). Impact: hard stall at sdd-spec for any consumer without Distillery; first fix attempt had to be retracted (compiler reuses the URL expression in the firewall allowDomains). +- **2026-06-18 · #287 · execute** — **safety-critical**: SDD PR #463 on the consumer repo (branch sdd/453-net-module — the networking module) regenerated connection.rs from a stale base and silently reverted merged fix #449, reinstating `.await.unwrap()` where `?`-propagation had been added; "In the guest, `` is pid-1, so a panic here kills init and takes down the whole VM." Impact: red CI, manual restore (ab7eca9a); a pid-1 VM-killer had CI not caught it. +- **2026-06-18 · #289 · dispatch** — fast-path /approve never dispatched: route job lacked `pull-requests:read`, the 403 was swallowed, the job stayed green, and "the spec PR merges but the implementation is never dispatched." Impact: fast path dead for every consumer; expensive diagnosis because the failing job succeeded. +- **2026-06-18 · #292 · dispatch** — next 403 in the same chain: fast-path dispatch used `workflow_dispatch` (needs App `actions:write`, absent from documented install; contradicts ADR-0014). Impact: fast path still dead on any documented install; docs-only fix leaves the ADR-0014 contradiction latent. +- **2026-06-18 · #294 · process/UX** — no acknowledgment between a slash command and first agent output: "They re-comment, assume it failed, or wait blind." Impact: duplicate commands and blind waiting on every command round-trip (enhancement, fixed with deterministic ack reactions). +- **2026-06-18 · #295 · triage** — "Confirmed 15/15 startup_failure on a staging consumer": plan/materialize wrapper jobs granted `pull-requests: read` while the called locks' nested jobs need `write`; workflow rejected at load, no logs/jobs. Impact: full path dead at spec→triage; declared v0.3.0 release blocker. +- **2026-06-18 · #298 · spec/derive/spike-reentry** — two more instances of the #295 class: spike-reentry "fired startup_failure on every issue close/unlabel — 20/20 on a staging consumer"; "/derive-spec was dead." Impact: silent dead surfaces plus failure noise; contract gate blind spot fixed (fa7df42). +- **2026-06-19 · #300 · triage** — App-applied `sdd:triage` never activated the agent (no `roles: all`; gate saw actor `spectacles-bot[bot]`, permission "none"); "run reports success — a silent no-op." Impact: full path silently stalled at spec→triage; human re-label required at every transition. +- **2026-06-19 · #301 · dispatch** — close-event cascade hardcoded a 2-hop walk vs ADR-0028's collapsed 1-hop trees: "Closed issue #36 is not a task sub-issue closure (walked 1 hops); ignoring." Impact: full-path E2E tracker (consumer/staging #29) sat with all work merged but not `sdd:done` until a manual /dispatch. +- **2026-06-19 · #303 · execute/dispatch** — haiku-tier fast-path execute concluded "success" with `agent_output.json = {"items":[]}`; per transcript it decided nobody had asked it to do anything and "replied 'What would you like to help with?'" Impact: fast-path implementation no-oped, issue stranded at `sdd:in-progress`, full run's tokens wasted; tier floored at sonnet. +- **2026-06-22 · #308 · execute (concurrency)** — re-triggered /execute run cancelled at activation by an unrelated same-tier task's PR events: "route → success / ack → success / activation → cancelled"; "touching one task's PR can kill another task's freshly-dispatched run." Impact: silent stranded task, no auto-recovery, manual re-bump. +- **2026-06-22 · #310 · execute/status** — concurrency-cancelled revise runs stamped thumbs-down on consumed review comments: "nearly every review comment on a PR… carried a [thumbs-down] reaction, reading as 'these findings failed'" while the PR progressed normally. Impact: misleading signal that also camouflages real failures. Open. +- **2026-06-22 · #311 · execute/review** — revise closes no loops: the resolver "blanket-resolves the App bot's own unresolved threads… resolves even findings that were never addressed — silently hiding them"; worked example: 7 findings needed manual triage (3 valid, 4 already fixed), red for ~an hour until manual takeover. Impact: throughput bottleneck absorbed by the maintainer on every review-fix PR of the cascade. Open. +- **2026-06-22 · #312 · execute** — Pre-PR gate omits consumer commitlint: "A 104-char commit header (`header-max-length` = 100)" and a subject-case violation each "left the PR `BLOCKED` on `commitlint` until a human reworded the offending commit (a force-push history rewrite)." Impact: merge-blocking rework, twice in one pilot run. Open. +- **2026-06-22 · #313 · dispatch/review** — human /revise silently gated behind `needs-human`: "No execute dispatches. The `/revise` sits silently behind the label" — despite validate reporting no blockers. Impact: every post-hand-off revise needed a manual label-clear with zero feedback. Open. +- **2026-06-22 · #314 · validate** — validate attempted egress-dependent builds in a sandbox with "no toolchain and no registry egress," classifying firewall blocks as blockers: the recurring "PR keeps getting `needs-human`, clear it, it comes back on the next push" loop. Impact: label churn and stalls with no real defect. Open. +- **2026-06-22 · #315 · distillery/status** — every scheduled distillery-sync run failed: Haiku read its own prompt as a spec, "`num_turns=1`, zero tool calls," asked a clarifying question headless; the status issue silently went stale (last updated 06-18, failures from 06-20). Impact: knowledge store stopped syncing; failure hidden from the status surface. +- **2026-06-22 · #317 · spec** — **safety-critical near-miss**: an sdd-spec derive PR "changed +3125 / -2665 across ~30 code files in addition to the spec document," gutting merged work (composer.rs -2188, RESOLUTION.md -194) behind an innocuous spec-derivation title. Impact: human force-push recovery; would have silently reverted merged code. Open. +- **2026-06-23 · #318 · execute** — Pre-PR gate ran cargo without `--locked`; a Cargo.toml dep change without lock regen shipped a PR that failed the consumer's `--locked` lane ("error: cannot update the lock file ... because --locked was passed" exit 101). Impact: already-red PR, misleading failure signature, human rework. +- **2026-06-23 · #320 · dispatch/execute/monitor** — "Two next-layer tasks both hit the 60m timeout, stayed `sdd:in-progress` with no PR, and could not be re-dispatched — a manual `/execute` no-op'd"; monitor recovery "just burns its 3 attempts and escalates to `needs-human` — a dead end." Impact: hard stall of two networking-feature tasks; operator hand-edited labels. Open. +- **2026-06-23 · #324 · distillery/status** — duplicate status issue plus a numberless `update_issue` ("Target is \"*\" but no item_number/issue_number specified") marked otherwise-successful sync runs red intermittently. Impact: false-red runs eroding trust in run status; human dedupe required. Open. +- **2026-06-24 · #326 · execute/validate/review** — **safety-critical class**: stale-base SDD PRs silently deleted/reverted out-of-scope merged work, "observed three times on the consumer pilot" (deleted another feature's merged spec doc; reverted a merged crate-split refactor; re-introduced a fixed bug). "Each was caught only because a human read the net-diff and hand-restored the file. The framework did not block the merge." Open. +- **2026-06-24 · #327 · review/validate** — full base..head re-scan on every push: "the same already-fixed findings re-surfaced run after run (re-resolved by hand each time)"; auto-revise churned PR heads, compounding CI/token waste; only stable state is "0 unresolved threads + green CI." Open. +- **2026-06-24 · #328 · spec (adoption)** — derive-spec offer copy reads as a callout ("you skipped a step") on human contributors' PRs on the consumer repo. Impact: social/adoption cost, no functional breakage. Open (fix in-flight, d9e6fc6). + +Filed in-window but not run failures: #275 (sdd-derive coverage-gap enhancement, 06-17), #307 (bot-filed spec-debt surface, 06-22 — the ADR 0027 backstop working, ~4,400 unspecced lines in the pipeline's own repo), #322 (premise outdated; v0.3.0 already correct, 06-23). + +--- + +## 2. Failure taxonomy + +All 39 issues, grouped by category. + +### permission-contract — #259, #269, #289, #292, #295, #298, #300 + +**Pattern:** a mismatch between what a job/token/actor is granted and what the code it runs actually needs — made invisible by error-swallowing (`#259`, `#289`: 403 → default-deny in a green job), by load-time rejection with no logs (`#295`, `#298`: caller job permissions < callee lock nested-job permissions), or by activation-gate semantics (`#300`: App actor fails `roles` check, run reports success). #292 is ADR drift (fast-path never migrated off `workflow_dispatch` per ADR-0014); #269 is the config-side sibling (consumer var silently re-models an agent). +**Status: fixed as instances; class partially guarded.** All seven closed (#269 not_planned — superseded by the Claude engine port, 6a0be81). The wrapper-lock contract gate now enforces secrets (#284-class) and caller>=callee permissions (fa7df42), but #292 was fixed docs-only — "the code still uses workflow_dispatch, so the ADR-0014 contradiction remains latent" — and no static API-call→scope lint exists yet. + +### safety/code-integrity — #287, #317, #326 + +**Pattern:** an agent writes from a base that trails origin/main (stale checkout/regeneration or `git add -A` scope capture) and the resulting PR silently deletes or reverts merged work; GitHub's 3-way merge raises no conflict, the pipeline raises no flag, and the bot's own description masks the damage. +**Status: OPEN as a class.** #287 fixed with a point guard in the execute Pre-PR gate (PR #288); #317 (spec-side variant) and #326 (the generalized net-diff guard, three recurrences post-#287) are open. #326 states it directly: the prior fix "patched one instance without generalizing." + +### state-machine/lifecycle — #257, #301, #313, #320 + +**Pattern:** the pipeline's event-driven state machine has holes — waits with no timeout (#257 CodeRabbit never reviews), transitions whose predicates encode stale structural invariants (#301's hardcoded `walked === 2` vs ADR-0028), labels that gate the very input they request (#313 human /revise behind `needs-human`), and optimistic labels with no failure-path reset (#320's permanent `sdd:in-progress` strand). +**Status: partially fixed.** #257 and #301 closed; #313 and #320 open — the two open ones are precisely the recovery paths (post-hand-off and post-crash). + +### infra/config — #269, #283, #285, #314, #318 + +**Pattern:** configuration surfaces treated as consistent when they were not: incomplete migrations (#283 wrappers vs locks), optional consumer vars interpolated into schema-validated config (#285), consumer-overridable model vars (#269), gate commands not mirroring consumer CI flags (#318 `--locked`), and sandbox constraints not encoded in classifiers (#314). +**Status: partially fixed.** #283, #285, #318 fixed; #269 superseded; #314 open. + +### prompt/model-behavior — #247, #258, #303, #315 + +**Pattern:** the compiled prompt and the model diverge: missing context assembly (#247 revise ignores PR threads), internally contradictory fragments where "the loudest stack-specific instruction won over the generic gate" (#258), and descriptive/procedural prompts that weaker tiers do not execute — haiku treating its directive as "all system context" and asking a question in a headless run (#303, #315). +**Status: fixed per instance; class recurring.** All four closed, but the haiku-fragility shape hit twice in five days (#303 → tier floored at sonnet; #315 → imperative reframe), with no compile-time contradiction lint or tier-capability gate yet. + +### cost/context-efficiency — #252, #271, #327 + +**Pattern:** fixed pipeline overhead and unbounded context dominate delivered work: per-task PR overhead exceeding the diff (#252, "~25M tokens" for one triage), a mega-prompt loading all phases/tools every run until the 25M rail (#271), and stateless full re-review on every push (#327). +**Status: partially fixed.** #252 (ADR 0022) and #271 (invocation cap #270, prefetch #278, per-phase split #279/ADR 0029) fixed; #327 open. + +### observability-gap — #254, #294, #310, #324 + +**Pattern:** the pipeline either emits no signal (no status surface #254, no command ack #294) or a wrong signal (#310 thumbs-down on consumed findings, #324 green work marked red by status bookkeeping). #254's framing generalizes: the gap "amplified every other failure class." +**Status: partially fixed.** #254 and #294 shipped; #310 and #324 open — both are false-signal defects in the very surfaces #254/#294 added. + +### process-design — #255, #256, #272, #275, #307, #311, #312, #322, #328 + +**Pattern:** the pipeline's process model diverges from how the work actually flows: ceremony exceeding value (#255's 4 gates/2 days → operator abandonment), verification framework-guessed instead of repo-defined (#256), sizing floors absent (#272), forward-only spec coverage (#275, #307), merge-unblock sweeps instead of reasoned thread resolution (#311), commit policy unenforced in-loop (#312), wrong-tool fit (#322, premise outdated), compliance-toned contributor copy (#328). +**Status: partially fixed.** #255, #256, #275, #322 closed; #272 (Levers 1–3 shipped, Lever 4 open), #307, #311, #312, #328 open. + +### concurrency/race — #308 + +**Pattern:** the execute wrapper's concurrency group was not strictly per-item across trigger types, so cancel-in-progress let cross-task same-tier events kill unrelated runs at activation. +**Status: fixed** (d641900, per-item keying) — but its downstream interaction effects remain open (#310's false thumbs-down, #311's cancelled-mid-revise churn). + +--- + +## 3. What the run cost in framework fixes + +**Issues filed during the run window (2026-06-15 → 2026-06-24): 30 of the 39** (#271, #272, #275, #283, #285, #287, #289, #292, #294, #295, #298, #300, #301, #303, #307, #308, #310, #311, #312, #313, #314, #315, #317, #318, #320, #322, #324, #326, #327, #328). + +**Severity distribution (impact-based; only #287 carries an explicit `severity:high` label):** + +- Safety/code-integrity (silent reversion of merged work): **3** — #287, #317, #326 (#326 alone records three live recurrences) +- Hard stall / pipeline-or-path-dead: **13** — #271, #283, #285, #289, #292, #295, #298, #300, #301, #303, #308, #313, #320 +- Degraded operation / rework / false signals: **9** — #272, #310, #311, #312, #314, #315, #318, #324, #327 +- Non-failure (enhancement, preventive, backstop-working, outdated premise, tone): **5** — #275, #294, #307, #322, #328 + +**startup_failure / permission-contract class:** 6 in-window issues — #283 (wrapper/lock secret contract; all 10 agentic wrappers dead on main-pinned installs), #289, #292, #295 ("15/15 startup_failure"), #298 ("20/20"), #300. Three of these (#283, #295, #298) are literal GitHub `startup_failure` at workflow load; the whole family reached main because the contract gate initially checked neither secrets nor caller>=callee permissions. Note the diagnostic tax of the sequential-403 chain: #289's fix exposed #292, whose fix exposed #295, whose class sweep found #298, and full App-driven operation then exposed #300 — five contract failures peeled one at a time across 06-18/06-19. + +**Silent-failure class (zero operator-visible signal at failure time): 11** — #287 (pipeline raised no flag; only consumer CI went red), #289 (swallowed 403 in a green job), #292 (merged spec PR, then nothing), #300 ("run reports success — a silent no-op"), #301 (route decline only in logs), #303 ("success" with empty output), #308 (stranded, "no run, no PR, no `needs-human`, no `[aw]`"), #313 ("total silence"), #317 (masked by innocuous PR title), #320 (no reset, no escalation that works), #326 (green checks on a revert). #315 aggravates the class: the status surface itself went silently stale while runs failed. + +**Still open as of 2026-07-01: 13 of 30 window issues** (43%) — #272, #307, #310, #311, #312, #313, #314, #317, #320, #324, #326, #327, #328. These are all 39 issues' open set; nothing pre-window remains open. Notably, **2 of the 3 safety-critical issues are open** (#317, #326), and the open set clusters in the revise/review convergence loop (#310, #311, #313, #327) and lifecycle recovery (#320). + +**Fix throughput:** roughly 20 framework fix PRs/commits landed in or immediately after the window against these issues (#270, #273, #276 + follow-ups, #278, #279, #284, #286, #288, #290, #293, #296, #297 + a0c2ee7, #299 incl. contract-gate fa7df42, #302, #304, #305, #309, #316, #323/d395045, 4e3559a), plus four ADRs (0026–0029). Direct waste on the consumer/staging side includes one ~25M-token dead triage run (#271), a wasted dispatch+execute run (#303), every scheduled distillery-sync run from 06-20 (#315), 15/15 + 20/20 red staging runs (#295, #298), and sustained maintainer manual triage across the review cascade (#311, #327). + +--- + +## 4. Safety-critical incidents + +These three dominate a correctness-first evaluation: the failure mode is not a stall but the agent **destroying merged work with green-looking artifacts**. + +### #287 — SDD pipeline silently reverted merged fix #449 (guest pid-1 panic-safety) in PR #463 — *closed, severity:high* + +- **What happened:** SDD-generated consumer PR #463 (branch `sdd/453-net-module` — the networking module itself) regenerated `crates//src/connection.rs` from a base that did not carry merged fix #449, reverting `Connection::from_socket` from `Result<…, ConnectionError>` + `.await?` back to a bare tuple + `.await.unwrap()`. "In the guest, `` is pid-1, so a panic here kills init and takes down the whole VM." The deleted code included the explanatory safety comment. "Nothing in the pipeline flagged that the generated diff reverted hunks from an already-merged commit." +- **Root cause:** regeneration from a working base trailing origin/main, with no guard diffing against origin/main — "the diff looked clean against the stale base." +- **Impact & recovery:** red CI (build/clippy/test) on consumer PR #463; human restore commit ab7eca9a. Had CI not caught it: a client that connects then drops kills init and the whole VM. +- **Fix:** PR #288 — merged-change revert guard in the execute Pre-PR gate that "blocks the PR when the branch is behind origin/main AND touches a file origin/main has independently advanced — the exact silent-revert signature"; remedy is rebase (silent revert → visible conflict) or needs-human. +- **Detection was luck:** red consumer CI plus a human diffing against origin/main instead of the stale base. + +### #317 — sdd-spec opened a non-docs-only PR: derive commit reverted merged files and added a code refactor — *OPEN* + +- **What happened:** a spec-derivation PR's single commit "changed **+3125 / -2665 across ~30 code files** in addition to the spec document": "`crates//src/core/compose.rs` **+1758** (new code) … `crates//src/client/composer.rs` **-2188** (gutted) … `crates//docs/RESOLUTION.md` **-194** (deleted)," plus Cargo churn and file moves. The bot's title/report described only a spec derivation, masking the corruption. +- **Root cause:** the derive commit captured the full working-tree diff (effectively `git add -A`) instead of only `docs/specs/**`; where base was ahead, divergence inverted into reverts. No docs-only assertion existed before publishing. Explicitly the spec-side analogue of #287. +- **Impact & recovery:** "Recovery required a human to force-push the branch back to `base + spec-doc-only` (the bot reported it as deriving the spec from the source issue, so the corruption was not obvious from the title)." Near-miss: merge would have silently reverted recently-merged code across ~30 files. +- **Fix state:** open. Proposed: branch from current default HEAD, stage only `docs/specs/**`, hard docs-only diff assertion before publishing. + +### #326 — Net-diff guard: stale-base PRs silently delete/revert out-of-scope merged work — *OPEN; the generalization the class needs* + +- **What happened:** three separate live recurrences on the consumer repo: "a PR deleted a merged spec document for a **different** feature (its branch predated that spec's merge)"; a PR reverted a merged crate-split refactor; a PR re-introduced a previously fixed bug by regenerating a file from a base lacking the fix. GitHub's 3-way merge honors the branch's explicit deletion "with no conflict and no signal." +- **Root cause:** no base-agnostic net-diff-vs-current-main guard. "#287 patched one instance without generalizing, #317 covers only the sdd-spec derive variant, and `sdd-validate` gate 2 **does detect** out-of-scope deletions — but emits a **Warning that escalates to `needs-human`**, not a blocking check, and never restores." +- **Impact:** "Each was caught only because a human read the net-diff and hand-restored the file. The framework did not block the merge." The issue calls this "the highest-severity class in the run: merged features/fixes would be erased with green checks." +- **Meta-finding for the retro:** #326's own eval signal notes that "an eval agent should flag when a closed 'fixed' issue (#287) shares a failure fingerprint with new incidents, indicating a point fix where a guard was needed." The class recurred three times *after* #287 was closed. + +**Correctness-first summary:** the integrity class was detected exclusively by red consumer CI or a human reading net-diffs; the pipeline itself never blocked one of these merges. The fully deterministic guard (`git diff origin/main...HEAD` deletions/reverts joined against declared files-in-scope, as a required status check) remains unshipped. + +--- + +## 5. Detection-gap matrix + +Consolidated from the `eval_agent_signal` fields. "Caught today by" reflects the recorded `detection_path` during the run. + +| Failure class (issues) | Caught today by | Deterministic signal that would catch it | Inference signal (if needed) | +|---|---|---|---| +| Silent revert / stale-base integrity loss (#287, #317, #326) | Red consumer CI + human net-diff read; nothing in-pipeline | Merge-base-behind + origin/main-advanced-file check pre-PR; `git diff origin/main...HEAD` deletions/reverts joined vs declared files-in-scope, as a required check re-run on every push; docs-only changed-files assertion for spec-agent PRs; >500-line/>3-file tripwire on "spec" PRs; recurrence-fingerprint match against closed "fixed" issues | LLM judge on PR-diff-vs-origin/main: "does this PR delete or undo code introduced by merged commits?" (deleted safety comment + `?`→`.unwrap()` is a high-precision tell) | +| Wrapper/lock contract breaks → startup_failure (#283, #295, #298) | #89 E2E harness / red staging runs (no lint pre-merge) | Static CI lint: wrapper `secrets:` ⊆ lock's `workflow_call.secrets` ("a wrapper passing an undeclared secret is always a startup failure"); caller job permissions >= per-scope max over callee nested jobs (now fa7df42); runtime: any `startup_failure` conclusion / ~1s zero-job run on sdd-*, especially fan-wide across agentic wrappers while pure-actions ones pass | None needed — class is fully enumerable from YAML | +| Swallowed 403 / token-scope mismatch (#259, #289, #292, #300) | Human staging repro + log reading; failing jobs were green | Static map of every `github.rest.*` call in composite actions → required scope ⊆ job `permissions`; docs-as-contract lint vs `docs/sdd/install.md` (also flags ADR-0014 drift mechanically); log scan for "Resource not accessible by integration" inside jobs that concluded success; stranded-label joins (`sdd:approved` + merged spec PR, no `sdd:in-progress` in N min); run-shape: success with agent/safe_outputs all skipped after `pre_activation` (#300) | Flag fix PRs adopting an option the issue labeled "not recommended" against an ADR (#292) | +| Lifecycle stalls / stranded states (#257, #301, #308, #313, #320) | Human watching the run, every time | Bounded-wait invariant on every external wait (reviewer bot, CI, human gate) — e.g. 30-min CodeRabbit SLA; `sdd:in-progress` + no open PR + no queued/running execute run beyond 60m+slack; route-decline log marker "walked 1 hops; ignoring" on `## Task` bodies; run shape route=success/ack=success/activation=cancelled with no agent job; human /revise while `needs-human` with no keyed run in N min (actor-type discrimination); monitor /dispatch comments yielding zero run starts | None needed — pure API/timeline joins | +| Model-tier no-op / prompt-framing failure (#303, #315) | Human reading transcripts after red or hollow runs | Run conclusion "success" + `agent_output.json` items==[] on a run whose context promises an implementation; `num_turns == 1` and `tool_calls == 0` on any headless run; per-tier safe-output emission rate telemetry; status-issue `updated_at` older than last scheduled run start | LLM judge (or regex for "Please clarify"/trailing "?") over the final message of non-interactive runs; "treated the directive as system context" single-turn shape | +| Context/token burn (#252, #271, #327) | Human reading token bills; 429 hard rail post-hoc | Budget-fraction alerts (e.g. 50% of 25M); monotonic per-call context growth (33K→122K) with no plateau; cache_read reset mid-run; duplicate identical reads (same issue ≥3x); loaded-vs-called MCP ratio (Serena: 0 calls); calls-completed vs safe-output-buffer; per-PR review token/duration growing linearly with pushes while diff-since-last-review stays small; cost-per-net-line ("aic") trend | LLM comparison of consecutive review outputs for semantic finding duplication | +| Gate/consumer-CI parity gaps (#256, #258, #312, #318) | Red consumer CI after the PR was already open | Gate-pass-then-first-CI-fail rate join per stack; transcript must show verify.sh / package-manager / toolchain invocations before PR creation; flag Cargo.toml (manifest) diff without matching lock hunk, or run `cargo fetch --locked` as a check; flag-parity linter (gate missing `--locked` / `--frozen-lockfile` the consumer enforces); pre-push commitlint of bot commit subjects against discovered config; "bot push → commitlint failure → human force-push" timeline shape | LLM lint over compiled locks for contradictory directives (generic gate vs stack fragment) plus fact-check of sandbox-capability claims against the firewall allowlist (#258) | +| Missing revise context / non-converging review loop (#247, #311, #327) | Human re-transcribing feedback / manual per-thread triage | Revise runs with zero comment/review fetch calls while unresolved threads > 0; GraphQL thread audit: bot-resolved threads whose last comment is not a bot reply with a sha or rebuttal ("resolved-without-reasoned-reply > 0 means findings are being buried"); unresolved third-party threads with zero bot replies over N cycles; finding-fingerprint recurrence after resolution with no touch to the anchored region; needs-human oscillation > K per PR | LLM judge: does the revise diff address the unresolved comments; per resolved thread, does the closing reply actually validate or rebut ("Fixed in \" contract) | +| False/missing operator signals (#254, #294, #310, #324) | Human field feedback / human noticing red bookkeeping | Time-to-first-ack per routed command (+ near-duplicate command pairs = retrying blind); reaction-vs-run join: thumbs-down backed by conclusion "cancelled" is a false failure signal; burst shape (≥2 thumbs-down same minute + one surviving revise run); exactly-one open "[distillery-sync] Status" issue invariant; agent/detection green + safe_outputs failed with "no item_number" signature; schema lint: every update-issue carries explicit issue_number; status-comment freshness vs last pipeline event | LLM judge of a PR's reaction trail vs run outcomes / thread state for signal-reality divergence | +| Over-decomposition / process ceremony (#252, #255, #272) | Human watching PR-to-overhead ratio; operator abandonment *was* the signal | Median task-PR net diff vs floors (100 vs 300/400); task count > ceil(total_diff / SDD_TRIAGE_MIN_TASK); sibling small PRs on same file set; single-task-Unit fraction via sub-issue tree walk; green sdd/ PRs closed unmerged by a human (+ superseding human PR) = abandonment; gate-latency ratio (>50% wall-clock in human-gate waits); triage token threshold (>10M for a sub-500-line feature) | LLM/static dead-code judge per task PR: new exports with zero in-repo call sites and no consumer wired in the same PR | +| Config interpolation / model drift (#269, #285) | E2E harness; #248 incident review | Lint: every `${{ vars.* }}` in an MCP url paired with an installer default or compile guard; gateway-schema log signature ("is not valid 'uri'") before agent step = config death, not agent failure; corrupted allowDomains grep (`${{` + truncated scheme); resolved-model-per-run (OTLP) diffed against a committed per-agent manifest; per-agent token-cost step-change alert | None needed | +| Validate false blockers (#314) | Human correlating label flaps with firewall-blocked build logs | Egress-denial log markers co-occurring with needs-human safe-output in the same validate run (any co-occurrence = policy violation); needs-human applied→removed→re-applied across pushes with no new findings; identical inputs yielding "No Blockers" vs needs-human flags the nondeterminism | None needed | +| Spec transcription drift (generated spec contradicts its source or itself; found post-window in the evaluated run — see `_retro-report.md` root-cause addendum) | Human remediation, 11 days after origin; every in-pipeline stage validates against the generated spec and is blind by construction | Spec-edit tripwire: any implementation PR whose diff touches a merged `docs/specs/**` spec file (the drift-spread vector — an execute agent deleted the spec's only correct sentence while re-scoping); on it, re-run the fidelity checks | LLM judge at the spec boundary: (a) requirement-vs-requirement contradiction scan of the generated spec (needs no source), (b) each normative requirement vs the source document's diagrams/topology sections (the drifted requirement inverted a topology stated ~200 lines from the diagram that seeded it) | +| Spec coverage / adoption (#275, #307, #328) | Automated (#307 — sdd-derive's own scan; the one machine-detected class) / human tone judgment | Lineage audit: merged non-bot PR ≥ SDD_SPEC_MIN_UNIT (400) without sdd/ lineage or spec reference; offer-comment idempotency (exactly one + needs-spec marker); staleness: open batch issue with unchecked items and zero /derive-spec events after N days; unspecced-lines ratio per release window (~3354-line outlier threshold); post-sync idempotency/doctype/chunk assertions (#322) | LLM tone-judge over contributor-facing bot copy (leads with value, no fault-list, cost-free decline); uptake/negative-reaction telemetry on offer comments | + +**Headline for the eval spec:** during the run, exactly **one** class was caught by automation the framework itself owned (#307's derive scan). Everything else was caught by red consumer CI (post-damage), the #89 E2E harness (staging), or a human watching. Nearly every row's primary detector is deterministic — timeline joins, run-shape checks, static lints, diff properties — with inference needed only as a backstop. + +--- + +## 6. Baseline expectations (from the pre-run design issues #247, #252, #254, #255, #257, #272) + +The framework's own pre-run history sets explicit yardsticks the networking run should be measured against. + +**Task sizing and decomposition (#252, #272):** + +- The pre-run standard from #252: "A feature whose total scope is ~250 lines across a handful of files produces 1–2 tasks, not 6." Its live counter-example: "These three are a single cohesive change (~265 lines) split across three task issues, three agent runs, three PRs, three CI cycles, three reviews, three merges." +- #272 raised the floors mid-window: "Demoable units land at ~100 net lines when an engineer's natural PR is ~500" — ADR 0026 set a ~400-net-line demoable-unit floor (SDD_SPEC_MIN_UNIT), SDD_TRIAGE_MIN_TASK moved 300→400, and ADR 0028 collapsed single-task Units. Yardstick: median merged sdd PR net diff should sit near 400, not ~100; single-task-Unit count should be zero; sub-issue arithmetic ("Sub-issue count = 2 + N + M") should be minimal for the feature's size. + +**Per-task PR overhead (#252, #272):** + +- Every task pays a fixed cost: "one sdd-execute run, one PR, one CI pipeline, one sdd-validate, one sdd-review, one merge" (#272), with review adding "up to SDD_MAX_REVIEW_ITERATIONS revise cycles" (#252). Cost yardsticks from #252: "The triage pass that produced the split consumed ~25M tokens; each `sdd-execute` run adds ~1M+ on top" — and #272's acceptance section demands the pipeline-cost ratio "(agent runs + CI pipelines + review passes) / net lines merged" be trended as an `experiments:` entry. The retro should compute exactly that ratio for issue #478. + +**Ceremony vs value / the abandonment threshold (#255):** + +- The known failure ceiling: "a feature an operator would ship as one PR took ~2 days across 4 human gates (merge spec PR, merge architecture PR, `/approve`, `/dispatch`), produced 6 task sub-issues, and the operator abandoned the pipeline midway — closed green task PRs and consolidated by hand, because one task PR shipped a primitive with no consumer wired up ('dead code')." +- Yardsticks shipped pre-run: SDD_AGILE_MAX "default ~800" for the single-PR path; a collapsed single /approve gate; gate-latency ratio (>50% of wall-clock in human-gate waits = ceremony overload); green agent PRs closed unmerged by a human = trust-loss event; no task PR should ship exports with zero in-repo call sites. Also the acknowledged trade-off: "One ~800-line PR is harder to review than three 250-line PRs." + +**Status surface and operator legibility (#254):** + +- Pre-run diagnosis: "State is scattered across labels, sub-issue trees, PR checks, and Actions runs — 'buried in a CI job somewhere'"; "Operators want one surface to check in with and nudge, like a single agent session." sdd-status shipped before the run (ADR 0023, ~500–650-line action). +- Yardsticks: exactly one status comment per tracking issue whose last-updated footer is newer than the last pipeline event; measurable interval between the pipeline entering a human-action-required state and the first human event. The run-window evidence (#300, #303, #308, #313, #315, #320 all found by "human watching") measures how far reality fell short. + +**Bounded waits on external dependencies (#257):** + +- The pre-run invariant: no open wait without an SLA — "head-commit age ≥ `SDD_CODERABBIT_STALL_MIN` (default 30 min) AND no review or comment by `coderabbitai[bot]` since that commit" triggers bounded nudges ("cap `SDD_CODERABBIT_NUDGE_MAX` (default 2) per head sha"), then needs-human. Yardstick for the retro: every wait state in the networking run (reviewer bot, CI, human gate, dispatch hand-off) should have had a bounded-wait invariant; #313 and #320 show where the generalization was missing. + +**Revision context discipline (#247):** + +- Pre-run contract: "The `/revise` agent should be smart enough to read the comments on the PR and use them as revision context, rather than relying on the user to restate intent," so that "the user does not have to manually paste or re-summarize feedback already present in the thread." The assembled-directive substrate shipped (PR #262) and demonstrably worked during the run (#310 notes "the surviving revise run's assembled directive from #247 consumes all unresolved threads") — but #311 and #327 show the closing half of the loop (per-thread resolution, incremental review state) was never specified, and became the run's throughput bottleneck. + +**Summary yardstick set for the retro:** median PR net diff ≈ 400 lines; 1–2 tasks per ~250-line scope; per-feature cost ratio trended (aic); <50% wall-clock in gate waits; zero green agent PRs closed unmerged; one fresh status comment per tracker; every external wait bounded (30-min-class SLA); every revise grounded in the PR thread and every thread closed with a reasoned verdict. The networking run should be scored against each, with sections 1 and 4 supplying the incidents and section 5 the signals a future eval agent needs so that "human watching" stops being the primary detector. diff --git a/docs/specs/03-spec-run-evaluation/evidence/_run-window-issue-records.json b/docs/specs/03-spec-run-evaluation/evidence/_run-window-issue-records.json new file mode 100644 index 0000000..26023c0 --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/evidence/_run-window-issue-records.json @@ -0,0 +1,872 @@ +[ + { + "group": "baselines-design", + "number": 247, + "title": "/revise agent should read PR comments for revision context", + "created_at": "2026-06-08T23:00:54Z", + "closed_at": "2026-06-12T16:36:34Z", + "state": "closed (completed)", + "stage": "review|execute", + "failure_mode": "When a user triggered /revise on a PR, the revise agent did not fetch or read the PR's existing comment thread (review comments and discussion). The user had to manually paste or re-summarize feedback that was already sitting in the thread, and revisions produced without that context could miss or ignore reviewer intent.", + "root_cause": "The manual /revise path assembled no PR-thread context for the agent; it relied entirely on the user restating intent in the /revise invocation, even though the authoritative feedback lived in the PR comments.", + "consumer_impact": "Rework and operator toil on a live run: every manual /revise required the human to re-transcribe reviewer feedback, and a revise pass run without that context risks a diff that addresses none of the open review threads, burning another full review cycle.", + "consumer_refs": [], + "fix": "spectacles PR #262 (branch feat/247-revise-comment-context; commits bcebdb6 'feat(sdd-execute): assemble PR thread context for manual /revise' and cc5760b; merge 63e3aed)", + "detection_path": "Human feedback \u2014 filed from a Slack conversation with Karl Godard, not caught by any automated signal.", + "eval_agent_signal": "Deterministic: on each /revise-triggered run, join the agent's tool-call log against the PR's comment/review API \u2014 flag any revise run that performs zero comment/review fetch calls before editing while the PR has >0 unresolved review threads. Inference-based backstop: LLM-judge the revise diff against the set of unresolved review comments at trigger time and flag runs where the diff addresses none of them.", + "category": "prompt/model-behavior", + "quotes": [ + "The `/revise` agent should be smart enough to read the comments on the PR and use them as revision context, rather than relying on the user to restate intent.", + "The user does not have to manually paste or re-summarize feedback already present in the thread.", + "Filed from Slack conversation with Karl Godard." + ] + }, + { + "group": "baselines-design", + "number": 252, + "title": "sdd-triage over-decomposes: per-task PR overhead exceeds the work delivered", + "created_at": "2026-06-10T16:33:24Z", + "closed_at": "2026-06-10T23:14:35Z", + "state": "closed (completed)", + "stage": "triage", + "failure_mode": "sdd-triage split feature tracking issues into per-function/per-component task sub-issues with no lower bound on task size. Each tiny task incurred the full fixed pipeline cost (one sdd-execute run, one PR, one CI cycle, one sdd-validate pass, one sdd-review pass plus up to SDD_MAX_REVIEW_ITERATIONS revise cycles, one merge decision), so for sub-100-line PRs the pipeline cost dwarfed the delivered diff.", + "root_cause": "The decomposition rule optimized for task isolation \u2014 one file / one requirement per task with clean depends-on edges \u2014 with no minimum-task-size rule and no accounting of the fixed per-PR overhead.", + "consumer_impact": "Wasted tokens and CI on the live pilot run: a single ~265-line cohesive change was delivered as three separate task issues, three agent runs, three PRs, three CI cycles, three reviews, and three merges; the triage pass alone burned ~25M tokens and each execute run added ~1M+. This fragmentation also fed directly into the operator abandonment documented in #255.", + "consumer_refs": [], + "fix": "spectacles PR #253 (branch feat/252-triage-task-sizing; commits 363b5cf 'bundle cohesive small tasks to cut per-task overhead' and bee9a1b 'default SDD_TRIAGE_MIN_TASK to 300'; merge 557d8ec) + ADR 0022 (decisions/0022-triage-task-sizing.md); later reinforced by ADR 0026 (demoable-unit sizing) and ADR 0028 (collapse single-task units)", + "detection_path": "Human watching \u2014 the operator observed the PR-to-overhead ratio and token bills on the pilot run; no automated cost/granularity check existed.", + "eval_agent_signal": "Deterministic, per tracking issue: join task sub-issue count against total merged diff size and flag when median task-PR net diff < ~100 lines or task count > ceil(total_diff / SDD_TRIAGE_MIN_TASK); a sibling-overlap check (multiple small task PRs touching the same file set within one tracking issue) catches the 'split one cohesive change' shape; a token-count threshold on the triage run (e.g. >10M tokens for a sub-500-line feature) catches the cost side early, before dispatch fans out.", + "category": "cost/context-efficiency", + "quotes": [ + "The triage pass that produced the split consumed ~25M tokens; each `sdd-execute` run adds ~1M+ on top.", + "These three are a single cohesive change (~265 lines) split across three task issues, three agent runs, three PRs, three CI cycles, three reviews, three merges.", + "A feature whose total scope is ~250 lines across a handful of files produces 1\u20132 tasks, not 6." + ] + }, + { + "group": "baselines-design", + "number": 254, + "title": "sdd-status: deterministic self-updating status surface per tracking issue", + "created_at": "2026-06-10T23:47:28Z", + "closed_at": "2026-06-12T16:48:17Z", + "state": "closed (completed)", + "stage": "status|monitor", + "failure_mode": "There was no single place to see where the pipeline stood on a feature: state was scattered across lifecycle labels, sub-issue trees, PR checks, and Actions runs \u2014 'buried in a CI job somewhere.' Operators could not tell whether the pipeline was working, stalled, or waiting on them, and had no one surface to check and nudge.", + "root_cause": "The pipeline emitted state as side effects of many GitHub primitives but never aggregated them; sdd-monitor could not fill the role because its repository-wide in-flight gate defers exactly when status matters most (during execute runs) and it has opposite concurrency semantics and a mutator trust posture.", + "consumer_impact": "Stalls and needs-human hand-offs went unnoticed on the live run \u2014 operator response latency ballooned because knowing 'whose move is it' required spelunking labels, sub-issues, and Actions logs; this observability gap amplified every other failure class (e.g. the CodeRabbit stalls in #257).", + "consumer_refs": [], + "fix": "spectacles PR #265 (branch feat/254-sdd-status; commit df5ca02 'feat(sdd-status): deterministic self-updating status surface per tracking issue'; merge f67e6b3) + ADR 0023 (decisions/0023-status-surface.md); ships .github/actions/sdd-status/action.yml (~500\u2013650 lines), wrappers/sdd-status.yml, shared/parent-walk.js extraction, /status command, SDD_STATUS opt-out", + "detection_path": "Human field feedback from the consumer pilot run \u2014 an operator complaint, not an automated detection.", + "eval_agent_signal": "Deterministic staleness/idle-stall metric per tracking issue: compute 'operator-actionable' state from the same decision table (lifecycle label + derived facts like green-but-unreviewed PR, needs-human present) and measure the interval between the pipeline entering a human-action-required state and any human event (comment, merge, label change); flag intervals above a threshold. Post-fix, the sentinel comment itself is machine-checkable: assert exactly one status comment per tracking issue whose last-updated footer is newer than the last pipeline event (label, PR, check_suite, workflow_run).", + "category": "observability-gap", + "quotes": [ + "State is scattered across labels, sub-issue trees, PR checks, and Actions runs \u2014 \"buried in a CI job somewhere.\"", + "Operators want one surface to check in with and nudge, like a single agent session.", + "`.github/actions/sdd-status/action.yml` | new | ~500\u2013650 lines" + ] + }, + { + "group": "baselines-design", + "number": 255, + "title": "Agile single-PR path: generalize the fast path (amends ADR 0012)", + "created_at": "2026-06-10T23:47:29Z", + "closed_at": "2026-06-12T16:56:24Z", + "state": "closed (completed)", + "stage": "spec|triage|dispatch|process-design", + "failure_mode": "For small-to-medium features the full pipeline was heavy-handed: a feature an operator would ship as one PR took ~2 days across 4 human gates (merge spec PR, merge architecture PR, /approve, /dispatch) and produced 6 task sub-issues. One task PR shipped a primitive with no consumer wired up ('dead code'), and the operator abandoned the pipeline midway \u2014 closing green task PRs and consolidating the work by hand.", + "root_cause": "The ADR 0012 fast-path classifier was too narrow (hard file-count and new-public-API vetoes) and the full path imposed two separate document gates plus /approve and /dispatch; per-task decomposition could ship a primitive without its consumer because the consumer landed in a different task PR.", + "consumer_impact": "Pipeline abandonment on the live run: green agent-produced task PRs were closed unmerged and the operator consolidated by hand, discarding agent work (wasted execute/validate/review runs) and losing pipeline lifecycle tracking for the feature; ~2 days of gate-waiting for a one-PR-sized change.", + "consumer_refs": [], + "fix": "spectacles PR #266 (branch feat/255-agile-single-pr; commits 73b0502, 2e6ec12, d9c7afe; merge 09b8af0) + ADR 0024 (decisions/0024-agile-single-pr-path.md) and ADR 0025 (decisions/0025-approve-auto-dispatch.md); adds SDD_AGILE_MAX (default ~800), /agile alias, sdd:approved marker, collapsed single /approve gate", + "detection_path": "Human field feedback \u2014 the operator's mid-run abandonment was itself the signal; no automation noticed that green PRs were being closed unmerged.", + "eval_agent_signal": "Two deterministic detectors: (1) abandonment pattern \u2014 agent-branch (sdd/) task PRs with green checks closed unmerged by a human, especially with a superseding human PR touching the same files, is a direct machine-checkable proxy for loss of operator trust; (2) gate-latency ratio \u2014 fraction of a feature's wall-clock spent in human-gate wait states (>50% over multiple gates flags ceremony overload). Inference-based: per task PR, an LLM/static 'dead code' judge \u2014 does the diff export new symbols with zero in-repo call sites and no consumer wired in the same PR.", + "category": "process-design", + "quotes": [ + "a feature an operator would ship as one PR took ~2 days across 4 human gates (merge spec PR, merge architecture PR, `/approve`, `/dispatch`), produced 6 task sub-issues, and the operator abandoned the pipeline midway \u2014 closed green task PRs and consolidated by hand, because one task PR shipped a primitive with no consumer wired up (\"dead code\").", + "estimated net diff \u2264 `SDD_AGILE_MAX` (new repo var, default ~800, plumbed as workflow_call input per the ADR 0022 `min_task` pattern)", + "One ~800-line PR is harder to review than three 250-line PRs." + ] + }, + { + "group": "baselines-design", + "number": 256, + "title": "Per-repo dev-env entrypoint: .github/sdd/verify.sh in-sandbox + dev-server smoke", + "created_at": "2026-06-10T23:47:30Z", + "closed_at": "2026-06-12T16:47:12Z", + "state": "closed (completed)", + "stage": "execute|validate", + "failure_mode": "Implementer agents had no project-specific way to verify their work: the pre-PR gate relied on generic command discovery (.github/workflows \u2192 CLAUDE.md \u2192 Makefile/justfile/package.json) and agents could not spin up the repo's dev server or tooling to check changes end-to-end, so unverified changes flowed to the PR where consumer CI or humans caught them late. (The issue notes the largest part of the original complaint was a separate prompt bug \u2014 the sdd-node-cleanup contradiction, tracked in #258 \u2014 to be fixed first.)", + "root_cause": "No consumer-authored dev-env entrypoint existed; verification was framework-guessed rather than repo-defined, and there was no sanctioned in-sandbox path to run a dev server. The design constraint that consumer code must never run on the host (firewall/tamper risk) meant any solution had to execute inside the sandbox.", + "consumer_impact": "Rework on the live run: changes that passed the generic in-sandbox gate could still fail the consumer's real verification, pushing failures into post-PR CI and human review cycles instead of the agent's own loop; agents could not self-check UI/dev-server behavior at all.", + "consumer_refs": [], + "fix": "spectacles PR #263 (branch feat/256-verify-sh; commits 32b7223 'per-repo .github/sdd/verify.sh gate entrypoint + dev-server smoke' and aa4ad4c; merge f4a15c3); prompt/docs-only change to sdd-execute-{haiku,sonnet,opus}.md step 6 plus install docs, with opt-in Playwright dev-server smoke via SDD_MCP_EXTRA=playwright", + "detection_path": "Human field feedback from the consumer pilot run (verbatim operator quote in the issue).", + "eval_agent_signal": "Deterministic run-status join: per task, correlate the agent's in-sandbox pre-PR gate result with the first consumer CI outcome on the resulting PR \u2014 a high gate-pass-then-CI-fail rate is the machine signal that the agent's verification is not equivalent to the repo's real checks. Secondary deterministic checks: when .github/sdd/verify.sh exists, assert the execute transcript contains its invocation before PR creation; flag execute runs whose transcript shows zero build/test command executions prior to opening a PR.", + "category": "process-design", + "quotes": [ + "it would be good if implementer agents were able to spin up the dev stack, or some task that gives them the local dev server and tooling, so they can check their work... perhaps there is some way to configure an entry point for a dev env for an autonomous agent, but that's very project specific, so might be hard to generalize.", + "Consumer-controlled code on the host runs outside the firewall with unrestricted egress and could tamper with the awf firewall config, digest-pinned MCP images, bind-mounted language servers, or the toolchain the agent subsequently trusts.", + "A buggy verify.sh bricks every task into `needs-human`" + ] + }, + { + "group": "baselines-design", + "number": 257, + "title": "sdd-monitor: CodeRabbit stall detection with bounded nudges", + "created_at": "2026-06-10T23:47:31Z", + "closed_at": "2026-06-12T16:34:59Z", + "state": "closed (completed)", + "stage": "monitor|review", + "failure_mode": "CodeRabbit sometimes never reviewed an agent-created PR at all (possibly due to its usage limits). The framework handled CodeRabbit CHANGES_REQUESTED reviews (implicit /revise, #128) but had no handler for the absence of a review, so the PR sat silently and the operator had to notice the stall and intervene by hand.", + "root_cause": "The pipeline's review loop was purely event-driven \u2014 it reacted to reviews that arrived but had no timeout/backstop for an external reviewer bot that never fires, leaving a hole in the state machine between 'PR open' and 'review received'.", + "consumer_impact": "Silent stall on the live run: agent PRs waited indefinitely for a review that never came, blocking the merge/auto-revise loop until a human happened to notice; unattended (overnight) operation was impossible for affected PRs.", + "consumer_refs": [], + "fix": "spectacles PR #264 (branch feat/257-coderabbit-stall; commits 9c93bfe 'detect CodeRabbit review stalls with bounded nudges' and 780c53f; merge ba8973f); adds a stall pass to .github/actions/sdd-monitor/action.yml (~130 lines) with SDD_CODERABBIT_STALL_MIN (30 min), SDD_CODERABBIT_NUDGE_MAX (2) per head sha, then needs-human escalation", + "detection_path": "Human watching \u2014 the operator noticed the silence during the pilot run; no monitor pass covered the no-review case.", + "eval_agent_signal": "Fully deterministic timeline-shape check, exactly the stall predicate the fix encodes: for each open non-draft sdd/ PR without needs-human, flag when head-commit age exceeds a threshold (default 30 min) with no review or comment by coderabbitai[bot] since that commit. Generalizable eval signal: for every external event the pipeline waits on (reviewer bot, CI check, human gate), assert a bounded-wait invariant \u2014 any wait state older than its SLA without a corresponding response event is a stall the eval agent reports.", + "category": "state-machine/lifecycle", + "quotes": [ + "CodeRabbit sometimes never reviews an agent-created PR at all (possibly its usage limits), and the operator has to notice the silence and intervene by hand.", + "head-commit age \u2265 `SDD_CODERABBIT_STALL_MIN` (default 30 min) AND no review or comment by `coderabbitai[bot]` since that commit.", + "cap `SDD_CODERABBIT_NUDGE_MAX` (default 2) per head sha (a new push resets the budget, matching CodeRabbit's per-push review model)" + ] + }, + { + "group": "pre-run-bugs", + "number": 258, + "title": "sdd-node-cleanup contradicts the pre-PR CI gate: Node agents skip install/typecheck/test", + "created_at": "2026-06-10T23:47:32Z", + "closed_at": "2026-06-11T00:51:07Z", + "state": "closed (completed)", + "stage": "execute", + "failure_mode": "Node-stack sdd-execute runs skipped install/typecheck/lint/test before opening PRs, so PRs landed already failing consumer CI. The compiled execute prompt contradicted itself: step 6's pre-PR CI gate requires running the target repo's declared CI commands green, but its only worked example and its supersede paragraph were Rust-only, while the imported shared/sdd-node-cleanup.md fragment asserted the opposite for Node ('no npm-registry egress... cannot run prettier/eslint/lockfile refresh' and 'test execution (npm test) and type-checking (tsc) are the consumer CI's job'). A Node run read the louder, stack-specific fragment and skipped the gate.", + "root_cause": "Stale prompt fragment contradiction, not a missing mechanism: both fragment claims were false at run time \u2014 the compiled lock's firewall allowlist includes registry.npmjs.org (gh-aw defaults) and the agent job runs actions/setup-node, so 'corepack enable && pnpm install --frozen-lockfile && pnpm check && pnpm test && pnpm build' was runnable in-sandbox. The loudest stack-specific instruction won over the generic gate.", + "consumer_impact": "Rework and red CI on the live run: every Node PR arrived failing consumer CI ('the agent never checks its work'), forcing revise cycles or human intervention instead of a self-verified green PR.", + "consumer_refs": [], + "fix": "PR #260 (branch fix/258-node-prepr-gate): commits e57053d (extend pre-PR CI gate to Node, drop false sandbox claims), 7ca094d (sdd-rust-cleanup: tests belong to in-sandbox gate), 35d39e1 (align lockfile detection order), e5e56e8 (yarn bootstrap egress); locks recompiled. Closed 2026-06-11.", + "detection_path": "Human field feedback from the consumer pilot run (pnpm/Astro repo) \u2014 a maintainer observed the agent never checks its work; no automated signal fired.", + "eval_agent_signal": "Two complementary checks. Deterministic: join each execute run's sandbox transcript with the resulting PR's first check-suite conclusion \u2014 flag any sdd/ PR whose first consumer CI run fails install/typecheck/test while the agent transcript contains zero package-manager/toolchain invocations (no corepack/npm/pnpm/yarn/cargo commands) for the detected stack; a rising per-stack first-push CI-failure rate is the early aggregate signal. Inference-based: an LLM lint over the compiled lock that hunts for mutually contradictory directives between the generic gate text and imported stack fragments (e.g. 'must run declared CI commands green' vs 'cannot run npm test / consumer CI's job'), plus a fact-check of sandbox-capability claims against the compiled firewall allowlist (fragment says 'no npm-registry egress' while lock allowlists registry.npmjs.org).", + "category": "prompt/model-behavior", + "quotes": [ + "the agent never checks its work.", + "Step 6's pre-PR CI gate (`sdd-execute-sonnet.md` ~lines 518\u2013564) requires running the target repo's declared CI commands green before opening a PR", + "`corepack enable && pnpm install --frozen-lockfile && pnpm check && pnpm test && pnpm build` is runnable inside the firewall now" + ] + }, + { + "group": "pre-run-bugs", + "number": 259, + "title": "Auto-revise fires only on required checks; protection lookup silently dead on GITHUB_TOKEN", + "created_at": "2026-06-10T23:47:33Z", + "closed_at": "2026-06-11T00:51:21Z", + "state": "closed (completed)", + "stage": "execute", + "failure_mode": "Failed-check auto-revise (#203) only fired for checks in the base branch's required set, and the required-set lookup itself was silently dead: the check_suite branch of sdd-route-execute fetched classic branch protection with the default github.token, and treated any error as 'no required contexts \u2192 do NOT trigger'. On repos where the call 403s/404s, no CI failure on any sdd PR ever triggered a revise. The lookup was also blind to repository rulesets, and even where it worked, non-required checks (commonly the expensive E2E/build-matrix ones) never triggered.", + "root_cause": "Permission/contract mismatch swallowed into a default-deny: the classic branch-protection endpoint needs administration:read, which GITHUB_TOKEN lacks (the wrapper's own auto-merge job mints an App token for exactly this reason); the error branch conflated 'cannot read protection' with 'nothing is required'. Fixed by defaulting the trigger to ALL failing checks (revise-on-checks=all) with an exclusion-glob input, and by querying rulesets (GET /repos/{o}/{r}/rules/branches/{branch}) in required mode.", + "consumer_impact": "Silent stall on the live run: PRs with failing CI sat red indefinitely with no revise and no escalation \u2014 'CI failures in my PRs would never be actioned on' \u2014 requiring a human to notice and manually /revise. Bounded blast radius of the fix: worst case 3 extra revise runs then a visible needs-human.", + "consumer_refs": [], + "fix": "PR #261 (branch fix/259-revise-all-checks): commits 14d17e8 (trigger auto-revise on all failed checks, not just required), 6d07597 (wrapper comment corrections). Default-on via @main action consumption, no lock recompiles. Closed 2026-06-11.", + "detection_path": "Human field feedback from the consumer pilot run; the failure was invisible in CI because the dead path exited as a clean no-op.", + "eval_agent_signal": "Deterministic timeline join: for every failure-conclusion check_suite event on an sdd/ head branch, expect within a bounded window either (a) a triggered execute/revise workflow run, (b) a hidden revise-iteration marker comment, or (c) a needs-human label; a failed suite followed by zero pipeline activity is the silent-drop signature. Second, generic anti-pattern detector: any swallowed API error (403/404) inside a routing action that resolves to a default-deny decision must emit a distinct step-output/log marker ('protection lookup failed'); an eval agent counts those markers vs revise triggers and alarms when lookup-failure rate is ~100% (path permanently dead) on a repo.", + "category": "permission-contract", + "quotes": [ + "CI failures in my PRs would never be actioned on.", + "That endpoint needs administration:read, which `GITHUB_TOKEN` lacks \u2014 the wrapper's own auto-merge job mints an App token for exactly this reason.", + "Worst case for a flaky advisory check: 3 extra revise runs then a visible escalation \u2014 strictly better than silence." + ] + }, + { + "group": "pre-run-bugs", + "number": 269, + "title": "Pin an explicit model per agent; stop floating on GH_AW_MODEL_AGENT_COPILOT", + "created_at": "2026-06-12T21:31:50Z", + "closed_at": "2026-06-18T03:45:00Z", + "state": "closed (not_planned)", + "stage": "spec|triage|dispatch|validate|review|infra", + "failure_mode": "Enhancement proposal: five agents (sdd-spec, sdd-triage, sdd-dispatch, sdd-validate, sdd-review) ran bare 'engine: copilot', so their model floated on the consumer-overridable GH_AW_MODEL_AGENT_COPILOT var (default claude-sonnet-4.6). PR #248 demonstrated the footgun: a consumer's model var silently resolved distillery-sync to opus and blew the per-run effective-token cap (429). Any unpinned agent was one consumer var away from a silent re-model \u2014 cost and behavior both change with no diff anywhere in the pipeline repo. Proposed per-agent pins: triage=opus (errors fan out multiplicatively), spec/validate/review=sonnet, dispatch=haiku.", + "root_cause": "Model selection treated as a global engine default instead of a per-agent fit decision; consumer-side vars had silent override authority over pipeline-repo behavior. Closed not_planned because the Copilot\u2192Claude engine port (commit 6a0be81, 2026-06-16) rebuilt the override surface (GH_AW_MODEL_AGENT_CLAUDE / GH_AW_MODEL_DETECTION_CLAUDE, documented in docs/sdd/install.md; execute tiers already pin via the model:* task label), mooting the Copilot-specific pinning table.", + "consumer_impact": "No direct live-run breakage from this issue itself (it is preventive), but the class it names had already burned a run: the #248 precedent was a silent re-model to opus that 429'd distillery-sync \u2014 wasted tokens and a dead run with no diff to explain it.", + "consumer_refs": [], + "fix": "Not planned \u2014 superseded by the Copilot\u2192Claude engine migration (commit 6a0be81, 2026-06-16); distillery-sync stayed pinned claude-haiku-4.5 per #248, execute tiers stay label-pinned, and the remaining float is documented in docs/sdd/install.md.", + "detection_path": "Human model-fit review of all agents, prompted by the #248 incident (429 on distillery-sync) observed during the consumer pilot.", + "eval_agent_signal": "Deterministic model-drift monitor: extract the resolved model ID from each run's OTLP spans / engine logs (ADR 0020 spans carry per-agent token usage and model) and diff it against a committed per-agent expected-model manifest; alert on any run whose resolved model differs from the manifest without a corresponding repo diff. Complementary config lint at compile time: every engine: block must either pin an explicit model literal in the lock or appear on a documented float allowlist; CI fails otherwise. Cost-side corroboration: per-agent token-cost baseline with alerting on step-change (an opus-priced token profile appearing under a haiku-manifested agent).", + "category": "infra/config", + "quotes": [ + "a consumer's model var silently resolved `distillery-sync` to opus and blew the per-run effective-token cap (429)", + "triage runs once per feature and its errors multiply downstream (one bad decomposition = N bad task PRs)", + "default `claude-sonnet-4.6`, consumer-overridable" + ] + }, + { + "group": "pre-run-bugs", + "number": 271, + "title": "sdd-triage burns context to the effective-token rail: structural context reduction", + "created_at": "2026-06-15T18:19:07Z", + "closed_at": "2026-06-17T19:12:29Z", + "state": "closed (completed)", + "stage": "triage", + "failure_mode": "A phase-A sdd-triage run on the consumer pilot died on the AWF hard rail: '429 Maximum effective tokens exceeded (25096706.10 / 25000000)'. It consumed 2.5M input tokens across 33 model API calls and produced zero safe output ({\"items\":[]}). Per-call context grew monotonically 33K \u2192 122K tokens; the prompt cache broke mid-run (calls 29/31/32 reset cache_read to the 32K base), re-billing the full context uncached; the run re-read the same tracking issue 4x via issue_read; Serena's MCP schemas were loaded but never called (0 calls; github=7, distillery=2).", + "root_cause": "sdd-triage was a single 1070-line, 7-situation mega-workflow whose ~103KB / ~33K-token compiled prompt loaded all three phases (A/B/C), six imported fragments including both heavy MCP fragments (sdd-mcp-serena.md 13.7KB, sdd-mcp-distillery.md 7.7KB), and the full GitHub default toolset on every run, regardless of which single phase actually executed \u2014 plus a mid-run cache break from static/dynamic prompt ordering.", + "consumer_impact": "Hard stall plus ~25M effective tokens wasted: the feature's triage produced nothing, halting the pipeline for that feature until the run was diagnosed and retried; before the #270 cap, the failure mode was silent rail-death rather than a recoverable noop.", + "consumer_refs": [], + "fix": "Backstop PR #270 (commit f3f25d2, cap AWF invocations at 25 to convert rail-death into a recoverable stop); structural cure PR #278 (commit 8e84c6f, deterministic entity pre-fetch + cache-hygiene, items B/D) and PR #279 (commit a7dd093, split into per-phase arch/plan/materialize workflows, ADR 0029). Closed 2026-06-17.", + "detection_path": "Red run: the 429 from the proxy's maxEffectiveTokens hard rail surfaced in the failed workflow run's logs; a human read the telemetry (per-call token growth, cache resets, tool-call counts) post-hoc.", + "eval_agent_signal": "This class is fully catchable from run telemetry long before the rail. Deterministic per-run monitors over OTLP spans / engine logs: (1) cumulative effective-token consumption crossing a fraction (e.g. 50%) of the 25M budget; (2) monotonic per-call context growth slope (33K\u2192122K) with no plateau; (3) cache_read dropping back to the base prefix mid-run (cache-break signature); (4) duplicate identical read calls (same issue_read target \u22653x); (5) loaded-vs-called tool-surface ratio (an MCP server with schemas in context and 0 invocations); (6) calls-completed vs safe-output-produced \u2014 a run past N calls with an empty items buffer is heading for zero output. Any one of these firing early converts a 25M-token death into a cheap abort/alert.", + "category": "cost/context-efficiency", + "quotes": [ + "429 Maximum effective tokens exceeded (25096706.10 / 25000000)", + "It consumed 2.5M input tokens across 33 model API calls and produced **zero safe output** (`{\"items\":[]}`). Per-call context grew monotonically 33K \u2192 122K tokens; the prompt cache broke mid-run (calls 29/31/32 reset `cache_read` to the 32K base)", + "a single 1070-line, 7-situation mega-workflow. The compiled prompt is ~103KB / ~33K base tokens ... Serena was loaded but never called (0 tool calls; github=7, distillery=2)" + ] + }, + { + "group": "pre-run-bugs", + "number": 272, + "title": "Reduce sub-issue count: larger demoable units + collapse single-task Units", + "created_at": "2026-06-15T19:37:57Z", + "closed_at": "", + "state": "open", + "stage": "spec|triage", + "failure_mode": "The pipeline over-decomposes work: demoable units land at ~100 net lines where an engineer's natural PR is ~500, and triage creates a grouping Unit sub-issue even when the Unit holds exactly one task. Sub-issue count = 2 + N + M per feature, and each task pays the full per-task pipeline cost (one sdd-execute run, one PR, one CI pipeline, one sdd-validate, one sdd-review, one merge) \u2014 the same poor cost-to-diff ratio ADR 0022 addressed, one layer up.", + "root_cause": "Three gaps: (A) sdd-spec step 5 gave no lower-bound unit sizing ('split as the work naturally falls'), so the model defaulted to fine-grained ~100-line units; (B) ADR 0022 folding was intra-Unit and cohesion-gated, never reducing N or folding across Units; (C) Units were mandatory grouping sub-issues even at 1:1 with tasks, making them pure overhead. Secondary: 5 binary fastpath gates hard-eject a 400-line feature into the full tree despite the 800-line SDD_AGILE_MAX ceiling.", + "consumer_impact": "Wasted tokens and human review load on the live run: a modest feature fanned out into many sub-issues and many tiny PRs, multiplying agent runs, CI pipelines, and review passes per net line shipped; no stall, but a structurally inflated cost per feature.", + "consumer_refs": [], + "fix": "Partial: Levers 1+3 shipped in PR #273 (ADR 0026 \u2014 ~400-net-line demoable-unit floor via SDD_SPEC_MIN_UNIT, unit-cohesion rule, spec gate 5, SDD_TRIAGE_MIN_TASK 300\u2192400); Lever 2 subsequently landed as ADR 0028 (commit 2245288, collapse single-task Units to feature-parented tasks, with tree-walker fixes 73631a6/77e4e65/96708a2); Lever 4 (soften fastpath ejection) still open \u2014 issue remains open.", + "detection_path": "Human observation of tree arithmetic and PR sizes during the consumer pilot (successor to the original over-decomposition report #252).", + "eval_agent_signal": "Purely deterministic repo-metric monitors: (1) net-diff-lines distribution per merged sdd PR \u2014 alert when the median falls well below the configured unit target (e.g. median ~100 vs target 400); (2) single-task-Unit fraction \u2014 walk each feature's sub-issue tree via the sub-issues API and count Units with exactly one child task (pure overhead nodes); (3) per-feature pipeline-cost ratio: (agent runs + CI pipelines + review passes) / net lines merged, i.e. the 'aic' metric the issue's own acceptance section demands as an experiments: entry, trended before/after sizing changes.", + "category": "process-design", + "quotes": [ + "Demoable units land at ~100 net lines when an engineer's natural PR is ~500", + "**Sub-issue count = 2 + N + M.**", + "each paying the full per-task pipeline cost (one `sdd-execute` run, one PR, one CI pipeline, one `sdd-validate`, one `sdd-review`, one merge)" + ] + }, + { + "group": "pre-run-bugs", + "number": 275, + "title": "Retrospective spec derivation for unspecced code (sdd-derive)", + "created_at": "2026-06-17T03:53:12Z", + "closed_at": "2026-06-17T14:29:47Z", + "state": "closed (completed)", + "stage": "spec|review|process-design", + "failure_mode": "Coverage-gap enhancement: SDD was forward-only (spec \u2192 triage \u2192 execute), so code explored directly on a feature branch with no tracking issue shipped with no spec, and sdd-review gates to sdd/ head branches \u2014 exploratory human PRs were exactly the case nothing covered. The spec artifact is still needed as the thing the team reviews and the entry distillery-sync mirrors into memory. Adds sdd-derive: deterministic size-gated detection (offer comment + needs-spec marker on PRs lacking SDD lineage with diff \u2265 SDD_SPEC_MIN_UNIT=400), a /derive-spec command that opens a spec/ docs PR with an in-spec Gap Analysis, and a weekly sdd-unspecced-scan roll-up with matrix fan-out.", + "root_cause": "Pipeline design assumed all shippable code enters through the SDD front door; human/exploratory PRs bypassed every gate and left the spec corpus and distillery memory incomplete, degrading future triage/dedup decisions.", + "consumer_impact": "No immediate live-run breakage; the impact is accumulating debt \u2014 unspecced merged code invisible to spec review and to distillery-sync's memory, weakening downstream duplication-avoidance and review coverage until derived retroactively.", + "consumer_refs": [], + "fix": "PR #276 (commit c009887, ADR 0027) plus follow-ups f0b52ff (CodeRabbit fixes on detection/scan), d9e6fc6 (reword offer as helpful, not a callout), 850e923 (installer installs sdd-derive). Closed 2026-06-17.", + "detection_path": "Human process review during the pilot \u2014 a gap analysis of what the review/spec gates do not cover, not a triggered failure.", + "eval_agent_signal": "Deterministic lineage audit an eval agent can run weekly: for every merged non-bot, non-docs-only PR with diff \u2265 SDD_SPEC_MIN_UNIT (400), verify SDD lineage (sdd/ head-branch prefix, linked tracking issue, or a docs/specs/ spec touching the same area); count unspecced merges as the coverage metric. Once sdd-derive is installed, verify the mechanism itself: each qualifying PR gets exactly one offer comment + needs-spec marker (idempotency check), /derive-spec from a write-scoped user yields a spec/ PR containing demoable units, R{unit}.{seq} acceptance criteria, and a populated Gap Analysis (LLM-judgeable), and non-write /derive-spec commands are ignored.", + "category": "process-design", + "quotes": [ + "on a PR lacking SDD lineage with a diff \u2265 `SDD_SPEC_MIN_UNIT` (default 400), post one offer comment + `needs-spec` marker", + "`sdd-review` gates to `sdd/` head branches, so exploratory human PRs are exactly the case nothing covers", + "Gaps (implementation gaps, missing failure paths, weak acceptance criteria, skipped demoable units) documented **in-spec only** in a new Gap Analysis section" + ] + }, + { + "group": "run-week-1", + "number": 283, + "title": "Consumer wrappers + installer still pass Copilot engine secret after Claude migration", + "created_at": "2026-06-18T03:41:51Z", + "closed_at": "2026-06-18T03:52:46Z", + "state": "closed (completed)", + "stage": "installer|infra (all 10 agentic wrappers: spec, triage, validate, review, derive, execute x3, spike-reentry, distillery-sync)", + "failure_mode": "The engine was migrated Copilot\u2192Claude in the compiled locks and shared fragments on main, but all 10 consumer-facing agentic wrappers still passed COPILOT_GITHUB_TOKEN to the reusable lock, and quick-setup.sh still told operators to provision the Copilot secret. Passing an undeclared secret to a workflow_call reusable workflow is a hard startup_failure, so the agent job never even starts. Every main-pinned install was completely dead; tag-pinned v0.2.0 installs were unaffected because wrapper and lock were internally consistent there.", + "root_cause": "Incomplete migration: locks declared CLAUDE_CODE_OAUTH_TOKEN and dropped COPILOT_GITHUB_TOKEN, but the consumer-facing surface (wrappers + quick-setup.sh) was never updated. No lint existed to check that a wrapper's secrets: map is a subset of the called lock's declared workflow_call.secrets.", + "consumer_impact": "Total pipeline stall on any main-pinned consumer install: sdd-spec, sdd-triage, sdd-validate all hit startup_failure in ~1s, so no agentic stage could run at all. Pure-actions wrappers (sdd-status, sdd-triage-dedupe-tasks) still ran, masking the breadth of the outage. Operators following quick-setup provisioned the wrong secret.", + "consumer_refs": [], + "fix": "spectacles PR #284 (branch fix/283-wrapper-claude-engine-secret, commit 1fd246b, merged 29ae79a): all wrappers pass CLAUDE_CODE_OAUTH_TOKEN; quick-setup.sh names the Claude engine secret. Prevention item: add lint that wrapper secrets \u2286 lock's workflow_call.secrets.", + "detection_path": "Surfaced by the issue #89 E2E harness on first contact with a fresh main-pinned install (not caught by any lint or unit-level check).", + "eval_agent_signal": "Two deterministic signals: (1) static CI lint asserting every wrapper's secrets: map is a subset of the called lock's declared workflow_call.secrets \u2014 'a wrapper passing an undeclared secret is always a startup failure', so this is a zero-false-positive check; (2) run-status join: any agentic workflow run concluding startup_failure (a machine-readable conclusion distinct from failure) with duration ~1s, especially when N different agentic workflows all show it while pure-actions workflows succeed \u2014 that fan-wide startup_failure pattern uniquely fingerprints a wrapper/lock contract break and requires no log parsing or inference.", + "category": "infra/config", + "quotes": [ + "Reproduced on a fresh install pinned to `main`: `sdd-spec`, `sdd-triage`, `sdd-validate` all `startup_failure` in ~1s; pure-actions wrappers (`sdd-status`, `sdd-triage-dedupe-tasks`) run fine.", + "Release `v0.2.0` is internally consistent (wrapper + lock both Copilot), so tag-pinned consumers are unaffected. Every `main`-pinned install is broken.", + "a wrapper passing an undeclared secret is always a startup failure" + ] + }, + { + "group": "run-week-1", + "number": 285, + "title": "Agents hard-fail at MCP gateway startup when Distillery is unconfigured", + "created_at": "2026-06-18T03:54:19Z", + "closed_at": "2026-06-18T04:10:01Z", + "state": "closed (completed)", + "stage": "spec|derive|triage|distillery|installer (any agent importing shared/sdd-mcp-distillery.md)", + "failure_mode": "When a consumer has not set the DISTILLERY_MCP_URL repo variable, the compiled lock's `url: ${{ vars.DISTILLERY_MCP_URL }}` resolves to empty string; the MCP gateway's config schema validation rejects the empty URL and exits 1 before the agent runs, killing the whole pipeline at sdd-spec. This contradicts the fragment's own outage contract, which already tolerates an unreachable store \u2014 the agent never gets the chance to apply it because 'unconfigured' fails harder than 'unreachable'. A follow-up comment retracted the issue's proposed in-expression `||` fallback: the gh-aw compiler injects the raw url expression into both the gateway config and the firewall allowDomains list, and a literal `://` inside the expression corrupts the allow-list.", + "root_cause": "Direct interpolation of an optional consumer variable into a schema-validated MCP gateway URL with no default; the degradation contract only covered unreachable stores, not unconfigured ones. Secondary root cause discovered during fixing: the gh-aw compiler reuses the URL expression in the firewall allowDomains, so fixes cannot live in the compiled expression.", + "consumer_impact": "Hard stall at sdd-spec for every consumer that does not run Distillery: route/activation pass, then the agent job fails at 'start MCP gateway' with a schema error \u2014 the pipeline is dead before the first agent token is spent. Also burned a first fix attempt (the fragment-level fallback) that had to be retracted and redesigned installer-side.", + "consumer_refs": [], + "fix": "spectacles PR #286 (branch fix/285-distillery-graceful-degrade, commit f5a3672 'fix: default DISTILLERY_MCP_URL to a placeholder so agents degrade gracefully', merged 21da064): quick-setup.sh writes the non-routable RFC-6761 placeholder https://distillery.disabled.invalid/ to the variable when unset, converting 'unconfigured' into the already-handled 'unreachable' path; fragment/locks untouched.", + "detection_path": "Surfaced by the issue #89 E2E harness on a fresh main-pinned install with no Distillery config; the compiler/allowDomains interaction was found only when implementing the first fix.", + "eval_agent_signal": "Deterministic, layered: (1) config lint \u2014 every `${{ vars.* }}` interpolated into an MCP server url in a compiled lock must be paired with an installer-written default or a compile-time guard (empty string can never reach the gateway schema); (2) runtime log-marker join \u2014 agent job failed AND log contains the gateway schema signature (\"is not valid 'uri'\" / \"does not match pattern '^https?://.+'\") before the agent step started, which cleanly separates gateway-config death from agent failure; (3) compiled-lock grep for a corrupted allowDomains entry (an allowDomains string containing `${{` and a truncated scheme) would have caught the retracted first fix before merge.", + "category": "infra/config", + "quotes": [ + "/mcpServers/distillery/url Error: '' is not valid 'uri' Error: does not match pattern '^https?://.+'", + "The gateway exits 1, so the `agent` job fails **before the agent runs**. A consumer that does not run Distillery cannot run any `sdd-*` agent \u2014 the whole pipeline is dead at `sdd-spec`.", + "\"allowDomains\":[\"${{ vars.DISTILLERY_MCP_URL || 'https\", ... # corrupted" + ] + }, + { + "group": "run-week-1", + "number": 287, + "title": "SDD pipeline silently reverted merged fix #449 (guest pid-1 panic-safety) in PR #463", + "created_at": "2026-06-18T04:12:49Z", + "closed_at": "2026-06-18T18:06:31Z", + "state": "closed (completed), labels: bug, severity:high", + "stage": "execute (regeneration step; validate/review also failed to flag it)", + "failure_mode": "SDD-generated consumer PR #463 (branch sdd/453-net-module) regenerated crates//src/connection.rs from a base state that did not carry merged fix #449, silently reverting Connection::from_socket from Result<\u2026, ConnectionError> + .await? back to a bare tuple + .await.unwrap() \u2014 reintroducing a guest pid-1 panic where a client that connects then drops kills init and takes down the whole VM. Nothing in the pipeline flagged that the generated diff undid hunks from an already-merged commit; the deleted code included the explanatory safety comment.", + "root_cause": "The execute/regenerate step rewrote the file from a working base that trailed origin/main (no rebase/refresh before regeneration), and no guard compared the regenerated diff against origin/main to detect reverted merged hunks \u2014 the diff looked clean against the stale base.", + "consumer_impact": "Silent revert of a merged safety-critical fix on the live consumer run: red CI (build/clippy/test) on PR #463, manual rework to restore the fix (commit ab7eca9a), and \u2014 had CI not caught it \u2014 a pid-1 panic that would crash the entire guest VM. This is the highest-severity class in the window: agent-caused code-integrity regression, not a stall.", + "consumer_refs": [ + "#449 \u2014 merged consumer fix (guest pid-1 panic-safety in connection.rs handshake path) that PR #463 silently reverted", + "#463 \u2014 SDD-generated net-module PR (branch sdd/453-net-module) containing the revert; went red on build/clippy/test", + "ab7eca9a \u2014 consumer restore commit 'fix(): restore handshake-failure resilience and guest PATH' re-applying #449 on the PR branch" + ], + "fix": "spectacles PR #288 (branch fix/287-merged-revert-guard, commit 0de6a28, merged f1ad1d4): deterministic merged-change revert guard added to step 6's Pre-PR gate (re-run from step 7's update path) in all three execute variants \u2014 fetches history, finds divergence point, blocks the PR when the branch is behind origin/main AND touches a file origin/main independently advanced; remedy is rebase (turning the silent revert into a visible conflict) or needs-human. Inlined rather than a shared fragment because a new fragment cannot resolve from @main in the same PR. Fix commit co-authored by Claude Opus 4.8 (1M context).", + "detection_path": "Red CI on consumer PR #463 plus human inspection of the diff; the pipeline itself raised no flag \u2014 the revert was only visible when diffing against origin/main instead of the stale base.", + "eval_agent_signal": "Deterministic diff property \u2014 the 'exact silent-revert signature' from the fix: PR branch merge-base is behind origin/main AND the PR touches a file that origin/main has independently advanced since the divergence point; stronger variant: 3-way check whether any PR hunk is the textual inverse of a hunk from a merged commit on the same lines (diff the regenerated file against origin/main, not the base). Inference-based complement: an LLM judge shown the PR-diff-vs-origin/main asked 'does this PR delete or undo code introduced by merged commits?' \u2014 deletion of a safety-explaining comment block plus replacement of `?`-propagation with `.unwrap()` is a high-precision tell. Either check runs pre-PR and needs no human.", + "category": "safety/code-integrity", + "quotes": [ + "In the guest, `` is pid-1, so a panic here kills init and takes down the whole VM.", + "Nothing in the pipeline flagged that the generated diff reverted hunks from an already-merged commit.", + "blocks the PR when the branch is behind origin/main AND touches a file origin/main has independently advanced \u2014 the exact silent-revert signature" + ] + }, + { + "group": "run-week-1", + "number": 289, + "title": "Fast-path /approve never dispatches: sdd-spec route job lacks pull-requests:read", + "created_at": "2026-06-18T15:15:42Z", + "closed_at": "2026-06-18T18:06:43Z", + "state": "closed (completed)", + "stage": "dispatch (fast-path route gate in wrappers/sdd-spec.yml)", + "failure_mode": "The fast-path single-PR flow (/agile \u2192 stub spec PR \u2192 /approve \u2192 merge \u2192 implementation dispatch, ADR 0012/0024) never reached sdd:in-progress. On the merged spec PR, the sdd-route-spec composite action calls pulls.listFiles with the route job's GITHUB_TOKEN, but the route job declares only contents:read + issues:read \u2014 no pull-requests scope. The 403 is swallowed by a catch block, approved_dispatch is never set, the approved-merge-dispatch job skips, and the agent-fallback path runs instead: the spec PR merges but the implementation is never dispatched, leaving the tracking issue stranded at sdd:fastpath-review + sdd:approved.", + "root_cause": "Permission-contract mismatch between a job's permissions block and the GitHub API calls made by the composite action it runs (pulls.listFiles at action.yml:320 needs pull-requests:read), made silent by an error-swallowing catch (action.yml:366-370). The route job intentionally runs on the unprivileged GITHUB_TOKEN because it gates App-token minting, so the scope had to be added there. Audit showed sdd-spec was the only wrapper broken in this class.", + "consumer_impact": "Fast-path flow dead on every consumer: spec PR merges, then nothing \u2014 no implementation PR, tracking issue stranded, requiring a human to notice the silence. Diagnosis cost real time because two plausible suspects (head-branch deletion 404; approve-after-merge ordering) had to be eliminated first; the failing job itself was green.", + "consumer_refs": [], + "fix": "spectacles PR #290 (branch fix/route-pr-read-perm, commit fce6edc 'fix(sdd-spec): grant route job pull-requests:read so fast-path dispatch works', merged 9f1bf07). Landed before the v0.3.0 tag as required.", + "detection_path": "Human-driven staging-consumer reproduction pinned to spectacles main (21da064), reading the route run log to find the swallowed 403 \u2014 no automated signal fired because the job succeeded.", + "eval_agent_signal": "Three machine checks: (1) static lint mapping every github.rest.* call inside each composite action to its required token scope and asserting it is \u2286 the calling job's permissions block (this class is fully enumerable from source); (2) runtime log scan for 'Resource not accessible by integration' inside a job that concluded success \u2014 a swallowed-403-in-green-job marker is the fingerprint of silent permission failures; (3) timeline-shape monitor: tracking issue carrying sdd:approved with its spec PR merged but no sdd:in-progress label event within N minutes = stranded fast-path, catchable by a label-event/PR-merge join with zero inference.", + "category": "permission-contract", + "quotes": [ + "Could not resolve the approved tracking issue from the spec file frontmatter: Resource not accessible by integration - https://docs.github.com/rest/pulls/pulls#list-pull-requests-files", + "the dispatch job **skips**, and the agent-fallback path runs instead \u2014 so the spec PR merges but the implementation is never dispatched", + "**sdd-spec is the only broken wrapper.**" + ] + }, + { + "group": "run-week-1", + "number": 292, + "title": "Fast-path implementation never dispatches: uses workflow_dispatch (needs App actions:write) against ADR-0014", + "created_at": "2026-06-18T16:13:04Z", + "closed_at": "2026-06-18T18:06:01Z", + "state": "closed (completed)", + "stage": "dispatch (fast-path approved-merge-dispatch; also installer docs)", + "failure_mode": "The next failure in the same fast-path chain after #289: sdd-fastpath-approve dispatches the single-PR implementation via github.rest.actions.createWorkflowDispatch, which requires App actions:write \u2014 a scope the documented install (docs/sdd/install.md:189: exactly contents/discussions/issues/pull-requests:write) does not grant, and which ADR-0014 had deliberately stopped relying on when it moved the cascade to /execute comments (after the identical 403 in issue #121). The fast-path dispatch was never migrated, so on any documented install the App mints its token and then 403s: spec PR merged, tracking issue stranded at sdd:fastpath-review + sdd:approved, no implementation PR.", + "root_cause": "Incomplete ADR-0014 migration \u2014 the cascade fan-out was moved off workflow_dispatch but the fast-path dispatch path kept it \u2014 combined with install docs that under-documented the App permission set the code actually needs (actions:write for fast-path, plus administration:read for the safe-outputs mint).", + "consumer_impact": "Fast-path blocked for every consumer installed per the documented permissions: same stranded-tracking-issue stall as #289, discovered only after #289's fix let the route gate resolve cleanly \u2014 a sequential-403 chain where each fix exposed the next. Documented-install operators must now add two App scopes and re-approve the installation.", + "consumer_refs": [ + "#19 \u2014 staging-consumer fast-path tracking issue in the reproduction ('Failed to dispatch sdd-execute-haiku.yml for fast-path #19: Resource not accessible by integration')" + ], + "fix": "spectacles PR #293 (branch fix/292-app-actions-write, commit 09bc822 'docs(install): document App actions:write (fast-path) + administration:read', merged 8b7c9a1). Notably this is the issue's own 'Alternative (not recommended)' \u2014 a docs-only fix documenting actions:write + administration:read \u2014 not the recommended fix of finishing the ADR-0014 migration to /execute comments; the code still uses workflow_dispatch, so the ADR-0014 contradiction remains latent.", + "detection_path": "Human sequential E2E validation on a staging consumer pinned to fix/route-pr-read-perm (main + PR #290): the route gate now resolved (approved_dispatch=true) and the approved-merge-dispatch job log showed the 403.", + "eval_agent_signal": "Deterministic: (1) docs-as-contract lint \u2014 enumerate every github.rest.actions.createWorkflowDispatch / privileged API call in the suite's actions, map to required App scopes, and diff against the permission set in docs/sdd/install.md (this also mechanically flags ADR drift: any workflow_dispatch usage while ADR-0014 declares the suite must not rely on actions:write); (2) runtime marker \u2014 'POST \u2026/actions/workflows/*/dispatches - 403' + 'Resource not accessible by integration' in the approved-merge-dispatch job; (3) same stranded-label timeline join as #289 (sdd:approved + merged spec PR, no sdd:in-progress), which catches the whole never-dispatches class regardless of which 403 caused it. An inference check on fix PRs could also flag when a fix adopts an option the issue explicitly labeled 'not recommended' against an ADR.", + "category": "permission-contract", + "quotes": [ + "POST /repos/.../actions/workflows/sdd-execute-haiku.yml/dispatches - 403 Failed to dispatch sdd-execute-haiku.yml for fast-path #19: Resource not accessible by integration", + "**ADR-0014** \u2026 moved the *cascade* (`sdd-dispatch`) off `workflow_dispatch` to posting `/execute` comments **precisely because** \"the App token's installation permissions did not include `actions: write` \u2026 `Resource not accessible by integration` on every fan-out cell (issue #121)\"", + "Breaks every consumer installed to the documented permission set \u2014 the fast path cannot complete." + ] + }, + { + "group": "run-week-2", + "number": 294, + "title": "feat(sdd): acknowledge human commands with an emoji reaction at routing time", + "created_at": "2026-06-18T17:25:36Z", + "closed_at": "2026-06-18T19:02:30Z", + "state": "closed (completed)", + "stage": "process-design (route step of every comment wrapper: spec, triage, dispatch, execute, status, derive)", + "failure_mode": "Enhancement closing a feedback gap: a human comments a slash command (/approve, /dispatch, /execute, /revise, /spec, /triage, /status, ...) and gets zero signal until the agent's first visible output minutes later, because runner spin-up + route job + agent container start + model latency all happen invisibly. Users re-comment, assume the command failed, or wait blind. The fix adds a deterministic host-side \ud83d\udc40 reaction at routing time (when should_run == true) and a terminal reaction (\ud83d\ude80 ran / \ud83d\ude15 noop / \ud83d\udc4e failed) mapped purely from job conclusions and declared outputs. A CodeRabbit follow-up fixed the terminal mapping under-reporting failures (upstream compute/lifecycle failure fell through to 'noop'; 'cancelled' mapped to 'ran').", + "root_cause": "No acknowledgment surface existed between command receipt and first agent output; the only feedback channel was the agent's own eventual comment/PR. Deliberately designed as deterministic wrapper code, never an agent action: 'an ack the model might forget, mis-time, or hallucinate is worse than none.' Implemented as composite action .github/actions/sdd-ack-reaction per ADR 0015 (no lock recompile).", + "consumer_impact": "No pipeline breakage; human-side friction on the live run \u2014 duplicate slash commands, assumed failures, blind waiting during every command round-trip. The terminal reaction also gives the pipeline its first per-command machine-readable outcome marker (ran/noop/failed) on the triggering comment itself.", + "consumer_refs": [], + "fix": "spectacles PR #297 (feat commit 6556e45 'acknowledge human commands with a deterministic emoji reaction', follow-up a0c2ee7 'terminal ack reports failed on upstream job failure/cancellation'; merge 7899cbc)", + "detection_path": "Human experience: the repo owner driving the live run felt the dead-air gap after each command; filed proactively as UX/observability design work, not from a red run.", + "eval_agent_signal": "Deterministic comment-to-response join: for every issue/PR comment matching the slash-command grammar where the route job set should_run==true, measure time-to-first-bot-visible-event (reaction, comment, label move, run start); flag any command with no acknowledgment within seconds and any pair of near-identical commands from the same user within a short window (evidence humans are retrying blind). Post-fix regression check: a routed command whose comment carries no \ud83d\udc40-then-terminal reaction sequence is a broken ack path; the terminal reaction itself becomes a machine-checkable per-command outcome (rocket/confused/-1) that a monitor can join against expected artifacts (e.g. created_pr_number) to catch noops and failures without reading logs.", + "category": "observability-gap", + "quotes": [ + "Between the comment and the agent's first visible output there is a gap: runner spin-up + route job + agent container start + model latency. During that gap the user has no signal the command was received. They re-comment, assume it failed, or wait blind.", + "an ack the model might forget, mis-time, or hallucinate is worse than none", + "\ud83d\ude80 (`rocket`) when the run produced its artifact (e.g. `created_pr_number != ''`), \ud83d\ude15 (`confused`) on a logged `noop`/skip, \ud83d\udc4e (`-1`) on job failure" + ] + }, + { + "group": "run-week-2", + "number": 295, + "title": "sdd-triage never starts: plan/materialize jobs under-grant pull-requests (startup_failure)", + "created_at": "2026-06-18T18:05:52Z", + "closed_at": "2026-06-18T18:58:37Z", + "state": "closed (completed)", + "stage": "triage", + "failure_mode": "Every invocation of sdd-triage was a GitHub Actions startup_failure \u2014 the workflow was rejected at load time, before any job ran, so there were no logs, jobs, or check-runs. The entire full (cascade) path was dead at the triage phase: a merged spec PR could never advance the tracking issue past sdd:spec. Shipped latent in the PR #279 per-phase triage split because the fast path never exercises triage.", + "root_cause": "Caller/callee permission contract violation: the plan and materialize jobs in wrappers/sdd-triage.yml granted pull-requests: read, but the reusable locks they call (sdd-triage-plan.lock.yml, sdd-triage-materialize.lock.yml) contain gh-aw safe_outputs/conclusion nested jobs that request pull-requests: write. A reusable callee's job may not exceed the caller job's grant, so GitHub fails the whole workflow at load. The arch job was correct (write); plan/materialize were scoped down on the false assumption that not opening PRs meant read was enough.", + "consumer_impact": "Hard stall of the live full-path run: spec\u2192triage transition impossible on any trigger (spec-PR merge, /triage, /revise on an arch PR); 15/15 startup_failure on the staging consumer. Declared a hard release blocker for v0.3.0.", + "consumer_refs": [], + "fix": "spectacles PR #296 (commit f18891b 'grant plan/materialize jobs pull-requests:write'; merge 314e400). Wrapper-only, no lock recompile. Prevention deferred to #298's gate extension.", + "detection_path": "Red runs surfaced by the issue #89 full-path multi-Unit E2E on a staging consumer; the only visible evidence was the GitHub UI 'Invalid workflow file' annotation, since startup_failure produces no jobs or logs.", + "eval_agent_signal": "Two-layer deterministic check. Static (pre-merge CI): parse every wrapper job that calls a local *.lock.yml and assert the caller job's effective permissions are >= the per-scope maximum across all nested jobs in the called lock (pure YAML analysis; exactly what the gate extension in fa7df42 later implemented). Runtime (monitor): alert on any workflow run with conclusion == 'startup_failure' or zero jobs on an sdd-* workflow \u2014 even a single occurrence is a hard signal because this class is 100% reproducible; plus a state-machine SLA: tracking issue still labeled sdd:spec N minutes after its spec PR merged with no triage run started.", + "category": "permission-contract", + "quotes": [ + "Confirmed 15/15 startup_failure on a staging consumer; surfaced by the issue #89 full-path multi-Unit E2E (the fast path never exercises triage, which is why this stayed latent).", + "The nested job 'conclusion' is requesting 'pull-requests: write', but is only allowed 'pull-requests: read'.", + "which GitHub rejects at workflow-load time (before any job runs \u2014 hence no logs/jobs/check-runs)" + ] + }, + { + "group": "run-week-2", + "number": 298, + "title": "Two more wrappers under-grant permissions vs called locks (startup_failure); contract gate misses the class", + "created_at": "2026-06-18T20:52:34Z", + "closed_at": "2026-06-18T23:57:52Z", + "state": "closed (completed)", + "stage": "spec (sdd-derive) + triage/monitor re-entry (sdd-spike-reentry) + infra (contract gate)", + "failure_mode": "Same caller>=callee permission class as #295, found in two more wrappers the fast/full happy path never exercised (issue body is truncated in the API; reconstructed from fix PR #299 commits). sdd-spike-reentry.yml's sdd-triage job granted pull-requests: read while calling sdd-triage-plan.lock \u2014 startup_failure on every issue close/unlabel, 20/20 on the staging consumer. sdd-derive.yml's derive job lacked discussions: write, which sdd-derive.lock's safe_outputs request \u2014 so /derive-spec was completely dead, and sdd-derive is default-installed as of #291. Three instances of the class (#295 + these two) had reached main.", + "root_cause": "The wrapper-lock contract gate added in #291 only validated workflow_call with:/secrets: (the #284 class), not caller-job permissions against the union of callee nested-job permissions \u2014 a blind spot letting the whole startup_failure class through CI repeatedly.", + "consumer_impact": "Silent dead surfaces on the live run: /derive-spec produced nothing (startup_failure leaves no jobs/logs), and every issue close/unlabel fired a failing spike-reentry run (20/20), breaking the spike re-entry path and generating failure noise. Would-be stall/needs-human on any flow touching those wrappers.", + "consumer_refs": [], + "fix": "spectacles PR #299 (commit 3cdaad8 'grant caller jobs the permissions their called locks need' + commit fa7df42 'ci(contract): enforce caller >= callee job permissions in wrapper-lock gate', verified to fail on the pre-fix tree; merge dbab37c)", + "detection_path": "Human class-sweep after #295: the owner audited all wrappers for the same pattern hours after the first instance, confirmed by 20/20 red startup_failures on the staging consumer.", + "eval_agent_signal": "Same static invariant as #295, now generalized: CI gate computing each lock's per-scope permission ceiling (max over nested jobs) and asserting every calling wrapper job grants >= it \u2014 machine-checkable from YAML alone, no run needed. The meta-signal for an eval agent: when a fix lands for a structural class (here, a caller/callee grant mismatch), automatically sweep all sibling artifacts for the identical pattern instead of waiting for the next instance to fire \u2014 #298 is literally 'two more of #295' found by manual sweep. Runtime backstop: any startup_failure conclusion, and coverage analysis flagging wrappers whose trigger paths (issue close/unlabel, /derive-spec) are never exercised by the happy-path E2E.", + "category": "permission-contract", + "quotes": [ + "fired startup_failure on every issue close/unlabel \u2014 20/20 on a staging consumer", + "sdd-derive is default-installed as of #291, so /derive-spec was dead", + "The wrapper->lock contract gate checked with:/secrets: (the #284 class) but not job permissions, so the caller>=callee permission class kept reaching main (#295 sdd-triage, #298 sdd-spike-reentry + sdd-derive \u2014 three startup_failures)." + ] + }, + { + "group": "run-week-2", + "number": 300, + "title": "Full path stalls at spec\u2192triage: triage agents lack roles:all, so App-applied sdd:triage can't activate", + "created_at": "2026-06-19T00:06:04Z", + "closed_at": "2026-06-19T04:51:03Z", + "state": "closed (completed)", + "stage": "triage (spec\u2192triage lifecycle transition)", + "failure_mode": "The autonomous full path silently stalled at spec\u2192triage: when sdd-spec applied the sdd:triage label under the App identity (the documented auto-start of the architecture phase), the sdd-triage-arch agent skipped silently \u2014 pre_activation set activated=false because github.actor was spectacles-bot[bot] (collaborator permission 'none'), and the run still reported overall SUCCESS, a silent no-op with no red signal. Only a human re-applying the label advanced the pipeline. Release-blocking for v0.3.0 since fully App-driven E2E was the acceptance condition.", + "root_cause": "The three per-phase triage agent sources (sdd-triage-arch.md, sdd-triage-plan.md, sdd-triage-materialize.md) declared no roles: key, so gh-aw compiled its restrictive activation gate (pre_activation running check_membership with GH_AW_REQUIRED_ROLES admin,maintainer,write) that evaluates the triggering actor \u2014 which on the issues:labeled event is the App bot. Every other automation-driven agent (execute, dispatch, derive, review, validate) declares roles: all and relies on its route action for actor authorization; triage was never given it. Pre-existing latent bug from before the per-phase split, only surfaced under fully App-driven operation.", + "consumer_impact": "Full-path stall requiring a human re-label at every spec\u2192triage transition \u2014 worse than a red failure because runs were green: the pipeline appeared healthy while doing nothing. Broke the 'no human action is required' contract of the App-driven cascade.", + "consumer_refs": [], + "fix": "spectacles PR #302 (commit edc150a 'fix(sdd-triage): roles:all so App-applied sdd:triage activates the agent' + lock recompile removing pre_activation from all three triage locks; merge 20dc641)", + "detection_path": "Human watching the #89 full-path App-driven E2E on staging: the label applied but nothing ran; diagnosis via job-level inspection of run 27785185404 (agent jobs skipped, gate log line) contrasted with human-actor run 27785333803 (all jobs success).", + "eval_agent_signal": "Silent no-op detector via run-status join: flag any sdd agent run whose overall conclusion is success but whose agent/safe_outputs jobs are all 'skipped' (especially with pre_activation success) \u2014 the exact shape of run 27785185404; deterministic log marker 'User permission ... does not meet requirements'. Label-event SLA: sdd:triage (or any activating lifecycle label) applied by the App with no corresponding agent job execution within N minutes. Static lock invariant: every agent source triggered by App-applied events must declare roles: all / every compiled lock on an automation path must contain no pre_activation job \u2014 checkable by grep over sources and locks in CI.", + "category": "permission-contract", + "quotes": [ + "Staging run 27785185404 (actor `spectacles-bot[bot]`): arch `pre_activation`\u2192success, `activation`/`agent`/`safe_outputs`\u2192skipped, log `User permission 'none' does not meet requirements: admin, maintainer, write`.", + "activated=false \u2192 arch activation/agent/safe_outputs all skipped (run reports success \u2014 a silent no-op)", + "Run 27785333803 (actor `norrietaylor`, human re-label): all arch jobs success." + ] + }, + { + "group": "run-week-2", + "number": 301, + "title": "Close-event cascade misses single-task (feature-parented) closures: hardcoded 2-hop walk vs ADR-0028", + "created_at": "2026-06-19T00:06:05Z", + "closed_at": "2026-06-19T15:01:30Z", + "state": "closed (completed)", + "stage": "dispatch (issues:closed cascade re-fire)", + "failure_mode": "When a single-task feature's only task closed (its impl PR merged), the issues:closed cascade did not advance the tracker toward sdd:done \u2014 the route declined the event and the tracking issue stalled until a manual /dispatch (or the sdd-monitor backstop). Filed non-blocking because the documented nudge/backstop covers it; fixed as fast-follow for v0.3.1.", + "root_cause": "Spec/code drift against ADR-0028: sdd-route-dispatch's close-event branch hardcoded a two-hop tree (Feature\u2192Unit\u2192task) via walkViaGraphql plus gates at ~line 495 (gqlWalked === 2) and ~line 544 (walked === 2). ADR-0028 collapses a single-task Unit into a feature-parented task \u2014 a direct 1-hop child of the tracker \u2014 so the walk resolved correctly in 1 hop but 1 !== 2 misclassified the task closure as a Unit/tracker closure and set should_run=false. Asymmetry: sdd-dispatch-compute was already ADR-0028-aware (admits a direct '## Task' child, ~lines 191-204), which is why a manual /dispatch succeeded.", + "consumer_impact": "Stall at the final done-transition of the live full-path E2E on the consumer/staging repo (tracking issue #29, task #36): tracker sat with all work merged but not sdd:done until a human /dispatch nudge \u2014 needs-human rework rather than a broken artifact.", + "consumer_refs": [ + "#29 \u2014 consumer/staging tracking issue on the full-path E2E whose cascade never re-armed", + "#36 \u2014 the single-task (feature-parented, '## Task', depends on: (none)) child whose closure was misclassified" + ], + "fix": "spectacles PR #305 (commit 77e4e65 'fix(dispatch): arm cascade on collapsed single-task (1-hop) closures (Gap B)' \u2014 accept walked in {1,2} when the chain terminates at a parentless tracker (stoppedBy==='none') and the closed body has a '## Task' block; composite-action only, no lock recompile; merge 73631a6)", + "detection_path": "Human observing the full-path E2E stall; the deterministic route decline was already in the logs of run 27788190031: 'Closed issue #36 is not a task sub-issue closure (walked 1 hops); ignoring.'", + "eval_agent_signal": "State-machine invariant check: for every closed issue whose body contains a '## Task' block and whose sub-issue parent chain terminates at a parentless tracker, a dispatch/cascade run must fire within N minutes \u2014 join issue-close timeline events against workflow-run starts. Cheap deterministic log marker: route decline lines matching 'walked 1 hops; ignoring' on issues whose body starts '## Task' are always this bug. Broader timeline shape: tracker whose child issues are all closed yet lacks sdd:done after an SLA. And a diff-time inference check: when an ADR changes a structural invariant (tree depth/shape, per ADR-0028), sweep all code paths encoding the old invariant (hardcoded hop counts, ===2 gates) for un-updated consumers.", + "category": "state-machine/lifecycle", + "quotes": [ + "route logged \"Closed issue #36 is not a task sub-issue closure (walked 1 hops); ignoring.\"", + "the GraphQL parent walk resolves correctly to the tracker in **1 hop**, but `1 !== 2`, so the code misclassifies the task closure as a Unit/tracker closure and sets `should_run=false`", + "the gates at ~line 495 (`gqlWalked === 2`) and ~line 544 (`walked === 2`)" + ] + }, + { + "group": "run-week-3", + "number": 303, + "title": "Fast-path execute no-ops on the haiku tier: agent doesn't follow the situation-1a entry, emits zero outputs", + "created_at": "2026-06-19T07:12:32Z", + "closed_at": "2026-06-19T15:01:39Z", + "state": "closed", + "stage": "execute|dispatch", + "failure_mode": "With the fast-path dispatch finally firing (B3 fixed), the dispatched fast-path execute agent classified to the haiku tier produced no implementation at all: the workflow run concluded 'success' but agent_output.json was {\"items\":[]}, detection/safe_outputs steps skipped, no sdd/ branch or PR was created, and the tracking issue stalled at sdd:in-progress. The fast path (/agile \u2192 /approve) therefore could not complete for exactly the small changes it exists for. The issue explicitly rules out a missing-directive bug: the execute prompt's situation-1a entry was fully followable and the dispatch input (entry:'fastpath', item_type:'issue', item_number:49, plan_comment_id, tier:'haiku') was correct.", + "root_cause": "Model-tier fragility: the haiku-tier agent did not execute the multi-step, descriptive/procedural situation-1a entry. Per its transcript it decided no one had asked it to do anything, ran a single turn, replied 'What would you like to help with?', and emitted zero safe outputs. The same execute prompt succeeded on sonnet (full-path task #45 produced a real merged impl PR), so the prompt exceeded what haiku reliably executes.", + "consumer_impact": "Silent stall on the live pilot: the fast-path implementation no-oped, the tracking issue (item 49) stranded at sdd:in-progress with no PR and no escalation, and the target change (--version flag in src/main.rs on staging) never landed until a human diagnosed the empty output and the tier floor was raised. Wasted a full dispatch+execute run's tokens for zero output.", + "consumer_refs": [ + "#49 \u2014 the fast-path tracking issue on the pilot/staging repo (aw_context item_number:49, tier haiku) whose implementation no-oped", + "#45 \u2014 the full-path task on the same repo that ran on sonnet (model:sonnet) and produced a real merged impl PR, used as the contrast case" + ], + "fix": "spectacles PR #304 (merge f8aad26), commit 25305c4 'fix(fastpath): floor the single-PR implementation tier at sonnet (B7)' \u2014 option 2 from the issue: never let the fast-path implementation step run on haiku", + "detection_path": "Human watching the pilot run: after confirming dispatch run 27810873003 succeeded, the maintainer inspected execute run 27810885047's artifacts, found agent_output.json empty and detection/safe_outputs skipped, then read the agent transcript to see the one-turn 'ask the user' behavior.", + "eval_agent_signal": "Deterministic join of run conclusion vs. safe-output emission: any sdd-execute run whose trigger context promises an implementation (entry='fastpath' or a task work-item) that concludes 'success' with agent_output.json items==[] (zero create_pull_request safe outputs) is an anomaly \u2014 flag immediately rather than letting the label state ('sdd:in-progress' with a completed run but no sdd/* branch/PR within N minutes) reveal it. Secondary: per-model-tier telemetry on output-emission rate (haiku vs sonnet success producing >=1 safe output); inference-based backstop: an LLM judge over the transcript flagging 'agent asked the user what they want / treated the directive as system context' single-turn no-op shapes.", + "category": "prompt/model-behavior", + "quotes": [ + "the run completes \"success\" but `agent_output.json = {\"items\":[]}` (no `create_pull_request`), `detection`/`safe_outputs` skip, the tracking issue stalls at `sdd:in-progress`", + "Per its transcript it reasoned \"they haven't actually asked me to do anything\u2026 it's all system context\u2026 I should ask the user what they want,\" ran one turn, replied \"What would you like to help with?\", and emitted zero safe outputs.", + "The only execute runs observed to succeed (the full-path multi-Unit run) ran on **sonnet** (task #45, `model:sonnet`) and produced a real impl PR." + ] + }, + { + "group": "run-week-3", + "number": 307, + "title": "Unspecced merged pull requests (bot-filed batch surface listing spec debt)", + "created_at": "2026-06-22T13:33:47Z", + "closed_at": "", + "state": "open", + "stage": "spec|process-design", + "failure_mode": "Not an outage but the ADR 0027 backstop working: spectacles-bot's retrospective spec-derivation scan filed a batch issue listing three PRs merged into spectacles itself with no accompanying spec (no sdd/ lineage, over the SDD_SPEC_MIN_UNIT=400-line floor): #291 (~403 lines, v0.3.0 release prep), #277 (~624 lines, triage single-task-Unit collapse), and #274 (~3354 lines, the copilot\u2192claude gh-aw workflow port). The pipeline's own repo bypassed its spec discipline for ~4,400 lines of merged change; the issue offers '/derive-spec #291 #277 #274' to open gap-analysis spec PRs. As of harvest it has zero comments \u2014 the spec debt is surfaced but unactioned.", + "root_cause": "The SDD pipeline is forward-only (spec \u2192 triage \u2192 execute); direct-to-code development \u2014 release chores, large migrations, exploratory branches \u2014 produces merged code with no spec, and until ADR 0027 nothing covered non-sdd/ head branches. The listed PRs predate or bypassed the sdd-derive offer flow, so the batch sweep caught them retroactively.", + "consumer_impact": "None on the live consumer run directly; the impact is latent \u2014 unspecced code (including the 3,354-line engine migration in #274) has no reviewable spec, no requirement-ID anchor, and no distillery-sync knowledge-store entry, degrading every downstream agent that grounds itself in specs. The open, unactioned checklist means the debt persists.", + "consumer_refs": [], + "fix": "open \u2014 the detection mechanism itself shipped earlier (sdd-derive, commit c009887 'feat(agents): add sdd-derive retrospective spec derivation (ADR 0027)', PR #276; tone later softened in d9e6fc6), but the listed specs have not been derived; the checklist items are unchecked", + "detection_path": "Automated: spectacles-bot's deterministic size-gated batch scan (sdd-derive detection per ADR 0027) filed the issue itself \u2014 no engine, mechanical check of lineage markers and net diff size.", + "eval_agent_signal": "The detection class is already machine-checked (diff property: merged PR with head ref not sdd/|spec/|arch/, no 'Closes #' in body, no docs/specs/** reference, net diff >= SDD_SPEC_MIN_UNIT). The eval-agent signal to add is staleness of the surface: an open bot-filed unspecced-PR batch issue with unchecked checklist items and zero '/derive-spec' comment events after N days is actionable spec debt; also a repo-level ratio metric (unspecced merged lines / total merged lines per release window) with the ~3354-line single-PR outlier as an alert threshold.", + "category": "process-design", + "quotes": [ + "These pull requests merged without an accompanying spec (no `sdd/` lineage, over the `SDD_SPEC_MIN_UNIT` floor).", + "#274 \u2014 Port gh-aw workflows from copilot to claude (CLAUDE_CODE_OAUTH_TOKEN) (~3354 lines)", + "#291 \u2014 chore: release-prep for v0.3.0 (contract gate, anonymization, installer fixes) (~403 lines)" + ] + }, + { + "group": "run-week-3", + "number": 308, + "title": "sdd-execute: a re-triggered run can be cancelled at the activation job by concurrent same-tier execute activity on a different task", + "created_at": "2026-06-22T17:02:46Z", + "closed_at": "2026-06-22T17:28:12Z", + "state": "closed", + "stage": "execute|dispatch", + "failure_mode": "On the consumer pilot run, a manually re-triggered sdd-execute run (/execute on a task sub-issue) was cancelled at its activation job in the same minute that unrelated sdd-execute activity (PR review / review-comment / revise events) fired for a different task of the same tier. Run shape: route=success, ack=success, activation=cancelled, agent job never started, run conclusion 'cancelled' after ~3 min. The task stranded at sdd:in-progress with no run, no PR, no needs-human, no [aw] marker, and did not auto-recover \u2014 the dispatch cascade only re-fires on task closure and skips in-flight-marked tasks.", + "root_cause": "The execute wrapper's workflow-level concurrency group was not strictly per-item across all trigger types: issue_comment /execute keyed on the item number, but PR-attached triggers (pull_request_review, pull_request_review_comment, implicit /revise) resolved a colliding group, so cancel-in-progress let cross-task same-tier events kill an unrelated in-flight run at activation.", + "consumer_impact": "Silent stall requiring human rework: the re-triggered task was killed by a sibling task's PR activity with no escalation and no auto-retry, and needed a manual /execute re-bump. More broadly it made parallel work on multiple same-tier tasks fragile \u2014 touching one task's PR could kill another task's freshly dispatched run.", + "consumer_refs": [], + "fix": "spectacles PR #309 (merge 79e9bf8), commit d641900 'fix(sdd-execute): key the workflow concurrency group per-item (#308)'", + "detection_path": "Human watching the consumer pilot run noticed the stranded task and inspected the cancelled run's job shape; the issue even documents the grep-able repro signal for future automation.", + "eval_agent_signal": "Deterministic run-status join, spelled out verbatim in the issue: flag any execute run with route=success, ack=success, ' / activation'=cancelled, run conclusion=cancelled, and no agent job \u2014 especially when time-correlated with another same-tier execute run's activation window. Complement with a lifecycle invariant check: an issue labeled sdd:in-progress whose most recent execute run is cancelled and which has no live run, PR, or needs-human label after M minutes is a stranded task. Both are pure API joins, no inference needed.", + "category": "concurrency/race", + "quotes": [ + "`route` \u2192 success / `ack` \u2192 success / `sdd-execute- / activation` \u2192 **cancelled** / agent job never started; run `conclusion: cancelled` (~3 min)", + "The task was left stranded \u2014 `sdd:in-progress`, no run, no PR, no `needs-human`, no `[aw]` \u2014 and did **not** auto-recover", + "touching one task's PR can kill another task's freshly-dispatched run" + ] + }, + { + "group": "run-week-3", + "number": 310, + "title": "sdd-ack-reaction: concurrency-cancelled revise runs stamp misleading thumbs-down on consumed review comments", + "created_at": "2026-06-22T19:43:55Z", + "closed_at": "", + "state": "open", + "stage": "execute|status", + "failure_mode": "On the consumer pilot run, nearly every review comment on a PR \u2014 both the external reviewer's and sdd-review's own findings \u2014 carried a thumbs-down reaction, reading as 'these findings failed / were rejected', when in fact every finding was consumed and the PR was progressing normally. No work was lost (the surviving revise run's assembled directive from #247 consumes all unresolved threads), but the visible signal said the opposite.", + "root_cause": "Interaction of three designed behaviors: (1) sdd-ack-reaction terminal mode maps run outcome onto the triggering comment as rocket=ran / confused=noop / thumbs-down=failed, treating concurrency cancellation as non-success; (2) the wrapper routes every review comment to sdd-execute as a revise trigger, so a reviewer posting N findings at once spawns N revise runs in the same tick; (3) the per-item concurrency group (#308/#309) serializes them, cancelling N-1 as superseded \u2014 and each cancelled sibling's comment gets stamped thumbs-down.", + "consumer_impact": "Misleading human signal on a live PR: a maintainer or reviewer scanning the PR sees a wall of thumbs-down on legitimate in-progress findings and reasonably concludes the review was rejected or agent runs failed; it also camouflages genuine failures, which use the identical reaction. No functional loss, but real trust/triage cost.", + "consumer_refs": [], + "fix": "open \u2014 no fix commit found in the checkout as of 2026-07-01; the offending mapping was introduced by a0c2ee7 'fix(sdd): terminal ack reports failed on upstream job failure/cancellation' (part of the #294/#297 ack-reaction feature); issue recommends collapsing a review burst into one revise run plus never stamping thumbs-down for a concurrency cancel", + "detection_path": "Human watching the consumer pilot run \u2014 a maintainer scanning the PR saw the wall of thumbs-down reactions on comments for a PR that was actually progressing.", + "eval_agent_signal": "Deterministic reaction-vs-run join: for each bot thumbs-down reaction, resolve the backing workflow run; if its conclusion is 'cancelled' (concurrency-superseded) rather than 'failure', the reaction is a false failure signal. Burst-shape heuristic: >=2 bot thumbs-down reactions stamped on review comments of one PR within the same minute, coincident with exactly one surviving successful revise run, is this class with near-certainty. Inference backstop: an LLM judge comparing a PR's reaction trail against its run outcomes and thread-consumption state to flag signal/reality divergence.", + "category": "observability-gap", + "quotes": [ + "nearly every review comment on a PR (both the external reviewer's and `sdd-review`'s own findings) carried a \ud83d\udc4e reaction, reading as \"these findings failed / were rejected\" \u2014 when in fact they were consumed and the PR was progressing normally", + "a burst of N review comments spawns N revise runs in the same tick", + "\ud83d\ude80 ran / \ud83d\ude15 noop / **\ud83d\udc4e failed**" + ] + }, + { + "group": "run-week-3", + "number": 311, + "title": "Auto-revise must resolve each addressed review thread with an inline resolution note (valid+sha / invalid+reason)", + "created_at": "2026-06-22T19:46:39Z", + "closed_at": "", + "state": "open", + "stage": "execute|review", + "failure_mode": "The revise pass closes no loops: step 7 of sdd-execute edits and pushes fixes but posts no per-thread note and resolves no individual threads, while the sdd-resolve-review-threads action blanket-resolves the App bot's own unresolved threads (to keep required_conversation_resolution from deadlocking merge) with no validity judgment \u2014 resolving even findings that were never addressed, silently hiding them. Third-party threads (CodeRabbit, humans) are never replied to or resolved at all. A maintainer cannot tell per finding whether it was fixed, dismissed, or buried. The 2026-06-24 bump reports this hit every review-fix PR in a live feature cascade and became a throughput bottleneck absorbed by manual maintainer effort.", + "root_cause": "Designed-behavior gap: thread resolution was implemented as a mechanical merge-unblock sweep rather than a reasoned validate\u2192reply\u2192resolve discipline (the coderabbit-loop pattern). The wrapper already holds both identifiers needed (REST databaseId for in_reply_to, GraphQL thread node id for resolveReviewThread) when assembling the #247 directive, so the gap is purely in agent behavior/prompt contract, compounded in the cascade by revise runs being repeatedly concurrency-cancelled before completing.", + "consumer_impact": "Sustained manual rework on the live cascade: a maintainer hand-did the agent's job on each converging review-fix PR \u2014 validating each thread against HEAD, manually fixing valid ones (clippy collapsible_if, commitlint subject reword, a dropped thiserror dep, a silently-deleted unrelated spec doc), replying with fixing shas, rebutting stale findings, resolving threads. In one worked example 7 findings needed manual triage (3 valid, 4 already fixed); findings sat red ~an hour until manual takeover; stale findings re-escalated to needs-human after every push. Review-fix PRs did not self-resolve.", + "consumer_refs": [], + "fix": "open \u2014 no fix commit in the checkout as of 2026-07-01; refs spectacles #247 (assembled revise directive) as the substrate the fix builds on", + "detection_path": "Human maintainer doing the manual per-thread triage across the consumer cascade; the 2026-06-24 owner bump comment documents recurrence and the escalation to throughput-bottleneck priority.", + "eval_agent_signal": "Deterministic thread-resolution audit via the GraphQL review-threads API after each revise run: count threads transitioned to resolved where the resolving actor is the bot and the thread's last comment is not a bot reply containing a verdict marker (a commit sha or a rebuttal citation) \u2014 resolved-without-reasoned-reply > 0 means findings are being buried. Companion invariants: unresolved third-party threads older than N revise cycles with zero bot replies; and threads resolved whose referenced finding text still matches current HEAD (never addressed). Inference backstop: an LLM judge per resolved thread scoring whether the closing reply actually validates or rebuts the finding (the issue's 'Fixed in ' / invalid+reason contract is directly judgeable).", + "category": "process-design", + "quotes": [ + "**blanket-resolves the App bot's own unresolved threads** so `required_conversation_resolution` does not deadlock merge. It applies no validity judgment and resolves **even findings that were never addressed** \u2014 silently hiding them.", + "a revise pass left 7 findings open; manual triage validated each (3 valid \u2192 `/revise`, 4 already fixed \u2192 inline note citing the fixing sha + resolve)", + "revise runs repeatedly concurrency-cancelled before completing) meant findings sat red for ~an hour until manual takeover" + ] + }, + { + "group": "run-week-4", + "number": 312, + "title": "Auto-revise commit subjects fail consumer commitlint \u2014 Pre-PR gate should run it", + "created_at": "2026-06-22T20:17:24Z", + "state": "open", + "closed_at": "", + "stage": "execute", + "failure_mode": "The sdd-execute Pre-PR CI gate runs fmt/clippy/test but not the consumer's commitlint, so the agent pushed commits whose subjects violated the consumer's commit rules (a 104-char header against a 100-char limit; an upper-case-start subject violating subject-case). Each violation left the PR BLOCKED on the consumer's commitlint check with no in-sandbox signal, and required a human force-push history rewrite to reword the offending commit since it was not always the tip commit.", + "root_cause": "The Pre-PR gate mirrors only code checks (fmt/clippy/test); consumer commit-message policy (commitlint.config.*, .commitlintrc*, package.json commitlint key) is neither executed in the gate nor encoded into the agent's subject generation, so violations are only discoverable after push via consumer CI. Step 7's generic conventional-commit guidance does not capture the consumer's specific header-max-length / subject-case / allowed types-scopes rules.", + "consumer_impact": "Merge-blocking rework, twice in one pilot run: the PR sat BLOCKED on the commitlint check until a human performed a force-push history rewrite to reword the bad commit. Stall plus needs-human-equivalent manual intervention each time.", + "consumer_refs": [], + "fix": "open (proposed: run consumer commitlint --from as a hard-fail Pre-PR gate step when a config is present; secondarily, read header-max-length/subject-case and constrain generated subjects)", + "detection_path": "Red consumer CI (commitlint check blocking merge) noticed by a human who then had to reword/force-push; no in-sandbox or pipeline-side detection.", + "eval_agent_signal": "Deterministic run-status join: a check-run named commitlint (or matching the consumer's commit-lint job) concluding failure on a PR whose head commits are bot-authored, occurring after a pipeline push event \u2014 flag on first occurrence, and escalate on repeat within the same PR. Cheaper pre-push variant: deterministically lint the agent's staged commit subjects against the consumer's discovered commitlint config (header length <= N, subject-case regex) and alert on any would-fail subject before push. A timeline shape of 'bot push -> commitlint check failure -> human force-push' recurring is the class signature.", + "category": "process-design", + "quotes": [ + "A 104-char commit header (`header-max-length` = 100).", + "A subject starting upper-case (`R1.4 exit signal, \u2026`) \u2192 `subject-case` violation (must not be sentence/start/pascal/upper-case).", + "Each left the PR `BLOCKED` on `commitlint` until a human reworded the offending commit (a force-push history rewrite, since the bad commit was not always the tip)." + ] + }, + { + "group": "run-week-4", + "number": 313, + "title": "A human-issued /revise is gated behind needs-human and silently no-ops until the label is cleared", + "created_at": "2026-06-22T20:41:11Z", + "state": "open", + "closed_at": "", + "stage": "dispatch|review", + "failure_mode": "When the auto-revise loop exhausts attempts it applies needs-human to hand off. A human then posts /revise with precise instructions \u2014 but while needs-human is present the /revise is gated: no execute dispatches, no reaction, no comment, total silence. The human must independently notice the label, manually remove it, and only then does the same /revise dispatch. The hand-off label and the very human input it requests collide.", + "root_cause": "Trigger routing unconditionally suppresses /revise (and pull_request_review_comment) dispatch while needs-human is set, without distinguishing human actors from the bot; a human /revise is not treated as the resolving input the documented contract describes ('A human resolves the comment and clears the label, which re-triggers this agent'), and the ignored trigger emits no acknowledgment.", + "consumer_impact": "Stall + wasted human effort on every post-hand-off revise: the human's validated instructions sat ignored for the duration until a manual label-clear; sdd-validate had already reported no blockers, so the PR was needlessly frozen. Recurring per hand-off, with zero feedback that the comment was dropped.", + "consumer_refs": [], + "fix": "open (proposed: treat a human /revise on a needs-human PR as the resolving input \u2014 clear the label and dispatch; if a trigger must be ignored, react/comment instead of silent no-op; distinguish human vs bot actors)", + "detection_path": "Human watching the PR: posted /revise ~6s after the hand-off, observed no execute dispatch, then manually removed the label to unblock.", + "eval_agent_signal": "Deterministic timeline join on the consumer PR: (a) needs-human labeled event, (b) a human-authored (non-bot) /revise or review comment posted while the label is present, (c) no workflow-run of the execute agent keyed to that comment within N minutes, then (d) a human unlabeled event followed by a dispatch of the same instruction. The (b)-without-(c) gap is the alert; the (d) pattern confirms the class. Requires joining issue-comment events, label events, and workflow-run starts with actor-type (User vs Bot) discrimination \u2014 fully machine-checkable, no LLM needed.", + "category": "state-machine/lifecycle", + "quotes": [ + "Human posts `/revise` with precise, validated instructions ~6s later.", + "No execute dispatches. The `/revise` sits silently behind the label; `sdd-validate` had already reported **no blockers**.", + "every human-directed revise after a hand-off needs a manual label-clear first, with no signal that the comment was ignored." + ] + }, + { + "group": "run-week-4", + "number": 314, + "title": "sdd-validate should not run build/proof tasks that require egress \u2014 firewall block must not trigger needs-human", + "created_at": "2026-06-22T20:45:24Z", + "state": "open", + "closed_at": "", + "stage": "validate", + "failure_mode": "sdd-validate Gate 1 tries to re-execute the consumer's build/test proofs inside the gh-aw sandbox, which by design has no Rust toolchain and no crates.io/package-registry egress. When the fetch/build is firewall-blocked, validate classifies it as a blocker and applies needs-human \u2014 a false positive on an infrastructure limitation. Behavior is inconsistent: some runs defer gracefully with no blocker, others flip to needs-human on the identical condition.", + "root_cause": "Gate 1 attempts egress-dependent proof execution in an environment where it can never succeed, and lacks a deterministic classifier that maps firewall/egress denial to the existing 'deferred to consumer CI' Info path instead of the blocker path; the authoritative proof gate (consumer CI test/netns/e2e jobs) is already acknowledged but not consistently relied on.", + "consumer_impact": "Recurring stall/label churn: combined with #313's gating, every push re-ran validate, which firewall-blocked and re-applied needs-human \u2014 the 'PR keeps getting needs-human, clear it, it comes back on the next push' loop, each cycle needing a human label-clear despite no real defect.", + "consumer_refs": [], + "fix": "open (proposed: detect missing toolchain/registry and deterministically defer to consumer CI without attempting the build; classify firewall/egress denial as Info-level infrastructure deferral, never needs-human; surface the deferral in the status comment)", + "detection_path": "Human watching the consumer PR observed the needs-human label re-appearing after every push and correlated it with firewall-blocked build attempts in validate run logs.", + "eval_agent_signal": "Two deterministic signals: (1) run-log/label join \u2014 a validate workflow run whose log contains egress-denial markers (firewall-blocked domain, 'no crates.io egress', registry fetch failure) AND whose safe-outputs include applying needs-human in the same run; any co-occurrence is a false-positive blocker by policy. (2) Label-flap detector \u2014 needs-human applied -> removed -> re-applied across consecutive pushes to the same PR with no new review findings between cycles (label-event timeline shape). Also an invariance check: identical inputs producing 'No Blockers' on one run and needs-human on another flags the nondeterminism directly.", + "category": "infra/config", + "quotes": [ + "The sandbox has **no toolchain and no registry egress** for the consumer's stack \u2014 re-executing `cargo build`/`cargo test` there can never succeed and is not the intended gate.", + "\"No Rust toolchain in gh-aw container (no crates.io egress) \u2192 deferred to consumer CI\", result \u2705 No Blockers", + "this produces the recurring \"PR keeps getting `needs-human`, clear it, it comes back on the next push\" loop seen on a consumer pilot run" + ] + }, + { + "group": "run-week-4", + "number": 315, + "title": "distillery-sync fails every run: Haiku reads the descriptive prompt as a spec and asks for clarification instead of syncing", + "created_at": "2026-06-22T21:01:08Z", + "state": "closed", + "closed_at": "2026-06-22T21:18:36Z", + "stage": "distillery|status", + "failure_mode": "Every scheduled/push distillery-sync run failed with 'No Safe Outputs Generated': the agent, pinned to claude-haiku-4-5, read its own third-person instruction prompt ('This agentic workflow keeps the Distillery knowledge store current\u2026') as a specification handed to it for review, asked a clarifying question in a headless run with nobody to answer, made zero tool calls, and never called noop. Compounding: the terminal status-issue upsert never ran, so the self-updating status issue silently went stale (last updated 2026-06-18 while failing runs began 2026-06-20), hiding the failure from anyone watching the status issue.", + "root_cause": "Prompt framing vs model: descriptive third-person prompt with no imperative opening, no headless directive ('never ask for clarification'), and no mandatory terminal safe-output requirement \u2014 against a Haiku model pinned intentionally for cost and the 25M effective-token firewall cap, which takes the descriptive framing literally instead of inferring the imperative. Not a connectivity problem: MCP registered fine, mcp_failures null.", + "consumer_impact": "No direct PR impact, but the Distillery knowledge store stopped syncing entirely \u2014 every scheduled run wasted (failed at num_turns=1) from 2026-06-20 onward, the sync cursor froze, and the status issue showed 'last synced 4 days ago' with no error surfaced. Silent knowledge-freshness degradation plus wasted scheduled-run cost.", + "consumer_refs": [], + "fix": "spectacles PR #316 (merge 44b9cb8): a496231 'fix(distillery-sync): reframe prompt imperatively to prevent empty-output runs' + 7086545 'fix(distillery-sync): declare noop in safe-outputs allowlist' \u2014 imperative second-person opening, headless never-ask directive, mandatory terminal status-issue upsert/noop; model stays pinned to claude-haiku-4-5.", + "detection_path": "Red run status ('No Safe Outputs Generated' on every scheduled run) plus reading the agent's transcript/thinking in run 27865550618; the stale status issue was a secondary human observation.", + "eval_agent_signal": "Deterministic telemetry check: any headless/scheduled agent run with num_turns == 1 and tool_calls == 0 (tool_types=0) is a hard anomaly regardless of conclusion; join with conclusion=failure or 'No Safe Outputs Generated' for certainty. Inference-based complement: an LLM judge (or even a regex for 'Please clarify' / trailing question mark) over the final assistant message on a non-interactive run flags clarification-question dead ends. Structural monitor: status-issue updated_at older than the most recent scheduled run's start time == guaranteed missed terminal upsert \u2014 catches the whole class even when the run is green.", + "category": "prompt/model-behavior", + "quotes": [ + "The agent ran **`num_turns=1`, zero tool calls** (`tool_types=0`, read-only), then ended.", + "Model is pinned to `claude-haiku-4-5` (intentionally \u2014 mechanical sync, cost, the 25M effective-token firewall cap on full-glob backfill).", + "its status issue was last updated 2026-06-18 and untouched since, while the failing scheduled runs began 2026-06-20" + ] + }, + { + "group": "run-week-4", + "number": 317, + "title": "sdd-spec opened a non-docs-only PR: the derive commit reverted merged files and added a code refactor alongside the spec", + "created_at": "2026-06-22T21:43:27Z", + "state": "open", + "closed_at": "", + "stage": "spec", + "failure_mode": "An sdd-spec derive run opened a spec PR whose single commit changed +3125/-2665 lines across ~30 code files in addition to the one legitimate spec document. Relative to base, the commit both reverted merged work (e.g. composer.rs gutted -2188, RESOLUTION.md deleted -194) and added an unrelated +1758-line code refactor plus Cargo.lock/Cargo.toml churn and client-to-core file moves. Merging it would have silently reverted recently-merged code; the bot's PR title/report described only a spec derivation, masking the corruption.", + "root_cause": "The derive commit was not scoped to the spec path \u2014 it captured the full working-tree diff (effectively git add -A) instead of only the authored file under docs/specs/**. Any checkout divergence from base (stale/contaminated checkout, merged-after-checkout changes, stray generated files) got committed, and where base was ahead the divergence inverted into reverts. No docs-only diff assertion existed before publishing. Spec-side analogue of the merged-change revert the execute agent already guards against (#287).", + "consumer_impact": "Near-miss silent revert: merging would have undone recently-merged code across ~30 files. Actual cost was human recovery \u2014 a force-push of the branch back to base + spec-doc-only \u2014 plus the risk that the innocuous bot description would have let it merge unnoticed.", + "consumer_refs": [], + "fix": "open (proposed: branch from current default HEAD and git add only docs/specs/** \u2014 never -A; pre-PR guard mirroring #287 that drops or needs-humans any out-of-scope working-tree change; hard assertion that the opened PR diff is docs-only, else hand off instead of publishing)", + "detection_path": "Human review of the opened spec PR diff \u2014 the +3125/-2665 size against an expected single-doc change made it visible; nothing in the pipeline flagged it.", + "eval_agent_signal": "Deterministic diff-property assertion, checkable from the PR files API at open time: for any PR authored by the spec agent, changed-files set must be a subset of docs/specs/** \u2014 a single out-of-scope path is a hard fail. Secondary deterministic revert detector for all bot PRs: any diff hunk that deletes content introduced on base after the branch's merge-base (i.e. the PR would revert commits it never touched) \u2014 computable via git log/merge-base without inference. A crude but effective early tripwire: additions+deletions on a 'spec' PR exceeding a small threshold (e.g. >500 lines or >3 files) triggers review before publish.", + "category": "safety/code-integrity", + "quotes": [ + "changed **+3125 / -2665 across ~30 code files** in addition to the spec document", + "`crates//src/core/compose.rs` **+1758** (new code) \u2026 `crates//src/client/composer.rs` **-2188** (gutted) \u2026 `crates//docs/RESOLUTION.md` **-194** (deleted)", + "Recovery required a human to force-push the branch back to `base + spec-doc-only` (the bot reported it as deriving the spec from the source issue, so the corruption was not obvious from the title)." + ] + }, + { + "group": "aftermath", + "number": 318, + "title": "sdd-execute Pre-PR gate misses Cargo.lock drift: a Cargo.toml dep change without a lock regen ships a PR that fails consumer --locked CI", + "created_at": "2026-06-23T05:06:46Z", + "state": "closed", + "closed_at": "2026-06-23T05:43:00Z", + "stage": "execute", + "failure_mode": "An execute run added a workspace dependency (`thiserror.workspace = true`) to a crate Cargo.toml without regenerating Cargo.lock. The sdd-execute Pre-PR gate ran cargo fmt/clippy/test without --locked, so cargo silently rewrote the lock in-memory and all three checks passed. The PR shipped and the consumer's --locked test lane failed with exit 101, presenting as a mysterious test failure rather than lockfile drift.", + "root_cause": "Gate/CI parity gap: the Pre-PR verification commands omitted --locked while the consumer CI enforced it (cargo nextest run --workspace with --locked), so lockfile drift was undetectable in-loop by construction.", + "consumer_impact": "Shipped an already-red PR to the consumer repo; consumer CI test job failed post-open with a misleading signature (looked like a test failure, was lock drift), requiring human diagnosis and a rework push to regenerate and commit Cargo.lock.", + "consumer_refs": [], + "fix": "spectacles commit 4e3559a 'fix(sdd-execute): run the Rust Pre-PR gate with --locked + regen Cargo.lock on manifest change' (Closes #318; adds --locked to the gate across sdd-execute-haiku/sonnet/opus and requires committing a regenerated Cargo.lock when a Cargo.toml changes, mirroring the Node frozen-lockfile rule)", + "detection_path": "Red consumer CI: the --locked test job failed after the PR was already open; a human traced the exit-101 error back to lockfile drift.", + "eval_agent_signal": "Deterministic diff property + run-status join: flag any pipeline PR whose diff touches a Cargo.toml (or package manifest) without a matching Cargo.lock hunk \u2014 or run `cargo fetch --locked` against the PR head as a required check. Also a parity linter over the gate definition: diff the Pre-PR gate's command flags against the consumer CI's (missing --locked / --frozen-lockfile when the consumer enforces it is an automatic finding). Early-warning join: Pre-PR gate green + consumer CI failing with the string 'cannot update the lock file ... because --locked was passed' identifies this class exactly.", + "category": "infra/config", + "quotes": [ + "`error: cannot update the lock file ... because --locked was passed` (exit 101)", + "The `sdd-execute` Pre-PR CI gate runs `cargo fmt` / `cargo clippy` / `cargo test` **without `--locked`**, so cargo silently rewrites `Cargo.lock` in-memory and all three pass.", + "Only the consumer's `test` lane (`cargo nextest run --workspace`, which enforces `--locked`) catches the drift \u2014 after the PR is already open and red." + ] + }, + { + "group": "aftermath", + "number": 320, + "title": "Failed/timed-out execute strands a task at sdd:in-progress with no reset \u2014 in-flight filter then blocks all re-dispatch", + "created_at": "2026-06-23T05:40:25Z", + "state": "open", + "closed_at": "", + "stage": "dispatch|execute|monitor", + "failure_mode": "When an sdd-execute run dies (60m wall-clock timeout, engine crash) without opening its sdd/ PR, nothing resets the lifecycle label, so the task sits at sdd:in-progress with no PR and no running job. The dispatcher's in-flight filter ('carries sdd:in-progress or has an open linked PR') then excludes it from the ready set forever: manual /execute no-ops, /dispatch skips it, and the sdd-monitor stranded-recovery posts /dispatch which also can't fire, burning its 3 attempts and dead-ending at needs-human.", + "root_cause": "Lifecycle label is set optimistically at claim time but has no failure-path reset, and the in-flight predicate trusts the bare label without corroborating an open PR or an actually running/queued execute run \u2014 so a crashed run's label is indistinguishable from genuine in-flight work.", + "consumer_impact": "Hard stall: two next-layer tasks in the consumer pilot hit the 60m timeout and became permanently un-redispatchable; the monitor's recovery mechanism was structurally ineffective (same filter), and recovery required an operator hand-editing labels.", + "consumer_refs": [], + "fix": "open", + "detection_path": "Human watching the pilot: operator noticed two tasks stuck, tried a manual /execute which no-op'd, then hand-edited the lifecycle labels to recover.", + "eval_agent_signal": "Deterministic label-event/run-status join: a task labeled sdd:in-progress with (a) no open linked PR and (b) no queued/in-progress sdd-execute workflow run for more than the execute wall-clock budget (60m + slack) is stranded \u2014 emit an alert or auto-reset. Equivalently, a timeline-shape check: 'labeled sdd:in-progress' event with no subsequent PR-opened cross-reference within N minutes, joined against workflow-run conclusions of failure/cancelled/timed_out for that task. A second signal for the dead-end: monitor posting /dispatch repeatedly with no resulting execute run start (comment\u2192run join yields zero) before escalating needs-human.", + "category": "state-machine/lifecycle", + "quotes": [ + "Two next-layer tasks both hit the 60m timeout, stayed `sdd:in-progress` with no PR, and could not be re-dispatched \u2014 a manual `/execute` no-op'd.", + "\"A task is in flight if it carries `sdd:in-progress` or has an open linked PR; the dispatcher removes in-flight tasks from the ready set before fanning out.\"", + "it just burns its 3 attempts and escalates to `needs-human` \u2014 a dead end, not a recovery" + ] + }, + { + "group": "aftermath", + "number": 322, + "title": "feat(distillery-sync): ingest docs via distillery_ingest_doc, not distillery_store", + "created_at": "2026-06-23T16:06:42Z", + "state": "closed", + "closed_at": "2026-06-23T16:26:08Z", + "stage": "distillery", + "failure_mode": "Enhancement: distillery-sync was believed to ingest consumer docs (specs, ADRs, decisions, READMEs) via distillery_store, a single-entry tool with fuzzy semantic dedup \u2014 non-idempotent on re-sync (duplicates or missed updates as docs evolve), no doctype facet (everything lands as default 'inbox' type), and no chunking (long specs stored as one blob, coarse retrieval). Proposed switching to the purpose-built distillery_ingest_doc (sha256 external_id, doctype tags, chunk-linked entries).", + "root_cause": "Wrong-tool fit in the agent design (ad-hoc memory store used for repeated document sync) \u2014 but the premise turned out to be outdated: v0.3.0 already used distillery_ingest_doc, verified by a live full-sync showing correct entry shape.", + "consumer_impact": "None observed on the live run \u2014 this was a latent data-quality risk (duplicate/uncategorized/unchunked knowledge-store entries degrading downstream retrieval), and the closing verification showed the deployed version already had the correct behavior.", + "consumer_refs": [], + "fix": "Closed via PR #323 / spectacles commit d395045 'feat(distillery-sync): ingest docs via distillery_ingest_doc, not distillery_store'; closing comment notes v0.3.0 already emitted ingest_doc-shaped entries, so the issue's premise was outdated. Depends on norrietaylor/distillery#627 (the distillery_ingest_doc tool).", + "detection_path": "Design review of the distillery-sync agent; resolved by inspecting a live full-sync into a fresh Distillery instance and finding entries already in ingest_doc shape.", + "eval_agent_signal": "Deterministic post-sync assertions on the knowledge store: (1) idempotency probe \u2014 re-run sync on an unchanged doc and assert zero new entries / dedup_action='skipped'; (2) entry-shape check \u2014 every synced entry carries doctype/ tag, metadata.doctype, metadata.source (repo path), metadata.external_id = sha256(text); (3) chunking check \u2014 a doc over the chunk threshold yields multiple chunk-linked entries (chunk_index/chunk_total present). Also a staleness guard for the eval agent itself: verify the issue's premise against the deployed version before acting (this issue was filed against outdated behavior).", + "category": "process-design", + "quotes": [ + "`external_id = sha256(text)` \u2192 **idempotent**: re-syncing identical content is a no-op (`dedup_action=\"skipped\"`)", + "`reference` entries tagged `doctype/doc` with `metadata.doctype`, `metadata.source` (repo path), `metadata.external_id` = SHA-256, and `chunk_index`/`chunk_total` (e.g. `crates//README.md` \u2192 3 chunks)", + "The premise (that distillery-sync uses `distillery_store`) was outdated." + ] + }, + { + "group": "aftermath", + "number": 324, + "title": "distillery-sync: deterministic status-issue upsert (dedupe + always emit issue_number) \u2014 safe_outputs fails on duplicate status issues", + "created_at": "2026-06-23T23:26:14Z", + "state": "open", + "closed_at": "", + "stage": "distillery|status", + "failure_mode": "Two compounding defects in the '[distillery-sync] Status' issue upsert: a step-7 search-miss created a duplicate status issue that persisted for days, and once two issues matched the title marker the pinned Haiku agent could not deterministically pick one, emitting an update-issue with no issue_number. The safe-outputs processor rejected it, and on push/schedule runs (no triggering issue to fall back to) the whole run was marked failed \u2014 even though the actual ingest succeeded.", + "root_cause": "Non-deterministic LLM-driven upsert with no tie-break rule, no duplicate self-heal, a fragile title-marker search that can miss the existing issue, and status bookkeeping wired as run-fatal instead of degradable.", + "consumer_impact": "Intermittent false-red pipeline runs on the consumer pilot (success dependent on search-result ordering): agent and detection passed, distillery_ingest_doc/gh_sync landed, but the run showed failed \u2014 eroding trust in run status and requiring a human to dedupe the status issues as a short-term fix.", + "consumer_refs": [], + "fix": "open", + "detection_path": "Red run in the consumer pilot: safe_outputs step failed with the numberless-update_issue error while the agent and detection jobs were green; a human deduplicated the status issues to restore operation.", + "eval_agent_signal": "Two cheap deterministic checks: (1) invariant probe \u2014 count of open issues whose title matches the '[distillery-sync] Status' marker must be exactly 1; >1 is an immediate violation, catchable the moment the duplicate is created rather than days later. (2) Run-status shape join \u2014 a workflow run where agent/detection jobs succeed but the safe_outputs job fails, especially with the log signature 'update_issue failed: Target is \"*\" but no item_number/issue_number specified', flags the bookkeeping-fails-green-work class. A schema lint on emitted safe-output items (every update-issue/add-comment must carry explicit issue_number) catches it pre-merge.", + "category": "observability-gap", + "quotes": [ + "\u2717 update_issue failed: Target is \"*\" but no item_number/issue_number specified in update_issue item", + "the pinned Haiku agent cannot deterministically pick one and emits an `update-issue` with no `issue_number`", + "The sync itself succeeds \u2014 `agent` and `detection` pass, the `distillery_ingest_doc`/`gh_sync` calls land \u2014 only the status-issue bookkeeping output fails, marking otherwise-successful runs red intermittently (depends on search-result ordering)." + ] + }, + { + "group": "aftermath", + "number": 326, + "title": "Net-diff guard: block execute/rebase PRs that silently delete or revert out-of-scope merged work (stale-base degrade)", + "created_at": "2026-06-24T15:41:39Z", + "state": "open", + "closed_at": "", + "stage": "execute|validate|review", + "failure_mode": "SDD PRs built on a base predating other merges to main silently delete or revert files merged after their base \u2014 files entirely outside the task's declared files-in-scope. GitHub's 3-way merge honors the branch's explicit deletion with no conflict and no signal. Observed three times on the consumer pilot: a PR deleted a merged spec doc for a different feature, a PR reverted a merged crate-split refactor, and a PR re-introduced a previously fixed bug by regenerating a file from a base lacking the fix.", + "root_cause": "No base-agnostic net-diff-vs-current-main guard: the prior fix (#287) patched one instance without generalizing, #317 covers only the sdd-spec derive variant, and sdd-validate gate 2 detects out-of-scope deletions but only emits a Warning escalating to needs-human \u2014 never a blocking check, never an auto-restore.", + "consumer_impact": "Silent reverts of merged work on the consumer repo \u2014 the highest-severity class in the run: merged features/fixes would be erased with green checks. Each instance cost a human net-diff read plus hand-restoration; had they merged unnoticed, other teams' work would have regressed.", + "consumer_refs": [], + "fix": "open", + "detection_path": "Human reading the PR's net diff against current main each time; the framework never blocked the merge.", + "eval_agent_signal": "Fully deterministic diff property, ideal as a required status check: compute `git diff origin/main...HEAD` for every pipeline PR, extract deletions and content-reverts (hunks restoring a main-present line to a prior state), and join each affected path against the task's declared files-in-scope block. Any deletion/revert of a path present on current main and not in files-in-scope = blocker. Re-run on every push and after rebases (keyed on head vs current main, not branch provenance). Recurrence of the class is itself a meta-signal: an eval agent should flag when a closed 'fixed' issue (#287) shares a failure fingerprint with new incidents, indicating a point fix where a guard was needed.", + "category": "safety/code-integrity", + "quotes": [ + "a PR deleted a merged spec document for a **different** feature (its branch predated that spec's merge)", + "Each was caught only because a human read the net-diff and hand-restored the file. The framework did not block the merge.", + "`sdd-validate` gate 2 **does detect** out-of-scope deletions \u2014 but emits a **Warning that escalates to `needs-human`**, not a blocking check, and never restores" + ] + }, + { + "group": "aftermath", + "number": 327, + "title": "sdd-review/sdd-validate: incremental review keyed to last-reviewed SHA (stop full re-scan churn every push)", + "created_at": "2026-06-24T18:47:35Z", + "state": "open", + "closed_at": "", + "stage": "review|validate", + "failure_mode": "Both sdd-review and sdd-validate re-scan the entire base..head diff on every trigger with no memory of what was already reviewed \u2014 proof artifacts re-executed, full scope checks re-run. Every push re-reviews unchanged code and surfaces new findings on lines nobody touched that iteration: already-fixed findings re-surfaced run after run, the auto-revise re-implemented items a prior pass had deliberately deferred (churning the PR head), and a stale finding re-applied needs-human on each push.", + "root_cause": "No last-reviewed-SHA state per PR: the #247 unresolved-thread aggregation only prevents duplicating open threads, but nothing records that a diff region already passed review, so each run starts from the whole diff and the review never converges except at the degenerate fixed point of 0 unresolved threads + green CI.", + "consumer_impact": "Steady throughput drain and rework loop on the multi-PR feature cascade: humans re-resolved the same findings repeatedly, auto-revise churned PR heads (each churn re-triggering CI and more review runs \u2014 compounding token/compute waste), and spurious needs-human labels stalled merges.", + "consumer_refs": [], + "fix": "open", + "detection_path": "Human watching the pilot's multi-PR cascade: noticed the same findings re-surfacing, hand re-resolving them each run, and the PR failing to converge.", + "eval_agent_signal": "Timeline-shape and diff-scope joins: (1) review comments anchored to files/lines not modified since the previous review run (comment path/position joined against the inter-run commit range) \u2014 a push touching one file must not yield findings on unrelated files; (2) finding-fingerprint recurrence \u2014 normalized finding text re-raised in run N+1 after being resolved in run N with no touch to the anchored region; (3) label oscillation \u2014 needs-human added/removed more than K times on one PR; (4) cost trend \u2014 review-run token/duration count per PR growing linearly with pushes while diff-since-last-review stays small. An LLM judge comparing consecutive review outputs for semantic duplication catches fuzzier re-raises.", + "category": "cost/context-efficiency", + "quotes": [ + "\u00a74 partitions a >200-line diff by *concern*, but the scope is always the full diff \u2014 never \"only the commits since my last review.\"", + "the same already-fixed findings re-surfaced run after run (re-resolved by hand each time)", + "a stale finding re-applied `needs-human` on each push. The only stable state is **0 unresolved threads + green CI**" + ] + }, + { + "group": "aftermath", + "number": 328, + "title": "sdd-derive: reword the derive-spec offer to read as useful help, not a callout for incomplete work", + "created_at": "2026-06-24T19:48:44Z", + "state": "open", + "closed_at": "", + "stage": "spec", + "failure_mode": "The bot comment offering /derive-spec on human contributors' unspecced PRs reads as a reprimand: it opens with a deficiency ('has no accompanying spec'), foregrounds a fault list ('implementation gaps, missing failure paths, weak acceptance criteria'), nags ('the weekly unspecced-PR scan will re-surface it'), and applies a 'needs-spec' label that tags the PR as deficient. Contributors' first impression of the SDD process becomes 'you skipped a step', discouraging adoption.", + "root_cause": "Contributor-facing copy in wrappers/sdd-derive.yml:99-107 was written from the pipeline's compliance perspective (missing requirement + report-card gap list + re-surface warning) rather than the developer's value perspective; the label taxonomy ('needs-spec') carries the same deficiency framing.", + "consumer_impact": "No functional pipeline impact \u2014 a social/adoption cost on the live run: human contributors on the consumer repo received callout-toned bot comments and deficiency labels on their PRs, risking resistance to the SDD process itself.", + "consumer_refs": [], + "fix": "In-flight: spectacles branch fix/derive-spec-offer-copy, commit d9e6fc6 'fix(sdd-derive): reword the derive-spec offer as helpful, not a callout' (unmerged to main as of harvest; issue still open)", + "detection_path": "Human watching: the offer comment was seen landing on contributors' PRs during the consumer pilot and judged off-putting by the operator.", + "eval_agent_signal": "Primarily inference-based: an LLM tone-judge over all bot-authored, human-contributor-facing comment templates (and live comments), scoring against rubric items like 'leads with recipient value, not a deficiency', 'no fault-list framing of the recipient's work', 'declining framed as cost-free (no re-surface nag)'. Deterministic proxies: lint for 'needs-*'-style labels applied to non-pipeline (human) PRs; engagement telemetry \u2014 offer comments with near-zero /derive-spec uptake or with negative-reaction emojis indicate the copy repels rather than invites.", + "category": "process-design", + "quotes": [ + "This pull request has no accompanying spec. Comment `/derive-spec` to have one derived retrospectively from the code", + "gap analysis (implementation gaps, missing failure paths, weak acceptance criteria)", + "the first impression of the SDD process is \"you skipped a step,\" which discourages adoption" + ] + } +] \ No newline at end of file diff --git a/docs/specs/03-spec-run-evaluation/rubric.md b/docs/specs/03-spec-run-evaluation/rubric.md new file mode 100644 index 0000000..72c10ea --- /dev/null +++ b/docs/specs/03-spec-run-evaluation/rubric.md @@ -0,0 +1,222 @@ +--- +version: v1-draft +status: unratified +--- + +# Run-evaluation scoring rubric + +> The repository is public: no consumer identity, no internal URL, no cost +> figure. Draft — takes effect only after ratification per spec R2.3 +> (`/ratify-rubric v1` recorded on the roll-up issue). + +Each item: stable id, the question, an anchored scale (every point cites +observable evidence), evidence sources, and a machine-check class — +`deterministic` (collector computes; judge quotes, never re-scores), +`assisted` (collector computes inputs, judge scores), `inference` +(judge-only). + +Scores are per feature (post-mortem tier) unless marked per-run. A scored +item always names its evidence; an item whose evidence is unavailable is +reported `unscored(reason)`, never guessed (per `shared/rigor.md`). + +--- + +## RB-fid — Spec fidelity + +**RB-fid-1 · R-ID delivery** — For each requirement ID: is the behavior +implemented **and wired into the live user-reachable path** at `sdd:done`? +Class: `assisted` (collector supplies R-ID → task → PR joins and diffs; +judge issues verdicts). +Scale: `delivered` = behavior exercised by a non-gated executable artifact +on the default branch; `partial` = code exists but unwired, feature-gated, +env-gated, or broken at integration; `missing` = no implementing change. +Evidence: spec R-ID text, merged PR diffs, proof-artifact audit +(RB-proof-1), task-issue re-scope decisions. + +**RB-fid-2 · Remediation split** — When post-`sdd:done` human PRs reference +the tracker: fraction of remediation lines that are (a) +specced-but-not-delivered, (b) needed-but-unspecced, (c) quality/refactor. +Class: `assisted` (collector detects and sizes the PRs; judge classifies +hunks). +Scale: reported as the (a)/(b)/(c) percentage split; no target — the trend +is the metric. A rising (a) indicts execution; a rising (b) indicts +spec/plan coverage. +Evidence: remediation PR diffs vs. spec R-IDs vs. pipeline PR diffs. + +## RB-proof — Proof and demoability + +**RB-proof-1 · Proof delivery** — For each proof artifact the spec declares: +executed, weakened, or missing at `sdd:done`? +Class: `deterministic` — grep merged diffs for the implementing test/command; +detect `#[ignore]`, env-var gates, `required-features`, CI-lane-only +execution. +Scale: `executed` = runs in default CI or a committed runnable artifact ran +in a check; `weakened` = exists but gated so no default path executes it; +`missing` = declared, not implemented. +Evidence: spec proof declarations, merged diffs, check-run history. + +**RB-proof-2 · Demoability at done** — Does an executed end-to-end, +user-visible demonstration exist at `sdd:done`? +Class: `assisted` (collector lists candidate artifacts; judge decides +whether any is genuinely end-to-end and user-visible, not a health check — +the empty-PR rule applied at feature scope). +Scale: 2 = an executed e2e artifact demonstrates the feature's headline user +story; 1 = e2e artifact exists but covers a fragment or requires unshipped +scaffolding; 0 = none (the evaluated run's state). +Evidence: proof audit, test-plan-style artifacts, CI runs at the done +boundary. + +## RB-int — Integrity + +**RB-int-1 · Silent-revert exposure** — Did any agent PR delete or revert +work that main had independently advanced, without a blocking signal? +Class: `deterministic` (net-diff-vs-main scan, docs-only assertion on spec +PRs, tripwires) with an `inference` backstop (judge reviews flagged diffs +for revert intent — deleted safety comments, `?` → `.unwrap()` tells). +Scale: count of incidents by outcome: blocked / warned-then-human-cleared / +merged-unnoticed. Any `merged-unnoticed` > 0 is the run's headline defect. +Evidence: per-PR net-diff scans, gate outputs, restore commits. + +**RB-int-2 · Recurrence fingerprints** — Did a failure signature matching a +closed "fixed" framework issue re-appear? +Class: `deterministic` (fingerprint join against the closed-issue corpus). +Scale: count; each recurrence names the closed issue it matches — a point +fix that needed a guard. +Evidence: snapshot fingerprints, framework issue tracker. + +## RB-loop — Convergence + +**RB-loop-1 · Review-loop closure** — Who closed review threads, and how? +Class: `deterministic` for the counts (threads resolved with a bot +reply carrying a sha or rebuttal vs. resolved bare vs. human-disposed); +`assisted` for the quality score of a sample of closing replies (does the +reply actually validate or rebut the finding). +Scale: closure-integrity ratio = reasoned-bot-closures / all closures; +human-disposition count reported alongside (the evaluated run: ~80). +Evidence: GraphQL thread audit, comment authorship. + +**RB-loop-2 · Finding recurrence** — Same finding re-raised after +resolution (within a PR or across sibling PRs)? +Class: `deterministic`. +Scale: recurrence count per feature; target 0 — each instance names the +fingerprint and passes. +Evidence: snapshot fingerprints. + +**RB-loop-3 · Revise grounding** — Did each revise run consume the +unresolved thread context, and did its diff address it? +Class: `assisted` (collector: did the run fetch threads; judge: does the +diff address the sampled findings). +Scale: 2 = grounded and addressing; 1 = grounded, diff partially addresses; +0 = revise ignored available thread context. +Evidence: run transcript fetch calls, revise diffs, thread texts. + +## RB-flow — Flow and autonomy + +**RB-flow-1 · Recovery efficacy** — Of stranding/monitor escalations, what +fraction recovered without a human? +Class: `deterministic`. +Scale: recovered-auto / escalations, with per-episode cause class (timeout, +quota, engine crash, dispatch gap, route decline). Evaluated-run baseline: +0/6. +Evidence: monitor audit lines, run index, human-touch inventory. + +**RB-flow-2 · needs-human discipline** — Episode count, dwell, and +position; late episodes (>50% of run) flagged. +Class: `deterministic`. +Scale: reported distribution; regression = late-episode count above the +prior runs' trend. The operator standard: early is acceptable, late is a +failure. +Evidence: label timelines. + +**RB-flow-3 · Human-touch inventory** — Manual `/execute`, `/dispatch`, +`/revise`, label surgery, and decision comments, split +design-intended-gate vs. framework-failure compensation. +Class: `deterministic` for counts, `assisted` for the split. +Scale: compensation touches per feature; trend down. +Evidence: comment streams, label events. + +## RB-input — Inputs + +**RB-input-1 · Ledger evidence quality** — Are assumption-ledger entries +"settled" on independent evidence? +Class: `assisted` (collector extracts entries + citations; judge flags +circular settlement — an entry whose only evidence is the spec it +parameterizes). +Scale: circular-settlement count; each names the entry and the risk it +buried. Evaluated-run baseline: the two costliest unknowns were +circular-settled and later caused the worst stall and an unshipped +mechanism. +Evidence: architecture ledger, spike docs, downstream incident joins. + +**RB-input-2 · Plan integration coverage** — Does the plan own the seams: +an integration/wiring/e2e task whose scope names the launch path and +entry-point files? +Class: `assisted`. +Scale: 2 = explicit integration task(s) with concrete files; 1 = wiring +mentioned but unowned; 0 = units only (the evaluated run — the two largest +unspecced remediation areas were files no pipeline PR touched). +Evidence: plan comment, task tree, remediation (b)-class concentration. + +**RB-input-3 · Source fidelity and internal consistency of the generated +spec** — (a) Do the generated requirements contradict *each other*? +(b) Does each generated normative requirement agree with the source document +it was derived from (diagrams and topology sections included)? +Class: `inference` (judge-only; run at the spec boundary, where a correction +is cheap and everything downstream inherits it — and **re-run whenever a +later PR edits a merged spec file**, since the evaluated run's drift was +*spread* by an implementation PR harmonizing a correct requirement toward a +wrong one). +Scale: contradiction count; each names the R-ID pair or the R-ID plus the +source passage it contradicts. Evaluated-run baseline: the generated spec +was internally contradictory from birth (one requirement resolved hostnames +to an internal-switch address the source's topology placed the daemon off +of; the adjacent requirement carried the source's correct published-port +model), survived every downstream gate — which validate conformance *to the +generated spec* and are blind to this class by construction — was +harmonized toward the wrong pole by a later spec-editing implementation PR, +and was corrected only inside a remediation PR ~11 days after origin. The +internal-consistency half needs no source document at all. +Evidence: generated spec R-IDs, source document, spec-file edit history, +spec-PR review thread. + +## RB-sig — Signals + +**RB-sig-1 · Signal truthfulness** — False or missing operator signals: +thumbs-down stamped by cancelled runs, red agent checks from concurrency +noise, stale `needs-human`, dead status surface. +Class: `deterministic`. +Scale: false-signal count by class; status-surface uptime (fresh comment at +last pipeline event: yes/no per boundary). +Evidence: reaction-vs-run joins, check-run conclusions, status comment +freshness. + +**RB-sig-2 · Contributor-facing tone** — Bot copy on human contributors' +items leads with value, no fault-listing, cost-free decline. +Class: `inference`. +Scale: 2/1/0 anchored on the derive-offer copy standard. +Evidence: bot comments on non-bot-authored items. + +## RB-cost — Economics + +**RB-cost-1 · Cost per feature / per net line** — Token usage per phase, +tier, feature; cost-per-net-line-kept. +Class: `deterministic` (both artifact layouts; degrade to +`unscored(retention)` when artifacts expired). +Scale: trended; regression = per-net-line cost rising across runs at equal +scope class. +Evidence: run artifacts, PR net diffs. + +**RB-cost-2 · Waste share** — Tokens in runs that produced no accepted +output (crashed, cancelled, no-op'd, quota-killed), as a share of feature +total. +Class: `deterministic`. +Scale: trended share; each top waste run named with cause. +Evidence: run conclusions × usage records. + +--- + +## Ratification ledger + +| Version | Ratified by | Where | Date | +|---|---|---|---| +| v1-draft | — (unratified) | — | — | diff --git a/docs/specs/04-spec-eval-actor/04-spec-eval-actor.md b/docs/specs/04-spec-eval-actor/04-spec-eval-actor.md new file mode 100644 index 0000000..9c17111 --- /dev/null +++ b/docs/specs/04-spec-eval-actor/04-spec-eval-actor.md @@ -0,0 +1,149 @@ +--- +id: spec-eval-actor +title: Eval actor — bounded actions on run-evaluation findings +kind: spec +status: planned +tracking-issue: +supersedes: +--- + +# 04-spec-eval-actor + +> The repository is public. This file carries no consumer identity, no +> internal URL, no cost figure. Companion sketch to +> `docs/specs/03-spec-run-evaluation/` — deliberately short; it becomes a +> full spec when the eval surfaces have run long enough to trust. + +## Context + +The run-evaluation suite (spec 03) is report-only by design: the collector +and judge write scorecards and snapshots, never labels or issues. The +evaluated run showed why the acting half matters — every detected condition +(stranded tasks, stale `needs-human`, recurrence fingerprints, integrity +tripwires) still required a human to translate detection into action. The +actor agent is that translation, kept separate so the reporting surface +stays side-effect-free and the mutation surface carries its own, stricter +trust posture. + +## Introduction / Overview + +`sdd-eval-actor` is a deterministic utility workflow (no `engine:` in v1 — +every v1 action is a mechanical mapping from a snapshot condition to a +bounded action; an engine-bearing escalation writer can come later). It +consumes `eval_snapshot.json` artifacts and the roll-up issue state, applies +an **action table**, and performs only allowlisted, idempotent, audited +actions. Eval reports; the actor acts; the human still decides anything +irreversible. + +## Goals + +1. Close the detection→action gap for the conditions the collector already + proves deterministically. +2. Keep every action bounded (rate-limited, capped, idempotent) and audited + (one ledger line per action on the roll-up issue). +3. Stay subordinate to the human: the actor never overrides an explicit + human decision recorded on the item, and `needs-human` remains a + human-owned hand-off per ADR 0001 — the actor may *apply* it with the + standard hand-off comment, never *clear* it. + +## User Stories + +- As an operator, I want stale `needs-human` labels re-validated and flagged + (not silently cleared) when their cause no longer holds, so I stop + clearing labels the run outgrew. +- As an operator, I want a stranded task re-dispatched once the blocking + condition (e.g. a quota window) has passed, instead of hand-typing + `/execute` two dozen times. +- As a suite maintainer, I want a recurrence fingerprint or integrity + tripwire to open exactly one well-evidenced framework issue, once, + with the snapshot attached. + +## Demoable Units of Work + +### Unit 1: Action table and actuator + +**Purpose:** The v1 action table, mechanically applied. Demoable: a seeded +snapshot condition produces exactly the mapped action plus a ledger line. +**Depends on:** spec 03 Units 1–2 (snapshots, roll-up) +**Affected areas:** `wrappers/sdd-eval-actor.yml` (new), +`.github/actions/sdd-eval-actor/action.yml` (new), +`shared/sdd-interaction.md`, `scripts/quick-setup.sh`. + +**Functional Requirements:** + +- **R1.1**: The actor shall trigger on collector completion (`workflow_run`) + and on a write-access `/act` comment; it shall be **opt-in** + (`SDD_EVAL_ACTOR=1`) — unlike the eval surfaces it mutates state, so it + follows the monitor's opt-in posture, with the spec-03 R1.3 unset-safe + predicate form. +- **R1.2**: v1 action table (exhaustive; anything not listed is out of + scope): + | Snapshot condition | Action | Bound | + |---|---|---| + | Stranded task, cause class retriable (quota window elapsed, timeout kill) | one `/dispatch` nudge comment on the tracker | ≤1 per task per 6 h, ≤3 total per task, then hand-off comment + `needs-human` | + | `needs-human` whose recorded cause tests false against current state | comment on the item naming the stale cause and the passing check | ≤1 per label application; never removes the label | + | Recurrence fingerprint matching a closed framework issue | open one issue on the framework repository, evidence per `shared/rigor.md`, deduped by fingerprint marker | one issue per fingerprint, ever | + | Integrity tripwire (net-diff deletion/revert of main-advanced files) on an open PR | request-changes-style comment on the PR quoting the scan; apply `needs-human` with hand-off comment | once per head sha | + | Judge post-mortem `merged-unnoticed` integrity incident | open one framework issue + comment on the merged PR | once per incident | +- **R1.3**: Every action shall be recorded as one ledger line (timestamp, + condition, action, target) in a dedicated section of the roll-up issue. +- **R1.4**: The actor shall never: merge, close, or block a PR; edit + tracking-issue bodies; write lifecycle labels; clear `needs-human`; act on + any item whose last human comment post-dates the snapshot it is acting on. +- **R1.5**: Identity: the suite App token; permissions the union of what the + table needs (`issues: write`, `pull-requests: write`, `actions: read`) + and nothing more. + +**Proof Artifacts:** + +- Test: seeded stranded-task snapshot → exactly one `/dispatch` nudge and + one ledger line; second run within the bound window → no action, one + "bound-suppressed" ledger note. Fails before this unit. +- Test: seeded recurrence fingerprint → exactly one framework issue with the + fingerprint marker; re-run → no duplicate. Fails before this unit. + +## Non-Goals + +- No engine in v1; no free-text judgment about what to do. +- No auto-remediation of code (no revert-fixing, no restore PRs) — those are + pipeline guards' and humans' work. +- No cross-repository actions beyond the single framework-issue table row. +- Not a replacement for sdd-monitor; it acts on eval snapshots, monitor + stays the dispatch backstop (their overlap — stranded-task nudges — is + resolved by the actor deferring when monitor attempts are still unspent). + +## Design Considerations + +The actor is deterministic because every v1 action is condition→action with +no judgment; this keeps it out of ADR 0020 scope and its blast radius +enumerable. The bound columns are the design: each is chosen so the worst +case (a wedged condition looping) produces bounded noise then a hand-off, +never a storm — the evaluated run's monitor demonstrated both the need +(0/6 recoveries) and the failure mode (attempts burned inside minutes +against multi-hour causes). + +## Security Considerations + +Same injection posture as the collector: snapshot and issue text are data; +commands are token-strict and write-access-gated. The framework-issue row +writes across repositories — its content is generated exclusively from +snapshot fields and never quotes consumer file contents beyond paths. + +## Open Questions + +- Whether the stranded-task nudge should invoke the dispatch cascade + directly instead of commenting `/dispatch` (a comment keeps the actor + inside the human command vocabulary and visible; a direct call is quieter + but less auditable). +- Whether a later engine-bearing tier should draft escalation summaries for + the hand-off comments (inference on top of the same bounded actions). + +## Gap Analysis + +Empty — forward-authored spec. + +## Verification + +Seed each table row's condition on a test repository and assert the mapped +action, its ledger line, and its bound; assert the four "never" rules of +R1.4 by attempting each and observing refusal; lint gates green.