FE-1251: Publish traceable comparison reports

.agents/skills/comparison-publish/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: comparison-publish
+description: Explicitly validate and idempotently publish one retained Brunch comparison report to the canonical Notion Comparison Reports database. Use only when the operator invokes /comparison-publish with a run directory.
+argument-hint: "<run-directory>"
+disable-model-invocation: true
+---
+
+# Comparison Publish
+
+Publish one completed comparison run with `/comparison-publish <run-directory>`. This skill is operator-invoked post-run reporting; it never conducts an experiment, starts a lane, judges a live run, or publishes automatically.
+
+Before proceeding, load [Comparison reporting](../comparison-reporting/SKILL.md) for validity-first evidence policy and [Notion reporting](../notion-reporting/SKILL.md) for safe mutation and verification. Comparison reporting owns the report grammar. Notion reporting owns formatting and the smallest safe mutation.
+
+## Fixed destination and identity
+
+- Hub: `brunch Testing Scenarios`, page id `3a53c81f-e024-8045-98e3-cf55aef279eb`.
+- Database: `Comparison Reports`, data source `collection://01130dd8-6a1c-420a-b84d-166766550163`.
+- Upsert key: the compound identity `Run ID + Phase`. Neither field alone is unique.
+- One database row is the report page. Never create a separate report page plus a register row.
+
+Always discover the current Notion MCP tool schema, fetch the hub, fetch the database/data source, and inspect its live properties before querying or writing. Names and option ids in this file are intent, not permission to assume a stale schema.
+
+## 1. Resolve and validate retained evidence
+
+Resolve the argument from the repository root to one real run directory. Reject an absent directory, symlink, traversal outside the repository, or ambiguous path. Require regular files named `report.md` and `provenance.json`. The report must be a completed retained report, not a draft assembled for publication.
+
+Parse `provenance.json` against `src/dev/comparison-provenance.ts`. Require:
+
+- schema version `1`;
+- one comparison kind: `elicitation`, `execution`, or `end_to_end`;
+- the exact run id and canonical capture timestamp;
+- root package name and release version;
+- exact tag or `null`;
+- full 40-character controller commit, matching clickable commit URL, branch or `null`, and dirty boolean.
+
+The run-directory identity and the provenance run id must agree. Treat a missing, malformed, or mismatched provenance artifact as a hard stop. Never infer historical provenance from the current checkout, current `HEAD`, current package version, a later tag, a report date, or operator memory.
+
+Validate the retained contracts that exist for the declared kind:
+
+- **Elicitation:** require the approved setup snapshot, lane-visible transcripts/interactions, lane terminal and cleanup evidence, and unchanged produced-document references claimed by `report.md`. Distinguish the approachable shared-actor procedure from a rigorous campaign.
+- **Execution:** parse every claimed attempt with `src/dev/execution-comparison/artifact-contract.ts`; use only audience-safe attempt packets and the two public case files named by comparison reporting. Require validity, terminal, cleanup, common command/browser, and retained-output references claimed by `report.md`.
+- **End-to-end:** validate the study contract, exact-byte handoffs, four-cell matrix or explicitly retained missing cells, requirement ledger, and underlying execution attempts with `src/dev/end-to-end-comparison/` contracts.
+
+Do not recursively inspect controller-only case directories. A missing contract or report claim unsupported by retained evidence stops publication; do not repair evidence, infer a result, or silently downgrade the gap.
+
+## 2. Produce the publication copy
+
+Read `report.md` using comparison reporting's evidence order: study design, provenance, validity before outcomes, common evidence, process evidence, lane-only diagnostics, then bounded interpretation. Preserve failed and invalid attempts.
+
+Create an audience-safe Notion body. Never publish controller-only oracle definitions, hidden fixtures, expected states, selector mappings, reveal material, private controller paths, secrets, or opaque reasoning. Include only aggregate hidden-oracle outcomes and portable evidence references. Private mission text may appear only when the fixed destination has been explicitly approved as an operator/controller audience; otherwise summarize its public concern without reproducing it.
+
+The page must include an **Identity** section containing:
+
+- Run ID and Phase;
+- capture timestamp;
+- root package name and release version;
+- exact tag, or `none`;
+- full controller commit SHA as its retained commit URL;
+- branch, or `detached`;
+- dirty state.
+
+Keep validity before outcomes in the page body. Do not add a winner, score, parity claim, causal claim, or broad benchmark claim that the frozen procedure and valid retained evidence do not support.
+
+## 3. Map live database properties
+
+Map the publication copy to the current live properties:
+
+- `Report`: report-page title;
+- `Date`: provenance capture date;
+- `Release`: root package version (the Identity section also records the exact tag);
+- `Commit`: provenance controller commit URL;
+- `Phase`: `Elicitation`, `Execution`, or `End-to-end` from comparison kind;
+- `Result`: the report's explicit bounded outcome, using an existing live option;
+- `Evidence`: retained-evidence completeness, using an existing live option;
+- `Status`: validity/publication status, using an existing live option;
+- `Run ID`: exact provenance run id;
+- `Case`: exact public case/mission identifier supported by the report.
+
+Do not invent an option or coerce an unsupported classification. If an exact mapping is absent, show the operator the mismatch and stop before mutation.
+
+## 4. Compound upsert
+
+Query the live data source for exact `Run ID + Phase` equality:
+
+- **Zero matches:** create one row in `Comparison Reports` with the mapped properties and the audience-safe report body.
+- **One match:** fetch that row page first. Update only changed properties and the generated report content. Preserve comments, child pages/databases, and unrelated child content. Do not replace a full page merely for convenience.
+- **More than one match:** stop and report the duplicate row URLs. Never pick one, merge them, or create another row.
+
+Use the smallest safe mutation. Keep the row's existing page identity on update so repeated publication is idempotent.
+
+## 5. Verify every write
+
+Re-query the database and fetch the report page after every write. Confirm:
+
+1. exactly one row matches `Run ID + Phase`;
+2. every mapped property has the intended typed value;
+3. the Identity section contains the immutable release, full SHA/link, tag, branch, and dirty state;
+4. validity precedes outcomes and retained evidence links resolve;
+5. controller-only material is absent;
+6. existing comments, child pages/databases, and unrelated content remain; and
+7. a second publication would update the same page rather than create another.
+
+Report completion only after this readback. Return the row URL, compound identity, create/update disposition, release, full commit SHA, and verification result.

.changeset/publish-comparison-provenance.md

@@ -0,0 +1,6 @@
+---
+"@hashintel/brunch": minor
+---
+
+- Build coherent execution plans by requiring frontier-verified multi-slice epics to converge through one terminal integration slice.
+- Add immutable comparison provenance and guarded Notion report publishing.

.pi/prompts/compare-execution.md

@@ -61,7 +61,18 @@ Display all of the following together before any target preparation or launch:
 
 Ask through ordinary typed text for explicit **approve**, **revise**, or **reject**. Ambiguity, questions, qualifications, partial approval, or silence are not approval. Revise and redisplay the complete setup, or reject and stop. Do not prepare a target, start a shell, invoke an executor, or invoke the oracle before explicit approval.
 
-After approval, save byte-identical `spec.md`, `public-contract.json`, and `run-setup.md` snapshots under the unused scratch run identity. Keep the two frozen inputs separate and unchanged.
+Immediately after approval and before saving snapshots or preparing the first lane, capture the controller checkout once:
+
+```sh
+npx tsx src/dev/comparison-provenance.ts capture \
+  --run-directory .fixtures/scratch/execution-comparisons/<run-id> \
+  --comparison-kind execution \
+  --run-id <run-id>
+```
+
+Treat an existing `provenance.json`, malformed metadata, or capture failure as a setup collision/failure and stop. Never overwrite it or reconstruct provenance from the checkout used later for reporting.
+
+Then save byte-identical `spec.md`, `public-contract.json`, and `run-setup.md` snapshots under the same scratch run identity. Keep the two frozen inputs separate and unchanged.
 
 ## Run approved lanes
 
@@ -154,8 +165,8 @@ The immutable writer rejects an existing attempt id. Never overwrite, delete, re
 
 ## Report
 
-After all selected lanes terminate and cleanup is proven, write `report.md` under the run scratch root. Present validity before outcomes for each lane. Then report factual terminal state, cleanup, produced output, common command results, and unchanged browser-oracle results.
+After all selected lanes terminate and cleanup is proven, write `report.md` beside the unchanged run-start `provenance.json` under the run scratch root. Present validity before outcomes for each lane. Then report factual terminal state, cleanup, produced output, common command results, and unchanged browser-oracle results.
 
 Keep common evidence and Brunch-only diagnostics visibly separate. Use `not_assessable` for unavailable common evidence. Do not score, rank, choose a winner, claim reliability, infer parity from unavailable or product-private evidence, or let Brunch-only run/Petri/debug data improve its common result. Do not turn ordinary visual hierarchy, clarity, or drag feel into an automatic mechanical verdict.
 
-Finish with the case id, run id, executor order, each attempt id and terminal/validity state, cleanup result, retained output paths, browser report paths, attempt-record paths, and `report.md`. State any remaining human witness plainly.
+Finish with the case id, run id, `provenance.json` path, executor order, each attempt id and terminal/validity state, cleanup result, retained output paths, browser report paths, attempt-record paths, and `report.md`. State any remaining human witness plainly.

.pi/prompts/compare-specs.md

@@ -61,7 +61,18 @@ Run setup is intentionally small, run-specific, and never written into the reusa
 5. Display together: the complete private mission; selected harnesses and order; exact per-harness target-visible framing; mission id; collision-safe run id; requested target document path; scratch and retained paths; and each adapter. Clearly label the mission **top-level-session-only** and the framing **harness-visible**.
 6. Ask through ordinary text for explicit **approve**, **revise**, or **reject**. Revise and redisplay the complete setup, or reject and stop; do not launch partially.
 
-Before the first harness, copy the exact approved private mission and separate setup into the unused scratch run identity as `private-mission.md` and `harness-setup.md`. They are retained with the run but never merged into one target packet.
+Immediately after approval and before copying snapshots or starting the first harness, capture the controller checkout once:
+
+```sh
+npx tsx src/dev/comparison-provenance.ts capture \
+  --run-directory .fixtures/scratch/comparisons/<run-id> \
+  --comparison-kind elicitation \
+  --run-id <run-id>
+```
+
+Run this from the Brunch repository root. Treat an existing `provenance.json`, malformed metadata, or capture failure as a setup collision/failure and stop; never overwrite it or reconstruct provenance from a later checkout.
+
+Then copy the exact approved private mission and separate setup into the same unused scratch run identity as `private-mission.md` and `harness-setup.md`. They are retained with `provenance.json` but never merged into one target packet.
 
 ## Run the approved mission
 
@@ -83,11 +94,11 @@ Run selected harnesses sequentially. For each harness:
 7. Acquire the harness-authored document if it exists. Never author, reconstruct, complete, rewrite, or improve it. Missing or partial output remains missing or partial.
 8. On every outcome, retain state, harness-visible interaction, target-cwd/session identity, final process status, document that exists, and cleanup notes. Kill remaining processes, dismiss the completed shell record, and verify no comparison-harness shell or process remains before starting another.
 
-Do not notify completion while any harness is ready, running, or waiting. After every selected harness is finished or failed, give one aggregate notification. Review scratch assembly, then deliberately copy it to the unused immutable run identity without changing snapshots, transcripts, or harness-authored documents.
+Do not notify completion while any harness is ready, running, or waiting. After every selected harness is finished or failed, give one aggregate notification. Review scratch assembly, then deliberately copy it to the unused immutable run identity without changing `provenance.json`, snapshots, transcripts, or harness-authored documents.
 
 ## Retained run and operator-only report
 
-The retained run contains distinct `private-mission.md` and `harness-setup.md` snapshots, every exact harness-visible transcript, every harness-authored document that exists, target identities/outcomes/cleanup, explanations for missing output, and `report.md`.
+The retained run contains the unchanged run-start `provenance.json`, distinct `private-mission.md` and `harness-setup.md` snapshots, every exact harness-visible transcript, every harness-authored document that exists, target identities/outcomes/cleanup, explanations for missing output, and `report.md`.
 
 Write `report.md` for an operator-only cold reader. It may reproduce the full private mission, but must keep these sections visibly separate:
 
@@ -99,4 +110,4 @@ Write `report.md` for an operator-only cold reader. It may reproduce the full pr
 
 Do not add a fixed rubric, score, scripted/API judge, statistics, recommendation, automatic winner, or inferred winner.
 
-Finish by reporting the saved mission path, scratch path, immutable run path, harness terminal states, cleanup result, and `report.md` path. Do not claim runtime correctness, isolation, user consistency, notification timing, or usefulness merely because static instructions or checks passed; those judgments belong to the operator reviewing the real interaction and artifacts.
+Finish by reporting the saved mission path, scratch path, immutable run path, `provenance.json` path, harness terminal states, cleanup result, and `report.md` path. Do not claim runtime correctness, isolation, user consistency, notification timing, or usefulness merely because static instructions or checks passed; those judgments belong to the operator reviewing the real interaction and artifacts.

docs/praxis/comparison-runs.md

@@ -47,6 +47,27 @@ npm run dev-cli -- export --workspace <workspace> --spec-id 1 --out <seed.json>
 
 `rpc` and `print` are deterministic observations. `mutate` is the explicit local write seam. `export` turns reviewed workbench state into a candidate fixture; review it before tracking it.
 
+## Immutable run-start provenance
+
+Every new elicitation, execution, or end-to-end comparison captures `provenance.json` immediately after the operator approves the setup and before the first lane starts. The snapshot records the root package release, exact tag when present, full controller commit and URL, branch, dirty state, run identity, and capture time:
+
+```sh
+npx tsx src/dev/comparison-provenance.ts capture \
+  --run-directory <unused-or-approved-run-directory> \
+  --comparison-kind <elicitation|execution|end_to_end> \
+  --run-id <run-id>
+```
+
+The writer is intentionally collision-failing. Never overwrite this file, recapture it after a checkout changes, or infer historical provenance from the checkout used to publish a report. Keep the snapshot when promoting a run from scratch to `.fixtures/runs/`. Existing historical runs without a retained snapshot remain unchanged and must not be backfilled from memory or current Git state.
+
+After the run is complete and its retained `report.md` is final, publish explicitly:
+
+```text
+/comparison-publish <run-directory>
+```
+
+The publication skill validates `report.md`, `provenance.json`, and the available kind-specific contracts; produces a validity-first, controller-safe copy; and upserts the existing Notion Comparison Reports database by `Run ID + Phase`. A repeated invocation updates the same report row, while duplicate matching rows stop publication for operator resolution. Publication never infers the release or commit from the publisher's checkout.
+
 ## Quick start: compare execution
 
 From a trusted top-level project Pi session, run:
@@ -81,6 +102,8 @@ immutable `ExecutionAttempt` format. Reports must present elicitation validity,
 execution validity, and requirement-level output evidence separately. This one-case tracer supports
 within-executor/spec contrasts only—not a winner, reliability estimate, or causal claim.
 
+After the complete end-to-end setup is approved, capture `end_to_end` provenance in its scratch run root before starting either elicitation lane. Promote that exact snapshot beside the handoffs, attempts, ledger, and report; the later execution matrix does not recapture it.
+
 The first promoted witness is
 [`petri-editor-e2e-20260721T132600Z`](../../.fixtures/runs/end-to-end-comparison/petri-editor-e2e-20260721T132600Z/).
 It retains both exact handoffs, all four valid failed execution cells, portable common evidence, the
@@ -125,10 +148,11 @@ The round-one materials remain useful for focused improvement and regression stu
 The rigorous campaign loop is:
 
 1. instantiate [`comparison-runs/mission-packet.md`](comparison-runs/mission-packet.md), keeping the reveal key outside every target cwd;
-2. drive one fresh lane with [the actor recipe](../../.agents/skills/agent-as-user-comparison/SKILL.md);
-3. retain target-visible interaction, validity/intervention notes, cleanup status, and the target-authored ready document under `.fixtures/scratch/comparisons/<campaign-id>/`;
-4. when the study needs structured judgment, run the masked-outcome and unblinded-process passes from the judgment prompt pack and have the named human adjudicator review the drafts; and
-5. after review, promote the portable bundle to `.fixtures/runs/agent-as-user-comparison/<campaign-id>/` and run `npm run check:promoted-run-paths` before commit.
+2. after setup approval, capture immutable `elicitation` provenance under `.fixtures/scratch/comparisons/<campaign-id>/` before the first lane;
+3. drive one fresh lane with [the actor recipe](../../.agents/skills/agent-as-user-comparison/SKILL.md);
+4. retain target-visible interaction, validity/intervention notes, cleanup status, and the target-authored ready document under `.fixtures/scratch/comparisons/<campaign-id>/`;
+5. when the study needs structured judgment, run the masked-outcome and unblinded-process passes from the judgment prompt pack and have the named human adjudicator review the drafts; and
+6. after review, promote the portable bundle, including unchanged `provenance.json`, to `.fixtures/runs/agent-as-user-comparison/<campaign-id>/` and run `npm run check:promoted-run-paths` before commit.
 
 Use push-driven hands-free control: project config lowers the query fallback floor to 5 seconds, forwards quiet output after roughly 3 seconds, and prunes superseded viewport reads before each LLM call. Send input and end the turn; act on the pushed quiet update. Query only when no push arrives or the incremental tail is ambiguous, and then read the current tail—never page historical scrollback.