diff --git a/.changeset/brave-rivers-compare.md b/.changeset/brave-rivers-compare.md new file mode 100644 index 000000000..bed65c4f2 --- /dev/null +++ b/.changeset/brave-rivers-compare.md @@ -0,0 +1,5 @@ +--- +"@hashintel/brunch": minor +--- + +Add isolated Brunch and Petrinaut brownfield comparison cases. diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 9abe38661..f85e81e26 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -35,6 +35,11 @@ jobs: git config --global user.name "Brunch CI" git config --global init.defaultBranch main + - name: Install native test dependencies + run: | + sudo apt-get update + sudo apt-get install --yes expect ripgrep zsh + - name: Install run: npm ci diff --git a/.pi/prompts/compare-execution.md b/.pi/prompts/compare-execution.md index 23863aa3b..5b22aff0a 100644 --- a/.pi/prompts/compare-execution.md +++ b/.pi/prompts/compare-execution.md @@ -39,7 +39,9 @@ For a supplied or selected id, run: npx tsx src/dev/execution-comparison-operator.ts inspect --case ``` -This is the only pre-approval case read. It validates the existing FE-1230 public case loader and returns only the exact frozen specification, exact public contract, hashes, and shared framing. If resolution or validation fails, report the error and stop; never guess, repair, normalize, or fall back to another case. +This is the only pre-approval case read. It validates the public case loader and returns only the exact frozen specification, exact public contract, packet hash, repository requirement, compiled oracle identity, and shared framing. If resolution or validation fails, report the error and stop; never guess, repair, normalize, or fall back to another case. + +For a case whose inspection reports `requiresSourceRepository: true`, ask for the absolute path to a trusted local checkout containing the pinned commit. This path is controller setup and is never target-visible. Greenfield cases require no source checkout. Ask which executors to run: **Brunch**, **Claude Code**, or both, and in which disclosed order. The default recommendation is both in the operator's chosen order. This approachable procedure does not claim order blinding, matched private reasoning, or statistical reliability. @@ -56,8 +58,9 @@ Display all of the following together before any target preparation or launch: 5. the run identity, case directory id, public case id, and public-packet hash; 6. exact scratch, attempt-record, per-lane target, process-log, final-tree, final-diff, browser-report, and visible-interaction output paths; 7. Brunch's pinned TUI entrypoint and Claude Code's structured adapter; -8. the fixed post-lane oracle identity `petri-editor-browser-v2`; and -9. the limits: sequential lanes, unchanged inputs, no substantive intervention, no landing, no score, no winner, and no reliability or parity claim. +8. the case's compiled post-lane oracle identity returned by `inspect`; +9. whether preparation is greenfield or a pinned remote-free brownfield snapshot; and +10. the limits: sequential lanes, unchanged inputs, no substantive intervention, no landing, no score, no winner, and no reliability or parity claim. Ask through ordinary typed text for explicit **approve**, **revise**, or **reject**. Ambiguity, questions, qualifications, partial approval, or silence are not approval. Revise and redisplay the complete setup, or reject and stop. Do not prepare a target, start a shell, invoke an executor, or invoke the oracle before explicit approval. @@ -82,13 +85,14 @@ Give both executors the exact same frozen specification and public contract with ### Brunch -Prepare through the existing FE-1230 workspace adapter: +Prepare through the case-aware workspace adapter. For greenfield cases omit `--source-repository`; for pinned brownfield cases include the approved absolute source checkout: ```sh npx tsx src/dev/execution-comparison-operator.ts prepare \ --case \ --lane brunch \ - --target + --target \ + [--source-repository ] ``` From the Brunch repository root, open one direct `interactive_shell` using the pinned project-local entrypoint reported by preparation: @@ -107,13 +111,14 @@ Keep Brunch run metadata, Petri journal/projection, JSONL, and `.brunch/debug/` ### Claude Code -Prepare through the execution wrapper: +Prepare through the same case-aware wrapper, using the same source checkout rule: ```sh npx tsx src/dev/execution-comparison-operator.ts prepare \ --case \ --lane claude_code \ - --target + --target \ + [--source-repository ] ``` Open one direct `interactive_shell` from the fresh target cwd through Claude Code's normal structured adapter: @@ -128,7 +133,7 @@ Send only the approved shared framing. Do not substitute a raw binary, another m After every lane terminates, first capture its final process status and all target-visible interaction evidence. Kill any still-running executor process, query the shell to a final status, dismiss the completed shell record, and verify that no executor process or interactive session remains. -Then run the unchanged controller-owned `petri-editor-browser-v2` oracle against the retained output: +Then run the unchanged controller-owned oracle identity returned by `inspect` against the retained output: ```sh npx tsx src/dev/execution-comparison-operator.ts oracle \ @@ -137,7 +142,7 @@ npx tsx src/dev/execution-comparison-operator.ts oracle \ --out /browser/report.json ``` -Do this after every lane terminates, including failed, exhausted, and invalid lanes. Let the existing wrapper run the target's declared `npm test` and `npm run build`, start the fixed static server, run the existing independent browser journeys, and close browser/server resources. Do not edit the frozen case, controller files, oracle manifest, fixtures, expected states, browser journey implementation, or report to make a lane pass. If the unchanged oracle cannot start, retain that setup failure and do not invent a product verdict. +Do this after every lane terminates, including failed, exhausted, and invalid lanes. Let the case-owned oracle run its fixed command, build, Git, TUI, or browser checks and close all resources. Do not edit the frozen case, controller files, oracle manifest, fixtures, expected states, journey implementation, or report to make a lane pass. If the unchanged oracle cannot start, retain that setup failure and do not invent a product verdict. After the oracle exits, verify cleanup again. Retain the target cwd; cleanup means no live process or session, not deletion of evidence. Do not launch the next lane until the prior executor shell and oracle resources are both fully clean. @@ -149,7 +154,7 @@ For every successful, failed, exhausted, and invalid attempt, retain immutably: - elapsed and intervention ledgers; - cleanup status and any residue; - the final tree and complete base-to-tip diff, plus honest notes for uncommitted or unavailable material; -- common `npm test`, `npm run build`, and browser results; +- common case-owned command and oracle results; - normalized visible interaction evidence; and - the unchanged retained output itself. @@ -165,7 +170,7 @@ The immutable writer rejects an existing attempt id. Never overwrite, delete, re ## Report -After all selected lanes terminate and cleanup is proven, write `report.md` beside the unchanged run-start `provenance.json` under the run scratch root. Present validity before outcomes for each lane. Then report factual terminal state, cleanup, produced output, common command results, and unchanged browser-oracle results. +After all selected lanes terminate and cleanup is proven, write `report.md` beside the unchanged run-start `provenance.json` under the run scratch root. Present study design and run identity, then validity before outcomes for each lane, factual terminal state, cleanup, produced output, common command results, unchanged oracle results, requirement findings, limitations, and recommendations. Keep every referenced path repository-relative so `/comparison-publish` can validate the retained bundle. Keep common evidence and Brunch-only diagnostics visibly separate. Use `not_assessable` for unavailable common evidence. Do not score, rank, choose a winner, claim reliability, infer parity from unavailable or product-private evidence, or let Brunch-only run/Petri/debug data improve its common result. Do not turn ordinary visual hierarchy, clarity, or drag feel into an automatic mechanical verdict. diff --git a/docs/archive/PLAN_HISTORY.md b/docs/archive/PLAN_HISTORY.md index 240b6e120..f08eede5d 100644 --- a/docs/archive/PLAN_HISTORY.md +++ b/docs/archive/PLAN_HISTORY.md @@ -3,6 +3,18 @@ This file is the active POC-line plan archive for `memory/PLAN.md`. Legacy pre-`next` history was moved out of the live docs tree with the old archived implementation. +## 2026-07-22 FE-1241 brownfield comparison closeout + +`brownfield-comparison-cases` ([FE-1241](https://linear.app/hash/issue/FE-1241/add-isolated-brownfield-comparison-cases), [PR #362](https://github.com/hashintel/brunch/pull/362)) completed the mechanical expansion from the sole greenfield Petri case to frozen Brunch backend and Petrinaut frontend replays. The branch is `ka/fe-1241-brownfield-comparison-cases`, based on `next` after FE-1240 landed. + +The frontier added content-addressed mission/baseline/registry/execution/oracle packets, exact packet handoff, lightweight pinned-source snapshots, closed case-owned dependency recipes, target-bounded Brunch/Claude launch policy, pinned Brunch graph seeding, and compile-time oracle dispatch without widening `ExecutionAttempt`. Brunch replays FE-1201/PR #336 host landing at parent `f5a423b19f76cf345d88053456870a126e451618`; Petrinaut replays PR #9051 optimization UI in the full HASH tree at parent `5c7a2d9db5caa851c38938f4b1bac19005b0e978`. Historical patches bootstrap claims and rivals but are never golden diffs or target-visible material. + +D137-L is the shared pinned preparation path. D136-L permits only the compiled immutable Petrinaut install before execution and the closed focused builds after lane termination. Brunch's public `/brunch:land` journey is judged by an independent full-range Git model; Petrinaut's standalone `/optimization` journey is judged through a same-origin deterministic optimizer, declared mechanical addresses, calibration-derived inputs, and contrastive rivals. Petri remains the greenfield case; Clay, arbitrary commands/plugins, source landing, aggregate winners, repetitions, and `ExecutionAttempt` schema changes remain out. + +Before merge, the implementation was reoriented from publication-grade benchmark proof to fast product learning. The automated merged-reference preflight, duplicate HASH installations, macOS network verifier, external-service probes, and receipt machinery were retired. Routine runs now use remote-free pinned snapshots and bounded tools as explicit experimental hygiene; source/reference calibration is repeated only when the source, public contract, or oracle changes. Current implementation state lives in [`src/dev/TOPOLOGY.md`](../../src/dev/TOPOLOGY.md); operational guidance lives in [`docs/praxis/comparison-runs.md`](../praxis/comparison-runs.md). + +The retained hardening binds the calibration seed into oracle identity, uses typed source-backed Petrinaut addresses without fixture fallback, aligns the Brunch public type with its backend runtime shape, and requires absolute oracle evidence output. The shared `/compare-execution` workflow now derives repository preparation and oracle identity from every selected case and retains publication-compatible provenance, attempts, cleanup, and validity-first reports for Petri, Brunch host landing, and Petrinaut. + ## 2026-07-22 Executor Petri sequence reconciliation `executor-slice-attempt-lifecycle` (FE-1192) merged in #324 on 2026-07-13. `petri-execution-parity` (FE-1195) then absorbed and completed the planned isolation/fan-in, durable parallel authority, Petrinaut-visible attempt topology, and epic integration sequence in #325. Their stale active/future entries and dependency chain were retired from `memory/PLAN.md`; current execution ownership and lifecycle semantics live in `src/executor/TOPOLOGY.md` under D127-L–D129-L. diff --git a/docs/praxis/comparison-guide.md b/docs/praxis/comparison-guide.md index 629d7783c..d6a72b7ba 100644 --- a/docs/praxis/comparison-guide.md +++ b/docs/praxis/comparison-guide.md @@ -24,9 +24,11 @@ From a trusted top-level project Pi session, run: ```text /compare-execution minimal-petri-net-editor +/compare-execution brunch-host-landing +/compare-execution petrinaut-optimization ``` -The workflow gives Brunch and Claude Code the same frozen specification and applies the same independent browser checks to both outputs. Alpha 10 records the Brunch version and commit automatically. +The workflow gives Brunch and Claude Code the same frozen specification and applies the selected case's independent checks to both outputs. Brownfield cases require a local checkout containing their pinned parent commit; the operator creates a fresh remote-free target from it. Alpha 10 records the Brunch version and commit automatically. ## End-to-end comparison diff --git a/docs/praxis/comparison-runs.md b/docs/praxis/comparison-runs.md index f19fc31f8..4bf71e99e 100644 --- a/docs/praxis/comparison-runs.md +++ b/docs/praxis/comparison-runs.md @@ -72,10 +72,12 @@ The publication skill validates `report.md`, `provenance.json`, and the availabl ## Quick start: compare execution -From a trusted top-level project Pi session, run: +From a trusted top-level project Pi session, run any frozen execution case: ```text /compare-execution minimal-petri-net-editor +/compare-execution brunch-host-landing +/compare-execution petrinaut-optimization ``` The project-local operator displays the frozen specification, public contract, selected Brunch/Claude @@ -84,10 +86,24 @@ fresh isolated lane at a time, cleans up each live shell before continuing, and controller-owned browser oracle to every retained outcome. With no case id, `/compare-execution` lists eligible cases under `testing/execution-comparisons/cases/`. +Greenfield cases prepare an empty repository. Brownfield cases ask for a trusted local source checkout, +materialize the pinned commit into a fresh remote-free repository, add the exact packet, and disable +web/MCP surfaces while bounding file access to the target. This prevents accidental contamination; it +does not claim adversarial isolation from every host capability. + This is developer/evaluation tooling under `.pi/prompts/`, not a shipped Brunch product command. Brunch stops at `promotion_prepared`; the operator never invokes `/brunch:land`, scores a lane, chooses a winner, or treats Brunch-only run/Petri/debug evidence as common comparison evidence. +### Petrinaut calibration + +The Petrinaut oracle uses the standalone `/optimization` route, closed focused builds, and a +deterministic loopback optimizer. Its source-backed addresses were calibrated against PR #9051 during +case construction. Recheck the pinned parent fails and the merged reference passes only when the +source commit, public contract, or oracle changes; record that bounded calibration in the change that +updates the case. Routine comparisons do not duplicate the HASH install or materialize a reference +checkout before every provider run. + ## End-to-end comparison tracer The end-to-end tracer composes the rigorous elicitation recipe with the execution-comparison contracts diff --git a/memory/PLAN.md b/memory/PLAN.md index 0d8b86515..9a5472079 100644 --- a/memory/PLAN.md +++ b/memory/PLAN.md @@ -21,7 +21,7 @@ Brunch-next has delivered the original composition spine: the host, sealed Pi pr **Parallel lanes.** Group 3 agent-layer work is pickup-ready. `agent-control-plane-closure` is one earned coverage frontier over foreground and background prompt ingress: it absorbs the former `graph-assurance-conduct` and `subagent-skill-access` items, reconciles the stale eager-context fossils left by D58-L's load-on-demand cutover, and closes prompt replacement/resource-invocation contradictions before capture experiments amplify them. FE-1187 retains only the concern-grouped review renderer. FE-1208's reshaped `automation-observability-dx` closure landed on the restacked base; its invalid warrant pilots remain historical evidence, while the useful evaluator/report primitives are available to `capture-ledger-tracer`. Two evaluation use cases stay deliberately distinct: **seed-based intra-product testing** (Brunch on its own terms — seeds encode Brunch graph/spec state only Brunch understands) and **mission-driven agent-as-user cross-product comparison** (`agent-as-user-comparison` — competitor CLIs share no Brunch state, so the comparable artifact is the "ready" spec/plan document a mission produces). FE-1210 closed the rigorous technical tracer on 2026-07-17; the separately planned operator workflow (`operator-comparison-workflow`, FE-1215) makes saved missions, visible comparison-harness framing, conversational launch, and a readable free-form report approachable without weakening the retained regression/evaluation machinery. Its first operator-led run falsified the nested fresh-actor topology: the top-level project Pi session must act as the simulated user and drive one direct harness subshell at a time, with stock-Pi text interaction as the portable baseline; the corrected real witness is deferred to `saved-mission-comparison-witness`. The KA stream owns executor/orchestrator/Execute-mode work. FE-1192 and FE-1195 completed the attempt, isolated fan-in, durable parallel authority, and epic integration sequence; `executor-plan-coherence` now uses those settled seams to test whether multi-slice plans converge on one working result. Broader instrumentation remains trigger-gated under Later. -**End-to-end comparison lane.** FE-1239 materializes the handoff that the completed reporting procedure names but does not run: two fresh rigorous elicitation lanes produce exact approved specifications, each specification crosses an immutable content-addressed handoff into both Brunch and Claude execution, and the unchanged Petri-editor oracle joins all four outputs into requirement-level evidence. Shared interface requirements are disclosed before elicitation and remain controlled baseline, while private/reveal requirements stay controller-only. The first tracer supports bounded within-case contrasts only — no winner, reliability claim, or cross-case causal conclusion. +**End-to-end comparison lane.** FE-1239 materializes the handoff that the completed reporting procedure names but does not run: two fresh rigorous elicitation lanes produce exact approved specifications, each specification crosses an immutable content-addressed handoff into both Brunch and Claude execution, and the unchanged Petri-editor oracle joins all four outputs into requirement-level evidence. Shared interface requirements are disclosed before elicitation and remain controlled baseline, while private/reveal requirements stay controller-only. The first tracer supports bounded within-case contrasts only — no winner, reliability claim, or cross-case causal conclusion. FE-1241 keeps that Petri editor as the greenfield case and adds frozen Brunch/Petrinaut brownfield packets plus controller-owned oracles. All three missions can now enter the same execution-only workflow and publication contract. Brownfield preparation uses pinned remote-free snapshots and target-bounded tools as experimental hygiene, not adversarial benchmark proof; Clay remains outside the campaign. **Current seams.** Brunch ships on the `1.0.0-alpha.x` line. One-shot `ask` is the only interactive structured-exchange terminal; D125-L's live ask registry provides headless discovery/answering, while the transcript-backed pending projection remains a compatibility surface for live offer tools after the legacy `present_question` pending branch retired. Sweep classification remains fail-closed and compile-time anchored to the exchange-schema terminal names (D117-L), while the larger capture-conditional watermark question remains A40-L. @@ -84,9 +84,9 @@ Close the entire first batch of walkthrough-related findings: remediation, the o ### Recently Completed +- 2026-07-22 `brownfield-comparison-cases` (FE-1241) — **✓ complete, learning-first:** frozen Brunch and Petrinaut packets, lightweight pinned-source preparation, deterministic oracles, publication-compatible attempt evidence, and portable CI are built. The Petrinaut calibration seed remains part of oracle identity and its source-backed addresses retain contrastive coverage. Expensive merged-reference preflight and adversarial admission were retired before a provider campaign because the current decision needs fast case-level product evidence, not an externally defensible benchmark. - 2026-07-21 `executor-slice-admission-parity` (FE-1240) — **✓ complete:** candidate admission now rejects scoped slices without executable criterion, design, or verification-machinery context; exact findings enter bounded repair, every admitted repaired slice survives preview/worker-context parsing, and the populated-plan execution guard remains fail-closed. - 2026-07-20 `comparison-reporting-skills` (FE-1232) — **✓ implementation complete, building on landed FE-1230:** added separate project-shared Notion publication and comparison-evidence reporting skills; elicitation, execution, end-to-end, and frozen campaign-strategy references; and executable guardrails for active-procedure precedence, safe mutation, validity-first interpretation, reproducible judging, failure retention, and audience-safe controller-only redaction. -- 2026-07-17 `agent-control-plane-closure` (FE-1216) — **✓ closed**: all seven required control-surface rows are built; foreground block replacement, load-on-demand context topology, distinct control ownership, observable resource reads, capability-honest background grants, process-lifetime immutable body/manifest caching, and one private posture renderer now satisfy D135-L/I68-L without schema, fixture, reviewer, or capture-experiment widening. Older completion history (including FE-1192/FE-1195 executor topology closure): [`docs/archive/PLAN_HISTORY.md`](../docs/archive/PLAN_HISTORY.md). ### Next — deferred operator evidence, shared host convergence, then Group 3 agent layer @@ -632,7 +632,7 @@ KA stream: lights_up: frozen spec -> Brunch/Claude isolated lanes -> hidden browser/Petri oracles -> adjudication excludes: host landing | product operator command | Cursor/Codex | broad benchmark claims -[hard]-> end-to-end-comparison-tracer (FE-1239) - status: active; staged 2x2 Petri-editor tracer + status: complete 2026-07-21; promoted staged 2x2 Petri-editor tracer reuses: FE-1210 fresh actor | FE-1230 attempt/oracle contracts | FE-1232 reporting grammar lights_up: mission -> exact elicited specs -> four execution cells -> requirement traceability excludes: winner | repetitions | multi-case causality | product command | ExecutionAttempt schema widening @@ -644,6 +644,11 @@ KA stream: lights_up: committed scope -> coherent candidate plan -> ordinary reconciliation slice reuses: canonical harness | isolated integration | epic verification excludes: browser-specific executor gate | durable plan kind | new lifecycle phase + -[hard]-> brownfield-comparison-cases + status: complete FE-1241 with learning-first pinned snapshots, deterministic case oracles, and publication-compatible execution runs + lights_up: same-base brownfield elicitation -> exact handoff -> crossed execution -> case oracle + cases: minimal-petri-net-editor = sole greenfield reference | FE-1201-derived brunch backend | PR #9051 petrinaut frontend + excludes: Clay | more greenfields | repetitions | aggregate winner | ExecutionAttempt widening # executor-run-environment (FE-1166) resolved 2026-07-15: policy merged (PR #302), # live remainder folded into executor-plan-synthesis (FE-1197), card consumed. diff --git a/memory/SPEC.md b/memory/SPEC.md index 2bf714c91..99b33e056 100644 --- a/memory/SPEC.md +++ b/memory/SPEC.md @@ -127,6 +127,8 @@ The POC's purpose is to prove three things: (a) that pi's coding-agent harness c | A46-L | A closed Zod schema can represent every `SessionPresentationDelta`, including `OpenAsk` questions with questionnaire questions present or exactly absent. **Validated 2026-07-15:** the contract composes the owned question schemas as exact alternatives and round-trips all delta variants without a cast or restated owner type. | medium | validated | D133-L; `src/rpc/__tests__/standalone-web-session-host.contract.test.ts` | | A47-L | Pi's valuable `InteractiveMode` TUI behavior can be preserved while one independent cwd-scoped Brunch session host owns the sole writable sealed Pi runtime, JSONL session manager, graph command authority, and semantic live-event fan-out used by both TUI and React clients. The unknown is the TUI attachment seam: Pi exports `InteractiveMode` over an in-process `AgentSessionRuntime`, not a remote TUI client. Validation must prove a real TUI + browser target without a second writable runtime, raw-Pi browser contract, or permanent second relay. Frontier: `shared-session-host-tracer`; retirement/cutover: `shared-session-host-cutover`. | medium | open | D39-L, D132-L, D133-L; I64-L, I65-L | | A48-L | A read-only semantic preflight can improve orientation-menu availability over deterministic graph-fact heuristics while returning within a ≤3-second interaction budget often enough to justify a model-backed path. Admission is limited to the configured soft recommended evaluator model; other foreground selections, missing evaluator auth, timeout, malformed output, or failure use the deterministic safe subset without restricting Pi-native `/model`. The path must consume no foreground turn, write no transcript/graph/session truth, and cache only in process by `{specId, lsn, operationalMode}`. A Brunch-owned reconciliation-blocker reader participates in gating now with an explicit empty implementation; injected non-empty blockers veto availability so future derived/persisted blocker wiring cannot be forgotten. Validation: a tracked human-approved contrastive catalog plus structured scratch tracer report; first run three uncached feasibility calls, then only if plausible run ten uncached labeled cases, requiring ≥8/10 within 3 seconds, exact flags on every completed response, at least one named deterministic-fallback miss corrected, and zero false-positive moves. If the budget or quality gate fails, retire only the model path, not the deterministic fallback. | low | open | D74-L, D109-L, D123-L; `walkthrough-remediation-2` | + + ### Active Decisions @@ -304,6 +306,8 @@ The POC's purpose is to prove three things: (a) that pi's coding-agent harness c | D132-L | Standalone interactive web uses one cwd-scoped combined Brunch host with a target-addressed inventory of sealed, in-process Pi `AgentSession`s (2026-07-14). The same process serves React assets/WebSocket Brunch RPC and owns coordinator/graph authority; it does not construct `InteractiveMode`, expose raw Pi RPC, spawn one Pi child per session, host multiple projects, or promise in-flight survival across host restart. One durable session target has one driver/many observers and cannot be opened as duplicate writable runtimes; write leases wait for real same-session contention. Hosted-session mutations return the complete `LiveSessionHostResult` discriminated `{status}` union as JSON-RPC success payloads, including domain refusals; only malformed boundary input and thrown host failures use JSON-RPC errors. FE-1200 materialized the one-target path and validated simultaneous target isolation (A43-L). Depends on: D5-L, D10-L, D33-L, D39-L, D84-L; req 4, req 31. Supersedes: D10-L/D72-L read-only-sidecar posture and D84-L singleton/TUI-owned target topology. | [`src/app/TOPOLOGY.md`](../src/app/TOPOLOGY.md), [`src/session/TOPOLOGY.md`](../src/session/TOPOLOGY.md), [`src/rpc/TOPOLOGY.md`](../src/rpc/TOPOLOGY.md), [`src/web/TOPOLOGY.md`](../src/web/TOPOLOGY.md) | active — target-addressed host and concurrent-session isolation materialized 2026-07-14 | | D133-L | Web and TUI share transport-neutral presentation semantics, not platform components or separately-authored domain decoders (2026-07-14). Validated Brunch `toolResult.details` project to a shared semantic presentation model; LLM-context, TUI, and React adapters render that meaning for their audiences. Web hydrates from a named JSONL-derived product projection and overlays target-addressed live events, then refetches canonical truth at settlement/reconnect. The host emits neither ANSI/TUI strings nor ready-made HTML, and no chat mirror/event store is introduced. FE-1200 materialized the full required persisted family inventory: ordinary text; free-text, choice, choices, and bounded-questionnaire terminal read-back; candidate, review-set, and digest offers/continuations; and receipt-bearing review settlement. Live React controls answer free text and listed single/multi choices; bounded questionnaires remain answerable headlessly through D38-L's schema-tagged string/JSON envelope, without a dedicated React questionnaire form. Depends on: A44-L, D17-L, D19-L, D104-L; req 12, req 17, req 32. | [`src/projections/TOPOLOGY.md`](../src/projections/TOPOLOGY.md), [`src/web/TOPOLOGY.md`](../src/web/TOPOLOGY.md) | active — full required-family coverage materialized 2026-07-15 | | D134-L | The approachable `/compare-specs` control topology is one top-level project Pi session acting as the simulated user and driving exactly one comparison-harness interactive subshell at a time. It does not spawn a Pi actor that then opens a nested interactive shell. The top-level agent alone receives the private mission; each harness receives only its approved minimal framing and the user's natural messages. Shared top-level context and lane order are acceptable and disclosed for this exploratory workflow; isolation-sensitive studies continue to use FE-1210's separate rigorous campaign recipe. Ordinary text interaction is the stock-Pi baseline for choices and approvals; a custom structured-question tool may enhance presentation but is never required. Setup checks are bounded to actual selected-harness prerequisites—no throwaway Pi/Claude provider turns or synthetic actor launches on every run. Supersedes: FE-1215's unmaterialized fresh-nested-actor design note. | [`.pi/prompts/compare-specs.md`](../.pi/prompts/compare-specs.md), [`testing/comparisons/missions/README.md`](../testing/comparisons/missions/README.md); PLAN `operator-comparison-workflow` | active — remediation materialized and focused Brunch smoke witnessed 2026-07-17; full comparison witness deferred | +| D136-L | Petrinaut's common mechanical gate uses the full pinned HASH checkout. Before a candidate lane, the controller materializes the parent tree and exact handoff, runs only `corepack yarn install --immutable --mode=skip-build`, and requires tracked source to remain clean. After the lane terminates at `promotion_prepared`, closed focused builds launch the standalone `/optimization` route; a deterministic loopback optimizer grades scenario-first configuration, fixed/optimized bindings, objective direction, request construction, progress/completion/error/cancellation, upstream abort, same-origin secrecy, and source-backed accessibility semantics. Synthetic fixtures prove sensitivity but do not author stronger semantics than the calibrated source behavior. The broad `/processes/draft` host/iframe shell remains non-gating outer evidence. Depends on: D134-L, A49-L retirement. | [`src/dev/TOPOLOGY.md`](../src/dev/TOPOLOGY.md) | active — learning-first gate materialized 2026-07-22 | +| D137-L | Historical replay preparation is experimental hygiene for fast product learning, not an adversarial security boundary (revised 2026-07-23). The controller materializes the pinned source tree into a fresh repository with no remote, adds the exact content-addressed packet, runs the one case-owned dependency recipe when required, checks source identity, packet bytes, and tracked cleanliness, then returns a lane-ready Brunch or Claude descriptor. Both lanes disable web/MCP surfaces where supported and bound file tools to the target, but the study reports this limitation rather than probing external services or proving host isolation. Depends on: A49-L retirement, D136-L. | [`src/dev/TOPOLOGY.md`](../src/dev/TOPOLOGY.md) | active — simplified for learning throughput | | D60-L | Agent context splits into pull / projection / render / surface, distinguishes graph-truth from active-context reads, and keeps `workspace.state` separate. Agent context (what the agent reasons over) spans `cwd` (filesystem kickoff heuristic — `.brunch?`, session count/length, README/markdown sizes, file counts), `graph` (overview/list/query), and `node` (variable-hop neighborhood). ... | [`src/graph/TOPOLOGY.md`](../src/graph/TOPOLOGY.md), [`src/projections/TOPOLOGY.md`](../src/projections/TOPOLOGY.md), [`src/agents/contexts/TOPOLOGY.md`](../src/agents/contexts/TOPOLOGY.md), [`src/app/TOPOLOGY.md`](../src/app/TOPOLOGY.md) | active | | D83-L | Context-render house style: a markdown frame (md-pen) with TOON for uniform data and a fenced ASCII tree for hierarchy, wrapped in `
` tags; agent context clusters into `` / `` / `` scopes. Refines D60-L's RENDER stage. LLM-facing agent-context renders adopt one consistent dialect instead of ad-hoc `[bracket]` + bullet lists: - **Audience scope.** `src/agents/contexts/` owns the LLM agent-context dialect (the `
` scope-clustering plus TOON data blocks and t ... | See archive snapshot for full rationale. | active | | D85-L | Suspended prompt-resource axis model: strategy/lens/method are no longer runtime state. A 2026-06-18 grill consolidation of the `agents/skills/` topology and the D58-L manifest axes, implemented across FE-893, FE-861, and FE-898, produced useful prompt-resource content and path topology. ... | [`src/agents/TOPOLOGY.md`](../src/agents/TOPOLOGY.md), [`src/agents/contexts/TOPOLOGY.md`](../src/agents/contexts/TOPOLOGY.md), [`src/agents/skills/TOPOLOGY.md`](../src/agents/skills/TOPOLOGY.md) | retired/suspended | @@ -688,6 +692,8 @@ For agent-as-user evaluation, the primary behavioral claim is **consequential-fa **FE-1230 greenfield execution-comparison assessment (2026-07-20).** Observability is **partial**: build/test output, git trees, browser DOM/accessibility state, console errors, JSON downloads, Brunch run/Petri artifacts, and target-visible interactions are text-native, while ordinary visual hierarchy and drag feel remain human judgments; product-private reasoning and diagnostics are excluded from common evidence. Reproducibility is **partial**: the approved Petri-editor specification, empty-repository base, public automation contract, Opus 4.8 model, budgets, and hidden oracle bytes freeze before the first valid lane, but generated implementation shape and model conduct vary; three Brunch repetitions detect gross instability rather than reliability tails. Controllability is **partial**: the controller can install and run unchanged build, browser, reference-model, round-trip, and negative-space checks against fresh lane outputs, while provider availability and product-native planning remain external. A minimal public accessibility contract gives both lanes stable roles/names without revealing hidden scenarios or expected results. +**Brownfield execution comparison assessment (updated 2026-07-23).** Observability is **partial**: pinned trees, diffs, build/test output, controller verdicts, temporary-host Git state, browser DOM/accessibility state, request/abort evidence, interventions, cleanup, and retained attempts are text-native; visual hierarchy and provider conduct remain external. Reproducibility is **high for deterministic mechanics and unproven for campaigns**: frozen Brunch/Petrinaut packets, remote-free pinned snapshots, case-owned recipes/oracles, the fake optimizer, and contrastive rivals are stable, while generated implementation and model conduct vary. Controllability is **partial**: Brunch host landing uses disposable repositories and an independent Git model; Petrinaut uses focused builds and same-origin browser checks after lane termination. Lightweight tool restrictions prevent accidental contamination but do not establish adversarial isolation; reports must preserve that limitation. + **Standalone-web tracer assessment (2026-07-14).** Observability is **partial**: JSONL, target-addressed RPC, semantic event frames, and accessible DOM/text states make structural session behavior observable; stream cadence, visual hierarchy, and error feel remain manual. Reproducibility is **high** for this tracer: paired temporary production boots use the deterministic faux provider and an existing JSONL session, not a live model or a static transcript golden. Controllability is **partial**: the middle loop controls the browser journey and cache/overlay loss, while a workbench walkthrough owns the bounded UX verdict. The normalizer may remove only declared nondeterministic ids/timestamps; it must not mask Brunch binding/runtime/exchange differences. **Combined trajectory/evaluation assessment.** The completed interactive TUI driver raises presentation-path controllability but does not by itself close causal attribution, real-provider reproducibility, or semantic ground truth. The first evaluation frontier should address those gaps with a minimal text-native legibility envelope over existing Pi lifecycle events, provider-payload introspection, Pi JSONL, TUI observations, and graph readback — not with OTel adoption. Its run/scenario/report contract must be product-neutral enough for later Claude Code or Cursor adapters, while Brunch-only trajectory enrichment remains optional diagnostic evidence. Campaigns are automated through a controlled user actor; deterministic checks own structural facts, and a small human-labeled set calibrates and audits semantic judgments. @@ -767,6 +773,10 @@ Dev-loop artifacts route to gitignored `.fixtures/scratch///`, res | Middle | **FE-1230 reference Petri model differential + metamorphic checks** | A tiny controller-owned P/T reference model independently computes enablement and weighted firing; the rendered marking/enablement agrees before and after firing, reset is idempotent, reload returns to the initial marking, export/import preserves structure, and disabled firing plus invalid operations leave state unchanged. | | Middle | **FE-1230 lane/process evidence contract** | Fresh repository identity, pinned model/product versions, budgets, start/end state, final git range, build/test/browser output, retries, interventions, terminal status, and cleanup are retained for every valid, failed, and invalid attempt. Only signals shared across lanes enter comparison packets; unavailable permission/cost/private-tool evidence is `not assessable`. | | Outer | **FE-1230 split execution judgment** | Identity-masked final trees/diffs plus common mechanical results receive criterion-level code/outcome review; a separate unblinded packet judges visible planning/execution conduct. Visual review mechanically rejects only catastrophic unusability (app absent, controls unreachable, interactions impossible); ordinary hierarchy, clarity, and feel remain qualitative and cannot override a mechanical failure. | +| Inner | **Brownfield case and repository-identity contracts** | The Petri editor remains the greenfield profile; Brunch and Petrinaut profiles name frozen axes, parent commit/tree identity, sanitized mission/baseline/registry hashes, compiled oracle ids, and exact handoffs. Separate targets start from the same remote-free pinned snapshot, keep controller material outside the cwd, and retain source/packet identity as experimental hygiene without claiming adversarial isolation. | +| Middle | **Brunch black-box `/brunch:land` journey plus independent Git model** | The controller creates disposable host/run/review repositories, resumes a settled candidate session under `PI_OFFLINE=1`, drives only the public TUI command, snapshots Git and run metadata before confirmation, then compares post-apply state with a controller-owned full-range reference model. This proves no pre-confirm mutation, complete multi-slice landing rather than final-commit-only application, mode-aware brownfield integration and greenfield materialization, bookkeeping exclusion, honest failure, and terminal `landed` without importing candidate host-landing modules or permitting a provider turn. | +| Middle | **Petrinaut standalone fake-provider browser and accessibility oracle** | In the full pinned HASH checkout, controller-prepared focused Petrinaut packages launch the public `/optimization` website route. A deterministic loopback fake optimizer proves capability-present Optimizations visibility, scenario-first configuration, fixed/optimized parameter selection, one metric plus direction, request construction, streamed trials/best-so-far/completion/error/cancellation, upstream abort, private-origin secrecy behind the same-origin proxy, and reachable accessible controls without requiring the real optimizer service. | +| Outer | **Brownfield live/qualitative evidence** | One fully provisioned HASH `/processes/draft` host/iframe smoke checks capability relay, optimizer proxying, iframe isolation, and adapter reality; one optional real-optimizer Petrinaut smoke checks live response compatibility. Identity-masked visual review judges Petrinaut hierarchy, clarity, and progress legibility; identity-masked code review judges both outputs. These are non-gating findings, are `not_assessable` when unavailable, and cannot override or repair common mechanical failures. | | Middle | **FE-1187 controlled provider conduct gate — paused** | On explicit re-entry, reconcile the extractor/oracle against the landed mixed-settlement contract, then run three fresh normalized-ingest samples. All must use free-text digest feedback, one bounded questionnaire when several questions exist, no combinatorial options, an honestly assigned per-node/per-edge review proposal when review is used, exact atomic settlement preservation, and no post-review mutation that completes or rewrites the approved proposal. Direct advisory mutation without review remains valid. Reports retain provider/model stamps and deterministic conduct markers; the stopped 2026-07-17 run is diagnostic and counts 0/3. | | Inner | **FE-1187 Impact Ledger golden + word-wrap-tolerant render-honesty** | Golden/inline snapshots at narrow/normal/wide widths lock populated-section-only canonical order, absence of empty heading/`None` pairs, per-node/per-edge settlement visibility, elision, `refs:` row shape, and the `obligation` fallback label for the borderless Impact Ledger renderer (D27-L/D131-L). `missingRenderedDetailsLeaves` is extended to reassemble `table`'s word-wrapped physical lines back into logical cells before leaf-presence checking, so a value silently split across wrapped rows cannot pass as "rendered" by accident. | | Middle | **FE-1187 Impact Ledger differential reference extractor** | A deliberately naive reference extractor (flat node/edge/term inventory, no styling or grouping) is compared against the ledger's code/connection inventory over the witnessed fixture plus hand-authored edge fixtures (empty group, single-node group, mixed-settlement group, term-only group, max-refs group). Proves item/status inventory completeness and empty-group omission independent of the "real" renderer's own logic. | @@ -842,6 +852,7 @@ The first required probe is M0: after manual TUI interaction, a checker proves ` - **Operator-led cross-product comparisons (FE-1215; D134-L remediation before later `saved-mission-comparison-witness` evidence).** The PM-facing comparison door is deliberately distinct from rigorous frozen-packet evaluation. One project Pi prompt conversationally creates or revises a rich private **agent-as-user mission** for a simulated user: their objective, context, priorities, preferences, constraints, knowledge, uncertainty, decision latitude, and conversational posture. The invoking top-level Pi agent receives that mission and directly performs the user's side of each interaction while driving exactly one comparison-harness subshell at a time; it must not spawn a Pi actor that opens another interactive shell. Each comparison harness receives only minimal visible framing plus the opening user message and subsequent mission-grounded answers. Harness selection and framing are run setup, not mission content. Ordinary conversational text is the portable baseline for operator choices and approvals; environment-specific structured-question tools are optional presentation only. Setup checks cover actual selected-harness prerequisites without synthetic actor/provider turns on every run. Editable missions live outside `.fixtures/` under `testing/comparisons/missions/`; each run snapshots the private mission, separately identified target-visible setup/interactions, and outputs under `.fixtures/runs/agent-as-user-comparison/` while temporary lane work stays in scratch. The readable operator report may expose the full private mission so elicitation can be compared against what each harness actually learned, but it keeps that baseline separate from target-visible evidence and declares no automatic winner or prescribed rubric. Because the top-level actor context spans sequential harnesses, this approachable workflow discloses order and does not claim the per-lane actor-process isolation required by rigorous frozen-packet studies; frozen reveal policies, matched budgets, blinding, fresh-per-lane actor sessions, structured adjudication, multi-run statistics, and scripted judges remain separate tools for focused improvement/regression claims. - **FE-1230 execution-comparison oracle boundary.** Execution cases are distinct from private elicitation missions: the human-approved specification and a minimal public runtime/accessibility contract are visible to every lane, while exact browser journeys, reference-model states, expected results, claim mapping, and adversarial fixtures remain controller-only and outside every lane cwd. The public contract requires a static production build at `dist/`, `npm run build`, `npm test`, and stable accessible roles/names for the canvas and named controls; it does not prescribe framework, source topology, implementation decomposition, test library, or internal state model. The versioned `petri-editor-browser-v2` suite tests/builds once, then runs every declared journey from a fresh browser context with public-only setup, per-journey runtime evidence, and non-blocking claim-linked verdicts that distinguish harness/setup failure from product assertion failure. Brunch stops at `promotion_prepared` and never lands. The retained first pair remains immutable; replay/promotion waits for its exact artifact paths. Mutants, masked/process judging, repetitions, and generalized campaign machinery are deferred until one valid end-to-end path exists. The tracer pins `anthropic/claude-opus-4-8` in both products, but same model does not imply equivalent hidden prompting or thinking controls. +- **Brownfield historical-replay oracle boundary (`brownfield-comparison-cases`).** Historical implementations bootstrap independent claims and focused rivals; matching code, decomposition, diff shape, or architecture is never a gate. Brunch replays only FE-1201's backend host-landing behavior through public `/brunch:land`, a controller-supplied settled session under `PI_OFFLINE=1`, and an independent temporary-repository Git model. Petrinaut replays only PR #9051's optimization UI in the full pinned HASH checkout; pre-existing optimizer backend/API capability is disclosed baseline. D136-L makes standalone `/optimization` the deterministic fake-provider gate after the one immutable install and closed focused builds; broad `/processes/draft` integration remains non-gating. D137-L creates the same pinned remote-free source plus exact packet for both executors and records its lightweight contamination ceiling. A merged-reference calibration is rerun only when the source, public contract, or oracle changes, not before every provider attempt. Real optimizer, fully provisioned host/iframe, repetitions, and masked reviews remain optional follow-up evidence. - **Shared-session-host convergence oracle (planned; `shared-session-host-tracer` → `shared-session-host-cutover`).** FE-1200's standalone proof is necessary but not sufficient for architectural replacement. The proving oracle must launch the production host as the sole owner of one writable sealed Pi runtime, attach a real TUI presentation and React observer/driver to the same durable target, exercise an ordinary turn plus one extension-owned structured ask and one TUI-only product interaction, detach/restart a client without ending the hosted runtime, and reject a duplicate writable open or conflicting driver. Durable settlement must converge through the same JSONL presentation, and browser traffic must remain Brunch semantic RPC/events rather than raw Pi RPC/events. Once A47-L retires, the cutover becomes a closed coverage sweep: every required TUI/web lifecycle, command/UI, exchange, transcript, graph-update, model/auth, and shutdown row has one host-owned path and a closure oracle; only then may `SessionEventRelay`, `brunch.sessionEvent`, `/rpc/driver`, and their sidecar harnesses be deleted. Human outer evidence must confirm that the TUI remains useful rather than becoming a thin degraded shell. - **Standalone-web compound oracle (2026-07-14; coverage completed 2026-07-15, `standalone-web-session-host`).** The initial five complementary oracles prove the host tracer: (1) inner RPC/host negative-space contracts require `(specId, sessionId)` on every lifecycle/driver/ask/event path and reject targetless fallback, duplicate writable opens, second drivers, and mismatched targets; (2) projection shape/malformed-detail tests keep JSONL and raw Pi/detail shapes behind the named semantic presentation; (3) a production-wired standalone-web browser journey, controlled only by a deterministic faux provider, asserts accessible hydration, streamed text, one `ask`, answer, and `agent_settled` states; (4) paired temporary web/TUI runs use a narrow declared normalizer to prove both live→settled→fresh JSONL hydration and equivalent Brunch binding/runtime/exchange semantics—no static JSONL golden; (5) a workbench manual checklist judges only stream cadence, busy/settled truthfulness, ask interaction, reload, and error feel. Browser cache/overlay loss is part of the middle-loop journey. The follow-on production-host concurrency differential retires A43-L with overlapping graph writes, asks, failure/recovery, target-local event sequences, reconnect, and separate JSONL readback; shared graph changes appear only through canonical `worldUpdate` continuity. The completed coverage pass adds projection no-loss/malformed tests for every required persisted ask terminal shape (including questionnaire read-back), React render/answer tests for free text and listed single/multi choices, headless schema-envelope questionnaire answering coverage (with no dedicated React questionnaire form), distinct candidate/review-set/digest production settlement/reconnect witnesses, concurrency/target isolation, and receipt-bearing review settlement. Live-provider conduct and process-restart survival remain explicitly deferred to their named later work; no second truth plane or raw Pi browser contract is introduced. - **Trace → eval → score → regression flywheel.** The first combined trajectory/evaluation proof is a controlled Brunch A/B over the existing consequential-fact claim, not a generic tracing platform or omnibus architecture score. One realistic non-inferable scenario carries a human-authored hidden-fact ledger, forbidden rivals, and a controlled reveal policy. The only intervention is an eval/dev ablation of the warrant-before-commit directive at the real prompt-composition seam; all other run conditions remain fixed. Three real-provider TUI-driven runs per arm produce a joined legibility envelope: stamped run configuration, directive inventory and content hashes, advertised/read/provider-visible directive state, ordered model/tool/exchange/TUI/graph effects, atomic evaluator judgments with evidence and rubric identity, and a replayable report. Completion requires promoted evidence that discriminates the full directive from the ablated rival; the report is an instrument, not the completion claim. This proves bounded evaluator discrimination, not competitor superiority or broad prompt quality. After the tracer, mine one real walkthrough failure into the same corpus. Keep OTel, broad subagent span joining, provider matrices, interaction-quality scoring, and competitor campaigns trigger-gated until a named claim requires them. @@ -876,6 +887,11 @@ The first required probe is M0: after manual TUI interaction, a checker proves ` | FE-1230 incomparable private process signals | Token accounting, permission prompts, hidden reasoning, and private tool traces differ by product or may be unavailable. | Compare only target-visible/common evidence; retain Brunch-only diagnostics outside judgment packets; mark unavailable cost/permission/tool metrics `not assessable` rather than fabricating parity. | | FE-1230 three-run reliability ceiling | Three Brunch repetitions can reveal catastrophic instability but cannot estimate tail reliability or generalize beyond one browser-app case. | State the ceiling in every report; make no broad speed/cost/reliability claim; add cases or repetitions only after the tracer proves the artifact/oracle contract and a named decision requires stronger evidence. | | FE-1230 same-model non-equivalence | Pinning Opus 4.8 does not equalize product-owned system prompts, tool policy, context management, thinking controls, or provider wrappers. | Frame the result as a same-base-model workflow/product comparison, record exact product/provider/model/configuration versions, and keep process interpretation separate from mechanical outcome correctness. | +| Brownfield historical-solution leakage | A target with later Git refs, remotes, GitHub/Linear/Notion access, or exact historical wording could retrieve the reference implementation and invalidate the replay. Package/dependency preparation may still need controlled network access. | D137-L materializes only the declared two-commit synthetic prefix (source identity root plus packet-only child), sanitizes missions, prepares selected dependencies controller-side, disables MCP/web/research surfaces, bounds filesystem tools, and denies target-command network. Admission adversarially rejects any third history, extra refs/remotes, path/symlink escape, weakened policy, reachable loopback/GitHub/Linear/Notion, unavailable isolation, or failing declared local checks; reachable solution sources invalidate the lane rather than reducing confidence. | +| Brunch derived-slice reference mismatch | The selected backend-only FE-1201 mission intentionally excludes TUI/RPC/web changes from the historical PR, so no historical whole-diff equality or completeness claim is valid. | Freeze a fresh requirement registry and independent temp-Git model/property oracle for the derived backend contract; use the historical implementation only to bootstrap fixtures and wrong rivals. | +| Brunch public-command oracle ceiling | `/brunch:land` is architecture-neutral and outcome-observable, but fresh sessions auto-kick a provider, settled-session/TUI setup can fail independently of candidate behavior, and a fixed temporary-Git fixture set cannot cover every repository topology. | Resume a controller-supplied settled session under `PI_OFFLINE=1`; separate `setup_failed` from `assertion_failed`; pair the black-box command journey with an independent full-range Git model and focused final-commit-only/bookkeeping rivals; make no reliability claim and add repository topologies only when a named failure or later campaign requires them. | +| Petrinaut fake-provider, host-integration, and visual ceiling | The focused `/optimization` hard gate proves the UI/provider contract but bypasses HASH's authenticated host/iframe bridge; the real `/processes/draft` shell requires unrelated GraphQL/codegen and broad platform/toolchain preparation. A deterministic fake also cannot prove every live optimizer response, while functional browser/accessibility assertions cannot judge ordinary visual hierarchy or progress feel. | Keep the standalone fake-provider oracle as the common mechanical gate. Run one fully provisioned HASH host/iframe smoke, one optional real-backend smoke, and masked human visual/code review as non-gating outer evidence; mark unavailable integration evidence `not_assessable`. Promote recurring live-shape or bridge failures into focused controller fixtures only when they can remain architecture-neutral; never let qualitative review override a mechanical failure. | +| Brownfield one-run portfolio ceiling | One Brunch and one Petrinaut 2×2 matrix broaden codebase/interface coverage but do not estimate reliability or establish an aggregate winner or causal advantage. | Report each case separately, retain `not_assessable`, and make no cross-case score, reliability, speed, or causal claim. Repetitions require a later named decision and separately frozen campaign. | | Full TUI automation | Cost exceeds value before the product state seams are proven, but startup-switcher regressions need a stronger visual signal than store-only checks. | Manual checklist plus artifact/query probe oracle; for FE-744 startup, add pty/ANSI-stripped capture assertions for the pre-Pi decision surface and absence of stale transcript bef … | | LLM elicitation quality and interaction flow | No stable deterministic ground truth for “good interview” early in the POC, and retired M1 scripted exchanges encoded only a thin obsolete exchange model. | Transcript-backed probe runs, human-reviewed probe reports, adversarial probe scenarios, expected structural coverage, … | | Subscription reconnect/resume | POC can prove initial state payload + live update without hardening network recovery yet. | Contract tests for initial state payload and ordered update sequence; **(2026-06-15)** reconnect/resume promoted to a `web-driver-streaming` battery claim — a turn-cut-point prope … | diff --git a/package.json b/package.json index d414d90df..9f684c07e 100644 --- a/package.json +++ b/package.json @@ -44,8 +44,8 @@ "db:generate": "drizzle-kit generate", "db:studio": "drizzle-kit studio", "test": "vitest --run --maxWorkers=4 --exclude='**/*.slow.test.ts'", - "test:full": "vitest --run --maxWorkers=4", - "test:slow": "vitest --run --maxWorkers=4 .slow.test.ts", + "test:full": "npm run test && npm run test:slow", + "test:slow": "vitest --run --maxWorkers=1 .slow.test.ts", "test:watch": "vitest", "test:context-surfaces": "vitest --run src/agents/contexts src/app/__tests__/print-workspace-state.test.ts src/session/__tests__/transcript-markdown.test.ts", "test:context-surfaces:watch": "vitest src/agents/contexts src/app/__tests__/print-workspace-state.test.ts src/session/__tests__/transcript-markdown.test.ts", diff --git a/src/.pi/extensions/__tests__/agent-runtime-runtime.test.ts b/src/.pi/extensions/__tests__/agent-runtime-runtime.test.ts index 291aff149..ee5979bff 100644 --- a/src/.pi/extensions/__tests__/agent-runtime-runtime.test.ts +++ b/src/.pi/extensions/__tests__/agent-runtime-runtime.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp } from 'node:fs/promises'; +import { mkdir, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -202,6 +202,88 @@ describe('Brunch agent runtime-state projection', () => { }); }); + it('keeps normal skill reads available while comparison roots remain bounded', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-read-policy-')); + const workspace = join(root, 'workspace'); + const packageRoot = join(root, 'package'); + const skillPath = join(packageRoot, 'SKILL.md'); + const workspaceFile = join(workspace, 'inside.md'); + const missingInsidePath = join(workspace, 'missing-inside.md'); + const missingOutsidePath = join(root, 'missing-outside.md'); + await mkdir(workspace); + await mkdir(packageRoot); + await writeFile(workspaceFile, '# Inside\n'); + await writeFile(skillPath, '# Skill\n'); + + type RegisteredRead = { + execute( + toolCallId: string, + params: { readonly path: string }, + signal: AbortSignal | undefined, + onUpdate: undefined, + ctx: { readonly cwd: string }, + ): Promise; + }; + const registerRead = (filesystemRoot?: string): RegisteredRead => { + let registered: RegisteredRead | undefined; + registerBrunchOperationalModePolicy( + { + registerTool: (tool: { readonly name: string }) => { + if (tool.name === 'read') registered = tool as unknown as RegisteredRead; + }, + getAllTools: () => ['read', 'grep', 'find', 'ls'].map((name) => ({ name })), + setActiveTools: (_tools: string[]) => {}, + on: (_event: string, _handler: (event: never, ctx?: never) => unknown) => {}, + } as never, + filesystemRoot === undefined ? {} : { filesystemRoot }, + ); + if (registered === undefined) throw new Error('read tool was not registered'); + return registered; + }; + + try { + await expect( + registerRead().execute('normal-read', { path: skillPath }, undefined, undefined, { + cwd: workspace, + }), + ).resolves.toBeDefined(); + await expect( + registerRead(await realpath(workspace)).execute( + 'aliased-in-root-read', + { path: workspaceFile }, + undefined, + undefined, + { cwd: workspace }, + ), + ).resolves.toBeDefined(); + await expect( + registerRead(workspace).execute( + 'missing-inside-read', + { path: missingInsidePath }, + undefined, + undefined, + { cwd: workspace }, + ), + ).rejects.toMatchObject({ code: 'ENOENT', syscall: 'access' }); + await expect( + registerRead(workspace).execute('bounded-read', { path: skillPath }, undefined, undefined, { + cwd: workspace, + }), + ).rejects.toThrow(`read-only tool path escapes target root: ${skillPath}`); + await expect( + registerRead(workspace).execute( + 'missing-outside-read', + { path: missingOutsidePath }, + undefined, + undefined, + { cwd: workspace }, + ), + ).rejects.toThrow(`read-only tool path escapes target root: ${missingOutsidePath}`); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + it('activates an explicit non-empty execute-mode tool set from registered tools', () => { const executeState: BrunchAgentState = { schemaVersion: 1, diff --git a/src/.pi/extensions/__tests__/registry.test.ts b/src/.pi/extensions/__tests__/registry.test.ts index e8c5c0562..4aa001ce9 100644 --- a/src/.pi/extensions/__tests__/registry.test.ts +++ b/src/.pi/extensions/__tests__/registry.test.ts @@ -84,7 +84,12 @@ import { registerBrunchSessionBoundary as sessionLifecycle } from '../session-ho import { hasBrunchDefaultRenderer } from '../shared/define-brunch-tool.js'; import { BRUNCH_TOOL_ACTIVITY_LABELS } from '../shared/tool-activity-labels.js'; import { assertProviderLegalToolSchema, hasToolParametersProvenance } from '../shared/tool-schema.js'; -import { parseSubagentMarkdown, type BrunchSubagentsDeps, type SubagentResult } from '../subagents/index.js'; +import { + BRUNCH_SUBAGENT_TOOL, + parseSubagentMarkdown, + type BrunchSubagentsDeps, + type SubagentResult, +} from '../subagents/index.js'; import { createSubagentToolCatalog } from '../subagents/session.js'; const extensionDefaults = { @@ -276,6 +281,45 @@ describe('Brunch explicit Pi extension registry', () => { expect(sessionStartIndexes[0]).toBeLessThan(sessionStartIndexes[1] ?? -1); }); + it('registers the comparison bundle without web or Specify subagent surfaces', async () => { + const recording = createRecordingExtensionApi(); + const definitions = new Map([ + ['planner', { name: 'planner' }], + ['worker', { name: 'worker' }], + ]) as BrunchSubagentsDeps['definitions']; + + await createBrunchPiExtensions(brunchChromeFixture, recording.onSessionBoundary, { + coordinator: {} as never, + graphMentionSource: { listMentionCandidates: () => [] }, + allowWebTools: false, + foregroundFilesystemRoot: '/tmp/comparison-target', + subagents: { + definitions, + delegatableAgents: [], + maxConcurrency: 1, + agentDir: '/tmp/agent', + createSettingsManager: () => ({}) as never, + resourceLoaderOptions: {} as never, + }, + })(recording.api); + + expect(recording.toolNames).not.toContain('web_fetch'); + expect(recording.toolNames).not.toContain('web_search'); + expect(recording.toolNames).not.toContain(BRUNCH_SUBAGENT_TOOL); + expect(recording.toolNames).toEqual( + expect.arrayContaining([ + 'read', + 'grep', + 'find', + 'ls', + BRUNCH_EXECUTE_ORCHESTRATE_TOOL, + BRUNCH_EXECUTE_AGENT_RESULT_TOOL, + BRUNCH_EXECUTE_TEST_RESULT_TOOL, + ]), + ); + expect([...definitions.keys()]).toEqual(['planner', 'worker']); + }); + it('registers execute_plan_check only with selected graph deps and returns side-effect-free findings', async () => { const registeredTools: Array<{ name: string; diff --git a/src/.pi/extensions/agent-runtime/runtime/index.ts b/src/.pi/extensions/agent-runtime/runtime/index.ts index de9da83ca..3fffd4e38 100644 --- a/src/.pi/extensions/agent-runtime/runtime/index.ts +++ b/src/.pi/extensions/agent-runtime/runtime/index.ts @@ -6,7 +6,9 @@ * than owning a second authority list. */ +import { realpath } from 'node:fs/promises'; import { homedir } from 'node:os'; +import { dirname, resolve, sep } from 'node:path'; import type { ExtensionAPI } from '@earendil-works/pi-coding-agent'; import { @@ -152,7 +154,10 @@ function supportsOperationalModePolicy(pi: ExtensionAPI): boolean { ); } -export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { +export function registerBrunchOperationalModePolicy( + pi: ExtensionAPI, + options: { readonly filesystemRoot?: string } = {}, +) { if (!supportsOperationalModePolicy(pi)) { return; } @@ -161,7 +166,11 @@ export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { ...getReadOnlyTools(process.cwd()).read, label: 'read', async execute(toolCallId, params, signal, onUpdate, ctx) { - return getReadOnlyTools(ctx.cwd).read.execute(toolCallId, params, signal, onUpdate); + const root = options.filesystemRoot ?? ctx.cwd; + if (options.filesystemRoot !== undefined) { + await assertBoundedReadPath(root, params.path); + } + return getReadOnlyTools(root).read.execute(toolCallId, params, signal, onUpdate); }, renderCall(args, theme) { const path = shortenPath(args.path || ''); @@ -189,7 +198,11 @@ export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { ...getReadOnlyTools(process.cwd()).grep, label: 'grep', async execute(toolCallId, params, signal, onUpdate, ctx) { - return getReadOnlyTools(ctx.cwd).grep.execute(toolCallId, params, signal, onUpdate); + const root = options.filesystemRoot ?? ctx.cwd; + if (options.filesystemRoot !== undefined) { + await assertBoundedReadPath(root, params.path ?? '.'); + } + return getReadOnlyTools(root).grep.execute(toolCallId, params, signal, onUpdate); }, renderCall(args, theme) { const path = shortenPath(args.path || '.'); @@ -214,7 +227,11 @@ export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { ...getReadOnlyTools(process.cwd()).find, label: 'find', async execute(toolCallId, params, signal, onUpdate, ctx) { - return getReadOnlyTools(ctx.cwd).find.execute(toolCallId, params, signal, onUpdate); + const root = options.filesystemRoot ?? ctx.cwd; + if (options.filesystemRoot !== undefined) { + await assertBoundedReadPath(root, params.path ?? '.'); + } + return getReadOnlyTools(root).find.execute(toolCallId, params, signal, onUpdate); }, renderCall(args, theme) { const path = shortenPath(args.path || '.'); @@ -238,7 +255,11 @@ export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { ...getReadOnlyTools(process.cwd()).ls, label: 'ls', async execute(toolCallId, params, signal, onUpdate, ctx) { - return getReadOnlyTools(ctx.cwd).ls.execute(toolCallId, params, signal, onUpdate); + const root = options.filesystemRoot ?? ctx.cwd; + if (options.filesystemRoot !== undefined) { + await assertBoundedReadPath(root, params.path ?? '.'); + } + return getReadOnlyTools(root).ls.execute(toolCallId, params, signal, onUpdate); }, renderCall(args, theme) { const path = shortenPath(args.path || '.'); @@ -295,3 +316,49 @@ export function registerBrunchOperationalModePolicy(pi: ExtensionAPI) { }; }); } + +async function assertBoundedReadPath(root: string, requestedPath: string): Promise { + const normalizedRoot = resolve(root); + const target = resolve(normalizedRoot, requestedPath); + const lexicallyContained = isContainedPath(normalizedRoot, target); + let realRoot: string; + let realTargetBoundary: string; + try { + [realRoot, realTargetBoundary] = await Promise.all([ + realpath(normalizedRoot), + realpathNearestExistingAncestor(target), + ]); + } catch (error) { + if (!lexicallyContained) { + throw new Error(`read-only tool path escapes target root: ${requestedPath}`); + } + throw error; + } + if (isContainedPath(realRoot, realTargetBoundary)) return; + if (!lexicallyContained) { + throw new Error(`read-only tool path escapes target root: ${requestedPath}`); + } + throw new Error(`read-only tool path escapes target root through symlink: ${requestedPath}`); +} + +async function realpathNearestExistingAncestor(target: string): Promise { + let candidate = target; + for (;;) { + try { + return await realpath(candidate); + } catch (error) { + if (!isErrorWithCode(error, 'ENOENT')) throw error; + const parent = dirname(candidate); + if (parent === candidate) throw error; + candidate = parent; + } + } +} + +function isErrorWithCode(error: unknown, code: string): error is NodeJS.ErrnoException { + return error instanceof Error && 'code' in error && error.code === code; +} + +function isContainedPath(root: string, target: string): boolean { + return target === root || target.startsWith(`${root}${sep}`); +} diff --git a/src/.pi/extensions/subagents/session.ts b/src/.pi/extensions/subagents/session.ts index bd158c40a..2b7f69441 100644 --- a/src/.pi/extensions/subagents/session.ts +++ b/src/.pi/extensions/subagents/session.ts @@ -20,7 +20,7 @@ * Its last assistant message is returned to the caller as tool-result content. */ -import { mkdir, writeFile } from 'node:fs/promises'; +import { mkdir, realpath, writeFile } from 'node:fs/promises'; import { dirname, isAbsolute, resolve, sep } from 'node:path'; import { @@ -206,13 +206,23 @@ export function createSubagentToolCatalog( injectedWorld?: SubagentInjectedWorld, ): Map { const pool = new Map(); - for (const definition of [ + for (const rawDefinition of [ createReadToolDefinition(cwd), createGrepToolDefinition(cwd), createFindToolDefinition(cwd), createLsToolDefinition(cwd), ]) { - pool.set(definition.name, definition as ToolDefinition); + const definition = rawDefinition as ToolDefinition; + const execute = definition.execute.bind(definition); + const boundedDefinition = { + ...definition, + execute: async (...args: Parameters) => { + const params = args[1] as { readonly path?: string }; + await assertBoundedExistingPath(cwd, params.path ?? '.'); + return await execute(...args); + }, + } as ToolDefinition; + pool.set(boundedDefinition.name, boundedDefinition); } for (const tool of [createWebSearchTool(), createWebFetchTool()]) { pool.set(tool.name, tool as unknown as ToolDefinition); @@ -239,7 +249,10 @@ function createWriteWorktreeFileTool(cwd: string) { parameters: toolParameters(WriteWorktreeFileParams), async execute(_toolCallId, params) { const target = boundedWorktreePath(cwd, params.path); + await assertBoundedExistingAncestor(cwd, target); await mkdir(dirname(target), { recursive: true }); + await assertBoundedExistingPath(cwd, dirname(target)); + await assertBoundedExistingTargetIfPresent(cwd, target); await writeFile(target, params.content, 'utf8'); return { content: [{ type: 'text' as const, text: `wrote ${params.path}` }], @@ -259,6 +272,43 @@ function boundedWorktreePath(cwd: string, rawPath: string): string { return target; } +async function assertBoundedExistingPath(cwd: string, rawPath: string): Promise { + const root = resolve(cwd); + const target = resolve(root, rawPath); + assertContainedPath(root, target); + const [realRoot, realTarget] = await Promise.all([realpath(root), realpath(target)]); + assertContainedPath(realRoot, realTarget); +} + +async function assertBoundedExistingAncestor(cwd: string, target: string): Promise { + let ancestor = dirname(target); + while (true) { + try { + await assertBoundedExistingPath(cwd, ancestor); + return; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + const parent = dirname(ancestor); + if (parent === ancestor) throw error; + ancestor = parent; + } + } +} + +async function assertBoundedExistingTargetIfPresent(cwd: string, target: string): Promise { + try { + await assertBoundedExistingPath(cwd, target); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } +} + +function assertContainedPath(root: string, target: string): void { + if (target !== root && !target.startsWith(`${root}${sep}`)) { + throw new Error('subagent filesystem path escapes the worktree'); + } +} + /** * Translate an agent's declared tool allowlist into SDK session options. * Throws on an unknown tool name (a Brunch authoring bug — fail loud). diff --git a/src/app/brunch-tui.ts b/src/app/brunch-tui.ts index 0236a02d1..806dc444c 100644 --- a/src/app/brunch-tui.ts +++ b/src/app/brunch-tui.ts @@ -1,4 +1,4 @@ -import { join } from 'node:path'; +import { join, resolve } from 'node:path'; import process from 'node:process'; import { @@ -114,6 +114,13 @@ export interface BrunchTuiLaunchContext { introspection?: BrunchTuiIntrospectionOptions; /** Product subagent tool registration for Specify mode; defaults on for normal launches. */ allowSubagents?: boolean; + /** + * Dev/eval-only historical-replay boundary. Presence selects the strict, + * non-configurable comparison composition rather than normal product grants. + */ + comparisonIsolation?: { + readonly targetRoot: string; + }; reportAsyncDiagnostic?: (diagnostic: { readonly type: 'warning'; readonly message: string }) => void; /** * Provider-backend substitution seam (faux provider in Tier-2 oracles). @@ -422,6 +429,10 @@ export function createBrunchAgentSessionRuntimeFactory( ): CreateAgentSessionRuntimeFactory { const { coordinator, productUpdates } = context; return async ({ cwd, agentDir: runtimeAgentDir, sessionManager, sessionStartEvent }) => { + const comparisonIsolation = context.comparisonIsolation; + if (comparisonIsolation && resolve(cwd) !== resolve(comparisonIsolation.targetRoot)) { + throw new Error('Brunch comparison runtime cwd must equal its isolated target root'); + } let currentWorkspace = await coordinator.bindCurrentSpecToReplacementSession(sessionManager); const graph = await openWorkspaceGraphRuntime(cwd); const graphDeps = { @@ -518,8 +529,9 @@ export function createBrunchAgentSessionRuntimeFactory( }), ); const agentState = projectBrunchAgentState(sessionManager.getBranch()); - const allowProductSubagents = context.allowSubagents !== false; - const shouldLoadSubagents = allowProductSubagents || agentState.operationalMode === 'execute'; + const allowProductSubagents = comparisonIsolation === undefined && context.allowSubagents !== false; + const shouldLoadSubagents = + comparisonIsolation !== undefined || allowProductSubagents || agentState.operationalMode === 'execute'; const subagents = shouldLoadSubagents ? await loadBrunchSubagents({ cwd, @@ -528,6 +540,7 @@ export function createBrunchAgentSessionRuntimeFactory( allowProductSubagents && agentState.operationalMode === 'specify' ? ['explorer', 'researcher', 'projector', 'reviewer'] : [], + ...(comparisonIsolation ? { includedAgents: ['planner', 'worker'] } : {}), world: { graph: { specId: currentWorkspace.spec.id, @@ -561,6 +574,12 @@ export function createBrunchAgentSessionRuntimeFactory( ...(context.liveExchange ? { liveExchange: context.liveExchange.opener } : {}), ...(context.introspection ? { introspection: context.introspection } : {}), ...(subagents ? { subagents } : {}), + ...(comparisonIsolation + ? { + allowWebTools: false, + foregroundFilesystemRoot: comparisonIsolation.targetRoot, + } + : {}), promptContext: () => { const specId = currentWorkspace.spec.id; const selectedSpec = selectedSpecContext(graph, specId); diff --git a/src/app/pi-extensions.ts b/src/app/pi-extensions.ts index 92963e4ac..c6002e605 100644 --- a/src/app/pi-extensions.ts +++ b/src/app/pi-extensions.ts @@ -250,6 +250,10 @@ export interface BrunchPiExtensionsOptions extends BrunchCommandsOptions { introspection?: BrunchPiIntrospectionOptions; continuityDrains?: () => readonly ContinuityDrain[]; executionPorts?: Partial; + /** Dev/eval-only filesystem boundary for foreground read-only tools. */ + foregroundFilesystemRoot?: string; + /** Defaults on; strict comparison launches set false. */ + allowWebTools?: boolean; /** * Optional subagent registry (D44-L/D92-L). When provided with a non-empty * code-owned delegatable set, the product `subagent` tool is registered and @@ -340,9 +344,13 @@ export function createBrunchPiExtensions( }), registerBrunchBranchPolicyHandlers, registerBrunchCompaction, - registerBrunchOperationalModePolicy, + (api) => + registerBrunchOperationalModePolicy( + api, + options.foregroundFilesystemRoot ? { filesystemRoot: options.foregroundFilesystemRoot } : undefined, + ), registerBrunchContext, - registerBrunchWebTools, + ...(options.allowWebTools === false ? [] : [registerBrunchWebTools]), registerBrunchExecuteStatus, ...(options.productUpdates ? [ diff --git a/src/app/pi-subagents.ts b/src/app/pi-subagents.ts index 68b222c9d..29f600aa3 100644 --- a/src/app/pi-subagents.ts +++ b/src/app/pi-subagents.ts @@ -24,6 +24,7 @@ export interface LoadBrunchSubagentsOptions { readonly cwd: string; readonly agentDir: string; readonly delegatableAgents: readonly string[]; + readonly includedAgents?: readonly string[]; readonly world?: LoadBrunchSubagentsWorld; } @@ -45,10 +46,20 @@ export interface LoadBrunchSubagentsWorld { * non-empty code-owned delegatable set. */ export async function loadBrunchSubagents(options: LoadBrunchSubagentsOptions): Promise { - const [definitions, config] = await Promise.all([ + const [loadedDefinitions, config] = await Promise.all([ loadSubagentDefinitions(subagentAgentsDir()), loadSubagentConfig(subagentConfigPath()), ]); + const definitions = + options.includedAgents === undefined + ? loadedDefinitions + : new Map( + options.includedAgents.flatMap((name) => { + const definition = loadedDefinitions.get(name); + if (!definition) throw new Error(`Brunch subagent definition is unavailable: ${name}`); + return [[name, definition] as const]; + }), + ); return { definitions, diff --git a/src/dev/TOPOLOGY.md b/src/dev/TOPOLOGY.md index 823430c95..207dca50f 100644 --- a/src/dev/TOPOLOGY.md +++ b/src/dev/TOPOLOGY.md @@ -11,14 +11,28 @@ This directory owns Brunch-only development loops and curation seams. Nothing he - faux/introspection/tier-2 harnesses used by tests and probes - dev-only witnesses such as `generate-fan-out-witness.ts` - controller-owned execution-comparison packets, lane adapters, immutable evidence contracts, and independently executable black-box oracle journeys (`execution-comparison/`) -- controller-owned end-to-end comparison composition (`end-to-end-comparison.ts` and `end-to-end-comparison/`): content-addressed study and exact-byte handoff contracts, isolated Brunch/Claude adapters, a closed four-cell matrix over unchanged `ExecutionAttempt` leaves, requirement traceability, and audience-safe redaction -- the shell-callable execution-comparison operator wrapper (`execution-comparison-operator.ts`), which lists/resolves frozen cases, prepares only Brunch/Claude targets, invokes the existing oracle after lane termination, and validates immutable attempt records for the project-local `/compare-execution` prompt +- controller-owned end-to-end comparison composition (`end-to-end-comparison.ts` and `end-to-end-comparison/`): content-addressed study and exact-byte handoff contracts, Brunch/Claude adapters, a closed four-cell matrix over unchanged `ExecutionAttempt` leaves, requirement traceability, audience-safe redaction, and D137-L pinned-source preparation +- the shell-callable execution-comparison operator wrapper (`execution-comparison-operator.ts`), which lists/resolves frozen cases, reports each case's repository and oracle identity, prepares Brunch/Claude targets, invokes the selected oracle after lane termination, and validates immutable attempt records for the project-local `/compare-execution` prompt - the shell-callable comparison provenance writer (`comparison-provenance.ts`), which captures one write-once release/controller snapshot after setup approval and before any comparison lane - the standalone component preview harness (`scripts/dev-components.ts` → `src/dev/component-preview.ts`) for previewing `.pi/components` in isolation on a real terminal, with no workspace/session/DB - the agent-drivable PTY walkthrough fallback (`npm run tui-driver` → `src/dev/tui-driver.ts`): named `expect`-pumped PTY sessions with guarded fifo control, headless-xterm screen rendering, and wait-for-text; use it when the canonical project-local `pi-interactive-shell` overlay cannot bind in a sandbox/headless host; sessions live under gitignored `.fixtures/scratch/tui-driver/` It does not own published CLI behavior, public RPC contracts, database imports from outside `graph/`, or the external `pi-interactive-shell`/`zigpty` runtime. The extension is permanent project development tooling declared under root `.pi` for host-capable manual runs; it never enters Brunch's shipped package manifest, sealed `src/.pi` profile, or runtime dependency graph. `docs/praxis/manual-testing.md` owns the measured priority order, install/health check, trust and auto-install implications, takeover/return, artifact bounds, and teardown procedure. +## Historical Replay Preparation + +`execution-comparison/historical-replay-target.ts` is D137-L's learning-first pinned-source operation: callers provide the frozen case, lane, source, target, and controller roots and receive a lane-ready descriptor after source materialization, exact packet freeze, optional case-owned dependency preparation, lightweight verification, and lane finalization. The target is a fresh repository with no remote; its root records the pinned source commit/tree and its packet child contains the exact approved public files. Preparation checks source identity, packet hashes, tracked cleanliness, and the selected dependency result. Brunch readiness includes a positive brownfield `specId`; Claude readiness omits `specId`; both include adapter-produced launch metadata. + +`end-to-end-comparison/solution-isolation.ts` owns this small hygiene boundary plus the lane tool-policy values. Claude receives empty MCP/settings/plugin state, disabled web tools, its native target policy, and denied reads for controller/source roots. Brunch disables foreground web tools, bounds foreground and child file tools to the target, and exposes only planner/worker execution subagents. The module does not probe GitHub/Linear/Notion, materialize a merged reference, brand a host verifier, or claim adversarial isolation. Reports name this ceiling explicitly; suspicious historical exposure invalidates an attempt rather than triggering hidden repair. + +## Brownfield Comparison Oracles + +`execution-comparison/host-landing-oracle.ts` is the public Brunch controller entry point; its same-named private subtree owns disposable Git fixtures, settled-session public-TUI actuation, the independent Git outcome model, and report types. The oracle resumes only controller-supplied settled sessions under `PI_OFFLINE=1`, invokes `/brunch:land` through the built candidate, and distinguishes setup invalidity from claim failure. + +Petrinaut preparation selected by D136-L is split across the historical-replay operation and oracle. `execution-comparison/operator-cli.ts` resolves every `pinned_git` contract into the shared operation; Petrinaut then runs the one immutable install (`corepack yarn install --immutable --mode=skip-build`) before either lane. Install failure or tracked-source mutation removes only the newly owned target. After lane termination, `execution-comparison/petrinaut-optimization-oracle.ts` runs the closed focused builds, deterministic loopback optimizer responses, and black-box browser journeys against `/optimization`. The compiled sequence includes `@hashintel/refractive` after design-system/core/optimizer prerequisites and before Petrinaut UI. Source-backed mechanical addresses and contrastive rivals remain frozen in the case. Recalibrate parent-fails/reference-passes only when the pinned source, public contract, or oracle changes; routine attempts do not duplicate the HASH install or reference checkout. + +`execution-comparison-operator.ts` dispatches both brownfield oracles and the retained Petri browser oracle through a closed compile-time registry and includes every implementation module in the immutable oracle-pack hash; manifests cannot select paths, commands, or plugins at runtime. + ## Launcher Surface `npm run dev-cli` is the front door for local workbenches; `npm run dev` directly launches the product CLI from TypeScript source. diff --git a/src/dev/end-to-end-comparison.ts b/src/dev/end-to-end-comparison.ts index c60631972..ca5e1a67f 100644 --- a/src/dev/end-to-end-comparison.ts +++ b/src/dev/end-to-end-comparison.ts @@ -52,3 +52,14 @@ export { type PreparedClaudeExecutionWorkspace, } from './end-to-end-comparison/claude-adapter.js'; export { retainExecutionCell } from './end-to-end-comparison/execution-cell.js'; +export { + createBrunchSolutionIsolationPolicy, + createClaudeSolutionIsolationPolicy, + materializePinnedSourceTree, + verifyPreparedHistoricalReplay, + type BrunchSolutionIsolationPolicy, + type ClaudeSolutionIsolationPolicy, + type MaterializedHistoricalReplayPrefix, + type MaterializedPinnedSourceTree, + type SolutionIsolationPolicy, +} from './end-to-end-comparison/solution-isolation.js'; diff --git a/src/dev/end-to-end-comparison/__tests__/case-profile.test.ts b/src/dev/end-to-end-comparison/__tests__/case-profile.test.ts new file mode 100644 index 000000000..cd4a171ac --- /dev/null +++ b/src/dev/end-to-end-comparison/__tests__/case-profile.test.ts @@ -0,0 +1,115 @@ +import { execFile } from 'node:child_process'; +import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +import { describe, expect, it } from 'vitest'; + +import { loadPublicCasePacket } from '../../execution-comparison/case-contract.js'; +import { loadEndToEndStudyContract } from '../study-contract.js'; + +const repositoryRoot = fileURLToPath(new URL('../../../../', import.meta.url)); +const petriStudy = join( + repositoryRoot, + 'testing/end-to-end-comparisons/cases/minimal-petri-net-editor/study-contract.json', +); +const brunchStudy = join( + repositoryRoot, + 'testing/end-to-end-comparisons/cases/brunch-host-landing/study-contract.json', +); +const petrinautStudy = join( + repositoryRoot, + 'testing/end-to-end-comparisons/cases/petrinaut-optimization/study-contract.json', +); +const brunchExecution = join(repositoryRoot, 'testing/execution-comparisons/cases/brunch-host-landing'); +const petrinautExecution = join(repositoryRoot, 'testing/execution-comparisons/cases/petrinaut-optimization'); +const execFileAsync = promisify(execFile); + +describe('compiled end-to-end comparison case profiles', () => { + it('loads the unchanged Petri case and exactly two pinned brownfield cases', async () => { + const [petri, brunch, petrinaut, brunchPacket, petrinautPacket] = await Promise.all([ + loadEndToEndStudyContract({ repositoryRoot, contractPath: petriStudy }), + loadEndToEndStudyContract({ repositoryRoot, contractPath: brunchStudy }), + loadEndToEndStudyContract({ repositoryRoot, contractPath: petrinautStudy }), + loadPublicCasePacket(brunchExecution), + loadPublicCasePacket(petrinautExecution), + ]); + + expect(petri.contract).toMatchObject({ + id: 'minimal-petri-net-editor-e2e-v1', + caseId: 'minimal-petri-net-editor-v1', + oracle: { id: 'minimal-petri-net-editor-oracles-v2' }, + }); + expect(brunch.contract).toMatchObject({ + id: 'brunch-host-landing-e2e-v1', + caseId: 'brunch-host-landing-v1', + oracle: { id: 'brunch-host-landing-oracles-v1' }, + source: { + parentCommit: 'f5a423b19f76cf345d88053456870a126e451618', + parentTree: 'a5709715a07faef0b96d3e05a7b6f9f8d693dd38', + }, + }); + expect(petrinaut.contract).toMatchObject({ + id: 'petrinaut-optimization-e2e-v1', + caseId: 'petrinaut-optimization-v1', + oracle: { id: 'petrinaut-optimization-oracles-v1' }, + source: { + parentCommit: '5c7a2d9db5caa851c38938f4b1bac19005b0e978', + parentTree: 'a3e08cf75e00cc9016c931f4665341506e03533e', + }, + }); + expect(brunchPacket.contract.case).toMatchObject({ + product: 'brunch', + mode: 'brownfield', + scope: 'single_feature', + surface: 'backend', + repository: { + substrate: 'pinned_git', + parentCommit: brunch.contract.source?.parentCommit, + parentTree: brunch.contract.source?.parentTree, + }, + }); + expect(petrinautPacket.contract.case).toMatchObject({ + product: 'petrinaut', + mode: 'brownfield', + scope: 'single_feature', + surface: 'frontend', + repository: { + substrate: 'pinned_git', + parentCommit: petrinaut.contract.source?.parentCommit, + parentTree: petrinaut.contract.source?.parentTree, + }, + }); + if (brunch.contract.source === undefined || petrinaut.contract.source === undefined) { + throw new Error('brownfield profiles must declare source identity'); + } + const actualBrunchTree = ( + await execFileAsync('git', ['rev-parse', `${brunch.contract.source.parentCommit}^{tree}`], { + cwd: repositoryRoot, + }) + ).stdout.trim(); + expect(actualBrunchTree).toBe(brunch.contract.source.parentTree); + }); + + it('keeps historical solution locators and controller expectations out of target-visible artifacts', async () => { + const visiblePaths = [ + 'testing/comparisons/missions/brunch-host-landing.md', + 'testing/end-to-end-comparisons/cases/brunch-host-landing/shared-baseline.md', + 'testing/execution-comparisons/cases/brunch-host-landing/spec.md', + 'testing/execution-comparisons/cases/brunch-host-landing/public-contract.json', + 'testing/comparisons/missions/petrinaut-optimization.md', + 'testing/end-to-end-comparisons/cases/petrinaut-optimization/shared-baseline.md', + 'testing/execution-comparisons/cases/petrinaut-optimization/spec.md', + 'testing/execution-comparisons/cases/petrinaut-optimization/public-contract.json', + ]; + const visible = ( + await Promise.all(visiblePaths.map((path) => readFile(join(repositoryRoot, path), 'utf8'))) + ).join('\n'); + + expect(visible).not.toMatch(/FE-1201|PR #336|pull\/336|0092a549|merged reference|historical solution/iu); + expect(visible).not.toMatch( + /FE-1162|PR #9051|pull\/9051|276e17d7|expected (?:tree|event|request)|final-commit-only|bookkeeping-retaining|historical solution/iu, + ); + }); +}); diff --git a/src/dev/end-to-end-comparison/__tests__/execution-adapters.test.ts b/src/dev/end-to-end-comparison/__tests__/execution-adapters.test.ts index 4b0474ecb..dac6de493 100644 --- a/src/dev/end-to-end-comparison/__tests__/execution-adapters.test.ts +++ b/src/dev/end-to-end-comparison/__tests__/execution-adapters.test.ts @@ -8,10 +8,12 @@ import { afterEach, describe, expect, it } from 'vitest'; import { runCommand, type CommandRunner } from '../../../app/command-runner.js'; import { prepareBrunchExecutionCell } from '../brunch-adapter.js'; import { + createClaudeExecutionLaunch, finalizeClaudeExecutionWorkspace, prepareClaudeExecutionWorkspace, runClaudeExecutionWorkspace, } from '../claude-adapter.js'; +import { createClaudeSolutionIsolationPolicy } from '../solution-isolation.js'; const roots: string[] = []; const contractTemplatePath = fileURLToPath( @@ -35,14 +37,29 @@ async function handoff(root: string): Promise<{ specificationPath: string; speci } describe('end-to-end execution adapters', () => { + it('encodes brownfield Claude tool lists as single CLI option values', () => { + const workspaceDir = '/tmp/brunch-brownfield-claude'; + const policy = createClaudeSolutionIsolationPolicy(workspaceDir, ['/tmp/controller']); + const launch = createClaudeExecutionLaunch({ workspaceDir, isolationPolicy: policy }); + const allowedIndex = launch.args.indexOf('--allowedTools'); + const disallowedIndex = launch.args.indexOf('--disallowedTools'); + + expect(launch.args[allowedIndex + 1]).toBe(policy.allowedTools.join(',')); + expect(launch.args[allowedIndex + 2]).toBe('--disallowedTools'); + expect(launch.args[disallowedIndex + 1]).toBe('WebFetch,WebSearch'); + expect(launch.args[disallowedIndex + 2]).toBe('--settings'); + }); + it('prepares Brunch from exact free-form bytes while preserving the legacy public packet', async () => { const root = await mkdtemp(join(tmpdir(), 'brunch-e2e-adapter-')); roots.push(root); const selected = await handoff(root); + const controllerRoot = join(root, 'controller'); + const repositoryRoot = fileURLToPath(new URL('../../../../', import.meta.url)); const prepared = await prepareBrunchExecutionCell({ cellRoot: join(root, 'cells', 'brunch-spec--brunch'), workspaceDir: join(root, 'targets', 'brunch'), - controllerRoot: join(root, 'controller'), + controllerRoot, specificationPath: selected.specificationPath, publicContractTemplatePath: contractTemplatePath, }); @@ -59,9 +76,8 @@ describe('end-to-end execution adapters', () => { '--spec-id', '1', ], - cwd: fileURLToPath(new URL('../../../../', import.meta.url)), + cwd: repositoryRoot, }); - expect(JSON.stringify(prepared)).not.toContain(join(root, 'controller')); }); it('prepares and finalizes an isolated Claude repository from the same exact packet', async () => { @@ -69,9 +85,10 @@ describe('end-to-end execution adapters', () => { roots.push(root); const selected = await handoff(root); const workspaceDir = join(root, 'targets', 'claude'); + const controllerRoot = join(root, 'controller'); const prepared = await prepareClaudeExecutionWorkspace({ workspaceDir, - controllerRoot: join(root, 'controller'), + controllerRoot, specificationPath: selected.specificationPath, publicContractTemplatePath: contractTemplatePath, }); @@ -92,7 +109,10 @@ describe('end-to-end execution adapters', () => { '--no-session-persistence', ]), ); - expect(JSON.stringify(prepared.launch)).not.toContain(join(root, 'controller')); + expect(prepared.launch.args).not.toEqual( + expect.arrayContaining(['--strict-mcp-config', '--settings', '--tools']), + ); + expect(prepared.launch.args.at(-1)).not.toContain(controllerRoot); await writeFile(join(workspaceDir, 'package.json'), '{"scripts":{"test":"true","build":"true"}}\n'); const finalized = await finalizeClaudeExecutionWorkspace({ workspaceDir }); diff --git a/src/dev/end-to-end-comparison/__tests__/factorial-browser-oracle.slow.test.ts b/src/dev/end-to-end-comparison/__tests__/factorial-browser-oracle.slow.test.ts index 27693eedd..a9c41dd05 100644 --- a/src/dev/end-to-end-comparison/__tests__/factorial-browser-oracle.slow.test.ts +++ b/src/dev/end-to-end-comparison/__tests__/factorial-browser-oracle.slow.test.ts @@ -33,14 +33,21 @@ afterAll(async () => { describe('synthetic end-to-end factorial oracle composition', () => { it('projects one unchanged known-good oracle result across all four synthetic cells', async () => { - const report = await runPetriEditorBrowserOracle({ appDir: fixtureDir, caseDir }); - const reports = MATRIX_CELL_IDS.map((cellId) => ({ cellId, report })); - - expect(reports.map(({ cellId }) => cellId)).toEqual(MATRIX_CELL_IDS); - for (const { report } of reports) { - expect(report.status).toBe('passed'); - expect(report.checks).toHaveLength(5); - expect(report.checks.every((check) => check.status === 'passed')).toBe(true); + const scratchRoot = await mkdtemp(join(tmpdir(), 'brunch-factorial-oracle-')); + try { + const isolatedFixtureDir = join(scratchRoot, 'known-good'); + await cp(fixtureDir, isolatedFixtureDir, { recursive: true }); + const report = await runPetriEditorBrowserOracle({ appDir: isolatedFixtureDir, caseDir }); + const reports = MATRIX_CELL_IDS.map((cellId) => ({ cellId, report })); + + expect(reports.map(({ cellId }) => cellId)).toEqual(MATRIX_CELL_IDS); + for (const { report } of reports) { + expect(report.status).toBe('passed'); + expect(report.checks).toHaveLength(5); + expect(report.checks.every((check) => check.status === 'passed')).toBe(true); + } + } finally { + await rm(scratchRoot, { recursive: true, force: true }); } }, 240_000); }); diff --git a/src/dev/end-to-end-comparison/__tests__/solution-isolation.test.ts b/src/dev/end-to-end-comparison/__tests__/solution-isolation.test.ts new file mode 100644 index 000000000..07ba7412d --- /dev/null +++ b/src/dev/end-to-end-comparison/__tests__/solution-isolation.test.ts @@ -0,0 +1,122 @@ +import { createHash } from 'node:crypto'; +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it } from 'vitest'; + +import { runCommand } from '../../../app/command-runner.js'; +import { + createBrunchSolutionIsolationPolicy, + createClaudeSolutionIsolationPolicy, + materializePinnedSourceTree, + verifyPreparedHistoricalReplay, +} from '../solution-isolation.js'; + +const roots: string[] = []; + +afterEach(async () => { + await Promise.all(roots.splice(0).map(async (root) => await rm(root, { recursive: true, force: true }))); +}); + +describe('learning-first historical snapshots', () => { + it('materializes a pinned tree into a fresh remote-free repository', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-comparison-source-')); + roots.push(root); + const source = join(root, 'source'); + const target = join(root, 'target'); + await run('git', ['init', '--initial-branch=main', source], root); + await writeFile(join(source, 'README.md'), 'pinned source\n'); + await run('git', ['add', '--all'], source); + await run( + 'git', + ['-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-m', 'source'], + source, + ); + const sourceCommit = await output('git', ['rev-parse', 'HEAD'], source); + + const materialized = await materializePinnedSourceTree({ + sourceRepositoryDir: source, + sourceCommit, + targetDir: target, + }); + + expect(await readFile(join(target, 'README.md'), 'utf8')).toBe('pinned source\n'); + expect(materialized.sourceCommit).toBe(sourceCommit); + expect(await output('git', ['remote'], target)).toBe(''); + expect(await output('git', ['rev-list', '--count', 'HEAD'], target)).toBe('1'); + }); + + it('checks frozen packet bytes and tracked cleanliness without claiming adversarial isolation', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-comparison-prefix-')); + roots.push(root); + const source = join(root, 'source'); + const target = join(root, 'target'); + await run('git', ['init', '--initial-branch=main', source], root); + await writeFile(join(source, 'README.md'), 'source\n'); + await run('git', ['add', '--all'], source); + await run( + 'git', + ['-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-m', 'source'], + source, + ); + const sourceCommit = await output('git', ['rev-parse', 'HEAD'], source); + const materialized = await materializePinnedSourceTree({ + sourceRepositoryDir: source, + sourceCommit, + targetDir: target, + }); + const packet = [ + ['public-contract.json', '{}\n'], + ['spec.md', '# Mission\n'], + ] as const; + for (const [path, bytes] of packet) await writeFile(join(target, path), bytes); + await run('git', ['add', '--all'], target); + await run( + 'git', + ['-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-m', 'packet'], + target, + ); + const baseSha = await output('git', ['rev-parse', 'HEAD'], target); + const prefix = { + ...materialized, + baseSha, + packetFiles: packet.map(([path, bytes]) => ({ path, sha256: sha256(bytes) })), + }; + + await expect(verifyPreparedHistoricalReplay({ prefix })).resolves.toBeUndefined(); + await writeFile(join(target, 'spec.md'), '# Drifted\n'); + await expect(verifyPreparedHistoricalReplay({ prefix })).rejects.toThrow( + /tracked source changes|drifted/u, + ); + }); + + it('retains lightweight target policies for both lanes', () => { + expect(createBrunchSolutionIsolationPolicy('/tmp/target')).toMatchObject({ + executor: 'brunch', + foregroundWebTools: false, + executionSubagents: ['planner', 'worker'], + }); + expect(createClaudeSolutionIsolationPolicy('/tmp/target', ['/tmp/controller'])).toMatchObject({ + executor: 'claude_code', + strictMcp: true, + webTools: false, + nativeSandbox: { deniedReadRoots: ['/tmp/controller'] }, + }); + }); +}); + +function sha256(value: string): string { + return `sha256:${createHash('sha256').update(value).digest('hex')}`; +} + +async function run(command: string, args: readonly string[], cwd: string): Promise { + const result = await runCommand(command, args, { cwd }); + if (result.exitCode !== 0) throw new Error(result.stderr || result.stdout); +} + +async function output(command: string, args: readonly string[], cwd: string): Promise { + const result = await runCommand(command, args, { cwd }); + if (result.exitCode !== 0) throw new Error(result.stderr || result.stdout); + return result.stdout.trim(); +} diff --git a/src/dev/end-to-end-comparison/brunch-adapter.ts b/src/dev/end-to-end-comparison/brunch-adapter.ts index eae129578..6bd125555 100644 --- a/src/dev/end-to-end-comparison/brunch-adapter.ts +++ b/src/dev/end-to-end-comparison/brunch-adapter.ts @@ -15,6 +15,24 @@ export interface ExecutionLaunch { readonly cwd: string; } +export function createBrunchExecutionLaunch(input: { + readonly workspaceDir: string; + readonly specId: number; +}): ExecutionLaunch { + return { + command: 'npx', + args: [ + 'tsx', + 'src/dev/execution-comparison-brunch.ts', + '--workspace', + input.workspaceDir, + '--spec-id', + String(input.specId), + ], + cwd: repositoryRoot(), + }; +} + export async function prepareBrunchExecutionCell(input: { readonly cellRoot: string; readonly workspaceDir: string; @@ -44,18 +62,10 @@ export async function prepareBrunchExecutionCell(input: { }); return { prepared, - launch: { - command: 'npx', - args: [ - 'tsx', - 'src/dev/execution-comparison-brunch.ts', - '--workspace', - input.workspaceDir, - '--spec-id', - String(prepared.specId), - ], - cwd: repositoryRoot(), - }, + launch: createBrunchExecutionLaunch({ + workspaceDir: input.workspaceDir, + specId: prepared.specId, + }), }; } diff --git a/src/dev/end-to-end-comparison/claude-adapter.ts b/src/dev/end-to-end-comparison/claude-adapter.ts index 4b091064e..db4dc8a20 100644 --- a/src/dev/end-to-end-comparison/claude-adapter.ts +++ b/src/dev/end-to-end-comparison/claude-adapter.ts @@ -4,6 +4,7 @@ import { join } from 'node:path'; import { runCommand, type CommandResult, type CommandRunner } from '../../app/command-runner.js'; import type { ExecutionLaunch } from './brunch-adapter.js'; import { materializeExactExecutionPacket } from './public-packet.js'; +import type { ClaudeSolutionIsolationPolicy } from './solution-isolation.js'; import { assertControllerIsolation } from './study-contract.js'; import { containedPath } from './validation.js'; @@ -20,6 +21,13 @@ export interface PreparedClaudeExecutionWorkspace { readonly launch: ExecutionLaunch; } +export interface ClaudeLaneReadyExecutionTarget { + readonly lane: 'claude_code'; + readonly targetDir: string; + readonly baseSha: string; + readonly launch: ExecutionLaunch; +} + export interface ClaudeExecutionRun { readonly startedAt: string; readonly endedAt: string; @@ -38,6 +46,87 @@ export interface ClaudeExecutionRun { }; } +export function createClaudeExecutionLaunch(input: { + readonly workspaceDir: string; + readonly isolationPolicy: ClaudeSolutionIsolationPolicy; +}): ExecutionLaunch { + return { + command: 'claude', + args: [ + '--print', + '--verbose', + '--output-format', + 'stream-json', + '--model', + 'claude-opus-4-8', + '--effort', + 'max', + '--permission-mode', + input.isolationPolicy.permissionMode, + '--no-session-persistence', + '--disable-slash-commands', + '--no-chrome', + '--strict-mcp-config', + '--mcp-config', + '{"mcpServers":{}}', + '--setting-sources', + '', + '--tools', + input.isolationPolicy.allowedTools.join(','), + '--allowedTools', + input.isolationPolicy.allowedTools.join(','), + '--disallowedTools', + 'WebFetch,WebSearch', + '--settings', + JSON.stringify({ + enabledPlugins: {}, + permissions: { + allow: input.isolationPolicy.allowedTools, + deny: ['WebFetch', 'WebSearch'], + }, + sandbox: { + enabled: input.isolationPolicy.nativeSandbox.enabled, + failIfUnavailable: input.isolationPolicy.nativeSandbox.failIfUnavailable, + autoAllowBashIfSandboxed: true, + allowUnsandboxedCommands: false, + filesystem: { + denyRead: input.isolationPolicy.nativeSandbox.deniedReadRoots, + }, + network: { + allowedDomains: input.isolationPolicy.nativeSandbox.allowedDomains, + deniedDomains: input.isolationPolicy.nativeSandbox.deniedDomains, + }, + }, + }), + implementationPrompt(), + ], + cwd: input.workspaceDir, + }; +} + +function createGreenfieldClaudeExecutionLaunch(workspaceDir: string): ExecutionLaunch { + return { + command: 'claude', + args: [ + '--print', + '--verbose', + '--output-format', + 'stream-json', + '--model', + 'claude-opus-4-8', + '--effort', + 'max', + '--permission-mode', + 'bypassPermissions', + '--no-session-persistence', + '--disable-slash-commands', + '--no-chrome', + implementationPrompt(), + ], + cwd: workspaceDir, + }; +} + export async function prepareClaudeExecutionWorkspace( input: { readonly workspaceDir: string; @@ -71,32 +160,13 @@ export async function prepareClaudeExecutionWorkspace( return { workspaceDir: input.workspaceDir, baseSha, - launch: { - command: 'claude', - args: [ - '--print', - '--verbose', - '--output-format', - 'stream-json', - '--model', - 'claude-opus-4-8', - '--effort', - 'max', - '--permission-mode', - 'bypassPermissions', - '--no-session-persistence', - '--disable-slash-commands', - '--no-chrome', - implementationPrompt(), - ], - cwd: input.workspaceDir, - }, + launch: createGreenfieldClaudeExecutionLaunch(input.workspaceDir), }; } export async function runClaudeExecutionWorkspace( input: { - readonly prepared: PreparedClaudeExecutionWorkspace; + readonly prepared: PreparedClaudeExecutionWorkspace | ClaudeLaneReadyExecutionTarget; readonly evidenceDir: string; readonly elapsedMinutes: number; }, @@ -106,10 +176,11 @@ export async function runClaudeExecutionWorkspace( throw new Error('Claude execution budget must be a positive whole number of minutes'); } await mkdir(input.evidenceDir); + const workspaceDir = 'targetDir' in input.prepared ? input.prepared.targetDir : input.prepared.workspaceDir; const startedAt = new Date().toISOString(); // ceiling: retain at most 10 MiB per provider stream; raise or stream to disk if real runs exceed it. const result = await runner(input.prepared.launch.command, input.prepared.launch.args, { - cwd: input.prepared.workspaceDir, + cwd: workspaceDir, timeoutMs: input.elapsedMinutes * 60_000, maxOutputBytes: 10 * 1024 * 1024, }); @@ -121,7 +192,7 @@ export async function runClaudeExecutionWorkspace( let repository: ClaudeExecutionRun['repository']; try { repository = await finalizeClaudeExecutionWorkspace( - { workspaceDir: input.prepared.workspaceDir }, + { workspaceDir, baseSha: input.prepared.baseSha }, runner, ); } catch { @@ -141,6 +212,7 @@ export async function runClaudeExecutionWorkspace( export async function finalizeClaudeExecutionWorkspace( input: { readonly workspaceDir: string; + readonly baseSha?: string; }, runner: CommandRunner = runCommand, ): Promise<{ @@ -148,11 +220,17 @@ export async function finalizeClaudeExecutionWorkspace( readonly reviewSha: string; readonly finalGitRange: string; }> { - const baseSha = ( - await gitChecked(runner, input.workspaceDir, ['rev-list', '--max-parents=0', 'HEAD']) - ).stdout - .trim() - .split('\n')[0]!; + const baseSha = + input.baseSha ?? + (await gitChecked(runner, input.workspaceDir, ['rev-list', '--max-parents=0', 'HEAD'])).stdout + .trim() + .split('\n')[0]!; + const resolvedBase = ( + await gitChecked(runner, input.workspaceDir, ['rev-parse', '--verify', `${baseSha}^{commit}`]) + ).stdout.trim(); + if (resolvedBase !== baseSha) { + throw new Error('Claude execution base does not match the lane-ready descriptor'); + } const status = await gitChecked(runner, input.workspaceDir, ['status', '--porcelain']); if (status.stdout.trim().length > 0) { await gitChecked(runner, input.workspaceDir, ['add', '--all']); @@ -192,7 +270,7 @@ function implementationPrompt(): string { 'Treat public-contract.json as the only additional delivery and interoperability baseline.', 'Work only inside the current Git repository. Do not inspect parent directories or external project files.', 'Do not reinterpret, normalize, or repair the specification.', - 'Run npm test and npm run build. Leave the complete implementation in the working tree when both pass.', + 'Follow the delivery, acceptance, and terminal rules in public-contract.json. Leave the complete implementation in the working tree when finished.', 'Do not create or use browser-oracle files; those are controller-owned and unavailable.', ].join(' '); } diff --git a/src/dev/end-to-end-comparison/public-packet.ts b/src/dev/end-to-end-comparison/public-packet.ts index e68ae141e..d59f1e738 100644 --- a/src/dev/end-to-end-comparison/public-packet.ts +++ b/src/dev/end-to-end-comparison/public-packet.ts @@ -17,16 +17,20 @@ export async function materializeExactExecutionPacket(input: { readonly packet: PublicCasePacket; }> { const specification = await readFile(input.specificationPath); - const template = parsePublicCaseContract( - parseJson(await readFile(input.publicContractTemplatePath, 'utf8')), - ); + const templateBytes = await readFile(input.publicContractTemplatePath); + const template = parsePublicCaseContract(parseJson(templateBytes.toString('utf8'))); + const specificationSha256 = createHash('sha256').update(specification).digest('hex'); const contract = parsePublicCaseContract({ ...template, case: { ...template.case, - specificationSha256: createHash('sha256').update(specification).digest('hex'), + specificationSha256, }, }); + const contractBytes = + template.case.specificationSha256 === specificationSha256 + ? templateBytes + : Buffer.from(`${JSON.stringify(contract, null, 2)}\n`); await mkdir(dirname(input.packetDir), { recursive: true }); try { @@ -41,10 +45,7 @@ export async function materializeExactExecutionPacket(input: { await writeFile(join(input.packetDir, contract.case.specification), specification, { flag: 'wx', }); - await writeFile(join(input.packetDir, 'public-contract.json'), `${JSON.stringify(contract, null, 2)}\n`, { - encoding: 'utf8', - flag: 'wx', - }); + await writeFile(join(input.packetDir, 'public-contract.json'), contractBytes, { flag: 'wx' }); const packet = await loadPublicCasePacket(input.packetDir); return { packetDir: input.packetDir, packet }; } catch (error) { diff --git a/src/dev/end-to-end-comparison/solution-isolation.ts b/src/dev/end-to-end-comparison/solution-isolation.ts new file mode 100644 index 000000000..743b639d0 --- /dev/null +++ b/src/dev/end-to-end-comparison/solution-isolation.ts @@ -0,0 +1,211 @@ +import { createHash } from 'node:crypto'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; + +import { runCommand, type CommandResult, type CommandRunner } from '../../app/command-runner.js'; + +const RECIPE_VERSION = 1 as const; +const SOURCE_IDENTITY_FILE = '.comparison-source.json'; +const COMPARISON_GIT_IDENTITY = [ + '-c', + 'user.name=Brunch Comparison', + '-c', + 'user.email=brunch-comparison@invalid.local', +] as const; + +export interface MaterializedPinnedSourceTree { + readonly recipeVersion: typeof RECIPE_VERSION; + readonly targetDir: string; + readonly sourceCommit: string; + readonly sourceTree: string; + readonly syntheticCommit: string; +} + +export interface MaterializedHistoricalReplayPrefix extends MaterializedPinnedSourceTree { + readonly baseSha: string; + readonly packetFiles: readonly { + readonly path: 'public-contract.json' | 'spec.md'; + readonly sha256: string; + }[]; +} + +export interface ClaudeSolutionIsolationPolicy { + readonly executor: 'claude_code'; + readonly recipeVersion: typeof RECIPE_VERSION; + readonly targetRoot: string; + readonly strictMcp: true; + readonly mcpServers: readonly []; + readonly webTools: false; + readonly ambientSettings: false; + readonly ambientPlugins: false; + readonly permissionMode: 'dontAsk'; + readonly nativeSandbox: { + readonly enabled: true; + readonly failIfUnavailable: true; + readonly allowedDomains: readonly []; + readonly deniedDomains: readonly []; + readonly deniedReadRoots: readonly string[]; + }; + readonly allowedTools: readonly ['Bash', 'Edit', 'Glob', 'Grep', 'Read', 'Write']; +} + +export interface BrunchSolutionIsolationPolicy { + readonly executor: 'brunch'; + readonly recipeVersion: typeof RECIPE_VERSION; + readonly targetRoot: string; + readonly foregroundWebTools: false; + readonly specifySubagents: false; + readonly foregroundFileRoot: string; + readonly executionSubagents: readonly ['planner', 'worker']; +} + +export type SolutionIsolationPolicy = ClaudeSolutionIsolationPolicy | BrunchSolutionIsolationPolicy; + +export async function materializePinnedSourceTree(input: { + readonly sourceRepositoryDir: string; + readonly sourceCommit: string; + readonly targetDir: string; + readonly runner?: CommandRunner; +}): Promise { + const runner = input.runner ?? runCommand; + const sourceCommit = ( + await gitChecked(runner, input.sourceRepositoryDir, [ + 'rev-parse', + '--verify', + `${input.sourceCommit}^{commit}`, + ]) + ).stdout.trim(); + const sourceTree = ( + await gitChecked(runner, input.sourceRepositoryDir, ['rev-parse', '--verify', `${sourceCommit}^{tree}`]) + ).stdout.trim(); + const archiveDir = await mkdtemp(join(tmpdir(), 'brunch-pinned-source-')); + const archivePath = join(archiveDir, 'source.tar'); + let targetCreated = false; + try { + await mkdir(dirname(input.targetDir), { recursive: true }); + await mkdir(input.targetDir); + targetCreated = true; + await commandChecked(runner, input.sourceRepositoryDir, 'git', [ + 'archive', + '--format=tar', + `--output=${archivePath}`, + sourceCommit, + ]); + await commandChecked(runner, input.targetDir, 'tar', ['-xf', archivePath]); + await writeFile( + join(input.targetDir, SOURCE_IDENTITY_FILE), + `${JSON.stringify({ recipeVersion: RECIPE_VERSION, sourceCommit, sourceTree }, null, 2)}\n`, + { encoding: 'utf8', flag: 'wx' }, + ); + await gitChecked(runner, input.targetDir, ['init', '--initial-branch=main']); + await gitChecked(runner, input.targetDir, ['add', '--all']); + await gitChecked(runner, input.targetDir, [ + ...COMPARISON_GIT_IDENTITY, + 'commit', + '-m', + 'Materialize pinned comparison source', + ]); + const syntheticCommit = (await gitChecked(runner, input.targetDir, ['rev-parse', 'HEAD'])).stdout.trim(); + return { + recipeVersion: RECIPE_VERSION, + targetDir: input.targetDir, + sourceCommit, + sourceTree, + syntheticCommit, + }; + } catch (error) { + if (targetCreated) await rm(input.targetDir, { recursive: true, force: true }); + throw error; + } finally { + await rm(archiveDir, { recursive: true, force: true }); + } +} + +export function createClaudeSolutionIsolationPolicy( + targetRoot: string, + forbiddenRoots: readonly string[] = [], +): ClaudeSolutionIsolationPolicy { + return { + executor: 'claude_code', + recipeVersion: RECIPE_VERSION, + targetRoot: resolve(targetRoot), + strictMcp: true, + mcpServers: [], + webTools: false, + ambientSettings: false, + ambientPlugins: false, + permissionMode: 'dontAsk', + nativeSandbox: { + enabled: true, + failIfUnavailable: true, + allowedDomains: [], + deniedDomains: [], + deniedReadRoots: [...new Set(forbiddenRoots.map((root) => resolve(root)))], + }, + allowedTools: ['Bash', 'Edit', 'Glob', 'Grep', 'Read', 'Write'], + }; +} + +export function createBrunchSolutionIsolationPolicy(targetRoot: string): BrunchSolutionIsolationPolicy { + const root = resolve(targetRoot); + return { + executor: 'brunch', + recipeVersion: RECIPE_VERSION, + targetRoot: root, + foregroundWebTools: false, + specifySubagents: false, + foregroundFileRoot: root, + executionSubagents: ['planner', 'worker'], + }; +} + +export async function verifyPreparedHistoricalReplay(input: { + readonly prefix: MaterializedHistoricalReplayPrefix; + readonly runner?: CommandRunner; +}): Promise { + const runner = input.runner ?? runCommand; + const identity = JSON.parse( + await readFile(join(input.prefix.targetDir, SOURCE_IDENTITY_FILE), 'utf8'), + ) as Partial; + if ( + identity.recipeVersion !== RECIPE_VERSION || + identity.sourceCommit !== input.prefix.sourceCommit || + identity.sourceTree !== input.prefix.sourceTree + ) { + throw new Error('materialized source identity does not match the frozen case'); + } + const remotes = (await gitChecked(runner, input.prefix.targetDir, ['remote'])).stdout.trim(); + if (remotes !== '') throw new Error('historical replay target must not have Git remotes'); + const trackedStatus = ( + await gitChecked(runner, input.prefix.targetDir, ['status', '--porcelain', '--untracked-files=no']) + ).stdout.trim(); + if (trackedStatus !== '') throw new Error('historical replay target has tracked source changes'); + for (const file of input.prefix.packetFiles) { + const digest = `sha256:${createHash('sha256') + .update(await readFile(join(input.prefix.targetDir, file.path))) + .digest('hex')}`; + if (digest !== file.sha256) throw new Error(`historical replay packet drifted: ${file.path}`); + } +} + +async function gitChecked( + runner: CommandRunner, + cwd: string, + args: readonly string[], +): Promise { + return await commandChecked(runner, cwd, 'git', args); +} + +async function commandChecked( + runner: CommandRunner, + cwd: string, + command: string, + args: readonly string[], +): Promise { + const result = await runner(command, args, { cwd }); + if (result.exitCode !== 0) { + throw new Error(`${command} ${args.join(' ')} failed: ${result.stderr || result.stdout}`); + } + return result; +} diff --git a/src/dev/end-to-end-comparison/study-contract.ts b/src/dev/end-to-end-comparison/study-contract.ts index e61719f9e..9ecedff0d 100644 --- a/src/dev/end-to-end-comparison/study-contract.ts +++ b/src/dev/end-to-end-comparison/study-contract.ts @@ -38,6 +38,10 @@ export interface EndToEndStudyContract { readonly manifestPath: string; readonly manifestSha256: string; }; + readonly source?: { + readonly parentCommit: string; + readonly parentTree: string; + }; readonly budgets: { readonly elicitation: { readonly qualifyingQuestions: number; @@ -66,16 +70,38 @@ export function parseEndToEndStudyContract(value: unknown): EndToEndStudyContrac const requirementRegistry = child(value, 'requirementRegistry'); const executionContractTemplate = child(value, 'executionContractTemplate'); const oracle = child(value, 'oracle'); + const source = value['source'] === undefined ? undefined : child(value, 'source'); const budgets = child(value, 'budgets'); const elicitationBudget = child(budgets, 'elicitation'); const executionBudget = child(budgets, 'execution'); const actorRecipes = child(value, 'actorRecipes'); const executionRecipes = child(actorRecipes, 'execution'); + const knownCase = + (value['id'] === 'minimal-petri-net-editor-e2e-v1' && + value['caseId'] === 'minimal-petri-net-editor-v1' && + oracle['id'] === 'minimal-petri-net-editor-oracles-v2' && + source === undefined) || + (value['id'] === 'petrinaut-optimization-e2e-v1' && + value['caseId'] === 'petrinaut-optimization-v1' && + oracle['id'] === 'petrinaut-optimization-oracles-v1' && + source !== undefined && + source['parentCommit'] === '5c7a2d9db5caa851c38938f4b1bac19005b0e978' && + source['parentTree'] === 'a3e08cf75e00cc9016c931f4665341506e03533e' && + gitObjectId(source['parentCommit']) && + gitObjectId(source['parentTree']) && + exactKeys(source, ['parentCommit', 'parentTree'])) || + (value['id'] === 'brunch-host-landing-e2e-v1' && + value['caseId'] === 'brunch-host-landing-v1' && + oracle['id'] === 'brunch-host-landing-oracles-v1' && + source !== undefined && + gitObjectId(source['parentCommit']) && + gitObjectId(source['parentTree']) && + exactKeys(source, ['parentCommit', 'parentTree'])); + if ( value['schemaVersion'] !== 1 || - !safeId(value['id']) || - !safeId(value['caseId']) || + !knownCase || !addressedPath(mission) || !addressedPath(baseline) || !addressedPath(requirementRegistry) || @@ -163,6 +189,16 @@ function addressedPath(value: Record): boolean { return safeRelativePath(value['path']) && sha256(value['sha256']); } +function gitObjectId(value: unknown): value is string { + return typeof value === 'string' && /^[a-f0-9]{40}$/u.test(value); +} + +function exactKeys(value: Record, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const wanted = [...expected].sort(); + return actual.length === wanted.length && actual.every((key, index) => key === wanted[index]); +} + function child(value: Record, key: string): Record { const selected = value[key]; if (!record(selected)) invalid(); diff --git a/src/dev/execution-comparison-brunch.ts b/src/dev/execution-comparison-brunch.ts index 9de34cef5..e43ea657b 100644 --- a/src/dev/execution-comparison-brunch.ts +++ b/src/dev/execution-comparison-brunch.ts @@ -13,6 +13,7 @@ import { runWithScopedBrunchOfflineDefault, type BrunchTuiLaunchContext, } from '../app/brunch-tui.js'; +import { openWorkspaceCommandExecutor } from '../graph/index.js'; export async function runPinnedBrunchExecutionTui(input: { readonly workspaceDir: string; @@ -21,26 +22,53 @@ export async function runPinnedBrunchExecutionTui(input: { readonly model: 'claude-opus-4-8'; }): Promise { const model = getBuiltinModel(input.provider, input.model); + const preflight = await resolvePinnedBrunchPreflight({ + workspaceDir: input.workspaceDir, + specId: input.specId, + }); await runBrunchTui({ cwd: input.workspaceDir, openWeb: false, webSidecarRunner: async () => null, - runWorkspaceDialogPreflight: async () => ({ - action: 'newSession', - specId: input.specId, - establish: { origin: 'greenfield' }, - }), + runWorkspaceDialogPreflight: async () => preflight, launchInteractive: async (context) => { await launchPinnedInteractive(context, model); }, }); } +export async function resolvePinnedBrunchPreflight(input: { + readonly workspaceDir: string; + readonly specId: number; +}): Promise< + | { readonly action: 'newSession'; readonly specId: number } + | { + readonly action: 'newSession'; + readonly specId: number; + readonly establish: { readonly origin: 'greenfield' }; + } +> { + const executor = await openWorkspaceCommandExecutor(input.workspaceDir); + const spec = executor.getSpec(input.specId); + if (spec === undefined) throw new Error(`prepared Brunch specification ${input.specId} is missing`); + return spec.origin === null + ? { + action: 'newSession', + specId: input.specId, + establish: { origin: 'greenfield' }, + } + : { action: 'newSession', specId: input.specId }; +} + async function launchPinnedInteractive(context: BrunchTuiLaunchContext, model: Model): Promise { const agentDir = getAgentDir(); const createRuntime = createBrunchAgentSessionRuntimeFactory({ ...context, + allowSubagents: false, + comparisonIsolation: { + targetRoot: context.workspace.cwd, + }, agentServices: { model }, }); const runtime = await createAgentSessionRuntime(createRuntime, { @@ -79,7 +107,8 @@ export function parseExecutionComparisonArgs(args: readonly string[]): { async function main(): Promise { const args = parseExecutionComparisonArgs(process.argv.slice(2)); await runPinnedBrunchExecutionTui({ - ...args, + workspaceDir: args.workspaceDir, + specId: args.specId, provider: 'anthropic', model: 'claude-opus-4-8', }); diff --git a/src/dev/execution-comparison-operator.ts b/src/dev/execution-comparison-operator.ts index 45e7c31d9..f60f643b0 100644 --- a/src/dev/execution-comparison-operator.ts +++ b/src/dev/execution-comparison-operator.ts @@ -1,5 +1,5 @@ import { mkdir, readFile, writeFile } from 'node:fs/promises'; -import { dirname, join, resolve } from 'node:path'; +import { dirname, isAbsolute, join, resolve } from 'node:path'; import process from 'node:process'; import { fileURLToPath } from 'node:url'; @@ -7,25 +7,138 @@ import { parseExecutionAttempt, writeExecutionAttemptImmutable, } from './execution-comparison/artifact-contract.js'; -import { runPetriEditorBrowserOracle } from './execution-comparison/browser-oracle.js'; +import { + runPetriEditorBrowserOracle, + type BrowserOracleReport, +} from './execution-comparison/browser-oracle.js'; +import { + runBrunchHostLandingOracle, + type HostLandingOracleReport, +} from './execution-comparison/host-landing-oracle.js'; import { listExecutionCases, prepareExecutionTarget, resolveExecutionCase, } from './execution-comparison/operator-cli.js'; -import { loadControllerOraclePack } from './execution-comparison/oracle-pack.js'; +import { + loadControllerOracleManifest, + loadControllerOraclePack, +} from './execution-comparison/oracle-pack.js'; +import { + runPetrinautOptimizationOracle, + type PetrinautOptimizationOracleReport, +} from './execution-comparison/petrinaut-optimization-oracle.js'; + +type CompiledOracleId = + | 'minimal-petri-net-editor-oracles-v2' + | 'brunch-host-landing-oracles-v1' + | 'petrinaut-optimization-oracles-v1'; + +interface CompiledOracle { + readonly implementationFiles: readonly string[]; + readonly run: (input: { + readonly appDir: string; + readonly caseDir: string; + }) => Promise; +} + +const COMPILED_ORACLES: Readonly> = { + 'minimal-petri-net-editor-oracles-v2': { + implementationFiles: [ + fileURLToPath(new URL('./execution-comparison/browser-oracle.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/browser-oracle/journey-runner.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/petri-reference.ts', import.meta.url)), + ], + run: runPetriEditorBrowserOracle, + }, + 'brunch-host-landing-oracles-v1': { + implementationFiles: [ + fileURLToPath(new URL('./execution-comparison/host-landing-oracle.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/host-landing-oracle/types.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/host-landing-oracle/git-model.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/host-landing-oracle/fixture.ts', import.meta.url)), + fileURLToPath(new URL('./execution-comparison/host-landing-oracle/runner.ts', import.meta.url)), + fileURLToPath(new URL('./tui-driver.ts', import.meta.url)), + fileURLToPath(new URL('./tui-driver/session.ts', import.meta.url)), + fileURLToPath(new URL('./tui-driver/screen.ts', import.meta.url)), + ], + run: async ({ appDir }) => await runBrunchHostLandingOracle({ candidateRoot: appDir }), + }, + 'petrinaut-optimization-oracles-v1': { + implementationFiles: [ + fileURLToPath(new URL('./execution-comparison/petrinaut-optimization-oracle.ts', import.meta.url)), + fileURLToPath( + new URL('./execution-comparison/petrinaut-optimization-oracle/types.ts', import.meta.url), + ), + fileURLToPath( + new URL('./execution-comparison/petrinaut-optimization-oracle/runner.ts', import.meta.url), + ), + fileURLToPath( + new URL('./execution-comparison/petrinaut-optimization-oracle/browser.ts', import.meta.url), + ), + fileURLToPath( + new URL( + './execution-comparison/petrinaut-optimization-oracle/calibration-seed.json', + import.meta.url, + ), + ), + fileURLToPath( + new URL('./execution-comparison/petrinaut-optimization-oracle/claims.ts', import.meta.url), + ), + fileURLToPath( + new URL('./execution-comparison/petrinaut-optimization-oracle/fake-optimizer.ts', import.meta.url), + ), + ], + run: async ({ appDir, caseDir }) => + await runPetrinautOptimizationOracle({ candidateRoot: appDir, caseDir }), + }, +}; + +export function resolveCompiledExecutionOracle(id: string): CompiledOracle { + if ( + id !== 'minimal-petri-net-editor-oracles-v2' && + id !== 'brunch-host-landing-oracles-v1' && + id !== 'petrinaut-optimization-oracles-v1' + ) { + throw new Error(`unknown compiled execution oracle id: ${id}`); + } + return COMPILED_ORACLES[id]; +} + +export async function retainCompiledOracleReport(input: { + readonly out: string; + readonly oracleId: CompiledOracleId; + readonly oraclePackSha256: string; + readonly report: BrowserOracleReport | HostLandingOracleReport | PetrinautOptimizationOracleReport; +}): Promise<{ + readonly out: string; + readonly status: 'passed' | 'failed' | 'assertion_failed' | 'setup_failed'; + readonly oraclePackSha256: string; + readonly oracleId: CompiledOracleId; +}> { + const out = resolve(input.out); + await mkdir(dirname(out), { recursive: true }); + await writeFile(out, `${JSON.stringify(input.report, null, 2)}\n`, { encoding: 'utf8', flag: 'wx' }); + return { + out, + status: input.report.status, + oraclePackSha256: input.oraclePackSha256, + oracleId: input.oracleId, + }; +} export const EXECUTION_COMPARISON_SHARED_FRAMING = [ 'Implement the frozen specification and public contract supplied in this isolated target.', 'Treat their bytes as immutable input: do not normalize, repair, or replace either file.', 'Work only in the target repository and do not inspect controller paths or seek hidden comparison material.', - 'Deliver the static browser application in dist/ and run npm test followed by npm run build.', - 'Stop after those commands and report the visible result; do not add a backend or runtime network dependency.', + 'Follow the case-specific delivery, acceptance, network, and terminal rules exactly.', + 'Stop at the contract-declared execution terminal and report only target-visible results.', ].join('\n'); const DEFAULT_CASES_ROOT = fileURLToPath( new URL('../../testing/execution-comparisons/cases/', import.meta.url), ); +const DEFAULT_CONTROLLER_ROOT = fileURLToPath(new URL('../../', import.meta.url)); export async function runExecutionComparisonOperatorCli(args: readonly string[]): Promise { const [command, ...rest] = args; @@ -41,6 +154,8 @@ export async function runExecutionComparisonOperatorCli(args: readonly string[]) case 'inspect': { assertOnlyOptions(options, ['case']); const selected = await resolveExecutionCase(required(options, 'case'), casesRoot); + const manifest = await loadControllerOracleManifest(selected.caseDir); + const repository = selected.packet.contract.case.repository; process.stdout.write( `${JSON.stringify( { @@ -48,6 +163,9 @@ export async function runExecutionComparisonOperatorCli(args: readonly string[]) caseId: selected.caseId, caseDir: selected.caseDir, publicPacketSha256: selected.packet.packetSha256, + oracleId: manifest.id, + repository, + requiresSourceRepository: repository.substrate === 'pinned_git', files: selected.packet.files, sharedFraming: EXECUTION_COMPARISON_SHARED_FRAMING, specification: await readFile(join(selected.caseDir, 'spec.md'), 'utf8'), @@ -60,46 +178,46 @@ export async function runExecutionComparisonOperatorCli(args: readonly string[]) return; } case 'prepare': { - assertOnlyOptions(options, ['case', 'lane', 'target']); + assertOnlyOptions(options, ['case', 'lane', 'target', 'source-repository']); const lane = required(options, 'lane'); if (lane !== 'brunch' && lane !== 'claude_code') { throw new Error('--lane must be brunch or claude_code'); } + const sourceRepository = options.has('source-repository') + ? requiredAbsolute(options, 'source-repository') + : undefined; const prepared = await prepareExecutionTarget({ lane, caseReference: required(options, 'case'), casesRoot, targetDir: resolve(required(options, 'target')), + controllerRoot: DEFAULT_CONTROLLER_ROOT, + ...(sourceRepository === undefined ? {} : { sourceRepositoryDir: sourceRepository }), }); process.stdout.write(`${JSON.stringify(prepared, null, 2)}\n`); return; } case 'oracle': { assertOnlyOptions(options, ['case', 'app', 'out']); + const out = requiredAbsolute(options, 'out'); const selected = await resolveExecutionCase(required(options, 'case'), casesRoot); + const manifest = await loadControllerOracleManifest(selected.caseDir); + const oracle = resolveCompiledExecutionOracle(manifest.id); const oraclePack = await loadControllerOraclePack({ caseDir: selected.caseDir, - implementationFiles: [ - fileURLToPath(new URL('./execution-comparison/browser-oracle.ts', import.meta.url)), - fileURLToPath(new URL('./execution-comparison/browser-oracle/journey-runner.ts', import.meta.url)), - fileURLToPath(new URL('./execution-comparison/petri-reference.ts', import.meta.url)), - ], + implementationFiles: oracle.implementationFiles, }); - const report = await runPetriEditorBrowserOracle({ + const report = await oracle.run({ appDir: resolve(required(options, 'app')), caseDir: selected.caseDir, }); - const out = resolve(required(options, 'out')); - await mkdir(dirname(out), { recursive: true }); - await writeFile(out, `${JSON.stringify(report, null, 2)}\n`, { encoding: 'utf8', flag: 'wx' }); - process.stdout.write( - `${JSON.stringify({ - out, - status: report.status, - oraclePackSha256: oraclePack.packSha256, - browserSuiteVersion: oraclePack.manifest.browserSuiteVersion, - })}\n`, - ); + const retained = await retainCompiledOracleReport({ + out, + oracleId: oraclePack.manifest.id, + oraclePackSha256: oraclePack.packSha256, + report, + }); + process.stdout.write(`${JSON.stringify(retained)}\n`); return; } case 'retain-attempt': { @@ -146,6 +264,14 @@ function required(options: ReadonlyMap, name: string): string { return value; } +function requiredAbsolute(options: ReadonlyMap, name: string): string { + const value = required(options, name); + if (!isAbsolute(value)) { + throw new Error(`--${name} must be an absolute path`); + } + return resolve(value); +} + if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { runExecutionComparisonOperatorCli(process.argv.slice(2)).catch((error: unknown) => { process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`); diff --git a/src/dev/execution-comparison/__tests__/accessibility-contract.test.ts b/src/dev/execution-comparison/__tests__/accessibility-contract.test.ts index fe8594669..2d63ea2f0 100644 --- a/src/dev/execution-comparison/__tests__/accessibility-contract.test.ts +++ b/src/dev/execution-comparison/__tests__/accessibility-contract.test.ts @@ -3,7 +3,7 @@ import { describe, expect, it } from 'vitest'; import { assertAccessibilityContract } from '../accessibility-contract.js'; -import type { ExecutionCasePublicContract } from '../case-contract.js'; +import type { BrowserExecutionCasePublicContract } from '../case-contract.js'; const accessibility = { application: { role: 'application', name: 'Petri net editor' }, @@ -31,7 +31,7 @@ const accessibility = { { role: 'spinbutton', name: 'Arc weight' }, ], feedbackRoles: ['status', 'alert'], -} satisfies ExecutionCasePublicContract['accessibility']; +} satisfies BrowserExecutionCasePublicContract['accessibility']; function buttons(): string { return accessibility.controls.map(({ name }) => ``).join(''); diff --git a/src/dev/execution-comparison/__tests__/brunch-lane.test.ts b/src/dev/execution-comparison/__tests__/brunch-lane.test.ts index d551eecc0..5b828f3eb 100644 --- a/src/dev/execution-comparison/__tests__/brunch-lane.test.ts +++ b/src/dev/execution-comparison/__tests__/brunch-lane.test.ts @@ -15,18 +15,23 @@ import { queryGraph } from '../../../graph/queries.js'; import { seedFixture } from '../../../graph/seed-fixtures.js'; import { buildBrunchExecutionSeed, + buildOpaqueBrownfieldExecutionSeed, buildOpaqueBrunchExecutionSeed, prepareBrunchExecutionWorkspace, } from '../brunch-lane.js'; -import { loadPublicCasePacket } from '../case-contract.js'; +import { isBrowserExecutionCaseContract, loadPublicCasePacket } from '../case-contract.js'; const caseDir = fileURLToPath( new URL('../../../../testing/execution-comparisons/cases/minimal-petri-net-editor/', import.meta.url), ); +const petrinautCaseDir = fileURLToPath( + new URL('../../../../testing/execution-comparisons/cases/petrinaut-optimization/', import.meta.url), +); describe('Brunch execution comparison lane adapter', () => { it('projects the frozen public packet into one complete greenfield execution scope', async () => { const packet = await loadPublicCasePacket(caseDir); + if (!isBrowserExecutionCaseContract(packet.contract)) throw new Error('expected browser case'); const specification = await readFile(`${caseDir}/spec.md`, 'utf8'); const fixture = buildBrunchExecutionSeed({ specification, contract: packet.contract }); expect(fixture.nodes.find((node) => node.source === 'approved-spec [D1]')).toMatchObject({ @@ -98,6 +103,7 @@ describe('Brunch execution comparison lane adapter', () => { it('preserves an arbitrary target-authored specification as one exact settled requirement', async () => { const packet = await loadPublicCasePacket(caseDir); + if (!isBrowserExecutionCaseContract(packet.contract)) throw new Error('expected browser case'); const specification = '# Target-authored specification\n\nSpacing stays exact. \n'; const fixture = buildOpaqueBrunchExecutionSeed({ specification, @@ -130,6 +136,32 @@ describe('Brunch execution comparison lane adapter', () => { expect(() => assertExecuteProjectionPlanReady(projection)).not.toThrow(); }); + it('projects an opaque brownfield specification without Petri-specific execution wording', async () => { + const packet = await loadPublicCasePacket(petrinautCaseDir); + const specification = await readFile(join(petrinautCaseDir, 'spec.md'), 'utf8'); + const fixture = buildOpaqueBrownfieldExecutionSeed({ + specification, + contract: packet.contract, + }); + expect(fixture.nodes.find((node) => node.source === 'e2e-handoff [exact-spec]')).toMatchObject({ + kind: 'requirement', + body: specification, + }); + expect(JSON.stringify(fixture)).not.toMatch(/Petri-net|static browser|npm install/iu); + + const db = createDb(':memory:'); + const seeded = seedFixture(new CommandExecutor(db), fixture); + const graph = queryGraph(db, seeded.specId); + const projection = projectExecuteGraph({ + specId: seeded.specId, + graphLsn: graph.lsn, + mode: 'brownfield', + nodes: graph.nodes, + edges: graph.edges, + }); + expect(() => assertExecuteProjectionPlanReady(projection)).not.toThrow(); + }); + it('prepares a fresh Brunch workspace with only the content-addressed public packet', async () => { const workspaceDir = await mkdtemp(join(tmpdir(), 'brunch-execution-lane-')); const prepared = await prepareBrunchExecutionWorkspace({ workspaceDir, caseDir }); diff --git a/src/dev/execution-comparison/__tests__/case-contract.test.ts b/src/dev/execution-comparison/__tests__/case-contract.test.ts index 7cd2e78ba..f49809310 100644 --- a/src/dev/execution-comparison/__tests__/case-contract.test.ts +++ b/src/dev/execution-comparison/__tests__/case-contract.test.ts @@ -5,14 +5,137 @@ import { fileURLToPath } from 'node:url'; import { describe, expect, it } from 'vitest'; -import { loadPublicCasePacket } from '../case-contract.js'; -import { loadControllerOraclePack } from '../oracle-pack.js'; +import { loadPublicCasePacket, parsePublicCaseContract } from '../case-contract.js'; +import { isPetriControllerOracleManifest, loadControllerOraclePack } from '../oracle-pack.js'; const caseDir = fileURLToPath( new URL('../../../../testing/execution-comparisons/cases/minimal-petri-net-editor/', import.meta.url), ); describe('execution comparison public case contract', () => { + it('accepts only the frozen greenfield and two exact brownfield profile variants', () => { + const brunchContract = { + schemaVersion: 1, + case: { + id: 'brunch-host-landing-v1', + specification: 'spec.md', + specificationSha256: 'a'.repeat(64), + provider: 'anthropic', + model: 'claude-opus-4-8', + product: 'brunch', + mode: 'brownfield', + scope: 'single_feature', + surface: 'backend', + repository: { + substrate: 'pinned_git', + parentCommit: '1'.repeat(40), + parentTree: '2'.repeat(40), + }, + }, + budgets: { + elapsedMinutes: 90, + mechanicalInterventions: 2, + substantiveHumanInterventions: 0, + }, + delivery: { + runtimeNetwork: 'forbidden', + dependencyInstallNetwork: 'forbidden', + }, + acceptance: { + publicCommand: '/brunch:land', + executionTerminal: 'promotion_prepared', + }, + rules: ['Work only in the target repository.'], + }; + + expect(parsePublicCaseContract(brunchContract)).toEqual(brunchContract); + for (const mutation of [ + { accessibility: { application: { role: 'application', name: 'Browser-only leak' } } }, + { case: { ...brunchContract.case, mode: 'greenfield' } }, + { case: { ...brunchContract.case, repository: { substrate: 'empty_dir' } } }, + { delivery: { ...brunchContract.delivery, test: { command: 'sh', args: ['oracle.sh'] } } }, + ]) { + expect(() => parsePublicCaseContract({ ...brunchContract, ...mutation })).toThrow( + 'invalid fixed public execution contract', + ); + } + + const petrinautContract = { + ...brunchContract, + case: { + ...brunchContract.case, + id: 'petrinaut-optimization-v1', + product: 'petrinaut', + surface: 'frontend', + repository: { + substrate: 'pinned_git', + parentCommit: '5c7a2d9db5caa851c38938f4b1bac19005b0e978', + parentTree: 'a3e08cf75e00cc9016c931f4665341506e03533e', + }, + }, + delivery: { + runtimeNetwork: 'forbidden', + dependencyInstallNetwork: 'controller_only', + }, + acceptance: { + publicRoute: '/optimization', + sameOriginApi: '/api/petrinaut-opt/optimize/all', + executionTerminal: 'promotion_prepared', + }, + mechanicalAddresses: { + skipTour: { kind: 'roleName', role: 'button', name: 'Skip tour' }, + dismissAssistant: { kind: 'roleName', role: 'button', name: 'Dismiss' }, + simulateMode: { kind: 'roleName', role: 'radio', name: 'Simulate' }, + optimizationsNav: { kind: 'roleValue', role: 'radio', value: 'optimizations' }, + viewTitle: { kind: 'exactText', text: 'Optimizations' }, + create: { kind: 'roleName', role: 'button', name: 'Create' }, + createDrawer: { kind: 'roleName', role: 'dialog', name: 'Create an optimization' }, + scenario: { kind: 'roleContents', role: 'combobox', contents: 'Select a scenario' }, + scenarioSelected: { kind: 'roleContents', role: 'combobox', contents: 'Seasonal Flu' }, + metric: { kind: 'roleContents', role: 'combobox', contents: 'Select a metric' }, + metricCustomOption: { kind: 'roleName', role: 'option', name: 'Custom code' }, + metricCode: { kind: 'roleName', role: 'textbox', name: 'Editor content' }, + optimizationName: { kind: 'roleName', role: 'textbox', name: 'Name' }, + directionMaximize: { kind: 'roleName', role: 'radio', name: 'Maximize' }, + directionMinimize: { kind: 'roleName', role: 'radio', name: 'Minimize' }, + run: { kind: 'roleName', role: 'button', name: 'Run' }, + cancel: { kind: 'roleName', role: 'button', name: 'Cancel' }, + statusComplete: { kind: 'exactText', text: 'Complete' }, + statusError: { kind: 'exactText', text: 'Error' }, + statusCancelled: { kind: 'exactText', text: 'Cancelled' }, + }, + }; + expect(parsePublicCaseContract(petrinautContract)).toEqual(petrinautContract); + for (const mutation of [ + { case: { ...petrinautContract.case, product: 'brunch' } }, + { case: { ...petrinautContract.case, surface: 'backend' } }, + { + case: { + ...petrinautContract.case, + repository: { ...petrinautContract.case.repository, parentTree: '2'.repeat(40) }, + }, + }, + { acceptance: { ...petrinautContract.acceptance, publicRoute: '/processes/draft' } }, + { + mechanicalAddresses: { + ...petrinautContract.mechanicalAddresses, + create: { kind: 'roleName', role: 'button', name: 'Create optimization' }, + }, + }, + { + mechanicalAddresses: { + ...petrinautContract.mechanicalAddresses, + create: { kind: 'roleName', role: 'button', name: 'Create' }, + extra: { kind: 'roleName', role: 'button', name: 'Extra' }, + }, + }, + ]) { + expect(() => parsePublicCaseContract({ ...petrinautContract, ...mutation })).toThrow( + 'invalid fixed public execution contract', + ); + } + }); + it('freezes only the approved specification and public contract', async () => { const packet = await loadPublicCasePacket(caseDir); @@ -83,6 +206,7 @@ describe('execution comparison public case contract', () => { browserSuiteVersion: 'petri-editor-browser-v2', referenceModelVersion: 'weighted-pt-v1', }); + if (!isPetriControllerOracleManifest(pack.manifest)) throw new Error('expected Petri manifest'); expect(pack.manifest.journeys.map((journey) => journey.id)).toEqual([ 'mount', 'node-lifecycle', diff --git a/src/dev/execution-comparison/__tests__/compare-execution-prompt.test.ts b/src/dev/execution-comparison/__tests__/compare-execution-prompt.test.ts index 0b702bfcf..92fb97a56 100644 --- a/src/dev/execution-comparison/__tests__/compare-execution-prompt.test.ts +++ b/src/dev/execution-comparison/__tests__/compare-execution-prompt.test.ts @@ -51,9 +51,11 @@ describe('/compare-execution operator prompt', () => { expect(prompt).toContain('process status'); expect(prompt).toContain('final tree and complete base-to-tip diff'); expect(prompt).toContain('visible interaction evidence'); - expect(prompt).toContain('`petri-editor-browser-v2`'); + expect(prompt).toContain('oracle identity returned by `inspect`'); + expect(prompt).toContain('[--source-repository ]'); + expect(prompt).toContain('`/comparison-publish`'); expect(prompt).toContain('after every lane terminates'); - expect(prompt).toContain('Present validity before outcomes'); + expect(prompt).toContain('validity before outcomes'); expect(prompt).toContain('Do not score'); expect(prompt).toContain('diagnostic-only'); }); diff --git a/src/dev/execution-comparison/__tests__/execution-comparison-brunch.test.ts b/src/dev/execution-comparison/__tests__/execution-comparison-brunch.test.ts index a344c5611..6558dcbe7 100644 --- a/src/dev/execution-comparison/__tests__/execution-comparison-brunch.test.ts +++ b/src/dev/execution-comparison/__tests__/execution-comparison-brunch.test.ts @@ -1,7 +1,20 @@ -import { describe, expect, it } from 'vitest'; +import { mkdir, mkdtemp, rm, symlink, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; + +import { createSubagentToolCatalog } from '../../../.pi/extensions/subagents/session.js'; +import { registerBrunchOperationalModePolicy } from '../../../app/pi-extensions.js'; +import { loadBrunchSubagents } from '../../../app/pi-subagents.js'; import { parseExecutionComparisonArgs } from '../../execution-comparison-brunch.js'; +const roots: string[] = []; + +afterEach(async () => { + await Promise.all(roots.splice(0).map(async (root) => await rm(root, { recursive: true, force: true }))); +}); + describe('execution comparison Brunch CLI arguments', () => { it('parses a complete workspace and positive specification id', () => { expect(parseExecutionComparisonArgs(['--workspace', '/tmp/petri-editor', '--spec-id', '17'])).toEqual({ @@ -21,4 +34,113 @@ describe('execution comparison Brunch CLI arguments', () => { ).toThrow('Usage:'); expect(() => parseExecutionComparisonArgs(['--unknown', 'value'])).toThrow(); }); + + it('bounds every foreground filesystem tool to the comparison target, including symlinks', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-comparison-tools-')); + roots.push(root); + const targetRoot = join(root, 'target'); + const outsideRoot = join(root, 'controller'); + await mkdir(targetRoot); + await mkdir(outsideRoot); + await writeFile(join(targetRoot, 'inside.txt'), 'inside\n'); + await writeFile(join(outsideRoot, 'excluded.txt'), 'outside comparison target\n'); + await symlink(join(outsideRoot, 'excluded.txt'), join(targetRoot, 'escaped-link.txt')); + await symlink(outsideRoot, join(targetRoot, 'escaped-directory')); + + const tools: Array<{ + readonly name: string; + readonly execute: ( + toolCallId: string, + params: Record, + signal: AbortSignal | undefined, + onUpdate: undefined, + ctx: { cwd: string }, + ) => Promise; + }> = []; + registerBrunchOperationalModePolicy( + { + registerTool: (tool: unknown) => tools.push(tool as (typeof tools)[number]), + getAllTools: () => tools, + setActiveTools: () => {}, + on: () => {}, + } as never, + { filesystemRoot: targetRoot }, + ); + + const parameters = new Map>([ + ['read', { path: join(targetRoot, 'inside.txt') }], + ['grep', { pattern: 'inside', path: targetRoot }], + ['find', { pattern: '*.txt', path: targetRoot }], + ['ls', { path: targetRoot }], + ]); + expect(tools.map(({ name }) => name)).toEqual(['read', 'grep', 'find', 'ls']); + for (const tool of tools) { + await expect( + tool.execute('allowed', parameters.get(tool.name)!, undefined, undefined, { + cwd: targetRoot, + }), + ).resolves.toBeDefined(); + await expect( + tool.execute( + 'parent-escape', + { ...parameters.get(tool.name), path: outsideRoot }, + undefined, + undefined, + { cwd: targetRoot }, + ), + ).rejects.toThrow('escapes target root'); + } + await expect( + tools[0]!.execute( + 'symlink-escape', + { path: join(targetRoot, 'escaped-link.txt') }, + undefined, + undefined, + { cwd: targetRoot }, + ), + ).rejects.toThrow('escapes target root through symlink'); + + const childRead = createSubagentToolCatalog(targetRoot).get('read')!; + await expect( + childRead.execute('child-allowed', { path: 'inside.txt' }, undefined, undefined, { + cwd: targetRoot, + } as never), + ).resolves.toBeDefined(); + await expect( + childRead.execute('child-parent-escape', { path: outsideRoot }, undefined, undefined, { + cwd: targetRoot, + } as never), + ).rejects.toThrow(); + await expect( + childRead.execute('child-symlink-escape', { path: 'escaped-link.txt' }, undefined, undefined, { + cwd: targetRoot, + } as never), + ).rejects.toThrow(); + const childWrite = createSubagentToolCatalog(targetRoot).get('write_worktree_file')!; + await expect( + childWrite.execute( + 'child-write-symlink-escape', + { path: 'escaped-directory/leak.txt', content: 'must stay sealed\n' }, + undefined, + undefined, + { cwd: targetRoot } as never, + ), + ).rejects.toThrow('escapes the worktree'); + }); + + it('loads only the sealed planner and worker for comparison execution', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-comparison-subagents-')); + roots.push(root); + const subagents = await loadBrunchSubagents({ + cwd: root, + agentDir: join(root, 'agent'), + delegatableAgents: [], + includedAgents: ['planner', 'worker'], + }); + + expect([...subagents.definitions.keys()]).toEqual(['planner', 'worker']); + expect(subagents.definitions.get('planner')?.tools).toEqual(['read']); + expect(subagents.definitions.get('worker')?.tools).toEqual(['read', 'write_worktree_file']); + expect(subagents.definitions.has('researcher')).toBe(false); + }); }); diff --git a/src/dev/execution-comparison/__tests__/historical-replay-target.test.ts b/src/dev/execution-comparison/__tests__/historical-replay-target.test.ts new file mode 100644 index 000000000..6ac3cc097 --- /dev/null +++ b/src/dev/execution-comparison/__tests__/historical-replay-target.test.ts @@ -0,0 +1,170 @@ +import { createHash } from 'node:crypto'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { afterEach, describe, expect, it } from 'vitest'; + +import { runCommand } from '../../../app/command-runner.js'; +import { openWorkspaceDb } from '../../../graph/index.js'; +import { queryGraph } from '../../../graph/queries.js'; +import { prepareHistoricalReplayTarget } from '../historical-replay-target.js'; +import { prepareExecutionTarget, resolveExecutionCase } from '../operator-cli.js'; + +const roots: string[] = []; + +afterEach(async () => { + await Promise.all(roots.splice(0).map(async (root) => await rm(root, { recursive: true, force: true }))); +}); + +describe('learning-first historical replay preparation', () => { + it('creates a remote-free Brunch target from the pinned tree and exact packet', async () => { + const fixture = await createBrunchFixture(); + const selected = await resolveExecutionCase('brunch-host-landing', fixture.casesRoot); + let installCalled = false; + const ready = await prepareHistoricalReplayTarget( + { + lane: 'brunch', + selectedCase: selected, + sourceRepositoryDir: fixture.sourceDir, + targetDir: fixture.targetDir, + controllerRoot: fixture.controllerDir, + }, + { + dependencyInstallRunner: async () => { + installCalled = true; + return { exitCode: 0, stdout: '', stderr: '' }; + }, + }, + ); + if (ready.lane !== 'brunch') throw new Error('expected Brunch target'); + + expect(ready).toMatchObject({ + status: 'ready', + caseId: 'brunch-host-landing-v1', + sourceCommit: fixture.sourceCommit, + sourceTree: fixture.sourceTree, + dependencyPreparation: { recipe: 'none', status: 'not_required' }, + launch: { + command: 'npx', + args: expect.arrayContaining(['--workspace', fixture.targetDir, '--spec-id']), + }, + }); + expect(ready.launch.args).not.toContain('--solution-isolation'); + expect(await git(fixture.targetDir, ['remote'])).toBe(''); + expect(await readFile(join(fixture.targetDir, 'spec.md'))).toEqual(fixture.specification); + expect(installCalled).toBe(false); + const graph = queryGraph(await openWorkspaceDb(fixture.targetDir), ready.specId); + expect(graph.nodes.find(({ source }) => source === 'e2e-handoff [exact-spec]')).toMatchObject({ + kind: 'requirement', + body: fixture.specification.toString('utf8'), + }); + }, 30_000); + + it('returns a restricted Claude launch from the same pinned packet', async () => { + const fixture = await createBrunchFixture(); + const ready = await prepareExecutionTarget({ + lane: 'claude_code', + caseReference: 'brunch-host-landing', + casesRoot: fixture.casesRoot, + sourceRepositoryDir: fixture.sourceDir, + targetDir: fixture.targetDir, + controllerRoot: fixture.controllerDir, + }); + if (ready.lane !== 'claude_code' || ready.preparation !== 'historical_replay') { + throw new Error('expected historical Claude target'); + } + + expect(ready.launch.args).toEqual( + expect.arrayContaining(['--strict-mcp-config', '--disallowedTools', 'WebFetch,WebSearch']), + ); + expect(JSON.stringify(ready.launch)).toContain(fixture.sourceDir); + expect(await git(fixture.targetDir, ['remote'])).toBe(''); + }, 30_000); + + it('removes an owned target when pinned source identity is wrong', async () => { + const fixture = await createBrunchFixture(); + const selected = await resolveExecutionCase('brunch-host-landing', fixture.casesRoot); + Object.assign(selected.packet.contract.case.repository, { parentTree: 'f'.repeat(40) }); + + await expect( + prepareHistoricalReplayTarget({ + lane: 'claude_code', + selectedCase: selected, + sourceRepositoryDir: fixture.sourceDir, + targetDir: fixture.targetDir, + controllerRoot: fixture.controllerDir, + }), + ).rejects.toMatchObject({ status: 'setup_failed', phase: 'source_materialization' }); + await expect(readFile(join(fixture.targetDir, 'spec.md'))).rejects.toMatchObject({ code: 'ENOENT' }); + }); +}); + +async function createBrunchFixture(): Promise<{ + readonly casesRoot: string; + readonly controllerDir: string; + readonly sourceDir: string; + readonly sourceCommit: string; + readonly sourceTree: string; + readonly targetDir: string; + readonly specification: Buffer; +}> { + const root = await mkdtemp(join(tmpdir(), 'brunch-historical-replay-')); + roots.push(root); + const sourceDir = join(root, 'source'); + const controllerDir = join(root, 'controller'); + const casesRoot = join(root, 'cases'); + const caseDir = join(casesRoot, 'brunch-host-landing'); + const targetDir = join(root, 'target'); + await Promise.all([mkdir(sourceDir), mkdir(controllerDir), mkdir(caseDir, { recursive: true })]); + await writeFile(join(sourceDir, 'package.json'), '{"name":"historical-source","private":true}\n'); + await writeFile(join(sourceDir, 'source.ts'), 'export const historical = true;\n'); + await git(sourceDir, ['init', '--initial-branch=main']); + await git(sourceDir, ['add', '--all']); + await git(sourceDir, [ + '-c', + 'user.name=Historical Fixture', + '-c', + 'user.email=historical@example.invalid', + 'commit', + '-m', + 'Pinned source', + ]); + const sourceCommit = await git(sourceDir, ['rev-parse', 'HEAD']); + const sourceTree = await git(sourceDir, ['rev-parse', 'HEAD^{tree}']); + const specification = Buffer.from('# Exact approved host landing specification\n\nBytes survive. \n'); + await writeFile(join(caseDir, 'spec.md'), specification); + await writeFile( + join(caseDir, 'public-contract.json'), + `${JSON.stringify( + { + schemaVersion: 1, + case: { + id: 'brunch-host-landing-v1', + specification: 'spec.md', + specificationSha256: createHash('sha256').update(specification).digest('hex'), + provider: 'anthropic', + model: 'claude-opus-4-8', + product: 'brunch', + mode: 'brownfield', + scope: 'single_feature', + surface: 'backend', + repository: { substrate: 'pinned_git', parentCommit: sourceCommit, parentTree: sourceTree }, + }, + budgets: { elapsedMinutes: 90, mechanicalInterventions: 2, substantiveHumanInterventions: 0 }, + delivery: { runtimeNetwork: 'forbidden', dependencyInstallNetwork: 'forbidden' }, + acceptance: { publicCommand: '/brunch:land', executionTerminal: 'promotion_prepared' }, + rules: ['Work only in the target repository.', 'Stop after promotion_prepared without landing.'], + }, + null, + 2, + )}\n`, + ); + return { casesRoot, controllerDir, sourceDir, sourceCommit, sourceTree, targetDir, specification }; +} + +async function git(cwd: string, args: readonly string[]): Promise { + const result = await runCommand('git', args, { cwd }); + if (result.exitCode !== 0) throw new Error(result.stderr || result.stdout); + return result.stdout.trim(); +} diff --git a/src/dev/execution-comparison/__tests__/host-landing-oracle.slow.test.ts b/src/dev/execution-comparison/__tests__/host-landing-oracle.slow.test.ts new file mode 100644 index 000000000..1c727da0a --- /dev/null +++ b/src/dev/execution-comparison/__tests__/host-landing-oracle.slow.test.ts @@ -0,0 +1,54 @@ +import { execFile } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +import { beforeAll, describe, expect, it } from 'vitest'; + +import { runBrunchHostLandingOracle } from '../host-landing-oracle.js'; + +const execFileAsync = promisify(execFile); +const candidateRoot = fileURLToPath(new URL('../../../../', import.meta.url)); + +describe('built Brunch host-landing oracle through the public TUI', () => { + beforeAll(async () => { + await execFileAsync('npm', ['run', 'build'], { + cwd: candidateRoot, + timeout: 240_000, + maxBuffer: 2 * 1024 * 1024, + }); + }, 250_000); + + it('rejects a fresh session before any provider-capable candidate launch', async () => { + const report = await runBrunchHostLandingOracle({ + candidateRoot, + sessionMode: 'fresh', + }); + + expect(report.status).toBe('setup_failed'); + expect(report.setupFailure).toContain('not settled'); + }); + + it.each([ + ['brownfield_success', 'passed'], + ['greenfield_success', 'passed'], + ['decline', 'passed'], + ['dirty_host', 'passed'], + ['conflict', 'passed'], + ['stale_acceptance', 'passed'], + ['final_commit_only', 'assertion_failed'], + ['bookkeeping_retained', 'assertion_failed'], + ] as const)( + 'judges %s as %s through /brunch:land', + async (scenario, expectedStatus) => { + const report = await runBrunchHostLandingOracle({ candidateRoot, scenario }); + + expect( + report.status, + [report.setupFailure, ...report.terminalEvidence].filter(Boolean).join('\n'), + ).toBe(expectedStatus); + expect(report.terminalEvidence.join('\n')).toContain('/brunch:land'); + expect(report.checks.find(({ id }) => id === 'public-tui-preflight')?.status).toBe('passed'); + }, + 90_000, + ); +}); diff --git a/src/dev/execution-comparison/__tests__/host-landing-oracle.test.ts b/src/dev/execution-comparison/__tests__/host-landing-oracle.test.ts new file mode 100644 index 000000000..bd83610f5 --- /dev/null +++ b/src/dev/execution-comparison/__tests__/host-landing-oracle.test.ts @@ -0,0 +1,135 @@ +import { execFile } from 'node:child_process'; +import { mkdir, mkdtemp, readFile, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +import { describe, expect, it } from 'vitest'; + +import { + evaluateHostLandingGitOutcome, + snapshotGitState, + type HostLandingScenario, +} from '../host-landing-oracle.js'; + +const execFileAsync = promisify(execFile); +const oracleSources = [ + '../host-landing-oracle.ts', + '../host-landing-oracle/types.ts', + '../host-landing-oracle/git-model.ts', + '../host-landing-oracle/fixture.ts', + '../host-landing-oracle/runner.ts', +].map((path) => fileURLToPath(new URL(path, import.meta.url))); +const IDENTITY = ['-c', 'user.name=oracle', '-c', 'user.email=oracle@invalid.local'] as const; + +describe('controller-owned host-landing oracle boundary', () => { + it('depends only on the public candidate launch, TUI driver, Git, and controller outputs', async () => { + const source = (await Promise.all(oracleSources.map((path) => readFile(path, 'utf8')))).join('\n'); + const imports = source + .split('\n') + .filter((line) => line.startsWith('import ') || line.startsWith('} from ')); + + expect(imports.join('\n')).not.toMatch( + /landing\.js|git-host-land-port|execute-land|executor\/|historical|FE-1201/iu, + ); + expect(source).toContain("join(input.candidateRoot, 'bin', 'brunch.js')"); + expect(source).toContain("'/usr/bin/env'"); + expect(source).toContain("'PI_OFFLINE=1'"); + }); + + it.each(['final_commit_only', 'bookkeeping_retained'] as const)( + 'rejects the focused %s rival against the independent expected tree', + async (scenario) => { + const fixture = await rivalFixture(scenario); + const report = await evaluateHostLandingGitOutcome({ + scenario, + hostDir: fixture.hostDir, + metadataPath: fixture.metadataPath, + canonicalExpectedTree: fixture.expectedTree, + before: fixture.snapshot, + preConfirm: fixture.snapshot, + terminalEvidence: ['3 commits across the complete range'], + providerActivity: false, + }); + + expect(report.status).toBe('assertion_failed'); + expect(report.checks.find(({ id }) => id === 'brownfield-full-range')?.status).toBe('failed'); + }, + ); + + it.each(['decline', 'dirty_host', 'conflict', 'stale_acceptance'] as const)( + 'accepts byte-identical %s refusal evidence without a landed status', + async (scenario) => { + const fixture = await refusalFixture(); + const report = await evaluateHostLandingGitOutcome({ + scenario, + hostDir: fixture.hostDir, + metadataPath: fixture.metadataPath, + canonicalExpectedTree: fixture.snapshot.tree, + before: fixture.snapshot, + preConfirm: fixture.snapshot, + terminalEvidence: ['complete range; Nothing changed'], + providerActivity: false, + }); + + expect(report.status).toBe('passed'); + expect(report.checks.find(({ id }) => id === 'refusal-safety')?.status).toBe('passed'); + }, + ); +}); + +async function rivalFixture( + scenario: Extract, +) { + const root = await mkdtemp(join(tmpdir(), `brunch-host-oracle-${scenario}-`)); + const model = join(root, 'model'); + await init(model); + await commit(model, 'src/a.ts', 'a\n', 'a'); + await commit(model, 'src/b.ts', 'b\n', 'b'); + await commit(model, 'src/c.ts', 'c\n', 'c'); + const expectedTree = await git(model, ['rev-parse', 'HEAD^{tree}']); + + const hostDir = join(root, 'host'); + await init(hostDir); + if (scenario === 'final_commit_only') { + await commit(hostDir, 'src/c.ts', 'c\n', 'tip only'); + } else { + await commit(hostDir, 'src/a.ts', 'a\n', 'a'); + await commit(hostDir, 'src/b.ts', 'b\n', 'b'); + await commit(hostDir, 'src/c.ts', 'c\n', 'c'); + await commit(hostDir, '.brunch/leak.json', '{}\n', 'bookkeeping'); + } + const metadataPath = join(hostDir, '.brunch/controller-run.json'); + await mkdir(dirname(metadataPath), { recursive: true }); + await writeFile(metadataPath, '{"status":"landed"}\n'); + const snapshot = await snapshotGitState(hostDir, metadataPath); + return { hostDir, metadataPath, expectedTree, snapshot }; +} + +async function refusalFixture() { + const root = await mkdtemp(join(tmpdir(), 'brunch-host-oracle-refusal-')); + const hostDir = join(root, 'host'); + await init(hostDir); + const metadataPath = join(hostDir, '.brunch/controller-run.json'); + await mkdir(dirname(metadataPath), { recursive: true }); + await writeFile(metadataPath, '{"status":"promotion_prepared"}\n'); + return { hostDir, metadataPath, snapshot: await snapshotGitState(hostDir, metadataPath) }; +} + +async function init(cwd: string): Promise { + await mkdir(cwd); + await execFileAsync('git', ['init', '-q', '-b', 'main'], { cwd }); + await execFileAsync('git', [...IDENTITY, 'commit', '--allow-empty', '-q', '-m', 'base'], { cwd }); +} + +async function commit(cwd: string, path: string, content: string, message: string): Promise { + await mkdir(dirname(join(cwd, path)), { recursive: true }); + await writeFile(join(cwd, path), content); + await execFileAsync('git', ['add', '--', path], { cwd }); + await execFileAsync('git', [...IDENTITY, 'commit', '-q', '-m', message], { cwd }); +} + +async function git(cwd: string, args: readonly string[]): Promise { + return (await execFileAsync('git', [...args], { cwd })).stdout.trim(); +} diff --git a/src/dev/execution-comparison/__tests__/operator-cli.test.ts b/src/dev/execution-comparison/__tests__/operator-cli.test.ts index a0c4b478c..4b65c528b 100644 --- a/src/dev/execution-comparison/__tests__/operator-cli.test.ts +++ b/src/dev/execution-comparison/__tests__/operator-cli.test.ts @@ -1,28 +1,42 @@ import { cp, mkdir, mkdtemp, readFile, readdir, rm, symlink } from 'node:fs/promises'; import { tmpdir } from 'node:os'; -import { join } from 'node:path'; +import { join, relative } from 'node:path'; import { fileURLToPath } from 'node:url'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; +import { runExecutionComparisonOperatorCli } from '../../execution-comparison-operator.js'; import { listExecutionCases, prepareExecutionTarget, resolveExecutionCase } from '../operator-cli.js'; const casesRoot = fileURLToPath(new URL('../../../../testing/execution-comparisons/cases/', import.meta.url)); const frozenCase = join(casesRoot, 'minimal-petri-net-editor'); +const controllerRoot = fileURLToPath(new URL('../../../../', import.meta.url)); describe('execution comparison operator case selection', () => { it('lists and resolves eligible case ids only inside the cases root', async () => { await expect(listExecutionCases(casesRoot)).resolves.toEqual([ + { + caseId: 'brunch-host-landing-v1', + directoryId: 'brunch-host-landing', + }, { caseId: 'minimal-petri-net-editor-v1', directoryId: 'minimal-petri-net-editor', }, + { + caseId: 'petrinaut-optimization-v1', + directoryId: 'petrinaut-optimization', + }, ]); await expect(resolveExecutionCase('minimal-petri-net-editor', casesRoot)).resolves.toMatchObject({ caseId: 'minimal-petri-net-editor-v1', directoryId: 'minimal-petri-net-editor', caseDir: frozenCase, }); + await expect(resolveExecutionCase('petrinaut-optimization-v1', casesRoot)).resolves.toMatchObject({ + caseId: 'petrinaut-optimization-v1', + directoryId: 'petrinaut-optimization', + }); }); it.each(['/tmp/minimal-petri-net-editor', '../minimal-petri-net-editor', 'controller', 'x/controller/y'])( @@ -84,7 +98,7 @@ describe('execution comparison target preparation', () => { expect(paths.some((path) => path.toLowerCase().includes('controller'))).toBe(false); expect(prepared.packet.files.map((file) => file.path)).toEqual(['public-contract.json', 'spec.md']); - if (prepared.lane === 'claude_code') { + if (prepared.preparation === 'empty_git') { expect(await readFile(join(targetDir, 'spec.md'), 'utf8')).toBe( await readFile(join(frozenCase, 'spec.md'), 'utf8'), ); @@ -92,7 +106,7 @@ describe('execution comparison target preparation', () => { await readFile(join(frozenCase, 'public-contract.json'), 'utf8'), ); expect(prepared.baseSha).toMatch(/^[a-f0-9]{40}$/u); - } else { + } else if (prepared.preparation === 'legacy_brunch') { expect(prepared.specId).toBe(1); } } finally { @@ -100,4 +114,131 @@ describe('execution comparison target preparation', () => { } }, ); + + it('requires the explicit source repository only for pinned cases', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-pinned-source-option-')); + try { + await expect( + runExecutionComparisonOperatorCli([ + 'prepare', + '--case', + 'petrinaut-optimization-v1', + '--lane', + 'claude_code', + '--target', + join(root, 'petrinaut'), + ]), + ).rejects.toThrow('requires --source-repository'); + await expect( + runExecutionComparisonOperatorCli([ + 'prepare', + '--case', + 'minimal-petri-net-editor-v1', + '--lane', + 'claude_code', + '--target', + join(root, 'petri'), + '--source-repository', + controllerRoot, + ]), + ).rejects.toThrow('valid only for pinned execution cases'); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + + it('rejects pinned targets overlapping controller or source roots before materialization', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-pinned-source-boundary-')); + const sourceRoot = join(root, 'source'); + const sourceLink = join(root, 'source-link'); + await mkdir(sourceRoot); + await symlink(sourceRoot, sourceLink); + try { + await expect( + prepareExecutionTarget( + { + lane: 'claude_code', + caseReference: 'petrinaut-optimization-v1', + casesRoot, + targetDir: join(root, 'relative-source-target'), + controllerRoot, + sourceRepositoryDir: 'relative-source', + }, + { + dependencyInstallRunner: async () => ({ exitCode: 0, stdout: '', stderr: '' }), + }, + ), + ).rejects.toThrow('must be an absolute path'); + await expect( + prepareExecutionTarget( + { + lane: 'claude_code', + caseReference: 'petrinaut-optimization-v1', + casesRoot, + targetDir: join(controllerRoot, '.unsafe-petrinaut-target'), + controllerRoot, + sourceRepositoryDir: sourceRoot, + }, + { + dependencyInstallRunner: async () => ({ exitCode: 0, stdout: '', stderr: '' }), + }, + ), + ).rejects.toThrow('controller and target roots must be disjoint'); + await expect( + prepareExecutionTarget( + { + lane: 'claude_code', + caseReference: 'petrinaut-optimization-v1', + casesRoot, + targetDir: join(root, 'linked-source-target'), + controllerRoot, + sourceRepositoryDir: sourceLink, + }, + { + dependencyInstallRunner: async () => ({ exitCode: 0, stdout: '', stderr: '' }), + }, + ), + ).rejects.toThrow('real directory, not a symlink'); + await expect( + prepareExecutionTarget( + { + lane: 'claude_code', + caseReference: 'petrinaut-optimization-v1', + casesRoot, + targetDir: join(sourceRoot, 'target'), + controllerRoot, + sourceRepositoryDir: sourceRoot, + }, + { + dependencyInstallRunner: async () => ({ exitCode: 0, stdout: '', stderr: '' }), + }, + ), + ).rejects.toThrow('controller and target roots must be disjoint'); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); + +describe('execution comparison oracle CLI', () => { + it('rejects a relative evidence path before oracle launch', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-relative-oracle-out-')); + const stdout = vi.spyOn(process.stdout, 'write').mockImplementation(() => true); + try { + await expect( + runExecutionComparisonOperatorCli([ + 'oracle', + '--case', + 'minimal-petri-net-editor-v1', + '--app', + root, + '--out', + relative(process.cwd(), join(root, 'report.json')), + ]), + ).rejects.toThrow('--out must be an absolute path'); + } finally { + stdout.mockRestore(); + await rm(root, { recursive: true, force: true }); + } + }); }); diff --git a/src/dev/execution-comparison/__tests__/operator-oracle-dispatch.test.ts b/src/dev/execution-comparison/__tests__/operator-oracle-dispatch.test.ts new file mode 100644 index 000000000..58f1e1fe9 --- /dev/null +++ b/src/dev/execution-comparison/__tests__/operator-oracle-dispatch.test.ts @@ -0,0 +1,138 @@ +import { mkdtemp, readFile, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { + EXECUTION_COMPARISON_SHARED_FRAMING, + resolveCompiledExecutionOracle, + retainCompiledOracleReport, +} from '../../execution-comparison-operator.js'; +import { loadControllerOraclePack } from '../oracle-pack.js'; + +const petrinautCaseDir = fileURLToPath( + new URL('../../../../testing/execution-comparisons/cases/petrinaut-optimization/', import.meta.url), +); + +describe('execution comparison compiled oracle dispatch', () => { + it('keeps shared framing neutral across browser and backend delivery contracts', () => { + expect(EXECUTION_COMPARISON_SHARED_FRAMING).toContain('case-specific delivery'); + expect(EXECUTION_COMPARISON_SHARED_FRAMING).not.toContain('static browser'); + expect(EXECUTION_COMPARISON_SHARED_FRAMING).not.toContain('do not add a backend'); + }); + + it('selects only the three compiled implementations and rejects unknown ids before launch', () => { + const petri = resolveCompiledExecutionOracle('minimal-petri-net-editor-oracles-v2'); + const brunch = resolveCompiledExecutionOracle('brunch-host-landing-oracles-v1'); + const petrinaut = resolveCompiledExecutionOracle('petrinaut-optimization-oracles-v1'); + + expect(petri.implementationFiles).toEqual( + expect.arrayContaining([expect.stringContaining('browser-oracle.ts')]), + ); + expect(brunch.implementationFiles).toEqual( + expect.arrayContaining([expect.stringContaining('host-landing-oracle.ts')]), + ); + expect(petrinaut.implementationFiles).toEqual( + expect.arrayContaining([ + expect.stringContaining('petrinaut-optimization-oracle.ts'), + expect.stringContaining('petrinaut-optimization-oracle/browser.ts'), + expect.stringContaining('petrinaut-optimization-oracle/calibration-seed.json'), + expect.stringContaining('petrinaut-optimization-oracle/claims.ts'), + expect.stringContaining('petrinaut-optimization-oracle/fake-optimizer.ts'), + ]), + ); + expect(() => resolveCompiledExecutionOracle('runtime-plugin')).toThrow( + 'unknown compiled execution oracle id', + ); + }); + + it('retains a claim-linked report beside its immutable oracle-pack hash', async () => { + const root = await mkdtemp(join(tmpdir(), 'brunch-oracle-dispatch-')); + const out = join(root, 'report.json'); + const report = { + schemaVersion: 1 as const, + caseId: 'brunch-host-landing-v1' as const, + oracleId: 'brunch-host-landing-oracles-v1' as const, + status: 'assertion_failed' as const, + scenario: 'final_commit_only' as const, + checks: [ + { + id: 'brownfield-full-range' as const, + claims: ['REQ2'], + status: 'failed' as const, + evidence: ['missing slice content'], + }, + ], + terminalEvidence: [], + gitEvidence: { + before: snapshot(), + preConfirm: snapshot(), + after: snapshot(), + expectedTree: 'a'.repeat(40), + actualTree: 'b'.repeat(40), + changedPaths: ['src/c.ts'], + }, + }; + const oraclePackSha256 = `sha256:${'c'.repeat(64)}`; + + await expect( + retainCompiledOracleReport({ + out, + oracleId: 'brunch-host-landing-oracles-v1', + oraclePackSha256, + report, + }), + ).resolves.toEqual({ + out, + status: 'assertion_failed', + oraclePackSha256, + oracleId: 'brunch-host-landing-oracles-v1', + }); + expect(JSON.parse(await readFile(out, 'utf8'))).toEqual(report); + await expect( + retainCompiledOracleReport({ + out, + oracleId: 'brunch-host-landing-oracles-v1', + oraclePackSha256, + report, + }), + ).rejects.toMatchObject({ code: 'EEXIST' }); + }); + + it('changes the Petrinaut oracle-pack identity when its calibration seed changes', async () => { + const oracle = resolveCompiledExecutionOracle('petrinaut-optimization-oracles-v1'); + const seedPath = oracle.implementationFiles.find((path) => path.endsWith('calibration-seed.json')); + expect(seedPath).toBeDefined(); + if (seedPath === undefined) return; + + const root = await mkdtemp(join(tmpdir(), 'brunch-oracle-seed-rival-')); + const rivalSeedPath = join(root, 'calibration-seed.json'); + await writeFile(rivalSeedPath, `${await readFile(seedPath, 'utf8')}\n`); + const [knownPack, rivalPack] = await Promise.all([ + loadControllerOraclePack({ + caseDir: petrinautCaseDir, + implementationFiles: oracle.implementationFiles, + }), + loadControllerOraclePack({ + caseDir: petrinautCaseDir, + implementationFiles: oracle.implementationFiles.map((path) => + path === seedPath ? rivalSeedPath : path, + ), + }), + ]); + + expect(rivalPack.packSha256).not.toBe(knownPack.packSha256); + }); +}); + +function snapshot() { + return { + head: '1'.repeat(40), + tree: '2'.repeat(40), + status: '', + runMetadataSha256: '3'.repeat(64), + runMetadataBytes: '{"status":"promotion_prepared"}\n', + }; +} diff --git a/src/dev/execution-comparison/__tests__/oracle-pack.test.ts b/src/dev/execution-comparison/__tests__/oracle-pack.test.ts new file mode 100644 index 000000000..774814e1b --- /dev/null +++ b/src/dev/execution-comparison/__tests__/oracle-pack.test.ts @@ -0,0 +1,90 @@ +import { readFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { + assertOracleClaimCoverage, + loadControllerOracleManifest, + parseControllerOracleManifest, +} from '../oracle-pack.js'; + +const casesRoot = fileURLToPath(new URL('../../../../testing/execution-comparisons/cases/', import.meta.url)); +const requirementsPath = fileURLToPath( + new URL( + '../../../../testing/end-to-end-comparisons/cases/brunch-host-landing/controller/requirement-registry.json', + import.meta.url, + ), +); +const petrinautRequirementsPath = fileURLToPath( + new URL( + '../../../../testing/end-to-end-comparisons/cases/petrinaut-optimization/controller/requirement-registry.json', + import.meta.url, + ), +); + +describe('compiled controller oracle manifests', () => { + it('accepts exactly three compiled variants with complete brownfield claim coverage', async () => { + const [petri, brunch, petrinaut, registry, petrinautRegistry] = await Promise.all([ + loadControllerOracleManifest(join(casesRoot, 'minimal-petri-net-editor')), + loadControllerOracleManifest(join(casesRoot, 'brunch-host-landing')), + loadControllerOracleManifest(join(casesRoot, 'petrinaut-optimization')), + readFile(requirementsPath, 'utf8').then( + (raw) => JSON.parse(raw) as { rows: readonly { id: string }[] }, + ), + readFile(petrinautRequirementsPath, 'utf8').then( + (raw) => JSON.parse(raw) as { rows: readonly { id: string }[] }, + ), + ]); + + expect(petri.id).toBe('minimal-petri-net-editor-oracles-v2'); + expect(brunch.id).toBe('brunch-host-landing-oracles-v1'); + expect(petrinaut.id).toBe('petrinaut-optimization-oracles-v1'); + expect(() => + assertOracleClaimCoverage( + brunch, + registry.rows.map(({ id }) => id), + ), + ).not.toThrow(); + expect(() => + assertOracleClaimCoverage( + petrinaut, + petrinautRegistry.rows.map(({ id }) => id), + ), + ).not.toThrow(); + expect(JSON.stringify(brunch)).not.toMatch(/manifestPath|command|plugin|implementationPath/u); + expect(JSON.stringify(petrinaut)).not.toMatch(/manifestPath|command|plugin|implementationPath/u); + expect(() => + parseControllerOracleManifest({ + ...petrinaut, + validityRules: [ + 'The candidate execution lane terminates at promotion_prepared before controller dependency preparation.', + 'The browser opens /optimization after preparation.', + ], + }), + ).toThrow('invalid fixed controller oracle manifest'); + }); + + it('rejects unknown ids and runtime implementation selectors', () => { + expect(() => + parseControllerOracleManifest({ + schemaVersion: 1, + id: 'unknown-oracle', + }), + ).toThrow('invalid fixed controller oracle manifest'); + expect(() => + parseControllerOracleManifest({ + schemaVersion: 1, + id: 'brunch-host-landing-oracles-v1', + publicCaseId: 'brunch-host-landing-v1', + runnerVersion: 'brunch-host-landing-v1', + referenceModelVersion: 'git-full-range-v1', + checks: [{ id: 'x', claims: ['REQ1'] }], + validityRules: ['promotion_prepared then controller landed'], + replacementRule: 'retain', + plugin: './oracle.js', + }), + ).toThrow('invalid fixed controller oracle manifest'); + }); +}); diff --git a/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.slow.test.ts b/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.slow.test.ts new file mode 100644 index 000000000..709cb12a3 --- /dev/null +++ b/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.slow.test.ts @@ -0,0 +1,178 @@ +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { afterAll, describe, expect, it } from 'vitest'; + +import { runCommand } from '../../../app/command-runner.js'; +import { isPetrinautOptimizationExecutionCaseContract, loadPublicCasePacket } from '../case-contract.js'; +import { loadControllerOracleManifest } from '../oracle-pack.js'; +import { + PETRINAUT_FOCUSED_PREPARATION, + runPetrinautOptimizationOracle, +} from '../petrinaut-optimization-oracle.js'; +import { runPetrinautBrowserChecks } from '../petrinaut-optimization-oracle/browser.js'; +import { + createInventedLabelsPetrinautCandidate, + createKnownGoodPetrinautCandidate, +} from '../petrinaut-optimization-oracle/fixture.js'; + +const caseDir = fileURLToPath( + new URL('../../../../testing/execution-comparisons/cases/petrinaut-optimization/', import.meta.url), +); +const roots: string[] = []; + +afterAll(async () => { + await Promise.all(roots.splice(0).map(async (root) => await rm(root, { recursive: true, force: true }))); +}); + +describe('standalone Petrinaut optimization browser oracle', () => { + it('proves the focused route, request, stream, failure, cancellation, origin, and accessibility leaves', async () => { + const candidateRoot = await mkdtemp(join(tmpdir(), 'brunch-petrinaut-known-good-')); + roots.push(candidateRoot); + await createKnownGoodPetrinautCandidate(candidateRoot); + + const report = await runPetrinautOptimizationOracle({ candidateRoot, caseDir }); + + expect(report.status, report.setupFailure ?? JSON.stringify(report, null, 2)).toBe('passed'); + expect(report.preparation.map(({ id }) => id)).toEqual(PETRINAUT_FOCUSED_PREPARATION.map(({ id }) => id)); + expect(report.preparation.every(({ status }) => status === 'passed')).toBe(true); + expect(report.checks.map(({ id, status }) => ({ id, status }))).toEqual([ + { id: 'route-and-accessibility', status: 'passed' }, + { id: 'scenario-configuration', status: 'passed' }, + { id: 'request-contract', status: 'passed' }, + { id: 'progress-and-completion', status: 'passed' }, + { id: 'service-error', status: 'passed' }, + { id: 'cancel-and-abort', status: 'passed' }, + { id: 'private-origin-secrecy', status: 'passed' }, + ]); + expect(report.checks.find(({ id }) => id === 'request-contract')?.evidence).toEqual( + expect.arrayContaining([ + 'captured flat fixed/optimized bindings', + 'captured saved and custom objectives with direction', + ]), + ); + expect(report.checks.find(({ id }) => id === 'progress-and-completion')?.evidence).toEqual( + expect.arrayContaining(['progressive trial rendered', 'best-so-far rendered', 'completion rendered']), + ); + expect(report.checks.find(({ id }) => id === 'cancel-and-abort')?.evidence).toEqual( + expect.arrayContaining(['host request aborted', 'cancelled state rendered']), + ); + expect(report.checks.find(({ id }) => id === 'private-origin-secrecy')?.evidence).toEqual( + expect.arrayContaining([ + 'browser traffic remained same-origin', + 'DOM omitted private optimizer origin', + ]), + ); + expect(report.consoleErrors).toEqual([]); + }, 120_000); + + it('uses semantic DOM readiness while background traffic remains non-idle beyond five seconds', async () => { + const candidateRoot = await mkdtemp(join(tmpdir(), 'brunch-petrinaut-background-readiness-')); + roots.push(candidateRoot); + await createKnownGoodPetrinautCandidate(candidateRoot, { + backgroundRequestDurationMs: 6_000, + }); + + const report = await runPetrinautOptimizationOracle({ + candidateRoot, + caseDir, + }); + + expect(report.status, report.setupFailure ?? JSON.stringify(report, null, 2)).toBe('passed'); + expect(report.failedRequests).toEqual([]); + expect(report.checks.every(({ status }) => status === 'passed')).toBe(true); + }, 120_000); + + it('fails an invented-labels rival that would satisfy the pre-D138 accessibility fiction', async () => { + const candidateRoot = await mkdtemp(join(tmpdir(), 'brunch-petrinaut-invented-labels-')); + roots.push(candidateRoot); + await createInventedLabelsPetrinautCandidate(candidateRoot); + + const report = await runPetrinautOptimizationOracle({ candidateRoot, caseDir }); + + expect(report.status).toBe('assertion_failed'); + expect(report.checks.find(({ id }) => id === 'route-and-accessibility')?.status).toBe('failed'); + }, 120_000); + + it('does not reuse the empty-state scenario address after selection', async () => { + const candidateRoot = await mkdtemp(join(tmpdir(), 'brunch-petrinaut-wrong-scenario-')); + roots.push(candidateRoot); + await createKnownGoodPetrinautCandidate(candidateRoot); + for (const step of PETRINAUT_FOCUSED_PREPARATION) { + const result = await runCommand(step.command, step.args, { + cwd: candidateRoot, + timeoutMs: 30_000, + maxOutputBytes: 16 * 1024, + }); + expect(result.exitCode, result.stderr).toBe(0); + } + const [packet, manifest] = await Promise.all([ + loadPublicCasePacket(caseDir), + loadControllerOracleManifest(caseDir), + ]); + if (!isPetrinautOptimizationExecutionCaseContract(packet.contract)) { + throw new Error('expected Petrinaut packet'); + } + if (manifest.id !== 'petrinaut-optimization-oracles-v1') { + throw new Error('expected Petrinaut manifest'); + } + const contract = { + ...packet.contract, + mechanicalAddresses: { + ...packet.contract.mechanicalAddresses, + scenarioSelected: { + kind: 'roleName', + role: 'combobox', + name: 'Missing selected scenario selector', + } as const, + }, + }; + + const result = await runPetrinautBrowserChecks({ candidateRoot, contract, manifest }); + + expect(result.checks.find(({ id }) => id === 'route-and-accessibility')?.status).toBe('passed'); + expect(result.checks.find(({ id }) => id === 'scenario-configuration')?.status).toBe('failed'); + }, 120_000); + + it.each([ + [ + 'omission', + (steps: typeof PETRINAUT_FOCUSED_PREPARATION) => steps.filter(({ id }) => id !== 'refractive-build'), + ], + [ + 'reordering', + (steps: typeof PETRINAUT_FOCUSED_PREPARATION) => { + const reordered = [...steps]; + const optimizerIndex = reordered.findIndex(({ id }) => id === 'optimizer-client-build'); + const refractiveIndex = reordered.findIndex(({ id }) => id === 'refractive-build'); + [reordered[optimizerIndex], reordered[refractiveIndex]] = [ + reordered[refractiveIndex]!, + reordered[optimizerIndex]!, + ]; + return reordered; + }, + ], + ] as const)( + 'rejects focused Refractive %s before browser setup', + async (_name, mutate) => { + const candidateRoot = await mkdtemp(join(tmpdir(), 'brunch-petrinaut-sensitive-')); + roots.push(candidateRoot); + await createKnownGoodPetrinautCandidate(candidateRoot); + const results = []; + for (const step of mutate(PETRINAUT_FOCUSED_PREPARATION)) { + const result = await runCommand(step.command, step.args, { + cwd: candidateRoot, + timeoutMs: 30_000, + maxOutputBytes: 16 * 1024, + }); + results.push(result); + if (result.exitCode !== 0) break; + } + expect(results.at(-1)).toMatchObject({ exitCode: 1 }); + expect(results.at(-1)?.stderr).toContain('focused preparation ran out of order'); + }, + 30_000, + ); +}); diff --git a/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.test.ts b/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.test.ts new file mode 100644 index 000000000..035b8f110 --- /dev/null +++ b/src/dev/execution-comparison/__tests__/petrinaut-optimization-oracle.test.ts @@ -0,0 +1,163 @@ +import { readFile, readdir } from 'node:fs/promises'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { describe, expect, it } from 'vitest'; + +import { PETRINAUT_FOCUSED_PREPARATION } from '../petrinaut-optimization-oracle.js'; +import { assessPetrinautFocusedObservation } from '../petrinaut-optimization-oracle/claims.js'; +import { startDeterministicFakeOptimizer } from '../petrinaut-optimization-oracle/fake-optimizer.js'; + +const oracleRoot = fileURLToPath(new URL('../petrinaut-optimization-oracle/', import.meta.url)); +const oracleEntry = fileURLToPath(new URL('../petrinaut-optimization-oracle.ts', import.meta.url)); + +describe('controller-owned Petrinaut optimization oracle boundary', () => { + it('frames deterministic optimizer events as the upstream Optuna SSE contract', async () => { + const fake = await startDeterministicFakeOptimizer(); + try { + const response = await fetch(`${fake.origin}/optimize/all`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ name: 'source fidelity' }), + }); + + expect(response.headers.get('content-type')).toBe('text/event-stream'); + const stream = await response.text(); + expect(stream).toContain('data: {"step":0,"params":{"rate":4},"metric":12,"state":"COMPLETE"}'); + expect(stream).toContain('data: {"step":1,"params":{"rate":6},"metric":10,"state":"COMPLETE"}'); + expect(stream).toContain('event: done\ndata: {}'); + } finally { + await fake.close(); + } + }); + + it('keeps the declared Refractive workspace build directly before Petrinaut UI', () => { + expect(PETRINAUT_FOCUSED_PREPARATION).toEqual([ + { + id: 'design-system-codegen', + command: 'yarn', + args: ['workspace', '@hashintel/ds-components', 'codegen'], + }, + { + id: 'design-system-build', + command: 'yarn', + args: ['workspace', '@hashintel/ds-components', 'build'], + }, + { + id: 'petrinaut-core-build', + command: 'yarn', + args: ['workspace', '@hashintel/petrinaut-core', 'build'], + }, + { + id: 'optimizer-client-build', + command: 'yarn', + args: ['workspace', '@local/petrinaut-optimizer-client', 'build'], + }, + { + id: 'refractive-build', + command: 'yarn', + args: ['workspace', '@hashintel/refractive', 'build'], + }, + { + id: 'petrinaut-ui-build', + command: 'yarn', + args: ['workspace', '@hashintel/petrinaut', 'build'], + }, + ]); + }); + + it('contains no historical, candidate-internal, or runtime-selected implementation dependency', async () => { + const sources = await sourceFiles(oracleRoot); + const source = ( + await Promise.all([ + readFile(oracleEntry, 'utf8'), + ...sources.map(async (path) => await readFile(path, 'utf8')), + ]) + ).join('\n'); + const imports = source + .split('\n') + .filter((line) => line.startsWith('import ') || line.startsWith('} from ')) + .join('\n'); + + expect(imports).not.toMatch( + /apps\/petrinaut|libs\/@hashintel\/petrinaut|create-optimization|optimizations-view|provider\.js/iu, + ); + expect(source).not.toMatch(/FE-1162|9051|276e17d7|historical solution|merged reference/iu); + expect(source).not.toMatch(/implementationPath|oraclePath|pluginPath|manifest\.(?:command|path)/u); + }); + + it('keeps D138 mechanical interactions on declared or calibration-derived addresses', async () => { + const browserSource = await readFile(join(oracleRoot, 'browser.ts'), 'utf8'); + expect(browserSource).toMatch(/mechanicalAddresses/u); + expect(browserSource).toMatch(/locate\(page, addresses\.create\)/u); + expect(browserSource).toMatch(/locate\(page, addresses\.metricCode\)/u); + expect(browserSource).toMatch(/addresses\.scenarioSelected/u); + expect(browserSource).toMatch(/addresses\.metricCustomOption/u); + expect(browserSource).toMatch(/addresses\.optimizationName/u); + expect(browserSource).toMatch(/parseCalibrationInputs/u); + expect(browserSource).not.toMatch(/Create optimization/u); + expect(browserSource).not.toMatch(/getByRole\('heading', \{ name: 'Optimizations'/u); + expect(browserSource).not.toMatch(/getByRole\('tab', \{ name: 'Optimizations'/u); + expect(browserSource).not.toMatch(/getByRole\('checkbox'/u); + expect(browserSource).not.toMatch(/getByRole\('combobox'\)\.first/u); + expect(browserSource).not.toMatch( + /Seasonal Flu|High Virulence Outbreak|Optimize infected_ratio|Infected Fraction/u, + ); + expect(browserSource).not.toMatch(/OPTIMIZATION_NAME_ADDRESS|CUSTOM_METRIC_OPTION/u); + }); + + it.each([ + [ + 'missing-route', + { + check: 'route-and-accessibility', + pathname: '/processes/draft', + expectedPathname: '/optimization', + } as const, + 'focused route missing', + ], + [ + 'final-only', + { + check: 'progress-and-completion', + progressiveTrialCount: 0, + bestSoFarVisible: false, + completionVisible: true, + } as const, + 'progressive trials missing', + ], + [ + 'direct-private-origin', + { + check: 'private-origin-secrecy', + candidateOrigin: 'http://candidate.test', + browserRequestUrls: ['http://candidate.test/optimization', 'http://optimizer.private/optimize/all'], + domText: 'Optimizations', + privateOrigin: 'http://optimizer.private', + } as const, + 'browser contacted private origin', + ], + [ + 'UI-only-cancel', + { + check: 'cancel-and-abort', + cancelControlVisible: true, + cancelledVisible: true, + hostRequestAborted: false, + } as const, + 'host request was not aborted', + ], + ])('%s rival fails its focused claim', (_name, observation, expectedFailure) => { + expect(assessPetrinautFocusedObservation(observation)).toContain(expectedFailure); + }); +}); + +async function sourceFiles(root: string): Promise { + const files: string[] = []; + for (const entry of await readdir(root, { withFileTypes: true })) { + const path = join(root, entry.name); + if (entry.isDirectory()) files.push(...(await sourceFiles(path))); + else if (entry.isFile() && entry.name.endsWith('.ts')) files.push(path); + } + return files; +} diff --git a/src/dev/execution-comparison/accessibility-contract.ts b/src/dev/execution-comparison/accessibility-contract.ts index 6f561652b..909cf6d11 100644 --- a/src/dev/execution-comparison/accessibility-contract.ts +++ b/src/dev/execution-comparison/accessibility-contract.ts @@ -3,7 +3,7 @@ import { queryAllByRole } from '@testing-library/dom'; import { compileAccessibleNamePattern, type AccessibleNameContract, - type ExecutionCasePublicContract, + type BrowserExecutionCasePublicContract, } from './case-contract.js'; export interface AccessibilityContractAssertion { @@ -15,9 +15,9 @@ export interface AccessibilityContractAssertion { export function assertAccessibilityContract( root: HTMLElement, - contract: ExecutionCasePublicContract['accessibility'], + contract: BrowserExecutionCasePublicContract['accessibility'], requirements: { - readonly dynamic?: readonly (keyof ExecutionCasePublicContract['accessibility']['dynamic'])[]; + readonly dynamic?: readonly (keyof BrowserExecutionCasePublicContract['accessibility']['dynamic'])[]; readonly inspectorFields?: readonly string[]; } = {}, ): AccessibilityContractAssertion { diff --git a/src/dev/execution-comparison/browser-oracle.ts b/src/dev/execution-comparison/browser-oracle.ts index c907000a5..5cde3b709 100644 --- a/src/dev/execution-comparison/browser-oracle.ts +++ b/src/dev/execution-comparison/browser-oracle.ts @@ -6,8 +6,16 @@ import { chromium, type Browser, type BrowserContext, type Locator, type Page } import { runCommand } from '../../app/command-runner.js'; import { runIndependentJourneys, type IndependentJourney } from './browser-oracle/journey-runner.js'; -import { loadPublicCasePacket, type ExecutionCasePublicContract } from './case-contract.js'; -import { loadControllerOracleManifest, type ControllerOracleManifest } from './oracle-pack.js'; +import { + isBrowserExecutionCaseContract, + loadPublicCasePacket, + type BrowserExecutionCasePublicContract, +} from './case-contract.js'; +import { + isPetriControllerOracleManifest, + loadControllerOracleManifest, + type PetriControllerOracleManifest, +} from './oracle-pack.js'; type AriaRole = Parameters[0]; @@ -40,6 +48,9 @@ export async function runPetriEditorBrowserOracle(input: { }): Promise { const packet = await loadPublicCasePacket(input.caseDir); const manifest = await loadControllerOracleManifest(input.caseDir); + if (!isBrowserExecutionCaseContract(packet.contract) || !isPetriControllerOracleManifest(manifest)) { + throw new Error('Petri browser oracle received a non-browser case'); + } const commands: BrowserOracleReport['commands'][number][] = []; for (const [id, command] of [ ['test', packet.contract.delivery.test], @@ -156,8 +167,8 @@ async function openJourneyEnvironment(browser: Browser, url: string): Promise[] { const definitions = new Map, 'id' | 'claims'>>([ @@ -397,7 +408,7 @@ async function assertRoundTripAndClear(page: Page): Promise { async function assertBaseAccessibility( page: Page, - contract: ExecutionCasePublicContract['accessibility'], + contract: BrowserExecutionCasePublicContract['accessibility'], ): Promise { await requireExactlyOne(page, contract.application.role, contract.application.name); await requireExactlyOne(page, contract.canvas.role, contract.canvas.name); diff --git a/src/dev/execution-comparison/brunch-lane.ts b/src/dev/execution-comparison/brunch-lane.ts index fe8c3bee3..8a62c4db5 100644 --- a/src/dev/execution-comparison/brunch-lane.ts +++ b/src/dev/execution-comparison/brunch-lane.ts @@ -9,7 +9,9 @@ import { type SeedFixtureNode, } from '../../graph/seed-fixtures.js'; import { + isBrowserExecutionCaseContract, loadPublicCasePacket, + type BrowserExecutionCasePublicContract, type ExecutionCasePublicContract, type PublicCasePacket, } from './case-contract.js'; @@ -27,6 +29,12 @@ export interface PreparedBrunchExecutionWorkspace { readonly packet: PublicCasePacket; } +export interface SeededBrownfieldBrunchWorkspace { + readonly workspaceDir: string; + readonly specId: number; + readonly packet: PublicCasePacket; +} + export async function prepareBrunchExecutionWorkspace(input: { readonly workspaceDir: string; readonly caseDir: string; @@ -39,6 +47,9 @@ export async function prepareBrunchExecutionWorkspace(input: { } const packet = await loadPublicCasePacket(input.caseDir); + if (!isBrowserExecutionCaseContract(packet.contract)) { + throw new Error('legacy Brunch execution seed supports only the Petri browser case'); + } const specification = await readFile(join(input.caseDir, packet.contract.case.specification), 'utf8'); const executor = await openWorkspaceCommandExecutor(input.workspaceDir); const seeded = seedFixture( @@ -72,9 +83,39 @@ export async function prepareBrunchExecutionWorkspace(input: { return { workspaceDir: input.workspaceDir, specId: seeded.specId, publicDir, packet }; } +export async function seedBrownfieldBrunchExecutionWorkspace(input: { + readonly workspaceDir: string; +}): Promise { + const packet = await loadPublicCasePacket(input.workspaceDir); + if (isBrowserExecutionCaseContract(packet.contract) || packet.contract.case.mode !== 'brownfield') { + throw new Error('brownfield Brunch execution seed requires a brownfield public contract'); + } + const specification = await readFile(join(input.workspaceDir, packet.contract.case.specification), 'utf8'); + const executor = await openWorkspaceCommandExecutor(input.workspaceDir); + const seeded = seedFixture( + executor, + buildOpaqueBrownfieldExecutionSeed({ + specification, + contract: packet.contract, + }), + ); + const established = executor.establishSpecPosture({ + specId: seeded.specId, + origin: 'brownfield', + }); + if (established.status !== 'success') { + throw new Error(`failed to establish brownfield Brunch execution posture: ${established.status}`); + } + return { + workspaceDir: input.workspaceDir, + specId: seeded.specId, + packet, + }; +} + export function buildBrunchExecutionSeed(input: { readonly specification: string; - readonly contract: ExecutionCasePublicContract; + readonly contract: BrowserExecutionCasePublicContract; }): SeedFixture { const sections = parseApprovedSpecification(input.specification); const nodes: SeedFixtureNode[] = sections.map((section, index) => sectionNode(section, index + 1)); @@ -86,7 +127,7 @@ export function buildBrunchExecutionSeed(input: { export function buildOpaqueBrunchExecutionSeed(input: { readonly specification: string; - readonly contract: ExecutionCasePublicContract; + readonly contract: BrowserExecutionCasePublicContract; }): SeedFixture { return buildExecutionSeed({ nodes: [ @@ -115,9 +156,106 @@ export function buildOpaqueBrunchExecutionSeed(input: { }); } +export function buildOpaqueBrownfieldExecutionSeed(input: { + readonly specification: string; + readonly contract: ExecutionCasePublicContract; +}): SeedFixture { + if (isBrowserExecutionCaseContract(input.contract) || input.contract.case.mode !== 'brownfield') { + throw new Error('opaque brownfield seed requires a brownfield public contract'); + } + return { + spec: { + slug: input.contract.case.id, + name: `Approved ${input.contract.case.product} brownfield execution`, + kind: 'feature', + }, + nodes: [ + { + local_id: 1, + plane: 'intent', + kind: 'requirement', + title: 'Approved target-authored specification', + body: input.specification, + basis: 'explicit', + settlement: 'settled', + source: 'e2e-handoff [exact-spec]', + }, + { + local_id: 2, + plane: 'plan', + kind: 'frontier', + title: 'Deliver the approved brownfield change', + body: 'Implement the approved feature in the existing target repository.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [frontier]', + }, + { + local_id: 3, + plane: 'plan', + kind: 'scope', + title: 'Implement and verify the approved brownfield change', + body: 'Execute the frozen specification as one coherent change while preserving the surrounding repository.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [scope]', + }, + { + local_id: 4, + plane: 'design', + kind: 'module', + title: 'Brownfield feature implementation', + body: 'The implementation location and internal design remain the executor’s responsibility.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [module]', + }, + { + local_id: 5, + plane: 'intent', + kind: 'criterion', + title: 'Approved brownfield behavior is satisfied', + body: 'The completed change satisfies the frozen approved specification in the existing repository.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [criterion]', + }, + { + local_id: 6, + plane: 'oracle', + kind: 'check', + title: 'Repository-local verification passes', + body: 'Use the prepared repository’s own focused build and test surfaces to verify the approved change.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [check]', + }, + { + local_id: 7, + plane: 'oracle', + kind: 'vv_method', + title: 'Prepared-target verification', + body: 'Run only repository-local verification available inside the prepared target under the execution isolation policy.', + basis: 'explicit', + settlement: 'settled', + source: 'execution-adapter [vv_method]', + }, + ], + edges: [ + edge('composition', 2, 3), + edge('realization', 1, 3), + edge('dependency', 5, 3), + edge('composition', 3, 4), + edge('dependency', 6, 3), + edge('witness', 6, 5, 'for'), + edge('realization', 7, 6), + ], + }; +} + function buildExecutionSeed(input: { readonly nodes: SeedFixtureNode[]; - readonly contract: ExecutionCasePublicContract; + readonly contract: BrowserExecutionCasePublicContract; }): SeedFixture { const nodes = [...input.nodes]; const next = nodes.length + 1; @@ -294,7 +432,7 @@ function kindForCode(code: string): { throw new Error(`unsupported approved specification code: ${code}`); } -function renderAccessibilityContract(contract: ExecutionCasePublicContract): string { +function renderAccessibilityContract(contract: BrowserExecutionCasePublicContract): string { const controls = contract.accessibility.controls .map((control) => `${control.role} "${control.name}"`) .join(', '); diff --git a/src/dev/execution-comparison/case-contract.ts b/src/dev/execution-comparison/case-contract.ts index d2074ca7b..3d2369bc2 100644 --- a/src/dev/execution-comparison/case-contract.ts +++ b/src/dev/execution-comparison/case-contract.ts @@ -2,7 +2,7 @@ import { createHash } from 'node:crypto'; import { readFile } from 'node:fs/promises'; import { basename, join } from 'node:path'; -export interface ExecutionCasePublicContract { +export interface BrowserExecutionCasePublicContract { readonly schemaVersion: 1; readonly case: { readonly id: 'minimal-petri-net-editor-v1'; @@ -39,6 +39,128 @@ export interface ExecutionCasePublicContract { readonly rules: readonly string[]; } +export interface BrunchHostLandingExecutionCasePublicContract { + readonly schemaVersion: 1; + readonly case: { + readonly id: 'brunch-host-landing-v1'; + readonly specification: 'spec.md'; + readonly specificationSha256: string; + readonly provider: 'anthropic'; + readonly model: 'claude-opus-4-8'; + readonly product: 'brunch'; + readonly mode: 'brownfield'; + readonly scope: 'single_feature'; + readonly surface: 'backend'; + readonly repository: { + readonly substrate: 'pinned_git'; + readonly parentCommit: string; + readonly parentTree: string; + }; + }; + readonly budgets: { + readonly elapsedMinutes: 90; + readonly mechanicalInterventions: 2; + readonly substantiveHumanInterventions: 0; + }; + readonly delivery: { + readonly runtimeNetwork: 'forbidden'; + readonly dependencyInstallNetwork: 'forbidden'; + }; + readonly acceptance: { + readonly publicCommand: '/brunch:land'; + readonly executionTerminal: 'promotion_prepared'; + }; + readonly rules: readonly string[]; +} + +export type PetrinautMechanicalAddress = + | { readonly kind: 'roleName'; readonly role: string; readonly name: string } + | { readonly kind: 'roleValue'; readonly role: string; readonly value: string } + | { readonly kind: 'roleContents'; readonly role: string; readonly contents: string } + | { readonly kind: 'exactText'; readonly text: string }; + +export type PetrinautMechanicalAddressKey = + | 'skipTour' + | 'dismissAssistant' + | 'simulateMode' + | 'optimizationsNav' + | 'viewTitle' + | 'create' + | 'createDrawer' + | 'scenario' + | 'scenarioSelected' + | 'metric' + | 'metricCustomOption' + | 'metricCode' + | 'optimizationName' + | 'directionMaximize' + | 'directionMinimize' + | 'run' + | 'cancel' + | 'statusComplete' + | 'statusError' + | 'statusCancelled'; + +export interface PetrinautOptimizationExecutionCasePublicContract { + readonly schemaVersion: 1; + readonly case: { + readonly id: 'petrinaut-optimization-v1'; + readonly specification: 'spec.md'; + readonly specificationSha256: string; + readonly provider: 'anthropic'; + readonly model: 'claude-opus-4-8'; + readonly product: 'petrinaut'; + readonly mode: 'brownfield'; + readonly scope: 'single_feature'; + readonly surface: 'frontend'; + readonly repository: { + readonly substrate: 'pinned_git'; + readonly parentCommit: '5c7a2d9db5caa851c38938f4b1bac19005b0e978'; + readonly parentTree: 'a3e08cf75e00cc9016c931f4665341506e03533e'; + }; + }; + readonly budgets: { + readonly elapsedMinutes: 90; + readonly mechanicalInterventions: 2; + readonly substantiveHumanInterventions: 0; + }; + readonly delivery: { + readonly runtimeNetwork: 'forbidden'; + readonly dependencyInstallNetwork: 'controller_only'; + }; + readonly acceptance: { + readonly publicRoute: '/optimization'; + readonly sameOriginApi: '/api/petrinaut-opt/optimize/all'; + readonly executionTerminal: 'promotion_prepared'; + }; + readonly mechanicalAddresses: Readonly>; + readonly rules: readonly string[]; +} + +export type ExecutionCasePublicContract = + | BrowserExecutionCasePublicContract + | BrunchHostLandingExecutionCasePublicContract + | PetrinautOptimizationExecutionCasePublicContract; + +export type PinnedExecutionCasePublicContract = Extract< + ExecutionCasePublicContract, + { readonly case: { readonly repository: { readonly substrate: 'pinned_git' } } } +>; + +export type PinnedExecutionCaseId = PinnedExecutionCasePublicContract['case']['id']; + +export function isBrowserExecutionCaseContract( + value: ExecutionCasePublicContract, +): value is BrowserExecutionCasePublicContract { + return value.case.id === 'minimal-petri-net-editor-v1'; +} + +export function isPetrinautOptimizationExecutionCaseContract( + value: ExecutionCasePublicContract, +): value is PetrinautOptimizationExecutionCasePublicContract { + return value.case.id === 'petrinaut-optimization-v1'; +} + export interface CommandContract { readonly command: string; readonly args: readonly string[]; @@ -102,6 +224,16 @@ export async function loadPublicCasePacket(caseDir: string): Promise): BrowserExecutionCasePublicContract { const caseValue = requiredRecord(value, 'case'); const repository = requiredRecord(caseValue, 'repository'); const budgets = requiredRecord(value, 'budgets'); @@ -160,7 +292,193 @@ export function parsePublicCaseContract(value: unknown): ExecutionCasePublicCont invalid(); } - return value as unknown as ExecutionCasePublicContract; + return value as unknown as BrowserExecutionCasePublicContract; +} + +function parseBrunchHostLandingContract( + value: Record, +): BrunchHostLandingExecutionCasePublicContract { + const caseValue = requiredRecord(value, 'case'); + const repository = requiredRecord(caseValue, 'repository'); + const budgets = requiredRecord(value, 'budgets'); + const delivery = requiredRecord(value, 'delivery'); + const acceptance = requiredRecord(value, 'acceptance'); + if ( + !exactKeys(value, ['schemaVersion', 'case', 'budgets', 'delivery', 'acceptance', 'rules']) || + !exactKeys(caseValue, [ + 'id', + 'specification', + 'specificationSha256', + 'provider', + 'model', + 'product', + 'mode', + 'scope', + 'surface', + 'repository', + ]) || + !exactKeys(repository, ['substrate', 'parentCommit', 'parentTree']) || + !exactKeys(budgets, ['elapsedMinutes', 'mechanicalInterventions', 'substantiveHumanInterventions']) || + !exactKeys(delivery, ['runtimeNetwork', 'dependencyInstallNetwork']) || + !exactKeys(acceptance, ['publicCommand', 'executionTerminal']) || + value['schemaVersion'] !== 1 || + caseValue['id'] !== 'brunch-host-landing-v1' || + caseValue['specification'] !== 'spec.md' || + !sha256HexValue(caseValue['specificationSha256']) || + caseValue['provider'] !== 'anthropic' || + caseValue['model'] !== 'claude-opus-4-8' || + caseValue['product'] !== 'brunch' || + caseValue['mode'] !== 'brownfield' || + caseValue['scope'] !== 'single_feature' || + caseValue['surface'] !== 'backend' || + repository['substrate'] !== 'pinned_git' || + !gitObjectId(repository['parentCommit']) || + !gitObjectId(repository['parentTree']) || + budgets['elapsedMinutes'] !== 90 || + budgets['mechanicalInterventions'] !== 2 || + budgets['substantiveHumanInterventions'] !== 0 || + delivery['runtimeNetwork'] !== 'forbidden' || + delivery['dependencyInstallNetwork'] !== 'forbidden' || + acceptance['publicCommand'] !== '/brunch:land' || + acceptance['executionTerminal'] !== 'promotion_prepared' || + !nonemptyStrings(value['rules']) + ) { + invalid(); + } + return value as unknown as BrunchHostLandingExecutionCasePublicContract; +} + +const PETRINAUT_MECHANICAL_ADDRESS_KEYS = [ + 'skipTour', + 'dismissAssistant', + 'simulateMode', + 'optimizationsNav', + 'viewTitle', + 'create', + 'createDrawer', + 'scenario', + 'scenarioSelected', + 'metric', + 'metricCustomOption', + 'metricCode', + 'optimizationName', + 'directionMaximize', + 'directionMinimize', + 'run', + 'cancel', + 'statusComplete', + 'statusError', + 'statusCancelled', +] as const satisfies readonly PetrinautMechanicalAddressKey[]; + +const PETRINAUT_MECHANICAL_ADDRESSES = { + skipTour: { kind: 'roleName', role: 'button', name: 'Skip tour' }, + dismissAssistant: { kind: 'roleName', role: 'button', name: 'Dismiss' }, + simulateMode: { kind: 'roleName', role: 'radio', name: 'Simulate' }, + optimizationsNav: { kind: 'roleValue', role: 'radio', value: 'optimizations' }, + viewTitle: { kind: 'exactText', text: 'Optimizations' }, + create: { kind: 'roleName', role: 'button', name: 'Create' }, + createDrawer: { kind: 'roleName', role: 'dialog', name: 'Create an optimization' }, + scenario: { kind: 'roleContents', role: 'combobox', contents: 'Select a scenario' }, + scenarioSelected: { kind: 'roleContents', role: 'combobox', contents: 'Seasonal Flu' }, + metric: { kind: 'roleContents', role: 'combobox', contents: 'Select a metric' }, + metricCustomOption: { kind: 'roleName', role: 'option', name: 'Custom code' }, + metricCode: { kind: 'roleName', role: 'textbox', name: 'Editor content' }, + optimizationName: { kind: 'roleName', role: 'textbox', name: 'Name' }, + directionMaximize: { kind: 'roleName', role: 'radio', name: 'Maximize' }, + directionMinimize: { kind: 'roleName', role: 'radio', name: 'Minimize' }, + run: { kind: 'roleName', role: 'button', name: 'Run' }, + cancel: { kind: 'roleName', role: 'button', name: 'Cancel' }, + statusComplete: { kind: 'exactText', text: 'Complete' }, + statusError: { kind: 'exactText', text: 'Error' }, + statusCancelled: { kind: 'exactText', text: 'Cancelled' }, +} as const satisfies Record; + +function parsePetrinautOptimizationContract( + value: Record, +): PetrinautOptimizationExecutionCasePublicContract { + const caseValue = requiredRecord(value, 'case'); + const repository = requiredRecord(caseValue, 'repository'); + const budgets = requiredRecord(value, 'budgets'); + const delivery = requiredRecord(value, 'delivery'); + const acceptance = requiredRecord(value, 'acceptance'); + const mechanicalAddresses = requiredRecord(value, 'mechanicalAddresses'); + if ( + !exactKeys(value, [ + 'schemaVersion', + 'case', + 'budgets', + 'delivery', + 'acceptance', + 'mechanicalAddresses', + 'rules', + ]) || + !exactKeys(caseValue, [ + 'id', + 'specification', + 'specificationSha256', + 'provider', + 'model', + 'product', + 'mode', + 'scope', + 'surface', + 'repository', + ]) || + !exactKeys(repository, ['substrate', 'parentCommit', 'parentTree']) || + !exactKeys(budgets, ['elapsedMinutes', 'mechanicalInterventions', 'substantiveHumanInterventions']) || + !exactKeys(delivery, ['runtimeNetwork', 'dependencyInstallNetwork']) || + !exactKeys(acceptance, ['publicRoute', 'sameOriginApi', 'executionTerminal']) || + !exactKeys(mechanicalAddresses, PETRINAUT_MECHANICAL_ADDRESS_KEYS) || + value['schemaVersion'] !== 1 || + caseValue['id'] !== 'petrinaut-optimization-v1' || + caseValue['specification'] !== 'spec.md' || + !sha256HexValue(caseValue['specificationSha256']) || + caseValue['provider'] !== 'anthropic' || + caseValue['model'] !== 'claude-opus-4-8' || + caseValue['product'] !== 'petrinaut' || + caseValue['mode'] !== 'brownfield' || + caseValue['scope'] !== 'single_feature' || + caseValue['surface'] !== 'frontend' || + repository['substrate'] !== 'pinned_git' || + repository['parentCommit'] !== '5c7a2d9db5caa851c38938f4b1bac19005b0e978' || + repository['parentTree'] !== 'a3e08cf75e00cc9016c931f4665341506e03533e' || + budgets['elapsedMinutes'] !== 90 || + budgets['mechanicalInterventions'] !== 2 || + budgets['substantiveHumanInterventions'] !== 0 || + delivery['runtimeNetwork'] !== 'forbidden' || + delivery['dependencyInstallNetwork'] !== 'controller_only' || + acceptance['publicRoute'] !== '/optimization' || + acceptance['sameOriginApi'] !== '/api/petrinaut-opt/optimize/all' || + acceptance['executionTerminal'] !== 'promotion_prepared' || + !petrinautMechanicalAddresses(mechanicalAddresses) || + !nonemptyStrings(value['rules']) + ) { + invalid(); + } + return value as unknown as PetrinautOptimizationExecutionCasePublicContract; +} + +function petrinautMechanicalAddresses( + value: Record, +): value is Record { + return PETRINAUT_MECHANICAL_ADDRESS_KEYS.every((key) => + sameMechanicalAddress(value[key], PETRINAUT_MECHANICAL_ADDRESSES[key]), + ); +} + +function sameMechanicalAddress(value: unknown, expected: PetrinautMechanicalAddress): boolean { + if (!record(value) || value['kind'] !== expected.kind) return false; + switch (expected.kind) { + case 'roleName': + return value['role'] === expected.role && value['name'] === expected.name; + case 'roleValue': + return value['role'] === expected.role && value['value'] === expected.value; + case 'roleContents': + return value['role'] === expected.role && value['contents'] === expected.contents; + case 'exactText': + return value['text'] === expected.text; + } } function parseJson(raw: string): unknown { @@ -247,6 +565,16 @@ function sha256HexValue(value: unknown): value is string { return typeof value === 'string' && /^[a-f0-9]{64}$/u.test(value); } +function gitObjectId(value: unknown): value is string { + return typeof value === 'string' && /^[a-f0-9]{40}$/u.test(value); +} + +function exactKeys(value: Record, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(); + const wanted = [...expected].sort(); + return actual.length === wanted.length && actual.every((key, index) => key === wanted[index]); +} + function sha256Hex(value: string): string { return createHash('sha256').update(value).digest('hex'); } diff --git a/src/dev/execution-comparison/historical-replay-target.ts b/src/dev/execution-comparison/historical-replay-target.ts new file mode 100644 index 000000000..07f8c6b69 --- /dev/null +++ b/src/dev/execution-comparison/historical-replay-target.ts @@ -0,0 +1,420 @@ +import { copyFile, lstat, mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { isAbsolute, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { runCommand, type CommandResult, type CommandRunner } from '../../app/command-runner.js'; +import { + createBrunchExecutionLaunch, + type ExecutionLaunch, +} from '../end-to-end-comparison/brunch-adapter.js'; +import { createClaudeExecutionLaunch } from '../end-to-end-comparison/claude-adapter.js'; +import { materializeExactExecutionPacket } from '../end-to-end-comparison/public-packet.js'; +import { + createBrunchSolutionIsolationPolicy, + createClaudeSolutionIsolationPolicy, + materializePinnedSourceTree, + verifyPreparedHistoricalReplay, + type BrunchSolutionIsolationPolicy, + type ClaudeSolutionIsolationPolicy, +} from '../end-to-end-comparison/solution-isolation.js'; +import { assertControllerIsolation } from '../end-to-end-comparison/study-contract.js'; +import { seedBrownfieldBrunchExecutionWorkspace } from './brunch-lane.js'; +import { + isBrowserExecutionCaseContract, + type PinnedExecutionCaseId, + type PublicCasePacket, +} from './case-contract.js'; + +const RECIPE_VERSION = 1 as const; +const COMPARISON_GIT_IDENTITY = [ + '-c', + 'user.name=Brunch Comparison', + '-c', + 'user.email=brunch-comparison@invalid.local', +] as const; +const NO_DEPENDENCY_RECIPE = { recipe: 'none' } as const; +const PETRINAUT_DEPENDENCY_RECIPE = { + recipe: 'petrinaut-yarn-immutable-v1', + command: 'corepack', + args: ['yarn', 'install', '--immutable', '--mode=skip-build'], +} as const; +type HistoricalReplayDependencyRecipe = typeof NO_DEPENDENCY_RECIPE | typeof PETRINAUT_DEPENDENCY_RECIPE; +const DEPENDENCY_RECIPE_BY_PINNED_CASE = { + 'brunch-host-landing-v1': NO_DEPENDENCY_RECIPE, + 'petrinaut-optimization-v1': PETRINAUT_DEPENDENCY_RECIPE, +} as const satisfies Record; + +type HistoricalReplayPreparationPhase = + | 'source_materialization' + | 'packet_freeze' + | 'dependency_preparation' + | 'admission' + | 'lane_finalization'; + +interface PetrinautDependencyPreparationBase { + readonly recipe: typeof PETRINAUT_DEPENDENCY_RECIPE.recipe; + readonly command: typeof PETRINAUT_DEPENDENCY_RECIPE.command; + readonly args: typeof PETRINAUT_DEPENDENCY_RECIPE.args; + readonly exitCode: number; +} + +export interface PetrinautDependencyPreparationResult extends PetrinautDependencyPreparationBase { + readonly status: 'passed'; + readonly exitCode: 0; +} + +export interface PetrinautDependencyPreparationFailure extends PetrinautDependencyPreparationBase { + readonly status: 'failed'; + readonly failureStage: 'install' | 'tracked_source_cleanliness'; +} + +export type PetrinautDependencyPreparationOutcome = + | PetrinautDependencyPreparationResult + | PetrinautDependencyPreparationFailure; + +export interface PetrinautDependencyPreparationObservation { + readonly outcome: PetrinautDependencyPreparationOutcome; + readonly commandResult: CommandResult; + readonly trackedSourceStatus?: string; +} + +export class PetrinautDependencyPreparationError extends Error { + readonly outcome: PetrinautDependencyPreparationFailure; + readonly observation: PetrinautDependencyPreparationObservation; + + constructor( + outcome: PetrinautDependencyPreparationFailure, + observation: PetrinautDependencyPreparationObservation, + ) { + super( + outcome.failureStage === 'install' + ? 'compiled Petrinaut dependency install failed' + : 'compiled Petrinaut dependency install modified tracked source', + ); + this.name = 'PetrinautDependencyPreparationError'; + this.outcome = outcome; + this.observation = observation; + } +} + +interface HistoricalReplayReadyBase { + readonly status: 'ready'; + readonly recipeVersion: typeof RECIPE_VERSION; + readonly caseId: string; + readonly targetDir: string; + readonly sourceCommit: string; + readonly sourceTree: string; + readonly materializedCommit: string; + readonly baseSha: string; + readonly dependencyPreparation: + | { + readonly recipe: 'none'; + readonly status: 'not_required'; + } + | PetrinautDependencyPreparationResult; + readonly launch: ExecutionLaunch; +} + +export interface BrunchHistoricalReplayReady extends HistoricalReplayReadyBase { + readonly lane: 'brunch'; + readonly specId: number; + readonly isolationPolicy: BrunchSolutionIsolationPolicy; +} + +export interface ClaudeHistoricalReplayReady extends HistoricalReplayReadyBase { + readonly lane: 'claude_code'; + readonly isolationPolicy: ClaudeSolutionIsolationPolicy; +} + +export type HistoricalReplayReady = BrunchHistoricalReplayReady | ClaudeHistoricalReplayReady; + +export interface HistoricalReplayCaseSelection { + readonly caseDir: string; + readonly packet: PublicCasePacket; +} + +export interface HistoricalReplayTargetDependencies { + readonly runner?: CommandRunner; + readonly dependencyInstallRunner?: CommandRunner; + readonly onPetrinautDependencyPreparation?: ( + observation: PetrinautDependencyPreparationObservation, + ) => Promise | void; +} + +export class HistoricalReplayTargetPreparationError extends Error { + readonly status = 'setup_failed' as const; + readonly phase: HistoricalReplayPreparationPhase; + override readonly cause: unknown; + + constructor(phase: HistoricalReplayPreparationPhase, cause: unknown) { + super(`historical replay target preparation failed during ${phase}: ${errorMessage(cause)}`, { + cause, + }); + this.name = 'HistoricalReplayTargetPreparationError'; + this.phase = phase; + this.cause = cause; + } +} + +export async function prepareHistoricalReplayTarget( + input: { + readonly lane: HistoricalReplayReady['lane']; + readonly selectedCase: HistoricalReplayCaseSelection; + readonly sourceRepositoryDir: string; + readonly targetDir: string; + readonly controllerRoot: string; + readonly forbiddenRoots?: readonly string[]; + }, + dependencies: HistoricalReplayTargetDependencies = {}, +): Promise { + const runner = dependencies.runner ?? runCommand; + let phase: HistoricalReplayPreparationPhase = 'source_materialization'; + let targetOwned = false; + try { + await validateInput(input); + const contract = input.selectedCase.packet.contract; + if (isBrowserExecutionCaseContract(contract) || contract.case.repository.substrate !== 'pinned_git') { + throw new Error('historical replay preparation requires a pinned brownfield case'); + } + const forbiddenRoots = uniqueRoots([ + input.sourceRepositoryDir, + input.controllerRoot, + repositoryRoot(), + ...(input.forbiddenRoots ?? []), + ]); + const materialized = await materializePinnedSourceTree({ + sourceRepositoryDir: input.sourceRepositoryDir, + sourceCommit: contract.case.repository.parentCommit, + targetDir: input.targetDir, + runner, + }); + targetOwned = true; + if ( + materialized.sourceCommit !== contract.case.repository.parentCommit || + materialized.sourceTree !== contract.case.repository.parentTree + ) { + throw new Error('pinned source identity does not match the frozen execution contract'); + } + + phase = 'packet_freeze'; + const packetRoot = await mkdtemp(join(tmpdir(), 'brunch-historical-replay-packet-')); + let packetFiles: typeof input.selectedCase.packet.files; + try { + const packet = await materializeExactExecutionPacket({ + specificationPath: join( + input.selectedCase.caseDir, + input.selectedCase.packet.contract.case.specification, + ), + publicContractTemplatePath: join(input.selectedCase.caseDir, 'public-contract.json'), + packetDir: join(packetRoot, 'packet'), + }); + if (packet.packet.packetSha256 !== input.selectedCase.packet.packetSha256) { + throw new Error('exact execution packet drifted from the frozen selected case'); + } + packetFiles = packet.packet.files; + for (const file of packetFiles) { + await copyFile(join(packet.packetDir, file.path), join(input.targetDir, file.path)); + } + } finally { + await rm(packetRoot, { recursive: true, force: true }); + } + await gitChecked(runner, input.targetDir, ['add', '--', 'public-contract.json', 'spec.md']); + await gitChecked(runner, input.targetDir, [ + ...COMPARISON_GIT_IDENTITY, + 'commit', + '-m', + 'Freeze exact comparison handoff', + ]); + const baseSha = (await gitChecked(runner, input.targetDir, ['rev-parse', 'HEAD'])).stdout.trim(); + + phase = 'dependency_preparation'; + const dependencyRecipe = DEPENDENCY_RECIPE_BY_PINNED_CASE[contract.case.id]; + const dependencyPreparation = + dependencyRecipe.recipe === 'none' + ? ({ ...dependencyRecipe, status: 'not_required' } as const) + : await preparePetrinautHistoricalReplayDependencies({ + targetDir: input.targetDir, + runner, + dependencyInstallRunner: dependencies.dependencyInstallRunner ?? runCommand, + ...(dependencies.onPetrinautDependencyPreparation === undefined + ? {} + : { + onObservation: dependencies.onPetrinautDependencyPreparation, + }), + }); + + phase = 'admission'; + const claudePolicy = createClaudeSolutionIsolationPolicy(input.targetDir, forbiddenRoots); + const brunchPolicy = createBrunchSolutionIsolationPolicy(input.targetDir); + await verifyPreparedHistoricalReplay({ + prefix: { + ...materialized, + baseSha, + packetFiles, + }, + runner, + }); + + phase = 'lane_finalization'; + const common = { + status: 'ready' as const, + recipeVersion: RECIPE_VERSION, + caseId: contract.case.id, + targetDir: input.targetDir, + sourceCommit: materialized.sourceCommit, + sourceTree: materialized.sourceTree, + materializedCommit: materialized.syntheticCommit, + baseSha, + dependencyPreparation, + }; + if (input.lane === 'brunch') { + const seeded = await seedBrownfieldBrunchExecutionWorkspace({ workspaceDir: input.targetDir }); + await assertTrackedSourceClean(runner, input.targetDir, 'Brunch graph preparation'); + return { + ...common, + lane: 'brunch', + specId: seeded.specId, + isolationPolicy: brunchPolicy, + launch: createBrunchExecutionLaunch({ + workspaceDir: input.targetDir, + specId: seeded.specId, + }), + }; + } + return { + ...common, + lane: 'claude_code', + isolationPolicy: claudePolicy, + launch: createClaudeExecutionLaunch({ + workspaceDir: input.targetDir, + isolationPolicy: claudePolicy, + }), + }; + } catch (error) { + if (targetOwned) { + await rm(input.targetDir, { recursive: true, force: true }); + } + throw new HistoricalReplayTargetPreparationError(phase, error); + } +} + +export async function preparePetrinautHistoricalReplayDependencies(input: { + readonly targetDir: string; + readonly runner: CommandRunner; + readonly dependencyInstallRunner: CommandRunner; + readonly onObservation?: (observation: PetrinautDependencyPreparationObservation) => Promise | void; +}): Promise { + const result = await input.dependencyInstallRunner( + PETRINAUT_DEPENDENCY_RECIPE.command, + PETRINAUT_DEPENDENCY_RECIPE.args, + { + cwd: input.targetDir, + timeoutMs: 30 * 60_000, + maxOutputBytes: 256 * 1024, + }, + ); + if (result.exitCode !== 0) { + const outcome: PetrinautDependencyPreparationFailure = { + ...PETRINAUT_DEPENDENCY_RECIPE, + status: 'failed', + exitCode: result.exitCode, + failureStage: 'install', + }; + const observation = { outcome, commandResult: result }; + await input.onObservation?.(observation); + throw new PetrinautDependencyPreparationError(outcome, observation); + } + const trackedSourceStatus = await readTrackedSourceStatus(input.runner, input.targetDir); + if (trackedSourceStatus.length > 0) { + const outcome: PetrinautDependencyPreparationFailure = { + ...PETRINAUT_DEPENDENCY_RECIPE, + status: 'failed', + exitCode: result.exitCode, + failureStage: 'tracked_source_cleanliness', + }; + const observation = { + outcome, + commandResult: result, + trackedSourceStatus, + }; + await input.onObservation?.(observation); + throw new PetrinautDependencyPreparationError(outcome, observation); + } + const outcome: PetrinautDependencyPreparationResult = { + ...PETRINAUT_DEPENDENCY_RECIPE, + status: 'passed', + exitCode: 0, + }; + await input.onObservation?.({ outcome, commandResult: result }); + return outcome; +} + +async function assertTrackedSourceClean( + runner: CommandRunner, + targetDir: string, + owner: string, +): Promise { + const trackedStatus = await readTrackedSourceStatus(runner, targetDir); + if (trackedStatus.length > 0) { + throw new Error(`${owner} modified tracked source: ${trackedStatus}`); + } +} + +async function readTrackedSourceStatus(runner: CommandRunner, targetDir: string): Promise { + return ( + await gitChecked(runner, targetDir, ['status', '--porcelain', '--untracked-files=no']) + ).stdout.trim(); +} + +async function validateInput(input: { + readonly sourceRepositoryDir: string; + readonly targetDir: string; + readonly controllerRoot: string; +}): Promise { + if (!isAbsolute(input.sourceRepositoryDir)) { + throw new Error('pinned source repository must be an absolute path'); + } + if (!isAbsolute(input.targetDir)) { + throw new Error('historical replay target must be an absolute path'); + } + if (!isAbsolute(input.controllerRoot)) { + throw new Error('historical replay preparation requires an absolute controller root'); + } + const source = await lstat(input.sourceRepositoryDir); + if (!source.isDirectory() || source.isSymbolicLink()) { + throw new Error('pinned source repository must be a real directory, not a symlink'); + } + assertControllerIsolation({ + controllerRoot: input.controllerRoot, + targetRoots: [input.targetDir], + }); + assertControllerIsolation({ + controllerRoot: input.sourceRepositoryDir, + targetRoots: [input.targetDir], + }); +} + +async function gitChecked( + runner: CommandRunner, + cwd: string, + args: readonly string[], +): ReturnType { + const result = await runner('git', args, { cwd }); + if (result.exitCode !== 0) { + throw new Error(`git ${args.join(' ')} failed: ${result.stderr || result.stdout}`); + } + return result; +} + +function uniqueRoots(roots: readonly string[]): string[] { + return [...new Set(roots.map((root) => resolve(root)))]; +} + +function repositoryRoot(): string { + return fileURLToPath(new URL('../../../', import.meta.url)); +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} diff --git a/src/dev/execution-comparison/host-landing-oracle.ts b/src/dev/execution-comparison/host-landing-oracle.ts new file mode 100644 index 000000000..fce5bd17b --- /dev/null +++ b/src/dev/execution-comparison/host-landing-oracle.ts @@ -0,0 +1,8 @@ +export { evaluateHostLandingGitOutcome, snapshotGitState } from './host-landing-oracle/git-model.js'; +export { runBrunchHostLandingOracle } from './host-landing-oracle/runner.js'; +export type { + GitStateSnapshot, + HostLandingOracleCheck, + HostLandingOracleReport, + HostLandingScenario, +} from './host-landing-oracle/types.js'; diff --git a/src/dev/execution-comparison/host-landing-oracle/fixture.ts b/src/dev/execution-comparison/host-landing-oracle/fixture.ts new file mode 100644 index 000000000..dcbbb22da --- /dev/null +++ b/src/dev/execution-comparison/host-landing-oracle/fixture.ts @@ -0,0 +1,207 @@ +import { execFile, spawn } from 'node:child_process'; +import { mkdir, mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { promisify } from 'node:util'; + +import { SessionManager } from '@earendil-works/pi-coding-agent'; + +import { + HOST_LANDING_REVIEW_REF, + HOST_LANDING_RUN_ID, + type HostLandingFixture, + type HostLandingScenario, +} from './types.js'; + +const execFileAsync = promisify(execFile); +const GIT_IDENTITY = [ + '-c', + 'user.name=Brunch Oracle', + '-c', + 'user.email=brunch-oracle@invalid.local', +] as const; + +export async function createHostLandingFixture( + candidateRoot: string, + scenario: HostLandingScenario, + sessionMode: 'settled' | 'fresh', +): Promise { + const root = await mkdtemp(join(tmpdir(), 'brunch-host-landing-oracle-')); + const hostDir = join(root, 'host'); + await mkdir(hostDir); + await git(hostDir, ['init', '-q', '-b', 'main']); + await commitFile(hostDir, 'base.txt', 'base\n', 'host base'); + const hostBaseSha = await gitOutput(hostDir, ['rev-parse', 'HEAD']); + const greenfield = scenario === 'greenfield_success'; + const runRepoDir = greenfield ? join(root, 'greenfield-run') : join(root, 'run-worktree'); + if (greenfield) { + await mkdir(runRepoDir); + await git(runRepoDir, ['init', '-q', '-b', 'main']); + await git(runRepoDir, [...GIT_IDENTITY, 'commit', '--allow-empty', '-q', '-m', 'empty run base']); + } else { + await git(hostDir, ['worktree', 'add', '--quiet', '--detach', runRepoDir, hostBaseSha]); + } + const runBaseSha = await gitOutput(runRepoDir, ['rev-parse', 'HEAD']); + await commitFile(runRepoDir, 'src/a.ts', 'export const a = 1;\n', 'integrate slice a'); + await commitFile(runRepoDir, 'src/b.ts', 'export const b = 2;\n', 'integrate slice b'); + await commitFile(runRepoDir, 'src/c.ts', 'export const c = 3;\n', `promote ${HOST_LANDING_RUN_ID}`); + const completeReviewSha = await gitOutput(runRepoDir, ['rev-parse', 'HEAD']); + const canonicalExpectedTree = await gitOutput(runRepoDir, ['rev-parse', `${completeReviewSha}^{tree}`]); + let reviewSha = completeReviewSha; + if (scenario === 'final_commit_only') { + await git(runRepoDir, ['checkout', '--quiet', '--detach', runBaseSha]); + await commitFile(runRepoDir, 'src/c.ts', 'export const c = 3;\n', 'tip-only rival'); + reviewSha = await gitOutput(runRepoDir, ['rev-parse', 'HEAD']); + } + if (scenario === 'bookkeeping_retained') { + await commitFile(runRepoDir, '.brunch/leak.json', '{"leaked":true}\n', 'bookkeeping rival'); + reviewSha = await gitOutput(runRepoDir, ['rev-parse', 'HEAD']); + } + await git(runRepoDir, ['update-ref', `refs/heads/${HOST_LANDING_REVIEW_REF}`, reviewSha]); + if (scenario === 'dirty_host') await writeFile(join(hostDir, 'base.txt'), 'dirty\n'); + if (scenario === 'conflict') { + await commitFile(hostDir, 'src/a.ts', 'export const a = 999;\n', 'host conflict'); + } + const promotionPath = join(hostDir, '.brunch', 'cook', 'runs', HOST_LANDING_RUN_ID, 'promotion.json'); + const metadataPath = join(hostDir, '.brunch', 'cook', 'runs', HOST_LANDING_RUN_ID, 'run.json'); + await mkdir(dirname(metadataPath), { recursive: true }); + await writeFile( + promotionPath, + `${JSON.stringify({ + runId: HOST_LANDING_RUN_ID, + specId: '1', + promotion: { + status: 'promoted', + commitSha: reviewSha, + reviewBranch: HOST_LANDING_REVIEW_REF, + }, + })}\n`, + ); + await writeFile( + metadataPath, + `${JSON.stringify({ + runId: HOST_LANDING_RUN_ID, + specId: '1', + planPath: join(root, 'plan.json'), + status: 'promotion_prepared', + substrate: greenfield ? 'empty_dir' : 'git_worktree', + worktreeDir: runRepoDir, + runBaseSha, + promotionPath, + promotionCommitSha: reviewSha, + promotionBranch: HOST_LANDING_REVIEW_REF, + })}\n`, + ); + const sessionFile = await createSessionFixture(candidateRoot, hostDir, sessionMode); + return { + root, + hostDir, + runRepoDir, + ...(greenfield ? { targetDir: join(root, 'materialized-target') } : {}), + runBaseSha, + reviewSha, + canonicalExpectedTree, + metadataPath, + sessionFile, + }; +} + +export async function advanceHostLandingReviewRef(fixture: HostLandingFixture): Promise { + await commitFile(fixture.runRepoDir, 'late.txt', 'late\n', 'late rival commit'); + await git(fixture.runRepoDir, [ + 'update-ref', + `refs/heads/${HOST_LANDING_REVIEW_REF}`, + await gitOutput(fixture.runRepoDir, ['rev-parse', 'HEAD']), + fixture.reviewSha, + ]); +} + +async function createSessionFixture( + candidateRoot: string, + cwd: string, + mode: 'settled' | 'fresh', +): Promise { + const responses = await runCandidateRpc(candidateRoot, cwd, [ + { + jsonrpc: '2.0', + id: 1, + method: 'workspace.activate', + params: { decision: { action: 'newSpec', title: 'Host landing oracle' } }, + }, + ]); + const result = responses.find((response) => response['id'] === 1)?.['result']; + if (!record(result) || !record(result['session']) || typeof result['session']['file'] !== 'string') { + throw new Error( + `candidate public RPC did not create a settled-session fixture: ${JSON.stringify(responses)}`, + ); + } + const sessionFile = result['session']['file']; + if (mode === 'fresh') return sessionFile; + const manager = SessionManager.open(sessionFile, dirname(sessionFile), cwd); + manager.appendMessage({ + role: 'assistant', + content: [{ type: 'text', text: 'Settled controller session.' }], + api: 'brunch-oracle', + provider: 'controller', + model: 'none', + usage: { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: 'stop', + timestamp: Date.now(), + }); + return sessionFile; +} + +async function runCandidateRpc( + candidateRoot: string, + cwd: string, + requests: readonly Record[], +): Promise[]> { + const child = spawn( + process.execPath, + [join(candidateRoot, 'bin', 'brunch.js'), '--cwd', cwd, '--mode', 'rpc'], + { + cwd: candidateRoot, + env: { ...process.env, PI_OFFLINE: '1', PI_SKIP_VERSION_CHECK: '1' }, + stdio: ['pipe', 'pipe', 'pipe'], + }, + ); + let stdout = ''; + let stderr = ''; + child.stdout.setEncoding('utf8').on('data', (chunk) => (stdout += chunk)); + child.stderr.setEncoding('utf8').on('data', (chunk) => (stderr += chunk)); + child.stdin.end(`${requests.map((request) => JSON.stringify(request)).join('\n')}\n`); + const exitCode = await new Promise((resolveExit) => child.on('close', resolveExit)); + if (exitCode !== 0) throw new Error(`candidate public RPC failed: ${stderr || stdout}`); + return stdout + .split('\n') + .filter(Boolean) + .map((line) => JSON.parse(line) as Record); +} + +async function commitFile(cwd: string, path: string, content: string, subject: string): Promise { + const selected = join(cwd, path); + await mkdir(dirname(selected), { recursive: true }); + await writeFile(selected, content); + await git(cwd, ['add', '--', path]); + await git(cwd, [...GIT_IDENTITY, 'commit', '-q', '-m', subject]); +} + +async function git(cwd: string, args: readonly string[]): Promise { + await execFileAsync('git', [...args], { cwd }); +} + +async function gitOutput(cwd: string, args: readonly string[]): Promise { + const result = await execFileAsync('git', [...args], { cwd }); + return result.stdout.trim(); +} + +function record(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} diff --git a/src/dev/execution-comparison/host-landing-oracle/git-model.ts b/src/dev/execution-comparison/host-landing-oracle/git-model.ts new file mode 100644 index 000000000..c677596b7 --- /dev/null +++ b/src/dev/execution-comparison/host-landing-oracle/git-model.ts @@ -0,0 +1,155 @@ +import { execFile } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { readFile } from 'node:fs/promises'; +import { promisify } from 'node:util'; + +import type { + GitStateSnapshot, + HostLandingOracleCheck, + HostLandingOracleReport, + HostLandingScenario, +} from './types.js'; + +const execFileAsync = promisify(execFile); +const REQUIRED_CONTENT_PATHS = ['src/a.ts', 'src/b.ts', 'src/c.ts'] as const; + +export async function evaluateHostLandingGitOutcome(input: { + readonly scenario: HostLandingScenario; + readonly hostDir: string; + readonly targetDir?: string; + readonly metadataPath: string; + readonly canonicalExpectedTree: string; + readonly before: GitStateSnapshot; + readonly preConfirm: GitStateSnapshot; + readonly terminalEvidence?: readonly string[]; + readonly providerActivity?: boolean; +}): Promise> { + const destination = input.targetDir ?? input.hostDir; + const after = await snapshotGitState(input.hostDir, input.metadataPath); + const destinationExists = await gitSucceeds(destination, ['rev-parse', '--git-dir']); + const actualTree = destinationExists ? await gitOutput(destination, ['rev-parse', 'HEAD^{tree}']) : ''; + const changedPaths = destinationExists + ? (await gitOutput(destination, ['ls-tree', '-r', '--name-only', 'HEAD'])).split('\n').filter(Boolean) + : []; + const metadata = parseRecord(after.runMetadataBytes); + const preflightPassed = + sameSnapshot(input.before, input.preConfirm) && + input.providerActivity !== true && + (input.terminalEvidence ?? []).some( + (line) => line.includes('complete') || line.includes('Nothing changed'), + ); + const refusal = ['decline', 'dirty_host', 'conflict', 'stale_acceptance'].includes(input.scenario); + const greenfield = input.scenario === 'greenfield_success'; + const fullRangePassed = + !refusal && + !greenfield && + actualTree === input.canonicalExpectedTree && + REQUIRED_CONTENT_PATHS.every((path) => changedPaths.includes(path)) && + !changedPaths.some((path) => path === '.brunch' || path.startsWith('.brunch/')) && + metadata['status'] === 'landed'; + const materializationPassed = + greenfield && + actualTree === input.canonicalExpectedTree && + (await gitOutput(destination, ['rev-list', '--count', 'HEAD'])) === '1' && + (await gitOutput(destination, ['log', '-1', '--format=%an <%ae>'])) === 'brunch ' && + !changedPaths.some((path) => path === '.brunch' || path.startsWith('.brunch/')) && + metadata['status'] === 'landed'; + const refusalPassed = + !refusal || (sameSnapshot(input.before, after) && metadata['status'] === 'promotion_prepared'); + const checks: HostLandingOracleCheck[] = [ + { + id: 'public-tui-preflight', + claims: ['AC1', 'INV1', 'REQ1'], + status: preflightPassed ? 'passed' : 'failed', + evidence: [ + `pre-confirm snapshot ${sameSnapshot(input.before, input.preConfirm) ? 'unchanged' : 'changed'}`, + `provider activity ${input.providerActivity === true ? 'observed' : 'absent'}`, + ], + }, + { + id: 'brownfield-full-range', + claims: ['REQ2'], + status: refusal || greenfield ? 'passed' : fullRangePassed ? 'passed' : 'failed', + evidence: [ + `expected tree ${input.canonicalExpectedTree}`, + `actual tree ${actualTree || '(missing)'}`, + `tracked paths ${changedPaths.join(', ') || '(none)'}`, + ], + }, + { + id: 'greenfield-materialization', + claims: ['REQ3'], + status: greenfield ? (materializationPassed ? 'passed' : 'failed') : 'passed', + evidence: [greenfield ? `materialized tree ${actualTree || '(missing)'}` : 'not selected'], + }, + { + id: 'refusal-safety', + claims: ['REQ4'], + status: refusalPassed ? 'passed' : 'failed', + evidence: [ + refusal ? `host ${sameSnapshot(input.before, after) ? 'unchanged' : 'changed'}` : 'not selected', + ], + }, + ]; + const passed = checks.every(({ status }) => status === 'passed'); + return { + status: passed ? 'passed' : 'assertion_failed', + checks, + terminalEvidence: input.terminalEvidence ?? [], + gitEvidence: { + before: input.before, + preConfirm: input.preConfirm, + after, + expectedTree: input.canonicalExpectedTree, + actualTree, + changedPaths, + }, + }; +} + +export async function snapshotGitState(cwd: string, metadataPath: string): Promise { + const runMetadataBytes = await readFile(metadataPath, 'utf8'); + return { + head: await gitOutput(cwd, ['rev-parse', 'HEAD']), + tree: await gitOutput(cwd, ['rev-parse', 'HEAD^{tree}']), + status: await gitOutput(cwd, ['status', '--porcelain=v1', '--untracked-files=no']), + runMetadataSha256: createHash('sha256').update(runMetadataBytes).digest('hex'), + runMetadataBytes, + }; +} + +export function emptyGitStateSnapshot(): GitStateSnapshot { + return { head: '', tree: '', status: '', runMetadataSha256: '', runMetadataBytes: '' }; +} + +function sameSnapshot(left: GitStateSnapshot, right: GitStateSnapshot): boolean { + return ( + left.head === right.head && + left.tree === right.tree && + left.status === right.status && + left.runMetadataSha256 === right.runMetadataSha256 && + left.runMetadataBytes === right.runMetadataBytes + ); +} + +async function gitOutput(cwd: string, args: readonly string[]): Promise { + const result = await execFileAsync('git', [...args], { cwd }); + return result.stdout.trim(); +} + +async function gitSucceeds(cwd: string, args: readonly string[]): Promise { + try { + await execFileAsync('git', [...args], { cwd }); + return true; + } catch { + return false; + } +} + +function parseRecord(raw: string): Record { + const value = JSON.parse(raw) as unknown; + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + throw new Error('run metadata must be an object'); + } + return value as Record; +} diff --git a/src/dev/execution-comparison/host-landing-oracle/runner.ts b/src/dev/execution-comparison/host-landing-oracle/runner.ts new file mode 100644 index 000000000..cbb059ab7 --- /dev/null +++ b/src/dev/execution-comparison/host-landing-oracle/runner.ts @@ -0,0 +1,179 @@ +import { randomUUID } from 'node:crypto'; +import { readFile, rm } from 'node:fs/promises'; +import { join, resolve } from 'node:path'; + +import { + removeSession, + sendKeys, + sendText, + sessionStatus, + startSession, + stopSession, + waitForScreenText, +} from '../../tui-driver.js'; +import { advanceHostLandingReviewRef, createHostLandingFixture } from './fixture.js'; +import { emptyGitStateSnapshot, evaluateHostLandingGitOutcome, snapshotGitState } from './git-model.js'; +import { + HOST_LANDING_CASE_ID, + HOST_LANDING_ORACLE_ID, + HOST_LANDING_RUN_ID, + type HostLandingFixture, + type HostLandingOracleReport, + type HostLandingScenario, +} from './types.js'; + +export async function runBrunchHostLandingOracle(input: { + readonly candidateRoot: string; + readonly scenario?: HostLandingScenario; + readonly sessionMode?: 'settled' | 'fresh'; + readonly keepFixture?: boolean; +}): Promise { + const scenario = input.scenario ?? 'brownfield_success'; + const candidateRoot = resolve(input.candidateRoot); + let fixture: HostLandingFixture | undefined; + try { + fixture = await createHostLandingFixture(candidateRoot, scenario, input.sessionMode ?? 'settled'); + return await driveCandidateTui({ candidateRoot, fixture, scenario }); + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + const empty = emptyGitStateSnapshot(); + return { + schemaVersion: 1, + caseId: HOST_LANDING_CASE_ID, + oracleId: HOST_LANDING_ORACLE_ID, + status: 'setup_failed', + scenario, + checks: [], + terminalEvidence: [], + gitEvidence: { + before: empty, + preConfirm: empty, + after: empty, + expectedTree: '', + actualTree: '', + changedPaths: [], + }, + setupFailure: detail, + }; + } finally { + if (fixture && !input.keepFixture) await rm(fixture.root, { recursive: true, force: true }); + } +} + +async function driveCandidateTui(input: { + readonly candidateRoot: string; + readonly fixture: HostLandingFixture; + readonly scenario: HostLandingScenario; +}): Promise { + if (!input.fixture.sessionFile) throw new Error('controller supplied no settled session'); + const sessionBefore = await readFile(input.fixture.sessionFile, 'utf8'); + if (!sessionBefore.includes('"role":"assistant"') && !sessionBefore.includes('"role": "assistant"')) { + throw new Error('controller session fixture is not settled'); + } + const before = await snapshotGitState(input.fixture.hostDir, input.fixture.metadataPath); + const name = `host-land-${randomUUID()}`; + const command = [ + '/usr/bin/env', + 'PI_OFFLINE=1', + 'PI_SKIP_VERSION_CHECK=1', + process.execPath, + join(input.candidateRoot, 'bin', 'brunch.js'), + '--cwd', + input.fixture.hostDir, + '--no-webui', + ]; + await startSession({ name, command, cols: 120, rows: 40, cwd: input.candidateRoot }); + const terminalEvidence: string[] = []; + try { + terminalEvidence.push(...(await wait(name, 'Continue your latest spec and session'))); + sendKeys(name, ['Up', 'Up', 'Up', 'Enter']); + await wait(name, 'What does this specification own?'); + sendKeys(name, ['Down', 'Enter']); + await wait(name, 'Does this build on the existing code here?'); + sendKeys(name, ['Enter']); + const settledOrModeChoice = await wait( + name, + /Choose how Specify mode should work|Settled controller session\./u, + ); + if (settledOrModeChoice.join('\n').includes('Choose how Specify mode should work')) { + sendKeys(name, ['Esc']); + terminalEvidence.push(...(await wait(name, 'Settled controller session.'))); + } else { + terminalEvidence.push(...settledOrModeChoice); + } + const commandText = + input.scenario === 'greenfield_success' + ? `/brunch:land ${HOST_LANDING_RUN_ID} ${input.fixture.targetDir}` + : `/brunch:land ${HOST_LANDING_RUN_ID}`; + terminalEvidence.push(`controller invoked ${commandText}`); + sendText(name, commandText); + sendKeys(name, ['Enter']); + const expectsConfirmation = !['dirty_host', 'conflict'].includes(input.scenario); + if (expectsConfirmation) { + terminalEvidence.push(...(await wait(name, 'Proceed with this host mutation?'))); + } else { + terminalEvidence.push(...(await wait(name, 'Nothing changed'))); + } + const preConfirm = await snapshotGitState(input.fixture.hostDir, input.fixture.metadataPath); + if (input.scenario === 'stale_acceptance') { + await advanceHostLandingReviewRef(input.fixture); + } + if (expectsConfirmation) { + if (input.scenario === 'decline') sendKeys(name, ['Down', 'Enter']); + else sendKeys(name, ['Enter']); + const terminal = await wait( + name, + input.scenario === 'decline' + ? 'declined; nothing changed' + : input.scenario === 'stale_acceptance' + ? 'ref_moved' + : `Landed ${HOST_LANDING_RUN_ID}`, + ); + terminalEvidence.push(...terminal); + } + const sessionAfter = await readFile(input.fixture.sessionFile, 'utf8'); + const providerActivity = + messageCount(sessionAfter) !== messageCount(sessionBefore) || + sessionAfter.includes('"customType":"brunch.kick"') || + sessionAfter.includes('"customType": "brunch.kick"'); + const evaluated = await evaluateHostLandingGitOutcome({ + scenario: input.scenario, + hostDir: input.fixture.hostDir, + ...(input.fixture.targetDir ? { targetDir: input.fixture.targetDir } : {}), + metadataPath: input.fixture.metadataPath, + canonicalExpectedTree: input.fixture.canonicalExpectedTree, + before, + preConfirm, + terminalEvidence, + providerActivity, + }); + return { + schemaVersion: 1, + caseId: HOST_LANDING_CASE_ID, + oracleId: HOST_LANDING_ORACLE_ID, + scenario: input.scenario, + ...evaluated, + }; + } finally { + await stopSession(name); + removeSession(name, { force: true }); + } +} + +async function wait(name: string, text: string | RegExp): Promise { + const status = sessionStatus(name); + if (!status) throw new Error(`TUI driver session ${name} disappeared`); + const result = await waitForScreenText(status.logPath, status.cols, status.rows, text, { + timeoutMs: 30_000, + }); + if (!result.matched) { + throw new Error(`candidate TUI did not render ${JSON.stringify(text)}\n${result.screen.join('\n')}`); + } + return result.screen; +} + +function messageCount(raw: string): number { + return raw + .split('\n') + .filter((line) => line.includes('"type":"message"') || line.includes('"type": "message"')).length; +} diff --git a/src/dev/execution-comparison/host-landing-oracle/types.ts b/src/dev/execution-comparison/host-landing-oracle/types.ts new file mode 100644 index 000000000..20ff5bfa2 --- /dev/null +++ b/src/dev/execution-comparison/host-landing-oracle/types.ts @@ -0,0 +1,64 @@ +export const HOST_LANDING_CASE_ID = 'brunch-host-landing-v1' as const; +export const HOST_LANDING_ORACLE_ID = 'brunch-host-landing-oracles-v1' as const; +export const HOST_LANDING_RUN_ID = 'run-1'; +export const HOST_LANDING_REVIEW_REF = `brunch/review/${HOST_LANDING_RUN_ID}`; + +export type HostLandingScenario = + | 'brownfield_success' + | 'greenfield_success' + | 'decline' + | 'dirty_host' + | 'conflict' + | 'stale_acceptance' + | 'final_commit_only' + | 'bookkeeping_retained'; + +export interface GitStateSnapshot { + readonly head: string; + readonly tree: string; + readonly status: string; + readonly runMetadataSha256: string; + readonly runMetadataBytes: string; +} + +export interface HostLandingOracleCheck { + readonly id: + | 'public-tui-preflight' + | 'brownfield-full-range' + | 'greenfield-materialization' + | 'refusal-safety'; + readonly claims: readonly string[]; + readonly status: 'passed' | 'failed'; + readonly evidence: readonly string[]; +} + +export interface HostLandingOracleReport { + readonly schemaVersion: 1; + readonly caseId: typeof HOST_LANDING_CASE_ID; + readonly oracleId: typeof HOST_LANDING_ORACLE_ID; + readonly status: 'passed' | 'assertion_failed' | 'setup_failed'; + readonly scenario: HostLandingScenario; + readonly checks: readonly HostLandingOracleCheck[]; + readonly terminalEvidence: readonly string[]; + readonly gitEvidence: { + readonly before: GitStateSnapshot; + readonly preConfirm: GitStateSnapshot; + readonly after: GitStateSnapshot; + readonly expectedTree: string; + readonly actualTree: string; + readonly changedPaths: readonly string[]; + }; + readonly setupFailure?: string; +} + +export interface HostLandingFixture { + readonly root: string; + readonly hostDir: string; + readonly runRepoDir: string; + readonly targetDir?: string; + readonly runBaseSha: string; + readonly reviewSha: string; + readonly canonicalExpectedTree: string; + readonly metadataPath: string; + readonly sessionFile: string; +} diff --git a/src/dev/execution-comparison/operator-cli.ts b/src/dev/execution-comparison/operator-cli.ts index fd70f80cb..e4b20b887 100644 --- a/src/dev/execution-comparison/operator-cli.ts +++ b/src/dev/execution-comparison/operator-cli.ts @@ -4,7 +4,16 @@ import { isAbsolute, join } from 'node:path'; import { promisify } from 'node:util'; import { prepareBrunchExecutionWorkspace } from './brunch-lane.js'; -import { loadPublicCasePacket, type PublicCasePacket } from './case-contract.js'; +import { + isBrowserExecutionCaseContract, + loadPublicCasePacket, + type PublicCasePacket, +} from './case-contract.js'; +import { + prepareHistoricalReplayTarget, + type HistoricalReplayReady, + type HistoricalReplayTargetDependencies, +} from './historical-replay-target.js'; const execFileAsync = promisify(execFile); const SAFE_CASE_ID = /^[a-z0-9][a-z0-9-]*$/u; @@ -27,15 +36,21 @@ export interface ResolvedExecutionCase extends ExecutionCaseSummary { export type PreparedExecutionTarget = | (ResolvedExecutionCase & { + readonly preparation: 'legacy_brunch'; readonly lane: 'brunch'; readonly targetDir: string; readonly specId: number; }) | (ResolvedExecutionCase & { + readonly preparation: 'empty_git'; readonly lane: 'claude_code'; readonly targetDir: string; readonly baseSha: string; - }); + }) + | (ResolvedExecutionCase & + HistoricalReplayReady & { + readonly preparation: 'historical_replay'; + }); export async function listExecutionCases(casesRoot: string): Promise { const entries = await readdir(casesRoot, { withFileTypes: true }); @@ -77,13 +92,49 @@ export async function resolveExecutionCase( }; } -export async function prepareExecutionTarget(input: { - readonly lane: 'brunch' | 'claude_code'; - readonly caseReference: string; - readonly casesRoot: string; - readonly targetDir: string; -}): Promise { +export async function prepareExecutionTarget( + input: { + readonly lane: 'brunch' | 'claude_code'; + readonly caseReference: string; + readonly casesRoot: string; + readonly targetDir: string; + readonly controllerRoot?: string; + readonly sourceRepositoryDir?: string; + }, + dependencies: HistoricalReplayTargetDependencies = {}, +): Promise { const selected = await resolveExecutionCase(input.caseReference, input.casesRoot); + const contract = selected.packet.contract; + if (!isBrowserExecutionCaseContract(contract) && contract.case.repository.substrate === 'pinned_git') { + if (input.sourceRepositoryDir === undefined) { + throw new Error('pinned execution case requires --source-repository'); + } + if (input.controllerRoot === undefined) { + throw new Error('pinned execution case requires an absolute controller root'); + } + const prepared = await prepareHistoricalReplayTarget( + { + lane: input.lane, + selectedCase: selected, + sourceRepositoryDir: input.sourceRepositoryDir, + targetDir: input.targetDir, + controllerRoot: input.controllerRoot, + }, + dependencies, + ); + return { + ...selected, + ...prepared, + preparation: 'historical_replay', + }; + } + if ( + input.sourceRepositoryDir !== undefined || + dependencies.runner !== undefined || + dependencies.dependencyInstallRunner !== undefined + ) { + throw new Error('--source-repository is valid only for pinned execution cases'); + } if (input.lane === 'brunch') { const prepared = await prepareBrunchExecutionWorkspace({ workspaceDir: input.targetDir, @@ -91,6 +142,7 @@ export async function prepareExecutionTarget(input: { }); return { ...selected, + preparation: 'legacy_brunch', lane: 'brunch', targetDir: input.targetDir, specId: prepared.specId, @@ -107,6 +159,7 @@ export async function prepareExecutionTarget(input: { const baseSha = await gitOutput(['rev-parse', 'HEAD'], input.targetDir); return { ...selected, + preparation: 'empty_git', lane: 'claude_code', targetDir: input.targetDir, baseSha, diff --git a/src/dev/execution-comparison/oracle-pack.ts b/src/dev/execution-comparison/oracle-pack.ts index ea6d0e59d..be7ef6273 100644 --- a/src/dev/execution-comparison/oracle-pack.ts +++ b/src/dev/execution-comparison/oracle-pack.ts @@ -2,7 +2,7 @@ import { createHash } from 'node:crypto'; import { readdir, readFile } from 'node:fs/promises'; import { basename, join, relative, sep } from 'node:path'; -export interface ControllerOracleManifest { +export interface PetriControllerOracleManifest { readonly schemaVersion: 1; readonly id: 'minimal-petri-net-editor-oracles-v2'; readonly publicCaseId: 'minimal-petri-net-editor-v1'; @@ -18,6 +18,58 @@ export interface ControllerOracleManifest { readonly catastrophicVisualGate: readonly string[]; } +export interface BrunchHostLandingControllerOracleManifest { + readonly schemaVersion: 1; + readonly id: 'brunch-host-landing-oracles-v1'; + readonly publicCaseId: 'brunch-host-landing-v1'; + readonly runnerVersion: 'brunch-host-landing-v1'; + readonly referenceModelVersion: 'git-full-range-v1'; + readonly checks: readonly { + readonly id: string; + readonly claims: readonly string[]; + }[]; + readonly validityRules: readonly string[]; + readonly replacementRule: string; +} + +export interface PetrinautOptimizationControllerOracleManifest { + readonly schemaVersion: 1; + readonly id: 'petrinaut-optimization-oracles-v1'; + readonly publicCaseId: 'petrinaut-optimization-v1'; + readonly runnerVersion: 'petrinaut-optimization-browser-v1'; + readonly fixtureVersion: 'deterministic-optimizer-v1'; + readonly checks: readonly { + readonly id: + | 'route-and-accessibility' + | 'scenario-configuration' + | 'request-contract' + | 'progress-and-completion' + | 'service-error' + | 'cancel-and-abort' + | 'private-origin-secrecy'; + readonly claims: readonly string[]; + }[]; + readonly validityRules: readonly string[]; + readonly replacementRule: string; +} + +export type ControllerOracleManifest = + | PetriControllerOracleManifest + | BrunchHostLandingControllerOracleManifest + | PetrinautOptimizationControllerOracleManifest; + +export function isPetriControllerOracleManifest( + value: ControllerOracleManifest, +): value is PetriControllerOracleManifest { + return value.id === 'minimal-petri-net-editor-oracles-v2'; +} + +export function isPetrinautOptimizationControllerOracleManifest( + value: ControllerOracleManifest, +): value is PetrinautOptimizationControllerOracleManifest { + return value.id === 'petrinaut-optimization-oracles-v1'; +} + export interface ControllerOraclePack { readonly manifest: ControllerOracleManifest; readonly files: readonly { @@ -72,6 +124,32 @@ export async function loadControllerOracleManifest(caseDir: string): Promise claims))].sort(codePointCompare); +} + +export function assertOracleClaimCoverage( + manifest: ControllerOracleManifest, + requirementClaimIds: readonly string[], +): void { + const assigned = oracleManifestClaimIds(manifest); + const required = [...new Set(requirementClaimIds)].sort(codePointCompare); + if (assigned.length !== required.length || assigned.some((claim, index) => claim !== required[index])) { + throw new Error('controller oracle manifest does not assign every requirement claim exactly'); + } +} + +function parsePetriManifest(value: Record): PetriControllerOracleManifest { const journeys = value['journeys']; if ( value['schemaVersion'] !== 1 || @@ -94,7 +172,109 @@ export function parseControllerOracleManifest(value: unknown): ControllerOracleM ) { invalid(); } - return value as unknown as ControllerOracleManifest; + return value as unknown as PetriControllerOracleManifest; +} + +function parseBrunchHostLandingManifest( + value: Record, +): BrunchHostLandingControllerOracleManifest { + const checks = value['checks']; + if ( + !exactKeys(value, [ + 'schemaVersion', + 'id', + 'publicCaseId', + 'runnerVersion', + 'referenceModelVersion', + 'checks', + 'validityRules', + 'replacementRule', + ]) || + value['schemaVersion'] !== 1 || + value['id'] !== 'brunch-host-landing-oracles-v1' || + value['publicCaseId'] !== 'brunch-host-landing-v1' || + value['runnerVersion'] !== 'brunch-host-landing-v1' || + value['referenceModelVersion'] !== 'git-full-range-v1' || + !Array.isArray(checks) || + checks.length === 0 || + !checks.every( + (check) => + record(check) && + exactKeys(check, ['id', 'claims']) && + nonempty(check['id']) && + nonemptyStrings(check['claims']), + ) || + new Set(checks.map((check) => (check as { id: string }).id)).size !== checks.length || + !nonemptyStrings(value['validityRules']) || + !(value['validityRules'] as string[]).join('\n').includes('promotion_prepared') || + !(value['validityRules'] as string[]).join('\n').includes('landed') || + !nonempty(value['replacementRule']) + ) { + invalid(); + } + return value as unknown as BrunchHostLandingControllerOracleManifest; +} + +const PETRINAUT_OPTIMIZATION_CHECKS = [ + 'route-and-accessibility', + 'scenario-configuration', + 'request-contract', + 'progress-and-completion', + 'service-error', + 'cancel-and-abort', + 'private-origin-secrecy', +] as const; + +function parsePetrinautOptimizationManifest( + value: Record, +): PetrinautOptimizationControllerOracleManifest { + const checks = value['checks']; + if ( + !exactKeys(value, [ + 'schemaVersion', + 'id', + 'publicCaseId', + 'runnerVersion', + 'fixtureVersion', + 'checks', + 'validityRules', + 'replacementRule', + ]) || + value['schemaVersion'] !== 1 || + value['id'] !== 'petrinaut-optimization-oracles-v1' || + value['publicCaseId'] !== 'petrinaut-optimization-v1' || + value['runnerVersion'] !== 'petrinaut-optimization-browser-v1' || + value['fixtureVersion'] !== 'deterministic-optimizer-v1' || + !Array.isArray(checks) || + checks.length !== PETRINAUT_OPTIMIZATION_CHECKS.length || + !checks.every( + (check, index) => + record(check) && + exactKeys(check, ['id', 'claims']) && + check['id'] === PETRINAUT_OPTIMIZATION_CHECKS[index] && + nonemptyStrings(check['claims']), + ) || + !nonemptyStrings(value['validityRules']) || + !petrinautValidityRules(value['validityRules'] as string[]) || + !nonempty(value['replacementRule']) + ) { + invalid(); + } + return value as unknown as PetrinautOptimizationControllerOracleManifest; +} + +function petrinautValidityRules(rules: readonly string[]): boolean { + const joined = rules.join('\n'); + return ( + joined.includes('Before candidate execution') && + joined.includes('compiled immutable dependency preparation') && + joined.includes('package-registry network') && + joined.includes('network-denied candidate execution lane') && + joined.includes('promotion_prepared') && + joined.includes('After lane termination') && + joined.includes('compiled focused build preparation') && + joined.includes('/optimization') + ); } async function listFiles(root: string): Promise { @@ -142,6 +322,12 @@ function record(value: unknown): value is Record { return typeof value === 'object' && value !== null && !Array.isArray(value); } +function exactKeys(value: Record, expected: readonly string[]): boolean { + const actual = Object.keys(value).sort(codePointCompare); + const wanted = [...expected].sort(codePointCompare); + return actual.length === wanted.length && actual.every((key, index) => key === wanted[index]); +} + function invalid(): never { throw new Error('invalid fixed controller oracle manifest'); } diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle.ts new file mode 100644 index 000000000..96a945468 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle.ts @@ -0,0 +1,10 @@ +export { + PETRINAUT_FOCUSED_PREPARATION, + runPetrinautOptimizationOracle, +} from './petrinaut-optimization-oracle/runner.js'; +export type { PetrinautOraclePreparationObservation } from './petrinaut-optimization-oracle/runner.js'; +export type { + PetrinautOptimizationCheckId, + PetrinautOptimizationOracleCheck, + PetrinautOptimizationOracleReport, +} from './petrinaut-optimization-oracle/types.js'; diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/browser.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/browser.ts new file mode 100644 index 000000000..c6d285666 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/browser.ts @@ -0,0 +1,622 @@ +import { spawn, type ChildProcess } from 'node:child_process'; +import { access, readFile } from 'node:fs/promises'; +import { createServer } from 'node:http'; +import { fileURLToPath } from 'node:url'; + +import { chromium, type Browser, type Locator, type Page } from 'playwright-core'; + +import type { + PetrinautMechanicalAddress, + PetrinautOptimizationExecutionCasePublicContract, +} from '../case-contract.js'; +import type { PetrinautOptimizationControllerOracleManifest } from '../oracle-pack.js'; +import { requirePetrinautFocusedObservation } from './claims.js'; +import { startDeterministicFakeOptimizer } from './fake-optimizer.js'; +import type { PetrinautOptimizationOracleCheck } from './types.js'; + +const SEMANTIC_ACTION_TIMEOUT_MS = 5_000; +const NAVIGATION_TIMEOUT_MS = 30_000; +const CALIBRATION_SEED_PATH = fileURLToPath(new URL('./calibration-seed.json', import.meta.url)); + +export async function runPetrinautBrowserChecks(input: { + readonly candidateRoot: string; + readonly contract: PetrinautOptimizationExecutionCasePublicContract; + readonly manifest: PetrinautOptimizationControllerOracleManifest; +}): Promise<{ + readonly checks: readonly PetrinautOptimizationOracleCheck[]; + readonly consoleErrors: readonly string[]; + readonly failedRequests: readonly string[]; +}> { + const fake = await startDeterministicFakeOptimizer(); + const port = await availablePort(); + const candidateOrigin = `http://127.0.0.1:${port}`; + const processEvidence: string[] = []; + const candidate = spawn( + 'yarn', + ['workspace', '@apps/petrinaut-website', 'dev', '--host', '127.0.0.1', '--port', String(port)], + { + cwd: input.candidateRoot, + env: { + ...process.env, + PETRINAUT_OPT_ORIGIN: fake.origin, + VITE_PETRINAUT_OPT_PROVIDER: 'service', + }, + detached: process.platform !== 'win32', + stdio: ['ignore', 'pipe', 'pipe'], + }, + ); + candidate.stdout?.on('data', (chunk: Buffer) => processEvidence.push(chunk.toString('utf8'))); + candidate.stderr?.on('data', (chunk: Buffer) => processEvidence.push(chunk.toString('utf8'))); + let browser: Browser | undefined; + const consoleErrors: string[] = []; + const failedRequests: string[] = []; + const calibrationSeed = JSON.parse(await readFile(CALIBRATION_SEED_PATH, 'utf8')) as unknown; + const calibration = parseCalibrationInputs(calibrationSeed); + try { + await waitForRoute( + `${candidateOrigin}${input.contract.acceptance.publicRoute}`, + candidate, + processEvidence, + ); + browser = await chromium.launch({ executablePath: await resolveChromeExecutable(), headless: true }); + const definitions = checkDefinitions({ + contract: input.contract, + calibration, + candidateOrigin, + fakeOrigin: fake.origin, + requests: fake.requests, + }); + const checks: PetrinautOptimizationOracleCheck[] = []; + for (const declared of input.manifest.checks) { + const definition = definitions.get(declared.id); + if (definition === undefined) throw new Error(`missing Petrinaut check implementation: ${declared.id}`); + const context = await browser.newContext(); + await context.addInitScript((seed) => { + localStorage.setItem('petrinaut-sdcpn', JSON.stringify({ [(seed as { id: string }).id]: seed })); + }, calibrationSeed); + const page = await context.newPage(); + page.setDefaultTimeout(SEMANTIC_ACTION_TIMEOUT_MS); + page.on('console', (message) => { + if (message.type() === 'error') consoleErrors.push(message.text()); + }); + page.on('pageerror', (error) => consoleErrors.push(error.message)); + page.on('requestfailed', (request) => + declared.id === 'cancel-and-abort' && request.failure()?.errorText === 'net::ERR_ABORTED' + ? undefined + : failedRequests.push( + `${request.method()} ${request.url()}: ${request.failure()?.errorText ?? 'failed'}`, + ), + ); + try { + await page.goto(`${candidateOrigin}${input.contract.acceptance.publicRoute}`, { + waitUntil: 'domcontentloaded', + timeout: NAVIGATION_TIMEOUT_MS, + }); + const evidence = await definition(page); + checks.push({ id: declared.id, claims: declared.claims, status: 'passed', evidence }); + } catch (error) { + checks.push({ + id: declared.id, + claims: declared.claims, + status: 'failed', + evidence: [error instanceof Error ? error.message : String(error)], + }); + } finally { + await context.close(); + } + } + return { checks, consoleErrors, failedRequests }; + } finally { + await browser?.close(); + await stopChild(candidate); + await fake.close(); + } +} + +type CheckDefinition = (page: Page) => Promise; + +interface CalibrationInputs { + readonly primaryScenarioName: string; + readonly resetScenarioName: string; + readonly optimizeParameterAddress: PetrinautMechanicalAddress; + readonly savedMetricName: string; +} + +function checkDefinitions(input: { + readonly contract: PetrinautOptimizationExecutionCasePublicContract; + readonly calibration: CalibrationInputs; + readonly candidateOrigin: string; + readonly fakeOrigin: string; + readonly requests: { readonly body: unknown; readonly aborted: boolean }[]; +}): ReadonlyMap { + const addresses = input.contract.mechanicalAddresses; + return new Map([ + [ + 'route-and-accessibility', + async (page) => { + await navigateToOptimizations(page, addresses); + await requireCount(locate(page, addresses.viewTitle), 1, 'viewTitle'); + await requireCount(locate(page, addresses.create), 1, 'create'); + await openCreateDrawer(page, addresses); + await requireCount(locate(page, addresses.createDrawer), 1, 'createDrawer'); + await requireCount(locate(page, addresses.scenario), 1, 'scenario'); + await selectScenarioOption(page, addresses, input.calibration.primaryScenarioName); + await requireCount(locate(page, addresses.metric), 1, 'metric'); + await requireCount(locate(page, addresses.directionMaximize), 1, 'directionMaximize'); + await requireCount(locate(page, addresses.run), 1, 'run'); + requirePetrinautFocusedObservation({ + check: 'route-and-accessibility', + pathname: new URL(page.url()).pathname, + expectedPathname: input.contract.acceptance.publicRoute, + }); + return [ + 'public /optimization route ready', + 'required controls resolve through declared mechanical addresses', + ]; + }, + ], + [ + 'scenario-configuration', + async (page) => { + await openConfiguration(page, addresses, input.calibration); + const optimize = locate(page, input.calibration.optimizeParameterAddress); + await optimize.click({ force: true }); + assert(await optimize.isChecked(), 'optimize toggle did not enable'); + await locate(page, addresses.directionMinimize).click({ force: true }); + await selectComboboxOption( + page, + addresses.scenarioSelected, + optionAddress(input.calibration.resetScenarioName), + ); + assert( + !(await locate(page, input.calibration.optimizeParameterAddress).isChecked()), + 'scenario change retained optimized binding', + ); + assert( + !(await locate(page, addresses.directionMinimize).isChecked()), + 'scenario change retained metric direction', + ); + return [ + 'configuration hidden until scenario selection', + 'scenario change reset bindings and objective', + ]; + }, + ], + [ + 'request-contract', + async (page) => { + await openConfiguration(page, addresses, input.calibration); + const savedRequestIndex = input.requests.length; + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, optionAddress(input.calibration.savedMetricName)); + await locate(page, addresses.directionMaximize).click({ force: true }); + await setOptimizationName(page, addresses, 'saved metric proof'); + await locate(page, addresses.run).click(); + await waitForAddress(page, addresses.statusComplete); + const savedBody = input.requests[savedRequestIndex]?.body; + assert(record(savedBody), 'fake optimizer did not capture the saved-metric request'); + const savedObjective = savedBody['objective']; + assert( + record(savedObjective) && + savedObjective['direction'] === 'maximize' && + typeof savedObjective['metricId'] === 'string', + 'saved objective missing', + ); + + await dismissOverlayDrawers(page); + await openCreateDrawer(page, addresses); + await selectScenarioOption(page, addresses, input.calibration.primaryScenarioName); + const customRequestIndex = input.requests.length; + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, addresses.metricCustomOption); + await locate(page, addresses.metricCode).fill('return 42;'); + await locate(page, addresses.directionMinimize).click({ force: true }); + await setOptimizationName(page, addresses, 'custom metric proof'); + await locate(page, addresses.run).click(); + await waitForAddress(page, addresses.statusComplete); + const body = input.requests[customRequestIndex]?.body; + assert(record(body), 'fake optimizer did not capture a JSON request'); + assert( + record(body['scenario']) && typeof body['scenario']['id'] === 'string', + 'scenario missing from request', + ); + const bindings = record(body['scenario']) ? body['scenario']['parameterBindings'] : undefined; + assert(record(bindings), 'parameter bindings missing from request'); + assert( + Object.values(bindings).some((binding) => record(binding) && binding['kind'] === 'optimize'), + 'optimized binding missing', + ); + assert( + Object.values(bindings).some((binding) => record(binding) && binding['kind'] === 'fixed'), + 'fixed binding missing', + ); + const objective = body['objective']; + assert(record(objective) && objective['direction'] === 'minimize', 'objective direction missing'); + assert(record(objective) && typeof objective['metricId'] === 'string', 'custom objective missing'); + const model = body['model']; + assert( + record(model) && + record(model['definition']) && + Array.isArray(model['definition']['metrics']) && + model['definition']['metrics'].some( + (metric) => record(metric) && metric['code'] === 'return 42;', + ), + 'custom metric code missing from request model', + ); + return [ + 'captured flat fixed/optimized bindings', + 'captured saved and custom objectives with direction', + ]; + }, + ], + [ + 'progress-and-completion', + async (page) => { + await openConfiguration(page, addresses, input.calibration); + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, optionAddress(input.calibration.savedMetricName)); + await locate(page, addresses.directionMaximize).click({ force: true }); + await setOptimizationName(page, addresses, 'progress proof'); + await locate(page, addresses.run).click(); + await waitForAddress(page, addresses.statusComplete); + const progressiveTrialCount = await page.getByText(/^\d+$/u).count(); + const bestSoFarVisible = (await page.getByText('Best', { exact: true }).count()) > 0; + const completionVisible = (await locate(page, addresses.statusComplete).count()) > 0; + requirePetrinautFocusedObservation({ + check: 'progress-and-completion', + progressiveTrialCount, + bestSoFarVisible, + completionVisible, + }); + return ['progressive trial rendered', 'best-so-far rendered', 'completion rendered']; + }, + ], + [ + 'service-error', + async (page) => { + await openConfiguration(page, addresses, input.calibration); + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, optionAddress(input.calibration.savedMetricName)); + await locate(page, addresses.directionMaximize).click({ force: true }); + await setOptimizationName(page, addresses, 'service failure'); + await locate(page, addresses.run).click(); + await waitForAddress(page, addresses.statusError); + return ['service error rendered distinctly']; + }, + ], + [ + 'cancel-and-abort', + async (page) => { + const requestIndex = input.requests.length; + await openConfiguration(page, addresses, input.calibration); + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, optionAddress(input.calibration.savedMetricName)); + await locate(page, addresses.directionMaximize).click({ force: true }); + await setOptimizationName(page, addresses, 'cancel proof'); + await locate(page, addresses.run).click(); + // Create closes and the view drawer opens on the active record (initializing/running). + await page + .getByText('Running', { exact: true }) + .filter({ visible: true }) + .or(page.getByText('Initializing', { exact: true }).filter({ visible: true })) + .first() + .waitFor(); + const cancel = locate(page, addresses.cancel); + await cancel.waitFor(); + await cancel.focus(); + assert( + await cancel.evaluate((element) => { + const active = document.activeElement; + return active === element || (active !== null && element.contains(active)); + }), + 'cancel is not focusable', + ); + await cancel.press('Enter'); + await waitForAddress(page, addresses.statusCancelled); + await waitFor( + () => input.requests[requestIndex]?.aborted === true, + 'upstream request was not aborted', + ); + requirePetrinautFocusedObservation({ + check: 'cancel-and-abort', + cancelControlVisible: true, + hostRequestAborted: input.requests[requestIndex]?.aborted === true, + cancelledVisible: (await locate(page, addresses.statusCancelled).count()) > 0, + }); + return ['cancel keyboard control rendered', 'host request aborted', 'cancelled state rendered']; + }, + ], + [ + 'private-origin-secrecy', + async (page) => { + const browserRequests: string[] = []; + page.on('request', (request) => browserRequests.push(request.url())); + await openConfiguration(page, addresses, input.calibration); + await locate(page, input.calibration.optimizeParameterAddress).click({ force: true }); + await selectComboboxOption(page, addresses.metric, optionAddress(input.calibration.savedMetricName)); + await locate(page, addresses.directionMaximize).click({ force: true }); + await setOptimizationName(page, addresses, 'secrecy proof'); + await locate(page, addresses.run).click(); + await waitForAddress(page, addresses.statusComplete); + requirePetrinautFocusedObservation({ + check: 'private-origin-secrecy', + candidateOrigin: input.candidateOrigin, + browserRequestUrls: browserRequests, + domText: await page.locator('body').innerText(), + privateOrigin: input.fakeOrigin, + }); + return ['browser traffic remained same-origin', 'DOM omitted private optimizer origin']; + }, + ], + ]); +} + +async function navigateToOptimizations( + page: Page, + addresses: PetrinautOptimizationExecutionCasePublicContract['mechanicalAddresses'], +): Promise { + const skip = locate(page, addresses.skipTour); + if ((await skip.count()) > 0) await skip.click({ force: true }); + const dismiss = locate(page, addresses.dismissAssistant); + if ((await dismiss.count()) > 0) await dismiss.click({ force: true }); + await locate(page, addresses.simulateMode).click({ force: true }); + await locate(page, addresses.optimizationsNav).click({ force: true }); + await locate(page, addresses.viewTitle).waitFor(); +} + +async function openConfiguration( + page: Page, + addresses: PetrinautOptimizationExecutionCasePublicContract['mechanicalAddresses'], + calibration: CalibrationInputs, +): Promise { + await navigateToOptimizations(page, addresses); + await openCreateDrawer(page, addresses); + assert( + (await locate(page, calibration.optimizeParameterAddress).count()) === 0, + 'configuration appeared before scenario selection', + ); + await selectScenarioOption(page, addresses, calibration.primaryScenarioName); +} + +async function openCreateDrawer( + page: Page, + addresses: PetrinautOptimizationExecutionCasePublicContract['mechanicalAddresses'], +): Promise { + if ((await locate(page, addresses.createDrawer).count()) === 0) { + await dismissOverlayDrawers(page); + await locate(page, addresses.create).click({ force: true }); + } + await locate(page, addresses.createDrawer).waitFor(); + await locate(page, addresses.run).waitFor(); + assert((await locate(page, addresses.run).count()) === 1, 'create drawer did not expose Run control'); +} + +async function dismissOverlayDrawers(page: Page): Promise { + for (let attempt = 0; attempt < 3; attempt += 1) { + if ((await page.getByRole('dialog').count()) === 0) return; + await page.keyboard.press('Escape'); + try { + await page.getByRole('dialog').first().waitFor({ state: 'hidden', timeout: 500 }); + } catch { + // Keep dismissing until no dialog remains or attempts exhaust. + } + } +} + +function locate(page: Page, address: PetrinautMechanicalAddress): Locator { + switch (address.kind) { + case 'roleName': + return page.getByRole(address.role as Parameters[0], { + name: address.name, + exact: true, + }); + case 'roleValue': + return page.locator(`${cssRoleSelector(address.role)}[value="${cssEscape(address.value)}"]`); + case 'roleContents': + return page.getByRole(address.role as Parameters[0]).filter({ + hasText: address.contents, + }); + case 'exactText': + // exactText never resolves role=tooltip nodes (nav tooltips reuse titles). + return page + .getByText(address.text, { exact: true }) + .and(page.locator(':not([role="tooltip"])')) + .filter({ visible: true }); + } +} + +async function selectScenarioOption( + page: Page, + addresses: PetrinautOptimizationExecutionCasePublicContract['mechanicalAddresses'], + optionText: string, +): Promise { + const combobox = locate(page, addresses.scenario); + await requireCount(combobox, 1, 'scenario'); + await selectComboboxLocatorOption(page, combobox, optionAddress(optionText)); +} + +async function selectComboboxLocatorOption( + page: Page, + combobox: Locator, + option: PetrinautMechanicalAddress, +): Promise { + const tagName = await combobox.evaluate((element) => element.tagName); + if (tagName === 'SELECT') { + assert( + option.kind === 'roleName' && option.role === 'option', + 'native select options require a roleName option address', + ); + await combobox.selectOption({ label: option.name }); + return; + } + await combobox.click({ force: true }); + await locate(page, option).click(); +} + +function cssRoleSelector(role: string): string { + if (role === 'radio') return 'input[type="radio"]'; + return `[role="${cssEscape(role)}"]`; +} + +function cssEscape(value: string): string { + return value.replaceAll('\\', '\\\\').replaceAll('"', '\\"'); +} + +async function selectComboboxOption( + page: Page, + address: PetrinautMechanicalAddress, + option: PetrinautMechanicalAddress, +): Promise { + await selectComboboxLocatorOption(page, locate(page, address), option); +} + +function optionAddress(name: string): PetrinautMechanicalAddress { + return { kind: 'roleName', role: 'option', name }; +} + +async function setOptimizationName( + page: Page, + addresses: PetrinautOptimizationExecutionCasePublicContract['mechanicalAddresses'], + name: string, +): Promise { + const nameField = locate(page, addresses.optimizationName); + await requireCount(nameField, 1, 'optimizationName'); + await nameField.fill(name); +} + +async function waitForAddress(page: Page, address: PetrinautMechanicalAddress): Promise { + await locate(page, address).first().waitFor(); +} + +async function requireCount(locator: Locator, expected: number, label: string): Promise { + const actual = await locator.count(); + assert(actual === expected, `${label}: expected ${expected}, received ${actual}`); +} + +function parseCalibrationInputs(value: unknown): CalibrationInputs { + assert(record(value), 'invalid Petrinaut calibration seed'); + const sdcpn = value['sdcpn']; + assert(record(sdcpn), 'calibration seed is missing sdcpn'); + const primary = recordById(sdcpn['scenarios'], 'scenario__seasonal_flu', 'scenario'); + const reset = recordById(sdcpn['scenarios'], 'scenario__high_virulence', 'scenario'); + const metric = recordById(sdcpn['metrics'], 'metric__infected_fraction', 'metric'); + assert(typeof primary['name'] === 'string', 'primary calibration scenario is missing its name'); + assert(typeof reset['name'] === 'string', 'reset calibration scenario is missing its name'); + assert(typeof metric['name'] === 'string', 'calibration metric is missing its name'); + assert(Array.isArray(primary['scenarioParameters']), 'primary scenario is missing parameters'); + const parameter = primary['scenarioParameters'].find( + (candidate) => record(candidate) && candidate['identifier'] === 'infected_ratio', + ); + assert(record(parameter), 'primary scenario is missing the optimized calibration parameter'); + assert( + typeof parameter['identifier'] === 'string', + 'optimized calibration parameter is missing its identifier', + ); + return { + primaryScenarioName: primary['name'], + resetScenarioName: reset['name'], + optimizeParameterAddress: { + kind: 'roleName', + role: 'checkbox', + name: `Optimize ${parameter['identifier']}`, + }, + savedMetricName: metric['name'], + }; +} + +function recordById(value: unknown, id: string, label: string): Record { + assert(Array.isArray(value), `calibration seed is missing ${label} rows`); + const found = value.find((candidate) => record(candidate) && candidate['id'] === id); + assert(record(found), `calibration seed is missing ${label} ${id}`); + return found; +} + +async function availablePort(): Promise { + const server = createServer(); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', () => { + server.off('error', reject); + resolve(); + }); + }); + const address = server.address(); + if (address === null || typeof address === 'string') throw new Error('port reservation failed'); + const port = address.port; + await new Promise((resolveClose, reject) => { + server.close((error) => (error === undefined ? resolveClose() : reject(error))); + }); + return port; +} + +async function waitForRoute(url: string, child: ChildProcess, evidence: readonly string[]): Promise { + for (let attempt = 0; attempt < 100; attempt += 1) { + if (child.exitCode !== null) { + throw new Error(`candidate process exited ${child.exitCode}: ${evidence.join('')}`); + } + try { + const response = await fetch(url); + if (response.ok) return; + } catch { + // Candidate is still starting. + } + await new Promise((resolve) => setTimeout(resolve, 50)); + } + throw new Error(`candidate /optimization route did not become ready: ${evidence.join('')}`); +} + +async function stopChild(child: ChildProcess): Promise { + if (child.exitCode !== null) return; + signalChildTree(child, 'SIGTERM'); + await Promise.race([ + new Promise((resolve) => child.once('exit', () => resolve())), + new Promise((resolve) => setTimeout(resolve, 2_000)), + ]); + if (child.exitCode === null) signalChildTree(child, 'SIGKILL'); +} + +function signalChildTree(child: ChildProcess, signal: NodeJS.Signals): void { + if (process.platform !== 'win32' && child.pid !== undefined) { + try { + process.kill(-child.pid, signal); + return; + } catch { + // The process group may already have exited. + } + } + child.kill(signal); +} + +async function resolveChromeExecutable(): Promise { + const candidates = [ + process.env['BRUNCH_CHROME_PATH'], + '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', + '/usr/bin/google-chrome', + '/usr/bin/chromium', + ].filter((candidate): candidate is string => candidate !== undefined && candidate.length > 0); + for (const candidate of candidates) { + try { + await access(candidate); + return candidate; + } catch { + // Continue through closed platform candidates. + } + } + throw new Error('no Chrome executable found; set BRUNCH_CHROME_PATH'); +} + +async function waitFor(condition: () => boolean, message: string): Promise { + for (let attempt = 0; attempt < 100; attempt += 1) { + if (condition()) return; + await new Promise((resolve) => setTimeout(resolve, 20)); + } + throw new Error(message); +} + +function record(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function assert(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message); +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/calibration-seed.json b/src/dev/execution-comparison/petrinaut-optimization-oracle/calibration-seed.json new file mode 100644 index 000000000..a549582a5 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/calibration-seed.json @@ -0,0 +1,165 @@ +{ + "id": "sir-calibration", + "title": "SIR Epidemic Model", + "sdcpn": { + "places": [ + { + "id": "place__susceptible", + "name": "Susceptible", + "colorId": null, + "dynamicsEnabled": false, + "differentialEquationId": null, + "showAsInitialState": true, + "x": -435, + "y": 150 + }, + { + "id": "place__infected", + "name": "Infected", + "colorId": null, + "dynamicsEnabled": false, + "differentialEquationId": null, + "showAsInitialState": true, + "x": -195, + "y": 285 + }, + { + "id": "place__recovered", + "name": "Recovered", + "colorId": null, + "dynamicsEnabled": false, + "differentialEquationId": null, + "x": 375, + "y": 195 + } + ], + "transitions": [ + { + "id": "transition__infection", + "name": "Infection", + "inputArcs": [ + { "placeId": "place__susceptible", "weight": 1, "type": "standard" }, + { "placeId": "place__infected", "weight": 1, "type": "standard" } + ], + "outputArcs": [{ "placeId": "place__infected", "weight": 2 }], + "lambdaType": "stochastic", + "lambdaCode": "// Mass-action infection: fires at the configured infection rate whenever a\n// Susceptible and an Infected are both present (the two standard input arcs).\nexport default Lambda((tokens, parameters) => parameters.infection_rate)", + "transitionKernelCode": "// Consumes 1 Susceptible + 1 Infected and produces 2 Infected (the output\n// arc has weight 2), encoding the S + I -> 2I reaction: the susceptible has\n// become newly infected. Places are untyped, so tokens carry no attributes.\nexport default TransitionKernel(() => {\n return {\n Infected: [{}, {}],\n };\n});", + "x": -150, + "y": 75 + }, + { + "id": "transition__recovery", + "name": "Recovery", + "inputArcs": [{ "placeId": "place__infected", "weight": 1, "type": "standard" }], + "outputArcs": [{ "placeId": "place__recovered", "weight": 1 }], + "lambdaType": "stochastic", + "lambdaCode": "// Each Infected recovers at the configured recovery rate. The ratio of\n// infection_rate to recovery_rate sets the basic reproduction number R0.\nexport default Lambda((tokens, parameters) => parameters.recovery_rate)", + "transitionKernelCode": "// Move one Infected to Recovered (1-to-1). Recovered individuals are immune,\n// so they never re-enter the Susceptible or Infected places.\nexport default TransitionKernel(() => {\n return {\n Recovered: [{}],\n };\n});", + "x": 90, + "y": 240 + } + ], + "types": [], + "differentialEquations": [], + "parameters": [ + { + "id": "param__infection_rate", + "name": "Infection Rate", + "variableName": "infection_rate", + "type": "real", + "defaultValue": "3" + }, + { + "id": "param__recovery_rate", + "name": "Recovery Rate", + "variableName": "recovery_rate", + "type": "real", + "defaultValue": "1" + } + ], + "scenarios": [ + { + "id": "scenario__seasonal_flu", + "name": "Seasonal Flu", + "description": "Moderate outbreak with R₀ ≈ 1.5. Models a typical seasonal influenza wave in a small community.", + "scenarioParameters": [ + { "type": "integer", "identifier": "population", "default": 1000 }, + { "type": "ratio", "identifier": "infected_ratio", "default": 0.01 } + ], + "parameterOverrides": { "param__infection_rate": "1.5", "param__recovery_rate": "0.8" }, + "initialState": { + "type": "per_place", + "content": { + "place__susceptible": "scenario.population * (1 - scenario.infected_ratio)", + "place__infected": "scenario.population * scenario.infected_ratio", + "place__recovered": "0" + } + } + }, + { + "id": "scenario__high_virulence", + "name": "High Virulence Outbreak", + "description": "Aggressive pathogen with R₀ ≈ 6 and slow recovery, modelling rapid spread before interventions.", + "scenarioParameters": [ + { "type": "integer", "identifier": "population", "default": 10000 }, + { "type": "ratio", "identifier": "infected_ratio", "default": 0.0001 } + ], + "parameterOverrides": { "param__infection_rate": "6", "param__recovery_rate": "0.5" }, + "initialState": { + "type": "per_place", + "content": { + "place__susceptible": "scenario.population * (1 - scenario.infected_ratio)", + "place__infected": "scenario.population * scenario.infected_ratio", + "place__recovered": "0" + } + } + }, + { + "id": "scenario__contained_outbreak", + "name": "Contained Outbreak", + "description": "Sub-threshold spread with R₀ < 1 (recovery outpaces infection), so the outbreak fizzles out instead of taking off.", + "scenarioParameters": [ + { "type": "integer", "identifier": "population", "default": 1000 }, + { "type": "ratio", "identifier": "infected_ratio", "default": 0.05 } + ], + "parameterOverrides": { "param__infection_rate": "0.6", "param__recovery_rate": "1.2" }, + "initialState": { + "type": "per_place", + "content": { + "place__susceptible": "scenario.population * (1 - scenario.infected_ratio)", + "place__infected": "scenario.population * scenario.infected_ratio", + "place__recovered": "0" + } + } + }, + { + "id": "scenario__pandemic_wave", + "name": "Pandemic Wave", + "description": "A large, mostly-susceptible population seeded with a few cases and R₀ ≈ 5 — useful for watching the classic epidemic curve build and burn out at scale.", + "scenarioParameters": [ + { "type": "integer", "identifier": "population", "default": 100000 }, + { "type": "ratio", "identifier": "infected_ratio", "default": 0.00005 } + ], + "parameterOverrides": { "param__infection_rate": "2.5", "param__recovery_rate": "0.5" }, + "initialState": { + "type": "per_place", + "content": { + "place__susceptible": "scenario.population * (1 - scenario.infected_ratio)", + "place__infected": "scenario.population * scenario.infected_ratio", + "place__recovered": "0" + } + } + } + ], + "metrics": [ + { + "id": "metric__infected_fraction", + "name": "Infected Fraction", + "description": "Share of the population currently infected.", + "code": "const s = state.places.Susceptible.count;\nconst i = state.places.Infected.count;\nconst r = state.places.Recovered.count;\nconst total = s + i + r;\nreturn total === 0 ? 0 : i / total;" + } + ] + }, + "lastUpdated": "2026-07-22T16:29:57.811Z" +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/claims.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/claims.ts new file mode 100644 index 000000000..2d3067521 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/claims.ts @@ -0,0 +1,58 @@ +export type PetrinautFocusedObservation = + | { + readonly check: 'route-and-accessibility'; + readonly pathname: string; + readonly expectedPathname: string; + } + | { + readonly check: 'progress-and-completion'; + readonly progressiveTrialCount: number; + readonly bestSoFarVisible: boolean; + readonly completionVisible: boolean; + } + | { + readonly check: 'cancel-and-abort'; + readonly cancelControlVisible: boolean; + readonly cancelledVisible: boolean; + readonly hostRequestAborted: boolean; + } + | { + readonly check: 'private-origin-secrecy'; + readonly candidateOrigin: string; + readonly browserRequestUrls: readonly string[]; + readonly domText: string; + readonly privateOrigin: string; + }; + +export function assessPetrinautFocusedObservation( + observation: PetrinautFocusedObservation, +): readonly string[] { + switch (observation.check) { + case 'route-and-accessibility': + return observation.pathname === observation.expectedPathname ? [] : ['focused route missing']; + case 'progress-and-completion': + return [ + ...(observation.progressiveTrialCount > 0 ? [] : ['progressive trials missing']), + ...(observation.bestSoFarVisible ? [] : ['best-so-far missing']), + ...(observation.completionVisible ? [] : ['completion missing']), + ]; + case 'cancel-and-abort': + return [ + ...(observation.cancelControlVisible ? [] : ['cancel control missing']), + ...(observation.cancelledVisible ? [] : ['cancelled state missing']), + ...(observation.hostRequestAborted ? [] : ['host request was not aborted']), + ]; + case 'private-origin-secrecy': + return [ + ...(observation.browserRequestUrls.every((url) => new URL(url).origin === observation.candidateOrigin) + ? [] + : ['browser contacted private origin']), + ...(observation.domText.includes(observation.privateOrigin) ? ['DOM exposed private origin'] : []), + ]; + } +} + +export function requirePetrinautFocusedObservation(observation: PetrinautFocusedObservation): void { + const failures = assessPetrinautFocusedObservation(observation); + if (failures.length > 0) throw new Error(failures.join('; ')); +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/fake-optimizer.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/fake-optimizer.ts new file mode 100644 index 000000000..6f6fba9f8 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/fake-optimizer.ts @@ -0,0 +1,98 @@ +import { createServer, type ServerResponse } from 'node:http'; + +export interface CapturedOptimizationRequest { + readonly body: unknown; + aborted: boolean; +} + +/** + * Deterministic Petrinaut Opt stand-in that speaks the upstream Optuna SSE + * contract decoded by `@local/petrinaut-optimizer-client`. + * + * Request `name` steers the branch: + * - includes `failure` → terminal `event: error` + * - includes `cancel` → hold the stream open (no terminal event) + * - otherwise → two COMPLETE trials + `event: done` + */ +export async function startDeterministicFakeOptimizer(): Promise<{ + readonly origin: string; + readonly requests: CapturedOptimizationRequest[]; + readonly close: () => Promise; +}> { + const requests: CapturedOptimizationRequest[] = []; + const openResponses = new Set(); + const server = createServer(async (request, response) => { + if (request.method !== 'POST' || request.url !== '/optimize/all') { + response.writeHead(404).end('Not found'); + return; + } + const chunks: Buffer[] = []; + for await (const chunk of request) chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + let body: unknown; + try { + body = JSON.parse(Buffer.concat(chunks).toString('utf8')) as unknown; + } catch { + response.writeHead(400).end('invalid JSON'); + return; + } + const captured: CapturedOptimizationRequest = { body, aborted: false }; + requests.push(captured); + openResponses.add(response); + const markAborted = () => { + if (!response.writableEnded) captured.aborted = true; + openResponses.delete(response); + }; + request.on('aborted', markAborted); + response.on('close', markAborted); + response.writeHead(200, { 'content-type': 'text/event-stream' }); + const name = record(body) && typeof body['name'] === 'string' ? body['name'] : ''; + if (name.includes('cancel')) return; + if (name.includes('failure')) { + writeEvent(response, { message: 'Deterministic optimizer failure' }, 'error'); + response.end(); + return; + } + writeEvent(response, { + step: 0, + params: { rate: 4 }, + metric: 12, + state: 'COMPLETE', + }); + await new Promise((resolve) => setTimeout(resolve, 20)); + writeEvent(response, { + step: 1, + params: { rate: 6 }, + metric: 10, + state: 'COMPLETE', + }); + writeEvent(response, {}, 'done'); + response.end(); + }); + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', () => { + server.off('error', reject); + resolve(); + }); + }); + const address = server.address(); + if (address === null || typeof address === 'string') throw new Error('fake optimizer has no TCP address'); + return { + origin: `http://127.0.0.1:${address.port}`, + requests, + close: async () => { + for (const open of openResponses) open.destroy(); + await new Promise((resolveClose, reject) => { + server.close((error) => (error === undefined ? resolveClose() : reject(error))); + }); + }, + }; +} + +function writeEvent(response: ServerResponse, event: unknown, name?: 'done' | 'error'): void { + response.write(`${name === undefined ? '' : `event: ${name}\n`}data: ${JSON.stringify(event)}\n\n`); +} + +function record(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/fixture.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/fixture.ts new file mode 100644 index 000000000..0cdf6aaa3 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/fixture.ts @@ -0,0 +1,403 @@ +import { mkdir, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; + +const PACKAGE_SCRIPTS = [ + ['packages/ds-components', '@hashintel/ds-components', ['codegen', 'build']], + ['packages/petrinaut-core', '@hashintel/petrinaut-core', ['build']], + ['packages/optimizer-client', '@local/petrinaut-optimizer-client', ['build']], + ['packages/refractive', '@hashintel/refractive', ['build']], + ['packages/petrinaut', '@hashintel/petrinaut', ['build']], +] as const; + +export async function createKnownGoodPetrinautCandidate( + root: string, + options: { + readonly backgroundRequestDurationMs?: number; + readonly inventedLabelsRival?: boolean; + } = {}, +): Promise { + await mkdir(join(root, 'scripts'), { recursive: true }); + await writeFile( + join(root, 'package.json'), + `${JSON.stringify( + { + private: true, + workspaces: ['packages/*', 'apps/*'], + }, + null, + 2, + )}\n`, + ); + await writeFile(join(root, 'scripts', 'mark.mjs'), preparationMarkerSource()); + await writeFile( + join(root, 'scripts', 'server.mjs'), + candidateServerSource({ + backgroundRequestDurationMs: options.backgroundRequestDurationMs ?? 0, + inventedLabelsRival: options.inventedLabelsRival ?? false, + }), + ); + for (const [path, name, scripts] of PACKAGE_SCRIPTS) { + await mkdir(join(root, path), { recursive: true }); + await writeFile( + join(root, path, 'package.json'), + `${JSON.stringify({ + name, + version: '0.0.0', + private: true, + scripts: Object.fromEntries( + scripts.map((script) => [script, `node ../../scripts/mark.mjs ${name}:${script}`]), + ), + })}\n`, + ); + } + await mkdir(join(root, 'apps', 'petrinaut-website'), { recursive: true }); + await writeFile( + join(root, 'apps', 'petrinaut-website', 'package.json'), + `${JSON.stringify({ + name: '@apps/petrinaut-website', + version: '0.0.0', + private: true, + scripts: { dev: 'node ../../scripts/server.mjs' }, + })}\n`, + ); +} + +/** Rival that retains the pre-D138 invented accessibility labels. */ +export async function createInventedLabelsPetrinautCandidate(root: string): Promise { + await createKnownGoodPetrinautCandidate(root, { inventedLabelsRival: true }); +} + +function preparationMarkerSource(): string { + return String.raw` +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; + +const root = resolve(import.meta.dirname, ".."); +const path = resolve(root, ".focused-preparation.json"); +const expected = [ + "@hashintel/ds-components:codegen", + "@hashintel/ds-components:build", + "@hashintel/petrinaut-core:build", + "@local/petrinaut-optimizer-client:build", + "@hashintel/refractive:build", + "@hashintel/petrinaut:build", +]; +let completed = []; +try { + completed = JSON.parse(await readFile(path, "utf8")); +} catch {} +const next = process.argv[2]; +if (next !== expected[completed.length]) { + throw new Error("focused preparation ran out of order: " + next); +} +completed.push(next); +await writeFile(path, JSON.stringify(completed)); +`; +} + +function candidateServerSource(options: { + readonly backgroundRequestDurationMs: number; + readonly inventedLabelsRival: boolean; +}): string { + const invented = options.inventedLabelsRival; + const viewTitle = invented ? '

Optimizations

' : 'Optimizations'; + const nav = invented + ? '
' + : `
+ + +
+
+ + + +
`; + const createLabel = invented ? 'Create optimization' : 'Create'; + const runLabel = invented ? 'Run optimization' : 'Run'; + const cancelLabel = invented ? 'Cancel optimization' : 'Cancel'; + const scenarioLabel = invented ? 'Scenario' : ''; + const scenarioPlaceholder = invented ? 'Select scenario' : 'Select a scenario'; + const metricLabel = invented ? 'Objective metric' : ''; + const metricPlaceholder = invented ? '' : 'Select a metric'; + const directionBlock = invented + ? `` + : `
+ + +
`; + const statusBlock = invented + ? '
Idle
' + : '
Idle
Best
'; + + return String.raw` +import { createServer, request as httpRequest } from "node:http"; + +const args = new Map(); +for (let index = 2; index < process.argv.length; index += 2) { + args.set(process.argv[index], process.argv[index + 1]); +} +const host = args.get("--host") ?? "127.0.0.1"; +const port = Number(args.get("--port")); +const optimizerOrigin = new URL(process.env.PETRINAUT_OPT_ORIGIN); +const optimizerProvider = process.env.VITE_PETRINAUT_OPT_PROVIDER; +const requiredPreparation = 6; +const backgroundRequestDurationMs = ${options.backgroundRequestDurationMs}; + +const html = String.raw${'`'} + +
+ ${nav} + + +

Create an optimization

+ + + + + + ${statusBlock} +
+ +${'`'}; + +const server = createServer(async (request, response) => { + if (request.method === "GET" && request.url === "/background-readiness-rival") { + setTimeout(() => { + response.writeHead(204).end(); + }, backgroundRequestDurationMs); + return; + } + if (request.method === "GET" && request.url === "/favicon.ico") { + response.writeHead(204).end(); + return; + } + if (request.method === "GET" && request.url === "/optimization") { + if (optimizerProvider !== "service") { + response.writeHead(404).end("Optimization provider disabled"); + return; + } + response.writeHead(200, { "content-type": "text/html; charset=utf-8" }).end(html); + return; + } + if (request.method === "POST" && request.url === "/api/petrinaut-opt/optimize/all") { + const upstream = httpRequest(new URL("/optimize/all", optimizerOrigin), { + method: "POST", + headers: { "content-type": "application/json", accept: "text/event-stream" }, + }, (upstreamResponse) => { + response.writeHead(upstreamResponse.statusCode ?? 500, { + "content-type": upstreamResponse.headers["content-type"] ?? "text/event-stream", + }); + upstreamResponse.on("error", () => { + if (!response.writableEnded) response.end(); + }); + upstreamResponse.pipe(response); + }); + upstream.on("error", () => { + if (!response.headersSent) response.writeHead(502); + if (!response.writableEnded) response.end(); + }); + request.on("error", () => upstream.destroy()); + request.pipe(upstream); + request.on("aborted", () => upstream.destroy()); + response.on("close", () => { + if (!response.writableEnded) upstream.destroy(); + }); + return; + } + response.writeHead(404).end("Not found"); +}); + +server.listen(port, host, async () => { + const { readFile } = await import("node:fs/promises"); + let prepared = []; + try { prepared = JSON.parse(await readFile(new URL("../.focused-preparation.json", import.meta.url), "utf8")); } catch {} + if (prepared.length !== requiredPreparation) { + console.error("focused preparation incomplete"); + process.exit(1); + } + console.log("PETRINAUT_READY"); +}); +`; +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/runner.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/runner.ts new file mode 100644 index 000000000..701e68d17 --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/runner.ts @@ -0,0 +1,124 @@ +import { runCommand, type CommandResult } from '../../../app/command-runner.js'; +import { isPetrinautOptimizationExecutionCaseContract, loadPublicCasePacket } from '../case-contract.js'; +import { + isPetrinautOptimizationControllerOracleManifest, + loadControllerOracleManifest, +} from '../oracle-pack.js'; +import { runPetrinautBrowserChecks } from './browser.js'; +import type { PetrinautOptimizationOracleReport } from './types.js'; + +export const PETRINAUT_FOCUSED_PREPARATION = [ + { + id: 'design-system-codegen', + command: 'yarn', + args: ['workspace', '@hashintel/ds-components', 'codegen'], + }, + { + id: 'design-system-build', + command: 'yarn', + args: ['workspace', '@hashintel/ds-components', 'build'], + }, + { + id: 'petrinaut-core-build', + command: 'yarn', + args: ['workspace', '@hashintel/petrinaut-core', 'build'], + }, + { + id: 'optimizer-client-build', + command: 'yarn', + args: ['workspace', '@local/petrinaut-optimizer-client', 'build'], + }, + { + id: 'refractive-build', + command: 'yarn', + args: ['workspace', '@hashintel/refractive', 'build'], + }, + { + id: 'petrinaut-ui-build', + command: 'yarn', + args: ['workspace', '@hashintel/petrinaut', 'build'], + }, +] as const; + +export interface PetrinautOraclePreparationObservation { + readonly id: (typeof PETRINAUT_FOCUSED_PREPARATION)[number]['id']; + readonly commandResult: CommandResult; +} + +export async function runPetrinautOptimizationOracle(input: { + readonly candidateRoot: string; + readonly caseDir: string; + readonly onPreparationResult?: (observation: PetrinautOraclePreparationObservation) => Promise | void; +}): Promise { + const [packet, manifest] = await Promise.all([ + loadPublicCasePacket(input.caseDir), + loadControllerOracleManifest(input.caseDir), + ]); + if ( + !isPetrinautOptimizationExecutionCaseContract(packet.contract) || + !isPetrinautOptimizationControllerOracleManifest(manifest) + ) { + throw new Error('Petrinaut optimization oracle received a different compiled case'); + } + + const preparation: PetrinautOptimizationOracleReport['preparation'][number][] = []; + for (const step of PETRINAUT_FOCUSED_PREPARATION) { + const result = await runCommand(step.command, step.args, { + cwd: input.candidateRoot, + timeoutMs: 10 * 60_000, + maxOutputBytes: 256 * 1024, + }); + await input.onPreparationResult?.({ + id: step.id, + commandResult: result, + }); + preparation.push({ + id: step.id, + status: result.exitCode === 0 ? 'passed' : 'failed', + exitCode: result.exitCode, + }); + if (result.exitCode !== 0) { + return { + schemaVersion: 1, + caseId: packet.contract.case.id, + oracleId: manifest.id, + status: 'setup_failed', + preparation, + checks: [], + setupFailure: `${step.id} exited ${result.exitCode}`, + consoleErrors: [], + failedRequests: [], + }; + } + } + + try { + const browser = await runPetrinautBrowserChecks({ + candidateRoot: input.candidateRoot, + contract: packet.contract, + manifest, + }); + return { + schemaVersion: 1, + caseId: packet.contract.case.id, + oracleId: manifest.id, + status: browser.checks.every(({ status }) => status === 'passed') ? 'passed' : 'assertion_failed', + preparation, + checks: browser.checks, + consoleErrors: browser.consoleErrors, + failedRequests: browser.failedRequests, + }; + } catch (error) { + return { + schemaVersion: 1, + caseId: packet.contract.case.id, + oracleId: manifest.id, + status: 'setup_failed', + preparation, + checks: [], + setupFailure: error instanceof Error ? error.message : String(error), + consoleErrors: [], + failedRequests: [], + }; + } +} diff --git a/src/dev/execution-comparison/petrinaut-optimization-oracle/types.ts b/src/dev/execution-comparison/petrinaut-optimization-oracle/types.ts new file mode 100644 index 000000000..6692a63cd --- /dev/null +++ b/src/dev/execution-comparison/petrinaut-optimization-oracle/types.ts @@ -0,0 +1,31 @@ +export type PetrinautOptimizationCheckId = + | 'route-and-accessibility' + | 'scenario-configuration' + | 'request-contract' + | 'progress-and-completion' + | 'service-error' + | 'cancel-and-abort' + | 'private-origin-secrecy'; + +export interface PetrinautOptimizationOracleCheck { + readonly id: PetrinautOptimizationCheckId; + readonly claims: readonly string[]; + readonly status: 'passed' | 'failed' | 'setup_failed'; + readonly evidence: readonly string[]; +} + +export interface PetrinautOptimizationOracleReport { + readonly schemaVersion: 1; + readonly caseId: 'petrinaut-optimization-v1'; + readonly oracleId: 'petrinaut-optimization-oracles-v1'; + readonly status: 'passed' | 'setup_failed' | 'assertion_failed'; + readonly preparation: readonly { + readonly id: string; + readonly status: 'passed' | 'failed'; + readonly exitCode: number; + }[]; + readonly checks: readonly PetrinautOptimizationOracleCheck[]; + readonly setupFailure?: string; + readonly consoleErrors: readonly string[]; + readonly failedRequests: readonly string[]; +} diff --git a/src/session/__tests__/active-branch-reader-inventory.test.ts b/src/session/__tests__/active-branch-reader-inventory.test.ts index 5fd9e01c1..efe3def86 100644 --- a/src/session/__tests__/active-branch-reader-inventory.test.ts +++ b/src/session/__tests__/active-branch-reader-inventory.test.ts @@ -71,6 +71,10 @@ const NON_SESSION_JSON_PARSERS: Record = { rationale: 'parses the dev-only normalized trajectory NDJSON artifact, never Pi session JSONL', requiredOwner: 'readEvents', }, + 'src/dev/execution-comparison/host-landing-oracle/fixture.ts': { + rationale: 'parses public candidate JSON-RPC stdout responses, never Pi session JSONL', + requiredOwner: 'runCandidateRpc', + }, }; const ALL_CLASSIFICATIONS: Record = { diff --git a/testing/comparisons/missions/README.md b/testing/comparisons/missions/README.md index ffc5dc268..c882cdbc2 100644 --- a/testing/comparisons/missions/README.md +++ b/testing/comparisons/missions/README.md @@ -9,6 +9,8 @@ Current library: - [`minimal-petri-net-editor.md`](minimal-petri-net-editor.md) — Petri-editor elicitation and end-to-end origin mission +- [`brunch-host-landing.md`](brunch-host-landing.md) — Brunch host-landing brownfield mission +- [`petrinaut-optimization.md`](petrinaut-optimization.md) — Petrinaut optimization brownfield mission This `README.md` is the directory's reserved control file, not a mission. `/compare-specs` must exclude it from mission listing, resolution, revision, and creation. @@ -31,4 +33,6 @@ The top-level session drives exactly one direct `interactive_shell` comparison h Missions remain editable. A revision changes only the mission and future runs; it never rewrites an existing run directory, private mission snapshot, harness-setup snapshot, transcript, target output, or report. Ephemeral assembly belongs under `.fixtures/scratch/comparisons/`; deliberately retained immutable evidence belongs under `.fixtures/runs/agent-as-user-comparison/`. +Each mission with a matching directory under `testing/execution-comparisons/cases/` may also be run as an execution-only comparison through `/compare-execution `. That path freezes `spec.md` and `public-contract.json`, captures `provenance.json` before the first lane, retains validated attempts and case-owned oracle reports, and writes the validity-first `report.md` consumed by `/comparison-publish`. Elicitation and execution remain separate phases unless an explicitly retained end-to-end matrix is run. + The retained operator-only report may reproduce the complete private mission as its baseline. It must visibly separate that top-level-session-only baseline from each harness's exact visible framing and transcript, outcomes, and unchanged harness-authored document so elicitation and leakage remain legible. It does not choose a winner or impose a fixed rubric. diff --git a/testing/comparisons/missions/brunch-host-landing.md b/testing/comparisons/missions/brunch-host-landing.md new file mode 100644 index 000000000..7b71df99e --- /dev/null +++ b/testing/comparisons/missions/brunch-host-landing.md @@ -0,0 +1,5 @@ +# Safely land a prepared run + +Add a user-confirmed workflow that can land a prepared implementation run into its host repository. +The workflow must show the complete proposed change before mutation, preserve repository safety on +decline or refusal, and make successful landing durable and inspectable. diff --git a/testing/comparisons/missions/petrinaut-optimization.md b/testing/comparisons/missions/petrinaut-optimization.md new file mode 100644 index 000000000..2b527eb19 --- /dev/null +++ b/testing/comparisons/missions/petrinaut-optimization.md @@ -0,0 +1,5 @@ +# Optimize a Petrinaut scenario + +Add a focused optimization interface to Petrinaut. A user must be able to choose a scenario, configure a +flat parameter search space and one objective metric, start an optimization, follow progressive results, +and cancel a run. Keep the optimizer address private behind the website's same-origin API. diff --git a/testing/end-to-end-comparisons/cases/brunch-host-landing/controller/requirement-registry.json b/testing/end-to-end-comparisons/cases/brunch-host-landing/controller/requirement-registry.json new file mode 100644 index 000000000..6d8dd4a31 --- /dev/null +++ b/testing/end-to-end-comparisons/cases/brunch-host-landing/controller/requirement-registry.json @@ -0,0 +1,64 @@ +{ + "schemaVersion": 1, + "caseId": "brunch-host-landing-v1", + "rows": [ + { + "id": "REQ1", + "publicConcern": "Complete-range preflight", + "origin": "public_baseline", + "publicWording": "Review the complete prepared change before host mutation.", + "controller": { + "wording": "The pre-confirmation snapshot names every commit and changed path in runBaseSha..reviewSha.", + "expectedState": "Host HEAD, tree, worktree, and run metadata are byte-identical before confirmation." + } + }, + { + "id": "REQ2", + "publicConcern": "Complete brownfield landing", + "origin": "controller_only", + "controller": { + "wording": "Every non-bookkeeping content change in the complete promoted range lands.", + "expectedState": "The host tree matches the controller Git model, .brunch is absent, and metadata is landed." + } + }, + { + "id": "REQ3", + "publicConcern": "Greenfield materialization", + "origin": "public_baseline", + "publicWording": "A greenfield run can land into a missing or empty target.", + "controller": { + "wording": "The complete review-tip tree becomes one Brunch-authored root commit.", + "expectedState": "The target has one commit and its tree equals reviewSha^{tree} without .brunch paths." + } + }, + { + "id": "REQ4", + "publicConcern": "Refusal safety", + "origin": "controller_only", + "controller": { + "wording": "Decline, dirty-host, conflict, and stale acceptance are inert.", + "expectedState": "The host snapshot is byte-identical and run status remains promotion_prepared." + } + }, + { + "id": "INV1", + "publicConcern": "User mutation authority", + "origin": "public_baseline", + "publicWording": "Landing requires confirmation after a read-only preflight.", + "controller": { + "wording": "No candidate or agent-callable surface mutates the host before confirmation.", + "expectedState": "Mutation begins only after the controller confirms the exact displayed review tip." + } + }, + { + "id": "AC1", + "publicConcern": "Public TUI address", + "origin": "public_baseline", + "publicWording": "The workflow is available as /brunch:land in the public TUI.", + "controller": { + "wording": "A resumed settled session reaches confirmation under PI_OFFLINE=1 without provider activity.", + "expectedState": "The real TUI renders the preflight and no fresh-session kick is observed." + } + } + ] +} diff --git a/testing/end-to-end-comparisons/cases/brunch-host-landing/shared-baseline.md b/testing/end-to-end-comparisons/cases/brunch-host-landing/shared-baseline.md new file mode 100644 index 000000000..4ef9bd840 --- /dev/null +++ b/testing/end-to-end-comparisons/cases/brunch-host-landing/shared-baseline.md @@ -0,0 +1,14 @@ +# Shared baseline: prepared-run landing + +The repository contains an execution system that produces immutable run metadata and a promoted review +reference. A successful execution lane stops at `promotion_prepared`; it does not mutate the host branch. + +The user-facing Brunch TUI is the public interaction surface. The landing workflow is addressed as +`/brunch:land` and requires explicit confirmation after a read-only preflight. + +Landing must work for both: + +- a brownfield run rooted at an existing host commit; and +- a greenfield run whose complete promoted tree is materialized into a missing or empty target. + +Implementation structure, module names, ports, and internal APIs are not part of the contract. diff --git a/testing/end-to-end-comparisons/cases/brunch-host-landing/study-contract.json b/testing/end-to-end-comparisons/cases/brunch-host-landing/study-contract.json new file mode 100644 index 000000000..65ae330d7 --- /dev/null +++ b/testing/end-to-end-comparisons/cases/brunch-host-landing/study-contract.json @@ -0,0 +1,52 @@ +{ + "schemaVersion": 1, + "id": "brunch-host-landing-e2e-v1", + "caseId": "brunch-host-landing-v1", + "mission": { + "path": "testing/comparisons/missions/brunch-host-landing.md", + "sha256": "sha256:1c9925dffbb4afcdb2b209558134d73bd6b0fbe4e78abe21a15d289e1308d89e" + }, + "sharedBaseline": { + "path": "testing/end-to-end-comparisons/cases/brunch-host-landing/shared-baseline.md", + "sha256": "sha256:286e5d6afbfc75a1662d5915647f1b130698b726b00fec377b92d01964baa8f3" + }, + "requirementRegistry": { + "path": "testing/end-to-end-comparisons/cases/brunch-host-landing/controller/requirement-registry.json", + "sha256": "sha256:b21214b0fa9b7dc6e5d602ebaae10fa2e0f136bab49b5e9205af9160e7b0c2e5" + }, + "executionContractTemplate": { + "path": "testing/execution-comparisons/cases/brunch-host-landing/public-contract.json", + "sha256": "sha256:48292ac96c44d91a487292ab9403dcd13adfb70848c8c8b3b9bbdd3aa2c0da18" + }, + "oracle": { + "id": "brunch-host-landing-oracles-v1", + "manifestPath": "testing/execution-comparisons/cases/brunch-host-landing/controller/oracle-manifest.json", + "manifestSha256": "sha256:ae5b9a0a4d7170471583471bd29ad619419d7190be33116d74aabfb487e07c21" + }, + "source": { + "parentCommit": "f5a423b19f76cf345d88053456870a126e451618", + "parentTree": "a5709715a07faef0b96d3e05a7b6f9f8d693dd38" + }, + "budgets": { + "elicitation": { + "qualifyingQuestions": 8, + "targetTurns": 16, + "elapsedMinutes": 45, + "mechanicalInterventions": 2 + }, + "execution": { + "elapsedMinutes": 90, + "mechanicalInterventions": 2, + "substantiveHumanInterventions": 0 + } + }, + "actorRecipes": { + "elicitation": "agent-as-user-comparison/v1", + "execution": { + "brunch": "brunch-pinned-git/v1", + "claude_code": "claude-code-pinned-git/v1" + } + }, + "specSources": ["brunch_spec", "claude_spec"], + "executors": ["brunch", "claude_code"] +} diff --git a/testing/end-to-end-comparisons/cases/petrinaut-optimization/controller/requirement-registry.json b/testing/end-to-end-comparisons/cases/petrinaut-optimization/controller/requirement-registry.json new file mode 100644 index 000000000..3dc19e428 --- /dev/null +++ b/testing/end-to-end-comparisons/cases/petrinaut-optimization/controller/requirement-registry.json @@ -0,0 +1,90 @@ +{ + "schemaVersion": 1, + "caseId": "petrinaut-optimization-v1", + "rows": [ + { + "id": "REQ1", + "publicConcern": "Scenario-first configuration", + "origin": "public_baseline", + "publicWording": "Select a scenario before configuration; changing it resets scenario-specific choices.", + "controller": { + "wording": "Configuration controls are unavailable before a scenario and reset after selecting another.", + "expectedState": "No stale parameter binding or metric remains after the scenario changes." + } + }, + { + "id": "REQ2", + "publicConcern": "Flat parameter bindings", + "origin": "controller_only", + "controller": { + "wording": "One fixed binding and one optimized binding with typed bounds enter the request.", + "expectedState": "The request contains the exact flat binding map configured through public controls." + } + }, + { + "id": "REQ3", + "publicConcern": "Objective metric", + "origin": "controller_only", + "controller": { + "wording": "Saved and custom metric paths each preserve one explicit objective direction.", + "expectedState": "The request contains one metric objective and maximize or minimize direction." + } + }, + { + "id": "REQ4", + "publicConcern": "Request construction", + "origin": "controller_only", + "controller": { + "wording": "The browser sends the complete configured request only to the same-origin proxy.", + "expectedState": "Captured JSON matches the selected scenario, bindings, metric, direction, execution, and study settings." + } + }, + { + "id": "REQ5", + "publicConcern": "Progressive results", + "origin": "controller_only", + "controller": { + "wording": "Deterministic trial, best-so-far, and complete events remain visibly distinct.", + "expectedState": "Progressive trials and current best render before the completed result." + } + }, + { + "id": "REQ6", + "publicConcern": "Service failure", + "origin": "controller_only", + "controller": { + "wording": "A deterministic service failure produces an error state rather than completion.", + "expectedState": "The rendered error is distinct from completed and cancelled results." + } + }, + { + "id": "REQ7", + "publicConcern": "Cancellation", + "origin": "controller_only", + "controller": { + "wording": "Cancel aborts the active proxy request and renders cancellation.", + "expectedState": "The controller observes request abortion and the DOM reports cancelled." + } + }, + { + "id": "REQ8", + "publicConcern": "Private upstream origin", + "origin": "public_baseline", + "publicWording": "Keep the optimizer address private behind the website's same-origin API.", + "controller": { + "wording": "No browser request or rendered text contains the private fake-optimizer origin.", + "expectedState": "Only the candidate origin appears in browser traffic and DOM evidence." + } + }, + { + "id": "AC1", + "publicConcern": "Accessible public workflow", + "origin": "public_baseline", + "publicWording": "Expose the Optimizations view, Create action, and Create an optimization drawer controls on /optimization.", + "controller": { + "wording": "Closed mechanical addresses for Simulate navigation, Create, scenario/metric selectors and metric editor, Maximize/Minimize, Run/Cancel, and Complete/Error/Cancelled remain exact.", + "expectedState": "Every required declared mechanical address resolves exactly once." + } + } + ] +} diff --git a/testing/end-to-end-comparisons/cases/petrinaut-optimization/shared-baseline.md b/testing/end-to-end-comparisons/cases/petrinaut-optimization/shared-baseline.md new file mode 100644 index 000000000..9dc9175be --- /dev/null +++ b/testing/end-to-end-comparisons/cases/petrinaut-optimization/shared-baseline.md @@ -0,0 +1,12 @@ +# Shared baseline: Petrinaut optimization + +The repository contains Petrinaut's editor, scenarios, parameter schemas, saved and custom metrics, and a +pre-existing optimizer API/client protocol. These capabilities are available to inspect and reuse; adding +the optimization user interface is the implementation task. + +The public mechanical address is the Petrinaut website's `/optimization` route. Its browser calls a +same-origin API path which may proxy to a separately configured optimizer. The upstream optimizer origin +must not appear in browser requests or rendered content. + +The execution scope is one frontend feature in the full repository. It does not require the authenticated +HASH application shell, an extracted Petrinaut repository, or a new optimizer backend. diff --git a/testing/end-to-end-comparisons/cases/petrinaut-optimization/study-contract.json b/testing/end-to-end-comparisons/cases/petrinaut-optimization/study-contract.json new file mode 100644 index 000000000..a9d4e0a8d --- /dev/null +++ b/testing/end-to-end-comparisons/cases/petrinaut-optimization/study-contract.json @@ -0,0 +1,52 @@ +{ + "schemaVersion": 1, + "id": "petrinaut-optimization-e2e-v1", + "caseId": "petrinaut-optimization-v1", + "mission": { + "path": "testing/comparisons/missions/petrinaut-optimization.md", + "sha256": "sha256:89daf3507df732405bad528e507328aede01b5125383c499aef8b97fff8b92b0" + }, + "sharedBaseline": { + "path": "testing/end-to-end-comparisons/cases/petrinaut-optimization/shared-baseline.md", + "sha256": "sha256:977947acf91d236397174a247436b77db0164f5b0deaf63ce78d6d8f12a46b9e" + }, + "requirementRegistry": { + "path": "testing/end-to-end-comparisons/cases/petrinaut-optimization/controller/requirement-registry.json", + "sha256": "sha256:9e8866d4a2b96a2e43c261f8514c349582547db958978cf13eb3a4654b73cd02" + }, + "executionContractTemplate": { + "path": "testing/execution-comparisons/cases/petrinaut-optimization/public-contract.json", + "sha256": "sha256:a8dedb7d6be4dfb706e47194de3bb32caf7724e2eb34f808c8a9ab139c235312" + }, + "oracle": { + "id": "petrinaut-optimization-oracles-v1", + "manifestPath": "testing/execution-comparisons/cases/petrinaut-optimization/controller/oracle-manifest.json", + "manifestSha256": "sha256:7f5050a0bae848023b01b68b22993934fe3bac691a90748659d894bfc76dd939" + }, + "source": { + "parentCommit": "5c7a2d9db5caa851c38938f4b1bac19005b0e978", + "parentTree": "a3e08cf75e00cc9016c931f4665341506e03533e" + }, + "budgets": { + "elicitation": { + "qualifyingQuestions": 8, + "targetTurns": 16, + "elapsedMinutes": 45, + "mechanicalInterventions": 2 + }, + "execution": { + "elapsedMinutes": 90, + "mechanicalInterventions": 2, + "substantiveHumanInterventions": 0 + } + }, + "actorRecipes": { + "elicitation": "agent-as-user-comparison/v1", + "execution": { + "brunch": "brunch-pinned-git/v1", + "claude_code": "claude-code-pinned-git/v1" + } + }, + "specSources": ["brunch_spec", "claude_spec"], + "executors": ["brunch", "claude_code"] +} diff --git a/testing/execution-comparisons/cases/brunch-host-landing/controller/oracle-manifest.json b/testing/execution-comparisons/cases/brunch-host-landing/controller/oracle-manifest.json new file mode 100644 index 000000000..63ebd0188 --- /dev/null +++ b/testing/execution-comparisons/cases/brunch-host-landing/controller/oracle-manifest.json @@ -0,0 +1,32 @@ +{ + "schemaVersion": 1, + "id": "brunch-host-landing-oracles-v1", + "publicCaseId": "brunch-host-landing-v1", + "runnerVersion": "brunch-host-landing-v1", + "referenceModelVersion": "git-full-range-v1", + "checks": [ + { + "id": "public-tui-preflight", + "claims": ["AC1", "INV1", "REQ1"] + }, + { + "id": "brownfield-full-range", + "claims": ["REQ2"] + }, + { + "id": "greenfield-materialization", + "claims": ["REQ3"] + }, + { + "id": "refusal-safety", + "claims": ["REQ4"] + } + ], + "validityRules": [ + "The candidate lane terminates at promotion_prepared before the controller creates any landing repository.", + "The controller resumes a supplied settled session under PI_OFFLINE=1; a fresh session or provider turn is setup_failed.", + "The controller invokes only the built candidate public TUI and inspects terminal, filesystem, Git, and retained run metadata.", + "Only disposable controller-created repositories may reach landed." + ], + "replacementRule": "Retain every report. Replace only harness setup invalidity under the same frozen packets; never replace an assertion failure." +} diff --git a/testing/execution-comparisons/cases/brunch-host-landing/public-contract.json b/testing/execution-comparisons/cases/brunch-host-landing/public-contract.json new file mode 100644 index 000000000..b8bebad7e --- /dev/null +++ b/testing/execution-comparisons/cases/brunch-host-landing/public-contract.json @@ -0,0 +1,38 @@ +{ + "schemaVersion": 1, + "case": { + "id": "brunch-host-landing-v1", + "specification": "spec.md", + "specificationSha256": "80dc03a071ee5bd03740afcecd90387e6d5e01c7d163d9ddf5d870599be181f3", + "provider": "anthropic", + "model": "claude-opus-4-8", + "product": "brunch", + "mode": "brownfield", + "scope": "single_feature", + "surface": "backend", + "repository": { + "substrate": "pinned_git", + "parentCommit": "f5a423b19f76cf345d88053456870a126e451618", + "parentTree": "a5709715a07faef0b96d3e05a7b6f9f8d693dd38" + } + }, + "budgets": { + "elapsedMinutes": 90, + "mechanicalInterventions": 2, + "substantiveHumanInterventions": 0 + }, + "delivery": { + "runtimeNetwork": "forbidden", + "dependencyInstallNetwork": "forbidden" + }, + "acceptance": { + "publicCommand": "/brunch:land", + "executionTerminal": "promotion_prepared" + }, + "rules": [ + "Work only in the target repository.", + "Do not inspect controller paths or seek hidden comparison material.", + "Do not use network access or external historical references.", + "Stop after the run reaches promotion_prepared; do not land it." + ] +} diff --git a/testing/execution-comparisons/cases/brunch-host-landing/spec.md b/testing/execution-comparisons/cases/brunch-host-landing/spec.md new file mode 100644 index 000000000..e5366912c --- /dev/null +++ b/testing/execution-comparisons/cases/brunch-host-landing/spec.md @@ -0,0 +1,35 @@ +# Host landing + +### G1 Land a prepared run safely + +Give the user one explicit TUI workflow for reviewing and landing a prepared run without allowing the +execution lane or an agent-callable tool to mutate the host. + +### REQ1 Complete-range review + +Before confirmation, show the full commit and path range from the run base through the promoted review +tip. Inspection must leave the host branch, tree, worktree, and run metadata unchanged. + +### REQ2 Complete brownfield landing + +After confirmation, land every content change in the promoted range into the active repository. Preserve +the review reference, exclude `.brunch` bookkeeping from tracked output, and record the run as landed. + +### REQ3 Greenfield materialization + +For a greenfield run, materialize the complete promoted tip tree into a missing or empty target as one +clean Brunch-authored initial commit. + +### REQ4 Refusal safety + +Decline, dirty-host, conflict, and stale-acceptance outcomes leave the host unchanged and do not record +the run as landed. + +### INV1 User authority + +Host mutation occurs only after an interactive confirmation bound to the exact promoted commit shown by +preflight. + +### AC1 Public TUI behavior + +The workflow is exercised through `/brunch:land` in the real public TUI without a provider turn. diff --git a/testing/execution-comparisons/cases/petrinaut-optimization/controller/oracle-manifest.json b/testing/execution-comparisons/cases/petrinaut-optimization/controller/oracle-manifest.json new file mode 100644 index 000000000..6181698be --- /dev/null +++ b/testing/execution-comparisons/cases/petrinaut-optimization/controller/oracle-manifest.json @@ -0,0 +1,45 @@ +{ + "schemaVersion": 1, + "id": "petrinaut-optimization-oracles-v1", + "publicCaseId": "petrinaut-optimization-v1", + "runnerVersion": "petrinaut-optimization-browser-v1", + "fixtureVersion": "deterministic-optimizer-v1", + "checks": [ + { + "id": "route-and-accessibility", + "claims": ["AC1"] + }, + { + "id": "scenario-configuration", + "claims": ["REQ1"] + }, + { + "id": "request-contract", + "claims": ["REQ2", "REQ3", "REQ4"] + }, + { + "id": "progress-and-completion", + "claims": ["REQ5"] + }, + { + "id": "service-error", + "claims": ["REQ6"] + }, + { + "id": "cancel-and-abort", + "claims": ["REQ7"] + }, + { + "id": "private-origin-secrecy", + "claims": ["REQ8"] + } + ], + "validityRules": [ + "Before candidate execution, the controller materializes the pinned source and completes the compiled immutable dependency preparation; only that declared step may use package-registry network.", + "The network-denied candidate execution lane terminates at promotion_prepared before browser launch.", + "After lane termination, the controller runs only the compiled focused build preparation and starts the candidate Petrinaut website.", + "The browser opens /optimization and sends optimization traffic only through the candidate same-origin proxy.", + "Controller fixtures and expected requests are created after lane termination in roots disjoint from the candidate." + ], + "replacementRule": "Retain every report. Replace only setup invalidity under the same frozen packets; never replace an assertion failure." +} diff --git a/testing/execution-comparisons/cases/petrinaut-optimization/public-contract.json b/testing/execution-comparisons/cases/petrinaut-optimization/public-contract.json new file mode 100644 index 000000000..59c3aeed5 --- /dev/null +++ b/testing/execution-comparisons/cases/petrinaut-optimization/public-contract.json @@ -0,0 +1,137 @@ +{ + "schemaVersion": 1, + "case": { + "id": "petrinaut-optimization-v1", + "specification": "spec.md", + "specificationSha256": "3fb0772f0e955f81240b58af9f53736393b1f8dccf55315371fc5755e801dc43", + "provider": "anthropic", + "model": "claude-opus-4-8", + "product": "petrinaut", + "mode": "brownfield", + "scope": "single_feature", + "surface": "frontend", + "repository": { + "substrate": "pinned_git", + "parentCommit": "5c7a2d9db5caa851c38938f4b1bac19005b0e978", + "parentTree": "a3e08cf75e00cc9016c931f4665341506e03533e" + } + }, + "budgets": { + "elapsedMinutes": 90, + "mechanicalInterventions": 2, + "substantiveHumanInterventions": 0 + }, + "delivery": { + "runtimeNetwork": "forbidden", + "dependencyInstallNetwork": "controller_only" + }, + "acceptance": { + "publicRoute": "/optimization", + "sameOriginApi": "/api/petrinaut-opt/optimize/all", + "executionTerminal": "promotion_prepared" + }, + "mechanicalAddresses": { + "skipTour": { + "kind": "roleName", + "role": "button", + "name": "Skip tour" + }, + "dismissAssistant": { + "kind": "roleName", + "role": "button", + "name": "Dismiss" + }, + "simulateMode": { + "kind": "roleName", + "role": "radio", + "name": "Simulate" + }, + "optimizationsNav": { + "kind": "roleValue", + "role": "radio", + "value": "optimizations" + }, + "viewTitle": { + "kind": "exactText", + "text": "Optimizations" + }, + "create": { + "kind": "roleName", + "role": "button", + "name": "Create" + }, + "createDrawer": { + "kind": "roleName", + "role": "dialog", + "name": "Create an optimization" + }, + "scenario": { + "kind": "roleContents", + "role": "combobox", + "contents": "Select a scenario" + }, + "scenarioSelected": { + "kind": "roleContents", + "role": "combobox", + "contents": "Seasonal Flu" + }, + "metric": { + "kind": "roleContents", + "role": "combobox", + "contents": "Select a metric" + }, + "metricCustomOption": { + "kind": "roleName", + "role": "option", + "name": "Custom code" + }, + "metricCode": { + "kind": "roleName", + "role": "textbox", + "name": "Editor content" + }, + "optimizationName": { + "kind": "roleName", + "role": "textbox", + "name": "Name" + }, + "directionMaximize": { + "kind": "roleName", + "role": "radio", + "name": "Maximize" + }, + "directionMinimize": { + "kind": "roleName", + "role": "radio", + "name": "Minimize" + }, + "run": { + "kind": "roleName", + "role": "button", + "name": "Run" + }, + "cancel": { + "kind": "roleName", + "role": "button", + "name": "Cancel" + }, + "statusComplete": { + "kind": "exactText", + "text": "Complete" + }, + "statusError": { + "kind": "exactText", + "text": "Error" + }, + "statusCancelled": { + "kind": "exactText", + "text": "Cancelled" + } + }, + "rules": [ + "Work only in the target repository.", + "Do not inspect controller paths or seek hidden comparison material.", + "Do not use network access or external references during implementation or verification.", + "Preserve the full repository and stop at promotion_prepared without landing." + ] +} diff --git a/testing/execution-comparisons/cases/petrinaut-optimization/spec.md b/testing/execution-comparisons/cases/petrinaut-optimization/spec.md new file mode 100644 index 000000000..f9f754761 --- /dev/null +++ b/testing/execution-comparisons/cases/petrinaut-optimization/spec.md @@ -0,0 +1,51 @@ +# Petrinaut optimization + +### G1 Configure and run an optimization + +Add a capability-present Optimizations view to Petrinaut's focused website route. The workflow must make +scenario selection explicit before optimization configuration and expose run progress without revealing +the private optimizer service. + +### REQ1 Scenario-first configuration + +Require the user to select a scenario before configuring an optimization. Changing the scenario resets +parameter and metric configuration that belongs to the previous scenario. + +### REQ2 Flat parameter bindings + +Show every selected scenario parameter in one flat configuration. Each parameter can be fixed at one +valid typed value or optimized over a valid typed domain. + +### REQ3 Objective metric + +Allow exactly one saved model metric or run-local custom metric as the objective, with an explicit +maximize or minimize direction. + +### REQ4 Request construction + +Starting a run sends the chosen scenario, fixed and optimized parameter bindings, metric objective and +direction, and execution/study settings through the website's same-origin optimization API. + +### REQ5 Progressive results + +Render streamed trials, best-so-far values, and successful completion as distinct progressive states. + +### REQ6 Service failure + +Render an optimizer service failure distinctly from successful completion and user cancellation. + +### REQ7 Cancellation + +Allow an active run to be cancelled. Cancellation aborts the browser's in-flight same-origin request and +renders a cancelled state. + +### REQ8 Private upstream origin + +Browser traffic and rendered content must not expose or contact the private upstream optimizer origin. + +### AC1 Accessible public workflow + +The `/optimization` route exposes an Optimizations view title and Create action. Opening Create reveals a +Create an optimization drawer whose scenario selector, metric selector, Maximize/Minimize direction +controls, Run/Cancel actions, and Complete/Error/Cancelled status text use the closed mechanical +addresses declared by the public contract.