Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/tricky-poems-do.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/brunch": patch
---

Add a deterministic full-stack prospect research regression case ([#369](https://github.com/hashintel/brunch/pull/369)).
7 changes: 7 additions & 0 deletions docs/praxis/comparison-runs.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,13 @@ audience-safe requirement ledger, and the bounded validity-first report.
This historical witness predates Alpha 10 provenance capture. Keep it as evidence and an example, but
do not backfill `provenance.json` or publish it through the current publication skill.

FE-1253 adds `prospect-research-workspace` as a deterministic full-stack regression case, not a second
end-to-end campaign profile. Its fixed public stack is React + Node.js + TypeScript + SQLite, with
server-side Pi/Clay-compatible fixtures and denied runtime network. The saved mission remains available
for exploratory specification work, while the compiled browser/HTTP/SQLite/export oracle is calibrated
against a known-good implementation and focused wrong rivals. A future campaign may compose this case
only through a separately authorized study contract.

## Agent recipe: drive, then join evidence

On an overlay-capable host, drive the real TUI with the pinned project-local `pi-interactive-shell` package. In a sandbox or headless environment, use `npm run tui-driver` as the fallback. Follow the bounded observation, named-key input, and deterministic cleanup protocol in [Manual Testing](manual-testing.md).
Expand Down
30 changes: 27 additions & 3 deletions memory/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Brunch-next has delivered the original composition spine: the host, sealed Pi pr

**Parallel lanes.** Group 3 agent-layer work is pickup-ready. `agent-control-plane-closure` is one earned coverage frontier over foreground and background prompt ingress: it absorbs the former `graph-assurance-conduct` and `subagent-skill-access` items, reconciles the stale eager-context fossils left by D58-L's load-on-demand cutover, and closes prompt replacement/resource-invocation contradictions before capture experiments amplify them. FE-1187 retains only the concern-grouped review renderer. FE-1208's reshaped `automation-observability-dx` closure landed on the restacked base; its invalid warrant pilots remain historical evidence, while the useful evaluator/report primitives are available to `capture-ledger-tracer`. Two evaluation use cases stay deliberately distinct: **seed-based intra-product testing** (Brunch on its own terms — seeds encode Brunch graph/spec state only Brunch understands) and **mission-driven agent-as-user cross-product comparison** (`agent-as-user-comparison` — competitor CLIs share no Brunch state, so the comparable artifact is the "ready" spec/plan document a mission produces). FE-1210 closed the rigorous technical tracer on 2026-07-17; the separately planned operator workflow (`operator-comparison-workflow`, FE-1215) makes saved missions, visible comparison-harness framing, conversational launch, and a readable free-form report approachable without weakening the retained regression/evaluation machinery. Its first operator-led run falsified the nested fresh-actor topology: the top-level project Pi session must act as the simulated user and drive one direct harness subshell at a time, with stock-Pi text interaction as the portable baseline; the corrected real witness is deferred to `saved-mission-comparison-witness`. The KA stream owns executor/orchestrator/Execute-mode work. FE-1192 and FE-1195 completed the attempt, isolated fan-in, durable parallel authority, and epic integration sequence; `executor-plan-coherence` now uses those settled seams to test whether multi-slice plans converge on one working result. Broader instrumentation remains trigger-gated under Later.

**End-to-end comparison lane.** FE-1239 materializes the handoff that the completed reporting procedure names but does not run: two fresh rigorous elicitation lanes produce exact approved specifications, each specification crosses an immutable content-addressed handoff into both Brunch and Claude execution, and the unchanged Petri-editor oracle joins all four outputs into requirement-level evidence. Shared interface requirements are disclosed before elicitation and remain controlled baseline, while private/reveal requirements stay controller-only. The first tracer supports bounded within-case contrasts only — no winner, reliability claim, or cross-case causal conclusion. FE-1241 keeps that Petri editor as the greenfield case and adds frozen Brunch/Petrinaut brownfield packets plus controller-owned oracles. All three missions can now enter the same execution-only workflow and publication contract. Brownfield preparation uses pinned remote-free snapshots and target-bounded tools as experimental hygiene, not adversarial benchmark proof; Clay remains outside the campaign.
**Comparison lane.** FE-1239 remains the sole rigorous greenfield 2×2 tracer: two independently approved Petri specifications cross exact handoffs into Brunch and Claude execution and one unchanged oracle joins all four outputs. FE-1241 keeps that Petri editor as the greenfield study profile and adds frozen Brunch/Petrinaut brownfield packets plus deterministic oracles under a learning-first workflow. FE-1253 is deliberately thinner: a saved exploratory prospect mission plus a deterministic React/Node/TypeScript/SQLite execution regression case, with no prospect study profile, requirement ledger, scored provider gate, or campaign claim.

**Current seams.** Brunch ships on the `1.0.0-alpha.x` line. One-shot `ask` is the only interactive structured-exchange terminal; D125-L's live ask registry provides headless discovery/answering, while the transcript-backed pending projection remains a compatibility surface for live offer tools after the legacy `present_question` pending branch retired. Sweep classification remains fail-closed and compile-time anchored to the exchange-schema terminal names (D117-L), while the larger capture-conditional watermark question remains A40-L.

Expand Down Expand Up @@ -119,6 +119,7 @@ Everything executor/orchestrator-shaped or Execute-mode-owned belongs to Kostand
- `executor-slice-admission-parity` ([FE-1240](https://linear.app/hash/issue/FE-1240/prevent-invalid-scoped-slices-from-reaching-execution)) — **complete on `ka/fe-1240-slice-admission-parity`, restacked on `next` after FE-1239 landed:** incomplete scoped-slice worker context is rejected during deterministic plan admission, exact findings enter bounded repair before `slice_execute`, and the execution boundary remains fail-closed. Definition below.
- `executor-plan-coherence` ([FE-1250](https://linear.app/hash/issue/FE-1250/build-coherent-execution-plans)) — **implementation complete on `ka/fe-1250-coherent-execution-plans`, based on FE-1240 via `next`:** frontier-verified multi-slice epics now require one ordinary terminal member over every sibling, planner conduct establishes shared foundations, and workers preserve cumulative public contracts for the canonical harness. Fast verification passes; next is the separately authorized unchanged Petri comparison rerun. No new browser gate, durable plan kind, or executor lifecycle phase. Definition below.
- `comparison-publication-workflow` ([FE-1251](https://linear.app/hash/issue/FE-1251/publish-traceable-comparison-reports)) — **active on `ka/fe-1251-comparison-publication`, stacked on FE-1250:** capture immutable controller and release provenance before comparison lanes start, then explicitly publish retained validity-first reports into the canonical Notion database through a guarded idempotent skill. Definition below.
- `prospect-research-workspace-regression` ([FE-1253](https://linear.app/hash/issue/FE-1253/thin-prospect-research-into-a-regression-case)) — **active on `ka/fe-1253-prospect-research-e2e`, stacked on FE-1241:** retain the exploratory mission and calibrated browser/HTTP/SQLite/export oracle while deleting the unexecuted prospect study profile, requirement-ledger expansion, and scored-provider completion gate.
- **Carved from FE-1167 (2026-07-13):** the Execute-mode evidence sub-list — Execute entry beats on thin vs rich seeds (assessment honesty: Ask on thin, Proceed on rich) and the FE-1107/KA residue (close-or-narrow, demo/walkthrough session via `TESTING_PLAN.md`, post-KA plan pass). The former sticky-posture question is no longer KA residue: FE-1187 `remediation-4` owns the persistent Specify elicitation-style audit/SPEC revision, and its Continue lexical audit owns the old `continue` ambiguity. Full context in the archived FE-1167 definition (`docs/archive/PLAN_HISTORY.md`).
- `planning-process-model` — **moved to the KA stream 2026-07-13; reshaped by D126-L**: the durable scope handoff is settled, so this item now owns only plan projection and epistemic-horizon questions beyond committed scopes. Definition below.
- **[1.x data-model handoff owed by FE-1187](../docs/architecture/BRUNCH_1X_DATA_MODEL_HANDOFF.md):** a concise colleague-facing note must distinguish current canon from directional vocabulary before the consolidated outer checkpoint. Current: `{milestone, frontier, scope}` with executor-derived slices; basis (`explicit | implicit`) orthogonal to settlement (`advisory | settled`); no persisted readiness grade or spec-global elicitation-gap table; active-branch Pi JSONL reads; one CommandExecutor mutation authority with spec-local LSN/change log; no new projected `vv_obligation` (legacy rows remain readable). Directional only: explain stored `thesis` as pitch/concept without renaming it yet, and avoid new coupling to spec-local `term` while its possible workspace lift remains future work. Persisted judgment-shaped reconciliation needs remain current, but are YAGNI-suspect: add no kinds/consumers/orchestration dependency without fresh evidence; re-evaluate derivation/removal when KA work first needs that table. Delivered note: [`Brunch 1.x data-model handoff`](../docs/architecture/BRUNCH_1X_DATA_MODEL_HANDOFF.md); it does not block the deterministic TUI queue.
Expand Down Expand Up @@ -456,6 +457,25 @@ Instrumentation experiments and far-horizon items. Each re-enters only via re-qu
- **Boundary:** no hidden-oracle exposure, inferred browser command, new plan-plane node, candidate command surface, execution-time plan repair, or FE-1241 comparison-framework change.
- **Current execution pointer:** none; the two-card scope is consumed. Re-enter only for the unchanged frozen Petri comparison rerun.

### prospect-research-workspace-regression

- **Name:** Thin prospect research into a regression case
- **Linear / branch:** [FE-1253](https://linear.app/hash/issue/FE-1253/thin-prospect-research-into-a-regression-case); `ka/fe-1253-prospect-research-e2e`, stacked on FE-1241 with no parent issue.
- **Kind:** earned evaluation simplification — retain a deterministic full-stack regression oracle and retire unproven campaign expansion.
- **Certainty:** earned.
- **Status:** active 2026-07-23. The mission, public packet, opaque Brunch seed, closed compiled oracle, known-good full stack, and focused rivals remain. The prospect-specific end-to-end study profile and provider campaign gate are being retired before any scored lane exists.
- **Objective:** provide one deterministic implementation-level acceptance case for the prospect research workflow without making ordinary regression value depend on a rigorous 2×2 campaign.
- **Closes:** the accidental coupling between a useful full-stack oracle and an unexecuted prospect-specific campaign.
- **Deletes / retires:** the prospect end-to-end study contract, shared baseline, reveal registry, matrix registration, requirement-ledger obligation, scored-provider gate, and campaign claims.
- **Stabilizes:** the fixed React + Node.js + TypeScript + SQLite public packet; `npm test` / `npm run build` / `npm start`; fresh database and fixture isolation; runtime-network denial; independent browser/API/SQLite/export journeys; focused rivals; and opaque specification seeding.
- **Depends on:** FE-1241 for the finalized execution-case registry and oracle dispatch only. No provider or strict greenfield Claude-isolation witness blocks completion.
- **Boundary:** manually initiated prospect research, evidence-backed qualification, deduplication/provenance, suppression, review, audited override, approval, export, provider failure, and restart persistence are in. Outreach delivery, live Pi/Clay quality, campaign composition, scored provider lanes, reliability claims, and `ExecutionAttempt` widening are out.
- **Acceptance:** the known-good full stack passes; focused rivals for unapproved research, confidence-only qualification, lost provenance, weak suppression, reasonless/destructive override, overbroad export, provider-failure laundering, non-durable state, and external runtime requests fail their owning claims; the prospect case is absent from end-to-end study registration; existing Petri and brownfield study bytes/oracles remain unchanged.
- **Verification:** inner — exact public/oracle parsers, oracle-pack hash, claim coverage, and opaque Brunch seed. Middle — independent browser + API + SQLite journeys over fresh database/fixture state, paired with the controller reference model and focused rivals. No outer campaign evidence is required.
- **Cross-cutting obligations:** controller fixtures and expected states remain outside targets; runtime network stays denied; compiled oracle dispatch remains fail-closed; historical comparison evidence remains unchanged.
- **Traceability:** D70-L, D139-L; FE-1230/FE-1241; [`docs/praxis/comparison-runs.md`](../docs/praxis/comparison-runs.md).
- **Current execution pointer:** thin the materialized branch to this regression boundary, then close through deterministic verification.

### comparison-reporting-skills

- **Name:** Report comparison evidence
Expand Down Expand Up @@ -647,8 +667,12 @@ KA stream:
-[hard]-> brownfield-comparison-cases
status: complete FE-1241 with learning-first pinned snapshots, deterministic case oracles, and publication-compatible execution runs
lights_up: same-base brownfield elicitation -> exact handoff -> crossed execution -> case oracle
cases: minimal-petri-net-editor = sole greenfield reference | FE-1201-derived brunch backend | PR #9051 petrinaut frontend
excludes: Clay | more greenfields | repetitions | aggregate winner | ExecutionAttempt widening
cases: minimal-petri-net-editor = FE-1241 greenfield reference | FE-1201-derived brunch backend | PR #9051 petrinaut frontend
excludes: Clay | more greenfields inside FE-1241 | repetitions | aggregate winner | ExecutionAttempt widening
-[hard]-> prospect-research-workspace-regression (FE-1253)
status: active; deterministic full-stack oracle calibrated, campaign expansion retiring
closes: useful regression oracle coupled to an unexecuted 2x2 campaign
excludes: provider lanes | requirement ledger | outreach delivery | reliability claims
# executor-run-environment (FE-1166) resolved 2026-07-15: policy merged (PR #302),
# live remainder folded into executor-plan-synthesis (FE-1197), card consumed.

Expand Down
Loading
Loading