feat(eval-harness): minimal live eval harness v0

.gitignore

@@ -11,6 +11,7 @@ vendor/
 nohup.out
 dumps/
 perf/reports/
+*.eval-results/
 TODO.md
 docs/inbox/
 .portal.pids

docs/proposals-impl/README.md

@@ -7,6 +7,7 @@ They are kept as historical design records, not as the active backlog. New or st
 Current implemented proposals:
 
 - [Cron Tool — Declarative Recurring Schedules](./cron-tool.md)
+- [Eval Harness](./eval-harness.md)
 - [Management Client and TUI Boundary Cleanup](./management-client-boundary-cleanup.md)
 - [npm Packaging and Embedded PilotSwarm Plugins](./npm-packaging-and-embedded-plugins.md)
 - [Prompt Layering and Framework Precedence](./prompt-layering-and-precedence.md)

docs/proposals-impl/eval-harness.md

@@ -0,0 +1,42 @@
+# Eval Harness
+
+Implemented status: `packages/eval-harness` v0.
+
+This proposal record now matches the minimal live PR scope. The active package
+docs live under `packages/eval-harness/`.
+
+## v0 Scope
+
+The v0 surface is intentionally small:
+
+- `live-smoke`
+- `live-critical-path`
+- `live-all` as the full bundled v0 corpus
+- 19 live-compatible JSON scenarios across runtime, durable, multi-turn,
+  and safety coverage — every scenario exercises a real PilotSwarm runtime
+  feature (CMS event capture, durable waits, worker-restart chaos,
+  multi-turn session memory, or provider-backed judge/safety integration)
+- console reporting
+- provider-backed LLM judge checks where scenarios request them
+
+Out of scope for v0: meta scenarios, prompt variants, ablations, model sweeps,
+sample expansion, post-run trajectory summaries, expanded reporters, and broad
+platform positioning.
+
+## Model
+
+```text
+Run config -> manifest -> scenario config
+```
+
+Run config owns driver, isolation, concurrency, timeout,
+reporters, output, and judge policy. Manifests select scenario files. Scenario
+files own prompt, tools, checks, per-scenario timeout, and chaos injection.
+
+## Kept Live Path
+
+The bundled plans run against managed live PilotSwarm workers and require
+`GITHUB_TOKEN`, `DATABASE_URL`, and PostgreSQL. The critical path covers basic
+runtime/CMS evidence, durable wait recovery, wait-tool-wait-tool sequencing,
+timer recovery after worker restart, multi-turn session memory, and safety
+behavior including direct/indirect injection and output-secret refusal.

package.json

@@ -13,7 +13,7 @@
     "test:mcp-server": "npm test --workspace=pilotswarm-mcp-server",
     "test:mcp-server:integration": "npm run test:integration --workspace=pilotswarm-mcp-server",
     "test:mcp-server:integration:all": "npm run test:integration:all --workspace=pilotswarm-mcp-server",
-    "build": "npm run build --workspace=packages/sdk && npm run build --workspace=pilotswarm-cli && npm run build --workspace=pilotswarm-mcp-server && npm run build --workspace=pilotswarm-web && npm run build --workspace=pilotswarm-ui-core && npm run build --workspace=pilotswarm-ui-react",
+    "build": "npm run build --workspace=packages/sdk && npm run build --workspace=pilotswarm-cli && npm run build --workspace=pilotswarm-mcp-server && npm run build --workspace=pilotswarm-web && npm run build --workspace=pilotswarm-ui-core && npm run build --workspace=pilotswarm-ui-react && npm run build --workspace=pilotswarm-eval-harness",
     "dev": "npm run dev --workspace=packages/sdk",
     "test": "npm test --workspace=packages/sdk",
     "test:local": "npm run test:local --workspace=packages/sdk",

packages/eval-harness/.eslintrc.cjs

@@ -0,0 +1,7 @@
+module.exports = {
+  root: true,
+  parserOptions: {
+    ecmaVersion: "latest",
+    sourceType: "module",
+  },
+};

packages/eval-harness/.gitignore

@@ -0,0 +1,3 @@
+dist/
+node_modules/
+.eval-results/

packages/eval-harness/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Affan Dar and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.