feat(lab): CL-07 task effectiveness evidence

devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md

@@ -26,7 +26,7 @@ independent review, blockers, and whether a later phase is authorized.
 | CL-04 | `feat/cl-04-lab-read-surfaces` | `68c71a4e9cdf882d812f09fd94783a28749db629` | NOT RECORDED | [#1378](https://github.com/lidge-jun/opencodex/pull/1378) | MERGED TO `dev` at `d517161aeaa3a974ad3c0360ff0c97b03b4c4520` |
 | CL-05 | `feat/cl-05-compatibility-matrix-ui` | `d517161aeaa3a974ad3c0360ff0c97b03b4c4520` | `2a159b8b7` (Models tab placement) | [#1384](https://github.com/lidge-jun/opencodex/pull/1384) | MERGED TO `dev` at `1072b9c39c48a4982229131613ac300560740742` |
 | CL-06 | `feat/cl-06-routing-profile-compatibility` | `1072b9c39c48a4982229131613ac300560740742` | `b96eae83f2a6d1654472aeeef84799070743aeb8` | [#1394](https://github.com/lidge-jun/opencodex/pull/1394) | MERGED TO `dev` at `b66e33ce7207d91014644d99317e456c992a3418`; ACCEPTED/CLOSED |
-
+| CL-07 | `feat/cl-07-task-effectiveness-producer` | `b66e33ce7207d91014644d99317e456c992a3418` | NOT RECORDED | [#1438](https://github.com/lidge-jun/opencodex/pull/1438) | DRAFT / PENDING CI; implementation head `ed8e0794189b214398d835e1af828905f29fbc53`; plan `007_cl07_task_effectiveness.md` |
 The CL-01 starting SHA is the exact CL-00 tip recorded when CL-01 began. Its
 moving base-ref name is not a substitute for that historical SHA.
 
@@ -162,7 +162,20 @@ Claims cannot produce `PROBED`/`VERIFIED`.
 - CL-03: **ACCEPTED/CLOSED** via [#1352](https://github.com/lidge-jun/opencodex/pull/1352), merged to `dev` at `68c71a4e9cdf882d812f09fd94783a28749db629`.
 - CL-04: **MERGED** via #1378 at `d517161aeaa3a974ad3c0360ff0c97b03b4c4520`.
 - CL-05: **MERGED** via #1384 at `1072b9c39c48a4982229131613ac300560740742`.
-- CL-06: **AUTHORIZED / IN PROGRESS** from CL-05 merge `1072b9c39c48a4982229131613ac300560740742`; branch `feat/cl-06-routing-profile-compatibility`; plan `006_cl06_routing_compatibility.md`.
+- CL-06: **ACCEPTED/CLOSED** via [#1394](https://github.com/lidge-jun/opencodex/pull/1394), merged to `dev` at `b66e33ce7207d91014644d99317e456c992a3418`.
+- CL-07: **AUTHORIZED / IN PROGRESS** from CL-06 merge `b66e33ce7207d91014644d99317e456c992a3418`; branch `feat/cl-07-task-effectiveness-producer`; plan `007_cl07_task_effectiveness.md`.
+- CL-08: **not started**.
+
+## CL-07 start log
+
+- **Starting/base SHA:** `b66e33ce7207d91014644d99317e456c992a3418` (exact CL-06 merge #1394)
+- **Branch:** `feat/cl-07-task-effectiveness-producer`
+- **Scope:** bounded Lab-owned task-effectiveness producer for `fabric-core` /
+  `fabric-core.task.synthetic-patch@1.0.0`; `exact-tree-diff-v1` verifier;
+  scratch sandbox; observation ingestion with `executionMode: fabric`; catalog
+  discovery for the task layer. No general Agent Fabric product API.
+- **Explicitly out of scope:** CL-08 automation/background execution; CL-06
+  routing semantic changes; user repositories/prompts; arbitrary shell.
 
 ## CL-03 implementation log (2026-08-09)
 

devlog/_plan/260807_compatibility_lab/007_cl07_task_effectiveness.md

@@ -0,0 +1,281 @@
+# CL-07 implementation record — Task-effectiveness evidence producer
+
+## Programme position
+
+| Field | Value |
+|---|---|
+| **Phase** | CL-07 |
+| **Starting SHA** | `b66e33ce7207d91014644d99317e456c992a3418` (CL-06 merge #1394) |
+| **Branch** | `feat/cl-07-task-effectiveness-producer` |
+| **Target** | `lidge-jun/opencodex:dev` |
+| **CL-08** | **Not started** (explicit non-goal) |
+
+## 0. Audit summary (repository reality)
+
+Inspected at starting SHA `b66e33ce7207d91014644d99317e456c992a3418`:
+
+| Area | Reality |
+|---|---|
+| `TaskSubjectV1` | Defined and validated (`src/lab/events/types.ts`, `validate.ts`); query DTO mapped |
+| Agent Fabric product | **Absent**. Only `devlog/_plan/800_agent-fabric/` planning notes and CL-00 reserved consumer semantics |
+| Lab ledger / artifacts / projection | Present; `task_effectiveness` already a legal evidence layer |
+| CL-04 catalog | Protocol + live only — no fabric suite discovery yet |
+| CL-05 matrix | Already lists `task_effectiveness` column; read-only |
+| CL-06 routing | `requiredSuites.evidenceLayer` is **only** `protocol_conformance` \| `live_route_compatibility` |
+| Live sandbox | Counter/env limits for probes; **not** a synthetic scratch-tree executor |
+| `fabric-core` / `synthetic-patch` / `exact-tree-diff-v1` | Reserved in `020_scenario_contract_and_catalogue.md`; **no runtime implementation** |
+
+**Decision:** CL-07 implements a **bounded Lab-owned task producer** under `src/lab/fabric/`. It does **not** invent a general Agent Fabric platform, ACP/A2A orchestration, background grind, or user-worktree execution.
+
+## A. Frozen producer boundary
+
+### A.1 Subject identity
+
+Evidence uses existing `TaskSubjectV1` only (no alternate task identity):
+
+```text
+subjectSchemaVersion      1
+subjectKind               task
+routeSubject              RouteSubjectV1   # exact nested route from CL-03/CL-06 builders
+taskClassId
+taskClassVersion
+taskFixtureDigest
+verifierManifestDigest
+fabricCompatibilityVersion
+sandboxProfileDigest
+```
+
+`subjectId = subjectIdForSubject(taskSubject)` (existing digest helper).
+
+Any material change to route, task class, fixture, verifier, fabric compatibility version, or sandbox profile yields a distinct subject. Evidence must not reuse across subjects.
+
+### A.2 Producer outcome schema (`FabricTaskOutcomeV1`)
+
+Closed, fail-closed on unknown fields. Schema version `1`.
+
+Required fields:
+
+| Field | Role |
+|---|---|
+| `schemaVersion` | `1` |
+| `taskClassId` / `taskClassVersion` | Exact class |
+| `routeSubject` / `taskSubject` / `subjectId` | Exact identities |
+| `taskFixtureDigest` | Fixture identity |
+| `verifierManifestDigest` | Verifier identity |
+| `fabricCompatibilityVersion` | Executor contract version |
+| `sandboxProfileDigest` | Sandbox policy identity |
+| `startedAt` / `completedAt` | Execution window (ms epoch integers) |
+| `limits` | Declared ceilings |
+| `usage` | Actual resource counters used for limit checks |
+| `outcome` | Normalized: `pass` \| `fail` \| `blocked` \| `inconclusive` |
+| `verifier` | Bounded `exact-tree-diff-v1` result |
+| `failure` | Optional typed failure (`class`, `code`, `retryable`, `attribution`) |
+| `artifactDigests` | Content-addressed digests only |
+| `sourceRefs` | Optional safe IDs (request / route-decision / attempt) — never bodies |
+
+**Forbidden in outcome and artifacts:** user repositories, arbitrary file trees, prompts, hidden reasoning, credentials, env secrets, host paths, unrestricted logs/stdout/stderr, raw model transcripts, arbitrary response bodies.
+
+### A.3 V1 executable scope (only)
+
+| Item | Value |
+|---|---|
+| Suite | `fabric-core@1.0.0` |
+| Scenario | `fabric-core.task.synthetic-patch@1.0.0` |
+| Evidence layer | `task_effectiveness` |
+| Execution mode | `fabric` |
+| Verifier | `exact-tree-diff-v1` |
+| Fixture | Lab-owned scratch with `src/value.txt` = UTF-8 `before\n`; requested final `after\n` |
+
+No additional task classes in this phase.
+
+### A.4 Limits (non-weakened)
+
+| Limit | Value |
+|---|---|
+| Files touched | 1 |
+| Aggregate input/output | 64 KiB |
+| Patch operations | 1 |
+| Total timeout | 30 s |
+| Inactivity timeout | 5 s |
+| Aggregate artifacts | 1 MiB |
+| Network | denied in scratch |
+| User MCP | unavailable |
+| Arbitrary shell | unavailable |
+| User repository | unreachable |
+
+### A.5 Patch producer seam
+
+Execution does **not** embed a general coding agent. Production evidence requires a host-issued `TrustedFabricPatchExecutor` invoked through `runFabricSyntheticPatchTaskForRoute`, with `RouteSubjectV1` built from `routeContext` + `destination` via `buildRouteSubjectV1`. Patch producers run in an isolated Bun child with hard termination on timeout.
+
+- Tests use `runFabricSyntheticPatchTaskHarness` with closed `FabricHarnessProducerKind` values or fixture executor modules; harness outcomes are not persistable via `persistFabricRunResult`.
+- A future live route adapter may call a provider **outside** the scratch sandbox and return only a validated `SyntheticPatchV1`; raw prompts/responses never enter Lab storage.
+- CL-07 does **not** ship automatic background execution (CL-08).
+
+## B. Verifier: `exact-tree-diff-v1`
+
+Deterministic, no LLM.
+
+1. Walk only the bounded scratch root (no follow).
+2. Reject symlinks, special files, path traversal (`..`, absolute, drive prefixes), unexpected paths.
+3. Sort repository-relative POSIX paths by UTF-8 bytes.
+4. Hash/read allowed file bytes under bounds.
+5. Pass iff the sole change is `src/value.txt: before\n → after\n` with no add/delete/rename.
+6. Emit bounded structured result: `{ verifierId, manifestDigest, passed, pathSummaries[], reason? }`.
+
+Verifier manifest bytes participate in `verifierManifestDigest` → `TaskSubjectV1`. Behavior changes require a new digest; historical observations are never reinterpreted with current bytes.
+
+## C. Sandbox / execution boundary
+
+Minimal deny-by-default scratch executor:
+
+- Create ephemeral Lab-owned directory under the Lab paths tree (not the user repo).
+- Materialize fixture files only.
+- Apply at most one validated patch operation via direct file write (no shell).
+- Enforce byte/time/inactivity ceilings.
+- Cleanup on success, failure, and timeout.
+- Freeze `sandboxProfileDigest` from a versioned sandbox profile object.
+
+Reuse patterns from `src/lab/artifacts/secure-fs.ts` / live sandbox env stripping where applicable; do not reuse live provider network transport inside the scratch.
+
+## D. Route identity
+
+Nested `RouteSubjectV1` must come from existing CL-03/CL-06 builders (`buildRouteSubjectV1` / policy subject helpers). Approximate `provider/model` strings are forbidden. Optional `sourceRefs` may cite `routeDecisionId` / attempt IDs without copying request content.
+
+## E. Lab ingestion
+
+`observationFromFabricOutcome` / internal `persistFabricOutcome` (not public):
+
+- `evidenceLayer: "task_effectiveness"`
+- `executionMode: "fabric"`
+- Exact scenario/suite IDs + manifest digests + fixture digests
+- Exact `TaskSubjectV1` + `subjectId`
+- Assertions from verifier
+- Typed `failure` attribution
+- Bounded environment metadata
+- Sanitized artifact refs via existing artifact store
+
+No second ledger, mutable task DB, or separate verdict store. JSONL canonical; SQLite rebuildable; verdicts projected.
+
+### E.1 Idempotency
+
+Event identity uses existing `assignEventId` content-addressing. Replaying the same outcome (same event payload identity) must not create contradictory evidence. A legitimate second attempt uses a distinct `attempt` / timing / outcome payload and remains distinct evidence.
+
+## F. Verdict / failure mapping
+
+Layer remains independent of protocol/live.
+
+| Condition | Observation outcome | Failure class → attribution | Projection effect |
+|---|---|---|---|
+| Verifier pass (current required task) | `pass` | — | May contribute to `PROBED`/`VERIFIED` per suite rule |
+| Verifier semantic fail | `fail` | `behavioral_failure` → `route` | `DEGRADED` path per suite |
+| Sandbox violation / containment | `blocked` or `inconclusive` | `sandbox_violation` → `harness`/`environment` | `BLOCKED` / none |
+| Timeout / inactivity / budget | `blocked` | matching blocker → `environment` | none (not route incompatibility) |
+| Harness/executor defect | `inconclusive` | `harness_failure` → `harness` | none |
+| Malformed producer outcome | reject (no event) | — | — |
+| Artifact integrity failure | reject or invalidate | `integrity_failure` → `harness` | invalidate |
+
+Registry/provider claims **cannot** create task-effectiveness `CLAIMED`.
+
+## G. Freshness / invalidation / artifacts
+
+Reuse existing Lab freshness, invalidation, sensitive purge, and rebuild. Missing historical contract artifacts make evidence unusable.
+
+Allowed artifacts: verifier summary, normalized tree-diff summary, bounded execution metadata, sanitized failure summary. Never file bodies, prompts, or raw logs.
+
+## H. Read surfaces / UI / CL-06 interaction
+
+- Extend CL-04 `queryLabCatalog` to discover `fabric-core` scenarios.
+- Projection/query already accept `task_effectiveness` subjects.
+- CL-05 matrix remains read-only; show task layer if already wired.
+- **CL-06 unchanged:** do **not** add `task_effectiveness` to routing `requiredSuites`. `TaskSubjectV1` cannot be resolved pre-dispatch (task class/fixture/verifier/sandbox unknown until execution). Historical task evidence may appear in Lab reads; it must not silently alter production routing without a future explicit contract (out of CL-07 scope).
+
+## I. Rejected alternatives
+
+| Alternative | Why rejected |
+|---|---|
+| General Agent Fabric API / worktree runner | Out of scope; unsafe surface area |
+| Shell-based patch apply | Violates no-arbitrary-shell; use direct write |
+| Pre-dispatch routing on task suites | Subject unknown before execution |
+| Numerical effectiveness score / leaderboard | Forbidden by CL-00 |
+| Second ledger / mutable verified flag | Breaks Lab persistence authority |
+| Embedding prompts in observations | Privacy violation |
+| Weakening byte/time limits for tests | Contract non-negotiable |
+| CL-08 auto/background probing | Explicit exclusion |
+
+## J. Security review checklist
+
+Scratch-root containment; symlink/path-traversal/special-file rejection; max files/bytes; artifact publication bounds; no network/MCP/shell in scratch; minimal env; no credentials/config exposure; cleanup; concurrent isolation; hostile patch paths. Fail closed. Adversarial tests required (§18 of programme request).
+
+## K. Implementation layout (proposed)
+
+```text
+src/lab/fabric/
+  constants.ts
+  types.ts
+  subject.ts
+  fixture.ts
+  sandbox-profile.ts
+  scratch.ts
+  patch.ts
+  verifier.ts
+  executor.ts
+  manifest.ts
+  observe.ts
+  index.ts
+tests/lab-fabric-task.test.ts
+```
+
+## L. Validation plan
+
+Typecheck; focused fabric/subject/sandbox/verifier/observe/ledger/projection/read-surface tests; routing regressions proving CL-06 unchanged; privacy scan; hygiene; GUI only if touched.
+
+## M. Status
+
+- **Authorization:** IN PROGRESS from CL-06 merge `b66e33ce7207d91014644d99317e456c992a3418`
+- **Implementation head:** pending push (authoritative route execution + isolated producer child)
+- **Accepted head:** NOT YET
+- **PR:** [#1438](https://github.com/lidge-jun/opencodex/pull/1438) (draft)
+
+### Authoritative route execution boundary
+
+- Production evidence uses `runFabricSyntheticPatchTaskForRoute({ routeContext, destination, patchExecutor })`.
+- `RouteSubjectV1` is built only via `buildRouteSubjectV1(routeContext, destination)` — callers cannot supply an independent route identity.
+- Patch production requires a host-issued `TrustedFabricPatchExecutor` (`createHostIssuedFabricPatchExecutor` in `src/lib/fabric-task-host.ts`).
+- **Public ingestion:** `persistFabricRunResult` only; it rejects harness runs (`executionAuthority !== "trusted_route"`).
+- `persistFabricOutcome` is internal to `observe.ts` and is **not** exported from the fabric public surface.
+- Harness runs (`runFabricSyntheticPatchTaskHarness`) use closed `FabricHarnessProducerKind` values only; they cannot create production ledger evidence.
+
+### Child isolation / IPC / timeouts
+
+- Patch producers run in a dedicated Bun child (`producer-child.ts`) spawned by `producer-isolate.ts` with minimal env (`TZ`, `NO_COLOR`, `OCX_FABRIC_SCRATCH_ROOT`).
+- Parent↔child protocol is newline-delimited JSON on stdout only (`activity`, `result`, `error` in `producer-protocol.ts`). Arbitrary logging is not mixed into protocol output.
+- Parent owns **both** total and inactivity timeouts; both terminate the child via `SIGKILL`. Classification: `timeout` vs `inactivity_timeout`.
+- Child stdout is capped at 64 KiB protocol bytes; stderr diagnostic capture capped at 4 KiB. Exceeding protocol limits → `budget_exhausted` and child kill.
+- `lastActivityAt` is authoritative in the parent; child `reportActivity()` emits `activity` IPC messages that reset the inactivity deadline.
+- `infinite_sync` harness disables inactivity ceiling extension so synchronous CPU spin is classified under total timeout.
+
+### Sandbox enforcement (honest scope)
+
+- Scratch containment, symlink/path-traversal/special-file rejection, byte/file limits, and cleanup are enforced in the parent via `scratch.ts`, `patch.ts` (`assertSafeRelativePosixPath`), and `applySyntheticPatch`.
+- Child isolation strips proxy env vars on the parent path and runs producers in a separate process with minimal env — **not** an OS-level network/shell sandbox.
+- Host-issued executor modules may still perform direct host filesystem operations outside the scratch tree; that is outside the scratch-apply boundary and is not claimed as blocked.
+- Declared deny flags (`fabricDeclaredSandboxPolicy`) document intent; runtime enforcement matches the scratch/patch/verifier containment above.
+
+### Failure attribution
+
+- Semantic verifier mismatch → `fail` / `behavioral_failure` / `route`.
+- Sandbox/containment (`FabricTaskError` from scratch/verifier/patch infrastructure) → `blocked` or `inconclusive` / `sandbox_violation` / `harness` or `environment` — never route-attributed `behavioral_failure`.
+- Timeouts / budget exhaustion → `blocked` / `environment`.
+- Harness defects → `inconclusive` / `harness`.
+
+### Inactivity accounting
+
+- `inactiveMs` = `completedAt - lastActivityAt` where `lastActivityAt` is updated only from parent-received `activity` IPC (or initial start).
+
+### Local validation (blocker fix head)
+
+- `bun x tsc --noEmit`: passed
+- `bun test tests/lab-fabric-task.test.ts`: 46/46 passed
+- `bun run privacy:scan`: passed
+- Windows sqlite projection flakes in `lab-evidence-ledger.test.ts` (EBUSY) — environmental, not CL-07
+- CL-08: **not started**

src/lab/artifacts/sanitize.ts

@@ -8,8 +8,8 @@ import { jcsStringify } from "../digest";
 import { redactSecretString } from "../../lib/redact";
 
 const FORBIDDEN_KEY = /^(?:authorization|proxy-authorization|cookie|set-cookie|api[-_]?key|x-api-key|token|secret|password|email|prompt|messages|content|body|url|hostname|baseUrl|path|account|alias)$/i;
-const SECRETISH = /sk-[a-z0-9]{10,}|Bearer\s+[A-Za-z0-9._\-]+|ghp_[A-Za-z0-9]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}/i;
-const SECRETISH_GLOBAL = /sk-[a-z0-9]{10,}|Bearer\s+[A-Za-z0-9._\-]+|ghp_[A-Za-z0-9]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}/gi;
+const SECRETISH = /sk-[a-z0-9]{10,}|credential-canary-[a-z0-9]{10,}|Bearer\s+[A-Za-z0-9._\-]+|ghp_[A-Za-z0-9]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}/i;
+const SECRETISH_GLOBAL = new RegExp(SECRETISH.source, "gi");
 
 export function redactForArtifact(artifactClass: ArtifactClass, payload: unknown): unknown {
   if (
@@ -583,4 +583,4 @@ export function sanitizeDiagnostic(value: unknown): string {
 
 export function sanitizedJsonBytes(value: unknown): Uint8Array {
   return new TextEncoder().encode(jcsStringify(scrubValue(value, 0)));
-}
+}

src/lab/conformance/types.ts

@@ -9,7 +9,11 @@ export type VerificationRole = "required" | "supplemental" | "negative_control";
 export type FailureClassification =
   | "harness_failure"
   | "timeout"
+  | "inactivity_timeout"
   | "budget_exhausted"
+  | "sandbox_violation"
+  | "malformed_producer_outcome"
+  | "layer_subject_mismatch"
   | "protocol_failure"
   | "capability_failure"
   | "behavioral_failure"
@@ -20,6 +24,25 @@ export type FailureClassification =
   | "provider_transient"
   | "inconclusive";
 
+export const FAILURE_CLASSIFICATIONS = [
+  "harness_failure",
+  "timeout",
+  "inactivity_timeout",
+  "budget_exhausted",
+  "sandbox_violation",
+  "malformed_producer_outcome",
+  "layer_subject_mismatch",
+  "protocol_failure",
+  "capability_failure",
+  "behavioral_failure",
+  "authentication_blocked",
+  "quota_blocked",
+  "region_blocked",
+  "network_failure",
+  "provider_transient",
+  "inconclusive",
+] as const satisfies readonly FailureClassification[];
+
 export interface FixtureRecord {
   id: string;
   role: "client_request" | "upstream_response" | "adapter_vector" | "synthetic_tool";