From b5f28e74fb5e0639d4e23d80eba7e57876c9a98a Mon Sep 17 00:00:00 2001 From: creayma Date: Fri, 31 Jul 2026 17:11:15 +0800 Subject: [PATCH 1/2] feat(loop-readiness): add fail-closed LC-01 loop readiness gate Implement roadmap LC-01: a versioned readiness contract (v1, 14 capabilities x 5 levels) plus a pure evaluator and CLI that decide whether a caller-declared capability assessment allows a loop run level. The gate never probes hosts, network, or workspace; absent required capabilities report `unavailable` and prevent the level, and any input the assessment contract rejects (unknown id/state/field, duplicate observation, empty evidence, unsupported version, malformed JSON) exits 1 without ever producing a decision. Exit codes: 0 allowed, 2 prevented, 1 invalid input. Wire a `loop` command group (audience advanced) with a `readiness` subcommand into the better-harness CLI registry and root help, add `references/loop-engineering/readiness-gate.md` to the doc routing graph, and regenerate the frozen CLI baselines that pin root help and inventory/schema hashes. Spec: docs/specs/2026-07-31-lc01-loop-readiness-gate.md (reviewed in three acceptance rounds; all P1/P2/P3 findings resolved). Validated with test/loop-readiness.test.mjs (25 tests: contract shape, allowed/ prevented envelopes, invalid-input matrix, import allowlist with obfuscation defenses, empty-tmpdir spawn, facade byte parity) and the full suite `npm test` at 1048/1048, plus mutation checks proving the no-arg and allowlist guards fail when their implementation is removed. Co-authored-by: QoderAI (Qwen 3.8 Max) --- .../2026-07-31-lc01-loop-readiness-gate.md | 264 ++++++++++++ references/loop-engineering/README.md | 5 + .../loop-engineering/automation-readiness.md | 4 + references/loop-engineering/readiness-gate.md | 69 +++ scripts/better-harness-cli/cli.mjs | 8 + scripts/better-harness-cli/registry.mjs | 16 + scripts/loop-readiness/cli.mjs | 157 +++++++ scripts/loop-readiness/contract.mjs | 73 ++++ scripts/loop-readiness/evaluate.mjs | 125 ++++++ .../loop-readiness/all-available.json | 76 ++++ .../loop-readiness/duplicate-observation.json | 16 + .../loop-readiness/empty-evidence.json | 11 + .../loop-readiness/empty-observations.json | 5 + .../loop-readiness/explicit-unavailable.json | 21 + test/fixtures/loop-readiness/malformed.json | 1 + .../loop-readiness/mixed-partial-failed.json | 41 ++ .../loop-readiness/non-required-blocked.json | 26 ++ test/fixtures/loop-readiness/one-absent.json | 16 + test/fixtures/loop-readiness/one-blocked.json | 51 +++ .../loop-readiness/unknown-capability.json | 16 + .../loop-readiness/unknown-state.json | 11 + .../loop-readiness/unsupported-version.json | 11 + .../scripts-refactor-contract/root-help.txt | 4 + test/loop-readiness.test.mjs | 396 ++++++++++++++++++ test/scripts-refactor-contract.test.mjs | 4 +- 25 files changed, 1425 insertions(+), 2 deletions(-) create mode 100644 docs/specs/2026-07-31-lc01-loop-readiness-gate.md create mode 100644 references/loop-engineering/readiness-gate.md create mode 100644 scripts/loop-readiness/cli.mjs create mode 100644 scripts/loop-readiness/contract.mjs create mode 100644 scripts/loop-readiness/evaluate.mjs create mode 100644 test/fixtures/loop-readiness/all-available.json create mode 100644 test/fixtures/loop-readiness/duplicate-observation.json create mode 100644 test/fixtures/loop-readiness/empty-evidence.json create mode 100644 test/fixtures/loop-readiness/empty-observations.json create mode 100644 test/fixtures/loop-readiness/explicit-unavailable.json create mode 100644 test/fixtures/loop-readiness/malformed.json create mode 100644 test/fixtures/loop-readiness/mixed-partial-failed.json create mode 100644 test/fixtures/loop-readiness/non-required-blocked.json create mode 100644 test/fixtures/loop-readiness/one-absent.json create mode 100644 test/fixtures/loop-readiness/one-blocked.json create mode 100644 test/fixtures/loop-readiness/unknown-capability.json create mode 100644 test/fixtures/loop-readiness/unknown-state.json create mode 100644 test/fixtures/loop-readiness/unsupported-version.json create mode 100644 test/loop-readiness.test.mjs diff --git a/docs/specs/2026-07-31-lc01-loop-readiness-gate.md b/docs/specs/2026-07-31-lc01-loop-readiness-gate.md new file mode 100644 index 0000000..33fb681 --- /dev/null +++ b/docs/specs/2026-07-31-lc01-loop-readiness-gate.md @@ -0,0 +1,264 @@ +# Fail-closed Loop Runtime Readiness Gate + +Add a versioned, machine-readable readiness contract and a deterministic +evaluator that decides whether a proposed loop run level is allowed, before any +mutating or scheduled workflow exists. This implements roadmap item `LC-01`. + +## Traceability + +- Spec ID: 2026-07-31-lc01-loop-readiness-gate +- Story: roadmap `LC-01` (roadmap ID, not a GitHub issue) +- Status: Implemented + +## Intent + +The [roadmap](../../roadmap.md) requires that no mutating runtime may bypass +`LC-01`, and `LC-06`/`LC-07` depend on it. Today the repository documents an +automation readiness contract in prose +([Automation Readiness](../../references/loop-engineering/automation-readiness.md)) +but has no executable owner: nothing can distinguish read-only observation from +plan-only or approved-apply runs, and nothing prevents a run when a required +capability is `blocked`, `partial`, `unavailable`, or `failed`. + +This spec adds the smallest executable gate: a versioned readiness-level +contract, a per-level required-capability matrix, and a pure +assessment-in/decision-out evaluator with a parser-safe CLI. The gate makes no +observations itself; callers declare capability observations and the gate fails +closed on anything missing or invalid. + +## Readiness Contract (v1) + +Readiness levels (ids are stable contract values; levels are independent +contract values under a partial order — `plan-only` and `scheduled-read-only` +are not comparable — so implementations must not introduce a numeric level +ranking): + +| Level id | Meaning | +| --- | --- | +| `read-only-observation` | Read workspace and provider evidence; write nothing outside the run directory. | +| `plan-only` | Additionally produce a plan artifact; apply nothing. | +| `human-approved-apply` | Execute a bounded, human-approved apply with isolation and rollback. | +| `scheduled-read-only` | Recurring observation without a human in the loop. | +| `scheduled-bounded-apply` | Recurring bounded apply; strictest requirement set. | + +Capability observation states: `available`, `partial`, `unavailable`, +`blocked`, `failed`. A required capability that is not `available` — including +one simply absent from the assessment — prevents the applicable level. + +Required-capability matrix (v1): + +| Capability id | read-only-observation | plan-only | human-approved-apply | scheduled-read-only | scheduled-bounded-apply | +| --- | --- | --- | --- | --- | --- | +| `workspace-read` | ✓ | ✓ | ✓ | ✓ | ✓ | +| `evidence-source` | ✓ | ✓ | ✓ | ✓ | ✓ | +| `privacy-boundary` | ✓ | ✓ | ✓ | ✓ | ✓ | +| `plan-artifact-write` | | ✓ | ✓ | | ✓ | +| `human-approval` | | | ✓ | | ✓ | +| `isolated-execution` | | | ✓ | | ✓ | +| `frozen-pre-state` | | | ✓ | | ✓ | +| `validation-route` | | | ✓ | | ✓ | +| `rollback-reference` | | | ✓ | | ✓ | +| `schedule-trigger` | | | | ✓ | ✓ | +| `stop-condition` | | | | ✓ | ✓ | +| `triage-path` | | | | ✓ | ✓ | +| `budget-policy` | | | | ✓ | ✓ | +| `idempotent-recovery` | | | | | ✓ | + +Capability id sources (one row per id, so reviewers can audit the derivation): + +| Capability id | Source clause | +| --- | --- | +| `workspace-read`, `evidence-source` | Automation Readiness fields 1 (Target) and 5 (Input pack) | +| `privacy-boundary` | Automation Readiness field 9 (Risk boundary); roadmap invariant "Raw private transcripts … never enter public artifacts" | +| `plan-artifact-write`, `validation-route` | Automation Readiness fields 6 (Sandbox) and 7 (Validation) | +| `human-approval` | Automation Readiness field 6; roadmap Open Decision "Should any provider support automatic apply?" | +| `isolated-execution`, `frozen-pre-state`, `rollback-reference` | Roadmap invariant "Every mutation has a frozen pre-state, explicit authority, isolated execution, objective validation, and a resolvable rollback or compensation boundary" | +| `schedule-trigger`, `stop-condition`, `triage-path`, `budget-policy` | Automation Readiness fields 2 (Trigger), 10 (Stop condition), 8 (Triage path); roadmap invariant "Every loop declares a trigger, … budget, stop condition, and human gate where required" | +| `idempotent-recovery` | Roadmap Non-goal "Do not enable scheduled bounded apply before the readiness, isolation, approval, idempotency, stop, and rollback contracts pass" and the `LC-06` acceptance row | + +Intentional v1 narrowing (explicit, so later versions do not treat these as +omissions): the loop-invariant items `stable input`, `state policy`, +`observability`, and `evaluation` are runtime-lifecycle concerns owned by +`LC-06` (state, journal) and `LC-04`/`LC-10` (evaluation, budget accounting +semantics); v1 gates only pre-run readiness. The Automation Readiness field +`Run scope` describes an execution shape, not a readiness capability, and maps +to `LC-06` run forms. `human-approved-apply` does not require `stop-condition` +or `triage-path` because a single approved bounded apply is not a loop; the +mutation invariant's five elements (frozen pre-state, authority, isolation, +validation, rollback) are all required instead. For +`scheduled-bounded-apply`, `human-approval` means the approval binds to the +bounded plan artifact the schedule executes; whether each run needs a fresh +approval is an `LC-06` wiring decision, not weakened here. + +The matrix is data, versioned as `readinessContractVersion: 1`, and owned by +the gate; assessments only supply observations. Additions in later versions +must not silently relax an existing level. + +### Assessment input contract (v1) + +The assessment is a JSON file with exactly these top-level fields: + +```json +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { "id": "workspace-read", "state": "available", "evidence": "..." } + ] +} +``` + +- `kind` must be `loop-readiness-assessment`; `readinessContractVersion` must + be an integer the evaluator supports (v1 evaluator supports only `1`). +- Each observation has exactly `id`, `state`, and `evidence`; `evidence` must + be a non-empty string. Empty or missing `evidence` is invalid input, not a + softer state. +- An empty `observations` array is valid input (every required capability is + then absent → `prevented`); a missing or non-array `observations` field is + malformed input. +- Unknown top-level fields, unknown observation fields, an `id` not defined by + the contract, an unknown `state` string, or the same `id` observed more than + once are all invalid input (exit 1), never a decision. Strict rejection is + the fail-closed choice: a misspelled capability must not silently degrade to + `unavailable`. + +### Evaluator semantics (v1) + +- Observations for contract-known capabilities that the requested level does + not require are ignored: they do not affect the decision and do not appear + in `blockingCapabilities` or the envelope's `observations` echo. +- A required capability absent from `observations` is reported in + `blockingCapabilities` with state `unavailable`. +- The boundary between the two non-zero exits is: a structurally valid + assessment that fails a requirement is a decision (`prevented`, exit 2); + anything the input contract rejects is invalid input (exit 1) and produces + no decision. + +Versioning rules: `schemaVersion` governs the decision-envelope shape; +`readinessContractVersion` governs level ids, capability ids, matrix +semantics, and the assessment observation semantics. They evolve +independently; the v1 evaluator accepts only contract version 1 and emits only +envelope schema version 1. + +Trust boundary: an `allowed` decision is necessary, never sufficient, +authority for a run. The gate cannot detect a caller that misdeclares +`available`; truthfulness and freshness of observations are the caller's +responsibility, and binding observations to an actual run window is `LC-06` +wiring. This mirrors the roadmap invariant "Configured presence never proves +observed use": a declared `available` does not prove the capability works. + +## Acceptance Scenarios + +- AC-1: A versioned contract module exports the five readiness levels, the five + capability states, and the v1 required-capability matrix; the evaluator and + CLI reject an assessment whose `readinessContractVersion` they do not + support. +- AC-2: Evaluating a level with every required capability observed `available` + returns an `allowed` decision envelope naming the level, contract version, + and the evidence strings of the observations it consumed. +- AC-3: Evaluating a level with any required capability observed `blocked`, + `partial`, `unavailable`, or `failed` — or not present in the assessment at + all — returns a `prevented` envelope listing every blocking capability with + its observed state (absent observations report `unavailable`). No allowed + decision is derivable from defaults. +- AC-4: The CLI (`better-harness loop readiness --level --assessment + --json`) keeps stdout parser-safe JSON in machine mode and uses exit + codes: `0` allowed, `2` prevented (documented non-success envelope), `1` for + every input the assessment contract rejects (invalid usage, + unreadable/malformed assessment, unknown level, unknown capability id, + unknown state, duplicate observation id, empty evidence, or unsupported + contract version). Invalid input never produces an `allowed` or `prevented` + decision. `--help` exits 0 and documents the levels and the exit-code + contract; invoking the CLI with no arguments prints the same help and exits + 0 (no decision is emitted); without `--json` the CLI prints a readable + decision summary for the same exit codes. +- AC-5: The gate reads only the assessment file passed to it. Verifiable + boundary: `contract.mjs` imports nothing, `evaluate.mjs` imports only the + contract module, and `cli.mjs` imports only `node:` built-ins plus the two + gate modules (asserted by a static import-allowlist test); the CLI succeeds + when spawned from an empty temporary working directory (no workspace + scanning). The gate CLI hands `--assessment` to the filesystem as given + (resolved against the caller's working directory); `path.join`/`path.resolve` + and argv-array spawn per the ARCHITECTURE CLI contract apply to the facade + and test layers. No further OS-specific behavior is claimed. +- AC-6: `npm test` passes, including a new `test/loop-readiness.test.mjs` + covering AC-1 through AC-5 with fixture assessments for at least: all + available; one `blocked`; one explicit `unavailable`; one required + capability absent; mixed `partial`+`failed`; a non-required capability + observed `blocked` (must still be `allowed`); empty `observations` array; + and invalid inputs — unknown level, unknown capability id, unknown state, + duplicate observation id, empty evidence, unsupported contract version, + malformed JSON, and an unreadable assessment path. Invalid-input tests + assert both the exit code and the error envelope/stderr content. +- AC-7: `node --test test/doc-link-graph.test.mjs` passes with this spec's + links resolving. + +## Non-goals + +- No loop runtime, scheduler, plan/apply executor, or side-effect journal + (`LC-06`), and no intervention experiment lifecycle (`LC-07`). +- No host/provider probing and no automatic capability discovery; observations + are declared by callers (adapter-supplied probes belong to the `HA-*` track). +- No verification of observation truthfulness or freshness; the gate judges + declared observations only (see Trust boundary). +- The gate writes no files and creates no run directory; the decision is + emitted on stdout only. +- No `schemas/` top-level promotion: the assessment/decision schema stays + capability-private under `scripts/loop-readiness/` per the directory ADR. +- No retrofit of existing commands (`harness analyze`, `checkup`, quickstart) + onto the gate in this change; wiring consumers is follow-up work under + `LC-06`. + +## Plan and Tasks + +New capability directory `scripts/loop-readiness/` (business-named, copyable, +testable, per [ARCHITECTURE](../ARCHITECTURE.md) conventions): + +1. `scripts/loop-readiness/contract.mjs` — exported level ids, capability + states, v1 matrix, `readinessContractVersion`; frozen data only, no + imports. +2. `scripts/loop-readiness/evaluate.mjs` — pure + `evaluateReadiness({ level, assessment })` returning the decision envelope + `{ kind: "loop-readiness-decision", schemaVersion: 1, + readinessContractVersion, level, status: "allowed" | "prevented", + blockingCapabilities: [{ id, state }], observations: [...] }`; throws typed + errors for every input the assessment contract rejects so the CLI can map + them to exit code 1. +3. `scripts/loop-readiness/cli.mjs` — argument parsing, `--help`, `--json` + machine mode and readable human mode, exit-code mapping per AC-4. +4. Register a `loop` command group (audience `advanced`, summary describing + the readiness gate) with a `readiness` subcommand (audience `advanced`) in + `scripts/better-harness-cli/registry.mjs`; the group later hosts `LC-06` + verbs. +5. `test/loop-readiness.test.mjs` — unit tests over `evaluate.mjs` plus CLI + spawn tests mirroring the style of `test/support-declarations.test.mjs` + (argv arrays, exit code plus error-content assertions); fixture assessments + under `test/fixtures/loop-readiness/`. +6. Docs: add `references/loop-engineering/readiness-gate.md` describing the + contract and linking it from `references/loop-engineering/README.md` and + `automation-readiness.md`; regenerate the doc link graph if routing changes. + +Decision rationale: the gate is caller-assessed rather than self-probing so it +stays deterministic, testable, and free of provider coupling; fail-closed +semantics follow the existing #11–#18 regression contract style (absence is +`unavailable`, never a default allow; structural doubt is rejection, never a +decision); exit code `2` distinguishes a valid `prevented` decision from +invalid input per the roadmap Definition of Done ("non-zero or a documented +non-success envelope"). + +## Test and Review Evidence + +- `node --test test/loop-readiness.test.mjs` → AC-1..AC-6. +- Mutation regression: removing one required capability observation from a + passing fixture must flip the decision to `prevented` and list exactly that + capability with state `unavailable`. +- `node scripts/better-harness.mjs loop readiness --help` exits 0 and documents + levels and exit codes; `--level bogus` exits 1 with a parser-safe error + envelope → AC-4. +- Static import-allowlist assertions over the three gate modules, and a CLI + spawn from an empty temporary directory → AC-5. +- `node --test test/doc-link-graph.test.mjs` → AC-7. +- `npm test` full suite → AC-6. +- Risk: purely additive (new directory, one registry entry, new test/fixtures, + one reference doc); no existing behavior changes, so blast radius is the CLI + registry table and doc link graph only. diff --git a/references/loop-engineering/README.md b/references/loop-engineering/README.md index ccde9d8..38f9399 100644 --- a/references/loop-engineering/README.md +++ b/references/loop-engineering/README.md @@ -43,6 +43,9 @@ agent loop. plugin/connector, subagent, and state support. - `automation-readiness.md`: schedule or automation readiness contract after Loop Discovery selects automation or `schedule-ready`. +- `readiness-gate.md`: executable fail-closed readiness gate (`LC-01`) that + decides whether a caller-declared capability assessment allows a loop run + level. - `loop-state-ledger.md`: state ledger contract for paused, recurring, background, or multi-run loops. - `learning-loop-patterns.md`: longitudinal learning pattern signatures, @@ -75,6 +78,8 @@ agent loop. - Use `loop-primitives.md` when a proven loop needs supporting primitives. - Use `automation-readiness.md` after the decision is automation, `schedule-ready`, or scheduled/background follow-up. +- Use `readiness-gate.md` when a proposed loop run level needs an executable + allowed/prevented decision from declared capability observations. - Use `loop-state-ledger.md` when a loop needs resumable, recurring, or cross-agent state. - Continue to `../agent-customize/skill-discovery.md` only after Loop Discovery selects diff --git a/references/loop-engineering/automation-readiness.md b/references/loop-engineering/automation-readiness.md index e193c40..568b7da 100644 --- a/references/loop-engineering/automation-readiness.md +++ b/references/loop-engineering/automation-readiness.md @@ -28,6 +28,10 @@ Recommend automation only when all fields are concrete: If any field is missing, return `needs more evidence` or recommend a one-time manual follow-up instead of recurring automation. +For an executable, fail-closed decision on a concrete loop run level, hand the +declared capability observations to the gate described in +[readiness-gate.md](readiness-gate.md) (`better-harness loop readiness`). + ## Automation Shapes - **Standalone automation**: use when each run should start fresh from stable diff --git a/references/loop-engineering/readiness-gate.md b/references/loop-engineering/readiness-gate.md new file mode 100644 index 0000000..88280b6 --- /dev/null +++ b/references/loop-engineering/readiness-gate.md @@ -0,0 +1,69 @@ +# Loop Readiness Gate Reference + +Use this after `automation-readiness.md` declares a concrete automation +contract and a caller needs an executable, fail-closed decision for a proposed +loop run level. The gate is the executable owner of roadmap item `LC-01`; the +implementation lives in `scripts/loop-readiness/` and is exposed as +`better-harness loop readiness`. + +## Contract (v1) + +Readiness levels are independent contract values under a partial order — do +not rank them numerically: + +- `read-only-observation`: read workspace and provider evidence; write nothing + outside the run directory. +- `plan-only`: additionally produce a plan artifact; apply nothing. +- `human-approved-apply`: execute a bounded, human-approved apply with + isolation and rollback. +- `scheduled-read-only`: recurring observation without a human in the loop. +- `scheduled-bounded-apply`: recurring bounded apply; strictest requirement + set. + +Capability observation states: `available`, `partial`, `unavailable`, +`blocked`, `failed`. A required capability that is not `available` — including +one absent from the assessment — prevents the applicable level. The v1 +required-capability matrix is data owned by the gate +(`scripts/loop-readiness/contract.mjs`, `readinessContractVersion: 1`); later +versions must not silently relax an existing level. + +## Assessment Input + +Callers declare observations; the gate never probes hosts or providers: + +```json +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { "id": "workspace-read", "state": "available", "evidence": "..." } + ] +} +``` + +Each observation carries exactly `id`, `state`, and a non-empty `evidence` +string. Unknown fields, unknown capability ids or states, duplicate ids, or an +unsupported contract version are rejected as invalid input and never become a +decision; a misspelled capability must not silently degrade to `unavailable`. + +## CLI and Exit Codes + +```text +better-harness loop readiness --level --assessment --json +``` + +- `0`: `allowed` decision envelope. +- `2`: `prevented` decision envelope listing every blocking capability with + its observed state. +- `1`: invalid usage or any input the assessment contract rejects. + +`--help` (or invoking the CLI with no arguments) prints the levels and this +exit-code contract and exits 0 without emitting a decision. + +## Trust Boundary + +An `allowed` decision is necessary, never sufficient, authority for a run. The +gate judges declared observations only; truthfulness and freshness stay with +the caller, and binding observations to an actual run window is `LC-06` +wiring. A declared `available` does not prove the capability works — configured +presence never proves observed use. diff --git a/scripts/better-harness-cli/cli.mjs b/scripts/better-harness-cli/cli.mjs index 705086e..2ec75f8 100644 --- a/scripts/better-harness-cli/cli.mjs +++ b/scripts/better-harness-cli/cli.mjs @@ -30,6 +30,11 @@ const HELP_GROUPS = [ audience: "workflow", commands: ["harness"], }, + { + title: "Loop Control", + audience: "advanced", + commands: ["loop"], + }, { title: "Project Evidence", audience: "advanced", @@ -81,6 +86,9 @@ const GROUP_EXAMPLES = { { audience: "advanced", text: "better-harness coding-agent-practices asset-baseline codex --workspace . --json" }, { audience: "advanced", text: "better-harness coding-agent-practices inventory qoder --workspace ." }, ], + "loop": [ + { audience: "advanced", text: "better-harness loop readiness --level read-only-observation --assessment --json" }, + ], }; function scriptPath(relativePath) { diff --git a/scripts/better-harness-cli/registry.mjs b/scripts/better-harness-cli/registry.mjs index c80acba..23ecb38 100644 --- a/scripts/better-harness-cli/registry.mjs +++ b/scripts/better-harness-cli/registry.mjs @@ -155,6 +155,22 @@ const COMMANDS = [ { name: "qoder-consistency-schema", audience: "maintainer", script: "core-change-watch/qoder-consistency-schema.mjs" }, ], }, + { + name: "loop", + kind: "group", + audience: "advanced", + summary: "Gate loop run levels with the fail-closed readiness contract.", + description: "Evaluate caller-declared capability assessments against the versioned LC-01 readiness contract; required capabilities that are not available prevent the applicable run level.", + subcommands: [ + { + name: "readiness", + audience: "advanced", + script: "loop-readiness/cli.mjs", + summary: "Decide whether a declared capability assessment allows a readiness level.", + description: "Read one assessment file, apply the v1 required-capability matrix, and return an allowed or prevented decision envelope with exit codes 0, 2, and 1.", + }, + ], + }, { name: "harness", kind: "group", diff --git a/scripts/loop-readiness/cli.mjs b/scripts/loop-readiness/cli.mjs new file mode 100644 index 0000000..d92a077 --- /dev/null +++ b/scripts/loop-readiness/cli.mjs @@ -0,0 +1,157 @@ +#!/usr/bin/env node + +// LC-01 readiness gate CLI. Exit codes: 0 allowed, 2 prevented (documented +// non-success envelope), 1 for every input the assessment contract rejects. +// The gate reads only the assessment file passed to it: no session stores, +// provider homes, network, or workspace scanning. + +import { readFileSync } from "node:fs"; +import { READINESS_CONTRACT_VERSION, READINESS_LEVELS } from "./contract.mjs"; +import { evaluateReadiness, ReadinessInputError } from "./evaluate.mjs"; + +const EXIT_ALLOWED = 0; +const EXIT_INVALID_INPUT = 1; +const EXIT_PREVENTED = 2; + +function helpText() { + const lines = [ + "Usage: better-harness loop readiness --level --assessment [--json]", + "", + "Decide whether a caller-declared capability assessment allows a readiness", + `level (readiness contract v${READINESS_CONTRACT_VERSION}). The gate reads only the assessment file,`, + "never probes hosts or providers, and fails closed on anything missing or", + "invalid. An allowed decision is necessary, never sufficient, authority for", + "a run; observation truthfulness stays with the caller.", + "", + "Levels:", + ...READINESS_LEVELS.map((level) => ` ${level}`), + "", + "Options:", + " --level Readiness level to evaluate", + " --assessment JSON capability assessment declared by the caller", + " --json Parser-safe JSON decision or error envelope on stdout", + " -h, --help Print this help", + "", + "Exit codes:", + " 0 allowed decision", + " 2 prevented decision (documented non-success envelope)", + " 1 invalid usage, unreadable or malformed assessment, unknown level,", + " unknown capability id or state, duplicate observation, empty evidence,", + " or unsupported contract version", + ]; + return `${lines.join("\n")}\n`; +} + +function parseArgs(argv) { + const options = { json: false, help: false }; + for (let index = 0; index < argv.length; index += 1) { + const value = argv[index]; + if (value === "--help" || value === "-h") { + options.help = true; + } else if (value === "--json") { + options.json = true; + } else if (value === "--level") { + options.level = argv[(index += 1)]; + } else if (value.startsWith("--level=")) { + options.level = value.slice("--level=".length); + } else if (value === "--assessment") { + options.assessment = argv[(index += 1)]; + } else if (value.startsWith("--assessment=")) { + options.assessment = value.slice("--assessment=".length); + } else { + throw new ReadinessInputError("INVALID_USAGE", `Unknown argument: ${value}`); + } + } + return options; +} + +function emitError(error, json) { + if (json) { + const envelope = { + kind: "loop-readiness-error", + code: error.code ?? "INVALID_INPUT", + message: error.message, + }; + process.stdout.write(`${JSON.stringify(envelope, null, 2)}\n`); + } else { + process.stderr.write(`loop readiness: ${error.message}\n`); + } + return EXIT_INVALID_INPUT; +} + +function emitDecision(decision, json) { + if (json) { + process.stdout.write(`${JSON.stringify(decision, null, 2)}\n`); + } else { + const lines = [ + `Decision: ${decision.status}`, + `Level: ${decision.level}`, + `Readiness contract: v${decision.readinessContractVersion}`, + `Consumed observations: ${decision.observations.length}`, + ]; + if (decision.blockingCapabilities.length > 0) { + lines.push("Blocking capabilities:"); + for (const capability of decision.blockingCapabilities) { + lines.push(` - ${capability.id}: ${capability.state}`); + } + } + process.stdout.write(`${lines.join("\n")}\n`); + } + return decision.status === "allowed" ? EXIT_ALLOWED : EXIT_PREVENTED; +} + +export function main(argv = process.argv.slice(2)) { + let options; + try { + options = parseArgs(argv); + } catch (error) { + if (error instanceof ReadinessInputError) { + return emitError(error, argv.includes("--json")); + } + throw error; + } + + if (options.help || argv.length === 0) { + process.stdout.write(helpText()); + return 0; + } + + try { + if (!options.level) { + throw new ReadinessInputError("INVALID_USAGE", "Missing required option: --level ."); + } + if (!options.assessment) { + throw new ReadinessInputError("INVALID_USAGE", "Missing required option: --assessment ."); + } + + let raw; + try { + raw = readFileSync(options.assessment, "utf8"); + } catch (error) { + throw new ReadinessInputError( + "UNREADABLE_ASSESSMENT", + `Cannot read assessment file ${options.assessment}: ${error.message}`, + ); + } + + let assessment; + try { + assessment = JSON.parse(raw); + } catch (error) { + throw new ReadinessInputError( + "MALFORMED_ASSESSMENT", + `Assessment file ${options.assessment} is not valid JSON: ${error.message}`, + ); + } + + const decision = evaluateReadiness({ level: options.level, assessment }); + return emitDecision(decision, options.json); + } catch (error) { + if (error instanceof ReadinessInputError) { + return emitError(error, options.json); + } + throw error; + } +} + +process.exitCode = main(); diff --git a/scripts/loop-readiness/contract.mjs b/scripts/loop-readiness/contract.mjs new file mode 100644 index 0000000..71750b5 --- /dev/null +++ b/scripts/loop-readiness/contract.mjs @@ -0,0 +1,73 @@ +// LC-01 readiness contract (v1): frozen data only, no imports. +// Levels are independent contract values under a partial order; implementations +// must not introduce a numeric level ranking. Additions in later contract +// versions must not silently relax an existing level. + +export const READINESS_CONTRACT_VERSION = 1; +export const DECISION_SCHEMA_VERSION = 1; +export const ASSESSMENT_KIND = "loop-readiness-assessment"; +export const DECISION_KIND = "loop-readiness-decision"; + +export const READINESS_LEVELS = Object.freeze([ + "read-only-observation", + "plan-only", + "human-approved-apply", + "scheduled-read-only", + "scheduled-bounded-apply", +]); + +export const CAPABILITY_STATES = Object.freeze([ + "available", + "partial", + "unavailable", + "blocked", + "failed", +]); + +const OBSERVATION_CAPABILITIES = Object.freeze([ + "workspace-read", + "evidence-source", + "privacy-boundary", +]); + +const MUTATION_CAPABILITIES = Object.freeze([ + "plan-artifact-write", + "human-approval", + "isolated-execution", + "frozen-pre-state", + "validation-route", + "rollback-reference", +]); + +const SCHEDULE_CAPABILITIES = Object.freeze([ + "schedule-trigger", + "stop-condition", + "triage-path", + "budget-policy", +]); + +export const CAPABILITY_IDS = Object.freeze([ + ...OBSERVATION_CAPABILITIES, + ...MUTATION_CAPABILITIES, + ...SCHEDULE_CAPABILITIES, + "idempotent-recovery", +]); + +export const REQUIRED_CAPABILITIES = Object.freeze({ + "read-only-observation": Object.freeze([...OBSERVATION_CAPABILITIES]), + "plan-only": Object.freeze([...OBSERVATION_CAPABILITIES, "plan-artifact-write"]), + "human-approved-apply": Object.freeze([ + ...OBSERVATION_CAPABILITIES, + ...MUTATION_CAPABILITIES, + ]), + "scheduled-read-only": Object.freeze([ + ...OBSERVATION_CAPABILITIES, + ...SCHEDULE_CAPABILITIES, + ]), + "scheduled-bounded-apply": Object.freeze([ + ...OBSERVATION_CAPABILITIES, + ...MUTATION_CAPABILITIES, + ...SCHEDULE_CAPABILITIES, + "idempotent-recovery", + ]), +}); diff --git a/scripts/loop-readiness/evaluate.mjs b/scripts/loop-readiness/evaluate.mjs new file mode 100644 index 0000000..c0b9fcf --- /dev/null +++ b/scripts/loop-readiness/evaluate.mjs @@ -0,0 +1,125 @@ +// LC-01 readiness evaluator: pure assessment-in/decision-out, no I/O. +// Fail-closed boundary: a structurally valid assessment that misses a +// requirement is a `prevented` decision; anything the input contract rejects +// raises a typed ReadinessInputError and never becomes a decision. + +import { + ASSESSMENT_KIND, + CAPABILITY_IDS, + CAPABILITY_STATES, + DECISION_KIND, + DECISION_SCHEMA_VERSION, + READINESS_CONTRACT_VERSION, + READINESS_LEVELS, + REQUIRED_CAPABILITIES, +} from "./contract.mjs"; + +export class ReadinessInputError extends Error { + constructor(code, message) { + super(message); + this.name = "ReadinessInputError"; + this.code = code; + } +} + +const ASSESSMENT_FIELDS = new Set(["kind", "readinessContractVersion", "observations"]); +const OBSERVATION_FIELDS = new Set(["id", "state", "evidence"]); + +function fail(code, message) { + throw new ReadinessInputError(code, message); +} + +function isPlainObject(value) { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function validateAssessment(assessment) { + if (!isPlainObject(assessment)) { + fail("MALFORMED_ASSESSMENT", "Assessment must be a JSON object."); + } + for (const field of Object.keys(assessment)) { + if (!ASSESSMENT_FIELDS.has(field)) { + fail("UNKNOWN_FIELD", `Unknown assessment field: ${field}.`); + } + } + if (assessment.kind !== ASSESSMENT_KIND) { + fail("MALFORMED_ASSESSMENT", `Assessment kind must be "${ASSESSMENT_KIND}".`); + } + if (assessment.readinessContractVersion !== READINESS_CONTRACT_VERSION) { + fail( + "UNSUPPORTED_CONTRACT_VERSION", + `Unsupported readinessContractVersion: ${JSON.stringify(assessment.readinessContractVersion)}. ` + + `This evaluator supports only version ${READINESS_CONTRACT_VERSION}.`, + ); + } + if (!Array.isArray(assessment.observations)) { + fail("MALFORMED_ASSESSMENT", "Assessment observations must be an array (it may be empty)."); + } + const seen = new Set(); + for (const observation of assessment.observations) { + if (!isPlainObject(observation)) { + fail("MALFORMED_ASSESSMENT", "Each observation must be a JSON object."); + } + for (const field of Object.keys(observation)) { + if (!OBSERVATION_FIELDS.has(field)) { + fail("UNKNOWN_FIELD", `Unknown observation field: ${field}.`); + } + } + if (!CAPABILITY_IDS.includes(observation.id)) { + // Strict rejection: a misspelled capability must not silently degrade + // to `unavailable`. + fail("UNKNOWN_CAPABILITY", `Unknown capability id: ${JSON.stringify(observation.id)}.`); + } + if (!CAPABILITY_STATES.includes(observation.state)) { + fail( + "UNKNOWN_STATE", + `Unknown capability state for ${observation.id}: ${JSON.stringify(observation.state)}.`, + ); + } + if (typeof observation.evidence !== "string" || observation.evidence.trim() === "") { + fail("EMPTY_EVIDENCE", `Observation ${observation.id} must carry a non-empty evidence string.`); + } + if (seen.has(observation.id)) { + fail("DUPLICATE_OBSERVATION", `Capability ${observation.id} is observed more than once.`); + } + seen.add(observation.id); + } +} + +export function evaluateReadiness({ level, assessment }) { + if (!READINESS_LEVELS.includes(level)) { + fail( + "UNKNOWN_LEVEL", + `Unknown readiness level: ${JSON.stringify(level)}. Known levels: ${READINESS_LEVELS.join(", ")}.`, + ); + } + validateAssessment(assessment); + + const observed = new Map(assessment.observations.map((observation) => [observation.id, observation])); + const blockingCapabilities = []; + const observations = []; + for (const id of REQUIRED_CAPABILITIES[level]) { + const observation = observed.get(id); + if (!observation) { + // Fail closed: an absent required capability reports `unavailable`. + blockingCapabilities.push({ id, state: "unavailable" }); + continue; + } + // Contract-known observations the level does not require are ignored and + // never echoed; only consumed observations appear in the envelope. + observations.push({ id: observation.id, state: observation.state, evidence: observation.evidence }); + if (observation.state !== "available") { + blockingCapabilities.push({ id, state: observation.state }); + } + } + + return { + kind: DECISION_KIND, + schemaVersion: DECISION_SCHEMA_VERSION, + readinessContractVersion: READINESS_CONTRACT_VERSION, + level, + status: blockingCapabilities.length === 0 ? "allowed" : "prevented", + blockingCapabilities, + observations, + }; +} diff --git a/test/fixtures/loop-readiness/all-available.json b/test/fixtures/loop-readiness/all-available.json new file mode 100644 index 0000000..136046f --- /dev/null +++ b/test/fixtures/loop-readiness/all-available.json @@ -0,0 +1,76 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "available", + "evidence": "fixture: evidence-source verified by caller" + }, + { + "id": "privacy-boundary", + "state": "available", + "evidence": "fixture: privacy-boundary verified by caller" + }, + { + "id": "plan-artifact-write", + "state": "available", + "evidence": "fixture: plan-artifact-write verified by caller" + }, + { + "id": "human-approval", + "state": "available", + "evidence": "fixture: human-approval verified by caller" + }, + { + "id": "isolated-execution", + "state": "available", + "evidence": "fixture: isolated-execution verified by caller" + }, + { + "id": "frozen-pre-state", + "state": "available", + "evidence": "fixture: frozen-pre-state verified by caller" + }, + { + "id": "validation-route", + "state": "available", + "evidence": "fixture: validation-route verified by caller" + }, + { + "id": "rollback-reference", + "state": "available", + "evidence": "fixture: rollback-reference verified by caller" + }, + { + "id": "schedule-trigger", + "state": "available", + "evidence": "fixture: schedule-trigger verified by caller" + }, + { + "id": "stop-condition", + "state": "available", + "evidence": "fixture: stop-condition verified by caller" + }, + { + "id": "triage-path", + "state": "available", + "evidence": "fixture: triage-path verified by caller" + }, + { + "id": "budget-policy", + "state": "available", + "evidence": "fixture: budget-policy verified by caller" + }, + { + "id": "idempotent-recovery", + "state": "available", + "evidence": "fixture: idempotent-recovery verified by caller" + } + ] +} diff --git a/test/fixtures/loop-readiness/duplicate-observation.json b/test/fixtures/loop-readiness/duplicate-observation.json new file mode 100644 index 0000000..1d25aa9 --- /dev/null +++ b/test/fixtures/loop-readiness/duplicate-observation.json @@ -0,0 +1,16 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "workspace-read", + "state": "blocked", + "evidence": "fixture: second observation for the same id" + } + ] +} diff --git a/test/fixtures/loop-readiness/empty-evidence.json b/test/fixtures/loop-readiness/empty-evidence.json new file mode 100644 index 0000000..55e80dc --- /dev/null +++ b/test/fixtures/loop-readiness/empty-evidence.json @@ -0,0 +1,11 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "" + } + ] +} diff --git a/test/fixtures/loop-readiness/empty-observations.json b/test/fixtures/loop-readiness/empty-observations.json new file mode 100644 index 0000000..d46533f --- /dev/null +++ b/test/fixtures/loop-readiness/empty-observations.json @@ -0,0 +1,5 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [] +} diff --git a/test/fixtures/loop-readiness/explicit-unavailable.json b/test/fixtures/loop-readiness/explicit-unavailable.json new file mode 100644 index 0000000..da78e3f --- /dev/null +++ b/test/fixtures/loop-readiness/explicit-unavailable.json @@ -0,0 +1,21 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "unavailable", + "evidence": "fixture: evidence root not found" + }, + { + "id": "privacy-boundary", + "state": "available", + "evidence": "fixture: privacy-boundary verified by caller" + } + ] +} diff --git a/test/fixtures/loop-readiness/malformed.json b/test/fixtures/loop-readiness/malformed.json new file mode 100644 index 0000000..4c68736 --- /dev/null +++ b/test/fixtures/loop-readiness/malformed.json @@ -0,0 +1 @@ +{ this is not json diff --git a/test/fixtures/loop-readiness/mixed-partial-failed.json b/test/fixtures/loop-readiness/mixed-partial-failed.json new file mode 100644 index 0000000..e024f4e --- /dev/null +++ b/test/fixtures/loop-readiness/mixed-partial-failed.json @@ -0,0 +1,41 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "available", + "evidence": "fixture: evidence-source verified by caller" + }, + { + "id": "privacy-boundary", + "state": "available", + "evidence": "fixture: privacy-boundary verified by caller" + }, + { + "id": "schedule-trigger", + "state": "available", + "evidence": "fixture: schedule-trigger verified by caller" + }, + { + "id": "stop-condition", + "state": "failed", + "evidence": "fixture: stop probe returned an error" + }, + { + "id": "triage-path", + "state": "available", + "evidence": "fixture: triage-path verified by caller" + }, + { + "id": "budget-policy", + "state": "partial", + "evidence": "fixture: budget declared without exhaustion policy" + } + ] +} diff --git a/test/fixtures/loop-readiness/non-required-blocked.json b/test/fixtures/loop-readiness/non-required-blocked.json new file mode 100644 index 0000000..60b70bc --- /dev/null +++ b/test/fixtures/loop-readiness/non-required-blocked.json @@ -0,0 +1,26 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "available", + "evidence": "fixture: evidence-source verified by caller" + }, + { + "id": "privacy-boundary", + "state": "available", + "evidence": "fixture: privacy-boundary verified by caller" + }, + { + "id": "plan-artifact-write", + "state": "blocked", + "evidence": "fixture: plan directory is read-only" + } + ] +} diff --git a/test/fixtures/loop-readiness/one-absent.json b/test/fixtures/loop-readiness/one-absent.json new file mode 100644 index 0000000..aea9971 --- /dev/null +++ b/test/fixtures/loop-readiness/one-absent.json @@ -0,0 +1,16 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "available", + "evidence": "fixture: evidence-source verified by caller" + } + ] +} diff --git a/test/fixtures/loop-readiness/one-blocked.json b/test/fixtures/loop-readiness/one-blocked.json new file mode 100644 index 0000000..b75646d --- /dev/null +++ b/test/fixtures/loop-readiness/one-blocked.json @@ -0,0 +1,51 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "evidence-source", + "state": "available", + "evidence": "fixture: evidence-source verified by caller" + }, + { + "id": "privacy-boundary", + "state": "available", + "evidence": "fixture: privacy-boundary verified by caller" + }, + { + "id": "plan-artifact-write", + "state": "available", + "evidence": "fixture: plan-artifact-write verified by caller" + }, + { + "id": "human-approval", + "state": "available", + "evidence": "fixture: human-approval verified by caller" + }, + { + "id": "frozen-pre-state", + "state": "available", + "evidence": "fixture: frozen-pre-state verified by caller" + }, + { + "id": "validation-route", + "state": "available", + "evidence": "fixture: validation-route verified by caller" + }, + { + "id": "rollback-reference", + "state": "available", + "evidence": "fixture: rollback-reference verified by caller" + }, + { + "id": "isolated-execution", + "state": "blocked", + "evidence": "fixture: no worktree isolation on this host" + } + ] +} diff --git a/test/fixtures/loop-readiness/unknown-capability.json b/test/fixtures/loop-readiness/unknown-capability.json new file mode 100644 index 0000000..4b920d9 --- /dev/null +++ b/test/fixtures/loop-readiness/unknown-capability.json @@ -0,0 +1,16 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + }, + { + "id": "workspace-write", + "state": "available", + "evidence": "fixture: misspelled capability" + } + ] +} diff --git a/test/fixtures/loop-readiness/unknown-state.json b/test/fixtures/loop-readiness/unknown-state.json new file mode 100644 index 0000000..78c656e --- /dev/null +++ b/test/fixtures/loop-readiness/unknown-state.json @@ -0,0 +1,11 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 1, + "observations": [ + { + "id": "workspace-read", + "state": "ready", + "evidence": "fixture: unknown state string" + } + ] +} diff --git a/test/fixtures/loop-readiness/unsupported-version.json b/test/fixtures/loop-readiness/unsupported-version.json new file mode 100644 index 0000000..411aa9c --- /dev/null +++ b/test/fixtures/loop-readiness/unsupported-version.json @@ -0,0 +1,11 @@ +{ + "kind": "loop-readiness-assessment", + "readinessContractVersion": 2, + "observations": [ + { + "id": "workspace-read", + "state": "available", + "evidence": "fixture: workspace-read verified by caller" + } + ] +} diff --git a/test/fixtures/scripts-refactor-contract/root-help.txt b/test/fixtures/scripts-refactor-contract/root-help.txt index 637c4d0..1237bce 100644 --- a/test/fixtures/scripts-refactor-contract/root-help.txt +++ b/test/fixtures/scripts-refactor-contract/root-help.txt @@ -20,6 +20,10 @@ Commands: preview-canvas, report-quality, repair-findings, record-fix-output, validate-canvas + Loop Control + loop Gate loop run levels with the fail-closed readiness contract + subcommands: readiness + Project Evidence session-analysis Collect and normalize Qoder, Codex, Claude, Cursor, Qwen, Copilot, Pi, and WorkBuddy session evidence diff --git a/test/loop-readiness.test.mjs b/test/loop-readiness.test.mjs new file mode 100644 index 0000000..eabefb5 --- /dev/null +++ b/test/loop-readiness.test.mjs @@ -0,0 +1,396 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; + +import { + CAPABILITY_IDS, + CAPABILITY_STATES, + READINESS_CONTRACT_VERSION, + READINESS_LEVELS, + REQUIRED_CAPABILITIES, +} from "../scripts/loop-readiness/contract.mjs"; +import { evaluateReadiness, ReadinessInputError } from "../scripts/loop-readiness/evaluate.mjs"; + +const repoRoot = process.cwd(); +const cliPath = path.join(repoRoot, "scripts", "loop-readiness", "cli.mjs"); +const facadePath = path.join(repoRoot, "scripts", "better-harness.mjs"); +const fixturesDir = path.join(repoRoot, "test", "fixtures", "loop-readiness"); + +function fixture(name) { + return path.join(fixturesDir, name); +} + +function loadFixture(name) { + return JSON.parse(readFileSync(fixture(name), "utf8")); +} + +function runCli(args, options = {}) { + return spawnSync(process.execPath, [cliPath, ...args], { + cwd: options.cwd ?? repoRoot, + encoding: "utf8", + }); +} + +function runFacade(args) { + return spawnSync(process.execPath, [facadePath, ...args], { + cwd: repoRoot, + encoding: "utf8", + }); +} + +function assertInvalid(fn, code) { + try { + fn(); + } catch (error) { + assert.ok(error instanceof ReadinessInputError, `expected ReadinessInputError, got ${error}`); + assert.equal(error.code, code, `expected ${code}, got ${error.code}: ${error.message}`); + return error; + } + return assert.fail(`expected ReadinessInputError ${code}, but no error was thrown`); +} + +// AC-1: versioned contract module. + +test("contract exports the five levels, five states, and a coherent v1 matrix", () => { + assert.equal(READINESS_CONTRACT_VERSION, 1); + assert.deepEqual([...READINESS_LEVELS], [ + "read-only-observation", + "plan-only", + "human-approved-apply", + "scheduled-read-only", + "scheduled-bounded-apply", + ]); + assert.deepEqual([...CAPABILITY_STATES], ["available", "partial", "unavailable", "blocked", "failed"]); + assert.equal(CAPABILITY_IDS.length, 14); + assert.deepEqual(Object.keys(REQUIRED_CAPABILITIES).sort(), [...READINESS_LEVELS].sort()); + for (const [level, required] of Object.entries(REQUIRED_CAPABILITIES)) { + assert.ok(required.length > 0, `${level} must require at least one capability`); + for (const id of required) { + assert.ok(CAPABILITY_IDS.includes(id), `${level} requires unknown capability ${id}`); + } + } + // The strictest level requires every contract capability. + assert.deepEqual([...REQUIRED_CAPABILITIES["scheduled-bounded-apply"]].sort(), [...CAPABILITY_IDS].sort()); + // Contract data is frozen. + assert.ok(Object.isFrozen(READINESS_LEVELS)); + assert.ok(Object.isFrozen(REQUIRED_CAPABILITIES)); + assert.ok(Object.isFrozen(REQUIRED_CAPABILITIES["read-only-observation"])); +}); + +test("evaluator rejects an unsupported readinessContractVersion", () => { + const error = assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("unsupported-version.json") }), + "UNSUPPORTED_CONTRACT_VERSION", + ); + assert.match(error.message, /supports only version 1/); +}); + +// AC-2: allowed decision envelope. + +test("all required capabilities available returns an allowed envelope with consumed evidence", () => { + for (const level of READINESS_LEVELS) { + const decision = evaluateReadiness({ level, assessment: loadFixture("all-available.json") }); + assert.equal(decision.kind, "loop-readiness-decision"); + assert.equal(decision.schemaVersion, 1); + assert.equal(decision.readinessContractVersion, 1); + assert.equal(decision.level, level); + assert.equal(decision.status, "allowed"); + assert.deepEqual(decision.blockingCapabilities, []); + assert.deepEqual( + decision.observations.map((observation) => observation.id).sort(), + [...REQUIRED_CAPABILITIES[level]].sort(), + `${level} must consume exactly its required observations`, + ); + for (const observation of decision.observations) { + assert.match(observation.evidence, /^fixture: /); + } + } +}); + +// AC-3: prevented decision envelope, absence reports unavailable. + +test("a blocked required capability prevents the level and reports its state", () => { + const decision = evaluateReadiness({ level: "human-approved-apply", assessment: loadFixture("one-blocked.json") }); + assert.equal(decision.status, "prevented"); + assert.deepEqual(decision.blockingCapabilities, [{ id: "isolated-execution", state: "blocked" }]); +}); + +test("an explicitly unavailable required capability prevents the level", () => { + const decision = evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("explicit-unavailable.json") }); + assert.equal(decision.status, "prevented"); + assert.deepEqual(decision.blockingCapabilities, [{ id: "evidence-source", state: "unavailable" }]); +}); + +test("an absent required capability prevents the level as unavailable", () => { + const decision = evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("one-absent.json") }); + assert.equal(decision.status, "prevented"); + assert.deepEqual(decision.blockingCapabilities, [{ id: "privacy-boundary", state: "unavailable" }]); +}); + +test("mixed partial and failed observations list every blocking capability", () => { + const decision = evaluateReadiness({ level: "scheduled-read-only", assessment: loadFixture("mixed-partial-failed.json") }); + assert.equal(decision.status, "prevented"); + assert.deepEqual( + decision.blockingCapabilities.sort((left, right) => left.id.localeCompare(right.id)), + [ + { id: "budget-policy", state: "partial" }, + { id: "stop-condition", state: "failed" }, + ], + ); +}); + +test("empty observations prevent the level listing every required capability", () => { + const decision = evaluateReadiness({ level: "plan-only", assessment: loadFixture("empty-observations.json") }); + assert.equal(decision.status, "prevented"); + assert.deepEqual( + decision.blockingCapabilities.map((capability) => capability.id).sort(), + [...REQUIRED_CAPABILITIES["plan-only"]].sort(), + ); + for (const capability of decision.blockingCapabilities) { + assert.equal(capability.state, "unavailable"); + } +}); + +test("a non-required contract capability observed blocked is ignored", () => { + const decision = evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("non-required-blocked.json") }); + assert.equal(decision.status, "allowed"); + assert.deepEqual(decision.blockingCapabilities, []); + assert.equal(decision.observations.some((observation) => observation.id === "plan-artifact-write"), false); +}); + +// Spec mutation regression: removing one required observation flips the +// decision to prevented and lists exactly that capability as unavailable. + +test("removing one required observation from a passing assessment flips to prevented", () => { + const assessment = loadFixture("all-available.json"); + assessment.observations = assessment.observations.filter((observation) => observation.id !== "rollback-reference"); + const decision = evaluateReadiness({ level: "scheduled-bounded-apply", assessment }); + assert.equal(decision.status, "prevented"); + assert.deepEqual(decision.blockingCapabilities, [{ id: "rollback-reference", state: "unavailable" }]); +}); + +// Invalid input never becomes a decision. + +test("evaluator rejects every input the assessment contract rejects", () => { + const valid = loadFixture("all-available.json"); + assertInvalid(() => evaluateReadiness({ level: "bogus", assessment: valid }), "UNKNOWN_LEVEL"); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("unknown-capability.json") }), + "UNKNOWN_CAPABILITY", + ); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("unknown-state.json") }), + "UNKNOWN_STATE", + ); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("duplicate-observation.json") }), + "DUPLICATE_OBSERVATION", + ); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: loadFixture("empty-evidence.json") }), + "EMPTY_EVIDENCE", + ); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: { ...valid, extra: true } }), + "UNKNOWN_FIELD", + ); + assertInvalid( + () => evaluateReadiness({ level: "read-only-observation", assessment: { kind: "loop-readiness-assessment", readinessContractVersion: 1 } }), + "MALFORMED_ASSESSMENT", + ); + assertInvalid(() => evaluateReadiness({ level: "read-only-observation", assessment: [] }), "MALFORMED_ASSESSMENT"); +}); + +// AC-4: CLI exit codes and parser-safe envelopes. + +test("CLI --help exits 0 and documents levels and exit codes", () => { + const result = runCli(["--help"]); + assert.equal(result.status, 0, result.stderr); + for (const level of READINESS_LEVELS) { + assert.ok(result.stdout.includes(level), `help must document level ${level}`); + } + assert.match(result.stdout, /Exit codes:/); + assert.match(result.stdout, /0 {2}allowed decision/); + assert.match(result.stdout, /2 {2}prevented decision/); +}); + +test("CLI with no arguments prints help and exits 0 without a decision", () => { + const result = runCli([]); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /Usage: better-harness loop readiness/); + assert.doesNotMatch(result.stdout, /Decision:/); +}); + +test("CLI allowed decision exits 0 with parser-safe JSON on stdout", () => { + const result = runCli(["--level", "read-only-observation", "--assessment", fixture("all-available.json"), "--json"]); + assert.equal(result.status, 0, result.stderr); + const decision = JSON.parse(result.stdout); + assert.equal(decision.kind, "loop-readiness-decision"); + assert.equal(decision.status, "allowed"); + assert.equal(result.stderr, ""); +}); + +test("CLI prevented decision exits 2 with the documented non-success envelope", () => { + const result = runCli(["--level", "human-approved-apply", "--assessment", fixture("one-blocked.json"), "--json"]); + assert.equal(result.status, 2, result.stderr); + const decision = JSON.parse(result.stdout); + assert.equal(decision.status, "prevented"); + assert.deepEqual(decision.blockingCapabilities, [{ id: "isolated-execution", state: "blocked" }]); +}); + +test("CLI human mode prints a readable allowed summary and exits 0", () => { + const result = runCli(["--level", "read-only-observation", "--assessment", fixture("all-available.json")]); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /Decision: allowed/); + assert.match(result.stdout, /Level: read-only-observation/); +}); + +test("CLI human mode prints a readable prevented summary and exits 2", () => { + const result = runCli(["--level", "read-only-observation", "--assessment", fixture("one-absent.json")]); + assert.equal(result.status, 2, result.stderr); + assert.match(result.stdout, /Decision: prevented/); + assert.match(result.stdout, /privacy-boundary: unavailable/); +}); + +test("CLI invalid inputs exit 1 with a parser-safe error envelope and never a decision", () => { + const cases = [ + { args: ["--level", "bogus", "--assessment", fixture("all-available.json"), "--json"], code: "UNKNOWN_LEVEL", match: /Unknown readiness level/ }, + { args: ["--level", "read-only-observation", "--assessment", fixture("malformed.json"), "--json"], code: "MALFORMED_ASSESSMENT", match: /not valid JSON/ }, + { args: ["--level", "read-only-observation", "--assessment", fixture("does-not-exist.json"), "--json"], code: "UNREADABLE_ASSESSMENT", match: /Cannot read assessment file/ }, + { args: ["--level", "read-only-observation", "--assessment", fixture("unsupported-version.json"), "--json"], code: "UNSUPPORTED_CONTRACT_VERSION", match: /supports only version 1/ }, + { args: ["--level", "read-only-observation", "--assessment", fixture("unknown-capability.json"), "--json"], code: "UNKNOWN_CAPABILITY", match: /Unknown capability id/ }, + { args: ["--level", "read-only-observation", "--json"], code: "INVALID_USAGE", match: /--assessment/ }, + { args: ["--unknown-flag", "--json"], code: "INVALID_USAGE", match: /Unknown argument/ }, + ]; + for (const { args, code, match } of cases) { + const result = runCli(args); + assert.equal(result.status, 1, `${args.join(" ")} must exit 1:\n${result.stdout}${result.stderr}`); + const envelope = JSON.parse(result.stdout); + assert.equal(envelope.kind, "loop-readiness-error", args.join(" ")); + assert.equal(envelope.code, code, `${args.join(" ")} expected ${code}, got ${envelope.code}`); + assert.match(envelope.message, match); + assert.equal(envelope.status, undefined, "invalid input must never carry a decision status"); + } +}); + +test("CLI human-mode invalid input reports the error on stderr", () => { + const result = runCli(["--level", "bogus", "--assessment", fixture("all-available.json")]); + assert.equal(result.status, 1); + assert.equal(result.stdout, ""); + assert.match(result.stderr, /Unknown readiness level/); +}); + +// AC-5: the gate reads only the assessment file. + +// Matches side-effect imports, default/named imports, and re-exports in both +// quote styles; plain `export const` declarations never match. +const IMPORT_PATTERN = /^import\s+["']([^"']+)["'];|^(?:import|export)\b[^;]*?\bfrom\s+["']([^"']+)["'];/gmu; + +function stripComments(source) { + return source.replace(/\/\*[\s\S]*?\*\//gu, " ").replace(/^\s*\/\/.*$/gmu, ""); +} + +function importSpecifiers(source) { + return [...source.matchAll(IMPORT_PATTERN)].map((match) => match[1] ?? match[2]); +} + +test("import allowlist pattern catches side-effect, single-quote, and re-export forms", () => { + assert.deepEqual(importSpecifiers('import "node:http";\n'), ["node:http"]); + assert.deepEqual(importSpecifiers("import x from 'node:net';\n"), ["node:net"]); + assert.deepEqual(importSpecifiers('export { x } from "node:dns";\n'), ["node:dns"]); + assert.deepEqual(importSpecifiers('import {\n a,\n} from "./contract.mjs";\n'), ["./contract.mjs"]); + assert.deepEqual(importSpecifiers('export const KIND = "value";\n'), []); +}); + +test("import allowlist defenses flag obfuscated static imports", () => { + // Same-line double import: only the first specifier is captured, so the + // leftover import token must trip the residue check. + const doubled = 'import a from "node:fs";import b from "node:net";\n'; + assert.deepEqual(importSpecifiers(doubled), ["node:fs"]); + assert.match(doubled.replace(IMPORT_PATTERN, ""), /\bimport\b/u); + // Line-prefix obfuscation survives comment stripping as a residue token. + const prefixed = stripComments('/**/import x from "node:net";\n'); + assert.match(prefixed.replace(IMPORT_PATTERN, ""), /\bimport\b/u); + const semiPrefixed = ';import x from "node:net";\n'; + assert.match(semiPrefixed.replace(IMPORT_PATTERN, ""), /\bimport\b/u); + // Re-export obfuscation carries no import token but leaves a from clause. + const reExport = ';export { x } from "node:dns";\n'; + assert.match(reExport.replace(IMPORT_PATTERN, ""), /\bfrom\s*["']/u); + // Indented and semicolon-free imports are caught by the declaration count. + for (const sneaky of [' import "node:os";\n', 'import "node:os"\n']) { + const lines = sneaky.split(/\r?\n/u).filter((line) => /^\s*(?:import\b|export\b[^;]*?\bfrom\b)/u.test(line)); + assert.ok(lines.length > importSpecifiers(sneaky).length, `declaration count must flag: ${sneaky.trim()}`); + } +}); + +test("gate modules keep the static import allowlist", () => { + const allowlists = { + "contract.mjs": [], + "evaluate.mjs": ["./contract.mjs"], + "cli.mjs": ["node:fs", "./contract.mjs", "./evaluate.mjs"], + }; + for (const [file, allowed] of Object.entries(allowlists)) { + const source = stripComments(readFileSync(path.join(repoRoot, "scripts", "loop-readiness", file), "utf8")); + const specifiers = importSpecifiers(source); + assert.deepEqual(specifiers.sort(), [...allowed].sort(), `${file} imports outside its allowlist`); + const declarationLines = source + .split(/\r?\n/u) + .filter((line) => /^\s*(?:import\b|export\b[^;]*?\bfrom\b)/u.test(line)); + assert.equal( + declarationLines.length, + specifiers.length, + `${file} has import/export-from lines the allowlist pattern did not capture`, + ); + const residue = source.replace(IMPORT_PATTERN, ""); + assert.doesNotMatch(residue, /\bimport\b/u, `${file} contains an import the allowlist pattern did not capture`); + assert.doesNotMatch(residue, /\bfrom\s*["']/u, `${file} contains a re-export the allowlist pattern did not capture`); + assert.doesNotMatch(source, /\brequire\s*\(/u, `${file} must not use require`); + assert.doesNotMatch(source, /import\s*\(/u, `${file} must not use dynamic import`); + } +}); + +test("CLI evaluates from an empty temporary working directory without workspace scanning", () => { + const tempDir = mkdtempSync(path.join(os.tmpdir(), "loop-readiness-")); + try { + const result = runCli( + ["--level", "read-only-observation", "--assessment", fixture("all-available.json"), "--json"], + { cwd: tempDir }, + ); + assert.equal(result.status, 0, result.stderr); + assert.equal(JSON.parse(result.stdout).status, "allowed"); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } +}); + +// Facade wiring: the loop group dispatches and reports its registry entry. + +test("better-harness loop readiness dispatches through the root facade", () => { + const result = runFacade(["loop", "readiness", "--level", "plan-only", "--assessment", fixture("all-available.json"), "--json"]); + assert.equal(result.status, 0, result.stderr); + assert.equal(JSON.parse(result.stdout).status, "allowed"); + + const direct = runCli(["--level", "plan-only", "--assessment", fixture("all-available.json"), "--json"]); + assert.equal(result.stdout, direct.stdout, "facade stdout must match direct invocation byte for byte"); + assert.equal(result.stderr, direct.stderr, "facade stderr must match direct invocation"); + assert.equal(result.status, direct.status, "facade exit code must match direct invocation"); + + const prevented = runFacade(["loop", "readiness", "--level", "plan-only", "--assessment", fixture("empty-observations.json"), "--json"]); + assert.equal(prevented.status, 2, prevented.stderr); +}); + +test("command inventory registers the loop group with the readiness subcommand", () => { + const result = runFacade(["commands", "--json", "--audience", "advanced"]); + assert.equal(result.status, 0, result.stderr); + const payload = JSON.parse(result.stdout); + const loop = payload.data.commands.find((command) => command.name === "loop"); + assert.ok(loop, "loop group must appear in the advanced command inventory"); + assert.equal(loop.kind, "group"); + assert.equal(loop.audience, "advanced"); + const readiness = loop.subcommands.find((subcommand) => subcommand.name === "readiness"); + assert.ok(readiness, "loop group must expose the readiness subcommand"); + assert.equal(readiness.script, "scripts/loop-readiness/cli.mjs"); +}); diff --git a/test/scripts-refactor-contract.test.mjs b/test/scripts-refactor-contract.test.mjs index 1dc3155..c2f7c22 100644 --- a/test/scripts-refactor-contract.test.mjs +++ b/test/scripts-refactor-contract.test.mjs @@ -107,12 +107,12 @@ test("scripts refactor contract freezes machine-readable CLI output", () => { { label: "command inventory", args: ["commands", "--json"], - sha256: "e86c8b3447f24cf1f6a94f7ca74c48c5d194aa54bad736ff02d7dd64e566dc99", + sha256: "4d76d8187af5e9c5574e8448bb422be07228821ff8a17477fcd93b60c5a08ac0", }, { label: "OpenCLI schema", args: ["schema"], - sha256: "b6b28fede8f135e598bc2278a36d2c94660edfe1b384c14f6e5da73c394a3347", + sha256: "26f868e9a7d5a53b4f57706c2277af739c8a9e22a7bcf8bcc811ce36847cfc00", }, { label: "Harness command description", From 0ea973ce545f8a768443961cdddc7e1b1b2d9ee9 Mon Sep 17 00:00:00 2001 From: creayma Date: Fri, 31 Jul 2026 18:30:35 +0800 Subject: [PATCH 2/2] docs(changelog): record loop readiness gate under Unreleased Record the new `better-harness loop readiness` fail-closed gate in the Unreleased Added section so the user-facing CLI addition from the LC-01 spec is covered by the changelog checklist on PR #38. Validated with the full suite `npm test`. Co-authored-by: QoderAI (Qwen 3.8 Max) --- CHANGELOG.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4a7268..5790de6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ observable behavior and compatibility, not every internal refactor. ## Unreleased +### Added + +- `better-harness loop readiness` (audience `advanced`) is a new fail-closed + gate for loop run levels. It evaluates a caller-declared capability + assessment against the versioned LC-01 readiness contract (v1, 14 + capabilities across 5 run levels) without probing hosts, network, or the + workspace: absent or non-`available` required capabilities prevent the + level (exit 2 with every blocking capability listed), and any input the + assessment contract rejects (unknown id/state/field, duplicate observation, + empty evidence, unsupported version, malformed JSON) exits 1 without ever + producing a decision. `--json` keeps stdout parser-safe. + ### Fixed - The Portable HTML report route in `templates/reporting/routing.md` now