diff --git a/docs/cyberbattlesim-researcher-command.md b/docs/cyberbattlesim-researcher-command.md index 3fd69c8..d3c3530 100644 --- a/docs/cyberbattlesim-researcher-command.md +++ b/docs/cyberbattlesim-researcher-command.md @@ -19,6 +19,14 @@ raes-adapters inspect --backend cyberbattlesim-chain pinned native source is available and byte-verified, along with the qualified source commit and supported RAES profiles. +Native `validate`, `run --mode smoke`, and `run --mode study` are currently +fail-closed for the selected task. Its attacker-action-log and +availability-series requirements cannot be verified at artifact-field and +data-quality granularity by the pinned RAES contract. These commands exit `3` +before runtime planning, native import, output reservation, or simulator +effects. Issue #86 owns the CyberBattleSim capture/manifest remediation; the +authored task is not weakened in the interim. + The native simulator remains separately installed because upstream publishes no selected index or release artifact. Build or acquire the wheel from commit `854d6966607fb68645651f55b0f97221bd293e0d`, verify the complete identity diff --git a/docs/decisions/capability-and-evidence-claim-guardrails.md b/docs/decisions/capability-and-evidence-claim-guardrails.md new file mode 100644 index 0000000..dade192 --- /dev/null +++ b/docs/decisions/capability-and-evidence-claim-guardrails.md @@ -0,0 +1,122 @@ +# Capability and evidence-claim integrity guardrails + +GitHub issue #84 is the authority for this repository-wide correction. This +note fixes the claim boundaries every adapter, shared helper, command, and +conformance lane must respect. It defines no new RAES contract, capability, +evidence type, status vocabulary, or implementation plan. + +## Keep four claims distinct + +An authored task requirement, a backend capability declaration, a captured +run artifact, and a post-run satisfaction claim are different facts with +different owners: + +- `ExperimentTaskModel` owns what evidence the author requires. A requirement + is demand, not proof that an adapter can capture it. +- The published RAES `BackendManifest` and capability models own what the + selected production target advertises. A capability is admissible only when + its production component and capture path are executable; a constraint, + source-ledger row, injected driver, or planned feature is not a substitute. +- `ExperimentEvidenceRecordModel`, `ExperimentDerivedMeasureModel`, and + `ExperimentArtifactRefModel` describe what one run actually emitted. A file + containing an evaluator summary is not an action log, availability series, + host-compromise series, or reward-component record merely because all of + them concern the same episode. +- `validate_experiment_run_against_task()` owns the task/run satisfaction join. + Conformance, qualification, source admission, cleanup, or a successful run + cannot bypass that join or manufacture its inputs. + +The pinned RAES contract validates a semantic `satisfies_refs` entry by +reference identity. `ExperimentEvidenceSatisfactionReferenceModel` cannot +express required-field coverage or the availability, redaction, withholding, +or loss status of those fields. Therefore a generic authored evidence concept +must not be placed in `satisfies_refs` until the applicable published RAES +contract can express and validate the complete artifact-and-field witness. +Current researcher tasks that require such a concept must fail closed even if +that makes an adapter or example temporarily non-runnable. An exact authored +artifact identity remains admissible only where the existing RAES validator +can verify its identity and any authored digest/path constraints directly. + +## Canonical incumbents + +| Concern | Canonical owner and required use | +| --- | --- | +| Manifest and capability shape | RAES `BackendManifest`, `BackendCapabilitySet`, component capability models, `backend_manifest_payload()`, capability-admission helpers, and `RuntimeTarget` presence/signature validation. Do not add an adapter capability schema or infer support from component existence alone. | +| Runtime admission | `RuntimeManager.plan()`, public target components, `ApplyResult`, snapshot-transition validation, participant admission/history validation, evaluator result validation, and cleanup receipt validation. Every affirmative production claim must survive its applicable public execution path. | +| Experiment evidence | RAES capture-spec, evidence-record, derived-measure, artifact, run, and task models plus `_experiment_evidence.py` and `_researcher_support.py` for mechanics only. Backend-local code owns source projection; shared helpers must not assign semantic evidence identities. | +| Task/run joins | `validate_experiment_run_against_task()` and `validate_experiment_study_against_tasks_and_runs()`. Do not copy their join logic or predeclare a positive result in `_BackendAdapter`, a manifest constraint, or a pack file. | +| Manifest conformance | `run_conformance_probe()`, canonical RAES report projection/writer, and executable adapter-local probes. `affirmative_capability_pointers()` is inventory only; a static pointer-to-reference table and one broad pass flag are not proof that each leaf was exercised. | +| Source truth | Backend qualification records, source admission, scenario/source ledgers, and loss disclosures. These establish provenance and bounded source facts; they do not satisfy per-run capture requirements. | +| Failures and disclosure | RAES `Diagnostic`/`DiagnosticModel`, `diagnostic_model()`, `ApplyResult`, stable command exit codes, and `base.redaction`. Missing, unavailable, withheld, redacted, lossy, unsupported, and failed must remain distinct non-success dispositions. | +| Persistence | `atomic_write_json_artifact()`, the RAES conformance report writer, exclusive confined output roots, and inventory-last sealing. Runtime state remains behind RAES runtime/control-plane ownership; no evidence registry, cache, or adapter store is introduced. | +| Verification | Existing repository tests, clean-installed distribution probes, the single nox graph, and the `PR Gate`. Negative tests must exercise every registered adapter and shared command path, not a hand-maintained subset that silently omits the next adapter. | + +Backend manifests may contain declarative values, but those values are not +self-authenticating. Shared constructors such as standard evaluator, +orchestrator, or cleanup capability builders may factor shape only after the +caller supplies truth established by the production path. They must not grant +support merely because several gym-style adapters are expected to share it. +Likewise, reporting every PrimAITE capability as an open conformance gap does +not make its affirmative production manifest truthful; an inadmissible adapter +is allowed to break. + +## Validation and security path + +| Layer the design passes | Required treatment | +| --- | --- | +| Authentication and authorization | The current researcher command is local and adds no auth surface. Participant authority still comes from exact manifest/selection/configuration joins and public participant admission. Any later network surface must use the RAES strict-default control-plane security, verified identity, role/target authorization, request limits, denial audit, and redacted exception handling; adapters do not add endpoints. | +| Secrets and environment bindings | No claim path reads credentials, a secret store, `.env`, or ambient configuration. Do not add token options or environment-selected capability/evidence overrides. Native credentials, action details, observations, rewards, argv, environment maps, and source paths never enter portable evidence or diagnostics. | +| Static input and config shape | Reuse closed `argparse` choices, per-backend required/foreign argument checks, pack digest validation, confined child resolution, duplicate-key-rejecting JSON loading, RAES SDL parsing, closed contract models, participant joins, target config normalizers, and selected-source admission. No arbitrary import, driver, profile, schema, or capture map is caller-selectable. | +| Runtime validators | Preserve manifest/component checks, capability admission, plan/resource/dependency validation, `ApplyResult` shape, snapshot transitions, participant action/result/history joins, evaluator/proposition checks, and cleanup verification. A native transition with an unverifiable projection is failure, not partial evidence satisfaction. | +| OS and process exposure | Keep relative confined outputs, exclusive mode-0700 creation, atomic publication, no shell interpolation or runtime download, clean-install isolation, cleared `PYTHONPATH`, `PYTHONSAFEPATH=1`, and discarded native stdout/stderr. Do not put evidence payloads, credentials, or native paths in argv or filenames. | +| Error envelopes and observability | Reuse bounded `_CommandFailure` messages, RAES diagnostics, canonical report projection, and default-deny redaction. Logs and terminal output may carry safe identities, pointers, counts, and dispositions only. Never serialize exception text, rejected values, native output, object representations, or tracebacks. | +| Artifact publication | Validate RAES models and task/run joins before sealing success; write the final inventory last. A checksum proves byte identity, not semantic completeness or safety. Failure, cleanup failure, or an unsatisfied requirement cannot be published with a successful disposition. | + +## Extension seam + +The extension seam is the existing backend strategy boundary, parameterized by +the live `BackendManifest`, the authored `ExperimentTaskModel`, and the actual +validated evidence records/artifact bytes from that run. A future published +RAES satisfaction contract may be consumed there without changing authored +task semantics or adding a repository schema. Until that owner can validate +field-level witnesses and negative data-quality states, the seam returns no +semantic satisfaction claim and lets the canonical task/run validator reject +the run. + +A future adapter registers with the existing command/target strategy and is +automatically included by repository-wide claim-integrity tests. It must not +require editing a global evidence allowlist, standard capability grant, copied +schema, or backend-name conditional. + +## Gotchas and anti-patterns + +- Do not retain `evidence_satisfies_refs`, a backend-name evidence allowlist, + unconditional `supports_* = True`, or a test-only/injected-driver bypass. +- Do not promote a source-ledger reference, capability pointer, conformance + evidence id, capture-spec declaration, content checksum, or evidence-record + existence into per-run satisfaction. +- Do not let an evaluator summary satisfy an action/observation/time-series + requirement when those records and required fields were not emitted. +- Do not treat missing, unavailable, redacted, withheld, lossy, unknown, + unsupported, partial, or unverified as aliases for satisfied. +- Do not define a local field-witness DTO, evidence status enum, validator, + exception hierarchy, manifest extension, profile, registry, or persistence + service to work around a missing RAES contract. +- Do not make positive tests depend only on current adapters. Mutation and + negative cases must catch a new manifest leaf, a new adapter registration, + an omitted artifact, missing required content, a negative data-quality state, + and a static reference reintroduced through any shared path. + +## Non-goals and boundaries + +Issue #84 does not implement missing action logs, availability or compromise +series, reward-component capture, source qualification, deterministic replay, +scientific equivalence, or new researcher backends. It does not rewrite +authored task semantics merely to keep current examples runnable. + +It also does not add or change a RAES schema, capability vocabulary, evidence +type, status model, validator, profile, diagnostic envelope, controller, store, +HTTP surface, console script, distribution, lockfile, or workflow. If a +published RAES contract cannot express and verify the required claim, the +repository records the gap by failing closed rather than creating local +authority. diff --git a/docs/decisions/cyberbattlesim-conformance-guardrails.md b/docs/decisions/cyberbattlesim-conformance-guardrails.md index 93b6a82..a6be4ba 100644 --- a/docs/decisions/cyberbattlesim-conformance-guardrails.md +++ b/docs/decisions/cyberbattlesim-conformance-guardrails.md @@ -74,14 +74,12 @@ off as a copied fixture family. The published realization harness is used only for realization-envelope questions it actually models; it is not a generic hook for unrelated source-protocol assertions. -Every affirmative manifest capability must join to passing executable evidence. -Derive the capability addresses from `backend_manifest_payload()` at runtime and -join them to stable adapter-probe evidence references. The join may be a small -module-local test/probe inventory, but it is not portable authority: it carries -only manifest JSON pointers and evidence references, contains no copied -capability values or expected manifest payload, and fails closed when a newly -declared affirmative capability has no passing evidence. Negative declarations -and limitations must also be exercised so absence is not mistaken for support. +Every affirmative manifest capability is derived from +`backend_manifest_payload()` at runtime and retained as unresolved inventory. +There is no module-local pointer-to-reference join: a broad conformance pass, +source validation, or adapter probe cannot certify each leaf. Negative +declarations and limitations must also be exercised so absence is not mistaken +for support. Where RAES already provides `evidence_refs`, `limitation_refs`, claim `limitations`, or `explicit_non_claims`, reuse those fields rather than creating adapter equivalents. @@ -98,7 +96,7 @@ constructed `RuntimeTarget` with an injected deterministic driver to run the published profile/fixtures and bounded local probes. It must cover manifest and source/profile identity, all four surfaces on success and failure, reset and stream dispositions, action/observation/evaluator separation, terminal -semantics, cleanup, capability-to-evidence closure, and portable serialization. +semantics, cleanup, unresolved capability inventory, and portable serialization. It must not import the native simulator or depend on network, user-home state, or an editable checkout. diff --git a/docs/decisions/cyberbattlesim-qualification-guardrails.md b/docs/decisions/cyberbattlesim-qualification-guardrails.md index 4ef4b74..e13454f 100644 --- a/docs/decisions/cyberbattlesim-qualification-guardrails.md +++ b/docs/decisions/cyberbattlesim-qualification-guardrails.md @@ -106,7 +106,8 @@ native-readiness plan must extend this same isolated source-native run instead of stopping at the upstream smoke. For issue #28, that means running the adapter conformance path with the real `CyberBattleSimDriver`, serializing the result through `backend_conformance_report_payload()`, collecting -`cyberbattlesim_source_protocol_diagnostics()` and manifest capability evidence, +`cyberbattlesim_source_protocol_diagnostics()` and unresolved manifest +capability inventory, and checking the emitted RAES payloads remain bounded and free of native action ids, observations, reward vectors, object representations, paths, environments, tracebacks, and hidden state. This is adapter-readiness evidence in this repo; diff --git a/docs/decisions/cyborg-conformance-guardrails.md b/docs/decisions/cyborg-conformance-guardrails.md index ca0c55d..674befa 100644 --- a/docs/decisions/cyborg-conformance-guardrails.md +++ b/docs/decisions/cyborg-conformance-guardrails.md @@ -86,22 +86,21 @@ construction. | Packaging and CI | The single `pyproject.toml`/`uv.lock`, `_verification_envs()`, `_tests()`, `_distributions()`, `probe_installed_identity.py`, the existing CI workflow and `PR Gate`, strict docs/policy gates, and canonical `nox -s verify`. Extend these paths; do not create a second lock, workflow, or unenforced verification graph. | The CyberBattleSim conformance module is the closest repository precedent for -canonical report projection, manifest-derived evidence closure, RAES -diagnostics, weakness references, and hostile-value tests. Reuse its composition -pattern, not its source protocol, action model, seed semantics, cleanup claims, -capability inventory, or backend-specific helper module. +canonical report projection, RAES diagnostics, weakness references, and +hostile-value tests. Reuse its composition pattern, not its source protocol, +action model, seed semantics, cleanup claims, capability inventory, or +backend-specific helper module. -## Local probes, capability evidence, and leakage +## Local probes, capability inventory, and leakage Derive affirmative capability JSON pointers from the live -`backend_manifest_payload()` and join them to stable references from probes that -actually passed. A small module-local pointer-to-evidence requirement map is a -closure check, not capability authority: it contains no copied capability -values, fails when a new affirmative surface has no evidence, and does not turn -constraints or component names into capabilities. Exercise negative and -unsupported declarations as well, especially replay, autonomous execution, -bounded concurrency, execution control, accounts, ACLs, generated artifacts, -persistent volumes, and cleanup when undeclared. +`backend_manifest_payload()` as unresolved inventory only. A broad published +conformance disposition, source-ledger validation, or adapter-local probe does +not prove every capability leaf, so there is no pointer-to-evidence requirement +map or positive join. Exercise negative and unsupported declarations as well, +especially replay, autonomous execution, bounded concurrency, execution +control, accounts, ACLs, generated artifacts, persistent volumes, and cleanup +when undeclared. Weaknesses are machine-resolvable references derived from qualification `admission.limitations`, `known_defects`, and the selected loss disclosures. @@ -225,11 +224,12 @@ does not rewrite, append, or locally reclassify that case. The executable adapter-local probes separately construct a supported switch/VM topology with a hostile injected native handle, validate the source selection and every declared runtime surface, verify cleanup, and inspect actual portable -projections. Capability evidence requires all three independent references: -the bounded published conformance disposition, validated source-ledger evidence, -and passing adapter-runtime diagnostics. A future RAES release that publishes a -constructive list-domain or equivalent governed witness seam can replace this -unsupported case without a local schema or profile. +projections. The bounded published conformance disposition, validated +source-ledger evidence, and passing adapter-runtime diagnostics remain three +independent facts; their conjunction is not per-leaf capability evidence. A +future RAES release that publishes a constructive list-domain or equivalent +governed witness seam can replace this unsupported case without a local schema +or profile. The checked-in full tier remains hermetic at ordered seeds `(3, 153)` and keeps `native_conformance=false`; the existing qualified-source reproducer owns native @@ -247,8 +247,8 @@ registry, or environment binding. A new qualified selection may add source resources, evidence bindings, seeds, or a native harness without changing the RAES profile corpus, report type, diagnostic model, control-plane persistence, or cross-simulator base API. A new -affirmative manifest claim automatically creates an evidence-closure gap until -a passing probe reference is joined to its derived pointer. +affirmative manifest claim automatically remains an unresolved inventory gap +until a published contract can verify the owning production path. ## Gotchas and anti-patterns diff --git a/docs/decisions/nasim-conformance-guardrails.md b/docs/decisions/nasim-conformance-guardrails.md index 9b50091..363ee93 100644 --- a/docs/decisions/nasim-conformance-guardrails.md +++ b/docs/decisions/nasim-conformance-guardrails.md @@ -51,7 +51,7 @@ Implementation must build on these incumbents: | Cleanup | `execute_nasim_cleanup()`, cleanup capability admission, base `execute_cleanup()`, `TrialCleanupPlanModel`, `TrialCleanupReceiptModel`, and `validate_trial_cleanup_receipt()`. A close boolean is not verified clean state. | | Diagnostics and failure hygiene | RAES `Diagnostic`, `diagnostic_model()`, `diagnostic_payload()`, `ApplyResult`; shared `diagnostic_address()`, `redact_native_value()`, and `bounded_context_label()`. Use stable pointer-addressed, input-free failures; add no exception or diagnostic hierarchy. | | Report and artifact persistence | `backend_conformance_report_payload()` and `write_backend_conformance_report()` own canonical report validation, redaction checking, safe run labels, and atomic persistence. Other JSON uses `atomic_write_json_artifact()` only after its content is validated; no checked-in generated report or new evidence repository. | -| Capability evidence | `_conformance_support` and `_gym_backend.conformance` own live-manifest traversal, fail-closed pointer coverage, passed-evidence accounting, source diagnostics, and weakness derivation. Add NASim-local passing evidence through their explicit configuration/input seam; do not fork the traversal or maintain a second capability catalog. | +| Capability inventory | `_conformance_support` and `_gym_backend.conformance` own live-manifest traversal, unresolved-pointer inventory, source diagnostics, and weakness derivation. They produce no positive per-leaf evidence join; do not fork the traversal or maintain a second capability catalog. | | Packaging and workflow | The single `pyproject.toml`/`uv.lock`, the isolated `nasim` extra, `_extras()`, `_verification_envs()`, `_tests()`, `_distributions()`, `probe_installed_identity.py`, `.github/workflows/ci.yml`, its `PR Gate`, and canonical `nox -s verify`. Extend these paths; do not add a lock, distribution, combined-extras environment, or parallel workflow. | The existing `tests/test_nasim_backend.py`, `tests/test_nasim_driver.py`, @@ -63,7 +63,7 @@ semantics in a second test-only controller. The current clean-wheel generalize to the installed suite, not a second inline suite to grow in parallel. -## Additive probes and evidence closure +## Additive probes and unresolved capability inventory Adapter-local probes cover facts RAES cannot know: the selected source/ledger join, installed-source identity, stochastic-stream dispositions, private action @@ -71,15 +71,12 @@ mapping, observation sealing, evaluator ownership, native terminal semantics, and verified close. They do not become `BackendConformanceReport.cases`, a fixture overlay, or a realization-harness misuse. -Every affirmative manifest capability must resolve from the live -`backend_manifest_payload()` to evidence references owned by probes that -passed. The existing shared pointer traversal remains the closure check. Stable -NASim-local probe references may be requirements for the applicable component -surface, but the requirement mapping contains only JSON pointers and evidence -references—not copied capability values—and must fail closed when a new -affirmative capability has no evidence. Source-ledger validation alone is not -runtime evidence; canonical target conformance alone is not source-protocol -evidence. +Every affirmative manifest capability is inventoried from the live +`backend_manifest_payload()` as unresolved. The shared pointer traversal is +inventory only: there is no static pointer-to-reference map and no broad pass +flag can certify a leaf. Source-ledger validation alone is not runtime evidence; +canonical target conformance alone is not source-protocol evidence; their +conjunction is still not field-verifiable capability evidence. Leakage coverage is the success/failure cross-product over the four declared backend surfaces—provisioner, orchestrator, participant runtime, and evaluator— @@ -106,7 +103,7 @@ impossible. The PR suite is fixed, deterministic, offline, and dependency-light. It uses the selected public seed `20260802`, a fully constructed `RuntimeTarget`, and an explicit deterministic injected driver. It runs the published profile/corpus, -the additive success/failure probes, capability-evidence closure, leakage +the additive success/failure probes, unresolved capability inventory, leakage checks, report projection, and cleanup. The suite never derives seed or case order from time, PR number, hashing, environment, test order, or global random state. @@ -127,8 +124,8 @@ qualified runtime from issue #32. It must explicitly construct a real inventory without substituting a stub/mock at any point, run `run_nasim_conformance()` through that target, serialize the canonical report through `backend_conformance_report_payload()`, collect -`nasim_source_protocol_diagnostics()` and manifest capability evidence, validate -every local diagnostic, close capability evidence, and verify cleanup. It does +`nasim_source_protocol_diagnostics()` and unresolved capability inventory, +validate every local diagnostic, and verify cleanup. It does not replace the deterministic injected-driver CI probe; it exercises the same composition against installed NASim. The emitted report, diagnostics, evidence references, and cleanup receipt must not contain native action coordinates, diff --git a/docs/decisions/primaite-conformance-guardrails.md b/docs/decisions/primaite-conformance-guardrails.md index 806ed69..f7f417c 100644 --- a/docs/decisions/primaite-conformance-guardrails.md +++ b/docs/decisions/primaite-conformance-guardrails.md @@ -53,7 +53,7 @@ CyberBattleSim/NASim/CybORG conformance suites are precedents for composition and hostile leakage tests. Reuse their shared mechanics and patterns, not their source semantics or private driver code. -## Additive probes and evidence closure +## Additive probes and unresolved capability inventory Adapter-local probes cover facts RAES cannot infer from the generic profile: selected source and profile identity, source-ledger coverage, reset and @@ -63,18 +63,12 @@ cutoff semantics, cleanup, and portable-output leakage. They must not become a second profile table, fixture corpus, schema registry, manifest authority, or copied capability catalog. Derive affirmative capability -JSON pointers from the live `backend_manifest_payload()` and join each pointer -to stable evidence references only when the owning probes passed. A newly -declared affirmative capability with no explicit evidence requirement must fail -closed as an evidence gap. - -Capability evidence is lane-scoped. A deterministic injected-driver PR run may -close the portable-mechanics evidence it actually exercises, with -`native_conformance=false` and explicit non-claims. It must not certify that the -live PrimAITE source runs safely in-process, that CPython 3.12 native evidence -exists, or that stochastic replay/equivalence was achieved. Any no-argument -production-live evidence API remains conservative unless a real, qualified, -isolated live driver supplies passing evidence. +JSON pointers from the live `backend_manifest_payload()` as unresolved inventory +only. A deterministic injected-driver PR run may prove the bounded mechanics it +actually exercises, with `native_conformance=false` and explicit non-claims, but +that broad result is not per-leaf capability evidence. It must not certify that +the live PrimAITE source runs safely in-process, that CPython 3.12 native evidence +exists, or that stochastic replay/equivalence was achieved. Leakage coverage is the success/failure cross-product over the four declared backend surfaces: provisioner, orchestrator, participant runtime, and evaluator. @@ -96,7 +90,7 @@ success cannot be synthesized after rollback is impossible. The PR suite is deterministic, offline, dependency-light, and clean-installable under the `primaite` extra. It uses a fully constructed `RuntimeTarget`, an explicit injected driver, fixed ordered seeds, published RAES conformance, -source-protocol diagnostics, adapter-local probes, capability-evidence closure, +source-protocol diagnostics, adapter-local probes, unresolved capability inventory, declared weaknesses, leakage checks, report projection, and cleanup. It must not derive seed, case order, corpus roots, or suite tier from time, PR number, hashing, global random state, test order, or environment variables. @@ -117,8 +111,9 @@ ownership: report writer; - local diagnostics are validated with `diagnostic_model()` and serialized with `diagnostic_payload()`; -- capability evidence is pointer-to-reference data derived from the live - manifest, not copied capability values; +- affirmative capability pointers are unresolved inventory derived from the + live manifest, never pointer-to-reference evidence manufactured from broad + passed probes; - declared weaknesses are derived from qualification limitations and selected loss disclosures, not a second list; and - reproduction commands are fixed argv arrays with no shell interpolation, diff --git a/docs/nasim-researcher-command.md b/docs/nasim-researcher-command.md index 2423824..287eafb 100644 --- a/docs/nasim-researcher-command.md +++ b/docs/nasim-researcher-command.md @@ -43,6 +43,14 @@ neither an `ExperimentRunModel` nor a native attacker episode merely because it shares the command. The conformance run reports `execution_basis` `installed-source-probe`. +Native `validate`, `run --mode smoke`, and `run --mode study` are currently +fail-closed for the packaged task. Its action-log and host-compromise-series +requirements cannot be verified at artifact-field and data-quality granularity +by the pinned RAES contract. These commands exit `3` before runtime planning, +native import, output reservation, or simulator effects. Issue #87 owns the +NASim capture/manifest remediation; the authored task is not weakened in the +interim. + The final line of each run prints only a bounded JSON summary. The newly reserved output root contains the canonical conformance reports and an `inventory.json` seal with relative artifact names, sizes, and SHA-256 digests. @@ -55,9 +63,8 @@ single autonomous red bruteforce attacker. NASim declares **no** red variant and **no** defender: the attacker is fixed by the admitted participant implementation, so the selection surface is the closed `--participant-*` set — there is no `--red-variant` and no `--blue-*` flag. Validation performs the -complete admission path without native construction: pack bytes, SDL, task/spec -joins, participant manifest/selection/configuration bindings, controls, seeds, -and runtime planning: +pack, SDL, task/spec, and participant joins, then rejects the current +unverifiable semantic evidence requirements before runtime planning: ```shell raes-adapters validate --backend nasim-tiny --mode study \ diff --git a/docs/researcher-command.md b/docs/researcher-command.md index 3a448c9..3321de7 100644 --- a/docs/researcher-command.md +++ b/docs/researcher-command.md @@ -20,12 +20,19 @@ root contains canonical conformance reports and an `inventory.json` seal with relative artifact names, sizes, and SHA-256 digests. An existing output path is always rejected. +Native `validate`, `run --mode smoke`, and `run --mode study` are currently +fail-closed for the packaged task. That task requires semantic reward-component +evidence, while the pinned RAES contract cannot verify required artifact fields +or negative data-quality states. These commands exit `3` before runtime +planning, native import, output reservation, or simulator effects. Issue #85 +owns the CybORG capture/manifest remediation; the authored task is not weakened +in the interim. + ## Validate the packaged example The wheel includes the symbolic environment-pack identity `cage2-research`. -Validation performs the complete non-native admission path: pack bytes, SDL, -task/spec joins, participant manifest/selection/configuration bindings, controls, -seeds, and runtime planning: +Validation performs pack, SDL, task/spec, and participant joins, then rejects +the current unverifiable semantic evidence requirement before runtime planning: ```shell raes-adapters validate \ diff --git a/mkdocs.yml b/mkdocs.yml index 4b60949..422d8c5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,6 +40,7 @@ nav: - Overview: decisions/adrs/README.md - Template: decisions/adrs/TEMPLATE.md - Shared simulator-adapter plumbing guardrails: decisions/base-plumbing-guardrails.md + - Capability and evidence-claim integrity guardrails: decisions/capability-and-evidence-claim-guardrails.md - CybORG/CAGE-2 backend qualification guardrails: decisions/cyborg-cage2-runtime-qualification-guardrails.md - CybORG/CAGE-2 source-ledger guardrails: decisions/cyborg-cage2-source-ledger-guardrails.md - CybORG/CAGE-2 Scenario2 SDL guardrails: decisions/cyborg-cage2-scenario2-sdl-guardrails.md diff --git a/noxfile.py b/noxfile.py index 728d416..dad1857 100644 --- a/noxfile.py +++ b/noxfile.py @@ -55,6 +55,7 @@ DriverResetReport, DriverStep, cyberbattlesim_backend_conformance_payload, + cyberbattlesim_manifest_capability_gaps, run_cyberbattlesim_conformance, ) from raes_adapters.cyberbattlesim.scenario_ledger import validate_all @@ -136,6 +137,7 @@ def verify_closed(self): assert payload["passed"] is True assert payload["native_conformance"] is False assert payload["cases"] +assert cyberbattlesim_manifest_capability_gaps() """ NASIM_CONFORMANCE_PROBE = r""" @@ -212,7 +214,7 @@ def verify_closed(self): # Compose the PR conformance evidence bundle from the installed wheel: the # published report projection (which drives this injected driver through the # constructed target's four surfaces on the published fixtures), source-protocol -# diagnostics, fail-closed capability evidence, and declared weaknesses. The +# diagnostics, unresolved capability inventory, and declared weaknesses. The # hostile-failure / leakage probes are injected-driver test constructs and live # in the PR test suite, not this happy-path installed-package proof. assert validate_all() == [] @@ -221,7 +223,7 @@ def verify_closed(self): assert bundle["backend_conformance"]["passed"] is True assert bundle["backend_conformance"]["cases"] assert bundle["source_diagnostics"] -assert bundle["capability_evidence"] +assert bundle["capability_gaps"] assert bundle["declared_weaknesses"] # Serializable to the portable JSON the evidence bundle claims to be. json.dumps(bundle, sort_keys=True) @@ -243,6 +245,7 @@ def verify_closed(self): assert index["adapter_diagnostics"] assert index["adapter_diagnostics"][0]["seed"] == 3 assert all(report["native_conformance"] is False for report in index["reports"]) +assert all(report["capability_gaps"] for report in index["reports"]) assert (Path(sys.argv[1]) / "index.json").is_file() json.dumps(index, sort_keys=True) """ @@ -323,21 +326,34 @@ def verify_closed(self): # Compose the PR conformance disclosure bundle from the installed wheel. PrimAITE # is fail-closed: the live driver is non-runnable in-process, so an injected # driver keeps native_conformance false, the canonical report keeps its single -# published no-witness case, capability evidence stays empty, and every -# affirmative capability is disclosed as a gap rather than certified. +# published no-witness case, and every affirmative capability is disclosed as +# a gap rather than certified. assert validate_all() == [] bundle = run_primaite_pr_conformance(driver=Driver()) assert bundle["native_conformance"] is False assert bundle["backend_conformance"]["native_conformance"] is False assert bundle["backend_conformance"]["cases"] assert bundle["source_diagnostics"] -assert bundle["capability_evidence"] == {} assert bundle["capability_gaps"] assert bundle["declared_weaknesses"] # Serializable to the portable JSON the evidence bundle claims to be. json.dumps(bundle, sort_keys=True) """ +EVIDENCE_REJECTION_PROBE = r""" +import subprocess +import sys + +completed = subprocess.run(sys.argv[1:], capture_output=True, check=False, text=True) +expected = ( + "researcher.validation.evidence-unverifiable: " + "task evidence requirements cannot be verified by the backend manifest\n" +) +assert completed.returncode == 3, completed +assert completed.stdout == "", completed +assert completed.stderr == expected, completed +""" + nox.options.default_venv_backend = "none" nox.options.reuse_existing_virtualenvs = True nox.options.sessions = ["verify"] @@ -352,6 +368,20 @@ def _uv_run_root(session: nox.Session, *args: str) -> None: _run(session, "uv", "run", "--frozen", "--project", str(REPO_ROOT), *args) +def _expect_evidence_rejection(session: nox.Session, python: Path, *command: str) -> None: + """Require the clean-installed CLI's exact fail-closed evidence diagnostic.""" + + _run( + session, + str(python), + "-I", + "-c", + EVIDENCE_REJECTION_PROBE, + *command, + env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, + ) + + def _tracked(session: nox.Session) -> list[str]: out = session.run( "git", @@ -576,8 +606,9 @@ def _distributions(session: nox.Session) -> None: "cyberbattlesim-chain", env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) - _run( + _expect_evidence_rejection( session, + conformance_venv / "bin" / "python", str(conformance_venv / "bin" / "raes-adapters"), "validate", "--mode", @@ -610,7 +641,6 @@ def _distributions(session: nox.Session) -> None: "20260729", "--run-id", "distribution-validation", - env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) _run( session, @@ -666,8 +696,9 @@ def _distributions(session: nox.Session) -> None: "nasim-tiny", env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) - _run( + _expect_evidence_rejection( session, + nasim_venv / "bin" / "python", str(nasim_venv / "bin" / "raes-adapters"), "validate", "--backend", @@ -700,7 +731,6 @@ def _distributions(session: nox.Session) -> None: "20260802", "--run-id", "distribution-validation", - env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) _run( session, @@ -768,8 +798,9 @@ def _distributions(session: nox.Session) -> None: "cyborg-cage2", env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) - _run( + _expect_evidence_rejection( session, + cyborg_venv / "bin" / "python", str(cyborg_venv / "bin" / "raes-adapters"), "validate", "--mode", @@ -804,7 +835,6 @@ def _distributions(session: nox.Session) -> None: "11", "--run-id", "distribution-validation", - env={"PYTHONPATH": "", "PYTHONSAFEPATH": "1"}, ) _run( session, diff --git a/src/raes_adapters/_conformance_support.py b/src/raes_adapters/_conformance_support.py index 22a630f..e9c4993 100644 --- a/src/raes_adapters/_conformance_support.py +++ b/src/raes_adapters/_conformance_support.py @@ -1,10 +1,8 @@ -"""Backend-neutral manifest capability-evidence composition. +"""Backend-neutral inventory of unresolved manifest capability claims. -Backends attest their affirmative manifest capability surfaces the same way: -enumerate the affirmative capability pointers in the manifest payload, and admit -each one that a passed executable probe backs. The traversal and the -passed-probe accounting are mechanical and RAES-payload-shaped; the probe -requirement map and the evidence-ref identities stay backend-local. +Traversal is mechanical and RAES-payload-shaped. It deliberately produces no +positive evidence join: a broad conformance result, source-ledger disposition, +or adapter probe cannot certify every affirmative manifest leaf. """ from __future__ import annotations @@ -12,11 +10,6 @@ from collections.abc import Iterable, Mapping from typing import cast -from raes_contracts.diagnostics import ( # type: ignore[import-untyped] - Diagnostic, - diagnostic_model, -) - from raes_adapters._diagnostics import escape_pointer_token _NON_CAPABILITY_KEYS = frozenset({"constraints", "name"}) @@ -46,73 +39,10 @@ def affirmative_capability_pointers(payload: Mapping[str, object]) -> tuple[str, return tuple(pointers) -def manifest_capability_evidence( - payload: Mapping[str, object], - *, - probe_requirements: Mapping[str, tuple[str, ...]], - passed_evidence_refs: Iterable[str], -) -> dict[str, tuple[str, ...]]: - """Return evidence references for declared affirmative manifest surfaces.""" - - passed = set(passed_evidence_refs) - evidence: dict[str, tuple[str, ...]] = {} - for pointer in affirmative_capability_pointers(payload): - requirements = probe_requirements.get(pointer) - if requirements is not None and set(requirements) <= passed: - evidence[pointer] = requirements - return evidence - - -def manifest_capability_evidence_gaps( - payload: Mapping[str, object], - *, - probe_requirements: Mapping[str, tuple[str, ...]], - passed_evidence_refs: Iterable[str], -) -> tuple[str, ...]: - """Return declared affirmative capability surfaces with no probe evidence.""" - - evidence = manifest_capability_evidence( - payload, - probe_requirements=probe_requirements, - passed_evidence_refs=passed_evidence_refs, - ) - return tuple( - pointer for pointer in affirmative_capability_pointers(payload) if pointer not in evidence - ) - - -def passed_probe_evidence_refs( - conformance_report: object | None, - source_diagnostics: Iterable[Diagnostic], - *, - backend_conformance_evidence: str, - source_protocol_evidence: str, - source_validation_failed_code: str, -) -> tuple[str, ...]: - """Return evidence refs only for executable probes that passed.""" - - refs: list[str] = [] - if _conformance_passed(conformance_report): - refs.append(backend_conformance_evidence) - source_models = [diagnostic_model(diagnostic) for diagnostic in source_diagnostics] - if source_models and all( - model.code != source_validation_failed_code for model in source_models - ): - refs.append(source_protocol_evidence) - return tuple(refs) - - -def _conformance_passed(conformance_report: object | None) -> bool: - """Return whether a conformance report passed with no gaps or failed cases.""" - - if conformance_report is None: - return False - return bool( - getattr(conformance_report, "passed", False) - and not getattr(conformance_report, "unsupported_contract_gaps", ()) - and not getattr(conformance_report, "unsupported_capability_gaps", ()) - and all(case.passed for case in getattr(conformance_report, "cases", ())) - ) +def manifest_capability_gaps(payload: Mapping[str, object]) -> tuple[str, ...]: + """Return every affirmative manifest leaf as unresolved inventory.""" + + return affirmative_capability_pointers(payload) def _iter_affirmative_capability_pointers( @@ -161,7 +91,5 @@ def _is_affirmative_capability_value(value: object) -> bool: __all__ = [ "affirmative_capability_pointers", - "manifest_capability_evidence", - "manifest_capability_evidence_gaps", - "passed_probe_evidence_refs", + "manifest_capability_gaps", ] diff --git a/src/raes_adapters/_gym_backend/conformance.py b/src/raes_adapters/_gym_backend/conformance.py index 210dfca..1cc720c 100644 --- a/src/raes_adapters/_gym_backend/conformance.py +++ b/src/raes_adapters/_gym_backend/conformance.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Callable, Iterable, Mapping +from collections.abc import Callable, Mapping from dataclasses import dataclass from typing import cast @@ -25,78 +25,18 @@ from raes_runtime.registry import RuntimeTarget # type: ignore[import-untyped] from raes_adapters._conformance_support import ( - manifest_capability_evidence, - manifest_capability_evidence_gaps, - passed_probe_evidence_refs, + manifest_capability_gaps, ) from raes_adapters._scenario_ledger import EvidenceSelection, LedgerProblem from raes_adapters.base import run_conformance_probe -# Capability surfaces every gym backend declares affirmatively. Backend -# conformance alone backs the control-plane surfaces; the provisioner and -# participant surfaces additionally require validated source-protocol evidence. -_BACKEND_ONLY_POINTERS = ( - "cleanup/supported_action_kinds", - "cleanup/supported_contract_versions", - "cleanup/supported_verification_methods", - "cleanup/supports_residual_state_disclosure", - "cleanup/supports_reusable_state", - "evaluator/preserves_binding_provenance", - "evaluator/supported_evidence_channels", - "evaluator/supported_predicate_families", - "evaluator/supported_quantifiers", - "evaluator/supported_sections", - "evaluator/supported_time_domains", - "evaluator/supported_truth_outcomes", - "evaluator/supports_objectives", - "evaluator/supports_scoring", - "orchestrator/supported_sections", - "orchestrator/supported_workflow_features", - "orchestrator/supports_workflows", -) -_PROVISIONER_DUAL_POINTERS = ( - "provisioner/max_total_nodes", - "provisioner/supported_account_features", - "provisioner/supported_node_types", - "provisioner/supported_os_families", - "provisioner/supports_accounts", -) - - -def standard_probe_requirements( - backend_evidence: str, - source_evidence: str, - *, - participant_dual: Iterable[str], -) -> dict[str, tuple[str, ...]]: - """Build the standard capability probe-requirement map for a gym backend. - - ``participant_dual`` names the participant-runtime capability sub-fields the - backend declares (each requires both backend and source evidence). - """ - - requirements: dict[str, tuple[str, ...]] = { - f"/capabilities/{pointer}": (backend_evidence,) for pointer in _BACKEND_ONLY_POINTERS - } - for pointer in _PROVISIONER_DUAL_POINTERS: - requirements[f"/capabilities/{pointer}"] = (backend_evidence, source_evidence) - for field in participant_dual: - requirements[f"/capabilities/participant_runtime/{field}"] = ( - backend_evidence, - source_evidence, - ) - return requirements - @dataclass(frozen=True) class GymConformanceConfig(object): """Backend-local identities and bindings for conformance composition.""" name: str - backend_evidence: str - source_evidence: str source_validation_failed: str - probe_requirements: Mapping[str, tuple[str, ...]] default_selection: EvidenceSelection create_target: Callable[..., RuntimeTarget] create_manifest: Callable[[], BackendManifest] @@ -121,36 +61,14 @@ def conformance_payload(report: BackendConformanceReport) -> dict[str, object]: return cast(dict[str, object], backend_conformance_report_payload(report)) -def capability_evidence( - config: GymConformanceConfig, - manifest: BackendManifest | None, - payload: Mapping[str, object] | None, - conformance_report: BackendConformanceReport | None, - source_diagnostics: Iterable[Diagnostic], -) -> dict[str, tuple[str, ...]]: - """Return evidence references for declared affirmative manifest surfaces.""" - - return manifest_capability_evidence( - _manifest_payload(config, manifest, payload), - probe_requirements=config.probe_requirements, - passed_evidence_refs=_passed_refs(config, conformance_report, source_diagnostics), - ) - - -def capability_evidence_gaps( +def capability_gaps( config: GymConformanceConfig, manifest: BackendManifest | None, payload: Mapping[str, object] | None, - conformance_report: BackendConformanceReport | None, - source_diagnostics: Iterable[Diagnostic], ) -> tuple[str, ...]: - """Return declared affirmative capability surfaces with no probe evidence.""" + """Return every affirmative manifest leaf as unresolved inventory.""" - return manifest_capability_evidence_gaps( - _manifest_payload(config, manifest, payload), - probe_requirements=config.probe_requirements, - passed_evidence_refs=_passed_refs(config, conformance_report, source_diagnostics), - ) + return manifest_capability_gaps(_manifest_payload(config, manifest, payload)) def declared_weaknesses( @@ -232,29 +150,11 @@ def _manifest_payload( ) -def _passed_refs( - config: GymConformanceConfig, - conformance_report: BackendConformanceReport | None, - source_diagnostics: Iterable[Diagnostic], -) -> tuple[str, ...]: - """Resolve passed-probe evidence refs under the backend's identities.""" - - return passed_probe_evidence_refs( - conformance_report, - source_diagnostics, - backend_conformance_evidence=config.backend_evidence, - source_protocol_evidence=config.source_evidence, - source_validation_failed_code=config.source_validation_failed, - ) - - __all__ = [ "GymConformanceConfig", - "capability_evidence", - "capability_evidence_gaps", + "capability_gaps", "conformance_payload", "declared_weaknesses", "run_conformance", "source_protocol_diagnostics", - "standard_probe_requirements", ] diff --git a/src/raes_adapters/_manifest_support.py b/src/raes_adapters/_manifest_support.py index 64d207c..f9787ff 100644 --- a/src/raes_adapters/_manifest_support.py +++ b/src/raes_adapters/_manifest_support.py @@ -24,7 +24,8 @@ CleanupCapabilities, EvaluatorCapabilities, OrchestratorCapabilities, - WorkflowFeature, + ParticipantRuntimeCapabilities, + ProvisionerCapabilities, ) from raes_contracts.apparatus import ( # type: ignore[import-untyped] ConceptBinding, @@ -139,57 +140,6 @@ def concept_bindings() -> tuple[ConceptBinding, ...]: ) -def standard_orchestrator_capabilities( - name: str, - constraints: Mapping[str, str], -) -> OrchestratorCapabilities: - """Declare the portable episode-orchestration support every gym backend shares.""" - - return OrchestratorCapabilities( - name=f"{name}-orchestrator", - supported_sections=frozenset({"events", "workflows"}), - supports_workflows=True, - supports_assertion_refs=False, - supports_inject_bindings=False, - supported_workflow_features=frozenset({WorkflowFeature.CALL}), - constraints=dict(constraints), - ) - - -def standard_evaluator_capabilities( - name: str, - constraints: Mapping[str, str], -) -> EvaluatorCapabilities: - """Declare the evaluator vocabulary every gym backend shares.""" - - return EvaluatorCapabilities( - name=f"{name}-evaluator", - supported_sections=frozenset({"conditions", "propositions", "assertions", "objectives"}), - supports_scoring=True, - supports_objectives=True, - supported_predicate_families=frozenset({"presence", "boolean", "string", "number"}), - supported_quantifiers=frozenset({"all", "any", "at_least"}), - supported_truth_outcomes=frozenset({"true", "false", "unknown", "unsupported"}), - supported_evidence_channels=frozenset({"api_response"}), - supported_time_domains=frozenset({"wall_clock"}), - preserves_binding_provenance=True, - constraints=dict(constraints), - ) - - -def standard_cleanup_capabilities(name: str) -> CleanupCapabilities: - """Declare the verified in-process cleanup support every gym backend shares.""" - - return CleanupCapabilities( - name=f"{name}-cleanup", - supported_contract_versions=CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, - supported_action_kinds=frozenset({"destroy", "reset", "verify"}), - supported_verification_methods=frozenset({"probe", "receipt"}), - supports_reusable_state=True, - supports_residual_state_disclosure=True, - ) - - def read_source_revision(load_qualification: Callable[[], Mapping[str, object]]) -> str: """Read the selected source revision from a backend qualification record.""" @@ -201,6 +151,50 @@ def read_source_revision(load_qualification: Callable[[], Mapping[str, object]]) return revision +def declared_cleanup_capabilities( + *, + name: str, + supported_contract_versions: frozenset[str], + supported_action_kinds: frozenset[str], + supported_verification_methods: frozenset[str], + supports_reusable_state: bool, + supports_residual_state_disclosure: bool, +) -> CleanupCapabilities: + """Build exactly the cleanup declaration supplied by one backend. + + Every value remains backend-local and explicit; this helper only projects + those values into the published RAES model. + """ + + return CleanupCapabilities( + name=name, + supported_contract_versions=supported_contract_versions, + supported_action_kinds=supported_action_kinds, + supported_verification_methods=supported_verification_methods, + supports_reusable_state=supports_reusable_state, + supports_residual_state_disclosure=supports_residual_state_disclosure, + ) + + +def compose_capability_set( + *, + provisioner: ProvisionerCapabilities, + orchestrator: OrchestratorCapabilities, + evaluator: EvaluatorCapabilities, + participant_runtime: ParticipantRuntimeCapabilities, + cleanup: CleanupCapabilities, +) -> BackendCapabilitySet: + """Compose backend-authored RAES capability declarations without defaults.""" + + return BackendCapabilitySet( + provisioner=provisioner, + orchestrator=orchestrator, + evaluator=evaluator, + participant_runtime=participant_runtime, + cleanup=cleanup, + ) + + def assemble_manifest( name: str, capabilities: BackendCapabilitySet, @@ -224,11 +218,10 @@ def assemble_manifest( __all__ = [ "adapter_version", "assemble_manifest", + "compose_capability_set", "concept_bindings", + "declared_cleanup_capabilities", "model_contract_id", "read_source_revision", - "standard_cleanup_capabilities", - "standard_evaluator_capabilities", - "standard_orchestrator_capabilities", "supported_contracts", ] diff --git a/src/raes_adapters/cli.py b/src/raes_adapters/cli.py index e0f1a1d..9f39ec6 100644 --- a/src/raes_adapters/cli.py +++ b/src/raes_adapters/cli.py @@ -163,6 +163,10 @@ def _single_participant_experiment_bindings_match( _RUNTIME_FAILURE_CODE = "researcher.runtime.failure" _CONTROLS_INVALID_CODE = "researcher.validation.controls-invalid" _CONTROLS_INVALID_MESSAGE = "run controls were not admitted" +_EVIDENCE_UNVERIFIABLE_CODE = "researcher.validation.evidence-unverifiable" +_EVIDENCE_UNVERIFIABLE_MESSAGE = ( + "task evidence requirements cannot be verified by the backend manifest" +) class _UsageFailure(Exception): @@ -177,6 +181,10 @@ class _ValidationFailure(Exception): """Internal pre-execution validation control flow.""" +class _EvidenceUnverifiableFailure(_ValidationFailure): + """Internal task-evidence admission control flow.""" + + @dataclass(frozen=True) class _CommandFailure(Exception): """Stable command failure safe to project to standard error.""" @@ -267,7 +275,6 @@ class _BackendAdapter(object): [str, ParticipantImplementationSelectionModel], ParticipantImplementationProvenanceModel ] evidence_source_label: str - evidence_satisfies_refs: tuple[str, ...] provenance_payload: Callable[[argparse.Namespace, _AdmittedRun], dict[str, object]] machine_software: Callable[[], dict[str, str]] @@ -954,7 +961,6 @@ def _cyberbattlesim_machine_software() -> dict[str, str]: build_controls=_cyborg_build_controls, episode_provenance=cyborg_researcher.blue_implementation_provenance, evidence_source_label="cyborg-cage2 evaluator projection", - evidence_satisfies_refs=("source-ledger:reward-components",), provenance_payload=_cyborg_provenance_payload, machine_software=_cyborg_machine_software, ), @@ -982,7 +988,6 @@ def _cyberbattlesim_machine_software() -> dict[str, str]: build_controls=_nasim_build_controls, episode_provenance=nasim_researcher.red_implementation_provenance, evidence_source_label="nasim-tiny evaluator projection", - evidence_satisfies_refs=("attacker-action-log", "host-compromise-series"), provenance_payload=_nasim_provenance_payload, machine_software=_nasim_machine_software, ), @@ -1010,7 +1015,6 @@ def _cyberbattlesim_machine_software() -> dict[str, str]: build_controls=_cyberbattlesim_build_controls, episode_provenance=cyberbattlesim_researcher.red_implementation_provenance, evidence_source_label="cyberbattlesim-chain evaluator projection", - evidence_satisfies_refs=("attacker-action-log", "availability-series"), provenance_payload=_cyberbattlesim_provenance_payload, machine_software=_cyberbattlesim_machine_software, ), @@ -1191,6 +1195,37 @@ def _validate_runtime_plan( raise _ValidationFailure +def _task_capture_admission_gaps( + task: ExperimentTaskModel, + manifest: BackendManifest, +) -> tuple[str, ...]: + """Return semantic task evidence refs the published manifest cannot verify. + + RAES 3.3's observation capability describes capture kinds, channels, + contracts, media, sealing, redaction, loss disclosure, and custody. It does + not bind a semantic task reference to an emitted artifact and its required + fields or negative data-quality states. Consequently, even a non-null + observation capability cannot admit these semantic references yet. + """ + + required = { + requirement.ref_id for requirement in task.evaluation_protocol.observation_requirements + } + required.update( + requirement.ref_id + for metric in task.evaluation_protocol.metric_definitions.values() + for requirement in metric.evidence_requirements + ) + if not required: + return () + # Validate/project the live manifest through its published owner. The + # resulting RAES 3.3 payload has no semantic-ref/field witness to inspect, + # so even observation-capability presence cannot promote the requirement. A + # later RAES contract owns that seam. + backend_manifest_payload(manifest) + return tuple(sorted(required)) + + def _admit_native_run(adapter: _BackendAdapter, args: argparse.Namespace) -> _AdmittedRun: """Admit all authoring, participant, and runtime controls before execution.""" @@ -1212,6 +1247,8 @@ def _admit_native_run(adapter: _BackendAdapter, args: argparse.Namespace) -> _Ad args, participant_manifest, participant_selection, participant_configuration ): raise _ValidationFailure + if _task_capture_admission_gaps(task, adapter.backend_manifest()): + raise _EvidenceUnverifiableFailure if not 1 <= len(args.run_id) <= 64 or not args.run_id.replace("-", "").isalnum(): raise _ValidationFailure seeds = adapter.admitted_seeds(args, spec) @@ -1313,17 +1350,29 @@ def _retain_failure(output: Path, code: str, message: str) -> None: atomic_write_json_artifact(output / "failure.json", diagnostic.model_dump(mode="json")) +def _admit_native_run_or_command_failure( + adapter: _BackendAdapter, args: argparse.Namespace +) -> _AdmittedRun: + """Project internal admission failures to their stable CLI diagnostics.""" + + try: + return _admit_native_run(adapter, args) + except _EvidenceUnverifiableFailure: + raise _CommandFailure( + EXIT_VALIDATION, _EVIDENCE_UNVERIFIABLE_CODE, _EVIDENCE_UNVERIFIABLE_MESSAGE + ) from None + except _ValidationFailure: + raise _CommandFailure( + EXIT_VALIDATION, _CONTROLS_INVALID_CODE, _CONTROLS_INVALID_MESSAGE + ) from None + + def _run_conformance(adapter: _BackendAdapter, args: argparse.Namespace) -> int: """Run the selected backend conformance suite and seal its evidence.""" admitted: _AdmittedRun | None = None if adapter.expected_pack_digest is not None: - try: - admitted = _admit_native_run(adapter, args) - except _ValidationFailure: - raise _CommandFailure( - EXIT_VALIDATION, _CONTROLS_INVALID_CODE, _CONTROLS_INVALID_MESSAGE - ) from None + admitted = _admit_native_run_or_command_failure(adapter, args) if util.find_spec(adapter.native_module) is None: raise _CommandFailure( EXIT_RUNTIME, @@ -1400,12 +1449,7 @@ def _native_environment( ) -> tuple[_AdmittedRun, Path]: """Admit native inputs, selected source, and the exclusive output root.""" - try: - admitted = _admit_native_run(adapter, args) - except _ValidationFailure: - raise _CommandFailure( - EXIT_VALIDATION, _CONTROLS_INVALID_CODE, _CONTROLS_INVALID_MESSAGE - ) from None + admitted = _admit_native_run_or_command_failure(adapter, args) if util.find_spec(adapter.native_module) is None: raise _CommandFailure( EXIT_RUNTIME, @@ -1485,9 +1529,6 @@ def _write_episode_evidence( size_bytes=len(evidence_bytes), created_at=result.evidence_records[0].captured_at, source=adapter.evidence_source_label, - satisfies_refs=[ - {"ref_kind": "evidence", "ref_id": ref} for ref in adapter.evidence_satisfies_refs - ], sensitivity="redacted", ) @@ -1633,12 +1674,7 @@ def _run_native(adapter: _BackendAdapter, args: argparse.Namespace) -> int: def _validated_admission(adapter: _BackendAdapter, args: argparse.Namespace) -> int: """Validate native controls and report the admitted run scope.""" - try: - admitted = _admit_native_run(adapter, args) - except _ValidationFailure: - raise _CommandFailure( - EXIT_VALIDATION, _CONTROLS_INVALID_CODE, _CONTROLS_INVALID_MESSAGE - ) from None + admitted = _admit_native_run_or_command_failure(adapter, args) print( json.dumps( { diff --git a/src/raes_adapters/cyberbattlesim/backend/__init__.py b/src/raes_adapters/cyberbattlesim/backend/__init__.py index cba457e..3a92f86 100644 --- a/src/raes_adapters/cyberbattlesim/backend/__init__.py +++ b/src/raes_adapters/cyberbattlesim/backend/__init__.py @@ -6,8 +6,7 @@ from .conformance import ( cyberbattlesim_backend_conformance_payload, cyberbattlesim_declared_weaknesses, - cyberbattlesim_manifest_capability_evidence, - cyberbattlesim_manifest_capability_evidence_gaps, + cyberbattlesim_manifest_capability_gaps, cyberbattlesim_source_protocol_diagnostics, run_cyberbattlesim_conformance, ) @@ -53,8 +52,7 @@ "EVALUATION_EVIDENCE_REF", "cyberbattlesim_backend_conformance_payload", "cyberbattlesim_declared_weaknesses", - "cyberbattlesim_manifest_capability_evidence", - "cyberbattlesim_manifest_capability_evidence_gaps", + "cyberbattlesim_manifest_capability_gaps", "cyberbattlesim_source_protocol_diagnostics", "create_cyberbattlesim_components", "create_cyberbattlesim_manifest", diff --git a/src/raes_adapters/cyberbattlesim/backend/conformance.py b/src/raes_adapters/cyberbattlesim/backend/conformance.py index edfc0f2..c4d403c 100644 --- a/src/raes_adapters/cyberbattlesim/backend/conformance.py +++ b/src/raes_adapters/cyberbattlesim/backend/conformance.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Iterable, Mapping +from collections.abc import Mapping from raes_backend_protocols.backend_manifest import ( # type: ignore[import-untyped] BackendManifest, @@ -18,10 +18,7 @@ from raes_contracts.diagnostics import Diagnostic # type: ignore[import-untyped] from raes_adapters._gym_backend import conformance as gym -from raes_adapters._gym_backend.conformance import ( - GymConformanceConfig, - standard_probe_requirements, -) +from raes_adapters._gym_backend.conformance import GymConformanceConfig from raes_adapters.base import run_conformance_probe from raes_adapters.cyberbattlesim import load_qualification from raes_adapters.cyberbattlesim.scenario_ledger import ( @@ -35,42 +32,10 @@ from .manifest import create_cyberbattlesim_manifest from .target import create_cyberbattlesim_target -_BACKEND_EVIDENCE = "evidence.cyberbattlesim.backend-conformance" -_SOURCE_EVIDENCE = "evidence.cyberbattlesim.source-protocol.validated" _SOURCE_FAILED = "cyberbattlesim.source-protocol.validation-failed" _CONFIG = GymConformanceConfig( name="cyberbattlesim", - backend_evidence=_BACKEND_EVIDENCE, - source_evidence=_SOURCE_EVIDENCE, source_validation_failed=_SOURCE_FAILED, - probe_requirements=standard_probe_requirements( - _BACKEND_EVIDENCE, - _SOURCE_EVIDENCE, - participant_dual=( - "execution_bindings", - "feature_support", - "max_autonomous_action_attempts", - "max_autonomous_burst_size", - "max_autonomous_in_flight", - "max_autonomous_occurrences", - "max_autonomous_participants", - "max_autonomous_retries_per_occurrence", - "max_concurrent_actions", - "max_execution_services", - "supported_autonomous_action_contracts", - "supported_autonomous_observation_boundaries", - "supported_autonomous_policy_profiles", - "supported_autonomous_selection_strategies", - "supported_autonomous_target_addresses", - "supported_behavior_features", - "supported_execution_control_actions", - "supported_interaction_features", - "supported_participant_roles", - "supports_autonomous_execution", - "supports_bounded_concurrency", - "supports_execution_control", - ), - ), default_selection=CYBERBATTLE_CHAIN, create_target=create_cyberbattlesim_target, create_manifest=create_cyberbattlesim_manifest, @@ -112,32 +77,14 @@ def cyberbattlesim_backend_conformance_payload( return gym.conformance_payload(report) -def cyberbattlesim_manifest_capability_evidence( - manifest: BackendManifest | None = None, - *, - payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), -) -> dict[str, tuple[str, ...]]: - """Return evidence references for declared affirmative manifest surfaces.""" - - return gym.capability_evidence( - _CONFIG, manifest, payload, conformance_report, source_diagnostics - ) - - -def cyberbattlesim_manifest_capability_evidence_gaps( +def cyberbattlesim_manifest_capability_gaps( manifest: BackendManifest | None = None, *, payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), ) -> tuple[str, ...]: - """Return declared affirmative capability surfaces with no probe evidence.""" + """Return every affirmative manifest leaf as unresolved inventory.""" - return gym.capability_evidence_gaps( - _CONFIG, manifest, payload, conformance_report, source_diagnostics - ) + return gym.capability_gaps(_CONFIG, manifest, payload) def cyberbattlesim_declared_weaknesses( @@ -159,8 +106,7 @@ def cyberbattlesim_source_protocol_diagnostics( __all__ = [ "cyberbattlesim_backend_conformance_payload", "cyberbattlesim_declared_weaknesses", - "cyberbattlesim_manifest_capability_evidence", - "cyberbattlesim_manifest_capability_evidence_gaps", + "cyberbattlesim_manifest_capability_gaps", "cyberbattlesim_source_protocol_diagnostics", "run_cyberbattlesim_conformance", ] diff --git a/src/raes_adapters/cyberbattlesim/backend/manifest.py b/src/raes_adapters/cyberbattlesim/backend/manifest.py index 2e5b0b7..51da96b 100644 --- a/src/raes_adapters/cyberbattlesim/backend/manifest.py +++ b/src/raes_adapters/cyberbattlesim/backend/manifest.py @@ -6,11 +6,15 @@ BackendManifest, ) from raes_backend_protocols.capabilities import ( # type: ignore[import-untyped] - BackendCapabilitySet, + CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + CleanupCapabilities, + EvaluatorCapabilities, + OrchestratorCapabilities, ParticipantExecutionBinding, ParticipantFeatureSupport, ParticipantRuntimeCapabilities, ProvisionerCapabilities, + WorkflowFeature, ) from raes_contracts.apparatus import ( # type: ignore[import-untyped] RealizationSupportDeclaration, @@ -22,10 +26,9 @@ from raes_adapters._manifest_support import ( assemble_manifest, + compose_capability_set, + declared_cleanup_capabilities, read_source_revision, - standard_cleanup_capabilities, - standard_evaluator_capabilities, - standard_orchestrator_capabilities, ) from raes_adapters.cyberbattlesim import load_qualification @@ -165,33 +168,58 @@ def _participant_capabilities() -> ParticipantRuntimeCapabilities: ) -def _capabilities() -> BackendCapabilitySet: - """Compose the complete selected backend capability declaration.""" +def _orchestrator_capabilities() -> OrchestratorCapabilities: + """Declare the CyberBattleSim orchestration surface.""" - return BackendCapabilitySet( - provisioner=_provisioner_capabilities(), - orchestrator=standard_orchestrator_capabilities( - "cyberbattlesim", - { - "native_transition_owner": "participant-runtime", - "workflow_scope": "episode lifecycle and serialized participant steps", - }, - ), - evaluator=standard_evaluator_capabilities( - "cyberbattlesim", - { - "reward_owner": "evaluator", - "outcome_reproduction": "stochastic-bounded", - "proposition_projection": ( - "binding-preserving unknown under lossy source evidence" - ), - "objective_terminal_state": ( - "running until a distinct mapped terminal cause is available" - ), - }, - ), - participant_runtime=_participant_capabilities(), - cleanup=standard_cleanup_capabilities("cyberbattlesim"), + return OrchestratorCapabilities( + name="cyberbattlesim-orchestrator", + supported_sections=frozenset({"events", "workflows"}), + supports_workflows=True, + supports_assertion_refs=False, + supports_inject_bindings=False, + supported_workflow_features=frozenset({WorkflowFeature.CALL}), + constraints={ + "native_transition_owner": "participant-runtime", + "workflow_scope": "episode lifecycle and serialized participant steps", + }, + ) + + +def _evaluator_capabilities() -> EvaluatorCapabilities: + """Declare the CyberBattleSim evaluator surface.""" + + return EvaluatorCapabilities( + name="cyberbattlesim-evaluator", + supported_sections=frozenset({"conditions", "propositions", "assertions", "objectives"}), + supports_scoring=True, + supports_objectives=True, + supported_predicate_families=frozenset({"presence", "boolean", "string", "number"}), + supported_quantifiers=frozenset({"all", "any", "at_least"}), + supported_truth_outcomes=frozenset({"true", "false", "unknown", "unsupported"}), + supported_evidence_channels=frozenset({"api_response"}), + supported_time_domains=frozenset({"wall_clock"}), + preserves_binding_provenance=True, + constraints={ + "reward_owner": "evaluator", + "outcome_reproduction": "stochastic-bounded", + "proposition_projection": ("binding-preserving unknown under lossy source evidence"), + "objective_terminal_state": ( + "running until a distinct mapped terminal cause is available" + ), + }, + ) + + +def _cleanup_capabilities() -> CleanupCapabilities: + """Declare the CyberBattleSim cleanup surface.""" + + return declared_cleanup_capabilities( + name="cyberbattlesim-cleanup", + supported_contract_versions=CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + supported_action_kinds=frozenset({"destroy", "reset", "verify"}), + supported_verification_methods=frozenset({"probe", "receipt"}), + supports_reusable_state=True, + supports_residual_state_disclosure=True, ) @@ -233,7 +261,13 @@ def create_cyberbattlesim_manifest() -> BackendManifest: source_revision = read_source_revision(load_qualification) return assemble_manifest( CYBERBATTLESIM_BACKEND_NAME, - _capabilities(), + compose_capability_set( + provisioner=_provisioner_capabilities(), + orchestrator=_orchestrator_capabilities(), + evaluator=_evaluator_capabilities(), + participant_runtime=_participant_capabilities(), + cleanup=_cleanup_capabilities(), + ), _realization_support(source_revision), { "source_identity": "qualified-complete-runtime-artifact-roots-attested", diff --git a/src/raes_adapters/cyberbattlesim/researcher.py b/src/raes_adapters/cyberbattlesim/researcher.py index 7b8bf47..03e1e15 100644 --- a/src/raes_adapters/cyberbattlesim/researcher.py +++ b/src/raes_adapters/cyberbattlesim/researcher.py @@ -70,7 +70,7 @@ from raes_adapters.cyberbattlesim.backend.conformance import ( cyberbattlesim_backend_conformance_payload, cyberbattlesim_declared_weaknesses, - cyberbattlesim_manifest_capability_evidence_gaps, + cyberbattlesim_manifest_capability_gaps, cyberbattlesim_source_protocol_diagnostics, run_cyberbattlesim_conformance, ) @@ -535,10 +535,7 @@ def cyberbattlesim_conformance_suite( or any(not case.passed for case in report.cases) ): raise RuntimeError("CyberBattleSim published conformance cases failed.") - if cyberbattlesim_manifest_capability_evidence_gaps( - conformance_report=report, source_diagnostics=diagnostics - ): - raise RuntimeError("CyberBattleSim manifest capability evidence is incomplete.") + capability_gaps = cyberbattlesim_manifest_capability_gaps() report_path = write_backend_conformance_report( payload, output_dir=output_dir, @@ -550,6 +547,7 @@ def cyberbattlesim_conformance_suite( "execution_basis": "installed-source-probe", "native_conformance": report.native_conformance, "report_path": report_path.relative_to(output_dir).as_posix(), + "capability_gaps": list(capability_gaps), } ) index: dict[str, object] = { diff --git a/src/raes_adapters/cyborg/__init__.py b/src/raes_adapters/cyborg/__init__.py index a42dcaa..a80792e 100644 --- a/src/raes_adapters/cyborg/__init__.py +++ b/src/raes_adapters/cyborg/__init__.py @@ -25,8 +25,7 @@ cyborg_backend_conformance_payload, cyborg_conformance_reproduction_commands, cyborg_declared_weaknesses, - cyborg_manifest_capability_evidence, - cyborg_manifest_capability_evidence_gaps, + cyborg_manifest_capability_gaps, cyborg_source_diagnostics, run_cyborg_conformance, run_cyborg_conformance_suite, @@ -68,8 +67,7 @@ "cyborg_backend_conformance_payload", "cyborg_conformance_reproduction_commands", "cyborg_declared_weaknesses", - "cyborg_manifest_capability_evidence", - "cyborg_manifest_capability_evidence_gaps", + "cyborg_manifest_capability_gaps", "cyborg_source_diagnostics", "run_cyborg_conformance", "run_cyborg_conformance_suite", @@ -108,8 +106,7 @@ def __getattr__(name: str) -> object: "cyborg_backend_conformance_payload", "cyborg_conformance_reproduction_commands", "cyborg_declared_weaknesses", - "cyborg_manifest_capability_evidence", - "cyborg_manifest_capability_evidence_gaps", + "cyborg_manifest_capability_gaps", "cyborg_source_diagnostics", "load_qualification", "read_compatibility_patch", diff --git a/src/raes_adapters/cyborg/conformance.py b/src/raes_adapters/cyborg/conformance.py index c2a698a..a768a18 100644 --- a/src/raes_adapters/cyborg/conformance.py +++ b/src/raes_adapters/cyborg/conformance.py @@ -5,7 +5,7 @@ import argparse import json import textwrap -from collections.abc import Iterable, Mapping, Sequence +from collections.abc import Mapping, Sequence from contextlib import suppress from dataclasses import replace from importlib.resources import files @@ -63,6 +63,7 @@ sample_participant_action_admission_request, ) +from raes_adapters._conformance_support import manifest_capability_gaps from raes_adapters.base import run_conformance_probe from ._diagnostics import diagnostic_address @@ -102,53 +103,6 @@ _CLOCK = "time.clock.cage2" _WORKFLOW = "orchestration.workflow.cage2-probe" -_PUBLISHED_CONFORMANCE_EVIDENCE = "evidence.cyborg.published-conformance.disclosed" -_SOURCE_LEDGER_EVIDENCE = "evidence.cyborg.source-ledger.validated" -_ADAPTER_RUNTIME_EVIDENCE = "evidence.cyborg.adapter-runtime.validated" -_ALL_EVIDENCE = ( - _PUBLISHED_CONFORMANCE_EVIDENCE, - _SOURCE_LEDGER_EVIDENCE, - _ADAPTER_RUNTIME_EVIDENCE, -) -_CAPABILITY_PROBE_REQUIREMENTS: Mapping[str, tuple[str, ...]] = { - "/capabilities/evaluator/preserves_binding_provenance": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_evidence_channels": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_predicate_families": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_quantifiers": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_sections": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_time_domains": _ALL_EVIDENCE, - "/capabilities/evaluator/supported_truth_outcomes": _ALL_EVIDENCE, - "/capabilities/evaluator/supports_objectives": _ALL_EVIDENCE, - "/capabilities/evaluator/supports_scoring": _ALL_EVIDENCE, - "/capabilities/orchestrator/supported_sections": _ALL_EVIDENCE, - "/capabilities/orchestrator/supported_workflow_features": _ALL_EVIDENCE, - "/capabilities/orchestrator/supports_workflows": _ALL_EVIDENCE, - "/capabilities/participant_runtime/supported_behavior_features": _ALL_EVIDENCE, - "/capabilities/participant_runtime/supported_interaction_features": _ALL_EVIDENCE, - "/capabilities/participant_runtime/supported_participant_roles": _ALL_EVIDENCE, - "/capabilities/provisioner/supported_account_features": _ALL_EVIDENCE, - "/capabilities/provisioner/supported_node_types": _ALL_EVIDENCE, - "/capabilities/provisioner/supported_os_families": _ALL_EVIDENCE, - "/capabilities/provisioner/supports_accounts": _ALL_EVIDENCE, - "/capabilities/provisioner/supports_acls": _ALL_EVIDENCE, - "/capabilities/time/max_clocks": _ALL_EVIDENCE, - "/capabilities/time/max_time_domains": _ALL_EVIDENCE, - "/capabilities/time/supported_advancement_modes": _ALL_EVIDENCE, - "/capabilities/time/supported_authority_kinds": _ALL_EVIDENCE, - "/capabilities/time/supported_constraint_kinds": _ALL_EVIDENCE, - "/capabilities/time/supported_contract_versions": _ALL_EVIDENCE, - "/capabilities/time/supported_domain_kinds": _ALL_EVIDENCE, - "/capabilities/time/supported_mapping_kinds": _ALL_EVIDENCE, - "/capabilities/time/supported_replay_behaviors": _ALL_EVIDENCE, - "/capabilities/time/supported_reset_behaviors": _ALL_EVIDENCE, - "/capabilities/time/supported_synchronization_modes": _ALL_EVIDENCE, - "/capabilities/time/supports_append_only_history": _ALL_EVIDENCE, - "/capabilities/time/supports_exact_rational_mappings": _ALL_EVIDENCE, - "/capabilities/time/supports_pause": _ALL_EVIDENCE, - "/capabilities/time/supports_run_provenance": _ALL_EVIDENCE, -} -_NON_CAPABILITY_KEYS = frozenset({"constraints", "name"}) - class _HermeticProbeDriver(object): """Deterministic, dependency-free driver used only for PR conformance.""" @@ -674,54 +628,22 @@ def cyborg_declared_weaknesses( return tuple(sorted(refs)) -def cyborg_manifest_capability_evidence( +def cyborg_manifest_capability_gaps( manifest: BackendManifest | None = None, *, payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), - adapter_diagnostics: Iterable[Diagnostic] = (), -) -> dict[str, tuple[str, ...]]: - """Return evidence references for declared affirmative manifest surfaces.""" - - passed_refs = set( - _passed_probe_evidence_refs( - conformance_report, - source_diagnostics, - adapter_diagnostics, - ) - ) - evidence: dict[str, tuple[str, ...]] = {} - for pointer in _affirmative_capability_pointers(_manifest_payload(manifest, payload)): - required = _CAPABILITY_PROBE_REQUIREMENTS.get(pointer) - if required is not None and set(required) <= passed_refs: - evidence[pointer] = required - return evidence - - -def cyborg_manifest_capability_evidence_gaps( - manifest: BackendManifest | None = None, - *, - payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), - adapter_diagnostics: Iterable[Diagnostic] = (), ) -> tuple[str, ...]: - """Return affirmative capability pointers lacking passing evidence.""" - - manifest_payload = _manifest_payload(manifest, payload) - evidence = cyborg_manifest_capability_evidence( - manifest, - payload=payload, - conformance_report=conformance_report, - source_diagnostics=source_diagnostics, - adapter_diagnostics=adapter_diagnostics, - ) - return tuple( - pointer - for pointer in _affirmative_capability_pointers(manifest_payload) - if pointer not in evidence + """Return every affirmative manifest leaf as unresolved inventory.""" + + resolved = ( + payload + if payload is not None + else cast( + Mapping[str, object], + backend_manifest_payload(manifest or create_cyborg_manifest()), + ) ) + return manifest_capability_gaps(resolved) def cyborg_conformance_reproduction_commands( @@ -775,17 +697,7 @@ def run_cyborg_conformance_suite( payload = cyborg_backend_conformance_payload(report) if not _report_has_only_passing_or_published_unsupported_cases(report): raise RuntimeError("CybORG published conformance cases failed.") - capability_evidence = cyborg_manifest_capability_evidence( - conformance_report=report, - source_diagnostics=diagnostics, - adapter_diagnostics=adapter_diagnostics, - ) - if cyborg_manifest_capability_evidence_gaps( - conformance_report=report, - source_diagnostics=diagnostics, - adapter_diagnostics=adapter_diagnostics, - ): - raise RuntimeError("CybORG manifest capability evidence is incomplete.") + capability_gaps = cyborg_manifest_capability_gaps() report_path = write_backend_conformance_report( payload, output_dir=output_dir, @@ -797,7 +709,7 @@ def run_cyborg_conformance_suite( "execution_basis": ExecutionBasis.HERMETIC_LIVE.value, "native_conformance": report.native_conformance, "report_path": report_path.relative_to(output_dir).as_posix(), - "capability_evidence": capability_evidence, + "capability_gaps": list(capability_gaps), } ) index: dict[str, object] = { @@ -825,43 +737,6 @@ def run_cyborg_conformance_suite( return index -def _manifest_payload( - manifest: BackendManifest | None, - payload: Mapping[str, object] | None, -) -> Mapping[str, object]: - """Return the provided payload or project the live CybORG manifest.""" - - if payload is not None: - return payload - return cast( - Mapping[str, object], backend_manifest_payload(manifest or create_cyborg_manifest()) - ) - - -def _passed_probe_evidence_refs( - report: BackendConformanceReport | None, - source_diagnostics: Iterable[Diagnostic], - adapter_diagnostics: Iterable[Diagnostic], -) -> tuple[str, ...]: - """Return only evidence references whose owning probes passed.""" - - refs: list[str] = [] - if ( - report is not None - and not report.unsupported_contract_gaps - and not report.unsupported_capability_gaps - and _report_has_only_passing_or_published_unsupported_cases(report) - ): - refs.append(_PUBLISHED_CONFORMANCE_EVIDENCE) - models = [diagnostic_model(item) for item in source_diagnostics] - if models and all(not model.code.endswith("validation-failed") for model in models): - refs.append(_SOURCE_LEDGER_EVIDENCE) - adapter_models = [diagnostic_model(item) for item in adapter_diagnostics] - if adapter_models and all(model.code.endswith(".validated") for model in adapter_models): - refs.append(_ADAPTER_RUNTIME_EVIDENCE) - return tuple(refs) - - def _report_has_only_passing_or_published_unsupported_cases( report: BackendConformanceReport, ) -> bool: @@ -884,71 +759,6 @@ def _report_has_only_passing_or_published_unsupported_cases( ) -def _affirmative_capability_pointers(payload: Mapping[str, object]) -> tuple[str, ...]: - """Derive JSON Pointers for every affirmative manifest capability.""" - - capabilities = payload.get("capabilities") - if not isinstance(capabilities, Mapping): - return () - pointers: list[str] = [] - for name, value in sorted(capabilities.items(), key=lambda item: str(item[0])): - if not _is_affirmative_capability_value(value): - continue - surface = f"/capabilities/{_escape_pointer_token(str(name))}" - if isinstance(value, Mapping): - nested = tuple( - _iter_affirmative_capability_pointers(cast(Mapping[object, object], value), surface) - ) - pointers.extend(nested or (surface,)) - else: - pointers.append(surface) - return tuple(pointers) - - -def _iter_affirmative_capability_pointers( - value: Mapping[object, object], - base_pointer: str, -) -> Iterable[str]: - """Yield affirmative leaves below one manifest capability mapping.""" - - for key, child in sorted(value.items(), key=lambda item: str(item[0])): - if key in _NON_CAPABILITY_KEYS or not _is_affirmative_capability_value(child): - continue - pointer = f"{base_pointer}/{_escape_pointer_token(str(key))}" - if isinstance(child, Mapping): - nested = tuple( - _iter_affirmative_capability_pointers(cast(Mapping[object, object], child), pointer) - ) - yield from nested or (pointer,) - else: - yield pointer - - -def _is_affirmative_capability_value(value: object) -> bool: - """Return whether a manifest capability value makes an affirmative claim.""" - - affirmative = False - if value is True: - affirmative = True - elif isinstance(value, str | int | float) and value is not False: - affirmative = bool(value) - elif isinstance(value, Sequence) and not isinstance(value, str | bytes): - affirmative = any(_is_affirmative_capability_value(item) for item in value) - elif isinstance(value, Mapping): - affirmative = any( - _is_affirmative_capability_value(child) - for key, child in value.items() - if key not in _NON_CAPABILITY_KEYS - ) - return affirmative - - -def _escape_pointer_token(token: str) -> str: - """Escape one JSON Pointer token.""" - - return token.replace("~", "~0").replace("/", "~1") - - def _cli_output_directory(value: str) -> Path: """Resolve a relative artifact directory beneath the invocation directory.""" @@ -997,8 +807,7 @@ def main(argv: Sequence[str] | None = None) -> int: "cyborg_backend_conformance_payload", "cyborg_conformance_reproduction_commands", "cyborg_declared_weaknesses", - "cyborg_manifest_capability_evidence", - "cyborg_manifest_capability_evidence_gaps", + "cyborg_manifest_capability_gaps", "cyborg_source_diagnostics", "run_cyborg_conformance", "run_cyborg_conformance_suite", diff --git a/src/raes_adapters/nasim/backend/__init__.py b/src/raes_adapters/nasim/backend/__init__.py index fd5d8b0..03a7f2b 100644 --- a/src/raes_adapters/nasim/backend/__init__.py +++ b/src/raes_adapters/nasim/backend/__init__.py @@ -7,8 +7,7 @@ PR_CONFORMANCE_SEED, nasim_backend_conformance_payload, nasim_declared_weaknesses, - nasim_manifest_capability_evidence, - nasim_manifest_capability_evidence_gaps, + nasim_manifest_capability_gaps, nasim_source_protocol_diagnostics, run_nasim_conformance, run_nasim_pr_conformance, @@ -50,8 +49,7 @@ "execute_nasim_cleanup", "nasim_backend_conformance_payload", "nasim_declared_weaknesses", - "nasim_manifest_capability_evidence", - "nasim_manifest_capability_evidence_gaps", + "nasim_manifest_capability_gaps", "nasim_source_protocol_diagnostics", "run_nasim_conformance", "run_nasim_pr_conformance", diff --git a/src/raes_adapters/nasim/backend/conformance.py b/src/raes_adapters/nasim/backend/conformance.py index b8649f7..2d08062 100644 --- a/src/raes_adapters/nasim/backend/conformance.py +++ b/src/raes_adapters/nasim/backend/conformance.py @@ -2,7 +2,7 @@ from __future__ import annotations -from collections.abc import Iterable, Mapping +from collections.abc import Mapping from raes_backend_protocols.backend_manifest import ( # type: ignore[import-untyped] BackendManifest, @@ -19,10 +19,7 @@ ) from raes_adapters._gym_backend import conformance as gym -from raes_adapters._gym_backend.conformance import ( - GymConformanceConfig, - standard_probe_requirements, -) +from raes_adapters._gym_backend.conformance import GymConformanceConfig from raes_adapters.nasim import load_qualification from raes_adapters.nasim.scenario_ledger import ( NASIM_TINY, @@ -40,23 +37,10 @@ # or global random state (see nasim-conformance-guardrails.md). PR_CONFORMANCE_SEED = 20260802 -_BACKEND_EVIDENCE = "evidence.nasim.backend-conformance" -_SOURCE_EVIDENCE = "evidence.nasim.source-protocol.validated" _SOURCE_FAILED = "nasim.source-protocol.validation-failed" _CONFIG = GymConformanceConfig( name="nasim", - backend_evidence=_BACKEND_EVIDENCE, - source_evidence=_SOURCE_EVIDENCE, source_validation_failed=_SOURCE_FAILED, - probe_requirements=standard_probe_requirements( - _BACKEND_EVIDENCE, - _SOURCE_EVIDENCE, - participant_dual=( - "supported_behavior_features", - "supported_interaction_features", - "supported_participant_roles", - ), - ), default_selection=NASIM_TINY, create_target=create_nasim_target, create_manifest=create_nasim_manifest, @@ -84,32 +68,14 @@ def nasim_backend_conformance_payload( return gym.conformance_payload(report) -def nasim_manifest_capability_evidence( - manifest: BackendManifest | None = None, - *, - payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), -) -> dict[str, tuple[str, ...]]: - """Return evidence references for declared affirmative manifest surfaces.""" - - return gym.capability_evidence( - _CONFIG, manifest, payload, conformance_report, source_diagnostics - ) - - -def nasim_manifest_capability_evidence_gaps( +def nasim_manifest_capability_gaps( manifest: BackendManifest | None = None, *, payload: Mapping[str, object] | None = None, - conformance_report: BackendConformanceReport | None = None, - source_diagnostics: Iterable[Diagnostic] = (), ) -> tuple[str, ...]: - """Return declared affirmative capability surfaces with no probe evidence.""" + """Return every affirmative manifest leaf as unresolved inventory.""" - return gym.capability_evidence_gaps( - _CONFIG, manifest, payload, conformance_report, source_diagnostics - ) + return gym.capability_gaps(_CONFIG, manifest, payload) def nasim_declared_weaknesses( @@ -138,7 +104,7 @@ def run_nasim_pr_conformance( Runs the published profile/corpus conformance for the profile inferred from the live manifest — which drives the supplied driver through the constructed ``RuntimeTarget``'s four surfaces on the published fixtures — then collects - source-protocol diagnostics, closes manifest capability evidence, and gathers + source-protocol diagnostics, inventories unresolved manifest claims, and gathers declared weaknesses into a single validated portable bundle. The three claims stay distinct: ``backend_conformance`` is the exact published report payload, ``source_diagnostics`` are RAES source evidence, and ``declared_weaknesses`` @@ -146,9 +112,8 @@ def run_nasim_pr_conformance( ``driver`` is required: the caller selects the lane explicitly (a deterministic injected driver for the PR/clean-install lane, a real - ``NasimDriver`` for the manual-live lane). It fails closed by raising when a - declared affirmative capability has no passing evidence, so a new surface can - never ship without an evidence join. + ``NasimDriver`` for the manual-live lane). No broad pass result is promoted + into per-leaf capability evidence. The hostile-failure and portable-output-leakage probes over the four surfaces are injected-driver constructs — they need drivers that raise on a @@ -161,26 +126,13 @@ def run_nasim_pr_conformance( report = run_nasim_conformance(driver=driver, seed=seed) diagnostics = nasim_source_protocol_diagnostics() payload = backend_manifest_payload(create_nasim_manifest()) - evidence = nasim_manifest_capability_evidence( - payload=payload, - conformance_report=report, - source_diagnostics=diagnostics, - ) - gaps = nasim_manifest_capability_evidence_gaps( - payload=payload, - conformance_report=report, - source_diagnostics=diagnostics, - ) - if gaps: - raise RuntimeError( - "NASim manifest capability evidence is incomplete for surfaces: " + ", ".join(gaps) - ) + gaps = nasim_manifest_capability_gaps(payload=payload) return { "seed": seed, "native_conformance": report.native_conformance, "backend_conformance": nasim_backend_conformance_payload(report), "source_diagnostics": [diagnostic_payload(item) for item in diagnostics], - "capability_evidence": {pointer: list(refs) for pointer, refs in sorted(evidence.items())}, + "capability_gaps": list(gaps), "declared_weaknesses": list(nasim_declared_weaknesses()), } @@ -189,8 +141,7 @@ def run_nasim_pr_conformance( "PR_CONFORMANCE_SEED", "nasim_backend_conformance_payload", "nasim_declared_weaknesses", - "nasim_manifest_capability_evidence", - "nasim_manifest_capability_evidence_gaps", + "nasim_manifest_capability_gaps", "nasim_source_protocol_diagnostics", "run_nasim_conformance", "run_nasim_pr_conformance", diff --git a/src/raes_adapters/nasim/backend/manifest.py b/src/raes_adapters/nasim/backend/manifest.py index 62096cd..834b332 100644 --- a/src/raes_adapters/nasim/backend/manifest.py +++ b/src/raes_adapters/nasim/backend/manifest.py @@ -13,9 +13,13 @@ BackendManifest, ) from raes_backend_protocols.capabilities import ( # type: ignore[import-untyped] - BackendCapabilitySet, + CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + CleanupCapabilities, + EvaluatorCapabilities, + OrchestratorCapabilities, ParticipantRuntimeCapabilities, ProvisionerCapabilities, + WorkflowFeature, ) from raes_contracts.apparatus import ( # type: ignore[import-untyped] RealizationSupportDeclaration, @@ -26,10 +30,9 @@ from raes_adapters._manifest_support import ( assemble_manifest, + compose_capability_set, + declared_cleanup_capabilities, read_source_revision, - standard_cleanup_capabilities, - standard_evaluator_capabilities, - standard_orchestrator_capabilities, ) from raes_adapters.nasim import load_qualification @@ -108,34 +111,61 @@ def _participant_capabilities() -> ParticipantRuntimeCapabilities: ) -def _capabilities() -> BackendCapabilitySet: - """Compose the complete selected backend capability declaration.""" +def _orchestrator_capabilities() -> OrchestratorCapabilities: + """Declare the NASim orchestration surface.""" - return BackendCapabilitySet( - provisioner=_provisioner_capabilities(), - orchestrator=standard_orchestrator_capabilities( - "nasim", - { - "native_transition_owner": "participant-runtime", - "workflow_scope": "episode lifecycle and serialized participant steps", - }, - ), - evaluator=standard_evaluator_capabilities( - "nasim", - { - "reward_owner": "evaluator", - "outcome_reproduction": "stochastic-bounded", - "goal_truth": ( - "goal attainment is evaluator-only truth, separate from reward and " - "from the participant observation boundary" - ), - "objective_terminal_state": ( - "running until a distinct mapped terminal cause is available" - ), - }, - ), - participant_runtime=_participant_capabilities(), - cleanup=standard_cleanup_capabilities("nasim"), + return OrchestratorCapabilities( + name="nasim-orchestrator", + supported_sections=frozenset({"events", "workflows"}), + supports_workflows=True, + supports_assertion_refs=False, + supports_inject_bindings=False, + supported_workflow_features=frozenset({WorkflowFeature.CALL}), + constraints={ + "native_transition_owner": "participant-runtime", + "workflow_scope": "episode lifecycle and serialized participant steps", + }, + ) + + +def _evaluator_capabilities() -> EvaluatorCapabilities: + """Declare the NASim evaluator surface.""" + + return EvaluatorCapabilities( + name="nasim-evaluator", + supported_sections=frozenset({"conditions", "propositions", "assertions", "objectives"}), + supports_scoring=True, + supports_objectives=True, + supported_predicate_families=frozenset({"presence", "boolean", "string", "number"}), + supported_quantifiers=frozenset({"all", "any", "at_least"}), + supported_truth_outcomes=frozenset({"true", "false", "unknown", "unsupported"}), + supported_evidence_channels=frozenset({"api_response"}), + supported_time_domains=frozenset({"wall_clock"}), + preserves_binding_provenance=True, + constraints={ + "reward_owner": "evaluator", + "outcome_reproduction": "stochastic-bounded", + "goal_truth": ( + "goal attainment is evaluator-only truth, separate from reward and " + "from the participant observation boundary" + ), + "objective_terminal_state": ( + "running until a distinct mapped terminal cause is available" + ), + }, + ) + + +def _cleanup_capabilities() -> CleanupCapabilities: + """Declare the NASim cleanup surface.""" + + return declared_cleanup_capabilities( + name="nasim-cleanup", + supported_contract_versions=CLEANUP_CAPABILITY_REQUIRED_CONTRACTS, + supported_action_kinds=frozenset({"destroy", "reset", "verify"}), + supported_verification_methods=frozenset({"probe", "receipt"}), + supports_reusable_state=True, + supports_residual_state_disclosure=True, ) @@ -183,7 +213,13 @@ def create_nasim_manifest() -> BackendManifest: source_revision = read_source_revision(load_qualification) return assemble_manifest( NASIM_BACKEND_NAME, - _capabilities(), + compose_capability_set( + provisioner=_provisioner_capabilities(), + orchestrator=_orchestrator_capabilities(), + evaluator=_evaluator_capabilities(), + participant_runtime=_participant_capabilities(), + cleanup=_cleanup_capabilities(), + ), _realization_support(source_revision), { "source_identity": "qualified-complete-runtime-artifact-roots-attested", diff --git a/src/raes_adapters/nasim/researcher.py b/src/raes_adapters/nasim/researcher.py index 86664fa..8db1361 100644 --- a/src/raes_adapters/nasim/researcher.py +++ b/src/raes_adapters/nasim/researcher.py @@ -72,7 +72,7 @@ PR_CONFORMANCE_SEED, nasim_backend_conformance_payload, nasim_declared_weaknesses, - nasim_manifest_capability_evidence_gaps, + nasim_manifest_capability_gaps, nasim_source_protocol_diagnostics, run_nasim_conformance, ) @@ -478,10 +478,7 @@ def nasim_conformance_suite(*, suite: str, output_dir: Path) -> dict[str, object or any(not case.passed for case in report.cases) ): raise RuntimeError("NASim published conformance cases failed.") - if nasim_manifest_capability_evidence_gaps( - conformance_report=report, source_diagnostics=diagnostics - ): - raise RuntimeError("NASim manifest capability evidence is incomplete.") + capability_gaps = nasim_manifest_capability_gaps() report_path = write_backend_conformance_report( payload, output_dir=output_dir, @@ -493,6 +490,7 @@ def nasim_conformance_suite(*, suite: str, output_dir: Path) -> dict[str, object "execution_basis": "installed-source-probe", "native_conformance": report.native_conformance, "report_path": report_path.relative_to(output_dir).as_posix(), + "capability_gaps": list(capability_gaps), } ) index: dict[str, object] = { diff --git a/src/raes_adapters/primaite/backend/__init__.py b/src/raes_adapters/primaite/backend/__init__.py index 63c5da0..f838ff6 100644 --- a/src/raes_adapters/primaite/backend/__init__.py +++ b/src/raes_adapters/primaite/backend/__init__.py @@ -7,8 +7,7 @@ PR_CONFORMANCE_SEED, primaite_backend_conformance_payload, primaite_declared_weaknesses, - primaite_manifest_capability_evidence, - primaite_manifest_capability_evidence_gaps, + primaite_manifest_capability_gaps, primaite_source_protocol_diagnostics, run_primaite_conformance, run_primaite_pr_conformance, @@ -57,8 +56,7 @@ "execute_primaite_cleanup", "primaite_backend_conformance_payload", "primaite_declared_weaknesses", - "primaite_manifest_capability_evidence", - "primaite_manifest_capability_evidence_gaps", + "primaite_manifest_capability_gaps", "primaite_source_protocol_diagnostics", "run_primaite_conformance", "run_primaite_pr_conformance", diff --git a/src/raes_adapters/primaite/backend/conformance.py b/src/raes_adapters/primaite/backend/conformance.py index 7d8736d..56efe88 100644 --- a/src/raes_adapters/primaite/backend/conformance.py +++ b/src/raes_adapters/primaite/backend/conformance.py @@ -23,7 +23,7 @@ diagnostic_payload, ) -from raes_adapters._conformance_support import affirmative_capability_pointers +from raes_adapters._conformance_support import manifest_capability_gaps from raes_adapters.base import run_conformance_probe from raes_adapters.primaite import load_qualification from raes_adapters.primaite.scenario_ledger import ( @@ -62,22 +62,7 @@ def primaite_backend_conformance_payload( return cast(dict[str, object], backend_conformance_report_payload(report)) -def primaite_manifest_capability_evidence() -> dict[str, tuple[str, ...]]: - """Return production capability evidence for the backend — deliberately none. - - No affirmative runtime capability is production-evidenced, and none can be - derived from a manifest or its payload: the live ``PrimaiteDriver`` fails closed - (it cannot execute in-process), so any passing target-conformance report - necessarily came from an injected test double, and a fake driver cannot upgrade - a live-runtime claim. Every declared capability is therefore reported as an open - gap by :func:`primaite_manifest_capability_evidence_gaps` rather than certified - here, so this evidence set is unconditionally empty. - """ - - return {} - - -def primaite_manifest_capability_evidence_gaps( +def primaite_manifest_capability_gaps( manifest: BackendManifest | None = None, *, payload: Mapping[str, object] | None = None, @@ -98,7 +83,7 @@ def primaite_manifest_capability_evidence_gaps( backend_manifest_payload(manifest or create_primaite_manifest()), ) ) - return affirmative_capability_pointers(resolved) + return manifest_capability_gaps(resolved) def primaite_declared_weaknesses( @@ -189,24 +174,22 @@ def run_primaite_pr_conformance( Unlike the NASim bundle this function does **not** raise on capability gaps. For PrimAITE every declared affirmative capability is an *expected* open gap: no affirmative runtime capability is production-evidenced while the live driver - is non-runnable, so ``capability_evidence`` is empty and ``capability_gaps`` - surfaces the full affirmative-pointer set as explicit non-claims rather than a - failure. Certifying any of those surfaces requires a real, qualified, isolated - live driver (blocked pending worker-process isolation and CPython 3.12 - qualification evidence), never a fake driver. + is non-runnable, so ``capability_gaps`` surfaces the full affirmative-pointer + set as explicit non-claims rather than manufacturing positive evidence. + Certifying any of those surfaces requires a real, qualified, isolated live + driver (blocked pending worker-process isolation and CPython 3.12 qualification + evidence), never a fake driver. """ report = run_primaite_conformance(driver=driver, seed=seed) diagnostics = primaite_source_protocol_diagnostics() payload = backend_manifest_payload(create_primaite_manifest()) - evidence = primaite_manifest_capability_evidence() - gaps = primaite_manifest_capability_evidence_gaps(payload=payload) + gaps = primaite_manifest_capability_gaps(payload=payload) return { "seed": seed, "native_conformance": report.native_conformance, "backend_conformance": primaite_backend_conformance_payload(report), "source_diagnostics": [diagnostic_payload(item) for item in diagnostics], - "capability_evidence": {pointer: list(refs) for pointer, refs in sorted(evidence.items())}, "capability_gaps": list(gaps), "declared_weaknesses": list(primaite_declared_weaknesses()), } @@ -216,8 +199,7 @@ def run_primaite_pr_conformance( "PR_CONFORMANCE_SEED", "primaite_backend_conformance_payload", "primaite_declared_weaknesses", - "primaite_manifest_capability_evidence", - "primaite_manifest_capability_evidence_gaps", + "primaite_manifest_capability_gaps", "primaite_source_protocol_diagnostics", "run_primaite_conformance", "run_primaite_pr_conformance", diff --git a/tests/test_claim_integrity.py b/tests/test_claim_integrity.py new file mode 100644 index 0000000..3d43360 --- /dev/null +++ b/tests/test_claim_integrity.py @@ -0,0 +1,196 @@ +"""Repository-wide capability and evidence-claim integrity checks.""" + +from __future__ import annotations + +import ast +import json +from importlib import resources +from pathlib import Path +from types import SimpleNamespace + +import pytest +from raes_backend_protocols.manifest import backend_manifest_payload +from raes_contracts.contracts import ExperimentTaskModel + +from raes_adapters import _conformance_support, cli +from raes_adapters.primaite.backend.manifest import create_primaite_manifest + +REPO_ROOT = Path(__file__).parents[1] + +_TASKS = { + "cyborg-cage2": ( + "raes_adapters.cyborg", + "examples/cage2-research/experiment/cage2-research.task.exp.json", + ), + "nasim-tiny": ( + "raes_adapters.nasim", + "examples/nasim-tiny/experiment/nasim-tiny.task.exp.json", + ), + "cyberbattlesim-chain": ( + "raes_adapters.cyberbattlesim", + "experiment/cyberbattle-chain.task.exp.yaml", + ), + "primaite": ( + "raes_adapters.primaite", + "experiment/data-manipulation.task.exp.yaml", + ), +} + +_MANIFESTS = { + **{name: adapter.backend_manifest for name, adapter in cli._BACKENDS.items()}, + "primaite": create_primaite_manifest, +} + +_RETIRED_EXACT_IDENTIFIERS = frozenset( + { + "_CAPABILITY_PROBE_REQUIREMENTS", + "evidence_satisfies_refs", + "standard_probe_requirements", + "standard_cleanup_capabilities", + "standard_evaluator_capabilities", + "standard_orchestrator_capabilities", + } +) + + +class _PortableModel: + """Small JSON-ready model used at the artifact-writing boundary.""" + + def __init__(self, payload: dict[str, object]) -> None: + self._payload = payload + + def model_dump(self, *, mode: str) -> dict[str, object]: + assert mode == "json" + return self._payload + + +def _task(backend: str) -> ExperimentTaskModel: + package, relative = _TASKS[backend] + source = resources.files(package).joinpath(relative).read_text(encoding="utf-8") + if relative.endswith(".json"): + return ExperimentTaskModel.model_validate_json(source) + import yaml + + return ExperimentTaskModel.model_validate(yaml.safe_load(source)) + + +def _identifier_names(path: Path) -> set[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + names = {node.id for node in ast.walk(tree) if isinstance(node, ast.Name)} + names.update(node.attr for node in ast.walk(tree) if isinstance(node, ast.Attribute)) + names.update( + alias.name + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) + for alias in node.names + ) + return names + + +def _retired_static_claim_gate_identifiers(names: set[str]) -> list[str]: + """Return retired exact names and the entire manifest-evidence helper family.""" + + return sorted( + name + for name in names + if name in _RETIRED_EXACT_IDENTIFIERS or "manifest_capability_evidence" in name + ) + + +@pytest.mark.parametrize( + "data_quality", + ("available", "missing", "unavailable", "redacted", "withheld", "lossy"), +) +def test_evaluator_summary_artifact_makes_no_semantic_satisfaction_claim( + tmp_path: Path, + data_quality: str, +) -> None: + run_ref = SimpleNamespace(ref_id="run.claim-integrity") + record = SimpleNamespace( + captured_at="2026-08-12T00:00:00Z", + run_ref=run_ref, + model_dump=lambda *, mode: { + "schema_version": "experiment-evidence-record/v1", + "run_ref": {"ref_kind": "experiment-run", "ref_id": run_ref.ref_id}, + "test_data_quality": data_quality, + }, + ) + result = SimpleNamespace( + evidence_records=(record,), + derived_measures=(), + diagnostics=(), + ) + provenance = _PortableModel({"schema_version": "participant-implementation-provenance/v1"}) + adapter = SimpleNamespace( + episode_provenance=lambda _run_id, _selection: provenance, + evidence_source_label="bounded evaluator summary", + ) + admitted = SimpleNamespace(participant_selection=object()) + + artifact = cli._write_episode_evidence( + adapter, + tmp_path, + run_ref.ref_id, + 1, + result, + admitted, + ) + + assert artifact.satisfies_refs == [] + assert json.loads((tmp_path / "evidence-records.json").read_text()) + + +@pytest.mark.parametrize("backend", sorted(_TASKS)) +def test_current_semantic_evidence_requirements_fail_closed_before_execution( + backend: str, +) -> None: + assert set(cli._BACKENDS) <= set(_TASKS) + gaps = cli._task_capture_admission_gaps(_task(backend), _MANIFESTS[backend]()) + + assert gaps + assert set(gaps) >= { + requirement.ref_id + for requirement in _task(backend).evaluation_protocol.observation_requirements + } + + +@pytest.mark.parametrize("backend", sorted(_TASKS)) +def test_every_affirmative_manifest_leaf_is_an_unresolved_inventory_gap(backend: str) -> None: + manifest = _MANIFESTS[backend]() + payload = backend_manifest_payload(manifest) + affirmative = _conformance_support.affirmative_capability_pointers(payload) + + assert affirmative + assert _conformance_support.manifest_capability_gaps(payload) == affirmative + + capabilities = payload["capabilities"] + assert isinstance(capabilities, dict) + provisioner = capabilities["provisioner"] + assert isinstance(provisioner, dict) + provisioner["supports_new_mode"] = True + assert "/capabilities/provisioner/supports_new_mode" in ( + _conformance_support.manifest_capability_gaps(payload) + ) + + +def test_source_tree_contains_no_retired_static_claim_gate_identifiers() -> None: + offenders: dict[str, list[str]] = {} + for path in sorted((REPO_ROOT / "src" / "raes_adapters").rglob("*.py")): + retired = _retired_static_claim_gate_identifiers(_identifier_names(path)) + if retired: + offenders[path.relative_to(REPO_ROOT).as_posix()] = retired + + assert offenders == {} + + +@pytest.mark.parametrize( + "identifier", + ( + "manifest_capability_evidence", + "manifest_capability_evidence_gaps", + "cyborg_manifest_capability_evidence", + "cyborg_manifest_capability_evidence_gaps", + ), +) +def test_retired_manifest_evidence_helper_family_is_closed(identifier: str) -> None: + assert _retired_static_claim_gate_identifiers({identifier}) == [identifier] diff --git a/tests/test_cyberbattlesim_conformance.py b/tests/test_cyberbattlesim_conformance.py index 16da71c..6de8bae 100644 --- a/tests/test_cyberbattlesim_conformance.py +++ b/tests/test_cyberbattlesim_conformance.py @@ -58,8 +58,7 @@ from raes_adapters.cyberbattlesim.backend.conformance import ( cyberbattlesim_backend_conformance_payload, cyberbattlesim_declared_weaknesses, - cyberbattlesim_manifest_capability_evidence, - cyberbattlesim_manifest_capability_evidence_gaps, + cyberbattlesim_manifest_capability_gaps, cyberbattlesim_source_protocol_diagnostics, run_cyberbattlesim_conformance, ) @@ -334,15 +333,9 @@ def test_source_protocol_diagnostics_and_declared_weaknesses_are_raes_models() - assert "loss:loss-unbound-random-streams:deterministic-replay" in weaknesses -def test_manifest_capability_evidence_is_derived_and_fails_closed() -> None: +def test_manifest_capability_inventory_is_derived_and_fails_closed() -> None: manifest = create_cyberbattlesim_manifest() - report = _run_selected_conformance() - diagnostics = cyberbattlesim_source_protocol_diagnostics() - inventory = cyberbattlesim_manifest_capability_evidence( - manifest, - conformance_report=report, - source_diagnostics=diagnostics, - ) + gaps = cyberbattlesim_manifest_capability_gaps(manifest) assert { "/capabilities/cleanup/supported_action_kinds", @@ -350,32 +343,13 @@ def test_manifest_capability_evidence_is_derived_and_fails_closed() -> None: "/capabilities/orchestrator/supports_workflows", "/capabilities/participant_runtime/supported_behavior_features", "/capabilities/provisioner/supported_node_types", - } <= set(inventory) - assert set(inventory["/capabilities/provisioner/supported_node_types"]) == { - "evidence.cyberbattlesim.backend-conformance", - "evidence.cyberbattlesim.source-protocol.validated", - } - assert inventory["/capabilities/cleanup/supported_action_kinds"] == ( - "evidence.cyberbattlesim.backend-conformance", - ) - assert ( - cyberbattlesim_manifest_capability_evidence_gaps( - manifest, - conformance_report=report, - source_diagnostics=diagnostics, - ) - == () - ) + } <= set(gaps) payload = backend_manifest_payload(manifest) payload["capabilities"]["provisioner"]["supports_new_mode"] = True - payload_gaps = cyberbattlesim_manifest_capability_evidence_gaps( - payload=payload, - conformance_report=report, - source_diagnostics=diagnostics, - ) - assert payload_gaps == ("/capabilities/provisioner/supports_new_mode",) + payload_gaps = cyberbattlesim_manifest_capability_gaps(payload=payload) + assert "/capabilities/provisioner/supports_new_mode" in payload_gaps def test_manual_native_readiness_protocol_covers_adapter_conformance_path() -> None: diff --git a/tests/test_cyberbattlesim_researcher_cli.py b/tests/test_cyberbattlesim_researcher_cli.py index 7e71480..c66aecf 100644 --- a/tests/test_cyberbattlesim_researcher_cli.py +++ b/tests/test_cyberbattlesim_researcher_cli.py @@ -175,15 +175,13 @@ def test_inspect_reports_external_pack_without_native_import( assert payload["examples"] == ["cyberbattlesim-chain (external release asset)"] -def test_validate_admits_exact_external_pack(capsys: pytest.CaptureFixture[str]) -> None: - assert cli.main(_args()) == 0 - assert json.loads(capsys.readouterr().out) == { - "disposition": "validated", - "pack": "admitted", - "participant": "cyberbattlesim-red-credential-cache", - "run_count": 1, - "scope": "run-admission", - } +def test_validate_rejects_task_with_unverifiable_semantic_evidence( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(_args()) == cli.EXIT_VALIDATION + captured = capsys.readouterr() + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err def test_validate_rejects_any_resealed_pack_digest( @@ -228,7 +226,7 @@ def test_episode_uses_source_policy_proposal_then_raes_admission() -> None: assert driver.closed -def test_run_suppresses_native_output_and_seals_portable_evidence( +def test_run_rejects_unverifiable_evidence_before_execution_or_output( tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch, @@ -246,26 +244,18 @@ def test_run_suppresses_native_output_and_seals_portable_evidence( ), ) - execute_episode = researcher.execute_episode - - def run_with_fake(scenario: object, controls: object) -> object: - print("native output must be suppressed") - return execute_episode(scenario, controls, driver=FakeDriver()) # type: ignore[arg-type] - - monkeypatch.setattr(researcher, "execute_episode", run_with_fake) + monkeypatch.setattr( + researcher, + "execute_episode", + lambda *args, **kwargs: pytest.fail("unverifiable evidence must not execute"), + ) before = Path.cwd() try: os.chdir(tmp_path) - assert cli.main(_args("run")) == 0 + assert cli.main(_args("run")) == cli.EXIT_VALIDATION finally: os.chdir(before) captured = capsys.readouterr() - assert "native output" not in captured.out - inventory = json.loads((tmp_path / "evidence" / "inventory.json").read_text()) - paths = {item["path"] for item in inventory["artifacts"]} - assert "provenance.json" in paths - assert "runs/cyberbattlesim-smoke-1/run.json" in paths - serialized = "\n".join(path.read_text() for path in (tmp_path / "evidence").rglob("*.json")) - assert str(PACK_ROOT) not in serialized - assert "must-not-cross" not in serialized - assert "Traceback" not in serialized + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err + assert not (tmp_path / "evidence").exists() diff --git a/tests/test_cyborg_conformance.py b/tests/test_cyborg_conformance.py index aaa66da..951ca47 100644 --- a/tests/test_cyborg_conformance.py +++ b/tests/test_cyborg_conformance.py @@ -29,8 +29,7 @@ cyborg_backend_conformance_payload, cyborg_conformance_reproduction_commands, cyborg_declared_weaknesses, - cyborg_manifest_capability_evidence, - cyborg_manifest_capability_evidence_gaps, + cyborg_manifest_capability_gaps, cyborg_source_diagnostics, register_cyborg_backend, run_cyborg_conformance, @@ -132,28 +131,11 @@ def test_pr_conformance_uses_the_exact_published_report_shape() -> None: assert report.unsupported_capability_gaps == () -def test_every_affirmative_manifest_capability_has_passing_evidence() -> None: - report = run_cyborg_conformance(seed=3) - diagnostics = cyborg_source_diagnostics() - adapter_diagnostics = cyborg_adapter_diagnostics(seed=3) +def test_affirmative_manifest_capabilities_remain_unresolved_inventory() -> None: + gaps = cyborg_manifest_capability_gaps() - evidence = cyborg_manifest_capability_evidence( - conformance_report=report, - source_diagnostics=diagnostics, - adapter_diagnostics=adapter_diagnostics, - ) - - assert evidence - assert ( - cyborg_manifest_capability_evidence_gaps( - conformance_report=report, - source_diagnostics=diagnostics, - adapter_diagnostics=adapter_diagnostics, - ) - == () - ) - assert all(pointer.startswith("/capabilities/") for pointer in evidence) - assert all(refs for refs in evidence.values()) + assert gaps + assert all(pointer.startswith("/capabilities/") for pointer in gaps) def test_adapter_probes_cover_every_runtime_surface_with_valid_diagnostics() -> None: @@ -333,6 +315,7 @@ def test_suite_index_preserves_canonical_reports_and_non_claims( persisted = json.loads(report_path.read_text(encoding="utf-8")) assert persisted["profile"] == "full-remote-control-plane" assert persisted["native_conformance"] is False + assert reports[0]["capability_gaps"] == list(cyborg_manifest_capability_gaps()) assert json.loads((tmp_path / "index.json").read_text(encoding="utf-8")) == index @@ -388,20 +371,3 @@ def test_suite_refuses_an_unexpected_published_failure( run_cyborg_conformance_suite(suite="pr", output_dir=tmp_path) assert not (tmp_path / "index.json").exists() - - -def test_suite_refuses_manifest_capability_evidence_gaps( - monkeypatch: pytest.MonkeyPatch, - tmp_path: Path, -) -> None: - monkeypatch.chdir(tmp_path.parent) - monkeypatch.setattr( - conformance_module, - "cyborg_manifest_capability_evidence_gaps", - lambda *args, **kwargs: ("/capabilities/provisioner/supported_node_types",), - ) - - with pytest.raises(RuntimeError, match="manifest capability evidence is incomplete"): - run_cyborg_conformance_suite(suite="pr", output_dir=tmp_path) - - assert not (tmp_path / "index.json").exists() diff --git a/tests/test_nasim_conformance.py b/tests/test_nasim_conformance.py index 86ea739..024c7ec 100644 --- a/tests/test_nasim_conformance.py +++ b/tests/test_nasim_conformance.py @@ -13,7 +13,6 @@ from dataclasses import dataclass, field, replace from pathlib import Path -import pytest from raes_backend_protocols.manifest import backend_manifest_payload from raes_conformance.conformance import BackendConformanceReport from raes_conformance.conformance.report import backend_conformance_report_payload @@ -49,8 +48,7 @@ from raes_adapters.nasim.backend.conformance import ( nasim_backend_conformance_payload, nasim_declared_weaknesses, - nasim_manifest_capability_evidence, - nasim_manifest_capability_evidence_gaps, + nasim_manifest_capability_gaps, nasim_source_protocol_diagnostics, run_nasim_conformance, run_nasim_pr_conformance, @@ -232,15 +230,9 @@ def test_source_protocol_diagnostics_and_declared_weaknesses_are_raes_models() - assert "loss:loss-unbound-action-rng:deterministic-replay" in weaknesses -def test_manifest_capability_evidence_is_derived_and_fails_closed() -> None: +def test_manifest_capability_inventory_is_derived_and_fails_closed() -> None: manifest = create_nasim_manifest() - report = run_nasim_conformance(driver=ProbeDriver(), seed=PR_CONFORMANCE_SEED) - diagnostics = nasim_source_protocol_diagnostics() - inventory = nasim_manifest_capability_evidence( - manifest, - conformance_report=report, - source_diagnostics=diagnostics, - ) + gaps = nasim_manifest_capability_gaps(manifest) assert { "/capabilities/cleanup/supported_action_kinds", @@ -248,39 +240,14 @@ def test_manifest_capability_evidence_is_derived_and_fails_closed() -> None: "/capabilities/orchestrator/supports_workflows", "/capabilities/participant_runtime/supported_behavior_features", "/capabilities/provisioner/supported_node_types", - } <= set(inventory) - # Provisioner/participant surfaces require both backend and source evidence; - # control-plane surfaces need only the canonical report. - assert set(inventory["/capabilities/provisioner/supported_node_types"]) == { - "evidence.nasim.backend-conformance", - "evidence.nasim.source-protocol.validated", - } - assert inventory["/capabilities/cleanup/supported_action_kinds"] == ( - "evidence.nasim.backend-conformance", - ) - assert ( - nasim_manifest_capability_evidence_gaps( - manifest, - conformance_report=report, - source_diagnostics=diagnostics, - ) - == () - ) + } <= set(gaps) # A newly declared affirmative capability with no probe evidence must fail # closed as a gap rather than be silently backed. payload = backend_manifest_payload(manifest) payload["capabilities"]["provisioner"]["supports_new_mode"] = True - payload_gaps = nasim_manifest_capability_evidence_gaps( - payload=payload, - conformance_report=report, - source_diagnostics=diagnostics, - ) - assert payload_gaps == ("/capabilities/provisioner/supports_new_mode",) - - # With no probe evidence at all, every affirmative surface is a gap. - assert nasim_manifest_capability_evidence(payload=backend_manifest_payload(manifest)) == {} - assert nasim_manifest_capability_evidence_gaps(payload=backend_manifest_payload(manifest)) + payload_gaps = nasim_manifest_capability_gaps(payload=payload) + assert "/capabilities/provisioner/supports_new_mode" in payload_gaps def test_pr_conformance_suite_composes_published_and_source_evidence() -> None: @@ -290,34 +257,13 @@ def test_pr_conformance_suite_composes_published_and_source_evidence() -> None: assert bundle["native_conformance"] is False assert bundle["backend_conformance"]["passed"] is True assert bundle["source_diagnostics"] - assert bundle["capability_evidence"]["/capabilities/provisioner/supported_node_types"] == [ - "evidence.nasim.backend-conformance", - "evidence.nasim.source-protocol.validated", - ] + assert bundle["capability_gaps"] == list(nasim_manifest_capability_gaps()) assert any(item.startswith("limitation:") for item in bundle["declared_weaknesses"]) # The composed bundle is the machine-readable evidence run by the # clean-install proof; it must carry no native leakage. assert not any(marker in _json_text(bundle) for marker in NATIVE_MARKERS) -def test_pr_conformance_suite_fails_closed_on_capability_evidence_gaps( - monkeypatch: pytest.MonkeyPatch, -) -> None: - # Drive the fail-closed branch: if any declared affirmative capability - # lacks passing evidence, the bundle must refuse rather than ship an - # unbacked surface (the guarantee run_nasim_pr_conformance's docstring - # exists to enforce). The success-path test above never reaches this - # branch because the real manifest has zero gaps. - monkeypatch.setattr( - "raes_adapters.nasim.backend.conformance.nasim_manifest_capability_evidence_gaps", - lambda *args, **kwargs: ("/capabilities/provisioner/supported_node_types",), - ) - - driver = ProbeDriver() - with pytest.raises(RuntimeError, match="incomplete"): - run_nasim_pr_conformance(driver=driver) - - def test_clock_control_is_unsupported_consistent_with_manifest() -> None: manifest = create_nasim_manifest() payload = backend_manifest_payload(manifest) diff --git a/tests/test_nasim_researcher_cli.py b/tests/test_nasim_researcher_cli.py index bab2ba1..c337655 100644 --- a/tests/test_nasim_researcher_cli.py +++ b/tests/test_nasim_researcher_cli.py @@ -14,7 +14,6 @@ import os import shutil import sys -from collections.abc import Iterator from dataclasses import dataclass, field from importlib import resources, util from pathlib import Path @@ -22,8 +21,6 @@ import pytest import raes from raes_contracts.contracts import ( - ExperimentRunModel, - ExperimentStudyModel, ParticipantConfigurationResultModel, ParticipantImplementationManifestModel, ParticipantImplementationSelectionModel, @@ -46,9 +43,6 @@ _RUN_ID = "nasim-research-example" _PACK_VALIDATOR_AVAILABLE = util.find_spec("raes_env_packs") is not None _EXAMPLE_ROOT = resources.files("raes_adapters.nasim") / "examples" / "nasim-tiny" -# The authored selection's default-deny exposure list legitimately names native -# NASim symbols; every other projection must withhold them. -_NATIVE_SYMBOLS = ("flatactionspace", "servicescan", "subnetscan", "osscan", "processscan") def _native_run_args( @@ -164,21 +158,6 @@ def _bypass_native(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(nasim_driver, "_verify_pre_import_source", lambda *args, **kwargs: None) -def _leaf_strings(payload: object, skip: frozenset[str]) -> Iterator[str]: - """Yield every string leaf except values nested under a skipped key.""" - - if isinstance(payload, dict): - for key, value in payload.items(): - if key in skip: - continue - yield from _leaf_strings(value, skip) - elif isinstance(payload, list): - for value in payload: - yield from _leaf_strings(value, skip) - elif isinstance(payload, str): - yield payload - - @dataclass class FakeNasimDriver: """Injected NASim seam that retains deliberately private native state. @@ -305,15 +284,13 @@ def test_validate_rejects_invalid_pack_with_bounded_code_only( @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the nasim extra") -def test_validate_admits_packaged_example(capsys: pytest.CaptureFixture[str]) -> None: - assert cli.main(_validate_args()) == 0 - assert json.loads(capsys.readouterr().out) == { - "disposition": "validated", - "pack": "admitted", - "participant": "nasim-red-bruteforce", - "run_count": 1, - "scope": "run-admission", - } +def test_validate_rejects_task_with_unverifiable_semantic_evidence( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(_validate_args()) == cli.EXIT_VALIDATION + captured = capsys.readouterr() + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err def test_existing_output_is_never_reused( @@ -627,36 +604,38 @@ def fail_after_native_output(*args: object, **kwargs: object) -> object: print("native secret error", file=sys.stderr) raise HostileFailure() - _bypass_native(monkeypatch) monkeypatch.setattr(nasim_researcher, "execute_episode", fail_after_native_output) - previous = Path.cwd() - try: - os.chdir(tmp_path) - assert cli.main(_native_run_args("failed-evidence")) == cli.EXIT_RUNTIME - finally: - os.chdir(previous) + output = tmp_path / "failed-evidence" + output.mkdir() + adapter = cli._BACKENDS["nasim-tiny"] + with pytest.raises(cli._CommandFailure) as failure: + cli._execute_quietly(adapter, object(), object(), output) captured = capsys.readouterr() assert captured.out == "" - assert captured.err.strip() == "researcher.runtime.failure: native execution or cleanup failed" - failure = json.loads((tmp_path / "failed-evidence" / "failure.json").read_text()) - assert failure["code"] == "researcher.runtime.failure" - assert "native secret" not in json.dumps(failure) + assert captured.err == "" + assert failure.value.exit_code == cli.EXIT_RUNTIME + retained = json.loads((output / "failure.json").read_text()) + assert retained["code"] == "researcher.runtime.failure" + assert "native secret" not in json.dumps(retained) @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the nasim extra") -def test_inventory_failure_uses_artifact_exit_without_traceback( +def test_capture_admission_preempts_execution_and_inventory_writes( tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch, ) -> None: - def run_with_fake_driver(scenario: object, controls: object) -> object: - return execute_episode(scenario, controls, driver=FakeNasimDriver()) # type: ignore[arg-type] - - _bypass_native(monkeypatch) - monkeypatch.setattr(nasim_researcher, "execute_episode", run_with_fake_driver) + monkeypatch.setattr(cli.util, "find_spec", lambda name: object()) + monkeypatch.setattr( + nasim_researcher, + "execute_episode", + lambda *args, **kwargs: pytest.fail("execution must not start"), + ) monkeypatch.setattr( - cli, "_seal_inventory", lambda output: (_ for _ in ()).throw(HostileFailure()) + cli, + "_seal_inventory", + lambda output: pytest.fail("inventory must not be written"), ) previous = Path.cwd() try: @@ -665,59 +644,36 @@ def run_with_fake_driver(scenario: object, controls: object) -> object: finally: os.chdir(previous) - assert result == cli.EXIT_ARTIFACT + assert result == cli.EXIT_VALIDATION captured = capsys.readouterr() assert captured.out == "" - assert ( - captured.err.strip() == "researcher.artifact.failure: portable evidence could not be sealed" - ) + assert "researcher.validation.evidence-unverifiable" in captured.err + assert not (tmp_path / "unsealed").exists() @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the nasim extra") -def test_study_run_seals_portable_raes_evidence_and_exact_inventory( +def test_study_run_rejects_unverifiable_evidence_before_execution_or_output( tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch, ) -> None: - def run_with_fake_driver(scenario: object, controls: object) -> object: - return execute_episode(scenario, controls, driver=FakeNasimDriver()) # type: ignore[arg-type] + def reject_execution(scenario: object, controls: object) -> object: + del scenario, controls + pytest.fail("unverifiable evidence must be rejected before execution") - _bypass_native(monkeypatch) - monkeypatch.setattr(nasim_researcher, "execute_episode", run_with_fake_driver) + monkeypatch.setattr(cli.util, "find_spec", lambda name: object()) + monkeypatch.setattr(nasim_researcher, "execute_episode", reject_execution) previous = Path.cwd() try: os.chdir(tmp_path) - assert cli.main(_native_run_args("evidence", mode="study", seeds=(_SEED,))) == 0 + assert ( + cli.main(_native_run_args("evidence", mode="study", seeds=(_SEED,))) + == cli.EXIT_VALIDATION + ) finally: os.chdir(previous) captured = capsys.readouterr() - assert json.loads(captured.out)["run_count"] == 1 - output = tmp_path / "evidence" - inventory = json.loads((output / "inventory.json").read_text()) - paths = {item["path"] for item in inventory["artifacts"]} - assert {"machine-inventory.json", "provenance.json", "summary.json"} <= paths - assert f"runs/{_RUN_ID}-1/evidence-records.json" in paths - assert f"runs/{_RUN_ID}-1/derived-measures.json" in paths - assert f"runs/{_RUN_ID}-1/run.json" in paths - assert "study.json" in paths - assert not any(Path(path).is_absolute() for path in paths) - assert all(len(item["sha256"]) == 64 for item in inventory["artifacts"]) - ExperimentRunModel.model_validate_json((output / f"runs/{_RUN_ID}-1/run.json").read_text()) - ExperimentStudyModel.model_validate_json((output / "study.json").read_text()) - - serialized = "\n".join(path.read_text() for path in output.rglob("*.json")) - assert "RESEARCHER_SECRET_SENTINEL" not in serialized - assert "must-not-cross" not in serialized - assert str(tmp_path) not in serialized - assert "Traceback" not in serialized - # Native NASim symbols may appear only in the authored selection's - # ``withheld_refs`` default-deny list; nowhere else in the projections. - projected = "\n".join( - leaf - for path in output.rglob("*.json") - for leaf in _leaf_strings(json.loads(path.read_text()), frozenset({"withheld_refs"})) - ) - for symbol in _NATIVE_SYMBOLS: - assert symbol not in projected - assert captured.err == "" + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err + assert not (tmp_path / "evidence").exists() diff --git a/tests/test_primaite_backend.py b/tests/test_primaite_backend.py index 6a94a02..4e0b6b6 100644 --- a/tests/test_primaite_backend.py +++ b/tests/test_primaite_backend.py @@ -56,8 +56,7 @@ create_primaite_target, execute_primaite_cleanup, primaite_declared_weaknesses, - primaite_manifest_capability_evidence, - primaite_manifest_capability_evidence_gaps, + primaite_manifest_capability_gaps, primaite_source_protocol_diagnostics, run_primaite_conformance, ) @@ -756,9 +755,7 @@ def test_bounded_conformance_probe_never_certifies_the_non_runnable_target() -> # No affirmative runtime capability is production-evidenced: the live driver # fails closed, so a fake-driver probe cannot certify it. Every affirmative # capability is disclosed as an open gap instead. - evidence = primaite_manifest_capability_evidence() - gaps = primaite_manifest_capability_evidence_gaps() - assert evidence == {} + gaps = primaite_manifest_capability_gaps() assert "/capabilities/cleanup/supported_action_kinds" in gaps assert "/capabilities/participant_runtime/feature_support" in gaps assert "/capabilities/evaluator/supports_scoring" not in gaps # no longer declared diff --git a/tests/test_primaite_conformance.py b/tests/test_primaite_conformance.py index c5a8209..695aa28 100644 --- a/tests/test_primaite_conformance.py +++ b/tests/test_primaite_conformance.py @@ -67,8 +67,7 @@ from raes_adapters.primaite.backend.conformance import ( primaite_backend_conformance_payload, primaite_declared_weaknesses, - primaite_manifest_capability_evidence, - primaite_manifest_capability_evidence_gaps, + primaite_manifest_capability_gaps, primaite_source_protocol_diagnostics, run_primaite_conformance, run_primaite_pr_conformance, @@ -393,15 +392,13 @@ def test_source_protocol_diagnostics_and_declared_weaknesses_are_raes_models() - assert any(item.startswith("loss:loss-abstracted-participant-interface") for item in weaknesses) -def test_manifest_capability_evidence_fails_closed_for_the_non_runnable_target() -> None: +def test_manifest_capability_inventory_fails_closed_for_the_non_runnable_target() -> None: manifest = create_primaite_manifest() # No affirmative runtime capability is production-evidenced: the live driver # fails closed, so an injected-driver probe cannot certify it. Evidence is # unconditionally empty and every affirmative capability is a disclosed gap. - evidence = primaite_manifest_capability_evidence() - gaps = primaite_manifest_capability_evidence_gaps(manifest) - assert evidence == {} + gaps = primaite_manifest_capability_gaps(manifest) assert { "/capabilities/cleanup/supported_action_kinds", "/capabilities/participant_runtime/feature_support", @@ -414,7 +411,7 @@ def test_manifest_capability_evidence_fails_closed_for_the_non_runnable_target() payload = backend_manifest_payload(manifest) payload["capabilities"]["provisioner"]["supports_new_mode"] = True assert "/capabilities/provisioner/supports_new_mode" in ( - primaite_manifest_capability_evidence_gaps(payload=payload) + primaite_manifest_capability_gaps(payload=payload) ) @@ -430,7 +427,6 @@ def test_pr_conformance_bundle_composes_published_and_source_evidence() -> None: assert bundle["source_diagnostics"] # PrimAITE is fail-closed: no capability evidence closes and every affirmative # capability is disclosed as an explicit non-claim rather than raising. - assert bundle["capability_evidence"] == {} assert bundle["capability_gaps"] assert any(str(item).startswith("limitation:") for item in bundle["declared_weaknesses"]) # The composed bundle is the machine-readable evidence the clean-install proof diff --git a/tests/test_researcher_cli.py b/tests/test_researcher_cli.py index 7593afe..d543c73 100644 --- a/tests/test_researcher_cli.py +++ b/tests/test_researcher_cli.py @@ -14,8 +14,6 @@ import pytest from raes import parse_sdl from raes_contracts.contracts import ( - ExperimentRunModel, - ExperimentStudyModel, ParticipantConfigurationResultModel, ParticipantImplementationManifestModel, ParticipantImplementationSelectionModel, @@ -266,15 +264,13 @@ def test_validate_rejects_invalid_pack_with_bounded_code_only( @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the cyborg extra") -def test_validate_admits_packaged_example(capsys: pytest.CaptureFixture[str]) -> None: - assert cli.main(_validate_args()) == 0 - assert json.loads(capsys.readouterr().out) == { - "disposition": "validated", - "pack": "admitted", - "participant": "cyborg-blue-sleep-policy", - "run_count": 2, - "scope": "run-admission", - } +def test_validate_rejects_task_with_unverifiable_semantic_evidence( + capsys: pytest.CaptureFixture[str], +) -> None: + assert cli.main(_validate_args()) == cli.EXIT_VALIDATION + captured = capsys.readouterr() + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the cyborg extra") @@ -713,66 +709,49 @@ def fail_after_native_output(*args: object, **kwargs: object) -> object: print("native secret error", file=sys.stderr) raise HostileFailure() - monkeypatch.setattr(cli.util, "find_spec", lambda name: object()) - monkeypatch.setattr(cli, "verify_selected_cyborg_source", lambda: None) monkeypatch.setattr(researcher, "execute_episode", fail_after_native_output) - previous = Path.cwd() - try: - import os - - os.chdir(tmp_path) - assert cli.main(_native_run_args("failed-evidence")) == cli.EXIT_RUNTIME - finally: - os.chdir(previous) + output = tmp_path / "failed-evidence" + output.mkdir() + adapter = cli._BACKENDS["cyborg-cage2"] + with pytest.raises(cli._CommandFailure) as failure: + cli._execute_quietly(adapter, object(), object(), output) captured = capsys.readouterr() assert captured.out == "" - assert captured.err.strip() == "researcher.runtime.failure: native execution or cleanup failed" - failure = json.loads((tmp_path / "failed-evidence" / "failure.json").read_text()) - assert failure["code"] == "researcher.runtime.failure" - assert "native secret" not in json.dumps(failure) + assert captured.err == "" + assert failure.value.exit_code == cli.EXIT_RUNTIME + retained = json.loads((output / "failure.json").read_text()) + assert retained["code"] == "researcher.runtime.failure" + assert "native secret" not in json.dumps(retained) @pytest.mark.skipif(not _PACK_VALIDATOR_AVAILABLE, reason="requires the cyborg extra") -def test_study_run_seals_portable_raes_evidence_and_exact_inventory( +def test_study_run_rejects_unverifiable_evidence_before_execution_or_output( tmp_path: Path, capsys: pytest.CaptureFixture[str], monkeypatch: pytest.MonkeyPatch, ) -> None: from raes_adapters.cyborg import researcher - def run_with_fake_driver(scenario: object, controls: RunControls) -> object: - return execute_episode(scenario, controls, driver=FakeResearchDriver()) + def reject_execution(scenario: object, controls: RunControls) -> object: + del scenario, controls + pytest.fail("unverifiable evidence must be rejected before execution") monkeypatch.setattr(cli.util, "find_spec", lambda name: object()) - monkeypatch.setattr(cli, "verify_selected_cyborg_source", lambda: None) - monkeypatch.setattr(researcher, "execute_episode", run_with_fake_driver) + monkeypatch.setattr(researcher, "execute_episode", reject_execution) previous = Path.cwd() try: import os os.chdir(tmp_path) - assert cli.main(_native_run_args("evidence", mode="study", seeds=(7, 11))) == 0 + assert ( + cli.main(_native_run_args("evidence", mode="study", seeds=(7, 11))) + == cli.EXIT_VALIDATION + ) finally: os.chdir(previous) captured = capsys.readouterr() - assert json.loads(captured.out)["run_count"] == 2 - output = tmp_path / "evidence" - inventory = json.loads((output / "inventory.json").read_text()) - paths = {item["path"] for item in inventory["artifacts"]} - assert {"machine-inventory.json", "provenance.json", "summary.json"} <= paths - assert "runs/research-example-1/evidence-records.json" in paths - assert "runs/research-example-2/derived-measures.json" in paths - assert "runs/research-example-1/run.json" in paths - assert "study.json" in paths - ExperimentRunModel.model_validate_json( - (output / "runs/research-example-1/run.json").read_text() - ) - ExperimentStudyModel.model_validate_json((output / "study.json").read_text()) - assert not any(Path(path).is_absolute() for path in paths) - serialized = "\n".join(path.read_text() for path in output.rglob("*.json")) - assert "RESEARCHER_SECRET_SENTINEL" not in serialized - assert str(tmp_path) not in serialized - assert "Traceback" not in serialized - assert captured.err == "" + assert captured.out == "" + assert "researcher.validation.evidence-unverifiable" in captured.err + assert not (tmp_path / "evidence").exists()