fix: remove hardcoded capability and evidence-satisfaction gates from every adapter

[Brad-Edwards]

Objective

Remove every hardcoded capability and evidence-satisfaction gate from every adapter, even when doing so breaks an adapter until its real capture path is implemented.

Problem

#30 exposed hardcoded evidence-reference claims for attacker-action-log and availability-series. The action log was not emitted, yet the static claims allowed the run to appear to satisfy the authored contract.

Scope

• Audit all adapters, shared CLI paths, and adapter plumbing for hardcoded capability claims, evidence-reference satisfaction lists, unconditional support flags, and equivalent bypasses.
• Remove every such claim.
• Derive advertised capture capabilities from the adapter's truthful backend manifest.
• Derive post-run satisfaction only from the artifacts and fields actually emitted for that run.
• Fail closed when the applicable RAES contract cannot express or verify the claim.
• Do not retain compatibility fallbacks, allowlists, backend-specific exceptions, or test-only bypasses. Adapter breakage is preferable to a false claim.

Acceptance criteria

• Repository-wide tests prove no adapter can satisfy a requirement through a hardcoded reference or unconditional capability flag.
• Every advertised capture capability is backed by an implemented capture path and executable evidence.
• Every post-run satisfied reference resolves to the actual artifact and required fields that support it.
• Missing, unavailable, redacted, withheld, and lossy data are reported honestly and cannot be promoted to satisfied.
• Adapters that cannot meet an admitted requirement fail closed.
• The canonical verification graph passes for the adapters that remain admissible.

References

• test(cyberbattlesim): reproduce the selected public baseline through RAES #30

[Brad-Edwards]

🛠️ Picked up by /implement - driver codex, branch 84-remove-hardcoded-gates, 2026-08-12T15:08:55.826Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #84). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #84). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation plan

Claim boundaries and fail-closed admission

• Remove _BackendAdapter.evidence_satisfies_refs and every backend-name evidence-reference allowlist. Episode evidence artifacts will describe only the evaluator-summary bytes actually emitted and will carry no semantic satisfies_refs claim.
• Add a pre-effect researcher admission check over the published ExperimentTaskModel and live BackendManifest. The current RAES 3.3 manifest/run contracts cannot bind a semantic evidence reference to required artifact fields or their unavailable/redacted/withheld/lossy status, so every current task with semantic evidence requirements is rejected before runtime planning, native source import, output reservation, or simulator effects. Do not weaken the authored tasks; issues 85–88 own real per-adapter remediation.
• Keep validate_experiment_run_against_task() and study validation as the authoritative post-run joins. Add no adapter schema, vocabulary, compatibility allowlist, backend exception, or test bypass.

Capability accounting

• Delete the static capability-pointer-to-evidence maps, broad passed-reference joins, and positive *_manifest_capability_evidence APIs from shared, CybORG, NASim, CyberBattleSim, and PrimAITE conformance composition.
• Retain manifest pointer traversal only as truthful inventory. Conformance bundles will expose every affirmative manifest leaf as an unresolved capability_gaps entry and will never turn a broad conformance/source/probe pass into per-leaf evidence. Canonical RAES conformance reports and backend-local diagnostics remain separate bounded facts.
• Remove unconditional “standard capability” grants from shared manifest plumbing. Backend modules must construct their own published RAES capability models explicitly; this keeps backend declarations local while issues 85–88 reconcile each declaration with production capture evidence.

TDD and repository-wide regression coverage

• First add repository-wide negative tests that fail on the current tree: no registered adapter may carry a static satisfaction list; emitted evaluator-summary artifacts have empty satisfies_refs; every current evidence-requiring task is rejected before native/runtime/output effects; every affirmative manifest leaf is automatically reported as a gap; and an injected new leaf is also a gap.
• Update focused CLI tests to assert the bounded validation failure and absence of output/effects for current tasks. Keep component/evaluator tests that prove the actual sanitized records and measures still work without claiming task satisfaction.
• Update conformance tests and installed-wheel probes to assert canonical report/diagnostic serialization plus unresolved capability inventory, never positive per-leaf capability evidence. Add an AST-backed repository invariant so reintroducing the retired allowlist/map APIs fails across every registered adapter.

Documentation and whole-repository fit

• Retain the architecture-preflight note and navigation entry, and update the four backend conformance guardrails where they currently endorse static pointer closure. The docs will distinguish finite conformance, source validation, emitted artifacts, manifest declarations, and semantic task satisfaction.
• Security: all rejection occurs before native import/effects/output; existing closed parsers, source/pack/participant joins, RAES validators, confined outputs, redaction, and bounded error envelopes remain active. No auth, secret, environment, argv, network, persistence, or native-output surface is added.
• Maintainability: reuse BackendManifest, backend_manifest_payload(), ExperimentTaskModel, the canonical task/run validators, and the existing affirmative-pointer traversal. Add no second validator family or cross-simulator semantic authority.
• Extensibility: future RAES field-verifiable contracts can replace the single fail-closed admission seam; a new adapter or capability leaf is automatically covered without editing an allowlist.
• Run focused claim-integrity, CLI, conformance, backend, and installed-probe tests during TDD, then the canonical requirement-free graph: uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify -- --skip-requirement. The Conventional Commit PR title will be fix: remove hardcoded capability and evidence gates; Release Please owns version/changelog changes.

[Brad-Edwards]

Execution obligation OBL-84-DISTRIBUTION-PROBE — Opened

Category: failing_check
Observed state: The implementation correctly rejected unverifiable task evidence before execution, but nox treated the expected exit code 3 as a distribution failure.
Impact: The canonical completion graph could not pass because its installed-wheel probes still required native validation success after semantic evidence admission became fail-closed.
Current obligation: Update every installed-wheel native validation probe to require the bounded validation exit while semantic evidence requirements remain unverifiable.

Evidence

• gc_implement_mechanical verify attempt issue84-verify-1 failed at completion_gate
• nox -s distributions reproduced the failure: clean-installed cyberbattlesim validate exited 3 after fail-closed task admission

[Brad-Edwards]

Execution obligation OBL-84-DISTRIBUTION-PROBE — Resolved

Category: failing_check
Observed state: The focused distribution session is green with all three native validation probes rejecting unverifiable evidence and every remaining package/conformance gate passing.
Impact: The distribution gate now proves both installed-package integrity and fail-closed native admission.
Current obligation: Update every installed-wheel native validation probe to require the bounded validation exit while semantic evidence requirements remain unverifiable.

Evidence

• noxfile.py clean-installed cyberbattlesim, NASim, and CybORG validation probes now require exit code 3
• uv tool run --from nox[uv]==2026.4.10 nox -s distributions completed successfully in 48 seconds

Disposition: fix
Corrective action: Added success_codes=[3] to the three clean-installed native validation probes so an unexpected success remains a failure while the intended bounded rejection passes.

Verification

• The CyberBattleSim probe exited 3 before pack gates, which then passed.
• The NASim probe exited 3 before conformance and pack gates, which then passed.
• The CybORG probe exited 3 before conformance, and the complete distributions session passed.

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #84 (branch 84-remove-hardcoded-gates)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: ship-with-fixes

The production shape is largely correct: it removes adapter-local satisfaction authority, centralizes unresolved capability inventory, keeps semantic evidence claims behind the RAES-owned task/run boundary, and fails closed before runtime effects. The backend-local capability construction preserves the ownership seam without introducing a competing schema and leaves room for a future published witness contract. The material weakness is in the canonical verification graph: the installed-wheel probes now accept every validation failure with exit code 3, not specifically the intended evidence-admission rejection.

Blocking findings (1):

1. [class] Installed-wheel probes accept unrelated validation regressions — noxfile.py:616
   Allowing exit code 3 as an unconditional success means these distribution probes pass for any validation failure, including missing packaged resources, broken task/participant joins, or another installed-wheel-only regression. Source-tree tests that inspect the expected diagnostic do not preserve the clean-install guarantee. Make the installed probe assert both the intended exit code and the bounded evidence-admission diagnostic so only the deliberate fail-closed outcome is accepted.

Security review

Verdict: ship

This change is shaped correctly around a fail-closed admission seam: authored evidence requirements, manifest capability declarations, runtime artifacts, and satisfaction claims remain distinct, while unverifiable semantic requirements are rejected before native import, output reservation, or simulator effects. The shared traversal now inventories unresolved affirmative capability leaves without manufacturing evidence, and backend-specific manifest construction remains within each adapter. The change touches validation, portable artifact projection, conformance reporting, and leakage boundaries without introducing a new input, authentication, execution, filesystem, or secrets surface. I found no concrete exploitable security issue introduced by the diff.

No blocking findings.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #84 on branch '84-remove-hardcoded-gates'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #84)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The production shape is largely correct: it removes adapter-local satisfaction authority, centralizes unresolved capability inventory, keeps semantic evidence claims behind the RAES-owned task/run boundary, and fails closed before runtime effects. The backend-local capability construction preserves the ownership seam without introducing a competing schema and leaves room for a future published witness contract. The material weakness is in the canonical verification graph: the installed-wheel probes now accept every validation failure with exit code 3, not specifically the intended evidence-admission rejection.

Security reviewer: This change is shaped correctly around a fail-closed admission seam: authored evidence requirements, manifest capability declarations, runtime artifacts, and satisfaction claims remain distinct, while unverifiable semantic requirements are rejected before native import, output reservation, or simulator effects. The shared traversal now inventories unresolved affirmative capability leaves without manufacturing evidence, and backend-specific manifest construction remains within each adapter. The change touches validation, portable artifact projection, conformance reporting, and leakage boundaries without introducing a new input, authentication, execution, filesystem, or secrets surface. I found no concrete exploitable security issue introduced by the diff.

Blocking findings: 1

Finding 1 — class (3 instances)

• ID: F1
• Title: [core] Installed-wheel probes accept unrelated validation regressions
• Location: noxfile.py:616
• Decision: fix
• Rationale: Allowing exit code 3 as an unconditional success means these distribution probes pass for any validation failure, including missing packaged resources, broken task/participant joins, or another installed-wheel-only regression. Source-tree …
• Instances:
  • noxfile.py:616
  • noxfile.py:707
  • noxfile.py:812

[Brad-Edwards]

gc_test_quality_review cycle 1 of 1 — issue #84

Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch: 84-remove-hardcoded-gates
Cycle: 1 / 1
Findings: 1

Finding 1 — [critical] tests/test_claim_integrity.py::test_source_tree_contains_no_retired_static_claim_gate_identifiers

Problem: The AST sweep's retired-name detection only catches the bare _manifest_capability_evidence suffix and a small exact-name set. It misses the _gaps-suffixed variant of the same gate (nasim_manifest_capability_evidence_gaps, cyborg_manifest_capability_evidence_gaps, cyberbattlesim_manifest_capability_evidence_gaps, primaite_manifest_capability_evidence_gaps), the base module names manifest_capability_evidence_gaps and passed_probe_evidence_refs in _conformance_support.py, and the shared capability_evidence / capability_evidence_gaps in _gym_backend/conformance.py that every backend's evidence gate was actually built on.
Why it matters: This test is the repository's designated backstop against exactly the regression class this PR removes (a static probe-requirement table certifying a manifest leaf). If a future change restores nasim_manifest_capability_evidence_gaps (or the shared gym.capability_evidence_gaps) with the old table-driven logic verbatim, this test still passes — the reintroduced hardcoded gate ships silently, which is the specific failure mode the guardrails doc calls out ("Do not retain ... a static reference reintroduced through any shared path").
Fix: Replace the suffix heuristic with a substring/regex check for manifest_capability_evidence anywhere in the identifier (not just as a trailing suffix), and add capability_evidence, capability_evidence_gaps, and passed_probe_evidence_refs to _RETIRED_EXACT_IDENTIFIERS (or an equivalent pattern) so every retired name across _gym_backend/conformance.py and _conformance_support.py is covered, not just the four backend-prefixed _evidence names.

[Brad-Edwards]

gc_test_quality_review cycle 1 of 1 complete for issue #84 on branch '84-remove-hardcoded-gates'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next gc_test_quality_review invocation to count cycles.

[Brad-Edwards]

Review decision record — test-quality cycle 1 (issue #84)

Reviewer: test-quality
Cycle: 1

Architectural read:

The change is shaped correctly at the architecture level: it deletes a whole family of "manufactured evidence" functions (manifest_capability_evidence, manifest_capability_evidence_gaps, passed_probe_evidence_refs, evidence_satisfies_refs, the standard_*_capabilities factories) and replaces the claim surface with an honest, unconditional inventory (manifest_capability_gaps / _task_capture_admission_gaps) that fails closed. The test suite was updated in lockstep across all four backends plus a new repository-wide test_claim_integrity.py, and the new admission gate (_EvidenceUnverifiableFailure) is exercised end-to-end for every registered adapter via _TASKS/_BACKENDS set-containment, which is exactly the kind of extensibility guard the design doc asks for. Where this falls short is the one test explicitly built to be the last line of defense against reintroducing the retired pattern (test_source_tree_contains_no_retired_static_claim_gate_identifiers): its identifier list and suffix heuristic cover only a minority of the actual retired names, so the regression it exists to catch — a future PR quietly restoring a static evidence-gate function — would slip through undetected in most backends and in the shared _gym_backend implementation itself.

Non-blocking observation: the new early-return branch in _task_capture_admission_gaps (if not required: return ()) has no test coverage anywhere in the diff — every current task requires evidence, so only the fail-closed branch is exercised. Given the whole point of the seam is to eventually let a fully-witnessed task pass, this pass-through path deserves at least one direct unit test constructing a task with no observation/metric evidence requirements.

Blocking findings: 1

Finding 1 — class (2 instances)

• ID: F1
• Title: (no title)
• Decision: fix
• Rationale: Addressed by next cycle
• Instances:
  • tests/test_claim_integrity.py:44 (_RETIRED_EXACT_IDENTIFIERS missing passed_probe_evidence_refs, capability_evidence, capability_evidence_gaps)
  • tests/test_claim_integrity.py:170-173 (endswith("_manifest_capability_evidence") never matches *_manifest_capability_evidence_gaps in nasim/cyborg/cyberbattlesim/primaite conformance modules or manifest_capability_evidence_gaps in _conformance_support.py)

[Brad-Edwards]

Pre-PR base synchronization

• Source: refs/remotes/origin/dev at d6aea6d6a751b8e3315103aff1ec31e136c2497e
• Outcome: already_current
• Published feature head: d9ae23f19ad2b6aa0cc890f5456715ae2c8b489c
• Verified tree: bf62689019ba5c9ca2265d684d0aa66dc8e82f03