test(cyborg): reproduce the public CAGE-2 protocol and emit equivalence evidence

[Brad-Edwards]

Objective

Demonstrate the adapter with a bounded, independently rerunnable reconstruction of the public CAGE Challenge 2 evaluation protocol.

Scope

• Reconstruct the evaluation matrix from pinned primary sources: trial lengths 30/50/100; B-line, Meander, and Sleep red policies; random.seed(153); 1,000 episodes per condition; and the upstream-shipped blue baseline agents selected in the frozen run plan.
• Preserve the upstream evaluator and score calculation as source facts while also emitting RAES objectives, evidence, derived measures, and run provenance.
• Predeclare baseline agents, exclusions, failure handling, confidence-interval method, tolerances, and criteria for exact, bounded, or failed reproduction.
• Drive the same authored scenario through the RAES reference backend for the equivalence tiers it genuinely supports. Unsupported runtime facts must weaken the claim rather than be synthesized.
• Generate immutable inputs, per-run inventories, aggregate tables, validation reports, environment metadata, and a one-command reproduction path into a fresh operator-selected output.
• Compare published and observed results without treating score similarity as conformance or semantic equivalence.

Acceptance criteria

• The frozen matrix and all source identities/digests are machine-readable.
• Every scheduled run is retained as valid, invalid, failed, or excluded; retries use new IDs.
• Aggregate scores and 95% confidence intervals are recomputable from retained run evidence.
• Authored-source, contract, execution-control, state/observation, outcome/evaluation, and disclosure tiers each have a pass/fail/weakened result.
• Reference-backend support and limitations are reported separately from CybORG outcomes.
• A clean environment can reproduce the aggregate artifact from an operator-supplied immutable export.
• The report states the exact bounded claim and does not claim deterministic replay, backend identity, or scientific reproducibility beyond the evidence.
• Tests are updated for the implemented behavior and the relevant targeted/native plus canonical verification commands are actually run and pass before acceptance.

References

• REP-005 — Drive and validate CAGE-2 replication across the CybORG sim and the ACES reference backend rae#639 (REP-005)
• https://github.com/cage-challenge/cage-challenge-2
• https://arxiv.org/abs/2309.07388

---

Data ownership and capture layout

The adapter owns the apparatus and writes a complete machine-readable result to a fresh explicit output selected by the operator. Generated study data is not checked into the adapter repository, packaged in the distribution, or bound to a downstream repository or filesystem location.

The generated export contains:

• protocol.json — the frozen matrix/oracle: conditions, agents, seeds, run counts, tolerances, and exact/bounded/failed reproduction criteria, frozen before mediated runs.
• source-ledger.json — pinned upstream sources with versions and content digests, plus loss and exclusion disclosures.
• runs/ — per-run inventory with never-reused IDs, terminal state, provenance, and evidence.
• aggregates.json — aggregate measures with 95% confidence intervals, recomputable from runs/.
• tiers.json — pass/fail/weakened results for authored-source, contract, execution-control, state/observation, outcome/evaluation, and disclosure, with cited evidence.
• validation-report.md — clean-environment recomputation and the one-command validation path.
• environment.json — exact software/source/pack/study identities and stated limitations.

Research projects and archives independently own retention and publication after validating an export. Export only reviewed, non-sensitive material; keep private, restricted, credential, provider-log, and native hidden-truth data out of portable artifacts.

Research purpose

The result supplies apparatus evidence for OpenRAE/research#14 and behavioral baseline outcome-reproduction evidence for OpenRAE/research#20. It is not a proof of submitted-agent identity, state equivalence, or general OpenRAE/CybORG research equivalence.

[Brad-Edwards]

🛠️ Picked up by /implement - driver codex, branch 22-cage2-protocol-reproduction, 2026-08-09T20:06:09.296Z.

[Brad-Edwards]

Execution obligation OBL-22-ARCH-PREFLIGHT-TIMEOUT — Opened

Category: workflow
Observed state: The first asynchronous architecture-preflight attempt returned job_failed after its hard timeout.
Impact: The implementation plan cannot be posted or treated as architecture-cleared until the canonical preflight writes its required marker.
Current obligation: Obtain a successful canonical architecture preflight before finalizing the issue 22 implementation plan.

Evidence

• Architecture preflight job job-msm8gqu5-1 exceeded the tool's 1,200,000 ms limit and was terminated without a result.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #22). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

Execution obligation OBL-22-ARCH-PREFLIGHT-TIMEOUT — Resolved

Category: workflow
Observed state: The initial preflight attempt timed out; the retry completed successfully.
Impact: The workflow can now proceed to codebase assessment and planning with architecture guidance in force.
Current obligation: Obtain a successful canonical architecture preflight before finalizing the issue 22 implementation plan.

Evidence

• Retry job job-msm97cfa-2 completed successfully and wrote the required preflight phase marker.
• The preflight returned binding guardrails and identified its durable design note.

Disposition: fix
Corrective action: Retried the canonical asynchronous architecture preflight with the same issue, project, and invocation checkout after the timed-out attempt.

Verification

• gc_codex_job reported status=done with result.phase_marker.phase=preflight.
• Preflight summary identified docs/decisions/cyborg-cage2-protocol-reproduction-guardrails.md as the durable guidance.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #22). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation plan

Freeze the experiment before execution

• Add a tracked, data-only revision at packages/cyborg_adapter/reproduction/cage2-26ce1c1-ccs-sleeper-v1/; keep all implementation under src/raes_adapters/cyborg and retain the repository's single pyproject.toml / uv.lock.
• Pin the CAGE-2 commit and selected-file digests, qualified evaluator, public README leaderboard, paper, RAES/adapter/reference-backend versions, full authored pack/scenario, and the existing blue Sleep participant artifacts. Preserve the source disagreement: evaluation.py runs 100 episodes, while the published validation protocol reports 1,000.
• Select the public CCS Sleeper baseline because it is both published and representable by the reviewed adapter Sleep participant without learned-policy or fallback-training bytes. Freeze the canonical matrix as trial lengths 30/50/100 × B-line/Meander/Sleep red policies × 1,000 episodes (9,000 admitted entries).
• Record random.seed(153) as a one-time ordered-stream augmentation, not a per-episode reset or deterministic-replay claim. Keep every unbound simulator/policy stream explicit.
• Predeclare the primary total-score estimand (sum of the nine condition means), descriptive condition means, deterministic decimal arithmetic/rounding, a two-sided normal-approximation 95% interval over independent condition means, eligibility and missing-data handling, no multiplicity correction for the single primary estimand, and exact/bounded/failed rules. Exact requires source-precision agreement; bounded requires the published and observed estimates to fall within each other's declared 95% bounds; otherwise the outcome comparison fails. Score agreement cannot upgrade any equivalence tier.

Implement the backend-local study and bundle paths

• Add src/raes_adapters/cyborg/reproduction.py to load bounded strict JSON, validate canonical RAES experiment/study/participant/admitted-plan models and digest joins, schedule deterministic sealed plan partitions with plan_batch_schedule(), and compose the existing CybORG target, participant lifecycle, evaluator, evidence, cleanup, provenance, and atomic-write helpers.
• Extend src/raes_adapters/cli.py with two closed surfaces: a full native/reference study command that writes only to a fresh mode-0700 staging root, and an offline reproduce command that validates a frozen public bundle and recomputes its aggregate in a fresh output root without CybORG or network access.
• Keep serial execution unless the sealed plan and live allocator evidence prove isolation. Do not resume, append, overwrite, allocate identities from time/PID, or let scheduling select conditions, identities, exclusions, randomness, or scores.
• Retain every admitted entry as valid, invalid, failed, or excluded. Preserve failed attempts and cleanup evidence; retries get a new execution-attempt identity and never replace the original run.
• Run the identical full authored scenario through CybORG and the RAES reference backend as separate apparatus/results. Unsupported reference policies, observations, reward facts, or temporal behavior produce cited weakened tiers, never synthetic facts.
• Gate scientific execution on the exact reviewed Scenario2 artifact from adapters docs(cyborg): author the validated CAGE-2 (Scenario2) SDL document #77 and immutable participant/source digests. The current two-step smoke pack must never be substituted. A missing prerequisite produces explicit exclusions/weakened evidence rather than simulated results.

Produce and verify the research bundle

• Materialize protocol.json, source-ledger.json, partitioned sealed plans, per-run evidence/inventories, aggregates.json, tiers.json, environment.json, validation-report.md, and a final relative integrity inventory. Keep published leaderboard values as separate baseline-result study members.
• Give authored-source, contract, execution-control, state/observation, outcome/evaluation, and disclosure independent pass/fail/weakened decisions with evidence references and limitations; never publish native objects, action ids, reward vectors, hidden truth, raw logs, environment/argv dumps, credentials, host paths, symlinks/special files, or tracebacks.
• Update tests/test_cyborg_reproduction.py and tests/test_researcher_cli.py for a bounded injected-driver matrix, strict/duplicate/non-finite/oversized JSON, digest/cardinality/terminal-state mismatches, all four dispositions, retry identity, cleanup/sealing failures, deterministic aggregate regeneration, tier separation, reference-backend limitations, output confinement, and leakage sentinels. The 9,000-run native study remains retained research evidence, not a PR/CI workload.
• Update the researcher guide, README/index/navigation, architecture note, and clean-install distribution probe so an installed wheel can recompute a copied checkout bundle without packaging packages/.
• Run targeted tests, then the canonical graph: uv tool run --from 'nox[uv]==2026.4.10' nox -f noxfile.py -s verify (the Makefile/policy path supplies the legitimate requirement-free skip for issue 22). Do not edit CHANGELOG.md or add a changelog fragment.

Research contribution and dependency boundary

• OpenRAE/research#14 receives exact apparatus/source/pack/participant identities, supported/unsupported control evidence, leakage and lifecycle probes, a clean short validation path, and a retained readiness/claim matrix. adapters release: publish and smoke-test the first usable CybORG adapter release #24 remains the owner of public-index release qualification; this bundle supplies evidence but does not claim that release gate.
• OpenRAE/research#20 receives the frozen public protocol/schedules, machine-readable runs and tables, deterministic derivation, claim/evidence mapping, limitations, publication screening, exact identities, and integrity manifest needed for the paper/artifact.
• adapters docs(cyborg): author the validated CAGE-2 (Scenario2) SDL document #77 is the claim-strength prerequisite for a real full-Scenario2 run. Until its reviewed SDL is available, implementation can build and test the apparatus, but the scientific study must not execute against or relabel the smoke pack.

[Brad-Edwards]

Execution obligation OBL-22-FULL-SCENARIO2-PREREQUISITE — Opened

Category: quality
Observed state: Issue 22 is planned, but its exact authored Scenario2 input is not present in the checkout.
Impact: Executing the 9,000-run protocol against the current two-step smoke pack would not reconstruct the public CAGE-2 experiment and would contaminate the evidence intended for OpenRAE/research#14 and #20.
Current obligation: Obtain and bind a reviewed full Scenario2 SDL/pack before scientific execution, or explicitly authorize absorbing the separately owned Scenario2 authoring scope into this implementation.

Evidence

• docs(cyborg): author the validated CAGE-2 (Scenario2) SDL document #77 is open and owns authoring the validated full CAGE-2 Scenario2 SDL.
• src/raes_adapters/cyborg/examples/cage2-research/docs/golden-readiness-checklist.md states that the bounded smoke SDL must be replaced before any full-CAGE or replication claim.
• The architecture preflight identifies the full authored Scenario2 pack as a claim-strength blocker.

[Brad-Edwards]

Execution obligation OBL-22-FULL-SCENARIO2-PREREQUISITE — Escalated

Category: quality
Observed state: The implementation plan is complete, but scientific execution is blocked on the missing full authored scenario.
Impact: Proceeding without this decision risks either invalid scientific evidence or an unexpectedly material expansion into separately owned scenario-authoring work.
Current obligation: Obtain and bind a reviewed full Scenario2 SDL/pack before scientific execution, or explicitly authorize absorbing the separately owned Scenario2 authoring scope into this implementation.

Evidence

• docs(cyborg): author the validated CAGE-2 (Scenario2) SDL document #77 is open and owns authoring the validated full CAGE-2 Scenario2 SDL.
• The current cage2-research pack is explicitly limited to a two-step smoke scenario.
• Issue 22's public research bundle must not relabel the smoke pack as Scenario2.

Pause class: unexpected_material_scope_expansion
Decision request: Should this issue 22 branch absorb and complete the full Scenario2 authoring work from #77 before implementing/running the reproduction, or should issue 22 remain paused until #77 lands through its own reviewed change?

This obligation remains open while the decision is pending.