Determinism And Stability Verification (ASR-514)

[Brad-Edwards]

ASR-514 | NON_FUNCTIONAL | SHOULD | Wave 3 | DRAFT

Statement

The ecosystem shall support verification of determinism, stability, or replay-consistency where repeatability claims are made.

Status

Independent validation technique. Follows the layered-validation framework in #97. CI branch UID: ASR-514.

Requirements

• ASR-514

---

Tracks Ground Control requirement ASR-514

[Brad-Edwards]

🛠️ Picked up by /implement - driver claude, branch 262-determinism-stability-verification, 2026-07-29T00:33:50.196Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #262). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #262). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation Plan — ASR-514 Determinism / Stability / Replay-Consistency Verification

Requirement (ASR-514, NON_FUNCTIONAL, SHOULD, Wave 3, DRAFT): "The ecosystem shall support verification of determinism, stability, or replay-consistency where repeatability claims are made."

Next technique in the layered-validation framework (#97), after ASR-512 (bounded behavioral probes) and ASR-513 (bounded counterfactual necessity). Follows the binding architecture preflight in docs/decisions/issue-262-asr-514-determinism-stability-verification-preflight.md.

Design boundary (per preflight)

The preflight's central decision: repeatability claims bind existing behavioral-relations, not a new one. Determinism/replay/stability already have canonical relations — canonical-artifact-identity (exact determinism / exact replay), participant-projected-history-equivalence (participant-history replay), statistical-similarity/statistical-equivalence (stability), trace-equivalence (strict equivalence). So:

• No new catalog relation, no taxonomy_revision bump, no schema/schema-publication change. The catalog stays at rev4; validate_behavioral_claim_binding() carrier/revision checks are not weakened. Relation carriers stay carriers — the projection identifier lives in its own observation_projection_ref field, never in a carrier.
• The one genuine gap is the executable cross-run join that has no portable home: given a producer's explicit repeatability claim over a cohort of ≥2 distinct admitted runs of one held-fixed subject, decide whether every repetition's projected outcome is identical under the bound relation's criterion — with ADR-068 distinct-run-identity, common lineage, controlled-variation, reset/cleanup, and typed run/evidence-join admission around the comparison. That join is a small frozen internal seam in raes_conformance, reusing the ASR-513 necessity_evidence discipline (digest-pinned validator authority, assembler-owned facts, stable diagnostics, sealed internal result) but none of its causal concepts. canonical-artifact-identity is binary, so a cohort >2 is decided as baseline-to-repetition comparisons, never reinterpreted as an n-ary relation.

Bounded scope of the first criterion: the shipped comparator implements exact canonical projected-outcome equality (canonical-projection-equality v1.0.0) bound to canonical-artifact-identity. Statistical stability (needs ExperimentStudyModel + statistical-*) and participant-history replay are declared future criteria and fail closed as unsupported, exactly as ASR-513 shipped one binary-but-for criterion and scoped others out.

New code (implementations/python/packages/raes_conformance/, each < 500 lines)

1. verification_authority.py — neutral shared primitives promoted (not copied) out of necessity_types.py: VerificationBinding, VerificationDisposition, VerificationRecordIdentity, digest/text validators. Honors the preflight's "promote neutral primitives once rather than copy" directive.
2. repeatability_types.py — ProjectionOutcome, per-repetition RepetitionProjection (decided → sha256 projected digest + run-bound evidence), and the sealed RepeatabilityConsistencyEvidence (module-owned _ASSEMBLY_TOKEN; direct construction raises).
3. repeatability_validation.py — RepetitionRef, VariationPolicy, RepeatabilityConsistencyCase (relation must be a supported repeatability relation; cohort of ≥2 distinct runs holding the subject fixed; canonical digest), RepeatabilityConsistencyOutcome (CONSISTENT/DIVERGENT/INCONCLUSIVE/UNSUPPORTED), compare_repeatability_consistency(), stable-coded diagnostics.
4. repeatability_evidence.py — trusted assembler assemble_repeatability_consistency_evidence(), caller-supplied variation/reset/cleanup records (run-bound refs only, no caller disposition/digest), a RepeatabilityEvidenceValidator Protocol, and the run cohort. Revalidates every run against its task, pins the validator to the case authority, resolves each projection through the exact run traceability, enforces distinct run identities + common family/lineage, then seals the evidence.

Decision rule (fail-closed): UNSUPPORTED for unauthenticated/unsealed evidence, case↔evidence digest mismatch, unsupported relation, stale catalog coordinate, unsupported criterion, missing capability, cohort/authority mismatch, or any UNSUPPORTED gate/outcome → INCONCLUSIVE for unverified variation/reset/cleanup, residual state, or indeterminate outcome → DIVERGENT when any repetition's projected digest differs from the cohort baseline (refuted, not an execution failure) → else CONSISTENT.

Modified code

• necessity_types.py — imports the promoted primitives from verification_authority, preserving __all__ so ASR-513 (necessity_evidence/necessity_validation + the 36-test suite) is byte-for-byte unchanged.

Docs

• docs/research/behavioral-validation/traceability-matrix-asr-514.md — clause matrix mirroring traceability-matrix-asr-513.md.
• specs/formal/behavioral-relations/README.md — new "### Determinism, stability, and replay-consistency verification" claim-surface subsection binding the existing relation; positive prose is relation-bound / boundary-bound, all equivalence terms stated as explicit nonclaims, so tools/check_behavioral_relation_claims.py passes. No relation-table row / no revision bump.
• Preflight note already in the tree.

Tests (test_repeatability_validation.py, TDD — 63 tests, 100% coverage of the new modules)

Positive equal-cohort → CONSISTENT; larger cohort; divergent digests → DIVERGENT (not failure); sealed-evidence direct construction / unsealed copy / authority / cohort tamper; every fail-closed admission and gate path (relation, stale catalog coordinate, criterion, capability, carriers+projection, non-finite boundary, distinct-run + held-fixed-subject cohort guards, assembler run/task/authority/projection-evidence/verification-evidence joins, variation/reset/cleanup dispositions and residual state, indeterminate/unsupported outcomes); stable diagnostics that never echo untrusted values; a claim-construction test proving the claim validates against canonical-artifact-identity.

Verification & plan-rule compliance

Before completion I run implementations/python/.venv/bin/python tools/check_repo_policy.py, tools/check_requirement_governance.py, and tools/verify_all.py (via nox -s verify) with RAES_REQUIREMENT_UID=ASR-514. No new authority-bearing artifacts outside specs//contracts//docs//implementations/; no published-schema change → no manifest ledger entry; no CHANGELOG/version edits (PR title will be feat: …, title-guard compliant). IMPLEMENTS/TESTS traceability for ASR-514 reconciled in Ground Control at Step 16.

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #262 (branch 262-determinism-stability-verification)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: don't-ship

The package placement and promotion of neutral verification-authority primitives fit the intended layering, but the load-bearing comparison seam is not shaped correctly. It represents an N-run cohort with one binary relation whose carriers identify the held-fixed subject rather than the compared artifacts, accepts lineage and canonical-input coordinates without resolving them against typed authorities, and bypasses the incumbent canonicalization path for identities that seal evidence. Those choices can produce a result that claims canonical-artifact identity without binding the actual artifact pair or exact execution lineage, and they would harden the wrong contract before replay or additional comparison criteria are added.

Blocking findings (4):

1. [class] Architecture-fit: The binary relation is bound to the input subject, not the compared artifacts — implementations/python/packages/raes_conformance/repeatability_validation.py:285
canonical-artifact-identity relates two named canonical artifacts, but admission requires both carriers to equal case.subject_ref; the actual compared artifacts exist only as unlabelled projected digests. The returned claim therefore asserts subject-to-itself identity while the comparator separately evaluates a cohort. For cohorts larger than two, the single binding also cannot preserve the individual baseline-to-repetition binary comparisons. Model explicit baseline-to-repetition bindings whose left and right carriers identify the projected artifacts, and retain those pair identities in the result and evidence.
2. [class] Architecture-fit: Common lineage and canonical input identity are only caller assertions — implementations/python/packages/raes_conformance/repeatability_validation.py:151
   The case proves common family and baseline lineage only by comparing caller-provided strings, while input_digest receives only syntactic SHA-256 validation. The assembler resolves run, task, and scenario-snapshot identities but never derives or checks these three coordinates against their typed authorities. A caller can therefore copy the same family/lineage labels and arbitrary input digest onto unrelated admitted runs and obtain a consistent result. Derive these coordinates from the task/run/study/traceability authorities during assembly, or remove redundant caller fields, and add a negative test using unrelated typed lineage with matching labels.
3. [class] Architecture-fit: Evidence-sealing identities bypass the canonical JCS path — implementations/python/packages/raes_conformance/repeatability_validation.py:183
   Both the case digest and verification-record digest introduce local json.dumps plus SHA-256 canonicalizers instead of the repository's RFC 8785/JCS authority. These digests are evidence identities, not local cache keys; for example, ensure_ascii=True produces different canonical bytes from JCS for non-ASCII values. Reuse or expose the neutral canonical JSON digest helper for both sites and pin it with canonicalization vectors.
4. [one-off] Code-quality: Repeatability diagnostics violate the JSON Pointer address contract — implementations/python/packages/raes_conformance/repeatability_validation.py:219
   The shared diagnostic factory emits a dotted address containing case_id, but the closed diagnostic boundary requires a JSON Pointer and a stable, non-caller-derived address. Failure results can therefore fail downstream DiagnosticModel normalization. Use a fixed or correctly pointer-escaped repeatability address and add a model-conversion assertion.

Security review

Verdict: don't-ship

The change is shaped around the right seam: a criterion-specific conformance comparator consumes admitted runs and evidence without adding execution, persistence, or transport concerns. It also centralizes producer/validator identity appropriately for future comparison criteria. However, evidence authenticity and run-to-evidence integrity are load-bearing security boundaries here, and the implementation currently represents both with caller-forgeable Python objects and mutable identifier joins. Those flaws allow a false consistency result without passing the advertised trusted assembly path.

Blocking findings (2):

1. [one-off] Importable private factory bypasses the trusted evidence assembler — implementations/python/packages/raes_conformance/repeatability_types.py:87
   Attacker model: a package consumer or extension can call the public comparison API but is not trusted to assert verification outcomes. Python underscores provide no access control: the caller can import _RepeatabilityConsistencyEvidenceParts and _new_repeatability_consistency_evidence(), which stamps the module's authentic assembly token without invoking assemble_repeatability_consistency_evidence(). The caller can therefore supply the case digest and publicly known authority identity, mark variation/reset/cleanup as verified, clear residual state, and provide equal projections; compare_repeatability_consistency() will accept the value as authentically assembled and return consistent. The module token is also directly importable, as the new test does. Do not use an importable object identity as evidence provenance. Resolve an opaque record through a trusted store/registry or verify a signature/MAC rooted outside caller-controlled Python objects; otherwise keep this entire API behind a trusted composition boundary and stop treating the token as a security control.
2. [class] Evidence joins authenticate identifiers but not evidence content — implementations/python/packages/raes_conformance/repeatability_evidence.py:224
   Attacker model: a caller controls the evidence_records supplied to the assembler but not the admitted run or trusted validator. _run_evidence_refs() reduces traceability references to only (ref_id, ref_version), and _record_matches_run() accepts any model carrying those coordinates and the expected run reference. The caller can substitute a different valid evidence record with the same ID/version/run coordinates; both projection and verification joins accept it, and the trusted validator then derives outcomes and gate dispositions from substituted content. This permits forged repeatability conclusions despite the run itself being digest-bound. Preserve and verify the owning reference's content digest, or resolve the immutable evidence record from trusted storage and compare its canonical digest before exposing it to the validator. Claims requiring this strength should fail closed when no integrity binding exists.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #262 on branch '262-determinism-stability-verification'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #262)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The package placement and promotion of neutral verification-authority primitives fit the intended layering, but the load-bearing comparison seam is not shaped correctly. It represents an N-run cohort with one binary relation whose carriers identify the held-fixed subject rather than the compared artifacts, accepts lineage and canonical-input coordinates without resolving them against typed authorities, and bypasses the incumbent canonicalization path for identities that seal evidence. Those choices can produce a result that claims canonical-artifact identity without binding the actual artifact pair or exact execution lineage, and they would harden the wrong contract before replay or additional comparison criteria are added.

Security reviewer: The change is shaped around the right seam: a criterion-specific conformance comparator consumes admitted runs and evidence without adding execution, persistence, or transport concerns. It also centralizes producer/validator identity appropriately for future comparison criteria. However, evidence authenticity and run-to-evidence integrity are load-bearing security boundaries here, and the implementation currently represents both with caller-forgeable Python objects and mutable identifier joins. Those flaws allow a false consistency result without passing the advertised trusted assembly path.

Blocking findings: 6

Finding 1 — class (4 instances)

• ID: F1
• Title: [core] Architecture-fit: The binary relation is bound to the input subject, not the compared artifacts
• Location: implementations/python/packages/raes_conformance/repeatability_validation.py:285
• Decision: fix
• Rationale: canonical-artifact-identity relates two named canonical artifacts, but admission requires both carriers to equal case.subject_ref; the actual compared artifacts exist only as unlabelled projected digests. The returned claim therefore a…
• Instances:
  • implementations/python/packages/raes_conformance/repeatability_validation.py:107
  • implementations/python/packages/raes_conformance/repeatability_validation.py:285
  • implementations/python/packages/raes_conformance/repeatability_validation.py:286
  • implementations/python/packages/raes_conformance/repeatability_validation.py:450

Finding 2 — class (6 instances)

• ID: F2
• Title: [core] Architecture-fit: Common lineage and canonical input identity are only caller assertions
• Location: implementations/python/packages/raes_conformance/repeatability_validation.py:151
• Decision: fix
• Rationale: The case proves common family and baseline lineage only by comparing caller-provided strings, while input_digest receives only syntactic SHA-256 validation. The assembler resolves run, task, and scenario-snapshot identities but never der…
• Instances:
  • implementations/python/packages/raes_conformance/repeatability_validation.py:68
  • implementations/python/packages/raes_conformance/repeatability_validation.py:69
  • implementations/python/packages/raes_conformance/repeatability_validation.py:117
  • implementations/python/packages/raes_conformance/repeatability_validation.py:139
  • implementations/python/packages/raes_conformance/repeatability_validation.py:151
  • implementations/python/packages/raes_conformance/repeatability_validation.py:153

Finding 3 — class (2 instances)

• ID: F3
• Title: [core] Architecture-fit: Evidence-sealing identities bypass the canonical JCS path
• Location: implementations/python/packages/raes_conformance/repeatability_validation.py:183
• Decision: fix
• Rationale: Both the case digest and verification-record digest introduce local json.dumps plus SHA-256 canonicalizers instead of the repository's RFC 8785/JCS authority. These digests are evidence identities, not local cache keys; for example, `ens…
• Instances:
  • implementations/python/packages/raes_conformance/repeatability_validation.py:183
  • implementations/python/packages/raes_conformance/repeatability_evidence.py:286

Finding 4 — one-off

• ID: F4
• Title: [core] Code-quality: Repeatability diagnostics violate the JSON Pointer address contract
• Location: implementations/python/packages/raes_conformance/repeatability_validation.py:219
• Decision: fix
• Rationale: The shared diagnostic factory emits a dotted address containing case_id, but the closed diagnostic boundary requires a JSON Pointer and a stable, non-caller-derived address. Failure results can therefore fail downstream DiagnosticModel…

Finding 5 — one-off

• ID: F5
• Title: [security] Importable private factory bypasses the trusted evidence assembler
• Location: implementations/python/packages/raes_conformance/repeatability_types.py:87
• Decision: fix
• Rationale: Attacker model: a package consumer or extension can call the public comparison API but is not trusted to assert verification outcomes. Python underscores provide no access control: the caller can import `_RepeatabilityConsistencyEvidencePa…

Finding 6 — class (4 instances)

• ID: F6
• Title: [security] Evidence joins authenticate identifiers but not evidence content
• Location: implementations/python/packages/raes_conformance/repeatability_evidence.py:224
• Decision: fix
• Rationale: Attacker model: a caller controls the evidence_records supplied to the assembler but not the admitted run or trusted validator. _run_evidence_refs() reduces traceability references to only (ref_id, ref_version), and `record_matches…
• Instances:
  • implementations/python/packages/raes_conformance/repeatability_evidence.py:220
  • implementations/python/packages/raes_conformance/repeatability_evidence.py:224
  • implementations/python/packages/raes_conformance/repeatability_evidence.py:233
  • implementations/python/packages/raes_conformance/repeatability_evidence.py:247