Define source-neutral SDL candidate synthesis and admission

[Brad-Edwards]

Problem

External sources may contain concepts, relationships, preconditions, orderings, and examples that are useful authoring inputs. Turning those inputs directly into executable scenario claims would confuse source assertions with RAES semantics and bypass ordinary authoring, validation, and admission.

Current ownership boundary

RAES owns the source-neutral candidate artifact, transformation provenance, semantic-loss representation, and validation/admission contracts.

• env-packs and source adapters own source retrieval, pack-aware orchestration, and scheme-specific authoring pipelines.
• Hub owns user journeys and demonstrations that make the capability discoverable; see Hub #25.
• No source-specific syntax, catalog client, or remote retrieval behavior belongs in the SDL core.

Outcome

Define a source-neutral candidate-synthesis pipeline that consumes governed, versioned external knowledge assertions and emits ordinary SDL candidate artifacts. Candidates must carry complete transformation provenance, expose unresolved choices and semantic loss, and pass the same parsing, composition, policy, and admission gates as manually authored content.

This work builds on the portable binding contract in #986.

Required pipeline contract

• Pin the source authority, revision or content digest, extraction query, and transformation version.
• Separate source facts, transformation assumptions, author decisions, and generated native constructs.
• Represent ambiguous relationships, missing preconditions, unsupported ordering, and unresolved parameterization explicitly.
• Emit stable candidate identities and a reproducible derivation record.
• Require human or governed policy decisions wherever the transformation cannot preserve meaning.
• Treat generated output as a candidate, never as admitted or executable merely because it came from an external authority.
• Preserve information-flow and provenance constraints through generation and later inspection.

Acceptance criteria

• One RAES contract handles candidate inputs from at least two unrelated external schemes or assertion formats.
• Positive fixtures reproduce byte-stable or semantically canonical candidates from pinned inputs.
• Negative fixtures cover ambiguous ordering, missing native semantics, unsupported relations, stale inputs, and non-reproducible transformations.
• Generated candidates pass the normal SDL validation and admission path with no privileged bypass.
• Reviewers can distinguish imported assertions, inferred structure, defaults, and explicit author decisions.
• Re-running with a changed source or transformation version produces an inspectable semantic diff.
• A source adapter can consume the contract without RAES knowing its retrieval protocol or authoring interface.

Non-goals

• Promising automatic semantic equivalence with an external source.
• Making an external assertion executable.
• Adding a source-specific authoring language, catalog client, or pack workflow to RAES.

[Brad-Edwards]

🛠️ Picked up by /implement - driver codex, branch 988-source-neutral-candidate-synthesis, 2026-08-12T15:03:26.819Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #988). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #988). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Implementation Plan

Issue #988 is requirement-free; its acceptance criteria and non-goals are the binding contract. This feature adds no repository/process structural gate and therefore does not require a new Ground Control requirement.

1. Drive the source-neutral contract and pipeline with failing tests first.

   • Add implementations/python/tests/test_candidate_synthesis.py and fixture sets for two unrelated scheme/format identities.
   • Cover deterministic successful synthesis, strict JSON/schema admission, byte/canonical stability, normal SDL parse/instantiate admission, explicit provenance origins, and semantic comparison.
   • Add negative fixtures/tests for ambiguous ordering, missing native semantics, unsupported relations, stale source pins, and unrecognized/non-reproducible transformation profiles; every material unresolved case must return no candidate.

2. Publish closed portable synthesis contracts.

   • Add input and record models under implementations/python/packages/raes_contracts/contracts/ with a discriminated source-assertion union, exact source/query/adapter/transformation/policy pins, typed assumptions/choices/decisions/construct traces, stable sorted identities, and all-or-none result invariants.
   • Compose ArtifactTransformationReportModel; keep imported assertions, inference, defaults, author decisions, generated constructs, and loss distinct.
   • Register versions/exports/schema bundle/conformance validators and publish hand-governed schemas, fixtures, and schema-publication ledger entries under contracts/.

3. Implement one trusted, source-neutral synthesis seam.

   • Add a pure raes candidate-synthesis module with fixed transformation-profile dispatch. It consumes supplied normalized assertions only—no retrieval, dynamic imports/plugins, environment selection, subprocesses, persistence, or scheme-specific branches.
   • Extend the canonical SDL formatting seam with deterministic model-to-sdl-yaml/v1 rendering, then reparse emitted bytes through production parse_sdl() with semantic validation. Successful results expose exact-byte and canonical SDL digests; refusals expose typed bounded diagnostics and no candidate.
   • Stable candidate identities derive from explicit governed author decisions, never source ordering or guessed semantics.

4. Integrate documentation and comparison/admission evidence.

   • Document the contract and ownership boundary in specs/, retaining the architecture preflight note.
   • Demonstrate changed source/transformation derivations separately from SDL semantic changes by using the existing semantic-comparison profile, request builder, owner adapter, and analyzer.
   • Demonstrate generated candidates follow ordinary parse, instantiation, and instantiated-admission paths with no generated-content bypass.

5. Verify the final tree.

   • Run focused candidate-synthesis tests during red/green/refactor.
   • Run the required final repository checks with RAES_REQUIREMENT_UID set for this issue branch: implementations/python/.venv/bin/python tools/check_repo_policy.py, tools/check_requirement_governance.py, and tools/verify_all.py; Ground Control will also run the configured completion and policy gates.

Whole-repository design checks

• Security: bounded duplicate-rejecting JSON, closed unions, safe inert locators, exact digest joins, fixed trusted profile dispatch, production YAML parser limits/semantic validator, sanitized bounded diagnostics, and no secrets/network/host/process side effects.
• Maintainability: reuse ContractModel, shared digest/scalar types, URI safety, transformation reports, canonical digest helpers, SDL formatter/parser, diagnostic models, schema generation/publication, conformance registries, and existing semantic comparison.
• Extensibility: schemes adapt into the same neutral assertion family; a new assertion meaning adds a closed union variant and governed rule, while new target/loss behavior adds a transformation-profile revision rather than a scheme branch.
• Whole-repo surfaces: raes_contracts, raes transformations/formatting/parser/admission, conformance, published schemas/fixtures/ledger, specs, schema generation, nox, repository policy, and semantic-comparison ownership are all included.

[Brad-Edwards]

Execution obligation ISSUE-988-REQUIREMENT-CONTEXT — Opened

Category: workflow
Observed state: The issue branch contains no requirement UID, and mechanical verification cannot export a UID for a requirement-free issue.
Impact: Requirement governance blocks the mandatory policy gate for governed contracts, specs, docs, and implementation paths.
Current obligation: Provide the repository-required DSL-114 context to mechanical verification without broadening issue #988's Ground Control requirement scope.

Evidence

• Mechanical verify job job-msq96oo4-3 passed repository policy and failed requirement governance.
• DSL-114 is ACTIVE and owns the author-facing external knowledge reference and mapping surface extended by issue Define source-neutral SDL candidate synthesis and admission #988.
• Issue Define portable external concept-scheme references and binding assertions #986 documents the accepted branch-UID fallback for the same requirement-free mechanical-verification edge case.

[Brad-Edwards]

Execution obligation ISSUE-988-REQUIREMENT-CONTEXT — Resolved

Category: workflow
Observed state: The issue branch originally contained no requirement UID, so requirement governance blocked mechanical policy verification.
Impact: The mandatory policy gate could not complete until the repository's requirement-context fallback was satisfied.
Current obligation: Provide the repository-required DSL-114 context to mechanical verification without broadening issue #988's Ground Control requirement scope.

Evidence

• The repository-controlled branch preparation boundary selected 988-dsl-114-candidate-synthesis.
• Mechanical verify job job-msq9c5ma-4 passed completion and policy on the renamed branch.

Disposition: fix
Corrective action: Moved the uncommitted issue work to the compliant branch 988-dsl-114-candidate-synthesis so branch-context detection supplies DSL-114 while the issue remains requirement-free.

Verification

• gc_implement_mechanical verify returned ok=true and phase=verification_complete.
• Completion passed in 177477 ms and policy passed in 14895 ms.

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #988 (branch 988-dsl-114-candidate-synthesis)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: don't-ship

The high-level shape is sound: portable contracts remain in raes_contracts, SDL-aware synthesis stays in raes, the implementation composes the incumbent transformation report, and emitted bytes re-enter the production parser. The design seam is the governed transformation profile, which should support the next source or profile without source-specific branches. However, the two authority-bearing mechanisms at that seam—the profile digest and the portable provenance record—do not actually close their claimed content and reference bindings, so the implementation is not yet production-ready.

Blocking findings (2):

1. [one-off] Transformation profile digest does not bind the transformation — implementations/python/packages/raes/candidate_synthesis.py:37
   SYNTHESIS_PROFILE_DIGEST hashes only the public profile id and version. It commits to neither a governed profile artifact nor the implemented rules, rendering behavior, or output-affecting limits. Consequently, transformation behavior can change while the same supposedly reproducible profile coordinate remains accepted and the derivation record continues reporting the old digest. Publish or otherwise define a canonical profile artifact containing every semantic/output-affecting parameter, derive the registered digest from that artifact, and test that the fixture coordinate matches it.
2. [class] Portable synthesis records admit unclosed provenance references — implementations/python/packages/raes_contracts/contracts/candidate_synthesis.py:383
   The record validates input_digest only by comparing it with the transformation report's co-carried source_digest; it cannot recompute or pair-validate that digest because the record omits input_version, target, and the transformation-profile coordinate. Construct traces likewise receive only sorted/unique target checks, while contribution ref_id values are never resolved against source assertions, decisions, assumptions, defaults, or registered profile rules. A standalone record can therefore pass conformance while attributing a candidate to unrelated input or dangling provenance. Close these joins at the contract boundary—either carry enough normalized input to recompute the digest or add a canonical input/record admission validator—and validate each contribution kind against its owning collection/profile.

Security review

Verdict: ship

This is shaped correctly: the change introduces a bounded, closed candidate-synthesis contract, keeps trusted dispatch behind an exact fixed transformation-profile coordinate, renders only ordinary SDL, and re-admits emitted bytes through the production parser and semantic-validation path. The main cross-cutting seams—source pinning, provenance, information-flow references, deterministic digests, transformation reporting, and later admission—remain distinct; the synthesis record does not itself become an authentication or authorization decision. The closed assertion union and governed profile boundary accommodate the obvious next variations without introducing dynamic imports, ambient retrieval, or a validation bypass.

No blocking findings.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #988 on branch '988-dsl-114-candidate-synthesis'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #988)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The high-level shape is sound: portable contracts remain in raes_contracts, SDL-aware synthesis stays in raes, the implementation composes the incumbent transformation report, and emitted bytes re-enter the production parser. The design seam is the governed transformation profile, which should support the next source or profile without source-specific branches. However, the two authority-bearing mechanisms at that seam—the profile digest and the portable provenance record—do not actually close their claimed content and reference bindings, so the implementation is not yet production-ready.

Security reviewer: This is shaped correctly: the change introduces a bounded, closed candidate-synthesis contract, keeps trusted dispatch behind an exact fixed transformation-profile coordinate, renders only ordinary SDL, and re-admits emitted bytes through the production parser and semantic-validation path. The main cross-cutting seams—source pinning, provenance, information-flow references, deterministic digests, transformation reporting, and later admission—remain distinct; the synthesis record does not itself become an authentication or authorization decision. The closed assertion union and governed profile boundary accommodate the obvious next variations without introducing dynamic imports, ambient retrieval, or a validation bypass.

Blocking findings: 2

Finding 1 — one-off

• ID: F1
• Title: [core] Transformation profile digest does not bind the transformation
• Location: implementations/python/packages/raes/candidate_synthesis.py:37
• Decision: fix
• Rationale: SYNTHESIS_PROFILE_DIGEST hashes only the public profile id and version. It commits to neither a governed profile artifact nor the implemented rules, rendering behavior, or output-affecting limits. Consequently, transformation behavior can …

Finding 2 — class (2 instances)

• ID: F2
• Title: [core] Portable synthesis records admit unclosed provenance references
• Location: implementations/python/packages/raes_contracts/contracts/candidate_synthesis.py:383
• Decision: fix
• Rationale: The record validates input_digest only by comparing it with the transformation report's co-carried source_digest; it cannot recompute or pair-validate that digest because the record omits input_version, target, and the transformation-profi…
• Instances:
  • implementations/python/packages/raes_contracts/contracts/candidate_synthesis.py:383
  • implementations/python/packages/raes_contracts/contracts/candidate_synthesis.py:409

[Brad-Edwards]

gc_test_quality_review cycle 1 of 1 — issue #988

Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch: 988-dsl-114-candidate-synthesis
Cycle: 1 / 1
Findings: 1

Finding 1 — [critical] implementations/python/tests/test_candidate_synthesis.py::test_incumbent_transformation_report_can_express_synthesis_semantic_loss

Problem: The test only asserts ArtifactTransformationLossKind.SEMANTIC_APPROXIMATION.value == "semantic-approximation" and the same for SEMANTIC_OMISSION — i.e. it restates the enum's own literal definition. It never constructs an ArtifactTransformationPreservationModel/loss entry with either kind, never validates one against the published artifact-transformation-report-v1 schema (which this PR also changed), and never drives any candidate_synthesis.py code path that actually produces such a loss — grepping the diff shows no production code anywhere emits SEMANTIC_APPROXIMATION or SEMANTIC_OMISSION; synthesize_sdl_candidate's only preservation outcomes are VERIFIED and NOT_APPLICABLE.
Why it matters: If the enum/schema pair drifts (e.g. schema regains only declaration-removed, or the model's ArtifactTransformationLossKind were renamed/removed and re-added with a typo'd value), or if the feature this enum exists to support is never actually wired up, this test still passes — it can only fail if someone edits the enum literal itself, which the same commit would already do. It provides no real regression protection for the schema change it accompanies.
Fix: Either wire the new loss kinds into an actual code path (e.g. have candidate synthesis emit a SEMANTIC_APPROXIMATION/SEMANTIC_OMISSION loss for a fixture that requires it) and assert on that produced record/report, or at minimum build an ArtifactTransformationPreservationModel instance using each new kind and validate it round-trips through the published artifact-transformation-report-v1 schema (Draft202012Validator(...).validate(...) plus validate_contract_payload), matching the pattern already used in test_profile_digest_binds_the_complete_published_transformation_artifact.