runtime: no affordance to inject a generated/rendered secret as a node environment value (env_file)

[Brad-Edwards]

Gap

A node often needs a runtime-generated secret as an environment value — created after another service boots, not authorable in the SDL. The motivating case: TheHive must receive Cortex's API key, which Cortex only mints once its org/user is bootstrapped at runtime; and TheHive's HTTPS keystore password is a generated artifact.

RAES today can declare:

• runtime.environment[] with value_classification: operator_secret — resolved from the operator environment, but the value must already exist out-of-band; there is no way to source it from a generated_artifact produced during realization.
• generated_artifacts with file outputs mounted at a mount_destination — file delivery only, no environment/env_file consumption.

There is no affordance for "consume generated-artifact output X as environment variable Y (or an env_file) on node Z."

Requested

An SDL affordance letting a node consume a generated-artifact output as an environment value / env_file — e.g. a generated_artifacts[].consumers[] env binding, or a runtime.environment[].value_from: {generated_artifact, output}.

Impact / current workaround

APTL (TechVault env-pack) declares TheHive's Cortex wiring (--cortex-hostnames/proto/port) so both services boot and connect, but the API key injection is unwired and documented in-SDL as a tracked refinement. Once this affordance lands, the key/keystore-password become declared generated-artifact env consumers instead of a post-boot manual step.

Requirements

• DSL-435

[Brad-Edwards]

🛠️ Picked up by /implement - driver claude, branch 1074-generated-artifact-env-consumers, 2026-08-04T04:47:52.347Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #1074). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #1074). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Plan — generated-artifact output as a node environment value / env_file (#1074)

Requirement: extends DSL-435 (Stateful Realization Resource Declarations). Binding design per the Step 2.5 preflight note docs/decisions/issue-1074-generated-artifact-env-consumers-preflight.md.

Design (single source of truth on the node env surface)

The author declares the binding once, on nodes.<node>.runtime.environment; the compiler derives the generated-artifact consumer projection. Generated artifacts stay producer-owned; runtime environment stays node-owned. Two delivery modes join the existing mount:

1. Scalar env var — runtime.environment[] entry carries a value-free value_from: {generated_artifact, output} reference instead of a raw value.
2. Env file — new runtime.environment_files[] entry: an env-file identity plus value_from, delivered as one opaque runtime-environment input (not a mount, not parsed into key/values).

A generated runtime value is not an operator_secret (that stays reserved for out-of-SDL operator material). Raw generated bytes never enter SDL/plans/snapshots/diagnostics/audit/HTTP; existing enforce_observed_value_redaction() remains the raw-value gate. producer_private outputs cannot be consumed by env/env_file delivery.

Model changes (implementations/python/packages/raes/)

• raes_contracts/vocabulary.py: new closed enum GeneratedArtifactDeliveryMode = mount | environment | env_file (shared by runtime, capabilities, compiler).
• runtime_configuration.py:
  • GeneratedArtifactValueSource(SDLModel): value-free {generated_artifact: str, output: PortableIdentifier} reference.
  • RuntimeEnvironmentVariable: add value_from: GeneratedArtifactValueSource | None; model validator forbids a raw value together with value_from.
  • RuntimeEnvironmentFile(SDLModel): {name, value_from, description}.
  • RuntimeConfiguration: add environment_files: list[RuntimeEnvironmentFile]; extend validate_unique_runtime_entries (unique env-file names).
• stateful_resources.py: relax GeneratedArtifact.consumers from min_length=1 to allow empty (an artifact consumed only via env/env_file has no file consumer); existing identity/SSH/uniqueness validators already tolerate an empty consumer list.

Semantic validation (raes/_stateful_resource_references.py, already receives nodes + generated_artifacts)

• New env-binding checks over every node's environment[].value_from and environment_files[].value_from: artifact ref resolves unambiguously (generated_artifacts.<id> or bare <id>), output exists, output is not producer_private.
• Replace the structural min_length=1 guarantee with a scenario-level invariant: every generated artifact is consumed by at least one file consumer OR at least one env/env_file binding (no orphan artifacts).

Compilation / planning (raes_processor/)

• compiler/stateful_resources.py: _stateful_spec tags existing file consumers delivery_mode: mount and derives synthesized consumer projections for env/env_file bindings (delivery mode, node target address, selected output, and the target env-var name / env-file identity) into the compiled generated-artifact payload. The runtime-environment payload carries the value_from reference (rides the model dump).
• Both generated-artifact and runtime-environment exact SEM-218 realization concerns must still pass (dropping a binding / changing an output ref / swapping env-file for mount is an exactness failure).

Capability + backend manifest (raes_backend_protocols/, raes_backend_stubs/, planner admission)

• provisioner_capabilities.py: add supported_generated_artifact_delivery_modes: frozenset[GeneratedArtifactDeliveryMode], mirroring supported_generated_artifact_kinds; normalize empty→{mount} when supports_generated_artifacts so existing backends keep mount-only semantics and must explicitly claim environment/env_file.
• Render/restore the field in backend-manifest v2 (manifest.py) + regenerate contracts/schemas/backend-manifest/backend-manifest-v2.json and fixtures.
• Admission (planner/manifest_validation.py / stateful_admission.py / control-plane submission): a compiled env/env_file consumer requires the backend to declare that delivery mode, else a bounded stateful-support Diagnostic.

Published schemas + ledger

• Regenerate the three parity schemas via tools/generate_contract_schemas.py: sdl-authoring-input-v1, instantiated-scenario-v1, instantiated-scenario-snapshot-v1 (+ backend-manifest-v2).
• Bump content_hash + add last_change.summary in contracts/schema-publication/entries/ for each changed contract (ADR-061 ledger discipline). tools/check_generated_schemas.py and tools/check_schema_publication.py must pass.

Specs / docs

• specs/sdl/stateful-resources.md: delivery modes; env/env_file authored on runtime env; compiler-derived consumer projection; consumers may be empty when delivery is env-only.
• specs/sdl/runtime-inventory.md: environment[].value_from and environment_files[], value-free reference, redaction/provenance interplay.
• Update the implementer-facing explain docs the ADR-054 surface map points at; keep specs/sdl/references.md consistent.

Tests (implementations/python/tests/)

• test_stateful_realization_resources.py: parse/compile/plan of a scalar value_from binding and an environment_files binding; compiled payload carries delivery_mode + env-var/env-file target; empty-consumers artifact consumed only via env is valid; rejection cases — unknown artifact, unknown output, producer_private output, value+value_from conflict, orphan artifact.
• Env model + redaction assertions (test_runtime_models.py / test_runtime_observed_values.py): value_from omits raw value; classification handling.
• Capability/manifest/admission: backend claiming environment delivery admits the payload; a mount-only backend is rejected with a bounded diagnostic. Schema Draft202012Validator acceptance/rejection for the new shapes.

Example

• Wire the motivating case into examples/scenarios/techvault-defensive-min.sdl.yaml (or techvault.sdl.yaml): a generated-artifact output projected as a scalar env var (e.g. the Cortex API key on TheHive), value-free.

Traceability

• Add IMPLEMENTS/TESTS links on DSL-435 for the changed code files, new tests, and the preflight ADR note.

Gates (plan-rules)

• check_repo_policy.py, check_requirement_governance.py, verify_all.py before completion; RAES_REQUIREMENT_UID=DSL-435 where the branch lacks a UID. No CHANGELOG.md/version hand-edits. PR title will be feat(sdl): … (Conventional Commit, passes title-guard).

Security / FAANG-L6 concerns addressed

Secret handling (no raw generated bytes anywhere portable; redaction gate reused; not operator_secret); OS-exposure (env vars/env files are inspectable — materialization must avoid argv/shell/stdout/world-readable files; documented as backend obligation); reference entitlement (unambiguous resolution, no producer_private leak); realization honesty (exact SEM-218 for both concerns); capability honesty (delivery-mode seam prevents a backend silently downgrading env injection to a mount); lifecycle (a runtime-generated value may exist only after a producer boots — a consumer ref does not imply an ordering edge; documented).

Non-goals (per preflight)

No operator-secret resolver / ambient env lookup; no real Cortex/TheHive bootstrap or Cortex org/user/API-key materialization profile; no account/app-auth/SSH/content/persistent-volume semantic change; no rotation/revocation/secret-store design.

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #1074 (branch 1074-generated-artifact-env-consumers)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: ship-with-fixes

The intended shape is right: generated artifacts remain producer-owned, runtime environment remains node-owned, and the compiler derives a provisioning projection while capability gating distinguishes mount/environment/env_file delivery. The design seam is the value-free {generated_artifact, output} reference plus a backend delivery-mode vocabulary, which leaves room for later delivery modes without inventing new top-level sections. The production-readiness gaps are at the contract boundaries: some new invariants exist only in Python validators or loose dict admission, so published schemas and direct provisioning payloads can disagree with the model-backed authority.

Blocking findings (2):

1. [class] Published schemas allow value and value_from together — contracts/schemas/sdl/sdl-authoring-input-v1.json:12879
   The Python model rejects runtime environment variables that set both a literal value and value_from, and also rejects operator_secret classification for generated-artifact values, but the published JSON schemas expose value_from as an independent optional property with no equivalent cross-field constraint. External clients validating against the contract can therefore emit payloads the canonical model rejects, and non-Python consumers have no contract-level signal that raw generated values are forbidden. The existing test named test_published_schema_accepts_env_value_from_and_rejects_value_conflict only checks schema shape; it never validates a conflicting payload. Encode the invariant in the schema generation/publication path or add an explicit contract-level semantic gate that is part of the published contract.
2. [one-off] Malformed environment_consumers bypass admission — implementations/python/packages/raes_processor/planner/stateful_admission.py:34
generated_artifact_payload_diagnostic pops environment_consumers out of the spec before the closed GeneratedArtifact model validation, then _delivery_modes_in_use validates it only when the value is a list. A present-but-wrong value such as an object or string is silently treated as no environment consumers, and list entries are checked only for mapping, target address, and enum value rather than the mode-specific required fields like environment_variable or environment_file. That leaves direct/control-plane submissions able to pass admission with a compiler-derived provisioning key that the backend cannot reliably realize. Reject non-null non-list values and validate each projection with a closed mode-specific shape before capability checks.

Security review

Verdict: ship-with-fixes

This change is shaped as a bounded extension of the existing generated-artifact and runtime-environment contracts: authors bind generated outputs at the node runtime surface, the compiler derives provisioning projections, semantic admission checks cross-resource resolution, and backend capability gating is extended by delivery mode. The security seam is the new value-free reference plus the derived payload handed to provisioners; that is the right seam, but it also means any newly introduced target identifiers need the same validation discipline as existing mount destinations and environment names.

Blocking findings (1):

1. [one-off] Env-file target accepts arbitrary path-like names — implementations/python/packages/raes/runtime_generated_value.py:39
   Attacker model: an SDL author whose scenario is admitted and sent to a provisioner. This PR introduces runtime.environment_files[].name and the compiler forwards it as environment_file in the generated-artifact consumer projection. The only model validation is non-empty/strip, so values like absolute paths, ../ traversal, newlines, or shell metacharacters can be admitted into the provisioning payload. If a backend materializes env-file delivery using this field as a file target or service env-file argument, the author can redirect the generated secret artifact outside the intended runtime env-file namespace or inject malformed service configuration. Existing generated-artifact file delivery has a dedicated mount-destination model surface; this new file-delivery target needs an equally bounded identifier/path contract before it crosses the planner/backend boundary.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #1074 on branch '1074-generated-artifact-env-consumers'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #1074)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The intended shape is right: generated artifacts remain producer-owned, runtime environment remains node-owned, and the compiler derives a provisioning projection while capability gating distinguishes mount/environment/env_file delivery. The design seam is the value-free {generated_artifact, output} reference plus a backend delivery-mode vocabulary, which leaves room for later delivery modes without inventing new top-level sections. The production-readiness gaps are at the contract boundaries: some new invariants exist only in Python validators or loose dict admission, so published schemas and direct provisioning payloads can disagree with the model-backed authority.

Security reviewer: This change is shaped as a bounded extension of the existing generated-artifact and runtime-environment contracts: authors bind generated outputs at the node runtime surface, the compiler derives provisioning projections, semantic admission checks cross-resource resolution, and backend capability gating is extended by delivery mode. The security seam is the new value-free reference plus the derived payload handed to provisioners; that is the right seam, but it also means any newly introduced target identifiers need the same validation discipline as existing mount destinations and environment names.

Blocking findings: 3

Finding 1 — class (4 instances)

• ID: F1
• Title: [core] Published schemas allow value and value_from together
• Location: contracts/schemas/sdl/sdl-authoring-input-v1.json:12879
• Decision: fix
• Rationale: The Python model rejects runtime environment variables that set both a literal value and value_from, and also rejects operator_secret classification for generated-artifact values, but the published JSON schemas expose value_from as…
• Instances:
  • contracts/schemas/sdl/sdl-authoring-input-v1.json:12879
  • contracts/schemas/sdl/instantiated-scenario-v1.json:15059
  • contracts/schemas/sdl/instantiated-scenario-snapshot-v1.json:15702
  • contracts/schemas/satisfiability/scenario-satisfiability-evidence-v1.json:15434

Finding 2 — one-off

• ID: F2
• Title: [core] Malformed environment_consumers bypass admission
• Location: implementations/python/packages/raes_processor/planner/stateful_admission.py:34
• Decision: fix
• Rationale: generated_artifact_payload_diagnostic pops environment_consumers out of the spec before the closed GeneratedArtifact model validation, then _delivery_modes_in_use validates it only when the value is a list. A present-but-wrong valu…

Finding 3 — one-off

• ID: F3
• Title: [security] Env-file target accepts arbitrary path-like names
• Location: implementations/python/packages/raes/runtime_generated_value.py:39
• Decision: fix
• Rationale: Attacker model: an SDL author whose scenario is admitted and sent to a provisioner. This PR introduces runtime.environment_files[].name and the compiler forwards it as environment_file in the generated-artifact consumer projection. The…

[Brad-Edwards]

gc_test_quality_review cycle 1 of 1 — issue #1074

Reviewer: test-quality (claude-sonnet-5 via gc_test_quality_review)
Branch: 1074-generated-artifact-env-consumers
Cycle: 1 / 1
Findings: 2

Finding 1 — [warning] implementations/python/tests/test_issue_1074_generated_artifact_env_consumers.py::test_runtime_environment_projection_pins_value_from_without_raw_bytes

Problem: The closing assertion assert not entry.get("value") is presented (via the inline comment) as verifying that project_environment discloses no raw literal value, but project_environment/_committed_value never write a "value" key into the returned entry under any input — value material only ever surfaces as value_present/value_commitment. The assertion is unrelated to the value_from feature under test and passes identically whether or not the value-from wiring is present or broken.
Why it matters: If a future change leaked the raw generated-artifact value under value_commitment, source, or any key other than a literal "value", this test would still pass, giving false confidence that the value-free contract is enforced.
Fix: Assert on the actual disclosure surface, e.g. assert "value_commitment" not in entry, or drop the assertion and rely on entry["value_from"] == {...} plus the existing parse-time rejection test for a raw value alongside value_from.

Finding 2 — [warning] implementations/python/packages/raes_backend_protocols/provisioner_capabilities.py:107-122

Problem: New production logic backfills supported_generated_artifact_delivery_modes to {MOUNT} when supports_generated_artifacts=True and no delivery modes are declared (legacy-manifest compatibility path). No test in the diff constructs a ProvisionerCapabilities/manifest with supports_generated_artifacts=True and an empty supported_generated_artifact_delivery_modes to check the backfill actually lands on {MOUNT}.
Why it matters: If the backfill defaulted to the full mode set (or stayed empty) instead of mount-only, a legacy backend manifest would silently be treated as capable of environment/env-file secret injection it never declared support for, and nothing in the test suite would catch it.
Fix: Add a test constructing ProvisionerCapabilities(supports_generated_artifacts=True, supported_generated_artifact_kinds={...}) without supported_generated_artifact_delivery_modes, and assert the result equals frozenset({GeneratedArtifactDeliveryMode.MOUNT}).

[Brad-Edwards]

gc_test_quality_review cycle 1 of 1 complete for issue #1074 on branch '1074-generated-artifact-env-consumers'. Posted by the MCP server to enforce the gc_test_quality_review hard-cap-1 contract (issue #884 follow-up, default lowered in #906). Do not edit or delete — used by the next gc_test_quality_review invocation to count cycles.

[Brad-Edwards]

Review decision record — test-quality cycle 1 (issue #1074)

Reviewer: test-quality
Cycle: 1

Architectural read:

This is a well-formed extension of the generated-artifact/consumer model: a new value-free GeneratedArtifactValueSource reference type flows cleanly through parse → semantic admission → compilation → provisioner-capability gating, with a new GeneratedArtifactDeliveryMode enum threaded consistently through all four layers. The new test file test_issue_1074_generated_artifact_env_consumers.py is unusually disciplined — it exercises real parse/validation/compile/plan pipelines rather than mocking internals, includes negative cases (unknown artifact, unknown output, producer-private rejection, path-traversal env-file names, malformed compiler-derived projections) and a schema-level roundtrip. The two gaps below are narrow and don't change that overall assessment.

Blocking findings: 2

Finding 1 — one-off

• ID: F1
• Title: (no title)
• Decision: fix
• Rationale: Addressed by next cycle

Finding 2 — one-off

• ID: F2
• Title: (no title)
• Decision: fix
• Rationale: Addressed by next cycle