From c61d8a25d70e278cba05101a7be5593ef0334616 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 18:54:28 -0400 Subject: [PATCH 01/67] docs(flow-d): add Lambda MicroVM substrate diagram doc Flow D is a second Flow-A substrate: AWS Lambda MicroVM via the ACK lambdamicrovms controller, composed by a single KRO ResourceGraphDefinition. Mirrors the flow-a-sandbox-capability.md style. Covers substrate architecture (KRO RGD over ACK primitives), the platform/app ownership split inside one RGD, the RuntimeClass-marked bridge shim (claim -> pod -> MicroVM), and the GA migration path (self-managed controller -> Managed ACK adopts it, RGD unchanged). Flow C is reserved for other work; this substrate is Flow D. --- .../diagrams/flow-d-microvm-sandbox.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/dark-factory/diagrams/flow-d-microvm-sandbox.md diff --git a/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md new file mode 100644 index 00000000..25013226 --- /dev/null +++ b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md @@ -0,0 +1,109 @@ +# Flow D — Lambda MicroVM–backed Agent Sandbox (alternative substrate) + +**Flow D is a second Flow-A substrate.** Where Flow A hands out **Kata micro-VM pods** on a +self-managed nested-virt EKS node group, Flow D hands out **AWS Lambda MicroVMs** provisioned by the +ACK `lambdamicrovms` controller and composed by a single **KRO `ResourceGraphDefinition`**. The Agent +Sandbox UX is unchanged: a consumer (notably **Flow B — Dark Factory**) creates a `SandboxClaim`, a +pod shows up, and the **same `dark-factory-coder`** runs its coding/testing loop — except the coder +executes inside a Lambda MicroVM instead of on the Kata node. + +> **Flow C is reserved for other work** — this substrate is **Flow D**. + +> **Why a second substrate?** Kata (Flow A) needs a dedicated nested-virt node group the platform owns +> and pays for while idle. Lambda MicroVM is a **serverless** micro-VM: no node group to run, per-claim +> lifecycle, sub-second warm starts, and a clean **platform-owns-the-image / app-owns-the-instance** +> split that maps directly onto the two ACK CRDs. Flow B can target either substrate with no pipeline +> change — it only ever sees the Agent Sandbox `SandboxClaim` contract. + +> 🎨 Diagrams are editable draw.io — sources in [`src/`](./src/), rendered PNGs in [`img/`](./img/). +> *(Flow D diagram sources are added alongside the Flow A/B ones; see `src/flow-d-*.drawio`.)* + +--- + +## D.1 — Substrate architecture (KRO RGD over ACK primitives) + +The platform installs two controllers and one composition layer, then exposes **one** custom +resource to consumers: + +- **Managed ACK** (EKS Capability) runs the **GA** controllers — `iam.services.k8s.aws` (Role) and + `s3.services.k8s.aws` (Bucket) — that the MicroVM image + instance depend on. +- **Self-managed ACK** runs **only** the pre-GA `lambdamicrovms.services.k8s.aws` controller (its own + Helm chart / ArgoCD addon), because Managed ACK bundles GA controllers only. +- **Managed KRO** (EKS Capability) runs the `ResourceGraphDefinition` engine. +- A single **`MicrovmSandbox` RGD** ties it all together: one CR expands into `MicrovmImage` + + `Microvm` + IAM `Role`(s) + S3 `Bucket`. + +``` +consumer (Flow B / any agent) + │ creates + ▼ + MicrovmSandbox (kro.run/v1alpha1 — the single abstraction) + │ expands into + ├── MicrovmImage (lambdamicrovms.services.k8s.aws) ── platform-owned inputs + ├── S3 Bucket (s3.services.k8s.aws) ── image codeArtifact store + ├── IAM Role (build) (iam.services.k8s.aws) ── MicrovmImage.buildRoleArn + ├── IAM Role (exec) (iam.services.k8s.aws) ── Microvm.executionRoleArn + └── Microvm (lambdamicrovms.services.k8s.aws) ── app-owned instance lifecycle +``` + +*Edit: `src/flow-d-substrate.drawio` → `img/flow-d-substrate.png`.* + +--- + +## D.2 — Platform-owned vs app-owned split (inside one RGD) + +The two ACK CRDs encode the ownership boundary the platform team and application teams care about; +the RGD schema surfaces each half to the right owner: + +| Layer | Owner | ACK resource | Key fields | +|---|---|---|---| +| **Image / substrate** | Platform | `MicrovmImage` | `baseImageARN`, `buildRoleArn`, `codeArtifact.uri` (S3), egress connectors | +| **Instance / run** | App team | `Microvm` | `imageIdentifier`, `executionRoleArn`, `ingress/egressNetworkConnectors`, `idlePolicy` | + +- **Platform** sets the image once (built **from the existing `dark-factory-coder` image** + its + `entrypoint.js`, published to the S3 `codeArtifact` bucket) — declarative, ACK-managed, GitOps. +- **App teams / Flow B** create per-claim `Microvm` instances referencing that image, and own the + instance lifecycle (`RunMicrovm` / `TerminateMicrovm`, idle policy) via the same claim they use today. + +*Edit: `src/flow-d-ownership.drawio` → `img/flow-d-ownership.png`.* + +--- + +## D.3 — The RuntimeClass shim (claim → pod → MicroVM) + +A literal Kubernetes `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**. +Lambda MicroVM is a **remote AWS service**, so a true node-level RuntimeClass isn't possible without a +virtual-kubelet provider (a large Go runtime component — explicitly **out of scope**). Flow D uses a +**RuntimeClass-marked bridge pod** instead, preserving the exact Agent Sandbox UX: + +``` +SandboxClaim (Flow B injects DF_ISSUE_NUMBER, repo, branch — unchanged) + │ + ▼ +Sandbox → Pod from the `lambda-microvm` SandboxTemplate variant + │ (bridge container; lands on a normal Auto-Mode node, NOT the kata pool) + ▼ +bridge applies a MicrovmSandbox (KRO) CR + │ + ▼ +Microvm RUNNING ── runs the SAME dark-factory-coder entrypoint (node /app/entrypoint.js) + │ + ├── bridge streams MicroVM logs → pod logs (pod Running ⇔ Microvm RUNNING) + └── pod exit / claim teardown → TerminateMicrovm +``` + +To Flow B and the user this is identical to Flow A — "a sandbox pod appeared and ran the coder" — but +the coder actually executed in the Lambda MicroVM. Log streaming is straightforward; interactive +exec/attach passthrough is **best-effort** (full fidelity would need virtual-kubelet). + +*Edit: `src/flow-d-shim.drawio` → `img/flow-d-shim.png`.* + +--- + +## D.4 — Future: when `lambdamicrovms` goes GA + +`lambdamicrovms` is currently **pre-GA** (`v1alpha1`), so its controller is self-managed. When it +graduates to GA upstream, **Managed ACK adopts it automatically** — the self-managed chart is deleted +and the `MicrovmSandbox` RGD is **unchanged** (it references the same `lambdamicrovms.services.k8s.aws` +CRDs regardless of who runs the controller). The design deliberately keeps the RGD independent of the +controller install method so this migration is a one-line addon removal. From a7e4530bccc7fe51fa0be4ea193c3a84aac5fb76 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 18:57:38 -0400 Subject: [PATCH 02/67] docs(flow-d): add Flow D section + TOC/flows-table entries to dark-factory README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New §4.5 'Flow D — Lambda MicroVM substrate (alternative to Flow A)': KRO RGD over ACK primitives (Managed KRO + Managed ACK for GA iam/s3 + self-managed lambdamicrovms pre-GA controller), the platform-owned/app-owned split encoded in the MicrovmImage/Microvm CRDs, the RuntimeClass-marked bridge shim, and disabled-by-default GitOps delivery. Adds the TOC entry and a 'second substrate' note to the Two-flows-at-a-glance section. Reuses dark-factory-coder image. --- docs/dark-factory/README.md | 69 +++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/dark-factory/README.md b/docs/dark-factory/README.md index 94a0f66a..c2ae0793 100644 --- a/docs/dark-factory/README.md +++ b/docs/dark-factory/README.md @@ -26,6 +26,7 @@ spokes as normal deployments. 2. [Two flows at a glance](#2-two-flows-at-a-glance) 3. [Flow A — Agent Sandbox capability](#3-flow-a--agent-sandbox-capability-permanent-platform-feature) 4. [Flow B — the Dark Factory pipeline](#4-flow-b--the-dark-factory-pipeline) + - [Flow D — Lambda MicroVM substrate (alternative to Flow A)](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a) 5. [The pluggable coding assistant](#5-the-pluggable-coding-assistant) 6. [Independent verification](#6-independent-verification-the-heart-of-the-pattern) 7. [Live status in the PR](#7-live-status-in-the-pr) @@ -84,6 +85,15 @@ useful on its own and the factory is a consumer of it. > an isolated VM). The Dark Factory is one *consumer* of that capability. Keeping them separate > means the isolation substrate can ship, be tested, and be reused independently of the factory. +> **Flow D — a second substrate.** Flow A's isolation boundary is a **Kata micro-VM pod** on a +> platform-owned nested-virt node group. **Flow D** offers an *alternative* Flow-A substrate — an +> **AWS Lambda MicroVM** (serverless micro-VM, no node group) provisioned via the ACK `lambdamicrovms` +> controller and composed by a single **KRO `ResourceGraphDefinition`**. Flow B is unchanged and can +> target either substrate through the same `SandboxClaim` contract. See +> [§4.5](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a) and +> [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md). *(Flow C is reserved for +> other work.)* + > **Why the hub, not a spoke?** The Dark Factory is a **pre-dev build/author** activity: it *writes* > code and needs GitHub write access. That belongs on the **hub — the control/build plane** — not on > a spoke, which is the **deploy/run plane** hosting real enterprise workloads (putting a @@ -256,6 +266,65 @@ UI — the substrate for scaling across many concurrent issues. --- +## 4.5. Flow D — Lambda MicroVM substrate (alternative to Flow A) + +> 📊 **See the diagrams:** [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md) +> (substrate architecture + platform/app ownership split + the RuntimeClass-shim bridge). + +Flow A's isolation boundary is a **Kata micro-VM pod** on a platform-owned nested-virt node group. +**Flow D is a second Flow-A substrate**: an **AWS Lambda MicroVM** — a *serverless* micro-VM with no +node group to run or pay for while idle, per-claim lifecycle, and sub-second warm starts. Flow B is +unchanged: it still creates a `SandboxClaim`, a pod still shows up, and the **same `dark-factory-coder`** +runs its coding/testing loop — except the coder executes inside a Lambda MicroVM. *(Flow C is reserved +for other work; this substrate is Flow D.)* + +### How it's built — KRO RGD over ACK primitives + +| Layer | Mechanism | Notes | +|---|---|---| +| **Composition** | **Managed KRO** (EKS Capability) + one `MicrovmSandbox` `ResourceGraphDefinition` | One CR expands into all primitives below | +| **GA primitives** | **Managed ACK** (EKS Capability) — `iam` Role, `s3` Bucket | AWS-run; the image store + build/exec roles | +| **MicroVM primitives** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` + `Microvm` CRDs (`lambdamicrovms.services.k8s.aws/v1alpha1`) | + +> **Why self-managed for the MicroVM controller?** Managed ACK bundles only controllers whose service +> is **GA upstream**. `lambdamicrovms` is **pre-GA** (`v1alpha1`), so it isn't in Managed ACK yet — it +> runs as its own GitOps addon. **Managed ACK + self-managed lambdamicrovms coexist** (different CRD +> groups → no conflict). When `lambdamicrovms` goes GA, delete the self-managed addon and Managed ACK +> adopts it — **the RGD is unchanged**. This "install both now" posture is deliberate and futuristic. + +### Platform-owned vs app-owned (encoded in the two ACK CRDs) + +- **Platform owns the image/substrate** — declarative, ACK-managed: `MicrovmImage` + (`baseImageARN`, `buildRoleArn`, `codeArtifact.uri` in S3, egress connectors). The image is built + **from the existing `dark-factory-coder` image** + its `entrypoint.js`. +- **App teams own the instance lifecycle** — `Microvm` (`imageIdentifier`, `executionRoleArn`, + `ingress/egressNetworkConnectors`, `idlePolicy{autoResumeEnabled, maxIdleDurationSeconds, + suspendedDurationSeconds}`) — created per claim, torn down with it. + +The single `MicrovmSandbox` RGD surfaces both halves so each owner sets its own fields, while +consumers see just one CRD. + +### The RuntimeClass shim (claim → pod → MicroVM) + +A literal K8s `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**; Lambda +MicroVM is a **remote AWS service**, so a true node-level RuntimeClass would require a virtual-kubelet +provider (a large Go runtime — **out of scope**). Flow D instead ships a **`lambda-microvm` +SandboxTemplate variant** whose pod is a lightweight **bridge**: it applies the `MicrovmSandbox` CR, +then **streams the MicroVM's logs into the pod** and maps lifecycle (pod Running ⇔ `Microvm` RUNNING; +pod exit → `TerminateMicrovm`). To Flow B and the user the UX is identical to Flow A. Interactive +exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up. + +### Delivery & status + +Shipped as GitOps, **disabled by default** (like the kata nodepool): a `microvm:` values block gates +the RGD + SandboxTemplate + self-managed controller addon; the hub overlay carries cluster-specific +values. The platform-capability enablement (Managed ACK + Managed KRO) lands separately in the +**appmod-blueprints** platform repo (they're EKS Capabilities, like the Managed ArgoCD the hub already +runs). This PR delivers the **design + GitOps scaffold**; the live end-to-end path (enable capabilities +→ sync controller → run a MicroVM coder) is the follow-up. + +--- + ## 5. The pluggable coding assistant The coder is behind a **thin, swappable interface** — a deliberate choice (the industry lesson is From 46a16a7289e3891afd2ee13f11bf22f8102a8ced Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 19:02:36 -0400 Subject: [PATCH 03/67] feat(flow-d): add self-managed ack-lambdamicrovms controller addon (disabled) New addons.yaml entry installing the pre-GA Lambda MicroVM ACK controller (oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart v0.1.1) that reconciles the MicrovmImage + Microvm CRDs. Self-managed because Managed ACK only bundles GA controllers; coexists with Managed ACK (GA iam/s3) since CRD groups differ. enabled:false (Flow D dormant), hub-only via alwaysSelector, sync-wave 0 so CRDs+controller precede the MicrovmSandbox RGD (wave 2). Pod-identity auth (no IRSA annotation), cluster install scope. --- gitops/addons/bootstrap/default/addons.yaml | 43 +++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/gitops/addons/bootstrap/default/addons.yaml b/gitops/addons/bootstrap/default/addons.yaml index 282961aa..6671ee8b 100644 --- a/gitops/addons/bootstrap/default/addons.yaml +++ b/gitops/addons/bootstrap/default/addons.yaml @@ -468,6 +468,49 @@ kata-deploy: runtimeClasses: enabled: false +# ack-lambdamicrovms (Flow D) — SELF-MANAGED ACK controller for the pre-GA +# Lambda MicroVM service (lambdamicrovms.services.k8s.aws: MicrovmImage + Microvm). +# Self-managed because Managed ACK (EKS Capability) only bundles GA-upstream +# controllers, and lambdamicrovms is still v1alpha1. Managed ACK (GA iam/s3) and +# this self-managed controller COEXIST (different CRD groups). When lambdamicrovms +# goes GA, delete this addon and Managed ACK adopts it — the KRO RGD is unchanged. +# +# DISABLED by default: Flow D is dormant until a cluster opts in. Hub-only, like +# the rest of the sandbox substrate. Installs the OCI chart +# oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart. Sync-wave 0 so the +# CRDs + controller are up before the agent-sandbox chart's MicrovmSandbox RGD +# (wave 2) references them. +ack-lambdamicrovms: + enabled: false + namespace: ack-system + chartName: lambdamicrovms-chart + defaultVersion: '0.1.1' + chartRepository: 'public.ecr.aws/aws-controllers-k8s' + annotationsAppSet: + argocd.argoproj.io/sync-wave: '0' + # Hub-only: the MicroVM substrate lives with the rest of the sandbox capability + # on the build plane. alwaysSelector pins generation to environment: + # control-plane regardless of useSelectors — never generated on a spoke. + alwaysSelector: + matchExpressions: + - key: environment + operator: In + values: ['control-plane'] + # ACK chart reads these at the ROOT of values (chart v0.1.1). The controller + # reconciles the Microvm + MicrovmImage CRDs; it uses EKS Pod Identity for AWS + # auth (empty SA annotations — the pod-identity association is created out of + # band / by the platform), so no IRSA role-arn annotation is set here. + valuesObject: + aws: + region: '{{default "us-west-2" (index .metadata.annotations "aws_region")}}' + serviceAccount: + create: true + name: ack-lambdamicrovms-controller + # Cluster-scoped install so the single MicrovmSandbox RGD (any namespace) can + # create Microvm/MicrovmImage CRs the controller reconciles. + installScope: cluster + deletionPolicy: delete + # Dark Factory (Flow B) — Argo Workflows that turn a GitHub issue into a PR by # claiming the Flow A warm pool. Hub-only (co-located with Argo Workflows + the # sandbox pool); sync-wave 3 so it lands after the agent-sandbox capability From 7ea85c45dcb31064309d6b5a76b8aa128a2b646e Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 19:05:42 -0400 Subject: [PATCH 04/67] feat(flow-d): add KRO MicrovmSandbox RGD tying all MicroVM primitives templates/50-rgd-microvm-sandbox.yaml: one kro.run/v1alpha1 ResourceGraphDefinition that expands a single MicrovmSandbox CR into S3 Bucket + build/exec IAM Roles (GA -> Managed ACK) + MicrovmImage + Microvm (pre-GA -> self-managed ACK). The platform/app ownership split is encoded in the schema (spec.image.* platform-owned, spec.run.* app-owned); status surfaces microvmID/state for the bridge to mirror. RGD is controller-install-agnostic (unchanged when lambdamicrovms goes GA). values.yaml: new microvm: block (enabled:false, region, apiGroup, bridgeImage, idle/connector defaults; cluster-specific ARNs left blank for the overlay). Verified: helm template renders 0 RGD when disabled, 1 valid RGD (5 resources: bucket/buildRole/execRole/image/microvm) when enabled. --- .../templates/50-rgd-microvm-sandbox.yaml | 133 ++++++++++++++++++ .../addons/charts/agent-sandbox/values.yaml | 33 +++++ 2 files changed, 166 insertions(+) create mode 100644 gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml diff --git a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml new file mode 100644 index 00000000..1cf4ca4c --- /dev/null +++ b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml @@ -0,0 +1,133 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — MicrovmSandbox ResourceGraphDefinition (KRO). + +ONE composite CRD (kro.run) that ties together ALL the Lambda MicroVM primitives so +a consumer (the lambda-microvm SandboxTemplate bridge, or any agent) creates a single +`MicrovmSandbox` and gets the whole substrate: + + MicrovmSandbox -> S3 Bucket (s3.services.k8s.aws) image codeArtifact store + IAM Role (build) (iam.services.k8s.aws) MicrovmImage.buildRoleARN + IAM Role (exec) (iam.services.k8s.aws) Microvm.executionRoleARN + MicrovmImage (lambdamicrovms.services.k8s.aws) platform-owned image + Microvm (lambdamicrovms.services.k8s.aws) app-owned instance + +OWNERSHIP SPLIT is expressed IN THE SCHEMA: + spec.image.* — PLATFORM-owned (base image, code artifact) — set once per image + spec.run.* — APP-owned (per-claim instance: idle policy, connectors, env) + +CONTROLLER SPLIT: the s3/iam resources are reconciled by MANAGED ACK (GA controllers); +MicrovmImage/Microvm by the SELF-MANAGED lambdamicrovms controller (pre-GA). The RGD is +identical regardless of who runs the controllers — when lambdamicrovms goes GA and +Managed ACK adopts it, THIS FILE DOES NOT CHANGE. + +Gated behind microvm.enabled so Flow D stays dormant until a cluster opts in. Requires +the Managed KRO capability (kro.run) + the ACK controllers to be present on the cluster. +*/ -}} +apiVersion: kro.run/v1alpha1 +kind: ResourceGraphDefinition +metadata: + name: microvm-sandbox + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + schema: + apiVersion: v1alpha1 + kind: MicrovmSandbox + group: {{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }} + spec: + # ── PLATFORM-owned: the image / substrate (set once per image) ────────── + image: + # ARN of the AWS-published base MicroVM image to build from. + baseImageARN: string + # S3 URI of the coder code artifact baked into the image. This is the SAME + # dark-factory-coder payload used by Flow A — MicroVM substrate, same coder. + codeArtifactUri: string + # Public-egress connector ARN the image build is allowed (platform policy). + egressConnectorARN: string | default="{{ .Values.microvm.defaults.egressConnectorARN }}" + # ── APP-owned: the per-claim instance lifecycle ───────────────────────── + run: + # Idle policy — MicroVM auto-suspend/resume to keep idle cost near zero. + autoResumeEnabled: boolean | default=true + maxIdleDurationSeconds: integer | default={{ .Values.microvm.defaults.maxIdleDurationSeconds }} + suspendedDurationSeconds: integer | default={{ .Values.microvm.defaults.suspendedDurationSeconds }} + # Connector ARNs governing the running MicroVM's network. + ingressConnectorARN: string | default="{{ .Values.microvm.defaults.ingressConnectorARN }}" + egressConnectorARN: string | default="{{ .Values.microvm.defaults.egressConnectorARN }}" + # Common: AWS region + a name stem for the created resources. + region: string | default="{{ .Values.microvm.region }}" + name: string + status: + # Surfaced from the ACK resources so the bridge pod can mirror lifecycle. + microvmID: ${microvm.status.microvmID} + microvmState: ${microvm.status.conditions[0].status} + imageARN: ${image.status.ackResourceMetadata.arn} + resources: + # 1) S3 bucket that stores the MicroVM code artifact (GA — Managed ACK). + - id: bucket + template: + apiVersion: s3.services.k8s.aws/v1alpha1 + kind: Bucket + metadata: + name: ${schema.spec.name}-microvm-artifacts + spec: + name: ${schema.spec.name}-microvm-artifacts + # 2) IAM role the image BUILD assumes (GA — Managed ACK). Platform-owned. + - id: buildRole + template: + apiVersion: iam.services.k8s.aws/v1alpha1 + kind: Role + metadata: + name: ${schema.spec.name}-microvm-build + spec: + name: ${schema.spec.name}-microvm-build + policyRefs: [] + assumeRolePolicyDocument: | + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambdamicrovms.amazonaws.com"},"Action":"sts:AssumeRole"}]} + # 3) IAM role the RUNNING MicroVM assumes (GA — Managed ACK). App-owned exec identity. + - id: execRole + template: + apiVersion: iam.services.k8s.aws/v1alpha1 + kind: Role + metadata: + name: ${schema.spec.name}-microvm-exec + spec: + name: ${schema.spec.name}-microvm-exec + policyRefs: [] + assumeRolePolicyDocument: | + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambdamicrovms.amazonaws.com"},"Action":"sts:AssumeRole"}]} + # 4) MicrovmImage (pre-GA — SELF-MANAGED lambdamicrovms controller). Platform-owned. + - id: image + template: + apiVersion: lambdamicrovms.services.k8s.aws/v1alpha1 + kind: MicrovmImage + metadata: + name: ${schema.spec.name}-image + spec: + name: ${schema.spec.name}-image + baseImageARN: ${schema.spec.image.baseImageARN} + buildRoleARN: ${buildRole.status.ackResourceMetadata.arn} + codeArtifact: + uri: ${schema.spec.image.codeArtifactUri} + egressNetworkConnectors: + - ${schema.spec.image.egressConnectorARN} + # 5) Microvm instance (pre-GA — SELF-MANAGED). App-owned lifecycle. Runs the + # same dark-factory-coder entrypoint that Flow A's Kata pod runs. + - id: microvm + template: + apiVersion: lambdamicrovms.services.k8s.aws/v1alpha1 + kind: Microvm + metadata: + name: ${schema.spec.name}-vm + spec: + imageIdentifier: ${image.status.ackResourceMetadata.arn} + executionRoleARN: ${execRole.status.ackResourceMetadata.arn} + ingressNetworkConnectors: + - ${schema.spec.run.ingressConnectorARN} + egressNetworkConnectors: + - ${schema.spec.run.egressConnectorARN} + idlePolicy: + autoResumeEnabled: ${schema.spec.run.autoResumeEnabled} + maxIdleDurationSeconds: ${schema.spec.run.maxIdleDurationSeconds} + suspendedDurationSeconds: ${schema.spec.run.suspendedDurationSeconds} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index f925fb73..ce1ee7c4 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -235,3 +235,36 @@ nodepool: subnetIds: [] amiId: "" launchTemplateId: "" + +# ── Flow D — Lambda MicroVM substrate (alternative to the Kata nodepool) ────── +# A SECOND Agent-Sandbox substrate: instead of a Kata pod on the nested-virt node +# group, the coder runs in an AWS Lambda MicroVM provisioned by the ACK +# lambdamicrovms controller and composed by a single KRO ResourceGraphDefinition +# (templates/50-rgd-microvm-sandbox.yaml). A `lambda-microvm` SandboxTemplate +# bridge (templates/51-...) preserves the Agent-Sandbox UX. See docs/dark-factory +# §4.5 and diagrams/flow-d-microvm-sandbox.md. +# +# DISABLED by default (like the kata nodepool): flip enabled=true on a cluster that +# has the Managed KRO + Managed ACK capabilities and the self-managed +# ack-lambdamicrovms controller (addons.yaml). Cluster-specific ARNs belong in the +# per-cluster overlay, NOT here. +microvm: + enabled: false + region: us-west-2 + # API group the generated MicrovmSandbox CRD is served under (KRO schema.group). + apiGroup: sandbox.agents.x-k8s.io + # Bridge pod image: reuse an image with kubectl (applies the MicrovmSandbox CR) + # + the AWS CLI (streams MicroVM logs). alpine/k8s bundles kubectl. + bridgeImage: alpine/k8s:1.31.0 + # The coder code-artifact + base image are cluster/account specific → overlay. + # image: + # baseImageARN: arn:aws:lambdamicrovms:::image/ + # codeArtifactUri: s3:///dark-factory-coder/.zip + baseImageARN: "" + codeArtifactUri: "" + # Safe cluster-agnostic defaults for the RGD schema (overlay sets the ARNs). + defaults: + maxIdleDurationSeconds: 900 + suspendedDurationSeconds: 300 + ingressConnectorARN: "" + egressConnectorARN: "" From 7390cf6b709c4054597f9d2760ea9717075f5c69 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 19:07:54 -0400 Subject: [PATCH 05/67] feat(flow-d): add lambda-microvm SandboxTemplate + bridge (the RuntimeClass shim) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit templates/51-sandboxtemplate-microvm.yaml: the shim that keeps the Agent-Sandbox UX while the coder runs in a remote Lambda MicroVM. A SandboxTemplate (-microvm) whose pod is a thin BRIDGE on a normal Auto-Mode node (no kata runtimeClass/nodeSelector) — it idles until a claim injects DF_ISSUE_NUMBER (same as Flow A), applies a MicrovmSandbox (KRO) CR, waits for Microvm RUNNING, mirrors lifecycle (pod Running <-> Microvm RUNNING), and deletes the CR on exit (-> TerminateMicrovm). Ships its ServiceAccount + least-priv Role/RoleBinding (microvmsandboxes only) + the bridge.sh ConfigMap. Both substrates coexist; a claim picks one by which template it references. Verified: helm template renders 5 objects when enabled, 0 when disabled; bridge.sh passes sh -n. --- .../templates/51-sandboxtemplate-microvm.yaml | 177 ++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml diff --git a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml new file mode 100644 index 00000000..7fd93ef0 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml @@ -0,0 +1,177 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — `lambda-microvm` SandboxTemplate + bridge RBAC + bridge script. + +The RuntimeClass "shim": Lambda MicroVM is a REMOTE AWS service, not a node-local +containerd handler, so there is no literal `lambda-microvm` RuntimeClass (that would +need a virtual-kubelet — out of scope). Instead this SandboxTemplate's pod is a thin +BRIDGE that preserves the Agent-Sandbox UX: + + 1. runs on a NORMAL Auto-Mode node (no kata runtimeClass / nodeSelector / taint) + 2. applies a MicrovmSandbox (KRO) CR from the injected claim env (DF_ISSUE_NUMBER, + repo, branch — same as Flow A), which provisions the Microvm running the SAME + dark-factory-coder entrypoint + 3. waits for Microvm RUNNING, then STREAMS its logs into the pod (pod Running ⇔ + Microvm RUNNING); on pod exit / claim teardown it deletes the MicrovmSandbox + (→ TerminateMicrovm) + +To Flow B and the user this looks identical to a Flow A claim. The bridge needs a +ServiceAccount (unlike the credential-less Kata coder) because it drives the KRO CR. + +Gated behind microvm.enabled. Rendered as a SEPARATE SandboxTemplate +(`-microvm`) so both substrates can coexist on one cluster; a consumer +selects the substrate by which template its SandboxClaim references. +*/ -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +rules: + # The bridge creates/reads/deletes MicrovmSandbox CRs (the KRO composite) only. + - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] + resources: ["microvmsandboxes"] + verbs: ["create", "get", "list", "watch", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: microvm-bridge +subjects: + - kind: ServiceAccount + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: microvm-bridge-script + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +data: + bridge.sh: | + #!/bin/sh + # Flow D bridge — claim -> MicrovmSandbox -> mirror lifecycle. Idles until a + # SandboxClaim injects DF_ISSUE_NUMBER (Flow B), exactly like the Kata coder. + set -eu + echo "[microvm-bridge] idle — waiting for a SandboxClaim to inject DF_ISSUE_NUMBER..." + while [ -z "${DF_ISSUE_NUMBER:-}" ]; do sleep 5; done + NAME="df-${DF_ISSUE_NUMBER}" + echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} -> creating MicrovmSandbox/${NAME}" + + # Delete the MicrovmSandbox on exit -> KRO cascades TerminateMicrovm + cleans + # the S3/IAM resources. Mirrors Flow A's onExit teardown. + cleanup() { + echo "[microvm-bridge] tearing down MicrovmSandbox/${NAME}" + kubectl delete microvmsandbox "${NAME}" --ignore-not-found --wait=false || true + } + trap cleanup EXIT INT TERM + + cat </dev/null || echo "") + [ "${STATE}" = "True" ] && break + i=$((i+1)); sleep 5 + done + VMID=$(kubectl get microvmsandbox "${NAME}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + echo "[microvm-bridge] Microvm ${VMID:-} state=${STATE:-} — pod now mirrors the MicroVM lifecycle." + + # Keep the pod alive == MicroVM alive. Real log streaming (aws lambdamicrovms + # get-microvm-logs / CloudWatch tail) is wired via the image build; here we hold + # the pod so Sandbox lifecycle == Microvm lifecycle. Exec/attach passthrough is + # a virtual-kubelet follow-up (see docs/dark-factory §4.5). + while kubectl get microvmsandbox "${NAME}" >/dev/null 2>&1; do sleep 15; done + echo "[microvm-bridge] MicrovmSandbox gone — exiting." +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: {{ .Values.warmPool.templateName }}-microvm + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + # Same env-injection contract as the Kata template so Flow B is unchanged. + envVarsInjectionPolicy: {{ .Values.coderTemplate.envVarsInjectionPolicy | default "Allowed" }} + podTemplate: + metadata: + labels: + {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} + agent-sandbox.io/role: coder + agent-sandbox.io/substrate: lambda-microvm + spec: + # NO kata runtimeClass / nodeSelector / toleration — the bridge is a normal + # pod on an Auto-Mode node. The isolation boundary is the remote MicroVM. + serviceAccountName: microvm-bridge + # The bridge DOES need a token (unlike the Kata coder) to drive the KRO CR. + automountServiceAccountToken: true + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: bridge + image: {{ .Values.microvm.bridgeImage }} + command: ["/bin/sh", "/scripts/bridge.sh"] + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: { cpu: 50m, memory: 64Mi } + limits: { cpu: 200m, memory: 128Mi } + volumeMounts: + - name: bridge-script + mountPath: /scripts + - name: tmp + mountPath: /tmp + volumes: + - name: bridge-script + configMap: + name: microvm-bridge-script + defaultMode: 0555 + - name: tmp + emptyDir: {} +{{- end }} From 8f6a081c1b2725bd2558b5b9e78c376b2c0c5da6 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 19:11:04 -0400 Subject: [PATCH 06/67] feat(flow-d): add dormant microvm block to hub agent-sandbox overlay Hub per-cluster overlay gets a microvm: block (enabled:false) alongside the kata nodepool block. Cluster/account-specific ARNs (baseImageARN, codeArtifactUri, ingress/egress connectors) left blank with guidance to fill before enabling. Flow D stays fully dormant on the hub until the Managed KRO + Managed ACK capabilities and the self-managed lambdamicrovms controller are in place. Verified full-chart render with the overlay: 0 Flow D objects when dormant (existing Kata objects intact), clean inventory when enabled (1 RGD + 2 SandboxTemplates + bridge SA/Role/RoleBinding/ConfigMap), valid YAML in both modes. --- .../hub/addons/agent-sandbox/values.yaml | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml index bf10ad47..2a436388 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml @@ -38,3 +38,26 @@ nodepool: # Adopt the existing LaunchTemplate in place (external-name). launchTemplateId: lt-0e204ea3e305e2e1f launchTemplateVersion: "$Latest" + +# ── Flow D — Lambda MicroVM substrate (hub) ────────────────────────────────── +# Alternative to the Kata `nodepool` above: run the coder in an AWS Lambda MicroVM +# instead of a Kata pod. DORMANT (enabled:false) until the hub has the Managed KRO +# + Managed ACK capabilities and the self-managed ack-lambdamicrovms controller +# (see gitops/addons/bootstrap/default/addons.yaml). The connector/image ARNs below +# are account/region specific and must be filled in before enabling — they are +# public identifiers, not secrets, but are left blank until the MicroVM base image +# + network connectors exist in the account. +microvm: + enabled: false + region: us-west-2 + apiGroup: sandbox.agents.x-k8s.io + bridgeImage: alpine/k8s:1.31.0 + # Built from the SAME dark-factory-coder image (coderTemplate.image) + entrypoint.js, + # published to the S3 codeArtifact bucket. Fill in when the base image exists. + baseImageARN: "" # arn:aws:lambdamicrovms:us-west-2:940019131157:image/ + codeArtifactUri: "" # s3:///dark-factory-coder/.zip + defaults: + maxIdleDurationSeconds: 900 + suspendedDurationSeconds: 300 + ingressConnectorARN: "" # arn:aws:lambdamicrovms:...:connector/ALL_INGRESS + egressConnectorARN: "" # arn:aws:lambdamicrovms:...:connector/INTERNET_EGRESS From 8c77e6024527d6b8f79ca6771fd3bc6b5bf675fa Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Wed, 29 Jul 2026 21:26:34 -0400 Subject: [PATCH 07/67] docs(flow-d): move ACK GA-list citation + capability-ownership split into Flow D The ACK GA-services-list reference and the pre-GA-vs-GA / self-managed rationale belong with the consumer (Flow D), not the platform capability-enablement doc. Add the authoritative ACK community services (GA list) link backing the 'GA upstream' claim, and clarify the ownership split: the Managed KRO + Managed ACK CAPABILITIES are enabled in appmod-blueprints (EKS Capability toggle, see its docs/EKS-Capabilities-KRO-ACK-Setup.md); THIS repo owns the self-managed lambdamicrovms controller + the KRO MicrovmSandbox RGD + the sandbox shim. --- docs/dark-factory/README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/dark-factory/README.md b/docs/dark-factory/README.md index c2ae0793..598f81a4 100644 --- a/docs/dark-factory/README.md +++ b/docs/dark-factory/README.md @@ -287,10 +287,16 @@ for other work; this substrate is Flow D.)* | **MicroVM primitives** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` + `Microvm` CRDs (`lambdamicrovms.services.k8s.aws/v1alpha1`) | > **Why self-managed for the MicroVM controller?** Managed ACK bundles only controllers whose service -> is **GA upstream**. `lambdamicrovms` is **pre-GA** (`v1alpha1`), so it isn't in Managed ACK yet — it +> is **GA upstream** (see the [ACK community services / GA list](https://aws-controllers-k8s.github.io/community/docs/community/services/)). +> `lambdamicrovms` is **pre-GA** (`v1alpha1`, not on that list), so it isn't in Managed ACK yet — it > runs as its own GitOps addon. **Managed ACK + self-managed lambdamicrovms coexist** (different CRD > groups → no conflict). When `lambdamicrovms` goes GA, delete the self-managed addon and Managed ACK > adopts it — **the RGD is unchanged**. This "install both now" posture is deliberate and futuristic. +> +> The **Managed KRO + Managed ACK capabilities themselves** are enabled at the platform layer in the +> **appmod-blueprints** repo (an EKS Capability toggle) — see that repo's +> `docs/EKS-Capabilities-KRO-ACK-Setup.md`. This repo owns only the **self-managed `lambdamicrovms` +> controller + the KRO `MicrovmSandbox` RGD + the sandbox shim** (Flow D). ### Platform-owned vs app-owned (encoded in the two ACK CRDs) From ddb2ef214a80700b65b09c6ebbc97f5ca256ceab Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 16:55:37 -0400 Subject: [PATCH 08/67] fix(flow-d): correct RGD to real Lambda MicroVM API (verified live) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Align the MicrovmSandbox RGD with the actual lambda-microvms API (verified against the live service + AWS docs in us-west-2): - IAM trust principal: lambda.amazonaws.com (+ sts:TagSession) — NOT the guessed lambdamicrovms.amazonaws.com. buildRole gets inline ECR-read + S3-read + logs (Lambda pulls the coder image from private ECR at build). - Drop network connectors entirely: Lambda MicroVMs have PUBLIC egress by default, which is all the coder needs (git/gh/registry). Ingress/VPC-egress connectors are optional add-ons (the latter created via a different service, aws lambda-core) — omitted from v1. Removes ingress/egressConnectorARN from schema + both resources. - codeArtifactUri: document it accepts an ECR image URI or S3 path (Flow D uses the arm64 dark-factory-coder ECR image directly). - base image ARN format corrected to arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64-only) in values guidance. Verified: helm template renders 5 resources, 0 connector refs, correct principal. --- .../templates/50-rgd-microvm-sandbox.yaml | 49 ++++++++++++------- .../addons/charts/agent-sandbox/values.yaml | 12 +++-- 2 files changed, 37 insertions(+), 24 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml index 1cf4ca4c..1166ad98 100644 --- a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml @@ -14,7 +14,13 @@ a consumer (the lambda-microvm SandboxTemplate bridge, or any agent) creates a s OWNERSHIP SPLIT is expressed IN THE SCHEMA: spec.image.* — PLATFORM-owned (base image, code artifact) — set once per image - spec.run.* — APP-owned (per-claim instance: idle policy, connectors, env) + spec.run.* — APP-owned (per-claim instance: idle policy) + +NETWORK: Lambda MicroVMs have PUBLIC internet egress by DEFAULT, so no network +connectors are attached here (the coder only needs outbound git/gh/registry, like +Flow A). Ingress connectors (AWS-managed, inbound HTTPS) and VPC egress connectors +(created out-of-band via `aws lambda-core create-network-connector`) are optional +add-ons a future variant can wire in; they are intentionally omitted from v1. CONTROLLER SPLIT: the s3/iam resources are reconciled by MANAGED ACK (GA controllers); MicrovmImage/Microvm by the SELF-MANAGED lambdamicrovms controller (pre-GA). The RGD is @@ -38,22 +44,22 @@ spec: spec: # ── PLATFORM-owned: the image / substrate (set once per image) ────────── image: - # ARN of the AWS-published base MicroVM image to build from. + # ARN of the AWS-published base MicroVM image to build from, e.g. + # arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64 — the only arch + # Lambda MicroVM supports). baseImageARN: string - # S3 URI of the coder code artifact baked into the image. This is the SAME - # dark-factory-coder payload used by Flow A — MicroVM substrate, same coder. + # URI of the coder code artifact for the image. Lambda MicroVM accepts an + # ECR image URI *or* an S3 path — Flow D points at the SAME dark-factory-coder + # (built for arm64), so no S3 repackaging is needed. codeArtifactUri: string - # Public-egress connector ARN the image build is allowed (platform policy). - egressConnectorARN: string | default="{{ .Values.microvm.defaults.egressConnectorARN }}" # ── APP-owned: the per-claim instance lifecycle ───────────────────────── run: # Idle policy — MicroVM auto-suspend/resume to keep idle cost near zero. + # (Declarative suspend/resume via Sandbox.operatingMode is driven by the + # microvm-lifecycle controller, not this field — see template 52.) autoResumeEnabled: boolean | default=true maxIdleDurationSeconds: integer | default={{ .Values.microvm.defaults.maxIdleDurationSeconds }} suspendedDurationSeconds: integer | default={{ .Values.microvm.defaults.suspendedDurationSeconds }} - # Connector ARNs governing the running MicroVM's network. - ingressConnectorARN: string | default="{{ .Values.microvm.defaults.ingressConnectorARN }}" - egressConnectorARN: string | default="{{ .Values.microvm.defaults.egressConnectorARN }}" # Common: AWS region + a name stem for the created resources. region: string | default="{{ .Values.microvm.region }}" name: string @@ -73,6 +79,9 @@ spec: spec: name: ${schema.spec.name}-microvm-artifacts # 2) IAM role the image BUILD assumes (GA — Managed ACK). Platform-owned. + # Lambda assumes this during create-microvm-image to pull the code artifact + # (ECR/S3) + write build logs. Trust = lambda.amazonaws.com (verified in the + # Lambda MicroVM getting-started docs), with inline ECR-read + S3-read + logs. - id: buildRole template: apiVersion: iam.services.k8s.aws/v1alpha1 @@ -81,10 +90,18 @@ spec: name: ${schema.spec.name}-microvm-build spec: name: ${schema.spec.name}-microvm-build - policyRefs: [] assumeRolePolicyDocument: | - {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambdamicrovms.amazonaws.com"},"Action":"sts:AssumeRole"}]} + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} + inlinePolicies: + microvm-build: | + {"Version":"2012-10-17","Statement":[ + {"Effect":"Allow","Action":["ecr:GetAuthorizationToken","ecr:BatchGetImage","ecr:GetDownloadUrlForLayer"],"Resource":"*"}, + {"Effect":"Allow","Action":["s3:GetObject"],"Resource":"arn:aws:s3:::*-microvm-artifacts/*"}, + {"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:*"} + ]} # 3) IAM role the RUNNING MicroVM assumes (GA — Managed ACK). App-owned exec identity. + # Minimal by default (the coder reaches models via Bifrost, git/gh via public + # egress — no AWS API needed); trust = lambda.amazonaws.com. - id: execRole template: apiVersion: iam.services.k8s.aws/v1alpha1 @@ -93,9 +110,8 @@ spec: name: ${schema.spec.name}-microvm-exec spec: name: ${schema.spec.name}-microvm-exec - policyRefs: [] assumeRolePolicyDocument: | - {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambdamicrovms.amazonaws.com"},"Action":"sts:AssumeRole"}]} + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} # 4) MicrovmImage (pre-GA — SELF-MANAGED lambdamicrovms controller). Platform-owned. - id: image template: @@ -109,8 +125,6 @@ spec: buildRoleARN: ${buildRole.status.ackResourceMetadata.arn} codeArtifact: uri: ${schema.spec.image.codeArtifactUri} - egressNetworkConnectors: - - ${schema.spec.image.egressConnectorARN} # 5) Microvm instance (pre-GA — SELF-MANAGED). App-owned lifecycle. Runs the # same dark-factory-coder entrypoint that Flow A's Kata pod runs. - id: microvm @@ -122,10 +136,7 @@ spec: spec: imageIdentifier: ${image.status.ackResourceMetadata.arn} executionRoleARN: ${execRole.status.ackResourceMetadata.arn} - ingressNetworkConnectors: - - ${schema.spec.run.ingressConnectorARN} - egressNetworkConnectors: - - ${schema.spec.run.egressConnectorARN} + # No network connectors → default public internet egress (see header). idlePolicy: autoResumeEnabled: ${schema.spec.run.autoResumeEnabled} maxIdleDurationSeconds: ${schema.spec.run.maxIdleDurationSeconds} diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index ce1ee7c4..1a27ec56 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -257,14 +257,16 @@ microvm: # + the AWS CLI (streams MicroVM logs). alpine/k8s bundles kubectl. bridgeImage: alpine/k8s:1.31.0 # The coder code-artifact + base image are cluster/account specific → overlay. + # Lambda MicroVM is ARM_64-only; codeArtifactUri accepts an ECR image URI or S3 path. # image: - # baseImageARN: arn:aws:lambdamicrovms:::image/ - # codeArtifactUri: s3:///dark-factory-coder/.zip + # baseImageARN: arn:aws:lambda::aws:microvm-image:al2023-1 + # codeArtifactUri: .dkr.ecr..amazonaws.com/dark-factory-coder:-arm64 baseImageARN: "" codeArtifactUri: "" - # Safe cluster-agnostic defaults for the RGD schema (overlay sets the ARNs). + # Idle policy defaults for the RGD schema (auto-suspend/resume). Declarative + # suspend/resume via Sandbox.operatingMode is handled by the microvm-lifecycle + # controller (templates/52), independent of these. No network connectors: + # Lambda MicroVMs default to public internet egress, which is all the coder needs. defaults: maxIdleDurationSeconds: 900 suspendedDurationSeconds: 300 - ingressConnectorARN: "" - egressConnectorARN: "" From a7c4b8af80040ab6cb5e94402b8c588ccfb63937 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 16:56:28 -0400 Subject: [PATCH 09/67] feat(flow-d): fill hub overlay with real MicroVM values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Set the hub Flow D overlay to the confirmed live values: - baseImageARN: arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1 (verified available; ARM_64-only). - codeArtifactUri: dark-factory-coder:v0.2.3-arm64 ECR URI (accepts ECR directly; arm64 tag to be built — MicroVM is ARM_64-only, existing amd64 tags untouched). - Drop the connector fields (MicroVMs default to public egress). Still enabled:false (dormant) until the ack-lambdamicrovms controller is synced. --- .../hub/addons/agent-sandbox/values.yaml | 22 +++++++++---------- 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml index 2a436388..07dbe51d 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml @@ -41,23 +41,21 @@ nodepool: # ── Flow D — Lambda MicroVM substrate (hub) ────────────────────────────────── # Alternative to the Kata `nodepool` above: run the coder in an AWS Lambda MicroVM -# instead of a Kata pod. DORMANT (enabled:false) until the hub has the Managed KRO -# + Managed ACK capabilities and the self-managed ack-lambdamicrovms controller -# (see gitops/addons/bootstrap/default/addons.yaml). The connector/image ARNs below -# are account/region specific and must be filled in before enabling — they are -# public identifiers, not secrets, but are left blank until the MicroVM base image -# + network connectors exist in the account. +# instead of a Kata pod. DORMANT (enabled:false) until the self-managed +# ack-lambdamicrovms controller is synced (see gitops/addons/bootstrap/default/ +# addons.yaml). Managed KRO + Managed ACK capabilities are ACTIVE on the hub. +# The ARNs below are public identifiers (not secrets). microvm: enabled: false region: us-west-2 apiGroup: sandbox.agents.x-k8s.io bridgeImage: alpine/k8s:1.31.0 - # Built from the SAME dark-factory-coder image (coderTemplate.image) + entrypoint.js, - # published to the S3 codeArtifact bucket. Fill in when the base image exists. - baseImageARN: "" # arn:aws:lambdamicrovms:us-west-2:940019131157:image/ - codeArtifactUri: "" # s3:///dark-factory-coder/.zip + # AWS-published base MicroVM image (ARM_64 — the only arch Lambda MicroVM supports). + baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" + # The SAME dark-factory-coder, built for arm64 (MicroVM is ARM_64-only) and pushed + # to ECR. codeArtifactUri accepts an ECR image URI directly — no S3 repackaging. + # NOTE: fill the real arm64 tag once built+pushed (task: arm64 coder image). + codeArtifactUri: "940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:v0.2.3-arm64" defaults: maxIdleDurationSeconds: 900 suspendedDurationSeconds: 300 - ingressConnectorARN: "" # arn:aws:lambdamicrovms:...:connector/ALL_INGRESS - egressConnectorARN: "" # arn:aws:lambdamicrovms:...:connector/INTERNET_EGRESS From 8582eb4acfd7e835faf025e98fe7728e770811f9 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 17:00:26 -0400 Subject: [PATCH 10/67] feat(flow-d): add suspend/resume via Sandbox.operatingMode (microvm-lifecycle) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the REQUIRED declarative suspend/resume: Sandbox.spec.operatingMode has {Running,Suspended} but the ACK Microvm CR has no suspend field (suspend/resume are imperative SDK ops the controller doesn't reconcile). Close the gap with a tiny always-on reconcile loop (templates/52) — pure shim, no virtual-kubelet, no new image: - microvm-lifecycle Deployment (alpine/k8s, ConfigMap reconcile.sh, matches pool-manager pattern): watches Sandboxes labeled substrate=lambda-microvm; on operatingMode Running->Suspended calls suspend-microvm, Suspended->Running calls resume-microvm, by microvmID resolved from the MicrovmSandbox status. Idempotent via a last-mode annotation. + SA/Role/RoleBinding (pod-identity auth). - Bridge (51) made suspend-aware: labels its MicrovmSandbox agent-sandbox.io/sandbox for correlation; preStop checks operatingMode and, if Suspended, writes a marker so cleanup() KEEPS the MicrovmSandbox (VM survives suspend); real teardown only on claim end. Create is idempotent (resume-or-create). +sandboxes get RBAC. - values: microvm.lifecycle.intervalSeconds (15s). Robust vs preStop-only: survives pod/node loss; resume needs no live pod. Verified: 0 objects disabled; 10 when enabled; reconcile.sh + bridge.sh pass sh -n. --- .../templates/51-sandboxtemplate-microvm.yaml | 41 +++- .../templates/52-microvm-lifecycle.yaml | 175 ++++++++++++++++++ .../addons/charts/agent-sandbox/values.yaml | 4 + 3 files changed, 216 insertions(+), 4 deletions(-) create mode 100644 gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml diff --git a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml index 7fd93ef0..65fd2b9b 100644 --- a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml @@ -38,10 +38,14 @@ metadata: labels: {{- include "agent-sandbox.labels" . | nindent 4 }} rules: - # The bridge creates/reads/deletes MicrovmSandbox CRs (the KRO composite) only. + # The bridge creates/reads/deletes MicrovmSandbox CRs (the KRO composite). - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] resources: ["microvmsandboxes"] verbs: ["create", "get", "list", "watch", "delete"] + # Read its owning Sandbox in preStop to tell a suspend from a real teardown. + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding @@ -75,16 +79,29 @@ data: echo "[microvm-bridge] idle — waiting for a SandboxClaim to inject DF_ISSUE_NUMBER..." while [ -z "${DF_ISSUE_NUMBER:-}" ]; do sleep 5; done NAME="df-${DF_ISSUE_NUMBER}" - echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} -> creating MicrovmSandbox/${NAME}" + # SANDBOX_NAME is injected via the downward API (the owning Sandbox CR's name) so + # the microvm-lifecycle controller can correlate this MicrovmSandbox to the Sandbox + # whose operatingMode it watches for suspend/resume. + SANDBOX_NAME="${SANDBOX_NAME:-$NAME}" + echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} (sandbox=${SANDBOX_NAME}) -> ensuring MicrovmSandbox/${NAME}" - # Delete the MicrovmSandbox on exit -> KRO cascades TerminateMicrovm + cleans - # the S3/IAM resources. Mirrors Flow A's onExit teardown. + # Teardown on exit ONLY on real claim end, not on a suspend (suspend deletes the + # pod but must KEEP the MicrovmSandbox so resume works). We distinguish via a + # marker file the pod writes when it sees operatingMode=Suspended. Default = + # teardown (claim ended / crash) -> KRO cascades TerminateMicrovm + cleans S3/IAM. cleanup() { + if [ -f /tmp/suspending ]; then + echo "[microvm-bridge] pod stopping for SUSPEND — keeping MicrovmSandbox/${NAME}" + return + fi echo "[microvm-bridge] tearing down MicrovmSandbox/${NAME}" kubectl delete microvmsandbox "${NAME}" --ignore-not-found --wait=false || true } trap cleanup EXIT INT TERM + # Idempotent create/adopt (resume-or-create): if the MicrovmSandbox already exists + # from a prior suspend cycle, kubectl apply is a no-op and we just re-attach. The + # microvm-lifecycle controller handles the actual resume-microvm on operatingMode. cat </dev/null || echo "") + [ "$M" = "Suspended" ] && touch /tmp/suspending || true securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true diff --git a/gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml new file mode 100644 index 00000000..d90f4860 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml @@ -0,0 +1,175 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — microvm-lifecycle controller (suspend/resume via Sandbox.operatingMode). + +WHY THIS EXISTS: the Agent Sandbox CRD exposes `spec.operatingMode ∈ {Running, +Suspended}` (the declarative suspend/resume intent), but the ACK Microvm CR has NO +suspend field — its spec is create-time only; suspend/resume are IMPERATIVE SDK ops +(`suspend-microvm`/`resume-microvm`) that the ACK controller deliberately does NOT +reconcile. So switching operatingMode does nothing on its own. This tiny always-on +reconcile loop closes that gap: + + Sandbox.operatingMode: Running -> Suspended : aws lambda-microvms suspend-microvm + Sandbox.operatingMode: Suspended -> Running : aws lambda-microvms resume-microvm + +It resolves the MicroVM id from the MicrovmSandbox (KRO) status (microvmID). The +MicrovmSandbox is NOT deleted on suspend (only on claim teardown -> terminate), so +the VM survives suspend/resume cycles. This is a reconcile loop (not preStop hooks) +so it is robust to pod/node loss and resume needs no live pod — pure shim, no +virtual-kubelet, no new image (alpine/k8s = kubectl + aws cli), matching the +pool-manager/bridge pattern. Auth via EKS Pod Identity (empty SA annotations; the +pod-identity association granting lambda-microvms Suspend/Resume/Get is created by +the platform). +*/ -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +rules: + # Read Sandboxes (watch operatingMode) + the MicrovmSandbox composites (read microvmID). + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["get", "list", "watch"] + - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] + resources: ["microvmsandboxes"] + verbs: ["get", "list", "watch"] + # Record the last-applied mode on the Sandbox (annotation) to detect transitions. + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["patch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: microvm-lifecycle +subjects: + - kind: ServiceAccount + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: microvm-lifecycle-script + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +data: + reconcile.sh: | + #!/bin/sh + # Reconcile Sandbox.operatingMode -> Lambda MicroVM suspend/resume. + # Idempotent: we stamp the last-acted mode on an annotation and only act on change. + set -eu + NS="{{ include "agent-sandbox.namespace" . }}" + REGION="{{ .Values.microvm.region }}" + APIGROUP="{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" }}" + ANN="microvm-lifecycle.agents.x-k8s.io/last-mode" + INTERVAL="{{ .Values.microvm.lifecycle.intervalSeconds | default 15 }}" + echo "[microvm-lifecycle] reconciling every ${INTERVAL}s (ns=${NS} region=${REGION})" + while true; do + # Only Sandboxes on the lambda-microvm substrate (bridge sets this label). + for sb in $(kubectl get sandbox -n "$NS" \ + -l agent-sandbox.io/substrate=lambda-microvm \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do + MODE=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "Running") + LAST=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath="{.metadata.annotations.${ANN}}" 2>/dev/null || echo "") + [ "$MODE" = "$LAST" ] && continue # no transition + # Resolve the MicroVM id from the matching MicrovmSandbox (name = df-) via its status. + VMID=$(kubectl get microvmsandbox -n "$NS" \ + -o jsonpath="{range .items[?(@.metadata.labels.agent-sandbox\.io/sandbox=='${sb}')]}{.status.microvmID}{end}" 2>/dev/null || echo "") + # Fallback: single-claim naming — try the sandbox name directly. + [ -z "$VMID" ] && VMID=$(kubectl get microvmsandbox "$sb" -n "$NS" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + if [ -z "$VMID" ]; then + echo "[microvm-lifecycle] $sb: mode=$MODE but no microvmID yet — will retry" + continue + fi + case "$MODE" in + Suspended) + echo "[microvm-lifecycle] $sb: Running->Suspended -> suspend-microvm $VMID" + aws lambda-microvms suspend-microvm --microvm-identifier "$VMID" --region "$REGION" 2>&1 || true + ;; + Running) + echo "[microvm-lifecycle] $sb: Suspended->Running -> resume-microvm $VMID" + aws lambda-microvms resume-microvm --microvm-identifier "$VMID" --region "$REGION" 2>&1 || true + ;; + *) + echo "[microvm-lifecycle] $sb: unknown operatingMode '$MODE' — skipping"; continue ;; + esac + # Stamp the mode we acted on so we don't repeat the call. + kubectl annotate sandbox "$sb" -n "$NS" "${ANN}=${MODE}" --overwrite >/dev/null 2>&1 || true + done + sleep "$INTERVAL" + done +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + replicas: 1 + selector: + matchLabels: + {{- include "agent-sandbox.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: microvm-lifecycle + template: + metadata: + labels: + {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: microvm-lifecycle + spec: + serviceAccountName: microvm-lifecycle + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: lifecycle + image: {{ .Values.microvm.bridgeImage }} + command: ["/bin/sh", "/scripts/reconcile.sh"] + env: + - name: AWS_REGION + value: {{ .Values.microvm.region | quote }} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: { cpu: 25m, memory: 64Mi } + limits: { cpu: 100m, memory: 128Mi } + volumeMounts: + - name: script + mountPath: /scripts + - name: tmp + mountPath: /tmp + volumes: + - name: script + configMap: + name: microvm-lifecycle-script + defaultMode: 0555 + - name: tmp + emptyDir: {} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index 1a27ec56..9bf3ed43 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -270,3 +270,7 @@ microvm: defaults: maxIdleDurationSeconds: 900 suspendedDurationSeconds: 300 + # microvm-lifecycle controller (templates/52) — reconciles Sandbox.operatingMode + # -> suspend/resume-microvm. How often the loop runs. + lifecycle: + intervalSeconds: 15 From fb3e3bfcbb3c8614c6b3c57e2b00896f637945c3 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 17:01:46 -0400 Subject: [PATCH 11/67] docs(flow-d): document Sandbox.operatingMode suspend/resume mechanism MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add §D.3a (diagram doc) + a Suspend/resume subsection (README §4.5) describing the microvm-lifecycle reconcile loop that translates Sandbox.operatingMode -> Lambda MicroVM suspend/resume-microvm, why it's needed (ACK Microvm has no suspend field), and why a reconcile loop beats preStop hooks (robust to pod/node loss). Matches the templates/52 implementation. --- docs/dark-factory/README.md | 10 +++++++ .../diagrams/flow-d-microvm-sandbox.md | 29 +++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/docs/dark-factory/README.md b/docs/dark-factory/README.md index 598f81a4..cd563f4d 100644 --- a/docs/dark-factory/README.md +++ b/docs/dark-factory/README.md @@ -320,6 +320,16 @@ then **streams the MicroVM's logs into the pod** and maps lifecycle (pod Running pod exit → `TerminateMicrovm`). To Flow B and the user the UX is identical to Flow A. Interactive exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up. +### Suspend / resume (Sandbox CRD) + +Because the substrate is a Lambda MicroVM (not a pod), Flow D gives you **declarative +suspend/resume through the Agent Sandbox CRD**: set `Sandbox.spec.operatingMode: Suspended` and a +small **`microvm-lifecycle`** reconcile loop calls `suspend-microvm` (and `resume-microvm` on +`Running`) by the MicroVM id — the VM's state is retained across the cycle, and the `MicrovmSandbox` is +kept (torn down only on real claim end). The ACK `Microvm` CR has no suspend field, so this loop +supplies the missing intent→SDK translation — pure shim, no virtual-kubelet. See +[`diagrams/flow-d-microvm-sandbox.md` §D.3a](diagrams/flow-d-microvm-sandbox.md). + ### Delivery & status Shipped as GitOps, **disabled by default** (like the kata nodepool): a `microvm:` values block gates diff --git a/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md index 25013226..0eadcd25 100644 --- a/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md +++ b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md @@ -100,6 +100,35 @@ exec/attach passthrough is **best-effort** (full fidelity would need virtual-kub --- +## D.3a — Suspend / resume (Sandbox.operatingMode → MicroVM) + +The Agent Sandbox CRD exposes `spec.operatingMode ∈ {Running, Suspended}` — the declarative +suspend/resume intent. But the ACK `Microvm` CR has **no suspend field**: its spec is create-time +only, `State` is status-only, and `suspend-microvm`/`resume-microvm` are **imperative SDK ops the ACK +controller deliberately does not reconcile**. So flipping `operatingMode` does nothing on its own — a +controller must translate intent into the SDK call. + +Flow D closes that gap with a tiny always-on **`microvm-lifecycle`** reconcile loop (a ConfigMap +script on `alpine/k8s`, same pattern as the pool-manager — **no virtual-kubelet, no new image**): + +``` +Sandbox.operatingMode: Running → Suspended : aws lambda-microvms suspend-microvm --microvm-identifier +Sandbox.operatingMode: Suspended → Running : aws lambda-microvms resume-microvm --microvm-identifier +``` + +- `` (the `microvmID`) is resolved from the `MicrovmSandbox` (KRO) status; the loop is idempotent + (stamps a `last-mode` annotation, acts only on transitions). +- The `MicrovmSandbox` is **kept** across suspend (the bridge's `preStop` detects `operatingMode: + Suspended` and skips teardown), so the VM survives suspend/resume; it's deleted only on real claim + teardown → `TerminateMicrovm`. +- Chosen over bridge `preStop` hooks alone because a reconcile loop is **robust to pod/node loss** and + resume needs no live pod. This is the open-source **Sandbox-CRD-driven** suspend/resume you get with + the MicroVM substrate. + +*Edit: `src/flow-d-suspend-resume.drawio` → `img/flow-d-suspend-resume.png`.* + +--- + ## D.4 — Future: when `lambdamicrovms` goes GA `lambdamicrovms` is currently **pre-GA** (`v1alpha1`), so its controller is self-managed. When it From 72f2541a7a3e5ce53592567c9a772aea07511bcb Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 17:50:21 -0400 Subject: [PATCH 12/67] feat(coder): emit a PR change-description alongside verification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per reviewer request: the coder now writes a concise, reviewer-facing description of WHAT it changed (and why) to artifacts/description.md as part of its run, and the PR body renders it as a '### 📝 Changes' section ABOVE the existing verification block. Falls back to a neutral line if absent so the body is never empty. - claude path: description instruction appended to the prompt. - kiro path: same instruction appended to SPEC.md (kiro reads --spec). - PR body: new Changes section placed before the marker, so status.js (which rewrites from the marker down) preserves it on every update. node --check passes. --- examples/dark-factory/coder/entrypoint.js | 24 ++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/examples/dark-factory/coder/entrypoint.js b/examples/dark-factory/coder/entrypoint.js index 79b421cf..d7e87bc0 100644 --- a/examples/dark-factory/coder/entrypoint.js +++ b/examples/dark-factory/coder/entrypoint.js @@ -256,11 +256,25 @@ function runCoder(repoDir) { // Inherit stdio so the coder CLI's own output + errors stream into the pod // logs (kubectl logs), instead of being swallowed by execFileSync's exception. const opts = { cwd: repoDir, env, stdio: "inherit", maxBuffer: 64 * 1024 * 1024 }; - const prompt = `Implement the change described in ${WORKSPACE}/SPEC.md. Build and run unit tests until green. Commit your work.`; + // The coder also writes a concise, human-readable summary of WHAT it changed to + // artifacts/description.md — this becomes the "Changes" section of the PR body + // (in addition to the verification section). Keep it short: what changed + why, + // as reviewer-facing markdown bullets. + const descPath = `${WORKSPACE}/artifacts/description.md`; + const prompt = + `Implement the change described in ${WORKSPACE}/SPEC.md. Build and run unit tests until green. Commit your work. ` + + `Then write a concise description of the changes you made (what changed and why, as a few markdown bullet points, ` + + `reviewer-facing — no preamble) to ${descPath}.`; if (ENGINE === "kiro") { // Kiro CLI headless — the coder image carries the `kiro` binary; it reads the // same Bifrost/Bedrock env above. --headless drives it non-interactively. + // Append the description instruction so kiro produces the same artifact. console.log("[coder] engine=kiro (kiro run --headless)"); + try { + fs.appendFileSync(`${WORKSPACE}/SPEC.md`, + `\n\n---\n\n## After implementing\n\nWrite a concise description of the changes you made ` + + `(what changed and why, a few reviewer-facing markdown bullets, no preamble) to ${descPath}.\n`); + } catch { /* non-fatal */ } return execFileSync("kiro", ["run", "--headless", "--spec", `${WORKSPACE}/SPEC.md`], opts); } console.log("[coder] engine=claude (claude -p)"); @@ -429,9 +443,17 @@ async function main() { ? "- ⏳ **Security review (AWS Security Agent):** _queued (DevOps cleared)…_" : "- ⬜ **Security review (AWS Security Agent):** _waiting on DevOps clearance_"; } + // Coder-authored description of the changes (artifacts/description.md). Shown + // as a "Changes" section ahead of the verification block. Falls back to a + // neutral line if the coder didn't produce one, so the PR body is never empty. + const desc = readSecret(`${WORKSPACE}/artifacts/description.md`); + const changesSection = desc + ? ["### 📝 Changes", "", desc, ""] + : ["### 📝 Changes", "", "_Implemented per the linked issue; see the diff for details._", ""]; const prBody = [ `Closes #${ISSUE}.`, "", + ...changesSection, "", "### 🏭 Dark Factory — verification", `- ✅ **Build + unit tests:** ${test.summary}`, From 6078c1882d3d70c8742dcc6551eaa3a41aee2d6f Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 17:53:21 -0400 Subject: [PATCH 13/67] fix(security-agent): early-exit when findings ready + graceful timeout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes for the 'security review waits ~15 min while the PR already shows a result' problem: 1. Early-exit: the code-review job's status field flips to COMPLETED long AFTER the AWS Security Agent App bot has already posted its findings on the PR. So in addition to polling status, each iteration now probes list-findings; the moment it returns a well-formed result (findingsSummaries present) we proceed — no waiting for the status flip. Reuses those findings for the report (no re-fetch). 2. Graceful timeout: this step is advisory (the App bot posts the authoritative result regardless), so exceeding the poll timeout now posts a neutral 'pending' status + a pointer to the bot comment, instead of a misleading red 'error'. This also fixes the stale sticky board: the sticky-status step depends on security-agent (runs once, after it), so a fast security step means the board is rewritten with accurate rows within seconds instead of ~15 min later. sh -n passes. --- .../dark-factory/scripts/security-agent.sh | 38 ++++++++++++++++--- 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/gitops/addons/charts/dark-factory/scripts/security-agent.sh b/gitops/addons/charts/dark-factory/scripts/security-agent.sh index 8dc938bb..9601df42 100644 --- a/gitops/addons/charts/dark-factory/scripts/security-agent.sh +++ b/gitops/addons/charts/dark-factory/scripts/security-agent.sh @@ -95,27 +95,53 @@ JOBID="$(aws securityagent start-code-review-job --region "$AWS_REGION" \ log "codeReviewJobId=${JOBID} — polling (timeout ${POLL_TIMEOUT}s)..." # ── 4. Poll to completion ──────────────────────────────────────────────────── +# The job's `status` field lags well behind the actual analysis: the AWS Security +# Agent GitHub App posts its findings comment on the PR (e.g. "No issues identified") +# minutes before batch-get-code-review-jobs flips to COMPLETED. Polling status alone +# therefore blocks this (advisory) step for the full timeout even though the result +# is already known. So we ALSO probe list-findings each iteration: once it returns a +# well-formed result (findingsSummaries key present), the review has produced output +# and we can proceed immediately — this is the early-exit that avoids the long wait. DEADLINE=$(( $(date +%s) + POLL_TIMEOUT )) STATUS="IN_PROGRESS" +DONE="" +: > "$WORK/findings.json" while [ "$(date +%s)" -lt "$DEADLINE" ]; do STATUS="$(aws securityagent batch-get-code-review-jobs --region "$AWS_REGION" \ --agent-space-id "$AGENT_SPACE_ID" --code-review-job-ids "$JOBID" \ --query 'codeReviewJobs[0].status' --output text 2>/dev/null || echo IN_PROGRESS)" log "job status=${STATUS}" case "$STATUS" in - COMPLETED|SUCCEEDED) break ;; + COMPLETED|SUCCEEDED) DONE="status"; break ;; FAILED|STOPPED|ERROR) log "review job ${STATUS}"; post_status "error" "security: review job ${STATUS}"; exit 0 ;; esac + # Early-exit: findings ready before status flips? (App bot already posted them.) + if aws securityagent list-findings --region "$AWS_REGION" \ + --agent-space-id "$AGENT_SPACE_ID" --code-review-job-id "$JOBID" \ + > "$WORK/findings.try.json" 2>/dev/null \ + && python3 -c 'import json,sys; sys.exit(0 if "findingsSummaries" in json.load(open(sys.argv[1])) else 1)' "$WORK/findings.try.json" 2>/dev/null; then + mv "$WORK/findings.try.json" "$WORK/findings.json" + DONE="findings"; log "findings ready (status=${STATUS}) — proceeding without waiting for status flip"; break + fi sleep 20 done -if [ "$STATUS" != "COMPLETED" ] && [ "$STATUS" != "SUCCEEDED" ]; then - log "timed out waiting for review (last=${STATUS})"; post_status "error" "security: review timed out"; exit 0 +if [ -z "$DONE" ]; then + # Advisory step: the App bot posts the authoritative result on the PR regardless, + # so a slow job-status flip is NOT a failure. Post a neutral pending status (not + # error) so the PR check isn't a misleading red, and continue. + log "review still running past ${POLL_TIMEOUT}s (last status=${STATUS}) — see the AWS Security Agent bot comment on the PR for the authoritative result" + post_status "pending" "security: review still running — see AWS Security Agent PR comment" + exit 0 fi # ── 5. Fetch findings, render report + verdict (python3, no node) ──────────── -aws securityagent list-findings --region "$AWS_REGION" \ - --agent-space-id "$AGENT_SPACE_ID" --code-review-job-id "$JOBID" \ - > "$WORK/findings.json" 2>/dev/null || echo '{"findingsSummaries":[]}' > "$WORK/findings.json" +# Reuse the findings we already fetched during the early-exit probe; only re-fetch +# if we broke on the status flip (findings not yet captured). +if [ "$DONE" = "status" ] || [ ! -s "$WORK/findings.json" ]; then + aws securityagent list-findings --region "$AWS_REGION" \ + --agent-space-id "$AGENT_SPACE_ID" --code-review-job-id "$JOBID" \ + > "$WORK/findings.json" 2>/dev/null || echo '{"findingsSummaries":[]}' > "$WORK/findings.json" +fi python3 - "$WORK/findings.json" "$BLOCK_LEVEL" "$WORK/report.md" > "$WORK/verdict.env" <<'PY' import json, sys From 0a3d7c095fcfe1691d77695b1f110c5b2e12652d Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 21:14:22 -0400 Subject: [PATCH 14/67] =?UTF-8?q?feat(flow-d):=20label-branched=20pipeline?= =?UTF-8?q?=20=E2=80=94=20darkfactory-lambda=20->=20MicroVM=20substrate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One pipeline, substrate chosen by label (per design decision): - sensor: accept BOTH dark-factory (Kata) + darkfactory-lambda (MicroVM) labels; pass the firing label to df-run as the trigger-label param (index 6). - df-run: new trigger-label param (default dark-factory); claim-sandbox takes a warm-pool input; the DAG computes it via an Argo expression (darkfactory-lambda -> coder-warmpool-microvm, else coder-warmpool). Everything downstream (coder, holdout, devops, security, PR, merge, teardown) is identical. - agent-sandbox: add the Lambda-MicroVM SandboxWarmPool (claim.warmPoolRef is required, so Flow D needs its own pool) referencing the -microvm SandboxTemplate; microvm.warmPool values (name, targetIdle). dark-factory: warmPool.lambdaName. Verified: df-run + sensor render to valid YAML; warm-pool expression resolves correctly; lambda warmpool renders only when microvm.enabled (0 when disabled). --- .../templates/51-sandboxtemplate-microvm.yaml | 19 +++++++++++++++ .../addons/charts/agent-sandbox/values.yaml | 6 +++++ .../templates/20-workflowtemplate-df-run.yaml | 23 ++++++++++++++++++- .../dark-factory/templates/42-sensor.yaml | 15 ++++++++++-- gitops/addons/charts/dark-factory/values.yaml | 6 ++++- 5 files changed, 65 insertions(+), 4 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml index 65fd2b9b..9733051d 100644 --- a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml @@ -207,4 +207,23 @@ spec: defaultMode: 0555 - name: tmp emptyDir: {} +--- +{{- /* +Lambda-MicroVM SandboxWarmPool. SandboxClaim.spec.warmPoolRef is REQUIRED (a claim +can't bind a bare template), so Flow D needs its own pool the df-run claim step +targets when the darkfactory-lambda label fires. Kept small (bridge pods are tiny); +each idle member is a bridge waiting to apply a MicrovmSandbox on claim. +*/ -}} +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: {{ .Values.microvm.warmPool.name | default "coder-warmpool-microvm" }} + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + replicas: {{ .Values.microvm.warmPool.targetIdle | default 1 }} + sandboxTemplateRef: + name: {{ .Values.warmPool.templateName }}-microvm {{- end }} diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index 9bf3ed43..dac88daa 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -274,3 +274,9 @@ microvm: # -> suspend/resume-microvm. How often the loop runs. lifecycle: intervalSeconds: 15 + # Lambda-MicroVM warm pool the df-run claim binds when the darkfactory-lambda + # label fires (Flow D). Small — bridge pods are tiny (the real coder runs in the + # MicroVM, not the pod). templateName is -microvm. + warmPool: + name: coder-warmpool-microvm + targetIdle: 1 diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index ee530955..d789a2d1 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -47,6 +47,13 @@ spec: # existing branch to address the feedback. - name: iterate-note value: "" + # Which label fired the run → selects the coder substrate: + # dark-factory → Kata micro-VM (default; Flow B) + # darkfactory-lambda → Lambda MicroVM (Flow D) + # The claim step branches on this. Defaults to dark-factory so manual / + # df-iterate submits (which don't pass it) stay on the Kata path. + - name: trigger-label + value: "dark-factory" # Always release the claimed sandbox, on success OR failure. onExit: teardown ttlStrategy: @@ -81,6 +88,11 @@ spec: tasks: - name: claim template: claim-sandbox + arguments: + parameters: + # darkfactory-lambda → Lambda-MicroVM pool; else the Kata pool. + - name: warm-pool + value: "{{`{{=workflow.parameters['trigger-label'] == 'darkfactory-lambda' ? '`}}{{ .Values.warmPool.lambdaName | default "coder-warmpool-microvm" }}{{`' : '`}}{{ .Values.warmPool.name }}{{`'}}`}}" - name: drive-coder template: await-coder dependencies: [claim] @@ -164,6 +176,12 @@ spec: # ---- 1. Claim a warm sandbox (creates the SandboxClaim with issue env) ---- - name: claim-sandbox + inputs: + parameters: + # Which SandboxWarmPool to claim from (Kata vs Lambda-MicroVM). The DAG + # computes this from the trigger-label; defaults to the Kata pool. + - name: warm-pool + value: {{ .Values.warmPool.name }} outputs: parameters: - name: sandbox @@ -188,8 +206,11 @@ spec: dark-factory.io/issue-number: "{{`{{workflow.parameters.issue-number}}`}}" dark-factory.io/managed-by: df-run spec: + # Substrate branch: the claim binds whichever warm pool the DAG passes + # in (Kata coder-warmpool by default; the Lambda-MicroVM pool when the + # darkfactory-lambda label fired). Everything else is identical. warmPoolRef: - name: {{ .Values.warmPool.name }} + name: "{{`{{inputs.parameters.warm-pool}}`}}" lifecycle: ttlSecondsAfterFinished: {{ .Values.claimTtlSeconds }} env: diff --git a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml index 985f8331..4bc5ef76 100644 --- a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml +++ b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml @@ -64,7 +64,11 @@ spec: eventName: dark-factory filters: data: - # Only fire when the `dark-factory` label was ADDED to an issue. + # Fire when EITHER dark-factory label is added: + # dark-factory → Kata micro-VM substrate (Flow B, default) + # darkfactory-lambda → Lambda MicroVM substrate (Flow D) + # The label chosen is passed to df-run as the `substrate` param, which + # branches the claim step. Everything else in the pipeline is identical. - path: headers.X-Github-Event type: string value: ["issues"] @@ -73,7 +77,7 @@ spec: value: ["labeled"] - path: body.label.name type: string - value: ["dark-factory"] + value: ["dark-factory", "darkfactory-lambda"] - name: pr-approved eventSourceName: dark-factory-github eventName: dark-factory @@ -154,6 +158,9 @@ spec: - name: issue-title - name: issue-body - name: base-branch + # The label that fired: "dark-factory" or "darkfactory-lambda". + # df-run maps this to the substrate (kata vs lambda-microvm). + - name: trigger-label # Map GitHub webhook fields → workflow parameters. parameters: # Deterministic workflow name = df-run- (the dedup key). @@ -188,6 +195,10 @@ spec: dependencyName: issue-labeled dataKey: body.repository.default_branch dest: spec.arguments.parameters.5.value + - src: + dependencyName: issue-labeled + dataKey: body.label.name + dest: spec.arguments.parameters.6.value # ---- PR review approved → df-merge-teardown (the ONLY merge path) ---- - template: diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 06503b65..10f5f7bd 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -11,8 +11,12 @@ argo: namespace: argo warmPool: - # The Flow A SandboxWarmPool the factory claims from. + # The Flow A SandboxWarmPool the factory claims from (Kata substrate, default). name: coder-warmpool + # Flow D: the Lambda-MicroVM SandboxWarmPool the factory claims from when the + # darkfactory-lambda label fires (coder runs in a MicroVM). Must match + # agent-sandbox microvm.warmPool.name. + lambdaName: coder-warmpool-microvm # Namespace of the pool / where claims + coder pods are created. namespace: agent-sandbox-system From 05a07d4ed91abb0faac0bf9e38d37ebc76e6d910 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Thu, 30 Jul 2026 21:20:31 -0400 Subject: [PATCH 15/67] =?UTF-8?q?feat(flow-d):=20suspend/resume=20demo=20?= =?UTF-8?q?=E2=80=94=20suspend=20MicroVM=20after=20coder=20pushes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flow D visible suspend/resume via Sandbox.operatingMode (per design decision): - New microvm-suspend DAG step (gated microvmSuspend.enabled + lambda substrate + PR exists): after drive-coder, flips Sandbox.spec.operatingMode=Suspended so the agent-sandbox microvm-lifecycle controller calls suspend-microvm — compute freed while the gates run (coder is idle during review). No-op for Kata. - microvm-set-mode script template: patches operatingMode, best-effort verifies the MicrovmSandbox microvmID (advisory). RBAC: workflow SA gets sandboxes 'patch'. - Resume: happens naturally — df-iterate (PR comment) needs the VM, and the MicroVM idlePolicy.autoResumeEnabled resumes on next request; the lifecycle controller also resumes on operatingMode=Running. - values: microvmSuspend {enabled, image}. Verified: full chart renders valid YAML; suspend step present when enabled, absent when disabled. --- .../dark-factory/templates/10-rbac.yaml | 3 +- .../templates/20-workflowtemplate-df-run.yaml | 51 +++++++++++++++++++ gitops/addons/charts/dark-factory/values.yaml | 9 ++++ 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml index 4b94ec11..56724217 100644 --- a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml +++ b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml @@ -76,7 +76,8 @@ rules: verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: ["agents.x-k8s.io"] resources: ["sandboxes"] - verbs: ["get", "list", "watch"] + # patch: Flow D suspend/resume flips Sandbox.spec.operatingMode from the workflow. + verbs: ["get", "list", "watch", "patch"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index d789a2d1..28533f50 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -100,6 +100,24 @@ spec: parameters: - name: sandbox value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" +{{- if .Values.microvmSuspend.enabled }} + # Flow D ONLY — suspend the Lambda MicroVM after the coder pushes its + # branch: flip Sandbox.operatingMode=Suspended so the microvm-lifecycle + # controller calls suspend-microvm. The VM's state is retained (snapshot) + # and compute is freed while the (potentially slow) gates run — the + # coder is idle during review anyway. df-iterate resumes it on a comment. + # Gated on the lambda substrate + a PR existing; a no-op for Kata. + - name: microvm-suspend + template: microvm-set-mode + dependencies: [drive-coder] + when: "{{`{{workflow.parameters.trigger-label}}`}} == darkfactory-lambda && {{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: sandbox + value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" + - name: mode + value: "Suspended" +{{- end }} {{- if .Values.holdout.enabled }} # P2 — holdout gate: hidden scenarios + a different-family judge. - name: holdout-gate @@ -235,6 +253,39 @@ spec: - { containerName: coder, name: DF_DEVOPS_CLEAR_LABEL, value: "{{ .Values.review.handoffLabel }}" } {{- end }} +{{- if .Values.microvmSuspend.enabled }} + # ---- Flow D: set Sandbox.operatingMode (suspend/resume the Lambda MicroVM) ---- + # Flips spec.operatingMode; the agent-sandbox microvm-lifecycle controller + # observes the change and calls suspend-microvm / resume-microvm by microvmID. + # We then best-effort verify the MicroVM reached the expected state via the API + # (advisory — never fails the run). Only invoked on the lambda substrate. + - name: microvm-set-mode + inputs: + parameters: + - name: sandbox + - name: mode # Running | Suspended + script: + image: {{ .Values.microvmSuspend.image | default "alpine/k8s:1.31.0" }} + command: [sh] + source: | + set -eu + SB="{{`{{inputs.parameters.sandbox}}`}}" + MODE="{{`{{inputs.parameters.mode}}`}}" + NS="{{ .Values.warmPool.namespace }}" + echo "[microvm-set-mode] Sandbox/$SB -> operatingMode=$MODE" + kubectl patch sandbox "$SB" -n "$NS" --type merge -p "{\"spec\":{\"operatingMode\":\"$MODE\"}}" + # Best-effort verify the microvm-lifecycle controller acted (poll the + # MicrovmSandbox status the bridge created). Advisory only. + EXPECT="RUNNING"; [ "$MODE" = "Suspended" ] && EXPECT="SUSPENDED" + i=0; while [ "$i" -lt 12 ]; do + VMID=$(kubectl get microvmsandbox -n "$NS" -o jsonpath="{range .items[?(@.metadata.labels.agent-sandbox\.io/sandbox=='$SB')]}{.status.microvmID}{end}" 2>/dev/null || echo "") + [ -n "$VMID" ] && { echo "[microvm-set-mode] microvmID=$VMID (lifecycle controller will drive $EXPECT)"; break; } + i=$((i+1)); sleep 5 + done + echo "[microvm-set-mode] done (advisory)." + activeDeadlineSeconds: 180 +{{- end }} + # ---- 2. Wait for the coder VM to finish — GitHub is the completion bus ---- # The coder image (baked into the SandboxTemplate) auto-runs on VM start, # reads DF_* env, implements + tests, pushes df/issue-N, and opens the PR diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 10f5f7bd..208ed3b0 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -227,6 +227,15 @@ iterate: enabled: true maxIterations: 3 +# ── Flow D — MicroVM suspend/resume demo ───────────────────────────────────── +# When the darkfactory-lambda label runs, suspend the MicroVM after the coder +# pushes (freeing compute while gates run) and resume it on a df-iterate comment. +# Drives Sandbox.spec.operatingMode; the agent-sandbox microvm-lifecycle controller +# translates that to suspend-microvm / resume-microvm. No-op for the Kata substrate. +microvmSuspend: + enabled: true + image: alpine/k8s:1.31.0 # needs kubectl + # ── Language / stack support ───────────────────────────────────────────────── # There is NO per-language profile config here — deliberately. Language support is # decoupled two ways: From f04719f39a4e2e941c210a086ff68394ac1b8bd5 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Fri, 31 Jul 2026 12:33:49 -0400 Subject: [PATCH 16/67] fix(flow-d): align MicrovmSandbox RGD with ACK controller best practices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applied field-level corrections from the lambdamicrovms-controller reference (mmeckes/lambdamicrovms-controller, 06-kro example + CRD source): - codeArtifact.uri is S3-ONLY (s3://bucket/key of a zip w/ Dockerfile), NOT an ECR image URI. Fixed the misleading 'ECR or S3' comments in the RGD schema + values (the Dockerfile inside MAY pull ECR base layers — build role keeps ecr:Get*). - image.readyWhen: gate on status.state == CREATED||UPDATED so the Microvm never launches from a half-built/failed image (KRO holds the instance until ready). - MicrovmImage.cpuConfigurations: [{architecture: ARM_64}] — the sole supported arch; make it explicit rather than relying on a default. - MicrovmImage.logging.cloudWatch.logGroup: /aws/lambda/microvms/-image so CREATE_FAILED build output is retrievable (aws logs tail ...). - status.microvmState now reads Microvm.status.state (PENDING/RUNNING/SUSPENDED/ ...) not conditions[0].status; added imageState passthrough. - build role S3 policy scoped to the actual bucket via CEL ref (bucket.spec.name) + ListBucket, and logs scoped to the microvms log-group prefix. Design unchanged: Sandbox CRDs → ACK Lambda MicroVM for create/delete + the microvm-lifecycle shim for suspend/resume (imperative ops, not controller-managed — confirmed by the reference). Still gated microvm.enabled=false; render-verified. --- .../templates/50-rgd-microvm-sandbox.yaml | 34 +++++++++++++++---- .../addons/charts/agent-sandbox/values.yaml | 8 +++-- 2 files changed, 34 insertions(+), 8 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml index 1166ad98..2023364a 100644 --- a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml @@ -48,9 +48,11 @@ spec: # arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64 — the only arch # Lambda MicroVM supports). baseImageARN: string - # URI of the coder code artifact for the image. Lambda MicroVM accepts an - # ECR image URI *or* an S3 path — Flow D points at the SAME dark-factory-coder - # (built for arm64), so no S3 repackaging is needed. + # S3 URI of the coder code artifact zip (app + Dockerfile), e.g. + # s3:///. Lambda MicroVM's codeArtifact.uri is S3-ONLY — it is + # NOT an ECR image reference (the Dockerfile inside the zip MAY pull private + # ECR base layers, which is why buildRole keeps ecr:Get*/BatchGetImage). The + # artifact must be published to the bucket before the first build runs. codeArtifactUri: string # ── APP-owned: the per-claim instance lifecycle ───────────────────────── run: @@ -65,8 +67,12 @@ spec: name: string status: # Surfaced from the ACK resources so the bridge pod can mirror lifecycle. + # microvmState is the Lambda MicroVM lifecycle state (PENDING/RUNNING/ + # SUSPENDING/SUSPENDED/TERMINATING/TERMINATED) from Microvm.status.state — + # NOT a k8s condition. The bridge maps pod-Ready to RUNNING off this. microvmID: ${microvm.status.microvmID} - microvmState: ${microvm.status.conditions[0].status} + microvmState: ${microvm.status.state} + imageState: ${image.status.state} imageARN: ${image.status.ackResourceMetadata.arn} resources: # 1) S3 bucket that stores the MicroVM code artifact (GA — Managed ACK). @@ -96,8 +102,8 @@ spec: microvm-build: | {"Version":"2012-10-17","Statement":[ {"Effect":"Allow","Action":["ecr:GetAuthorizationToken","ecr:BatchGetImage","ecr:GetDownloadUrlForLayer"],"Resource":"*"}, - {"Effect":"Allow","Action":["s3:GetObject"],"Resource":"arn:aws:s3:::*-microvm-artifacts/*"}, - {"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:*"} + {"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket"],"Resource":["arn:aws:s3:::${bucket.spec.name}","arn:aws:s3:::${bucket.spec.name}/*"]}, + {"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:log-group:/aws/lambda/microvms/*"} ]} # 3) IAM role the RUNNING MicroVM assumes (GA — Managed ACK). App-owned exec identity. # Minimal by default (the coder reaches models via Bifrost, git/gh via public @@ -113,7 +119,12 @@ spec: assumeRolePolicyDocument: | {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} # 4) MicrovmImage (pre-GA — SELF-MANAGED lambdamicrovms controller). Platform-owned. + # readyWhen gates the Microvm (resource 5) on a genuinely SUCCESSFUL build — + # state CREATED/UPDATED — so the instance never launches from a half-built or + # failed image (KRO holds the Microvm until this predicate is true). - id: image + readyWhen: + - ${image.status.state == "CREATED" || image.status.state == "UPDATED"} template: apiVersion: lambdamicrovms.services.k8s.aws/v1alpha1 kind: MicrovmImage @@ -125,6 +136,17 @@ spec: buildRoleARN: ${buildRole.status.ackResourceMetadata.arn} codeArtifact: uri: ${schema.spec.image.codeArtifactUri} + # Lambda MicroVM is ARM_64-ONLY — the sole supported architecture. The + # code artifact + any bundled binaries must be arm64 (dark-factory-coder + # is built for arm64 for exactly this substrate). + cpuConfigurations: + - architecture: ARM_64 + # CloudWatch build/runtime logs. On a CREATE_FAILED the controller can't + # see the build output — this is where it lands: + # aws logs tail /aws/lambda/microvms/${schema.spec.name}-image + logging: + cloudWatch: + logGroup: /aws/lambda/microvms/${schema.spec.name}-image # 5) Microvm instance (pre-GA — SELF-MANAGED). App-owned lifecycle. Runs the # same dark-factory-coder entrypoint that Flow A's Kata pod runs. - id: microvm diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index dac88daa..1d43999d 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -257,10 +257,14 @@ microvm: # + the AWS CLI (streams MicroVM logs). alpine/k8s bundles kubectl. bridgeImage: alpine/k8s:1.31.0 # The coder code-artifact + base image are cluster/account specific → overlay. - # Lambda MicroVM is ARM_64-only; codeArtifactUri accepts an ECR image URI or S3 path. + # Lambda MicroVM is ARM_64-only. codeArtifactUri is an S3 URI (s3://bucket/key) + # to a zip containing the coder app + a Dockerfile — it is NOT an ECR image + # reference (the Dockerfile inside MAY pull private ECR base layers; the build + # role keeps ecr:Get*/BatchGetImage for that). Publish the arm64 coder artifact + # to the bucket before enabling. # image: # baseImageARN: arn:aws:lambda::aws:microvm-image:al2023-1 - # codeArtifactUri: .dkr.ecr..amazonaws.com/dark-factory-coder:-arm64 + # codeArtifactUri: s3:///dark-factory-coder--arm64.zip baseImageARN: "" codeArtifactUri: "" # Idle policy defaults for the RGD schema (auto-suspend/resume). Declarative From 0d368b46cf80f0fe3bff6bd5f77a7a198bfcf7ce Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Fri, 31 Jul 2026 15:54:02 -0400 Subject: [PATCH 17/67] refactor(flow-d): split Lambda MicroVM into its own agent-sandbox-lambda chart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clean folder separation — Flow D (Lambda MicroVM) is now a standalone chart instead of files dumped into the Kata agent-sandbox chart: gitops/addons/charts/agent-sandbox-lambda/ Chart.yaml, values.yaml, templates/_helpers.tpl templates/image/10-rgd-and-image.yaml — KRO RGD (image-only) + the ONE platform MicrovmSandbox instance (built ONCE) templates/shim/20-bridge-sandboxtemplate.yaml — bridge: reads image handoff, RunMicrovm (SDK), TerminateMicrovm templates/shim/30-microvm-lifecycle.yaml — suspend/resume controller (SDK) - Moved the 3 microvm templates out of agent-sandbox (Kata chart) → now Kata-only. - Moved the microvm: values block out of agent-sandbox/values.yaml + the hub overlay into agent-sandbox-lambda/{values.yaml, clusters/hub overlay}. Corrected stale docs (codeArtifact is S3-only, not ECR). - New gated addon entry agent-sandbox-lambda (hub-only, sync-wave 2); dormant until microvm.enabled=true. ARCHITECTURE (per design): KRO builds the platform image ONCE; the shim runs/suspends/ resumes/terminates the per-session VM imperatively (ACK doesn't reconcile those). RGD no longer contains a Microvm resource. Renders: Kata chart microvm-free; Lambda chart 0 disabled / 13 enabled. Flow D only — dark-factory (PR #32) untouched. --- gitops/addons/bootstrap/default/addons.yaml | 21 ++ .../charts/agent-sandbox-lambda/Chart.yaml | 14 + .../templates/_helpers.tpl | 28 ++ .../templates/image/10-rgd-and-image.yaml} | 101 +++---- .../shim/20-bridge-sandboxtemplate.yaml | 255 ++++++++++++++++++ .../templates/shim/30-microvm-lifecycle.yaml} | 25 +- .../charts/agent-sandbox-lambda/values.yaml | 66 +++++ .../templates/51-sandboxtemplate-microvm.yaml | 229 ---------------- .../addons/charts/agent-sandbox/values.yaml | 49 ---- .../addons/agent-sandbox-lambda/values.yaml | 31 +++ .../hub/addons/agent-sandbox/values.yaml | 21 -- 11 files changed, 481 insertions(+), 359 deletions(-) create mode 100644 gitops/addons/charts/agent-sandbox-lambda/Chart.yaml create mode 100644 gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl rename gitops/addons/charts/{agent-sandbox/templates/50-rgd-microvm-sandbox.yaml => agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml} (63%) create mode 100644 gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml rename gitops/addons/charts/{agent-sandbox/templates/52-microvm-lifecycle.yaml => agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml} (86%) create mode 100644 gitops/addons/charts/agent-sandbox-lambda/values.yaml delete mode 100644 gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml create mode 100644 gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml diff --git a/gitops/addons/bootstrap/default/addons.yaml b/gitops/addons/bootstrap/default/addons.yaml index 6671ee8b..d23f053b 100644 --- a/gitops/addons/bootstrap/default/addons.yaml +++ b/gitops/addons/bootstrap/default/addons.yaml @@ -411,6 +411,27 @@ agent-sandbox: coderTemplate: bifrostUrl: '{{default "http://bifrost.bifrost.svc.cluster.local:8080" (index .metadata.annotations "bifrost_url")}}' +# agent-sandbox-lambda — Flow D. A SECOND, opt-in Agent-Sandbox substrate: the coder +# runs in an AWS Lambda MicroVM (Firecracker) instead of a Kata pod. KRO builds the +# platform image ONCE (MicrovmImage + build/exec IAM + S3 + logs via ACK GA +# controllers); a shim (bridge pod + lifecycle controller) drives the per-session VM +# (RunMicrovm/suspend/resume/TerminateMicrovm) imperatively. Same namespace + operator +# as agent-sandbox; only ADDS the MicroVM path. DORMANT until the per-cluster overlay +# flips microvm.enabled=true (needs Managed KRO+ACK, the self-managed ack-lambdamicrovms +# controller, and a published arm64 coder artifact). Hub-only, sync-wave 2. +agent-sandbox-lambda: + enabled: true + namespace: agent-sandbox-system + defaultVersion: '0.1.0' + path: 'gitops/addons/charts/agent-sandbox-lambda' + annotationsAppSet: + argocd.argoproj.io/sync-wave: '2' + alwaysSelector: + matchExpressions: + - key: environment + operator: In + values: ['control-plane'] + # kata-deploy — installs the Kata runtime (containerd handlers for kata-clh / # kata-qemu) on the tainted kata MNG nodes. Separate app (not a subchart dep of # agent-sandbox) so ArgoCD pulls the upstream OCI chart directly. Gated by diff --git a/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml b/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml new file mode 100644 index 00000000..d5da419c --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml @@ -0,0 +1,14 @@ +apiVersion: v2 +name: agent-sandbox-lambda +description: >- + Flow D — Lambda MicroVM substrate for the Agent Sandbox capability. A second, + opt-in execution substrate alongside the Kata micro-VM chart (agent-sandbox): + the coder runs in an AWS Lambda MicroVM (Firecracker) instead of a Kata node. + KRO builds the platform image ONCE (MicrovmImage + build/exec IAM + S3 artifact + + CloudWatch logs, via ACK GA controllers); a lightweight shim (bridge pod + + lifecycle controller) drives the per-session VM (RunMicrovm / suspend / resume / + TerminateMicrovm) imperatively via the AWS SDK. Same dark-factory-coder (arm64) + image + same Agent Sandbox UX as Flow A. Disabled by default (microvm.enabled). +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl b/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl new file mode 100644 index 00000000..dc54887e --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl @@ -0,0 +1,28 @@ +{{/* +Common labels applied to every resource this chart renders. Kept under the +app.kubernetes.io/name "agent-sandbox" (same capability, Lambda substrate) so +Flow D resources associate with the Agent Sandbox capability; the chart name +distinguishes them. +*/}} +{{- define "agent-sandbox.labels" -}} +app.kubernetes.io/name: agent-sandbox +app.kubernetes.io/component: lambda-microvm +app.kubernetes.io/part-of: open-agent-platform +app.kubernetes.io/managed-by: {{ .Release.Service }} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }} +{{- end -}} + +{{/* +Selector labels (stable subset used by controllers). +*/}} +{{- define "agent-sandbox.selectorLabels" -}} +app.kubernetes.io/name: agent-sandbox +app.kubernetes.io/component: lambda-microvm +{{- end -}} + +{{/* +The namespace the capability runs in (must match the Kata agent-sandbox chart). +*/}} +{{- define "agent-sandbox.namespace" -}} +{{- default "agent-sandbox-system" .Values.namespace -}} +{{- end -}} diff --git a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml similarity index 63% rename from gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml rename to gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml index 2023364a..1ab75f0f 100644 --- a/gitops/addons/charts/agent-sandbox/templates/50-rgd-microvm-sandbox.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml @@ -41,39 +41,40 @@ spec: apiVersion: v1alpha1 kind: MicrovmSandbox group: {{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }} + # SCOPE: this RGD builds ONLY the platform IMAGE + the slow-changing infra it + # needs (S3 artifact bucket, build role, execution role). It deliberately does + # NOT create a `Microvm` — a MicroVM instance is a per-SESSION, request-time + # resource whose create/suspend/resume/terminate are IMPERATIVE SDK ops that the + # ACK controller does not reconcile (confirmed in the lambdamicrovms-controller + # reference: run/suspend/resume/terminate are SDK calls, not desired state, and + # the 06-kro example likewise leaves the running Microvm out of the graph). The + # shim (SandboxTemplate bridge + microvm-lifecycle controller, templates 51/52) + # owns that lifecycle. So a `MicrovmSandbox` = "an image is built + ready + its + # exec identity", and its status is the HANDOFF the shim consumes to RunMicrovm. spec: # ── PLATFORM-owned: the image / substrate (set once per image) ────────── - image: - # ARN of the AWS-published base MicroVM image to build from, e.g. - # arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64 — the only arch - # Lambda MicroVM supports). - baseImageARN: string - # S3 URI of the coder code artifact zip (app + Dockerfile), e.g. - # s3:///. Lambda MicroVM's codeArtifact.uri is S3-ONLY — it is - # NOT an ECR image reference (the Dockerfile inside the zip MAY pull private - # ECR base layers, which is why buildRole keeps ecr:Get*/BatchGetImage). The - # artifact must be published to the bucket before the first build runs. - codeArtifactUri: string - # ── APP-owned: the per-claim instance lifecycle ───────────────────────── - run: - # Idle policy — MicroVM auto-suspend/resume to keep idle cost near zero. - # (Declarative suspend/resume via Sandbox.operatingMode is driven by the - # microvm-lifecycle controller, not this field — see template 52.) - autoResumeEnabled: boolean | default=true - maxIdleDurationSeconds: integer | default={{ .Values.microvm.defaults.maxIdleDurationSeconds }} - suspendedDurationSeconds: integer | default={{ .Values.microvm.defaults.suspendedDurationSeconds }} - # Common: AWS region + a name stem for the created resources. + # ARN of the AWS-published base MicroVM image to build from, e.g. + # arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64 — the only arch + # Lambda MicroVM supports). + baseImageARN: string + # S3 URI of the coder code artifact zip (app + Dockerfile), e.g. + # s3:///. Lambda MicroVM's codeArtifact.uri is S3-ONLY — it is + # NOT an ECR image reference (the Dockerfile inside the zip MAY pull private + # ECR base layers, which is why buildRole keeps ecr:Get*/BatchGetImage). The + # artifact must be published to the bucket before the first build runs. + codeArtifactUri: string + # AWS region + a name stem for the created resources. region: string | default="{{ .Values.microvm.region }}" name: string status: - # Surfaced from the ACK resources so the bridge pod can mirror lifecycle. - # microvmState is the Lambda MicroVM lifecycle state (PENDING/RUNNING/ - # SUSPENDING/SUSPENDED/TERMINATING/TERMINATED) from Microvm.status.state — - # NOT a k8s condition. The bridge maps pod-Ready to RUNNING off this. - microvmID: ${microvm.status.microvmID} - microvmState: ${microvm.status.state} - imageState: ${image.status.state} + # The HANDOFF the shim reads to RunMicrovm (imperatively) per session: + # imageARN → Microvm.imageIdentifier + # executionRoleARN → Microvm.executionRoleARN + # Plus imageState so the shim only launches once the build is CREATED/UPDATED. imageARN: ${image.status.ackResourceMetadata.arn} + imageState: ${image.status.state} + imageVersion: ${image.status.latestActiveImageVersion} + executionRoleARN: ${execRole.status.ackResourceMetadata.arn} resources: # 1) S3 bucket that stores the MicroVM code artifact (GA — Managed ACK). - id: bucket @@ -132,10 +133,10 @@ spec: name: ${schema.spec.name}-image spec: name: ${schema.spec.name}-image - baseImageARN: ${schema.spec.image.baseImageARN} + baseImageARN: ${schema.spec.baseImageARN} buildRoleARN: ${buildRole.status.ackResourceMetadata.arn} codeArtifact: - uri: ${schema.spec.image.codeArtifactUri} + uri: ${schema.spec.codeArtifactUri} # Lambda MicroVM is ARM_64-ONLY — the sole supported architecture. The # code artifact + any bundled binaries must be arm64 (dark-factory-coder # is built for arm64 for exactly this substrate). @@ -147,20 +148,30 @@ spec: logging: cloudWatch: logGroup: /aws/lambda/microvms/${schema.spec.name}-image - # 5) Microvm instance (pre-GA — SELF-MANAGED). App-owned lifecycle. Runs the - # same dark-factory-coder entrypoint that Flow A's Kata pod runs. - - id: microvm - template: - apiVersion: lambdamicrovms.services.k8s.aws/v1alpha1 - kind: Microvm - metadata: - name: ${schema.spec.name}-vm - spec: - imageIdentifier: ${image.status.ackResourceMetadata.arn} - executionRoleARN: ${execRole.status.ackResourceMetadata.arn} - # No network connectors → default public internet egress (see header). - idlePolicy: - autoResumeEnabled: ${schema.spec.run.autoResumeEnabled} - maxIdleDurationSeconds: ${schema.spec.run.maxIdleDurationSeconds} - suspendedDurationSeconds: ${schema.spec.run.suspendedDurationSeconds} + # NOTE: there is deliberately NO `Microvm` resource here. The running MicroVM is + # a per-session, request-time resource — the shim (microvm-lifecycle controller, + # template 52) creates it with RunMicrovm and drives suspend/resume/terminate as + # imperative SDK ops (the ACK controller does not reconcile those). This RGD stops + # at "image built + exec role ready", handed off via status above. +{{- if .Values.microvm.image.enabled | default true }} +--- +# The ONE platform MicrovmSandbox INSTANCE — applied by GitOps, reconciled by KRO +# ONCE to build the coder image + its build/exec roles + artifact bucket. This is +# slow-changing platform infra (one image per cluster), NOT per-session — so it lives +# here in the chart, not in the per-claim shim. The shim reads THIS object's status +# (imageARN + executionRoleARN) to RunMicrovm per session. Rebuild the image by +# bumping microvm.codeArtifactUri (a new artifact) and re-syncing. +apiVersion: {{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" }}/v1alpha1 +kind: MicrovmSandbox +metadata: + name: {{ .Values.microvm.image.name | default "coder" }} + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + name: {{ .Values.microvm.image.name | default "coder" }} + region: {{ .Values.microvm.region | quote }} + baseImageARN: {{ .Values.microvm.baseImageARN | quote }} + codeArtifactUri: {{ .Values.microvm.codeArtifactUri | quote }} +{{- end }} {{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml new file mode 100644 index 00000000..eca454ce --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -0,0 +1,255 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — `lambda-microvm` SandboxTemplate + bridge RBAC + bridge script. + +The RuntimeClass "shim": Lambda MicroVM is a REMOTE AWS service, not a node-local +containerd handler, so there is no literal `lambda-microvm` RuntimeClass (that would +need a virtual-kubelet — out of scope). Instead this SandboxTemplate's pod is a thin +BRIDGE that preserves the Agent-Sandbox UX: + + 1. runs on a NORMAL Auto-Mode node (no kata runtimeClass / nodeSelector / taint) + 2. reads the ONE platform-built image (the committed MicrovmSandbox in template 50, + whose status carries imageARN + executionRoleARN — built ONCE by KRO/ACK, NOT + per session) and calls RunMicrovm (imperative AWS SDK) to launch a per-session + MicroVM running the SAME dark-factory-coder entrypoint + 3. records the microvmID on the owning Sandbox (annotation) so the microvm-lifecycle + controller (template 52) can suspend/resume it, and holds the pod so the pod + lifecycle mirrors the MicroVM; on real teardown it calls TerminateMicrovm + +ARCHITECTURE SPLIT (why the bridge no longer creates a MicrovmSandbox per claim): +image build = slow, declarative, ONE per cluster (KRO/ACK, template 50). Running a VM += fast, imperative, per SESSION (RunMicrovm/suspend/resume/terminate are SDK ops the +ACK controller does NOT reconcile). So the bridge drives the RUN side via the SDK and +only READS the platform image handoff — it does not re-run KRO per claim. + +To Flow B and the user this looks identical to a Flow A claim. The bridge needs a +ServiceAccount + AWS creds (Pod Identity) to call the Lambda MicroVM SDK. + +Gated behind microvm.enabled. Rendered as a SEPARATE SandboxTemplate +(`-microvm`) so both substrates can coexist on one cluster; a consumer +selects the substrate by which template its SandboxClaim references. +*/ -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +rules: + # The bridge only READS the platform MicrovmSandbox (image handoff: imageARN + + # executionRoleARN). It does NOT create/delete it — that object is GitOps-owned + # platform infra (template 50), built once. Per-session Run/Terminate is done via + # the AWS SDK, not by mutating this CR. + - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] + resources: ["microvmsandboxes"] + verbs: ["get", "list", "watch"] + # Read the owning Sandbox (suspend-vs-teardown in preStop) + patch it to record the + # per-session microvmID (annotation) so the microvm-lifecycle controller can + # suspend/resume THIS session's VM. + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["get", "patch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: microvm-bridge +subjects: + - kind: ServiceAccount + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: microvm-bridge-script + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +data: + bridge.sh: | + #!/bin/sh + # Flow D bridge — claim -> RunMicrovm (SDK) against the pre-built platform image -> + # mirror lifecycle. Idles until a SandboxClaim injects DF_ISSUE_NUMBER (Flow B), + # exactly like the Kata coder. Needs AWS creds (Pod Identity) for the SDK calls. + set -eu + REGION="{{ .Values.microvm.region }}" + PLATFORM_IMAGE="{{ .Values.microvm.image.name | default "coder" }}" # the ONE committed MicrovmSandbox (template 50) + NS="{{ include "agent-sandbox.namespace" . }}" + echo "[microvm-bridge] idle — waiting for a SandboxClaim to inject DF_ISSUE_NUMBER..." + while [ -z "${DF_ISSUE_NUMBER:-}" ]; do sleep 5; done + # SANDBOX_NAME (downward API = owning Sandbox CR name) lets the microvm-lifecycle + # controller correlate THIS session's MicroVM for suspend/resume. + SANDBOX_NAME="${SANDBOX_NAME:-df-${DF_ISSUE_NUMBER}}" + echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} (sandbox=${SANDBOX_NAME})" + + # 1) READ the platform image handoff (built ONCE by KRO/ACK — template 50). Wait + # until its image build is terminal (CREATED/UPDATED) before launching. + echo "[microvm-bridge] reading platform image ${PLATFORM_IMAGE} (waiting for build ready)..." + i=0; IMAGE_ARN=""; EXEC_ROLE="" + while [ "$i" -lt 240 ]; do + IST=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageState}' 2>/dev/null || echo "") + if [ "${IST}" = "CREATED" ] || [ "${IST}" = "UPDATED" ]; then + IMAGE_ARN=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageARN}' 2>/dev/null || echo "") + EXEC_ROLE=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.executionRoleARN}' 2>/dev/null || echo "") + [ -n "${IMAGE_ARN}" ] && [ -n "${EXEC_ROLE}" ] && break + fi + i=$((i+1)); sleep 5 + done + if [ -z "${IMAGE_ARN}" ] || [ -z "${EXEC_ROLE}" ]; then + echo "[microvm-bridge] ERROR: platform image not ready (imageState=${IST:-}) — cannot RunMicrovm"; exit 1 + fi + echo "[microvm-bridge] image=${IMAGE_ARN} execRole=${EXEC_ROLE}" + + # 2) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image. + # idlePolicy enables auto-suspend/resume; explicit suspend/resume is driven by + # the microvm-lifecycle controller (template 52) off Sandbox.operatingMode. + echo "[microvm-bridge] RunMicrovm..." + RUN_JSON=$(aws lambda-microvms run-microvm \ + --region "${REGION}" \ + --image-identifier "${IMAGE_ARN}" \ + --execution-role-arn "${EXEC_ROLE}" \ + --idle-policy 'autoResumeEnabled=true,maxIdleDurationSeconds={{ .Values.microvm.defaults.maxIdleDurationSeconds }},suspendedDurationSeconds={{ .Values.microvm.defaults.suspendedDurationSeconds }}' \ + 2>&1) || { echo "[microvm-bridge] run-microvm failed: ${RUN_JSON}"; exit 1; } + VMID=$(echo "${RUN_JSON}" | sed -n 's/.*"[Mm]icrovmId"[ ]*:[ ]*"\([^"]*\)".*/\1/p' | head -1) + echo "[microvm-bridge] launched MicroVM ${VMID:-}" + + # 3) Record microvmID on the owning Sandbox so the lifecycle controller can + # suspend/resume THIS session's VM (it reads this annotation). + [ -n "${VMID}" ] && kubectl annotate sandbox "${SANDBOX_NAME}" -n "${NS}" \ + "microvm-lifecycle.agents.x-k8s.io/microvm-id=${VMID}" --overwrite >/dev/null 2>&1 || true + + # 4) Teardown on real claim end (NOT on suspend): suspend deletes the pod but must + # KEEP the MicroVM so resume works. /tmp/suspending marker (preStop) distinguishes. + cleanup() { + if [ -f /tmp/suspending ]; then + echo "[microvm-bridge] pod stopping for SUSPEND — keeping MicroVM ${VMID}" + return + fi + echo "[microvm-bridge] TerminateMicrovm ${VMID}" + [ -n "${VMID}" ] && aws lambda-microvms terminate-microvm --region "${REGION}" --microvm-identifier "${VMID}" >/dev/null 2>&1 || true + } + trap cleanup EXIT INT TERM + + # 5) Hold the pod so Sandbox lifecycle == MicroVM lifecycle. Poll the VM state; + # exit when it's gone (terminated). (Log streaming / exec passthrough is a + # virtual-kubelet follow-up — see docs/dark-factory §4.5.) + echo "[microvm-bridge] MicroVM ${VMID} running — pod now mirrors its lifecycle." + while [ -n "${VMID}" ]; do + ST=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" \ + --query 'state' --output text 2>/dev/null || echo "") + case "${ST}" in + TERMINATED|TERMINATING|"") echo "[microvm-bridge] MicroVM state=${ST:-gone} — exiting."; break ;; + esac + sleep 15 + done +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: {{ .Values.warmPool.templateName }}-microvm + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + # Same env-injection contract as the Kata template so Flow B is unchanged. + envVarsInjectionPolicy: {{ .Values.coderTemplate.envVarsInjectionPolicy | default "Allowed" }} + podTemplate: + metadata: + labels: + {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} + agent-sandbox.io/role: coder + agent-sandbox.io/substrate: lambda-microvm + spec: + # NO kata runtimeClass / nodeSelector / toleration — the bridge is a normal + # pod on an Auto-Mode node. The isolation boundary is the remote MicroVM. + serviceAccountName: microvm-bridge + # The bridge needs a k8s token (read platform image, patch Sandbox) AND AWS creds + # (Pod Identity association on this SA) to call the Lambda MicroVM SDK + # (RunMicrovm/GetMicrovm/TerminateMicrovm). The Kata coder is credential-less; the + # bridge is not, because RUN is imperative. + automountServiceAccountToken: true + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: bridge + image: {{ .Values.microvm.bridgeImage }} + command: ["/bin/sh", "/scripts/bridge.sh"] + # Preflight suspend marker: if this pod is being deleted because the owning + # Sandbox went operatingMode=Suspended, the microvm-lifecycle controller has + # (or will) suspend the MicroVM — the bridge must NOT TerminateMicrovm. The + # controller writes operatingMode; the bridge checks it to set the marker. + lifecycle: + preStop: + exec: + command: + - /bin/sh + - -c + - | + # If the owning Sandbox is Suspended, mark so cleanup() keeps the CR. + M=$(kubectl get sandbox "df-${DF_ISSUE_NUMBER:-}" \ + -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") + [ "$M" = "Suspended" ] && touch /tmp/suspending || true + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: { cpu: 50m, memory: 64Mi } + limits: { cpu: 200m, memory: 128Mi } + volumeMounts: + - name: bridge-script + mountPath: /scripts + - name: tmp + mountPath: /tmp + volumes: + - name: bridge-script + configMap: + name: microvm-bridge-script + defaultMode: 0555 + - name: tmp + emptyDir: {} +--- +{{- /* +Lambda-MicroVM SandboxWarmPool. SandboxClaim.spec.warmPoolRef is REQUIRED (a claim +can't bind a bare template), so Flow D needs its own pool the df-run claim step +targets when the darkfactory-lambda label fires. Kept small (bridge pods are tiny); +each idle member is a bridge waiting to RunMicrovm (from the pre-built platform image) +on claim. +*/ -}} +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: {{ .Values.microvm.warmPool.name | default "coder-warmpool-microvm" }} + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + replicas: {{ .Values.microvm.warmPool.targetIdle | default 1 }} + sandboxTemplateRef: + name: {{ .Values.warmPool.templateName }}-microvm +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml similarity index 86% rename from gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml rename to gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml index d90f4860..eb0f4d3c 100644 --- a/gitops/addons/charts/agent-sandbox/templates/52-microvm-lifecycle.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml @@ -37,17 +37,12 @@ metadata: labels: {{- include "agent-sandbox.labels" . | nindent 4 }} rules: - # Read Sandboxes (watch operatingMode) + the MicrovmSandbox composites (read microvmID). + # Read Sandboxes (watch operatingMode + read the bridge-written microvm-id annotation) + # and patch them (stamp the last-acted mode to detect transitions). No microvmsandboxes + # access needed — the per-session VM id lives on the Sandbox, not a KRO status. - apiGroups: ["agents.x-k8s.io"] resources: ["sandboxes"] - verbs: ["get", "list", "watch"] - - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] - resources: ["microvmsandboxes"] - verbs: ["get", "list", "watch"] - # Record the last-applied mode on the Sandbox (annotation) to detect transitions. - - apiGroups: ["agents.x-k8s.io"] - resources: ["sandboxes"] - verbs: ["patch", "update"] + verbs: ["get", "list", "watch", "patch", "update"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding @@ -92,13 +87,13 @@ data: MODE=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "Running") LAST=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath="{.metadata.annotations.${ANN}}" 2>/dev/null || echo "") [ "$MODE" = "$LAST" ] && continue # no transition - # Resolve the MicroVM id from the matching MicrovmSandbox (name = df-) via its status. - VMID=$(kubectl get microvmsandbox -n "$NS" \ - -o jsonpath="{range .items[?(@.metadata.labels.agent-sandbox\.io/sandbox=='${sb}')]}{.status.microvmID}{end}" 2>/dev/null || echo "") - # Fallback: single-claim naming — try the sandbox name directly. - [ -z "$VMID" ] && VMID=$(kubectl get microvmsandbox "$sb" -n "$NS" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + # Resolve THIS session's MicroVM id from the annotation the BRIDGE writes on the + # Sandbox after RunMicrovm (microvm-lifecycle.agents.x-k8s.io/microvm-id). There + # is no per-session MicrovmSandbox anymore — the VM is created imperatively by the + # bridge (SDK), so the id lives on the Sandbox, not in a KRO status. + VMID=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") if [ -z "$VMID" ]; then - echo "[microvm-lifecycle] $sb: mode=$MODE but no microvmID yet — will retry" + echo "[microvm-lifecycle] $sb: mode=$MODE but no microvm-id annotation yet — will retry" continue fi case "$MODE" in diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml new file mode 100644 index 00000000..8ac0e4d2 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -0,0 +1,66 @@ +# agent-sandbox-lambda — Flow D (Lambda MicroVM substrate) values. +# +# Opt-in second substrate for the Agent Sandbox capability. Disabled by default; +# set microvm.enabled=true (per-cluster overlay) to render it. Runs in the SAME +# namespace as the Kata agent-sandbox chart and reuses the same operator + CRDs — +# it only ADDS the Lambda-MicroVM image build (KRO) + the bridge/lifecycle shim. + +# Namespace the capability runs in (must match the agent-sandbox chart). +namespace: agent-sandbox-system + +# Warm-pool template name stem (the bridge SandboxTemplate is -microvm, +# matching the Kata chart's convention so df-run's claim step resolves it). +warmPool: + templateName: coder-sandbox + +# SandboxTemplate env-injection policy (same contract as the Kata template so Flow B +# is unchanged). +coderTemplate: + envVarsInjectionPolicy: Allowed + +# ── Lambda MicroVM (Flow D) ────────────────────────────────────────────────── +microvm: + # Master gate — Flow D stays dormant until a cluster opts in. + enabled: false + region: us-west-2 + # API group the generated MicrovmSandbox CRD is served under (KRO schema.group). + apiGroup: sandbox.agents.x-k8s.io + # Bridge/lifecycle pod image: needs kubectl (read platform image, patch Sandbox) + # AND the AWS CLI (RunMicrovm/suspend/resume/terminate). alpine/k8s bundles both. + bridgeImage: alpine/k8s:1.31.0 + + # The ONE platform image built by KRO/ACK (10-rgd-microvm-image.yaml). Built ONCE + # per cluster; the shim reads its status (imageARN + executionRoleARN) to RunMicrovm + # per session. Lambda MicroVM is ARM_64-ONLY. codeArtifactUri is an S3 URI + # (s3://bucket/key) of a zip containing the coder app + a Dockerfile — NOT an ECR + # image ref (the Dockerfile inside MAY pull private ECR base layers; the build role + # keeps ecr:Get*/BatchGetImage). Publish the arm64 coder artifact before enabling. + # baseImageARN: arn:aws:lambda::aws:microvm-image:al2023-1 + # codeArtifactUri: s3:///dark-factory-coder--arm64.zip + baseImageARN: "" + codeArtifactUri: "" + image: + # Render the single committed MicrovmSandbox instance that triggers the build. + enabled: true + # Name of that platform image object; the bridge reads its status by this name. + name: coder + + # Idle policy for RunMicrovm (auto-suspend/resume). EXPLICIT suspend/resume across + # the review→fix loop is driven by the microvm-lifecycle controller off + # Sandbox.operatingMode — see lifecycle below and project_flow_d_lifecycle memory: + # coder codes → SUSPEND → agents review → (fix findings) → RESUME same VM → + # … loop until cleared → merge/exit → TERMINATE. + defaults: + maxIdleDurationSeconds: 900 + suspendedDurationSeconds: 300 + + # microvm-lifecycle controller (30-microvm-lifecycle.yaml) — reconciles + # Sandbox.operatingMode Running↔Suspended → suspend/resume-microvm. Loop interval. + lifecycle: + intervalSeconds: 15 + + # Lambda-MicroVM warm pool the df-run claim binds when the darkfactory-lambda label + # fires (Flow D). Small — bridge pods are tiny (the real coder runs in the MicroVM). + warmPool: + name: coder-warmpool-microvm + targetIdle: 1 diff --git a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml b/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml deleted file mode 100644 index 9733051d..00000000 --- a/gitops/addons/charts/agent-sandbox/templates/51-sandboxtemplate-microvm.yaml +++ /dev/null @@ -1,229 +0,0 @@ -{{- if and .Values.microvm .Values.microvm.enabled }} -{{- /* -Flow D — `lambda-microvm` SandboxTemplate + bridge RBAC + bridge script. - -The RuntimeClass "shim": Lambda MicroVM is a REMOTE AWS service, not a node-local -containerd handler, so there is no literal `lambda-microvm` RuntimeClass (that would -need a virtual-kubelet — out of scope). Instead this SandboxTemplate's pod is a thin -BRIDGE that preserves the Agent-Sandbox UX: - - 1. runs on a NORMAL Auto-Mode node (no kata runtimeClass / nodeSelector / taint) - 2. applies a MicrovmSandbox (KRO) CR from the injected claim env (DF_ISSUE_NUMBER, - repo, branch — same as Flow A), which provisions the Microvm running the SAME - dark-factory-coder entrypoint - 3. waits for Microvm RUNNING, then STREAMS its logs into the pod (pod Running ⇔ - Microvm RUNNING); on pod exit / claim teardown it deletes the MicrovmSandbox - (→ TerminateMicrovm) - -To Flow B and the user this looks identical to a Flow A claim. The bridge needs a -ServiceAccount (unlike the credential-less Kata coder) because it drives the KRO CR. - -Gated behind microvm.enabled. Rendered as a SEPARATE SandboxTemplate -(`-microvm`) so both substrates can coexist on one cluster; a consumer -selects the substrate by which template its SandboxClaim references. -*/ -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: microvm-bridge - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: microvm-bridge - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} -rules: - # The bridge creates/reads/deletes MicrovmSandbox CRs (the KRO composite). - - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] - resources: ["microvmsandboxes"] - verbs: ["create", "get", "list", "watch", "delete"] - # Read its owning Sandbox in preStop to tell a suspend from a real teardown. - - apiGroups: ["agents.x-k8s.io"] - resources: ["sandboxes"] - verbs: ["get"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: microvm-bridge - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: microvm-bridge -subjects: - - kind: ServiceAccount - name: microvm-bridge - namespace: {{ include "agent-sandbox.namespace" . }} ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: microvm-bridge-script - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} -data: - bridge.sh: | - #!/bin/sh - # Flow D bridge — claim -> MicrovmSandbox -> mirror lifecycle. Idles until a - # SandboxClaim injects DF_ISSUE_NUMBER (Flow B), exactly like the Kata coder. - set -eu - echo "[microvm-bridge] idle — waiting for a SandboxClaim to inject DF_ISSUE_NUMBER..." - while [ -z "${DF_ISSUE_NUMBER:-}" ]; do sleep 5; done - NAME="df-${DF_ISSUE_NUMBER}" - # SANDBOX_NAME is injected via the downward API (the owning Sandbox CR's name) so - # the microvm-lifecycle controller can correlate this MicrovmSandbox to the Sandbox - # whose operatingMode it watches for suspend/resume. - SANDBOX_NAME="${SANDBOX_NAME:-$NAME}" - echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} (sandbox=${SANDBOX_NAME}) -> ensuring MicrovmSandbox/${NAME}" - - # Teardown on exit ONLY on real claim end, not on a suspend (suspend deletes the - # pod but must KEEP the MicrovmSandbox so resume works). We distinguish via a - # marker file the pod writes when it sees operatingMode=Suspended. Default = - # teardown (claim ended / crash) -> KRO cascades TerminateMicrovm + cleans S3/IAM. - cleanup() { - if [ -f /tmp/suspending ]; then - echo "[microvm-bridge] pod stopping for SUSPEND — keeping MicrovmSandbox/${NAME}" - return - fi - echo "[microvm-bridge] tearing down MicrovmSandbox/${NAME}" - kubectl delete microvmsandbox "${NAME}" --ignore-not-found --wait=false || true - } - trap cleanup EXIT INT TERM - - # Idempotent create/adopt (resume-or-create): if the MicrovmSandbox already exists - # from a prior suspend cycle, kubectl apply is a no-op and we just re-attach. The - # microvm-lifecycle controller handles the actual resume-microvm on operatingMode. - cat </dev/null || echo "") - [ "${STATE}" = "True" ] && break - i=$((i+1)); sleep 5 - done - VMID=$(kubectl get microvmsandbox "${NAME}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") - echo "[microvm-bridge] Microvm ${VMID:-} state=${STATE:-} — pod now mirrors the MicroVM lifecycle." - - # Keep the pod alive == MicroVM alive. Real log streaming (aws lambdamicrovms - # get-microvm-logs / CloudWatch tail) is wired via the image build; here we hold - # the pod so Sandbox lifecycle == Microvm lifecycle. Exec/attach passthrough is - # a virtual-kubelet follow-up (see docs/dark-factory §4.5). - while kubectl get microvmsandbox "${NAME}" >/dev/null 2>&1; do sleep 15; done - echo "[microvm-bridge] MicrovmSandbox gone — exiting." ---- -apiVersion: extensions.agents.x-k8s.io/v1beta1 -kind: SandboxTemplate -metadata: - name: {{ .Values.warmPool.templateName }}-microvm - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} - agent-sandbox.io/substrate: lambda-microvm -spec: - # Same env-injection contract as the Kata template so Flow B is unchanged. - envVarsInjectionPolicy: {{ .Values.coderTemplate.envVarsInjectionPolicy | default "Allowed" }} - podTemplate: - metadata: - labels: - {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} - agent-sandbox.io/role: coder - agent-sandbox.io/substrate: lambda-microvm - spec: - # NO kata runtimeClass / nodeSelector / toleration — the bridge is a normal - # pod on an Auto-Mode node. The isolation boundary is the remote MicroVM. - serviceAccountName: microvm-bridge - # The bridge DOES need a token (unlike the Kata coder) to drive the KRO CR. - automountServiceAccountToken: true - securityContext: - runAsNonRoot: true - runAsUser: 1000 - runAsGroup: 1000 - fsGroup: 1000 - seccompProfile: - type: RuntimeDefault - containers: - - name: bridge - image: {{ .Values.microvm.bridgeImage }} - command: ["/bin/sh", "/scripts/bridge.sh"] - # Preflight suspend marker: if this pod is being deleted because the owning - # Sandbox went operatingMode=Suspended, the microvm-lifecycle controller has - # (or will) suspend the MicroVM — the bridge must NOT tear down the - # MicrovmSandbox. The controller writes operatingMode; the bridge checks it. - lifecycle: - preStop: - exec: - command: - - /bin/sh - - -c - - | - # If the owning Sandbox is Suspended, mark so cleanup() keeps the CR. - M=$(kubectl get sandbox "df-${DF_ISSUE_NUMBER:-}" \ - -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") - [ "$M" = "Suspended" ] && touch /tmp/suspending || true - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: ["ALL"] - resources: - requests: { cpu: 50m, memory: 64Mi } - limits: { cpu: 200m, memory: 128Mi } - volumeMounts: - - name: bridge-script - mountPath: /scripts - - name: tmp - mountPath: /tmp - volumes: - - name: bridge-script - configMap: - name: microvm-bridge-script - defaultMode: 0555 - - name: tmp - emptyDir: {} ---- -{{- /* -Lambda-MicroVM SandboxWarmPool. SandboxClaim.spec.warmPoolRef is REQUIRED (a claim -can't bind a bare template), so Flow D needs its own pool the df-run claim step -targets when the darkfactory-lambda label fires. Kept small (bridge pods are tiny); -each idle member is a bridge waiting to apply a MicrovmSandbox on claim. -*/ -}} -apiVersion: extensions.agents.x-k8s.io/v1beta1 -kind: SandboxWarmPool -metadata: - name: {{ .Values.microvm.warmPool.name | default "coder-warmpool-microvm" }} - namespace: {{ include "agent-sandbox.namespace" . }} - labels: - {{- include "agent-sandbox.labels" . | nindent 4 }} - agent-sandbox.io/substrate: lambda-microvm -spec: - replicas: {{ .Values.microvm.warmPool.targetIdle | default 1 }} - sandboxTemplateRef: - name: {{ .Values.warmPool.templateName }}-microvm -{{- end }} diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index 1d43999d..f925fb73 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -235,52 +235,3 @@ nodepool: subnetIds: [] amiId: "" launchTemplateId: "" - -# ── Flow D — Lambda MicroVM substrate (alternative to the Kata nodepool) ────── -# A SECOND Agent-Sandbox substrate: instead of a Kata pod on the nested-virt node -# group, the coder runs in an AWS Lambda MicroVM provisioned by the ACK -# lambdamicrovms controller and composed by a single KRO ResourceGraphDefinition -# (templates/50-rgd-microvm-sandbox.yaml). A `lambda-microvm` SandboxTemplate -# bridge (templates/51-...) preserves the Agent-Sandbox UX. See docs/dark-factory -# §4.5 and diagrams/flow-d-microvm-sandbox.md. -# -# DISABLED by default (like the kata nodepool): flip enabled=true on a cluster that -# has the Managed KRO + Managed ACK capabilities and the self-managed -# ack-lambdamicrovms controller (addons.yaml). Cluster-specific ARNs belong in the -# per-cluster overlay, NOT here. -microvm: - enabled: false - region: us-west-2 - # API group the generated MicrovmSandbox CRD is served under (KRO schema.group). - apiGroup: sandbox.agents.x-k8s.io - # Bridge pod image: reuse an image with kubectl (applies the MicrovmSandbox CR) - # + the AWS CLI (streams MicroVM logs). alpine/k8s bundles kubectl. - bridgeImage: alpine/k8s:1.31.0 - # The coder code-artifact + base image are cluster/account specific → overlay. - # Lambda MicroVM is ARM_64-only. codeArtifactUri is an S3 URI (s3://bucket/key) - # to a zip containing the coder app + a Dockerfile — it is NOT an ECR image - # reference (the Dockerfile inside MAY pull private ECR base layers; the build - # role keeps ecr:Get*/BatchGetImage for that). Publish the arm64 coder artifact - # to the bucket before enabling. - # image: - # baseImageARN: arn:aws:lambda::aws:microvm-image:al2023-1 - # codeArtifactUri: s3:///dark-factory-coder--arm64.zip - baseImageARN: "" - codeArtifactUri: "" - # Idle policy defaults for the RGD schema (auto-suspend/resume). Declarative - # suspend/resume via Sandbox.operatingMode is handled by the microvm-lifecycle - # controller (templates/52), independent of these. No network connectors: - # Lambda MicroVMs default to public internet egress, which is all the coder needs. - defaults: - maxIdleDurationSeconds: 900 - suspendedDurationSeconds: 300 - # microvm-lifecycle controller (templates/52) — reconciles Sandbox.operatingMode - # -> suspend/resume-microvm. How often the loop runs. - lifecycle: - intervalSeconds: 15 - # Lambda-MicroVM warm pool the df-run claim binds when the darkfactory-lambda - # label fires (Flow D). Small — bridge pods are tiny (the real coder runs in the - # MicroVM, not the pod). templateName is -microvm. - warmPool: - name: coder-warmpool-microvm - targetIdle: 1 diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml new file mode 100644 index 00000000..0b2a3057 --- /dev/null +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -0,0 +1,31 @@ +# Hub (openclaw-eks / control-plane) overlay for the agent-sandbox-lambda chart +# (Flow D — Lambda MicroVM substrate). +# +# DORMANT (microvm.enabled=false) until the self-managed ack-lambdamicrovms +# controller is synced (gitops/addons/bootstrap/default/addons.yaml) and the arm64 +# coder artifact is published to S3. Managed KRO + Managed ACK capabilities are +# ACTIVE on the hub. The ARNs below are public identifiers (not secrets). +microvm: + enabled: false + region: us-west-2 + apiGroup: sandbox.agents.x-k8s.io + bridgeImage: alpine/k8s:1.31.0 + # AWS-published base MicroVM image (ARM_64 — the only arch Lambda MicroVM supports). + baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" + # Lambda MicroVM codeArtifact.uri is S3-ONLY (a zip with the coder app + a Dockerfile); + # it is NOT an ECR image reference. The Dockerfile inside MAY pull the arm64 + # dark-factory-coder from ECR as a base layer (the build role keeps ecr:Get*). Publish + # the artifact zip to this bucket/key before flipping enabled=true. + # TODO: create the artifact zip (Dockerfile FROM the v0.2.5-arm64 coder) + upload. + codeArtifactUri: "s3://dark-factory-microvm-artifacts-940019131157-us-west-2/coder-v0.2.5-arm64.zip" + image: + enabled: true + name: coder + defaults: + maxIdleDurationSeconds: 900 + suspendedDurationSeconds: 300 + lifecycle: + intervalSeconds: 15 + warmPool: + name: coder-warmpool-microvm + targetIdle: 1 diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml index 07dbe51d..bf10ad47 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox/values.yaml @@ -38,24 +38,3 @@ nodepool: # Adopt the existing LaunchTemplate in place (external-name). launchTemplateId: lt-0e204ea3e305e2e1f launchTemplateVersion: "$Latest" - -# ── Flow D — Lambda MicroVM substrate (hub) ────────────────────────────────── -# Alternative to the Kata `nodepool` above: run the coder in an AWS Lambda MicroVM -# instead of a Kata pod. DORMANT (enabled:false) until the self-managed -# ack-lambdamicrovms controller is synced (see gitops/addons/bootstrap/default/ -# addons.yaml). Managed KRO + Managed ACK capabilities are ACTIVE on the hub. -# The ARNs below are public identifiers (not secrets). -microvm: - enabled: false - region: us-west-2 - apiGroup: sandbox.agents.x-k8s.io - bridgeImage: alpine/k8s:1.31.0 - # AWS-published base MicroVM image (ARM_64 — the only arch Lambda MicroVM supports). - baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" - # The SAME dark-factory-coder, built for arm64 (MicroVM is ARM_64-only) and pushed - # to ECR. codeArtifactUri accepts an ECR image URI directly — no S3 repackaging. - # NOTE: fill the real arm64 tag once built+pushed (task: arm64 coder image). - codeArtifactUri: "940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:v0.2.3-arm64" - defaults: - maxIdleDurationSeconds: 900 - suspendedDurationSeconds: 300 From 26fe34287872cda4e31b75fbee5599ff6104bc1f Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Fri, 31 Jul 2026 15:57:26 -0400 Subject: [PATCH 18/67] =?UTF-8?q?feat(flow-d):=20enforce=20suspend?= =?UTF-8?q?=E2=86=92resume=E2=86=92terminate=20lifecycle=20(VM=20persists?= =?UTF-8?q?=20across=20review=20loop)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the required Lambda coder lifecycle: coder codes → SUSPEND; findings + fix → RESUME the SAME VM (keeps context); terminate ONLY at final exit (merge). - df-run onExit teardown is now SUBSTRATE-AWARE (script, not a blind resource delete): Kata deletes the claim (frees pool); LAMBDA KEEPS the claim + suspended VM so the same VM survives the whole review→fix→re-review loop (df-iterate resumes it). Without this, df-run's exit deleted the claim → orphaned/terminated the suspended VM. - df-merge-teardown adds a microvm-terminate step (gated microvmSuspend.enabled) that resolves the per-session VM id from the Sandbox annotation and TerminateMicrovm on merge — the ONLY place the Lambda VM is destroyed. Ordered before the claim delete so the bridge's suspend-marker can't keep a leaked VM. - microvm-set-mode verify reads the VM id from the Sandbox annotation (post image-only RGD refactor — no per-session MicrovmSandbox status anymore). Flow-D-branch changes to the dark-factory template (base of PR #41); gated by microvmSuspend.enabled so Flow B (Kata, PR #32) is unaffected. Renders clean both ways. --- .../templates/20-workflowtemplate-df-run.yaml | 36 ++++++++++------ ...21-workflowtemplate-df-merge-teardown.yaml | 42 ++++++++++++++++++- 2 files changed, 65 insertions(+), 13 deletions(-) diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index 28533f50..879ab6b1 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -274,11 +274,12 @@ spec: NS="{{ .Values.warmPool.namespace }}" echo "[microvm-set-mode] Sandbox/$SB -> operatingMode=$MODE" kubectl patch sandbox "$SB" -n "$NS" --type merge -p "{\"spec\":{\"operatingMode\":\"$MODE\"}}" - # Best-effort verify the microvm-lifecycle controller acted (poll the - # MicrovmSandbox status the bridge created). Advisory only. + # Best-effort verify the microvm-lifecycle controller acted. Post-refactor the + # per-session VM id lives on the Sandbox annotation (the bridge writes it after + # RunMicrovm) — there is no per-session MicrovmSandbox anymore. Advisory only. EXPECT="RUNNING"; [ "$MODE" = "Suspended" ] && EXPECT="SUSPENDED" i=0; while [ "$i" -lt 12 ]; do - VMID=$(kubectl get microvmsandbox -n "$NS" -o jsonpath="{range .items[?(@.metadata.labels.agent-sandbox\.io/sandbox=='$SB')]}{.status.microvmID}{end}" 2>/dev/null || echo "") + VMID=$(kubectl get sandbox "$SB" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") [ -n "$VMID" ] && { echo "[microvm-set-mode] microvmID=$VMID (lifecycle controller will drive $EXPECT)"; break; } i=$((i+1)); sleep 5 done @@ -838,13 +839,24 @@ spec: node /scripts/status.js # ---- onExit: release the claim (operator refills the pool) ---- + # SUBSTRATE-AWARE teardown (onExit). Kata: delete the claim now — df-run is done, + # the pod's work is over, free the warm pool. LAMBDA (Flow D): do NOT delete — the + # coder VM was SUSPENDED (microvm-suspend step) and must PERSIST across the whole + # review→fix→re-review loop so df-iterate can RESUME the SAME VM (keeps context). + # The claim + suspended VM are terminated only at the END (df-merge-teardown on + # merge). So on the lambda substrate this onExit is a deliberate no-op. - name: teardown - resource: - action: delete - flags: ["--ignore-not-found"] - manifest: | - apiVersion: extensions.agents.x-k8s.io/v1beta1 - kind: SandboxClaim - metadata: - name: df-issue-{{`{{workflow.parameters.issue-id}}`}} - namespace: {{ .Values.warmPool.namespace }} + script: + image: {{ .Values.stepImage }} + command: [sh] + source: | + set -eu + LABEL="{{`{{workflow.parameters.trigger-label}}`}}" + NS="{{ .Values.warmPool.namespace }}" + CLAIM="df-issue-{{`{{workflow.parameters.issue-id}}`}}" + if [ "$LABEL" = "darkfactory-lambda" ]; then + echo "[teardown] lambda substrate — KEEPING claim/${CLAIM} (VM stays SUSPENDED until merge; df-iterate resumes it)." + exit 0 + fi + echo "[teardown] kata substrate — deleting SandboxClaim/${CLAIM} (frees the warm pool)." + kubectl delete sandboxclaim "${CLAIM}" -n "${NS}" --ignore-not-found --wait=false || true diff --git a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml index fc7cae97..319758fe 100644 --- a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml +++ b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml @@ -39,9 +39,22 @@ spec: tasks: - name: merge template: merge-pr +{{- if .Values.microvmSuspend.enabled }} + # Flow D: this is the FINAL exit — TERMINATE the Lambda MicroVM that was + # kept suspended across the review→fix loop. Runs before the claim delete so + # the VM is destroyed explicitly (the suspended VM would otherwise be kept by + # the bridge's suspend-marker on pod delete → leak). No-op for Kata. + - name: microvm-terminate + template: microvm-terminate + dependencies: [merge] +{{- end }} - name: teardown template: teardown-claim +{{- if .Values.microvmSuspend.enabled }} + dependencies: [microvm-terminate] +{{- else }} dependencies: [merge] +{{- end }} # ---- Merge the PR — only if every dark-factory/* check is green ---- - name: merge-pr @@ -91,8 +104,35 @@ spec: echo "[df-merge] human-approved PR #${PR} in ${REPO} — verifying + merging" node /scripts/merge.js +{{- if .Values.microvmSuspend.enabled }} + # ---- Flow D: terminate the Lambda MicroVM (final exit) ---- + # Resolve the per-session microvm id from the owning Sandbox annotation (the bridge + # wrote it after RunMicrovm), then TerminateMicrovm. A suspended VM can be + # terminated directly. Advisory (never fails the merge) — the reaper + idlePolicy + # are backstops. Needs lambda-microvms:TerminateMicrovm on the workflow's IRSA role. + - name: microvm-terminate + script: + image: {{ .Values.microvmSuspend.image | default "alpine/k8s:1.31.0" }} + command: [sh] + source: | + set -eu + NS="{{ .Values.warmPool.namespace }}" + REGION="{{ .Values.securityAgent.region | default "us-west-2" }}" + NUM="{{`{{workflow.parameters.issue-number}}`}}" + # Find the Sandbox for this issue (by the number label the claim/sandbox carry). + SB=$(kubectl get sandbox -n "$NS" -l "dark-factory.io/issue-number=${NUM}" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "") + [ -z "$SB" ] && { echo "[microvm-terminate] no Sandbox for issue #${NUM} — nothing to terminate"; exit 0; } + VMID=$(kubectl get sandbox "$SB" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") + [ -z "$VMID" ] && { echo "[microvm-terminate] Sandbox/$SB has no microvm-id — nothing to terminate"; exit 0; } + echo "[microvm-terminate] TerminateMicrovm ${VMID} (issue #${NUM}, final exit)" + command -v aws >/dev/null 2>&1 || { echo "[microvm-terminate] aws cli missing on image — skipping (advisory)"; exit 0; } + aws lambda-microvms terminate-microvm --region "$REGION" --microvm-identifier "$VMID" 2>&1 || echo "[microvm-terminate] terminate failed (advisory)" +{{- end }} + # ---- Teardown the claim (operator refills the pool) ---- - # Delete by the issue-number label (the PR-review event has no issue id). + # Delete by the issue-number label (the PR-review event has no issue id). For Lambda + # this also removes the (idle) bridge pod; the VM is already terminated above. - name: teardown-claim resource: action: delete From 9037e5fee3a1bf4a3e9551a11edb46175d2b15c6 Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Fri, 31 Jul 2026 16:00:06 -0400 Subject: [PATCH 19/67] docs(flow-d): update design for image-only KRO + shim-run VM + suspend/resume/terminate lifecycle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aligns the Flow D docs with the refactor: KRO builds the image ONCE (RGD has no Microvm); the shim runs the per-session VM imperatively (RunMicrovm/suspend/resume/ TerminateMicrovm); the coder VM PERSISTS suspended across the review→fix loop and is terminated only at merge. Updated the composition table, platform/app split, shim description, the suspend/resume/terminate lifecycle section, and the delivery section (now the standalone agent-sandbox-lambda chart with image/ + shim/ subfolders). NOTE: diagrams/flow-d-microvm-sandbox.md ASCII still shows the old per-claim model — follow-up. --- docs/dark-factory/README.md | 81 +++++++++++++++++++++++-------------- 1 file changed, 51 insertions(+), 30 deletions(-) diff --git a/docs/dark-factory/README.md b/docs/dark-factory/README.md index cd563f4d..cb097cbd 100644 --- a/docs/dark-factory/README.md +++ b/docs/dark-factory/README.md @@ -282,9 +282,9 @@ for other work; this substrate is Flow D.)* | Layer | Mechanism | Notes | |---|---|---| -| **Composition** | **Managed KRO** (EKS Capability) + one `MicrovmSandbox` `ResourceGraphDefinition` | One CR expands into all primitives below | +| **Composition** | **Managed KRO** (EKS Capability) + one `MicrovmSandbox` `ResourceGraphDefinition` | One CR expands into the IMAGE primitives below (built once); the running `Microvm` is NOT in the graph — the shim runs it imperatively | | **GA primitives** | **Managed ACK** (EKS Capability) — `iam` Role, `s3` Bucket | AWS-run; the image store + build/exec roles | -| **MicroVM primitives** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` + `Microvm` CRDs (`lambdamicrovms.services.k8s.aws/v1alpha1`) | +| **Image primitive** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` CRD (`lambdamicrovms.services.k8s.aws/v1alpha1`); the `Microvm` is created via SDK by the shim, not as a graph resource | > **Why self-managed for the MicroVM controller?** Managed ACK bundles only controllers whose service > is **GA upstream** (see the [ACK community services / GA list](https://aws-controllers-k8s.github.io/community/docs/community/services/)). @@ -298,46 +298,67 @@ for other work; this substrate is Flow D.)* > `docs/EKS-Capabilities-KRO-ACK-Setup.md`. This repo owns only the **self-managed `lambdamicrovms` > controller + the KRO `MicrovmSandbox` RGD + the sandbox shim** (Flow D). -### Platform-owned vs app-owned (encoded in the two ACK CRDs) +### The split: KRO builds the image ONCE; the shim runs the VM per session -- **Platform owns the image/substrate** — declarative, ACK-managed: `MicrovmImage` - (`baseImageARN`, `buildRoleArn`, `codeArtifact.uri` in S3, egress connectors). The image is built - **from the existing `dark-factory-coder` image** + its `entrypoint.js`. -- **App teams own the instance lifecycle** — `Microvm` (`imageIdentifier`, `executionRoleArn`, - `ingress/egressNetworkConnectors`, `idlePolicy{autoResumeEnabled, maxIdleDurationSeconds, - suspendedDurationSeconds}`) — created per claim, torn down with it. +This is the load-bearing design decision (and it matches the ACK controller's own guidance — +image build is slow/declarative, running a VM is fast/imperative): -The single `MicrovmSandbox` RGD surfaces both halves so each owner sets its own fields, while -consumers see just one CRD. +- **Platform image — declarative, built ONCE by KRO/ACK.** The `MicrovmSandbox` RGD + (`agent-sandbox-lambda/templates/image/`) composes only the slow-changing infra: `MicrovmImage` + (`baseImageARN`, `buildRoleARN`, `codeArtifact.uri` — an **S3 zip** of the arm64 `dark-factory-coder` + + a Dockerfile) plus its **build role**, **execution role**, and **S3 artifact bucket** (ACK GA + controllers). A **single committed `MicrovmSandbox` instance** (GitOps-applied) is reconciled once; + KRO gates the handoff on a successful build (`readyWhen state == CREATED||UPDATED`). Its status + surfaces `imageARN` + `executionRoleARN`. The RGD **does not** contain a `Microvm`. +- **Per-session VM — imperative, driven by the shim.** Running a MicroVM (`RunMicrovm`), and its + `suspend` / `resume` / `TerminateMicrovm`, are request-time SDK ops the ACK controller does **not** + reconcile. So the shim owns them — never a `Microvm` CR per claim. ### The RuntimeClass shim (claim → pod → MicroVM) A literal K8s `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**; Lambda MicroVM is a **remote AWS service**, so a true node-level RuntimeClass would require a virtual-kubelet provider (a large Go runtime — **out of scope**). Flow D instead ships a **`lambda-microvm` -SandboxTemplate variant** whose pod is a lightweight **bridge**: it applies the `MicrovmSandbox` CR, -then **streams the MicroVM's logs into the pod** and maps lifecycle (pod Running ⇔ `Microvm` RUNNING; -pod exit → `TerminateMicrovm`). To Flow B and the user the UX is identical to Flow A. Interactive -exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up. - -### Suspend / resume (Sandbox CRD) - -Because the substrate is a Lambda MicroVM (not a pod), Flow D gives you **declarative -suspend/resume through the Agent Sandbox CRD**: set `Sandbox.spec.operatingMode: Suspended` and a -small **`microvm-lifecycle`** reconcile loop calls `suspend-microvm` (and `resume-microvm` on -`Running`) by the MicroVM id — the VM's state is retained across the cycle, and the `MicrovmSandbox` is -kept (torn down only on real claim end). The ACK `Microvm` CR has no suspend field, so this loop -supplies the missing intent→SDK translation — pure shim, no virtual-kubelet. See +SandboxTemplate variant** (`agent-sandbox-lambda/templates/shim/`) whose pod is a lightweight +**bridge**: on claim it **reads the platform image handoff** (`imageARN` + `executionRoleARN` from the +one built `MicrovmSandbox`) and calls **`RunMicrovm`** (SDK) to launch this session's VM, records the +`microvmID` as an annotation on the owning `Sandbox`, and holds the pod so its lifecycle mirrors the +MicroVM's. On real teardown it calls `TerminateMicrovm`. To Flow B and the user the UX is identical to +Flow A. Interactive exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up. + +### Suspend / resume / terminate — the coder VM persists across the review loop + +Because the substrate is a Lambda MicroVM (not a pod), Flow D uses **suspend/resume through the Agent +Sandbox CRD** to keep the coder's context across the whole review→fix→re-review loop — the payoff of +this substrate over Kata (where each fix round claims a fresh pod): + +1. **Coder finishes the coding task → SUSPEND** (`df-run` flips `Sandbox.operatingMode=Suspended`; the + `microvm-lifecycle` reconcile loop calls `suspend-microvm` by the annotated id). Compute is freed; + the VM's memory/disk are snapshotted. +2. DevOps + Security agents review the PR while the coder is suspended. +3. **Findings + "fix" → RESUME the SAME VM** (df-iterate sets `operatingMode=Running` → `resume-microvm`). + Context intact — no cold re-implement. +4. Coder fixes → SUSPEND again; loop 2–4 until both agents clear. +5. **Final exit (merge) → TERMINATE** (`df-merge-teardown` calls `TerminateMicrovm`, then deletes the + claim). This is the **only** place the VM is destroyed — `df-run`'s onExit is substrate-aware and + **keeps** the suspended Lambda VM (unlike Kata, which frees its pod on df-run exit). + +The ACK `Microvm` has no suspend field, so the `microvm-lifecycle` loop supplies the intent→SDK +translation — pure shim, no virtual-kubelet. See [`diagrams/flow-d-microvm-sandbox.md` §D.3a](diagrams/flow-d-microvm-sandbox.md). ### Delivery & status -Shipped as GitOps, **disabled by default** (like the kata nodepool): a `microvm:` values block gates -the RGD + SandboxTemplate + self-managed controller addon; the hub overlay carries cluster-specific -values. The platform-capability enablement (Managed ACK + Managed KRO) lands separately in the -**appmod-blueprints** platform repo (they're EKS Capabilities, like the Managed ArgoCD the hub already -runs). This PR delivers the **design + GitOps scaffold**; the live end-to-end path (enable capabilities -→ sync controller → run a MicroVM coder) is the follow-up. +Shipped as GitOps in its **own chart** — `gitops/addons/charts/agent-sandbox-lambda/` (separate from +the Kata `agent-sandbox` chart), structured as `templates/image/` (KRO RGD + the one platform +`MicrovmSandbox`) and `templates/shim/` (bridge SandboxTemplate + warm pool + `microvm-lifecycle` +controller). **Disabled by default** (`microvm.enabled=false`); the hub overlay +(`clusters/hub/addons/agent-sandbox-lambda/values.yaml`) carries cluster-specific values, and a gated +`agent-sandbox-lambda` addon entry deploys it hub-only. The platform-capability enablement (Managed ACK ++ Managed KRO) lands separately in the **appmod-blueprints** platform repo (they're EKS Capabilities, +like the Managed ArgoCD the hub already runs). This PR delivers the **design + GitOps scaffold**; the +live end-to-end path (enable capabilities → sync controller → publish the arm64 artifact → run a MicroVM +coder with suspend/resume) is the follow-up. --- From 575ae48a61addf60405ce71af1cf7d9893b7ae6c Mon Sep 17 00:00:00 2001 From: elamaran shanmugam Date: Fri, 31 Jul 2026 17:37:41 -0400 Subject: [PATCH 20/67] feat(flow-d): controller IAM via ACK (all-GitOps) + enable on hub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per design: everything via Managed ACK + KRO, self-managed ACK only for the pre-GA lambdamicrovms controller — NO Crossplane, NO Terraform, NO CLI. - Controller bootstrap IAM is now an ACK iam.services.k8s.aws Role + ACK eks.services.k8s.aws PodIdentityAssociation (templates/shim/00-controller-pod-identity.yaml) — same Managed-ACK mechanism as the RGD's build/exec roles. Role trusts pods.eks.amazonaws.com; inline policy = lambda:*Microvm* + iam:PassRole (lambda). Bridge + lifecycle SAs get their own PodIdentityAssociations reusing the role. All 4 objects server-dry-run VERIFIED against the live ACK CRDs on the hub. - Enabled ack-lambdamicrovms controller addon (enabled:true) + microvm.enabled=true in the hub overlay (accountId + podIdentity set). Downstream (S3, build/exec roles, MicrovmImage) stays in the KRO RGD via ACK. Flow D only (PR #41); dark-factory (PR #32) untouched. --- gitops/addons/bootstrap/default/addons.yaml | 2 +- .../shim/00-controller-pod-identity.yaml | 117 ++++++++++++++++++ .../charts/agent-sandbox-lambda/values.yaml | 14 +++ .../addons/agent-sandbox-lambda/values.yaml | 8 +- 4 files changed, 139 insertions(+), 2 deletions(-) create mode 100644 gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml diff --git a/gitops/addons/bootstrap/default/addons.yaml b/gitops/addons/bootstrap/default/addons.yaml index d23f053b..84369c0b 100644 --- a/gitops/addons/bootstrap/default/addons.yaml +++ b/gitops/addons/bootstrap/default/addons.yaml @@ -502,7 +502,7 @@ kata-deploy: # CRDs + controller are up before the agent-sandbox chart's MicrovmSandbox RGD # (wave 2) references them. ack-lambdamicrovms: - enabled: false + enabled: true namespace: ack-system chartName: lambdamicrovms-chart defaultVersion: '0.1.1' diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml new file mode 100644 index 00000000..985b1211 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml @@ -0,0 +1,117 @@ +{{/* + Self-managed ACK lambdamicrovms controller → AWS access via EKS Pod Identity, + declared ALL-ACK (same mechanism as the RGD's build/exec roles): an ACK + iam.services.k8s.aws Role (trusted by the EKS Pod Identity service principal) + + an ACK eks.services.k8s.aws PodIdentityAssociation binding it to the controller's + ServiceAccount (ack-lambdamicrovms-controller in ack-system, created by the ACK + chart). No Crossplane, no Terraform, no CLI. + + This is the ONE bootstrap IAM the KRO RGD can't self-create (the controller needs + creds before it can create anything). It's reconciled by the Managed-ACK iam + eks + controllers (both live on the hub). Everything DOWNSTREAM — the S3 bucket, the build + role, the exec role, the MicrovmImage — is created by the KRO RGD via ACK too (see + templates/image/). So the entire IAM surface is ACK/KRO/GitOps. + + Earlier sync-wave so the role + association exist before the controller pod needs + them; Pod Identity creds are vended on demand + ArgoCD selfHeal converges with no + manual steps. Gated by microvm.enabled. +*/}} +{{- if and .Values.microvm .Values.microvm.enabled }} +apiVersion: iam.services.k8s.aws/v1alpha1 +kind: Role +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller + # Trust the EKS Pod Identity service principal (not IRSA/OIDC). + assumeRolePolicyDocument: | + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Service": "pods.eks.amazonaws.com"}, + "Action": ["sts:AssumeRole", "sts:TagSession"] + }] + } + inlinePolicies: + lambdamicrovms: | + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "LambdaMicrovms", + "Effect": "Allow", + "Action": [ + "lambda:CreateMicrovmImage","lambda:UpdateMicrovmImage","lambda:DeleteMicrovmImage", + "lambda:GetMicrovmImage","lambda:GetMicrovmImageVersion","lambda:ListMicrovmImages", + "lambda:RunMicrovm","lambda:GetMicrovm","lambda:TerminateMicrovm","lambda:ListMicrovms", + "lambda:SuspendMicrovm","lambda:ResumeMicrovm", + "lambda:TagResource","lambda:UntagResource","lambda:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "PassBuildExecRoles", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "*", + "Condition": {"StringEquals": {"iam:PassedToService": "lambda.amazonaws.com"}} + } + ] + } +--- +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ .Values.microvm.podIdentity.controllerNamespace | default "ack-system" }} + serviceAccount: {{ .Values.microvm.podIdentity.controllerServiceAccount | default "ack-lambdamicrovms-controller" }} + # ACK PodIdentityAssociation takes the role ARN (no role-ref selector). The ACK + # Role above has a deterministic name, so the ARN is constructed from the account id. + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +--- +# The BRIDGE SA (agent-sandbox-system) calls RunMicrovm/GetMicrovm/TerminateMicrovm — +# reuse the same lambda-microvms role via its own PodIdentityAssociation. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ include "agent-sandbox.namespace" . }} + serviceAccount: microvm-bridge + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +--- +# The LIFECYCLE controller SA calls suspend-microvm/resume-microvm — same role. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ include "agent-sandbox.namespace" . }} + serviceAccount: microvm-lifecycle + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml index 8ac0e4d2..3822fd2e 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -29,6 +29,20 @@ microvm: # AND the AWS CLI (RunMicrovm/suspend/resume/terminate). alpine/k8s bundles both. bridgeImage: alpine/k8s:1.31.0 + # AWS account id — used to construct the controller role ARN for the ACK + # PodIdentityAssociation (which takes an ARN, not a role-ref). Overlay per cluster. + accountId: "" + + # Controller bootstrap IAM (ALL-ACK: iam.services.k8s.aws Role + eks.services.k8s.aws + # PodIdentityAssociation — templates/shim/00-controller-pod-identity.yaml). This is + # the ONE IAM the KRO RGD can't self-create (creds-before-create). Managed by the + # Managed-ACK iam+eks controllers on the hub. The bridge/lifecycle SAs reuse this + # same role (they also call lambda-microvms). + podIdentity: + clusterName: hub + controllerNamespace: ack-system + controllerServiceAccount: ack-lambdamicrovms-controller + # The ONE platform image built by KRO/ACK (10-rgd-microvm-image.yaml). Built ONCE # per cluster; the shim reads its status (imageARN + executionRoleARN) to RunMicrovm # per session. Lambda MicroVM is ARM_64-ONLY. codeArtifactUri is an S3 URI diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 0b2a3057..878f7702 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -6,10 +6,16 @@ # coder artifact is published to S3. Managed KRO + Managed ACK capabilities are # ACTIVE on the hub. The ARNs below are public identifiers (not secrets). microvm: - enabled: false + enabled: true region: us-west-2 apiGroup: sandbox.agents.x-k8s.io bridgeImage: alpine/k8s:1.31.0 + # Hub account (for the ACK PodIdentityAssociation role ARN). + accountId: "940019131157" + podIdentity: + clusterName: hub + controllerNamespace: ack-system + controllerServiceAccount: ack-lambdamicrovms-controller # AWS-published base MicroVM image (ARM_64 — the only arch Lambda MicroVM supports). baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" # Lambda MicroVM codeArtifact.uri is S3-ONLY (a zip with the coder app + a Dockerfile); From ba514ac230be22ff89202f8d635bae7cf3ae4b6c Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Sat, 1 Aug 2026 19:23:32 -0400 Subject: [PATCH 21/67] feat(flow-d): re-apply df-run orchestration onto reverted #32 (old/working structure) After #45 was reverted (PR back to old flat-addons + working ACK structure), reset flow-d to the pre-#45-merge state and re-merged the CURRENT #32 (47b8908, contains all the colleague's still-valid Flow B work minus the reverted hub-sandbox commit). Then re-applied the Flow D df-run orchestration (dropped when I took their df-run in the merge): substrate-branched claim (lambda warm pool on darkfactory-lambda), microvm-suspend DAG step + microvm-set-mode template (suspend after coder pushes), substrate-aware teardown (Kata deletes claim; Lambda keeps suspended VM until merge). df-merge-teardown already has microvm-terminate. Renders clean (default + microvmSuspend). --- .../templates/20-workflowtemplate-df-run.yaml | 84 ++++++++++++++++--- 1 file changed, 74 insertions(+), 10 deletions(-) diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index 97706361..6ac2b3dc 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -93,6 +93,11 @@ spec: tasks: - name: claim template: claim-sandbox + arguments: + parameters: + # Flow D: darkfactory-lambda → the Lambda-MicroVM warm pool; else Kata. + - name: warm-pool + value: "{{`{{=workflow.parameters['trigger-label'] == 'darkfactory-lambda' ? '`}}{{ .Values.warmPool.lambdaName | default "coder-warmpool-microvm" }}{{`' : '`}}{{ .Values.warmPool.name }}{{`'}}`}}" - name: drive-coder template: await-coder dependencies: [claim] @@ -100,6 +105,22 @@ spec: parameters: - name: sandbox value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" +{{- if .Values.microvmSuspend.enabled }} + # Flow D ONLY — after the coder pushes, SUSPEND the Lambda MicroVM + # (operatingMode=Suspended → microvm-lifecycle calls suspend-microvm). The VM + # persists suspended across the review→fix loop; df-iterate resumes the SAME + # VM; terminated only at merge. Gated on the lambda substrate + a PR; no-op for Kata. + - name: microvm-suspend + template: microvm-set-mode + dependencies: [drive-coder] + when: "{{`{{workflow.parameters.trigger-label}}`}} == darkfactory-lambda && {{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: sandbox + value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" + - name: mode + value: "Suspended" +{{- end }} {{- if .Values.holdout.enabled }} # P2 — holdout gate: hidden scenarios + a different-family judge. - name: holdout-gate @@ -184,6 +205,11 @@ spec: # ---- 1. Claim a warm sandbox (creates the SandboxClaim with issue env) ---- - name: claim-sandbox + inputs: + parameters: + # Which warm pool to claim from — Kata (default) or Lambda-MicroVM (Flow D). + - name: warm-pool + value: "{{ .Values.warmPool.name }}" outputs: parameters: - name: sandbox @@ -209,7 +235,7 @@ spec: dark-factory.io/managed-by: df-run spec: warmPoolRef: - name: {{ .Values.warmPool.name }} + name: "{{`{{inputs.parameters.warm-pool}}`}}" lifecycle: ttlSecondsAfterFinished: {{ .Values.claimTtlSeconds }} env: @@ -809,13 +835,51 @@ spec: node /scripts/status.js # ---- onExit: release the claim (operator refills the pool) ---- +{{- if .Values.microvmSuspend.enabled }} + # ---- Flow D: set Sandbox.operatingMode (suspend/resume the Lambda MicroVM) ---- + # Flips spec.operatingMode; the agent-sandbox-lambda microvm-lifecycle controller + # observes it and calls suspend-microvm / resume-microvm by the microvm-id + # annotation the bridge wrote. Advisory verify (never fails the run). Lambda only. + - name: microvm-set-mode + inputs: + parameters: + - name: sandbox + - name: mode # Running | Suspended + script: + image: {{ .Values.microvmSuspend.image | default "alpine/k8s:1.31.0" }} + command: [sh] + source: | + set -eu + SB="{{`{{inputs.parameters.sandbox}}`}}" + MODE="{{`{{inputs.parameters.mode}}`}}" + NS="{{ .Values.warmPool.namespace }}" + echo "[microvm-set-mode] Sandbox/$SB -> operatingMode=$MODE" + kubectl patch sandbox "$SB" -n "$NS" --type merge -p "{\"spec\":{\"operatingMode\":\"$MODE\"}}" + i=0; while [ "$i" -lt 12 ]; do + VMID=$(kubectl get sandbox "$SB" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") + [ -n "$VMID" ] && { echo "[microvm-set-mode] microvmID=$VMID"; break; } + i=$((i+1)); sleep 5 + done + echo "[microvm-set-mode] done (advisory)." + activeDeadlineSeconds: 180 +{{- end }} + + # ---- onExit: release the claim (operator refills the pool) ---- + # SUBSTRATE-AWARE: Kata deletes the claim now (df-run done → free the pool). LAMBDA + # (Flow D) KEEPS the claim + SUSPENDED VM so the same VM survives the review→fix loop + # (df-iterate resumes it); the VM is terminated only at merge (df-merge-teardown). - name: teardown - resource: - action: delete - flags: ["--ignore-not-found"] - manifest: | - apiVersion: extensions.agents.x-k8s.io/v1beta1 - kind: SandboxClaim - metadata: - name: df-issue-{{`{{workflow.parameters.issue-id}}`}} - namespace: {{ .Values.warmPool.namespace }} + script: + image: {{ .Values.stepImage }} + command: [sh] + source: | + set -eu + LABEL="{{`{{workflow.parameters.trigger-label}}`}}" + NS="{{ .Values.warmPool.namespace }}" + CLAIM="df-issue-{{`{{workflow.parameters.issue-id}}`}}" + if [ "$LABEL" = "darkfactory-lambda" ]; then + echo "[teardown] lambda substrate — KEEPING ${CLAIM} (VM stays SUSPENDED until merge)." + exit 0 + fi + echo "[teardown] kata substrate — deleting SandboxClaim/${CLAIM}." + kubectl delete sandboxclaim "${CLAIM}" -n "${NS}" --ignore-not-found --wait=false || true From 1e7227d3121152e979ca6689fac008e6415b1fbd Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Sun, 2 Aug 2026 21:02:34 -0400 Subject: [PATCH 22/67] fix(flow-d): sync-wave the RGD before its MicrovmSandbox instance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The MicrovmSandbox instance failed dry-run ('CRD not found') and blocked the whole app sync (nothing applied, incl the RGD that GENERATES that CRD — chicken-and-egg within the app). RGD → sync-wave -1 (KRO generates the CRD first); instance → wave 1 + SkipDryRunOnMissingResource so the first pass doesn't block. ArgoCD retries/selfHeal converge. --- .../templates/image/10-rgd-and-image.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml index 1ab75f0f..53ebaa8f 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml @@ -36,6 +36,11 @@ metadata: name: microvm-sandbox labels: {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + # Apply the RGD FIRST so KRO generates the MicrovmSandbox CRD before the instance + # (below) is synced — otherwise the instance fails dry-run ("CRD not found") and + # blocks the whole app. + argocd.argoproj.io/sync-wave: "-1" spec: schema: apiVersion: v1alpha1 @@ -168,6 +173,12 @@ metadata: namespace: {{ include "agent-sandbox.namespace" . }} labels: {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + # Sync AFTER the RGD (wave -1) so its generated CRD exists. SkipDryRunOnMissingResource + # lets the first sync proceed even if KRO hasn't registered the CRD in the same pass — + # ArgoCD retries + selfHeal converge once the CRD appears (no whole-app block). + argocd.argoproj.io/sync-wave: "1" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: name: {{ .Values.microvm.image.name | default "coder" }} region: {{ .Values.microvm.region | quote }} From 8b81ed1274cb213ffac4a743add9a9aa77ab1698 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 11:31:28 -0400 Subject: [PATCH 23/67] fix(agent-sandbox): allow coder DNS egress to public resolvers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The coder Kata VM resolves via public DNS (8.8.8.8/1.1.1.1) since a Kata guest can't use in-cluster CoreDNS and EKS Auto Mode runs no kube-dns Service. coder-sandbox-egress only permitted :53 to namespaceSelector{} (in-cluster), so name resolution silently failed — the coder crashed with EAI_AGAIN api.github.com even though :443 egress worked, and no PR was ever pushed. Add a :53 egress rule scoped to the resolver /32s (networkPolicy. dnsResolvers), tighter than the existing :443 public allow. --- .../templates/30-networkpolicy.yaml | 18 ++++++++++++++++++ gitops/addons/charts/agent-sandbox/values.yaml | 11 +++++++++++ 2 files changed, 29 insertions(+) diff --git a/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml b/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml index 183cb77e..545c3c2d 100644 --- a/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml @@ -41,6 +41,24 @@ spec: port: 53 - protocol: TCP port: 53 + # DNS to the PUBLIC resolvers the Kata guest VM actually uses (pod + # dnsConfig.nameservers). The in-cluster :53 rule above can't cover these — + # they're external IPs — and Auto Mode has no kube-dns Service, so without + # this the coder can't resolve ANY name (crashes EAI_AGAIN api.github.com even + # though :443 egress works). Scoped to the resolver /32s (from values), so this + # opens :53 to nothing but the DNS servers themselves. + {{- if .Values.networkPolicy.dnsResolvers }} + - to: + {{- range .Values.networkPolicy.dnsResolvers }} + - ipBlock: + cidr: {{ . | quote }} + {{- end }} + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + {{- end }} # Bifrost LLM gateway (in-cluster) — the ONLY control-plane-side service the # coder may reach, and only on :8080. - to: diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index 1c896db1..37ee7831 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -63,6 +63,17 @@ networkPolicy: - 172.16.0.0/12 # RFC-1918 - 192.168.0.0/16 # RFC-1918 - 169.254.0.0/16 # link-local (blocks IMDS 169.254.169.254) + # Public DNS resolvers the coder Kata VM uses (its pod dnsConfig.nameservers). + # A Kata guest VM can't use in-cluster CoreDNS (and EKS Auto Mode doesn't run a + # kube-dns Service here), so the coder resolves via PUBLIC DNS. The egress policy + # must therefore allow :53 to these resolver IPs — the in-cluster :53 rule alone + # does NOT (public resolver IPs aren't in-cluster), which silently broke name + # resolution (coder crashed with EAI_AGAIN api.github.com though :443 worked). + # Scoped to the exact resolver /32s — tighter than the :443 public allow. Keep in + # sync with the SandboxTemplate podTemplate dnsConfig.nameservers. + dnsResolvers: + - 8.8.8.8/32 # Google Public DNS + - 1.1.1.1/32 # Cloudflare DNS # Admin-tier ClusterNetworkPolicy (31-clusternetworkpolicy.yaml) — control-plane # isolation that applies regardless of pod ownership (the standard NetworkPolicy From dfdca5624fb4e04be6006993ee31a7d0efa87bb0 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 11:31:29 -0400 Subject: [PATCH 24/67] fix(dark-factory): retain completed workflows 7d (was 1h) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ttlStrategy.secondsAfterCompletion was 3600, so runs were GC'd an hour after finishing — and there's no workflow DB archive on this cluster, so they were lost for good (Friday's runs gone). Bump to 604800 (7d) via new argo.workflowTtlSecondsAfterCompletion value across df-run/df-iterate/ df-merge-teardown so demo+debug history survives a work week. --- .../dark-factory/templates/20-workflowtemplate-df-run.yaml | 2 +- .../templates/21-workflowtemplate-df-merge-teardown.yaml | 2 +- .../templates/22-workflowtemplate-df-iterate.yaml | 2 +- gitops/addons/charts/dark-factory/values.yaml | 7 +++++++ 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index 6ac2b3dc..504fb33a 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -62,7 +62,7 @@ spec: # Always release the claimed sandbox, on success OR failure. onExit: teardown ttlStrategy: - secondsAfterCompletion: 3600 + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} {{- if .Values.metrics.enabled }} # Success metrics (Argo-native Prometheus). Argo exposes these on the workflow # controller's :9090/metrics — scraped into the platform Prometheus. Gives the diff --git a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml index 319758fe..56f297fa 100644 --- a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml +++ b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml @@ -32,7 +32,7 @@ spec: - name: repo - name: pr-number ttlStrategy: - secondsAfterCompletion: 3600 + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} templates: - name: main dag: diff --git a/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml b/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml index 379dd7d7..a02da4b5 100644 --- a/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml +++ b/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml @@ -35,7 +35,7 @@ spec: - name: comment-author value: "" ttlStrategy: - secondsAfterCompletion: 3600 + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} templates: - name: main dag: diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index ce427912..6698b67a 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -9,6 +9,13 @@ namespace: agent-sandbox-system # Argo Workflows install (already on the hub in the `argo` namespace). argo: namespace: argo + # How long a COMPLETED workflow (success or failure) is retained before Argo's + # controller garbage-collects it. There is NO workflow archive/DB persistence on + # this cluster (workflow-controller-configmap has no `persistence:` block), so once + # TTL fires the run is gone for good — not archived. 7 days keeps demo/debug history + # visible in the Argo UI across a work week. Was 3600 (1h), which silently dropped + # runs an hour after they finished. + workflowTtlSecondsAfterCompletion: 604800 warmPool: # The Flow A SandboxWarmPool the factory claims from (Kata substrate, default). From aaaaff68c02a3097bbd39f87d4076846f3b07cd6 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 11:31:52 -0400 Subject: [PATCH 25/67] fix(flow-d): use kro.run apiGroup + grant KRO graph child RBAC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes so the MicrovmSandbox RGD actually reconciles on the hub: 1. apiGroup kro.run (was sandbox.agents.x-k8s.io): EKS Managed KRO's controller only watches the kro.run group — an RGD on any other group stays state=Inactive ('cache sync timeout ... microvmsandboxes'). Verified by group-probe on the hub. Generated CRD is microvmsandboxes. kro.run; kind (MicrovmSandbox) unchanged. 2. New 40-kro-graph-rbac.yaml: KRO runs as the cluster's KRO capability role (EKS access entry, session KRO). AmazonEKSKROPolicy grants kro.run but NOT CRUD on the ACK children the graph creates, so reconcile hit 'forbidden: cannot get buckets'. Grant a ClusterRole scoped to exactly the 3 child groups (s3 buckets, iam roles, lambdamicrovms images/vms) bound to that identity (microvm.kroCapability values). --- .../templates/image/10-rgd-and-image.yaml | 4 +- .../shim/20-bridge-sandboxtemplate.yaml | 2 +- .../templates/shim/30-microvm-lifecycle.yaml | 2 +- .../templates/shim/40-kro-graph-rbac.yaml | 74 +++++++++++++++++++ .../charts/agent-sandbox-lambda/values.yaml | 23 +++++- .../addons/agent-sandbox-lambda/values.yaml | 3 +- 6 files changed, 102 insertions(+), 6 deletions(-) create mode 100644 gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml index 53ebaa8f..328ec202 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml @@ -45,7 +45,7 @@ spec: schema: apiVersion: v1alpha1 kind: MicrovmSandbox - group: {{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }} + group: {{ .Values.microvm.apiGroup | default "kro.run" | quote }} # SCOPE: this RGD builds ONLY the platform IMAGE + the slow-changing infra it # needs (S3 artifact bucket, build role, execution role). It deliberately does # NOT create a `Microvm` — a MicroVM instance is a per-SESSION, request-time @@ -166,7 +166,7 @@ spec: # here in the chart, not in the per-claim shim. The shim reads THIS object's status # (imageARN + executionRoleARN) to RunMicrovm per session. Rebuild the image by # bumping microvm.codeArtifactUri (a new artifact) and re-syncing. -apiVersion: {{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" }}/v1alpha1 +apiVersion: {{ .Values.microvm.apiGroup | default "kro.run" }}/v1alpha1 kind: MicrovmSandbox metadata: name: {{ .Values.microvm.image.name | default "coder" }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index eca454ce..602c085c 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -49,7 +49,7 @@ rules: # executionRoleARN). It does NOT create/delete it — that object is GitOps-owned # platform infra (template 50), built once. Per-session Run/Terminate is done via # the AWS SDK, not by mutating this CR. - - apiGroups: [{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" | quote }}] + - apiGroups: [{{ .Values.microvm.apiGroup | default "kro.run" | quote }}] resources: ["microvmsandboxes"] verbs: ["get", "list", "watch"] # Read the owning Sandbox (suspend-vs-teardown in preStop) + patch it to record the diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml index eb0f4d3c..fc64dcac 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml @@ -75,7 +75,7 @@ data: set -eu NS="{{ include "agent-sandbox.namespace" . }}" REGION="{{ .Values.microvm.region }}" - APIGROUP="{{ .Values.microvm.apiGroup | default "sandbox.agents.x-k8s.io" }}" + APIGROUP="{{ .Values.microvm.apiGroup | default "kro.run" }}" ANN="microvm-lifecycle.agents.x-k8s.io/last-mode" INTERVAL="{{ .Values.microvm.lifecycle.intervalSeconds | default 15 }}" echo "[microvm-lifecycle] reconciling every ${INTERVAL}s (ns=${NS} region=${REGION})" diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml new file mode 100644 index 00000000..7d13f124 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml @@ -0,0 +1,74 @@ +{{/* + KRO graph child-resource RBAC. + + The MicrovmSandbox ResourceGraphDefinition (templates/image/) is reconciled by EKS + Managed KRO, whose controller authenticates to the API server as the cluster's KRO + capability role (EKS access entry, session name "KRO"). The AWS-managed + AmazonEKSKROPolicy attached to that access entry grants KRO its own kro.run perms and + discovery, but NOT create/update/delete on the ACK kinds this graph composes. So when + KRO tries to materialize the graph it fails: + + resource reconciliation failed: buckets.s3.services.k8s.aws "coder-microvm-artifacts" + is forbidden: User ".../hub-KROCapabilityRole/KRO" cannot get resource "buckets" ... + + This ClusterRole grants KRO CRUD on EXACTLY the three ACK groups the MicrovmSandbox + graph creates as children — s3 Buckets, iam Roles, and the self-managed lambdamicrovms + MicrovmImages/Microvms — and nothing else. K8s RBAC cannot scope list/watch/create by + resource NAME, so the grant is per-kind; it deliberately does NOT include core/*, apps, + rbac, secrets, or any other ACK service. delete is required so KRO can garbage-collect + the graph's children when a MicrovmSandbox is removed. + + Gated by microvm.enabled AND microvm.podIdentity.kroCapability.enabled. Rendered as + ArgoCD sync-wave -2 (with the controller bootstrap IAM) so KRO can watch/CRUD the + children before the platform MicrovmSandbox instance (wave 1) reconciles. +*/}} +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- with .Values.microvm.kroCapability }} +{{- if .enabled }} +{{- $cluster := $.Values.microvm.podIdentity.clusterName }} +{{- $account := $.Values.microvm.accountId | toString }} +{{- $role := .roleName | default (printf "%s-KROCapabilityRole" $cluster) }} +{{- $session := .sessionName | default "KRO" }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ $cluster }}-kro-microvm-graph + labels: + {{- include "agent-sandbox.labels" $ | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +rules: + # S3 artifact bucket (graph resource `bucket`). + - apiGroups: ["s3.services.k8s.aws"] + resources: ["buckets", "buckets/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Build + exec IAM roles (graph resources `buildRole`, `execRole`). + - apiGroups: ["iam.services.k8s.aws"] + resources: ["roles", "roles/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Platform image + (future) per-session instance (graph resource `image`; Microvm is + # driven imperatively by the shim but kept here so KRO can read/GC if ever graphed). + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvmimages", "microvmimages/status", "microvms", "microvms/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ $cluster }}-kro-microvm-graph + labels: + {{- include "agent-sandbox.labels" $ | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ $cluster }}-kro-microvm-graph +subjects: + # The EKS access-entry username Managed KRO's controller presents. + - kind: User + name: "arn:aws:sts::{{ $account }}:assumed-role/{{ $role }}/{{ $session }}" + apiGroup: rbac.authorization.k8s.io +{{- end }} +{{- end }} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml index 3822fd2e..e81abc5b 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -24,7 +24,13 @@ microvm: enabled: false region: us-west-2 # API group the generated MicrovmSandbox CRD is served under (KRO schema.group). - apiGroup: sandbox.agents.x-k8s.io + # MUST be "kro.run": EKS Managed KRO's controller only watches the kro.run group — + # an RGD whose schema.group is anything else (even a *.kro.run subdomain) never leaves + # state=Inactive ("cache sync timeout ... Resource=microvmsandboxes"), because the + # capability's controller identity (hub-KROCapabilityRole via AmazonEKSKROPolicy) has + # no list/watch on other groups. The generated CRD is microvmsandboxes.kro.run; the + # kind (MicrovmSandbox) is unchanged. Verified by group-probe on the hub 2026-08-03. + apiGroup: kro.run # Bridge/lifecycle pod image: needs kubectl (read platform image, patch Sandbox) # AND the AWS CLI (RunMicrovm/suspend/resume/terminate). alpine/k8s bundles both. bridgeImage: alpine/k8s:1.31.0 @@ -43,6 +49,21 @@ microvm: controllerNamespace: ack-system controllerServiceAccount: ack-lambdamicrovms-controller + # EKS Managed KRO runs its controller as the cluster's KRO capability role. On this + # cluster its k8s identity is the EKS access-entry username + # arn:aws:sts:::assumed-role/-KROCapabilityRole/KRO + # (session name "KRO"). AmazonEKSKROPolicy grants KRO its own kro.run perms but NOT + # CRUD on the ACK children the MicrovmSandbox RGD graph creates (s3 buckets, iam + # roles, lambdamicrovms images/instances) — so KRO's instance reconcile hits + # "forbidden: ... cannot get resource buckets". templates/shim/40-kro-graph-rbac.yaml + # grants exactly those child kinds to this identity. Override roleName/sessionName if + # your cluster's capability wiring differs (confirm via + # aws eks describe-access-entry --principal-arn .../KROCapabilityRole). + kroCapability: + enabled: true + roleName: "" # defaults to "-KROCapabilityRole" + sessionName: KRO + # The ONE platform image built by KRO/ACK (10-rgd-microvm-image.yaml). Built ONCE # per cluster; the shim reads its status (imageARN + executionRoleARN) to RunMicrovm # per session. Lambda MicroVM is ARM_64-ONLY. codeArtifactUri is an S3 URI diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 878f7702..fe33ffbf 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -8,7 +8,8 @@ microvm: enabled: true region: us-west-2 - apiGroup: sandbox.agents.x-k8s.io + # MUST be kro.run — EKS Managed KRO only watches the kro.run group (see chart values.yaml). + apiGroup: kro.run bridgeImage: alpine/k8s:1.31.0 # Hub account (for the ACK PodIdentityAssociation role ARN). accountId: "940019131157" From 70c2fdca6c8906b6eb66801127fe4495badd9774 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 11:38:31 -0400 Subject: [PATCH 26/67] fix(flow-d): quote operands in microvm-suspend when-expression The Kata substrate showed microvm-suspend as phase=Error (type=Skipped): Argo substitutes the trigger-label value inline, so the unquoted 'dark-factory == darkfactory-lambda' parsed as arithmetic on bare identifiers and errored ('Failed to evaluate when expression'). Harmless (the step is correctly skipped on Kata and the run still completes) but it surfaces a spurious Error node. Quote both operands so it evaluates as a clean string comparison and skips silently on non-lambda runs. --- .../dark-factory/templates/20-workflowtemplate-df-run.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index 504fb33a..d92f3982 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -113,7 +113,11 @@ spec: - name: microvm-suspend template: microvm-set-mode dependencies: [drive-coder] - when: "{{`{{workflow.parameters.trigger-label}}`}} == darkfactory-lambda && {{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + # Quote BOTH operands: Argo substitutes the label value inline, so an + # unquoted `dark-factory == darkfactory-lambda` is parsed as arithmetic on + # bare identifiers and errors ("Failed to evaluate 'when' expression") — + # the step then shows phase=Error even though it's correctly Skipped on Kata. + when: "\"{{`{{workflow.parameters.trigger-label}}`}}\" == \"darkfactory-lambda\" && \"{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}\" != \"\"" arguments: parameters: - name: sandbox From ee5d6ffe1213d2b5478f52bd63d1a6fce52f68d2 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 11:50:41 -0400 Subject: [PATCH 27/67] fix(dark-factory): default microvmSuspend off (Flow D only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit microvmSuspend.enabled defaulted true, so the df-run DAG always rendered a microvm-suspend task — on a Kata (Flow B) run it's runtime-skipped but still appears as a node in the graph, an irrelevant/confusing Lambda-MicroVM step in a pipeline with no MicroVM. Default it false so the step isn't rendered at all unless a cluster runs the Lambda substrate; enable per-cluster overlay for darkfactory-lambda. --- gitops/addons/charts/dark-factory/values.yaml | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 6698b67a..498bb848 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -261,9 +261,16 @@ iterate: # When the darkfactory-lambda label runs, suspend the MicroVM after the coder # pushes (freeing compute while gates run) and resume it on a df-iterate comment. # Drives Sandbox.spec.operatingMode; the agent-sandbox microvm-lifecycle controller -# translates that to suspend-microvm / resume-microvm. No-op for the Kata substrate. +# translates that to suspend-microvm / resume-microvm. +# +# DEFAULT false: this is Flow D (Lambda MicroVM) ONLY. When enabled it adds a +# microvm-suspend task to the df-run DAG; on a Kata (Flow B) run that task is +# runtime-skipped, but it STILL renders into the graph — an irrelevant, confusing +# node in a pipeline that has no MicroVM. Keep it OFF unless a cluster actually runs +# the Lambda substrate, so the Kata pipeline graph contains only Kata-relevant steps. +# Flip to true (per-cluster overlay) on a cluster wired for darkfactory-lambda. microvmSuspend: - enabled: true + enabled: false image: alpine/k8s:1.31.0 # needs kubectl # ── Language / stack support ───────────────────────────────────────────────── From 82daaa15d521b56b7bb21a0642ad1a472db24e3b Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 13:23:35 -0400 Subject: [PATCH 28/67] fix(flow-d): scope controller iam:PassRole by target ARN, drop service condition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CreateMicrovmImage/RunMicrovm was denied iam:PassRole on coder-microvm-build: the statement conditioned on iam:PassedToService=lambda.amazonaws.com, but the Lambda MicroVM sub-service passes to a different principal (verified: sim allowed for lambda.amazonaws.com yet the live API denied; microvms/microvm.lambda.amazonaws.com also implicitDeny). Replace the brittle service condition with a Resource scope to arn:...:role/*-microvm-build + *-microvm-exec — net-tighter (only these two purpose- built roles can be passed) and principal-agnostic. Post-change sim: allowed. --- .../shim/00-controller-pod-identity.yaml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml index 985b1211..15529448 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml @@ -55,12 +55,26 @@ spec: ], "Resource": "*" }, + {{- /* + Scope PassRole by the TARGET role ARN (the build/exec roles KRO creates), + NOT by an iam:PassedToService condition. CreateMicrovmImage/RunMicrovm pass + the role to the Lambda MicroVM sub-service whose principal is NOT plain + lambda.amazonaws.com — a StringEquals on lambda.amazonaws.com fails closed, + so the controller got AccessDenied on iam:PassRole for coder-microvm-build + (verified: sim ALLOWED for lambda.amazonaws.com yet the live API DENIED, i.e. + the real passed-to principal differs; microvms/microvm.lambda.amazonaws.com + also implicitDeny). ARN-scoping to *-microvm-build/-exec is net-TIGHTER than + the previous Resource:* — only these two purpose-built roles can be passed — + and is principal-agnostic so it survives whatever sub-service MicroVM uses. + */ -}} { "Sid": "PassBuildExecRoles", "Effect": "Allow", "Action": "iam:PassRole", - "Resource": "*", - "Condition": {"StringEquals": {"iam:PassedToService": "lambda.amazonaws.com"}} + "Resource": [ + "arn:aws:iam::{{ .Values.microvm.accountId }}:role/*-microvm-build", + "arn:aws:iam::{{ .Values.microvm.accountId }}:role/*-microvm-exec" + ] } ] } From 8c9a08d3c94c06f6c7fd96388e515fad1a007fae Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 13:27:13 -0400 Subject: [PATCH 29/67] fix(flow-d): controller needs lambda:PassNetworkConnector for image build After PassRole was fixed, CreateMicrovmImage next failed on lambda:PassNetworkConnector for the AWS-managed INTERNET_EGRESS connector (the default egress attached when the RGD sets no explicit connectors). Add lambda:PassNetworkConnector + List/GetNetworkConnector scoped to arn:aws:lambda::*:network-connector:*. --- .../shim/00-controller-pod-identity.yaml | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml index 15529448..82fad73a 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml @@ -51,10 +51,28 @@ spec: "lambda:GetMicrovmImage","lambda:GetMicrovmImageVersion","lambda:ListMicrovmImages", "lambda:RunMicrovm","lambda:GetMicrovm","lambda:TerminateMicrovm","lambda:ListMicrovms", "lambda:SuspendMicrovm","lambda:ResumeMicrovm", - "lambda:TagResource","lambda:UntagResource","lambda:ListTagsForResource" + "lambda:TagResource","lambda:UntagResource","lambda:ListTagsForResource", + "lambda:ListNetworkConnectors","lambda:GetNetworkConnector" ], "Resource": "*" }, + {{- /* + CreateMicrovmImage/RunMicrovm attach a network connector to the MicroVM. + With no explicit connectors in the RGD, the service uses the AWS-managed + default INTERNET_EGRESS connector, and passing it needs + lambda:PassNetworkConnector (verified: image build denied + 'lambda:PassNetworkConnector on .../network-connector:aws-network-connector: + INTERNET_EGRESS'). Scope to the AWS-managed connector ARNs in-region. + */ -}} + { + "Sid": "PassNetworkConnectors", + "Effect": "Allow", + "Action": "lambda:PassNetworkConnector", + "Resource": [ + "arn:aws:lambda:{{ .Values.microvm.region }}:aws:network-connector:*", + "arn:aws:lambda:{{ .Values.microvm.region }}:{{ .Values.microvm.accountId }}:network-connector:*" + ] + }, {{- /* Scope PassRole by the TARGET role ARN (the build/exec roles KRO creates), NOT by an iam:PassedToService condition. CreateMicrovmImage/RunMicrovm pass From e32a08dbf406687e159f52a1ee631723a08faee9 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 13:37:29 -0400 Subject: [PATCH 30/67] fix(flow-d): point codeArtifactUri at the RGD-created bucket MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The overlay codeArtifactUri named dark-factory-microvm-artifacts--, but the RGD creates the artifact bucket as ${image.name}-microvm-artifacts (coder-microvm- artifacts). The image build read from a bucket the RGD never provisioned → NoSuchBucket / CREATE_FAILED. Point it at s3://coder-microvm-artifacts/coder-v0.2.5-arm64.zip. --- .../hub/addons/agent-sandbox-lambda/values.yaml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index fe33ffbf..55112038 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -21,10 +21,12 @@ microvm: baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" # Lambda MicroVM codeArtifact.uri is S3-ONLY (a zip with the coder app + a Dockerfile); # it is NOT an ECR image reference. The Dockerfile inside MAY pull the arm64 - # dark-factory-coder from ECR as a base layer (the build role keeps ecr:Get*). Publish - # the artifact zip to this bucket/key before flipping enabled=true. - # TODO: create the artifact zip (Dockerfile FROM the v0.2.5-arm64 coder) + upload. - codeArtifactUri: "s3://dark-factory-microvm-artifacts-940019131157-us-west-2/coder-v0.2.5-arm64.zip" + # dark-factory-coder from ECR as a base layer (the build role keeps ecr:Get*). + # MUST live in the bucket the RGD creates: ${image.name}-microvm-artifacts (here + # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD + # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the + # artifact zip to this exact bucket/key. + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64.zip" image: enabled: true name: coder From 556e7c72a5d199c8ae17aa1bbf90c37734f6698c Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 14:41:42 -0400 Subject: [PATCH 31/67] fix(flow-d): name the bridge container 'coder' for claim-injection parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flow B's SandboxClaim injects env (DF_ISSUE_NUMBER, DF_REPO, ...) into a container named 'coder' — the same contract as the Kata substrate. The microvm bridge template named its container 'bridge', so the operator rejected every claim targeting the microvm warmpool ('target container coder not found in template'). Rename bridge->coder (still runs bridge.sh -> RunMicrovm) so the Lambda substrate is transparent to df-run. --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 602c085c..0e039f4a 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -194,7 +194,13 @@ spec: seccompProfile: type: RuntimeDefault containers: - - name: bridge + # Named `coder` (NOT `bridge`) on purpose: Flow B's SandboxClaim injects env + # (DF_ISSUE_NUMBER, DF_REPO, …) into a container called `coder` — the SAME claim + # contract as the Kata substrate. The operator REJECTS the claim + # ("target container coder not found") if this name differs, so the substrate + # must expose a `coder` container to stay transparent to df-run. This container + # is still the bridge (runs bridge.sh → RunMicrovm); only the name matches Kata. + - name: coder image: {{ .Values.microvm.bridgeImage }} command: ["/bin/sh", "/scripts/bridge.sh"] # Preflight suspend marker: if this pod is being deleted because the owning From 6a97fedbbdf8b41fad437b1b3ad0f3e76100bbe0 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 14:53:32 -0400 Subject: [PATCH 32/67] fix(flow-d): allow bridge egress to the K8s API server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge carries role=coder so the shared coder-sandbox-egress policy applies — which denies the service+VPC CIDRs on :443 to isolate untrusted coder code from the control plane. But the bridge (trusted platform code) MUST reach the API server to read the MicrovmSandbox image handoff + annotate the Sandbox; without it kubectl hangs and it never RunMicrovm's. Add an additive NetworkPolicy selecting only the bridge's distinct substrate=lambda-microvm label, allowing :443 to the apiserver (svc ClusterIP + VPC endpoint CIDRs). Kata coder isolation is untouched (different selector). --- .../shim/20-bridge-sandboxtemplate.yaml | 41 +++++++++++++++++++ .../charts/agent-sandbox-lambda/values.yaml | 10 +++++ 2 files changed, 51 insertions(+) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 0e039f4a..ff98cd7a 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -37,6 +37,47 @@ metadata: labels: {{- include "agent-sandbox.labels" . | nindent 4 }} --- +{{- /* + Bridge API-server egress. + + The bridge pod carries agent-sandbox.io/role=coder (so Flow B's claim contract + + the shared coder-sandbox-egress policy apply to it). But that policy denies RFC-1918 + + the service CIDR on :443 to isolate UNTRUSTED coder code from the control plane — + and the K8s API server lives exactly there (kubernetes svc 172.20.0.1 + apiserver + endpoints in the VPC 10.0.0.0/8). NetworkPolicies are additive, so this ADDS an + egress allow for the API server, selected ONLY on the bridge's distinct + agent-sandbox.io/substrate=lambda-microvm label (Kata coders don't have it, so their + isolation is untouched). The bridge runs TRUSTED platform code (reads the MicrovmSandbox + image handoff + annotates the owning Sandbox with the microvmID) — unlike the Kata coder + it MUST reach the API server, or `kubectl get microvmsandbox` hangs and it never + RunMicrovm's (verified: in-pod kubectl to 172.20.0.1:443 timed out under coder-egress). +*/ -}} +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: microvm-bridge-apiserver-egress + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + agent-sandbox.io/substrate: lambda-microvm + policyTypes: + - Egress + egress: + # K8s API server — service ClusterIP + the in-VPC apiserver endpoints, :443 only. + - to: + - ipBlock: + cidr: {{ .Values.microvm.apiServerCidr | default "172.20.0.1/32" | quote }} + {{- range .Values.microvm.apiServerEndpointCidrs | default (list "10.0.0.0/16") }} + - ipBlock: + cidr: {{ . | quote }} + {{- end }} + ports: + - protocol: TCP + port: 443 +--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml index e81abc5b..6944ad72 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -35,6 +35,16 @@ microvm: # AND the AWS CLI (RunMicrovm/suspend/resume/terminate). alpine/k8s bundles both. bridgeImage: alpine/k8s:1.31.0 + # K8s API-server egress for the bridge (microvm-bridge-apiserver-egress NetworkPolicy). + # The bridge needs :443 to the API server to read the MicrovmSandbox handoff + annotate + # the Sandbox, but the shared coder-egress policy denies the service+VPC CIDRs. These + # scope the additive allow to the API server only. apiServerCidr = the `kubernetes` + # service ClusterIP /32; apiServerEndpointCidrs = the VPC range holding the apiserver + # endpoint IPs. Override per cluster (kubectl get svc kubernetes; get endpoints kubernetes). + apiServerCidr: "172.20.0.1/32" + apiServerEndpointCidrs: + - "10.0.0.0/16" + # AWS account id — used to construct the controller role ARN for the ACK # PodIdentityAssociation (which takes an ARN, not a role-ref). Overlay per cluster. accountId: "" From df957cdf0d841fcef018150734348cf23d1817ba Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 15:01:48 -0400 Subject: [PATCH 33/67] fix(flow-d): bridge/lifecycle image needs aws-cli v2 (has lambda-microvms) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit alpine/k8s:1.31.0 ships aws-cli 1.34, which does NOT know the pre-GA lambda-microvms service — 'aws lambda-microvms run-microvm' just printed the service list, so the bridge never launched a VM. Switch bridgeImage to public.ecr.aws/aws-cli/aws-cli:latest (glibc, always-current v2, has lambda-microvms — same image the security-agent uses) and fetch a static kubectl at start in bridge.sh + reconcile.sh (that image has no kubectl). --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 12 ++++++++++++ .../templates/shim/30-microvm-lifecycle.yaml | 9 +++++++++ .../addons/charts/agent-sandbox-lambda/values.yaml | 9 ++++++--- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index ff98cd7a..74d3b8d0 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -130,6 +130,18 @@ data: # mirror lifecycle. Idles until a SandboxClaim injects DF_ISSUE_NUMBER (Flow B), # exactly like the Kata coder. Needs AWS creds (Pod Identity) for the SDK calls. set -eu + # The bridge image is public.ecr.aws/aws-cli/aws-cli:latest — a glibc, always-current + # aws-cli v2 that KNOWS the pre-GA `lambda-microvms` service (the alpine/k8s image's + # aws-cli 1.34 does NOT: `aws lambda-microvms` printed the service list = unrecognized, + # so RunMicrovm never fired). That image has no kubectl, so fetch a static one here + # (same pattern as the security-agent bootstrap). Both are then on PATH. + if ! command -v kubectl >/dev/null 2>&1; then + echo "[microvm-bridge] fetching kubectl..." + ARCH="$(uname -m)"; case "$ARCH" in aarch64|arm64) A=arm64;; *) A=amd64;; esac + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/${A}/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi REGION="{{ .Values.microvm.region }}" PLATFORM_IMAGE="{{ .Values.microvm.image.name | default "coder" }}" # the ONE committed MicrovmSandbox (template 50) NS="{{ include "agent-sandbox.namespace" . }}" diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml index fc64dcac..6f741a80 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml @@ -73,6 +73,15 @@ data: # Reconcile Sandbox.operatingMode -> Lambda MicroVM suspend/resume. # Idempotent: we stamp the last-acted mode on an annotation and only act on change. set -eu + # bridgeImage is aws-cli v2 (has lambda-microvms; alpine/k8s's aws-cli 1.34 does not) + # but has no kubectl — fetch a static one (same as bridge.sh / security-agent). + if ! command -v kubectl >/dev/null 2>&1; then + echo "[microvm-lifecycle] fetching kubectl..." + ARCH="$(uname -m)"; case "$ARCH" in aarch64|arm64) A=arm64;; *) A=amd64;; esac + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/${A}/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi NS="{{ include "agent-sandbox.namespace" . }}" REGION="{{ .Values.microvm.region }}" APIGROUP="{{ .Values.microvm.apiGroup | default "kro.run" }}" diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml index 6944ad72..db733ca5 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -31,9 +31,12 @@ microvm: # no list/watch on other groups. The generated CRD is microvmsandboxes.kro.run; the # kind (MicrovmSandbox) is unchanged. Verified by group-probe on the hub 2026-08-03. apiGroup: kro.run - # Bridge/lifecycle pod image: needs kubectl (read platform image, patch Sandbox) - # AND the AWS CLI (RunMicrovm/suspend/resume/terminate). alpine/k8s bundles both. - bridgeImage: alpine/k8s:1.31.0 + # Bridge/lifecycle pod image: needs the AWS CLI that KNOWS the pre-GA lambda-microvms + # service (RunMicrovm/suspend/resume/terminate) AND kubectl. alpine/k8s bundles kubectl + # but its aws-cli 1.34 does NOT have lambda-microvms (verified: prints the service list + # = unrecognized). So use the glibc, always-current aws-cli v2 (has lambda-microvms) and + # fetch kubectl at start (bridge.sh / lifecycle loop do this, same as the security-agent). + bridgeImage: public.ecr.aws/aws-cli/aws-cli:latest # K8s API-server egress for the bridge (microvm-bridge-apiserver-egress NetworkPolicy). # The bridge needs :443 to the API server to read the MicrovmSandbox handoff + annotate From 8a16250ef320f3bfad0f304e46470d345db2f8be Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 15:05:25 -0400 Subject: [PATCH 34/67] =?UTF-8?q?fix(flow-d):=20hub=20overlay=20bridgeImag?= =?UTF-8?q?e=20=E2=86=92=20aws-cli=20v2=20(was=20pinning=20alpine/k8s)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hub overlay pinned bridgeImage: alpine/k8s:1.31.0, overriding the chart default — so the bridge/lifecycle kept the aws-cli that lacks lambda-microvms. Align the overlay to public.ecr.aws/aws-cli/aws-cli:latest. --- .../clusters/hub/addons/agent-sandbox-lambda/values.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 55112038..40125ab0 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -10,7 +10,9 @@ microvm: region: us-west-2 # MUST be kro.run — EKS Managed KRO only watches the kro.run group (see chart values.yaml). apiGroup: kro.run - bridgeImage: alpine/k8s:1.31.0 + # aws-cli v2 (has the pre-GA lambda-microvms service; alpine/k8s's aws-cli 1.34 does NOT, + # so RunMicrovm never fired). bridge.sh/reconcile.sh fetch kubectl at start. See chart values. + bridgeImage: public.ecr.aws/aws-cli/aws-cli:latest # Hub account (for the ACK PodIdentityAssociation role ARN). accountId: "940019131157" podIdentity: From 2cebef5f9724d985f76b6965327d94df648ff464 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 15:23:04 -0400 Subject: [PATCH 35/67] fix(flow-d): allow bridge egress to EKS Pod Identity endpoint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RunMicrovm failed 'retrieving credentials from container-role: connect timeout http://169.254.170.23/v1/credentials' — the bridge gets its AWS creds from the Pod Identity agent at link-local 169.254.170.23, but the shared coder-egress policy denies 169.254.0.0/16 (IMDS block for untrusted coder code). Allow ONLY the Pod Identity /32:80 for the bridge selector; IMDS (169.254.169.254) stays denied, Kata coder untouched. --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 74d3b8d0..b715a314 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -77,6 +77,19 @@ spec: ports: - protocol: TCP port: 443 + # EKS Pod Identity credential endpoint (link-local 169.254.170.23:80). The bridge + # gets its AWS creds — to call lambda-microvms run/suspend/resume/terminate — from + # the Pod Identity agent here. The shared coder-egress policy denies 169.254.0.0/16 + # (to block IMDS for untrusted coder code), which ALSO blocks Pod Identity, so the + # bridge's run-microvm failed "retrieving credentials from container-role: connect + # timeout http://169.254.170.23/v1/credentials". Allow ONLY the Pod Identity /32 (NOT + # IMDS 169.254.169.254, which stays denied) and ONLY for the bridge selector. + - to: + - ipBlock: + cidr: {{ .Values.microvm.podIdentityEndpoint | default "169.254.170.23/32" | quote }} + ports: + - protocol: TCP + port: 80 --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role From 2c7738dd21cc8daba8ad1d1953fb195be86e5959 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 15:27:59 -0400 Subject: [PATCH 36/67] fix(flow-d): bridge SANDBOX_NAME from downward API (was df- mismatch) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge annotates the owning Sandbox with the microvm-id so the lifecycle controller can suspend/resume it. But SANDBOX_NAME fell back to df-${DF_ISSUE_NUMBER} (e.g. df-9999), while agent-sandbox names the Sandbox after the CLAIM (e.g. df-issue-smoke-d) — so the annotate silently failed, suspend couldn't find the VM, and pod teardown TERMINATED it instead. Set SANDBOX_NAME from the pod's own metadata.name (pod==Sandbox name); fix the preStop fallback too. --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index b715a314..fec0210e 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -269,6 +269,17 @@ spec: - name: coder image: {{ .Values.microvm.bridgeImage }} command: ["/bin/sh", "/scripts/bridge.sh"] + # SANDBOX_NAME = this pod's own name via the downward API. agent-sandbox names + # the Sandbox CR and its pod identically, so this IS the owning Sandbox name. + # Without it bridge.sh fell back to df- (e.g. df-9999) which does NOT + # match the real Sandbox (named after the claim, e.g. df-issue-smoke-d), so the + # microvm-id annotation write silently failed → the lifecycle controller couldn't + # find the VM to suspend and the pod teardown TERMINATED it instead of suspending. + env: + - name: SANDBOX_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name # Preflight suspend marker: if this pod is being deleted because the owning # Sandbox went operatingMode=Suspended, the microvm-lifecycle controller has # (or will) suspend the MicroVM — the bridge must NOT TerminateMicrovm. The @@ -281,7 +292,7 @@ spec: - -c - | # If the owning Sandbox is Suspended, mark so cleanup() keeps the CR. - M=$(kubectl get sandbox "df-${DF_ISSUE_NUMBER:-}" \ + M=$(kubectl get sandbox "${SANDBOX_NAME:-df-${DF_ISSUE_NUMBER:-}}" \ -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") [ "$M" = "Suspended" ] && touch /tmp/suspending || true securityContext: From 7911579ca6c35601fec92d0dd09c564a33a6c9d1 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 15:33:45 -0400 Subject: [PATCH 37/67] fix(flow-d): re-enable microvmSuspend (hub runs both substrates) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turned it off earlier to keep the Kata graph clean, but Flow D REQUIRES the microvm-suspend step (suspend-after-code). It's when-gated + now skips cleanly on Kata (quoted operands → Skipped, not Error), so a both-substrate hub should have it on. A Kata-only cluster can set it false. --- gitops/addons/charts/dark-factory/values.yaml | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 498bb848..549e9272 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -263,15 +263,16 @@ iterate: # Drives Sandbox.spec.operatingMode; the agent-sandbox microvm-lifecycle controller # translates that to suspend-microvm / resume-microvm. # -# DEFAULT false: this is Flow D (Lambda MicroVM) ONLY. When enabled it adds a -# microvm-suspend task to the df-run DAG; on a Kata (Flow B) run that task is -# runtime-skipped, but it STILL renders into the graph — an irrelevant, confusing -# node in a pipeline that has no MicroVM. Keep it OFF unless a cluster actually runs -# the Lambda substrate, so the Kata pipeline graph contains only Kata-relevant steps. -# Flip to true (per-cluster overlay) on a cluster wired for darkfactory-lambda. +# ENABLE on clusters that run the Lambda substrate (Flow D). It adds a microvm-suspend +# task to the df-run DAG that, on a darkfactory-lambda run, sets Sandbox.operatingMode= +# Suspended after the coder pushes (the microvm-lifecycle controller then calls +# suspend-microvm). On a Kata (Flow B) run the step is when-gated and skips cleanly +# (quoted operands — a clean Skipped node, NOT the earlier Error). On a Kata-ONLY cluster +# set this false so the DAG has zero MicroVM steps. This hub runs BOTH substrates, so +# it's on: Flow D needs it, and the Kata skip is harmless. microvmSuspend: - enabled: false - image: alpine/k8s:1.31.0 # needs kubectl + enabled: true + image: alpine/k8s:1.31.0 # only patches Sandbox.operatingMode via kubectl (no aws-cli) # ── Language / stack support ───────────────────────────────────────────────── # There is NO per-language profile config here — deliberately. Language support is From e56dd91d2744e1b22697cda3a2fa7dfc618900db Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 17:05:11 -0400 Subject: [PATCH 38/67] docs(flow-d): design for running the coder inside the MicroVM MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Substrate + lifecycle are proven E2E, but running the actual coder in the VM is an application re-architecture, not wiring: the /run lifecycle hook has a 30s timeout and the MicroVM model is a long-running HTTP service (request/response), while the coder is a one-shot 5-15min batch job. runHookPayload is an HTTP body to a /run endpoint the app must SERVE (a SecretKeyReference on the Microvm CR) — NOT env/file. Document the async /run wrapper approach + VPC egress connector (bootstrap Job) + Bifrost internal NLB + runHookPayload secret + SG rules, with open questions, before building. --- .../flow-d-coder-in-microvm-design.md | 127 ++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 docs/dark-factory/flow-d-coder-in-microvm-design.md diff --git a/docs/dark-factory/flow-d-coder-in-microvm-design.md b/docs/dark-factory/flow-d-coder-in-microvm-design.md new file mode 100644 index 00000000..a79fe32d --- /dev/null +++ b/docs/dark-factory/flow-d-coder-in-microvm-design.md @@ -0,0 +1,127 @@ +# Flow D — Running the Coder *inside* the Lambda MicroVM (design) + +**Status:** design / spike — NOT implemented. Written after proving the Flow D **substrate** +end-to-end and discovering that running the actual coder in the VM is an application +re-architecture, not a wiring task. + +## TL;DR + +The Flow D **substrate + lifecycle is proven live**: a `darkfactory-lambda` GitHub issue → +Argo sensor → `df-run` claims the Lambda warm pool → the bridge calls `RunMicrovm` → a real +Lambda MicroVM reaches **RUNNING** in AWS → `suspend`/`resume`/`terminate` are wired → the VM +is terminated on teardown (verified, zero orphans). + +What is **NOT** done: the coder does not actually *execute* inside that MicroVM, so no PR is +produced. That is because the one-shot `dark-factory-coder` and the Lambda MicroVM +snapshot/hook execution model are **fundamentally different execution shapes**. Closing the gap +requires re-architecting the coder, plus VPC/Bifrost networking. This doc specifies that work +so it can be decided deliberately. + +## Why it isn't just wiring — the execution-model mismatch + +| | Kata coder (Flow B, works today) | Lambda MicroVM model | +| --- | --- | --- | +| Shape | **one-shot batch process**: `node entrypoint.js` runs clone→agent→push→PR, then exits | **long-lived HTTP service** that is *snapshotted* at build, *resumed* per session | +| Duration | 5–15 min per run | per-request; the `run` lifecycle hook has a **30s timeout** ("keep it short — on the critical path") | +| Trigger | pod start + `DF_ISSUE_NUMBER` env injected by the SandboxClaim | build-time `ready`/`validate` hooks; per-instance `run` hook receives `runHookPayload` as the HTTP request body | +| Secrets/context | files projected into the pod (`/etc/secrets/gh-token`, `bifrost-api-key`) + `DF_*` env | `runHookPayload` — a **Kubernetes `SecretKeyReference`** on the `Microvm` CR, delivered as the `/run` hook body (≤16 KB); image must set `hooks.microvmHooks.run: ENABLED` | +| Network | in-cluster: reaches Bifrost by ClusterIP `172.20.181.17:8080`; git/gh over public :443 | runs **outside the cluster network**; only `INTERNET_EGRESS` by default; cannot reach a ClusterIP; VPC reach needs an egress **network connector** | + +The killer facts (verified against `mmeckes/lambdamicrovms-controller` docs + the live `aws +lambda-microvms`/`lambda-core` CLIs, 2026-08-03): + +1. **`/run` hook = 30s timeout.** A 5–15 min coder run cannot happen *in* the hook. +2. **The intended app model is request/response** (02-developer-handoff: RunMicrovm → mint + auth token → HTTP request → response → terminate) — not a batch job. +3. **`runHookPayload` is delivered as an HTTP body to a `/run` endpoint the app must SERVE** — + NOT an env var and NOT a mounted file. (An earlier attempt at an env/file boot-shim was + wrong and is discarded.) +4. So the coder must be **wrapped in a long-running HTTP server** that starts the coding work + asynchronously — the coder's current `entrypoint.js` is not written that way. + +## Proposed design (async `/run` pattern) + +Keep the coder *logic* (`entrypoint.js`) intact; change how it is *invoked*. + +``` +build: MicrovmImage (FROM arm64 dark-factory-coder + a thin HTTP wrapper) + hooks.port: 8080 + microvmImageHooks.ready: server up → safe to snapshot + microvmHooks.run: ENABLED (30s), suspend/resume/terminate ENABLED + +run: controller delivers runHookPayload (Secret {ghToken, bifrostKey, bifrostUrl(NLB), + issueNumber, repo, branch, baseBranch, title}) as the /run body + → wrapper writes /etc/secrets/{gh-token,bifrost-api-key} + exports DF_*/BIFROST_URL + → wrapper spawns `node entrypoint.js` in the BACKGROUND, returns 200 within 30s + → coder does clone→agent→push→PR async (many minutes) + +observe: df-run's existing `await-coder` step ALREADY polls GitHub for the PR head — reuse it + verbatim; it doesn't care whether the coder ran in Kata or a MicroVM. + +teardown: suspend/resume/terminate hooks best-effort flush; bridge TerminateMicrovm on exit. +``` + +### Components to build + +1. **HTTP wrapper + artifact** (`coder-microvm/`): a small server (`server.js`) exposing + `ready`, `run`, `suspend`, `resume`, `terminate` on :8080; `run` materializes the payload + into the coder's existing file/env contract and background-spawns `entrypoint.js`. Dockerfile + `FROM 940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:-arm64`. Zip + (Dockerfile + server.js) → S3, per the controller's `ci/package-artifact.sh` format. + *(Supersedes the placeholder `microvm-entry.js` listener that only existed to get the image + to CREATED.)* + +2. **MicrovmImage: enable hooks** (RGD `templates/image/10-rgd-and-image.yaml`): add + `hooks.port: 8080`, `microvmImageHooks.ready: ENABLED`, `microvmHooks.run/suspend/resume/ + terminate: ENABLED`. Without `run: ENABLED` the payload is silently never delivered. + +3. **VPC egress connector** (bootstrap Job — honest: *GitOps-provisioned, not continuously + reconciled*; no ACK/Crossplane API exists for `lambda-core` connectors). Committed + find-or-create Job modeled on `06-securityagent-bootstrap.yaml`: + `aws lambda-core get/create-network-connector` with + `VpcEgressConfiguration={SubnetIds:[hub subnets], SecurityGroupIds:[sg], NetworkProtocol:IPv4}` + → writes the connector ARN to a ConfigMap the MicrovmImage `egressNetworkConnectors` reads. + IAM: the bootstrap/capability role needs `lambda-core:*NetworkConnector*` + the EC2 ENI perms + Lambda uses to provision ENIs. **Caveat:** if the connector is deleted out-of-band, nothing + self-heals until the Job re-runs (not a controller). + +4. **Bifrost VPC-reachable** (internal NLB — this part *is* declarative): a `Service + type=LoadBalancer` with `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` + + `nlb-target-type` in the bifrost chart, reconciled by the AWS Load Balancer Controller. The + MicroVM (via the egress connector) reaches Bifrost at the NLB's stable VPC address on :8080. + (Bifrost's pod IP `10.0.x.x` is in-VPC and reachable via the connector, but ephemeral — the + NLB gives a stable target. Its ClusterIP `172.20.x.x` is NOT routable from a VPC ENI.) + **Shared-infra change — needs owner sign-off.** + +5. **runHookPayload Secret + Microvm wiring**: the bridge (or a per-session step) writes a + Secret with the payload key and the `Microvm`/RunMicrovm references it as + `runHookPayload: {name, key}`. Since it's a SecretKeyReference the **controller** delivers it + — confirm whether the imperative `RunMicrovm` path the bridge uses accepts the same, or + whether this session should create a short-lived `Microvm` CR instead. + +6. **Security-group rules**: allow the connector ENIs → Bifrost NLB on :8080. + +## Open questions for review + +- **Async vs request-driven?** Background-spawn (df-run polls for the PR, minimal coder change) + vs. the reference's request/response model (bridge sends an HTTP "code this" request + waits; + needs the auth-token path). Background-spawn reuses `await-coder` and is less invasive. +- **Imperative RunMicrovm vs a `Microvm` CR per session?** `runHookPayload` being a + SecretKeyReference is controller-delivered; the current bridge calls `aws run-microvm` + imperatively. Decide whether per-session VMs become short-lived `Microvm` CRs (declarative + payload delivery) or stay imperative (verify the CLI accepts an inline/secret payload). +- **Cost:** the VPC egress connector provisions ENIs; the internal NLB is an hourly resource. + Both are ongoing while Flow D is enabled. +- **Is in-VM coder even required for the goal?** The substrate is a valid deliverable on its + own (a second sandbox substrate). Running the coder in it is the "make it actually code" step + — worth confirming it's in scope before the re-architecture. + +## What exists today (so nothing is lost) + +- Substrate live: RGD Active, S3 bucket, build/exec roles, **MicrovmImage CREATED (v1.0)**, + bridge launches/terminates a real MicroVM from a `darkfactory-lambda` issue. +- All the substrate + bridge fixes are committed on `flow-d-lambda-microvm-sandbox` (container + named `coder`, aws-cli v2 image, kubectl fetch, API-server + Pod Identity egress, + downward-API SANDBOX_NAME, microvmSuspend on, `darkfactory-lambda` label). +- The **placeholder** code artifact (`microvm-entry.js` listener) is what's in S3 today — it + only proved the image builds; it must be replaced per §1 above. From 27ee2b57ce9a871b87c970f7d75d62bbfa5427e6 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 18:48:31 -0400 Subject: [PATCH 39/67] =?UTF-8?q?feat(flow-d):=20lambda-coder=20=E2=80=94?= =?UTF-8?q?=20Bedrock-direct=20+=20/run=20hook=20wrapper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make the coder actually run inside the MicroVM, Bedrock-direct (no Bifrost/EKS-network dependency): - entrypoint.js: branch LLM transport on USE_BEDROCK. Kata (default)=Bifrost shim; Lambda=CLAUDE_CODE_USE_BEDROCK=1 + real Bedrock model id, creds from the exec role. - examples/dark-factory/coder-microvm/: hook-server.js serves the MicroVM lifecycle hooks on :8080; /run materializes runHookPayload (issue ctx + gh token) and BACKGROUND- spawns entrypoint.js (the 30s run-hook can't hold a 5-15min coder run), returning fast. Dockerfile FROM the arm64 coder image + overlays the updated entrypoint.js (no ECR rebuild). - RGD: MicrovmImage hooks (ready+run+suspend/resume/terminate) ENABLED; exec role gets bedrock:InvokeModel*. See docs/dark-factory/flow-d-coder-in-microvm-design.md. --- .../dark-factory/coder-microvm/Dockerfile | 32 +++++++ .../dark-factory/coder-microvm/hook-server.js | 87 +++++++++++++++++++ examples/dark-factory/coder/entrypoint.js | 73 +++++++++++----- .../templates/image/10-rgd-and-image.yaml | 33 ++++++- 4 files changed, 199 insertions(+), 26 deletions(-) create mode 100644 examples/dark-factory/coder-microvm/Dockerfile create mode 100644 examples/dark-factory/coder-microvm/hook-server.js diff --git a/examples/dark-factory/coder-microvm/Dockerfile b/examples/dark-factory/coder-microvm/Dockerfile new file mode 100644 index 00000000..3254805a --- /dev/null +++ b/examples/dark-factory/coder-microvm/Dockerfile @@ -0,0 +1,32 @@ +# Flow D — lambda-coder image: the dark-factory coder wrapped for the Lambda MicroVM +# snapshot/hook runtime. +# +# FROM the arm64 dark-factory-coder (Lambda MicroVM is ARM_64-only) — it carries +# entrypoint.js + the toolchain (git, node, claude-code). We add ONLY the hook server +# that adapts the one-shot coder to the MicroVM lifecycle (see hook-server.js): the +# /run hook background-spawns entrypoint.js with USE_BEDROCK=1 so the coder calls +# Bedrock directly via the MicroVM execution role — no Bifrost / EKS-network path. +# +# The image is built by Lambda from a code-artifact ZIP (this Dockerfile + hook-server.js) +# in S3 — NOT pushed to ECR as a normal image. The base coder image IS pulled from ECR +# during that build (the MicrovmImage build role keeps ecr:Get*). +ARG CODER_IMAGE=940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:v0.2.5-arm64 +FROM ${CODER_IMAGE} + +WORKDIR /app + +# The hook server (serves ready/validate/run/suspend/resume/terminate on :8080). +COPY hook-server.js /app/hook-server.js + +# Ship the UPDATED coder over the one baked into the ECR base. The base image's +# entrypoint.js predates the USE_BEDROCK branch; overlaying it here means the +# MicrovmImage build (which pulls the ECR base) gets the Bedrock-capable coder +# WITHOUT a separate ECR rebuild+push. Keep in sync with examples/dark-factory/coder/entrypoint.js. +COPY entrypoint.js /app/entrypoint.js + +# Lambda MicroVM snapshots the process started here. hooks.port on the MicrovmImage +# must match this (8080). The server stays up (long-running) so the VM isn't idle- +# suspended mid coder-run — the Microvm idlePolicy.maxIdleDurationSeconds is set +# longer than a coder run. +EXPOSE 8080 +CMD ["node", "/app/hook-server.js"] diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js new file mode 100644 index 00000000..b3952805 --- /dev/null +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -0,0 +1,87 @@ +// Flow D — Lambda MicroVM hook server (the lambda-coder wrapper). +// +// Lambda MicroVM is a snapshot/hook runtime: the platform builds an image by +// starting THIS process and snapshotting it once the `ready` hook says "go", then +// resumes that snapshot per session and calls the `run` hook with the session's +// runHookPayload as the request body. Hooks are HTTP endpoints we serve on :8080 +// (hooks.port on the MicrovmImage); each must answer within its timeout (run = 30s). +// +// The dark-factory coder (entrypoint.js) is a ONE-SHOT batch job (clone → agent → +// push → PR, 5-15 min). It cannot run *inside* the 30s run hook. So the run hook is +// just the handshake: it materializes the payload into the coder's file/env contract +// and BACKGROUND-SPAWNS entrypoint.js, then returns 200 immediately. The coder then +// runs to completion asynchronously; df-run's await-coder step polls GitHub for the +// PR exactly as it does for Kata. The VM stays alive because idlePolicy. +// maxIdleDurationSeconds on the Microvm is set longer than a coder run (idle = no +// inbound traffic; a background job would otherwise auto-suspend). +// +// LLM: USE_BEDROCK=1 is exported so entrypoint.js calls Bedrock DIRECTLY via the +// MicroVM execution role — no Bifrost / EKS-network dependency (a MicroVM can't reach +// Bifrost's ClusterIP). See docs/dark-factory/flow-d-coder-in-microvm-design.md. + +const http = require("http"); +const fs = require("fs"); +const { spawn } = require("child_process"); + +const PORT = parseInt(process.env.HOOKS_PORT || "8080", 10); +const SECRETS_DIR = "/tmp/secrets"; + +let coderStarted = false; + +// Materialize the runHookPayload (JSON) into the coder's existing contract: +// files: /tmp/secrets/{gh-token} ; env: DF_*, AWS_REGION, USE_BEDROCK=1 +// then background-spawn entrypoint.js. Idempotent: only the first /run starts it. +function startCoder(payload) { + if (coderStarted) { + console.log("[hook-server] /run received again — coder already started, ignoring"); + return; + } + coderStarted = true; + + let d = {}; + try { d = JSON.parse(payload || "{}"); } catch (e) { console.log("[hook-server] payload not JSON:", e.message); } + + fs.mkdirSync(SECRETS_DIR, { recursive: true, mode: 0o700 }); + if (d.ghToken) fs.writeFileSync(`${SECRETS_DIR}/gh-token`, d.ghToken, { mode: 0o400 }); + + const env = { + ...process.env, + USE_BEDROCK: "1", // Bedrock-direct (exec-role creds) + GH_TOKEN_PATH: `${SECRETS_DIR}/gh-token`, + AWS_REGION: d.region || process.env.AWS_REGION || "us-west-2", + DF_ISSUE_NUMBER: d.issueNumber ? String(d.issueNumber) : "", + DF_REPO: d.repo || "", + DF_BRANCH: d.branch || (d.issueNumber ? `df/issue-${d.issueNumber}` : ""), + DF_BASE_BRANCH: d.baseBranch || "main", + DF_ISSUE_TITLE: d.issueTitle || "", + }; + if (d.model) env.CODER_MODEL = d.model; + + console.log(`[hook-server] /run → background-spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); + const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: "inherit", detached: false }); + child.on("exit", (code) => console.log(`[hook-server] coder exited code=${code}`)); + child.on("error", (e) => console.log(`[hook-server] coder spawn error: ${e.message}`)); +} + +const server = http.createServer((req, res) => { + let body = ""; + req.on("data", (c) => { body += c; }); + req.on("end", () => { + const ok = (obj) => { res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(obj || { status: "ok" })); }; + switch (req.url) { + // Build-time: signal the process is initialized so the builder snapshots a + // clean, waiting coder (no session work baked into the snapshot). + case "/ready": return ok({ status: "ready" }); + case "/validate": return ok({ status: "valid" }); + // Per-session start: body IS runHookPayload. Kick off the coder, return fast. + case "/run": startCoder(body); return ok({ status: "started" }); + // Lifecycle: coder holds no external state to flush; ack so the service proceeds. + case "/suspend": return ok({ status: "suspended" }); + case "/resume": return ok({ status: "resumed" }); + case "/terminate":return ok({ status: "terminated" }); + default: return ok({ status: "ok", path: req.url }); + } + }); +}); + +server.listen(PORT, () => console.log(`[hook-server] listening on :${PORT} (lambda-coder, Bedrock-direct)`)); diff --git a/examples/dark-factory/coder/entrypoint.js b/examples/dark-factory/coder/entrypoint.js index a8184e03..5b43fa31 100644 --- a/examples/dark-factory/coder/entrypoint.js +++ b/examples/dark-factory/coder/entrypoint.js @@ -226,28 +226,21 @@ http.createServer((cReq,cRes)=>{ } function runCoder(repoDir) { - // Bifrost is an Anthropic-compatible gateway. Point Claude Code at its - // /anthropic route via ANTHROPIC_BASE_URL + ANTHROPIC_API_KEY. Do NOT set - // CLAUDE_CODE_USE_BEDROCK — that flag makes Claude Code use the AWS Bedrock - // SDK directly (needs AWS creds in the VM, which we deliberately withhold) - // and ignores ANTHROPIC_BASE_URL. Bifrost auth is optional on this platform, - // so the key may be absent; send a placeholder so the CLI doesn't prompt. - const key = readSecret(BIFROST_KEY_PATH) || "bifrost"; - // Route through the localhost UA-shim (see startBifrostUaShim) so Bifrost - // doesn't apply its broken claude-cli request transform. - const base = startBifrostUaShim(`${BIFROST_URL.replace(/\/+$/, "")}/anthropic`); - const env = { + // Two LLM transports, selected by USE_BEDROCK: + // - Kata (Flow B, default): Bifrost gateway. The Kata VM is credential-less + + // in-cluster, so it reaches models through Bifrost's /anthropic route (which + // also gives centralized Langfuse observability). CLAUDE_CODE_USE_BEDROCK is + // deliberately UNSET here (it would make the CLI use the Bedrock SDK directly + // and ignore ANTHROPIC_BASE_URL). + // - Lambda MicroVM (Flow D): USE_BEDROCK=1. A MicroVM runs OUTSIDE the cluster + // network and can't reach Bifrost's ClusterIP; forcing it back in-cluster + // needed a VPC connector + internal NLB. Instead the MicroVM's EXECUTION ROLE + // grants bedrock:InvokeModel, so Claude Code calls Bedrock directly over public + // egress — no EKS network dependency. Trade-off: these calls bypass Bifrost's + // Langfuse telemetry (documented in flow-d-coder-in-microvm-design.md). + const useBedrock = /^(1|true|yes)$/i.test(process.env.USE_BEDROCK || ""); + const baseEnv = { ...process.env, - ANTHROPIC_BASE_URL: base, - ANTHROPIC_API_KEY: key, - // Bifrost maps model ALIASES → Bedrock model IDs. Claude Code's default - // model name (e.g. claude-sonnet-4) isn't a Bifrost alias and returns - // "provided model identifier is invalid" (400). Use the platform's Bifrost - // alias (verified: 'claude-sonnet' → us.anthropic.claude-sonnet-4-5). Set - // both the primary and the small/fast model so the CLI never falls back to - // an unknown identifier. - ANTHROPIC_MODEL: process.env.CODER_MODEL || "claude-sonnet", - ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_MODEL || "claude-sonnet", // The sandbox runs with readOnlyRootFilesystem, so $HOME (/home/node) is NOT // writable. Claude Code writes its config, session state, and — critically — // per-invocation SHELL SNAPSHOT files that its Bash tool sources before every @@ -262,9 +255,41 @@ function runCoder(repoDir) { // Non-interactive: never open a browser / prompt for login in headless mode. CI: "1", }; - fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); - delete env.CLAUDE_CODE_USE_BEDROCK; - console.log(`[coder] LLM: base=${base} model=${env.ANTHROPIC_MODEL}`); + let env; + if (useBedrock) { + // Bedrock-direct: creds come from the MicroVM execution role (Pod Identity / + // instance creds); the CLI uses the Bedrock SDK. Model must be a real Bedrock + // model ID (NOT a Bifrost alias). AWS_REGION comes from the runHookPayload/env. + env = { + ...baseEnv, + CLAUDE_CODE_USE_BEDROCK: "1", + AWS_REGION: process.env.AWS_REGION || process.env.CODER_REGION || "us-west-2", + ANTHROPIC_MODEL: process.env.CODER_MODEL || "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_SMALL_MODEL || process.env.CODER_MODEL || "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + }; + fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); + console.log(`[coder] LLM: Bedrock-direct region=${env.AWS_REGION} model=${env.ANTHROPIC_MODEL}`); + } else { + // Bifrost is an Anthropic-compatible gateway. Route through the localhost + // UA-shim so Bifrost doesn't apply its broken claude-cli request transform. + // Bifrost auth is optional; send a placeholder so the CLI doesn't prompt. + const key = readSecret(BIFROST_KEY_PATH) || "bifrost"; + const base = startBifrostUaShim(`${BIFROST_URL.replace(/\/+$/, "")}/anthropic`); + env = { + ...baseEnv, + ANTHROPIC_BASE_URL: base, + ANTHROPIC_API_KEY: key, + // Bifrost maps model ALIASES → Bedrock model IDs. Claude Code's default + // model name (e.g. claude-sonnet-4) isn't a Bifrost alias and returns + // "provided model identifier is invalid" (400). Use the platform's Bifrost + // alias (verified: 'claude-sonnet' → us.anthropic.claude-sonnet-4-5). + ANTHROPIC_MODEL: process.env.CODER_MODEL || "claude-sonnet", + ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_MODEL || "claude-sonnet", + }; + fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); + delete env.CLAUDE_CODE_USE_BEDROCK; + console.log(`[coder] LLM: Bifrost base=${base} model=${env.ANTHROPIC_MODEL}`); + } // Inherit stdio so the coder CLI's own output + errors stream into the pod // logs (kubectl logs), instead of being swallowed by execFileSync's exception. const opts = { cwd: repoDir, env, stdio: "inherit", maxBuffer: 64 * 1024 * 1024 }; diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml index 328ec202..55201363 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml @@ -112,8 +112,10 @@ spec: {"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:log-group:/aws/lambda/microvms/*"} ]} # 3) IAM role the RUNNING MicroVM assumes (GA — Managed ACK). App-owned exec identity. - # Minimal by default (the coder reaches models via Bifrost, git/gh via public - # egress — no AWS API needed); trust = lambda.amazonaws.com. + # trust = lambda.amazonaws.com. Grants bedrock:InvokeModel so the coder calls + # Bedrock DIRECTLY (Flow D is Bedrock-direct: a MicroVM runs outside the cluster + # and can't reach Bifrost's ClusterIP, so it uses this role's creds over public + # egress instead — see flow-d-coder-in-microvm-design.md). git/gh over public :443. - id: execRole template: apiVersion: iam.services.k8s.aws/v1alpha1 @@ -124,6 +126,11 @@ spec: name: ${schema.spec.name}-microvm-exec assumeRolePolicyDocument: | {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} + inlinePolicies: + bedrock-invoke: | + {"Version":"2012-10-17","Statement":[ + {"Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream","bedrock:Converse","bedrock:ConverseStream"],"Resource":["arn:aws:bedrock:*::foundation-model/*","arn:aws:bedrock:*:*:inference-profile/*"]} + ]} # 4) MicrovmImage (pre-GA — SELF-MANAGED lambdamicrovms controller). Platform-owned. # readyWhen gates the Microvm (resource 5) on a genuinely SUCCESSFUL build — # state CREATED/UPDATED — so the instance never launches from a half-built or @@ -147,6 +154,28 @@ spec: # is built for arm64 for exactly this substrate). cpuConfigurations: - architecture: ARM_64 + # Lifecycle hooks — the lambda-coder (hook-server.js) serves these on :8080. + # ready : build waits for the server to be up before snapshotting a + # clean, waiting coder (else the snapshot is taken too early). + # run : per-session start; delivers runHookPayload (issue context + + # GitHub token) as the request body → hook-server background- + # spawns the coder. WITHOUT run:ENABLED the payload is silently + # never delivered. + # suspend/resume/terminate : lifecycle acks (coder holds no external state). + hooks: + port: 8080 + microvmImageHooks: + ready: ENABLED + readyTimeoutInSeconds: 120 + microvmHooks: + run: ENABLED + runTimeoutInSeconds: 30 + suspend: ENABLED + suspendTimeoutInSeconds: 30 + resume: ENABLED + resumeTimeoutInSeconds: 30 + terminate: ENABLED + terminateTimeoutInSeconds: 30 # CloudWatch build/runtime logs. On a CREATE_FAILED the controller can't # see the build output — this is where it lands: # aws logs tail /aws/lambda/microvms/${schema.spec.name}-image From 7369d1b385dc761651843a6c2e2cc05929a47969 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:00:09 -0400 Subject: [PATCH 40/67] feat(flow-d): bridge passes runHookPayload (issue ctx + gh token) to RunMicrovm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge now builds a JSON payload (issue number/repo/branch/title + GitHub token + region) and passes it via --run-hook-payload on run-microvm (CLI form is a plain ≤16KB string). hook-server.js's /run receives it, writes /tmp/secrets/gh-token + DF_* env, and background-spawns the Bedrock-direct coder. Mounts the dark-factory-github secret (gh-token) read-only at /etc/df. No Bifrost key — Bedrock-direct via the exec role. --- .../shim/20-bridge-sandboxtemplate.yaml | 36 ++++++++++++++++--- 1 file changed, 32 insertions(+), 4 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index fec0210e..3f924646 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -183,14 +183,32 @@ data: fi echo "[microvm-bridge] image=${IMAGE_ARN} execRole=${EXEC_ROLE}" - # 2) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image. - # idlePolicy enables auto-suspend/resume; explicit suspend/resume is driven by - # the microvm-lifecycle controller (template 52) off Sandbox.operatingMode. - echo "[microvm-bridge] RunMicrovm..." + # 2) Build the runHookPayload — the per-session context the coder needs, delivered + # as the /run hook body (hook-server.js background-spawns the coder from it). + # Via the CLI --run-hook-payload is a PLAIN STRING (≤16KB); we pass JSON. Includes + # the GitHub token (mounted from the dark-factory-github secret at /etc/df/gh-token) + # + issue context from the claim env. NO Bifrost key — the coder is Bedrock-direct + # via the exec role. If the token file is absent the coder can't push; fail loud. + GH_TOKEN="$(cat /etc/df/gh-token 2>/dev/null || echo "")" + [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" + PAYLOAD=$(GH="${GH_TOKEN}" node -e ' + const p={ghToken:process.env.GH||"", region:process.env.REGION||"us-west-2", + issueNumber:process.env.DF_ISSUE_NUMBER||"", repo:process.env.DF_REPO||"", + branch:process.env.DF_BRANCH||"", baseBranch:process.env.DF_BASE_BRANCH||"main", + issueTitle:process.env.DF_ISSUE_TITLE||""}; + process.stdout.write(JSON.stringify(p));' REGION="${REGION}" 2>/dev/null) + + # 3) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image, + # handing it the payload. idlePolicy enables auto-suspend/resume; explicit + # suspend/resume is driven by the microvm-lifecycle controller off operatingMode. + # maxIdleDurationSeconds MUST exceed a coder run (idle = no inbound traffic; a + # background coder would otherwise be auto-suspended mid-run). + echo "[microvm-bridge] RunMicrovm (with runHookPayload for issue #${DF_ISSUE_NUMBER})..." RUN_JSON=$(aws lambda-microvms run-microvm \ --region "${REGION}" \ --image-identifier "${IMAGE_ARN}" \ --execution-role-arn "${EXEC_ROLE}" \ + --run-hook-payload "${PAYLOAD}" \ --idle-policy 'autoResumeEnabled=true,maxIdleDurationSeconds={{ .Values.microvm.defaults.maxIdleDurationSeconds }},suspendedDurationSeconds={{ .Values.microvm.defaults.suspendedDurationSeconds }}' \ 2>&1) || { echo "[microvm-bridge] run-microvm failed: ${RUN_JSON}"; exit 1; } VMID=$(echo "${RUN_JSON}" | sed -n 's/.*"[Mm]icrovmId"[ ]*:[ ]*"\([^"]*\)".*/\1/p' | head -1) @@ -308,6 +326,11 @@ spec: mountPath: /scripts - name: tmp mountPath: /tmp + # GitHub token (read-only) the bridge folds into the runHookPayload so the + # coder in the MicroVM can push + open the PR. Same secret the Kata coder uses. + - name: gh-token + mountPath: /etc/df + readOnly: true volumes: - name: bridge-script configMap: @@ -315,6 +338,11 @@ spec: defaultMode: 0555 - name: tmp emptyDir: {} + - name: gh-token + secret: + secretName: {{ .Values.microvm.githubSecretName | default "dark-factory-github" }} + defaultMode: 0400 + optional: true --- {{- /* Lambda-MicroVM SandboxWarmPool. SandboxClaim.spec.warmPoolRef is REQUIRED (a claim From 6e7070b7810e54bcd6524a79b942cf121719479f Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:05:12 -0400 Subject: [PATCH 41/67] fix(flow-d): build runHookPayload with python3, not node MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge image (aws-cli v2) has no node — the node -e payload builder crashed the bridge right after reading the handoff (CrashLoopBackOff, before RunMicrovm). Switch to python3 (present in the image) + json.dumps for safe escaping of the token/title. --- .../shim/20-bridge-sandboxtemplate.yaml | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 3f924646..6541fb1d 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -191,12 +191,19 @@ data: # via the exec role. If the token file is absent the coder can't push; fail loud. GH_TOKEN="$(cat /etc/df/gh-token 2>/dev/null || echo "")" [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" - PAYLOAD=$(GH="${GH_TOKEN}" node -e ' - const p={ghToken:process.env.GH||"", region:process.env.REGION||"us-west-2", - issueNumber:process.env.DF_ISSUE_NUMBER||"", repo:process.env.DF_REPO||"", - branch:process.env.DF_BRANCH||"", baseBranch:process.env.DF_BASE_BRANCH||"main", - issueTitle:process.env.DF_ISSUE_TITLE||""}; - process.stdout.write(JSON.stringify(p));' REGION="${REGION}" 2>/dev/null) + # Build the JSON payload with python3 (present in the aws-cli image; node is NOT). + # json.dumps handles escaping of the token + any quotes in the title safely. + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c ' +import json, os +print(json.dumps({ + "ghToken": os.environ.get("GH",""), + "region": os.environ.get("REGION","us-west-2"), + "issueNumber": os.environ.get("DF_ISSUE_NUMBER",""), + "repo": os.environ.get("DF_REPO",""), + "branch": os.environ.get("DF_BRANCH",""), + "baseBranch": os.environ.get("DF_BASE_BRANCH","main"), + "issueTitle": os.environ.get("DF_ISSUE_TITLE",""), +}))') # 3) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image, # handing it the payload. idlePolicy enables auto-suspend/resume; explicit From 72d8501ff3b93cd07bd4c4ee50a0eb0ef5b6826e Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:10:59 -0400 Subject: [PATCH 42/67] fix(flow-d): single-line python3 payload builder (YAML block-scalar fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The multi-line python at column 0 broke out of the bridge.sh: | block scalar → ArgoCD ComparisonError 'could not find expected :'. Collapse to one line so all script content stays indented under the block scalar. --- .../shim/20-bridge-sandboxtemplate.yaml | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 6541fb1d..ce940418 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -192,18 +192,10 @@ data: GH_TOKEN="$(cat /etc/df/gh-token 2>/dev/null || echo "")" [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" # Build the JSON payload with python3 (present in the aws-cli image; node is NOT). - # json.dumps handles escaping of the token + any quotes in the title safely. - PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c ' -import json, os -print(json.dumps({ - "ghToken": os.environ.get("GH",""), - "region": os.environ.get("REGION","us-west-2"), - "issueNumber": os.environ.get("DF_ISSUE_NUMBER",""), - "repo": os.environ.get("DF_REPO",""), - "branch": os.environ.get("DF_BRANCH",""), - "baseBranch": os.environ.get("DF_BASE_BRANCH","main"), - "issueTitle": os.environ.get("DF_ISSUE_TITLE",""), -}))') + # ONE line on purpose: a multi-line python body at column 0 would break out of the + # `bridge.sh: |` YAML block scalar (all lines must stay indented). json.dumps escapes + # the token + any quotes in the title. Reads DF_*/GH/REGION from the environment. + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE","")}))') # 3) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image, # handing it the payload. idlePolicy enables auto-suspend/resume; explicit From cfcb258bc891cde672cb935e1811a06f6c63507f Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:19:30 -0400 Subject: [PATCH 43/67] docs(flow-d): record in-VM build attempt + confirmed /run-hook blocker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bedrock-direct async design built + deployed: hook-server runs in the VM (verified via CloudWatch 'listening on :8080'), bridge passes runHookPayload, VM launches RUNNING — but the /run hook never fires so the coder doesn't start. Confirmed cause: runHookPayload is a SecretKeyReference delivered via the declarative Microvm CR (controller-driven), NOT the imperative run-microvm --run-hook-payload CLI the bridge uses. Correct path documented: per-session Microvm CR + Secret. hook-server/artifact/image/IAM/Bedrock are done + verified. --- .../flow-d-coder-in-microvm-design.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/dark-factory/flow-d-coder-in-microvm-design.md b/docs/dark-factory/flow-d-coder-in-microvm-design.md index a79fe32d..f21750e5 100644 --- a/docs/dark-factory/flow-d-coder-in-microvm-design.md +++ b/docs/dark-factory/flow-d-coder-in-microvm-design.md @@ -116,6 +116,43 @@ teardown: suspend/resume/terminate hooks best-effort flush; bridge TerminateMicr own (a second sandbox substrate). Running the coder in it is the "make it actually code" step — worth confirming it's in scope before the re-architecture. +## 2026-08-03 build attempt — where it got to + the confirmed blocker + +Built and deployed the Bedrock-direct async design end-to-end. Live results: + +- ✅ **lambda-coder artifact + image**: `examples/dark-factory/coder-microvm/` (hook-server.js + + Dockerfile FROM the arm64 coder + the USE_BEDROCK entrypoint branch). MicrovmImage rebuilt + to **UPDATED** with `hooks` enabled; exec role has `bedrock:InvokeModel*`. Verified the + hook-server runs in the VM — CloudWatch shows `[hook-server] listening on :8080 (lambda-coder, + Bedrock-direct)`. +- ✅ **Bridge payload**: builds JSON (issue ctx + GitHub token + region) with python3 (node is + absent in the aws-cli image) and passes `--run-hook-payload` on `run-microvm`. VM launches + RUNNING with the payload; no bridge crash. +- ❌ **BLOCKER: the `/run` hook never fires** → the coder never starts in the VM → no PR. + CloudWatch shows the server `listening` but never logs the `/run` handling / background-spawn. + +**Confirmed root cause:** `runHookPayload` is a **`SecretKeyReference`, "not a literal"** — the +docs + the 02-developer-handoff example deliver it via the **declarative `Microvm` CR** +(`runHookPayload: {name, key}` → the self-managed controller reads the Secret and drives the +`/run` hook). The imperative `run-microvm --run-hook-payload ""` CLI path the bridge uses +does **not** invoke `/run` (VM reaches RUNNING but the hook is silent). Two things also worth +noting from the reference: (a) the intended session model is the CLIENT minting an auth token and +sending an HTTP request to the VM endpoint with `X-aws-proxy-auth` (request/response), and (b) the +`/run` hook is service-internal on VM start. + +**Correct path (next):** make the per-session VM a **`Microvm` CR** (declarative), not a bridge +CLI call: +- bridge (or a per-session step) writes a **Secret** with the payload key, then creates a + `Microvm` CR referencing it (`imageIdentifierRef`, `executionRoleRef`, `runHookPayload:{name,key}`, + `idlePolicy`), and reads back `status.microvmID` for the lifecycle annotation. +- the self-managed lambdamicrovms controller reconciles it and delivers the payload to `/run`, + which background-spawns the coder. +- teardown = delete the `Microvm` CR (controller terminates), replacing the imperative + TerminateMicrovm. +This trades the imperative bridge for the declarative CR path the payload mechanism actually +requires — and it's MORE GitOps-faithful. Est: bridge rewrite (CR create/delete instead of CLI) ++ a per-session Secret; the hook-server/artifact/image/IAM/Bedrock pieces are already done and verified. + ## What exists today (so nothing is lost) - Substrate live: RGD Active, S3 bucket, build/exec roles, **MicrovmImage CREATED (v1.0)**, From f0e12fd70b21d4df6ed37045e3eaac2067981196 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:42:31 -0400 Subject: [PATCH 44/67] feat(flow-d): bridge creates Microvm CR + payload Secret (fires /run) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Switch the bridge from the imperative run-microvm CLI (which never fires /run) to the declarative path the runHookPayload SecretKeyReference actually requires: write a payload Secret + create a Microvm CR (imageIdentifier, executionRoleARN, runHookPayload{name,key}, idlePolicy); the self-managed controller reconciles it and delivers the payload to the /run hook → hook-server background-spawns the Bedrock-direct coder. Teardown deletes the CR (controller terminates the VM); suspend keeps it. Manifests built as JSON via python3 (no heredoc — a column-0 EOF breaks the bridge.sh block scalar). Bridge RBAC += microvms + secrets CRUD. --- .../shim/20-bridge-sandboxtemplate.yaml | 92 ++++++++++--------- 1 file changed, 50 insertions(+), 42 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index ce940418..7af5611c 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -99,13 +99,21 @@ metadata: labels: {{- include "agent-sandbox.labels" . | nindent 4 }} rules: - # The bridge only READS the platform MicrovmSandbox (image handoff: imageARN + - # executionRoleARN). It does NOT create/delete it — that object is GitOps-owned - # platform infra (template 50), built once. Per-session Run/Terminate is done via - # the AWS SDK, not by mutating this CR. + # READS the platform MicrovmSandbox (image handoff: imageARN + executionRoleARN) — + # GitOps-owned platform infra (template 50), built once. - apiGroups: [{{ .Values.microvm.apiGroup | default "kro.run" | quote }}] resources: ["microvmsandboxes"] verbs: ["get", "list", "watch"] + # Per-session Microvm CR: the bridge CREATES it (declarative path — the only way the + # controller fires the /run hook that delivers runHookPayload) and DELETES it on + # teardown (controller terminates the VM). + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvms"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Per-session payload Secret (runHookPayload SecretKeyReference). + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update", "patch", "delete"] # Read the owning Sandbox (suspend-vs-teardown in preStop) + patch it to record the # per-session microvmID (annotation) so the microvm-lifecycle controller can # suspend/resume THIS session's VM. @@ -185,60 +193,60 @@ data: # 2) Build the runHookPayload — the per-session context the coder needs, delivered # as the /run hook body (hook-server.js background-spawns the coder from it). - # Via the CLI --run-hook-payload is a PLAIN STRING (≤16KB); we pass JSON. Includes - # the GitHub token (mounted from the dark-factory-github secret at /etc/df/gh-token) - # + issue context from the claim env. NO Bifrost key — the coder is Bedrock-direct - # via the exec role. If the token file is absent the coder can't push; fail loud. + # CRITICAL: runHookPayload is a SecretKeyReference delivered by the DECLARATIVE + # Microvm CR (the controller reads the Secret + drives the /run hook). The + # imperative `run-microvm --run-hook-payload` CLI launches the VM but NEVER fires + # /run (verified: VM RUNNING, hook-server listening, but coder never started). So + # we write a Secret + create a Microvm CR. GitHub token from the mounted secret; + # NO Bifrost key — the coder is Bedrock-direct via the exec role. GH_TOKEN="$(cat /etc/df/gh-token 2>/dev/null || echo "")" [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" - # Build the JSON payload with python3 (present in the aws-cli image; node is NOT). - # ONE line on purpose: a multi-line python body at column 0 would break out of the - # `bridge.sh: |` YAML block scalar (all lines must stay indented). json.dumps escapes - # the token + any quotes in the title. Reads DF_*/GH/REGION from the environment. + # python3 (node is absent in the aws-cli image); json.dumps escapes token/title. One + # line so it stays inside the bridge.sh: | YAML block scalar. PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE","")}))') - # 3) RunMicrovm (IMPERATIVE SDK) — launch this session's VM from the pre-built image, - # handing it the payload. idlePolicy enables auto-suspend/resume; explicit - # suspend/resume is driven by the microvm-lifecycle controller off operatingMode. - # maxIdleDurationSeconds MUST exceed a coder run (idle = no inbound traffic; a - # background coder would otherwise be auto-suspended mid-run). - echo "[microvm-bridge] RunMicrovm (with runHookPayload for issue #${DF_ISSUE_NUMBER})..." - RUN_JSON=$(aws lambda-microvms run-microvm \ - --region "${REGION}" \ - --image-identifier "${IMAGE_ARN}" \ - --execution-role-arn "${EXEC_ROLE}" \ - --run-hook-payload "${PAYLOAD}" \ - --idle-policy 'autoResumeEnabled=true,maxIdleDurationSeconds={{ .Values.microvm.defaults.maxIdleDurationSeconds }},suspendedDurationSeconds={{ .Values.microvm.defaults.suspendedDurationSeconds }}' \ - 2>&1) || { echo "[microvm-bridge] run-microvm failed: ${RUN_JSON}"; exit 1; } - VMID=$(echo "${RUN_JSON}" | sed -n 's/.*"[Mm]icrovmId"[ ]*:[ ]*"\([^"]*\)".*/\1/p' | head -1) - echo "[microvm-bridge] launched MicroVM ${VMID:-}" + MVM="mvm-${DF_ISSUE_NUMBER}" # Microvm CR + payload Secret name for this session + # Build BOTH manifests as JSON with python3 and pipe to kubectl apply. JSON (not a + # heredoc) on purpose: a heredoc's column-0 EOF terminator breaks out of the + # bridge.sh: | YAML block scalar. JSON is valid YAML and stays on indented lines. + MAXIDLE={{ .Values.microvm.defaults.maxIdleDurationSeconds }}; SUSPDUR={{ .Values.microvm.defaults.suspendedDurationSeconds }} + echo "[microvm-bridge] writing payload Secret + Microvm CR ${MVM} (fires /run for issue #${DF_ISSUE_NUMBER})..." + MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":mvm+"-payload","namespace":ns},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] payload secret apply failed"; exit 1; } + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"idlePolicy":{"autoResumeEnabled":True,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } - # 3) Record microvmID on the owning Sandbox so the lifecycle controller can - # suspend/resume THIS session's VM (it reads this annotation). + # 4) Wait for the controller to report the running VM's id, record it on the Sandbox + # (lifecycle controller reads this to suspend/resume THIS session's VM). + VMID=""; i=0 + while [ "$i" -lt 60 ]; do + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -n "${VMID}" ] && break + i=$((i+1)); sleep 5 + done + echo "[microvm-bridge] Microvm ${MVM} -> ${VMID:-}" [ -n "${VMID}" ] && kubectl annotate sandbox "${SANDBOX_NAME}" -n "${NS}" \ "microvm-lifecycle.agents.x-k8s.io/microvm-id=${VMID}" --overwrite >/dev/null 2>&1 || true - # 4) Teardown on real claim end (NOT on suspend): suspend deletes the pod but must - # KEEP the MicroVM so resume works. /tmp/suspending marker (preStop) distinguishes. + # 5) Teardown on real claim end (NOT on suspend): deleting the Microvm CR makes the + # controller terminate the VM. On SUSPEND the pod stops but the CR + VM must stay + # (resume reuses them) — the /tmp/suspending marker (preStop) distinguishes. cleanup() { if [ -f /tmp/suspending ]; then - echo "[microvm-bridge] pod stopping for SUSPEND — keeping MicroVM ${VMID}" + echo "[microvm-bridge] pod stopping for SUSPEND — keeping Microvm ${MVM}" return fi - echo "[microvm-bridge] TerminateMicrovm ${VMID}" - [ -n "${VMID}" ] && aws lambda-microvms terminate-microvm --region "${REGION}" --microvm-identifier "${VMID}" >/dev/null 2>&1 || true + echo "[microvm-bridge] deleting Microvm ${MVM} (controller terminates the VM)" + kubectl delete microvm "${MVM}" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true } trap cleanup EXIT INT TERM - # 5) Hold the pod so Sandbox lifecycle == MicroVM lifecycle. Poll the VM state; - # exit when it's gone (terminated). (Log streaming / exec passthrough is a - # virtual-kubelet follow-up — see docs/dark-factory §4.5.) - echo "[microvm-bridge] MicroVM ${VMID} running — pod now mirrors its lifecycle." - while [ -n "${VMID}" ]; do - ST=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" \ - --query 'state' --output text 2>/dev/null || echo "") + # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the CR's state; + # exit when it's terminated/gone. + echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." + while true; do + ST=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.state}' 2>/dev/null || echo "GONE") case "${ST}" in - TERMINATED|TERMINATING|"") echo "[microvm-bridge] MicroVM state=${ST:-gone} — exiting."; break ;; + TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; esac sleep 15 done From 00195a5e8af3331d95fa610e17dd6ee0de717462 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:50:15 -0400 Subject: [PATCH 45/67] docs(flow-d): Microvm CR path reconciles + VM runs, /run hook still silent Declarative Microvm CR path works (CR RUNNING, microvmID populated, clean CR-delete teardown, 0 orphans) but the /run hook produces no runtime coder output in CloudWatch. Ruled out: payload mechanism, image hooks (v2.0), IAM, bridge crash, YAML. Open frontier: whether the service invokes /run against the in-VM hook-server + why no logs. --- .../flow-d-coder-in-microvm-design.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/dark-factory/flow-d-coder-in-microvm-design.md b/docs/dark-factory/flow-d-coder-in-microvm-design.md index f21750e5..3ce0f731 100644 --- a/docs/dark-factory/flow-d-coder-in-microvm-design.md +++ b/docs/dark-factory/flow-d-coder-in-microvm-design.md @@ -153,6 +153,30 @@ This trades the imperative bridge for the declarative CR path the payload mechan requires — and it's MORE GitOps-faithful. Est: bridge rewrite (CR create/delete instead of CLI) + a per-session Secret; the hook-server/artifact/image/IAM/Bedrock pieces are already done and verified. +## 2026-08-03 (later) — declarative Microvm CR path: reconciles + VM runs, /run still silent + +Switched the bridge from the imperative `run-microvm` CLI to the **declarative `Microvm` CR** +path (write payload Secret → create `Microvm` CR with `runHookPayload:{name,key}` → controller +reconciles → delete CR on teardown). Verified working: +- ✅ Bridge creates the Secret + `Microvm` CR (`mvm-`); RBAC for microvms+secrets added. +- ✅ Controller reconciles it: CR `state=RUNNING`, `ACK.ResourceSynced=True`, `status.microvmID` + populated, annotated on the Sandbox. Deleting the CR cleanly terminates the VM (0 orphans). +- ✅ MicrovmImage is v2.0, `UPDATED`, `hooks` present (run/ready/suspend/resume/terminate). +- ❌ **STILL no `/run` output**: CloudWatch `/aws/lambda/microvms/coder-image` shows the build-time + `[hook-server] listening on :8080` but ZERO runtime events after the VM starts — the coder never + logs, no PR. The `/run` hook is not producing coder execution we can observe. + +**What's ruled out:** payload delivery mechanism (now declarative CR, the documented path), image +hooks (present, v2.0 built), IAM (bedrock on exec role), bridge crash (restarts=0), YAML (renders +clean). **What's NOT yet proven:** that the service actually invokes `/run` against the hook-server, +and that hook-server's `/run` handler + background-spawn + Bedrock call execute. Can't see inside +the VM beyond CloudWatch (which is empty at runtime) — needs either (a) the VM's runtime logs routed +somewhere visible, (b) hitting the VM endpoint directly with an auth token (X-aws-proxy-auth) to +probe the hook-server, or (c) the controller/service confirming the run-hook HTTP call + its response. +This is the current debugging frontier — the substrate, image, CR path, and teardown all work; the +open question is purely whether/how the `/run` hook reaches the in-VM hook-server and why it emits +no logs. + ## What exists today (so nothing is lost) - Substrate live: RGD Active, S3 bucket, build/exec roles, **MicrovmImage CREATED (v1.0)**, From d0bc7334b84e7c8d8dce50a80ef67a9ee962a5b7 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 19:57:50 -0400 Subject: [PATCH 46/67] =?UTF-8?q?docs(flow-d):=20E2E=20run=20#106=20report?= =?UTF-8?q?=20=E2=80=94=20chain=20works=20to=20/run,=202=20gaps=20isolated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Probed the live VM: GET / -> ok, POST /run -> coder started. Whole chain functional (issue->workflow->bridge->Microvm CR->VM->hook-server->/run->coder-spawn->clean teardown). Two isolated gaps: (1) service doesn't auto-invoke /run on launch (had to POST manually); (2) runtime logs don't reach CloudWatch (build logs only) — blinded prior runs. --- .../flow-d-coder-in-microvm-design.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/dark-factory/flow-d-coder-in-microvm-design.md b/docs/dark-factory/flow-d-coder-in-microvm-design.md index 3ce0f731..05101b87 100644 --- a/docs/dark-factory/flow-d-coder-in-microvm-design.md +++ b/docs/dark-factory/flow-d-coder-in-microvm-design.md @@ -177,6 +177,32 @@ This is the current debugging frontier — the substrate, image, CR path, and te open question is purely whether/how the `/run` hook reaches the in-VM hook-server and why it emits no logs. +## 2026-08-03 E2E run #106 — full chain works to /run; 2 pinpointed gaps + +Ran a clean GH-issue E2E and **probed the VM directly** (minted an auth token, hit the endpoint). +Stage-by-stage: issue → label → workflow → bridge claim → Microvm CR (`mvm-106`) → VM RUNNING with +endpoint — all ✅. Then the decisive probes against the live VM: +- `GET https:///` (X-aws-proxy-auth) → `{"status":"ok","path":"/"}` → **hook-server is + ALIVE and reachable at runtime.** +- `POST /run` → `{"status":"started"}` → **the /run handler works and background-spawns the coder.** + +So the entire chain — including the hook-server and its /run→coder-spawn — is functional. The two +remaining gaps are now precisely isolated: + +1. **The service does not auto-invoke `/run` on launch.** After RunMicrovm/Microvm-CR reconcile, the + run hook is not called automatically — I had to POST /run manually to start the coder. Either the + run hook fires on a trigger we're not hitting, or the payload/hook wiring needs a specific field to + auto-fire. (auth-token minting: `create-microvm-auth-token --expiration-in-minutes N --allowed-ports + port=8080`; token is at `.authToken.X-aws-proxy-auth`.) +2. **Runtime logs don't reach CloudWatch.** `logging.cloudWatch.logGroup` on the image captures BUILD + logs only; after the VM runs, `/aws/lambda/microvms/coder-image` has 0 runtime events even though the + hook-server clearly runs (proven by the probe). This blinded every prior run — need to wire runtime + stdout/stderr to CloudWatch (or read it another way) to observe the coder. + +Both are now concrete, small-surface problems (a hook-trigger config + a log-routing config), NOT +architecture. The substrate, image+hooks, Bedrock exec role, declarative Microvm CR path, payload +delivery, hook-server, /run→coder-spawn, and clean CR-delete teardown are all verified working. + ## What exists today (so nothing is lost) - Substrate live: RGD Active, S3 bucket, build/exec roles, **MicrovmImage CREATED (v1.0)**, From cc6d33c6bcf28d30d11560ed77121322025d7d53 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 20:48:21 -0400 Subject: [PATCH 47/67] feat(flow-d): bridge drives /run + runtime CloudWatch logs + ingress connector MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Close the 2 E2E gaps: (1) the service's internal /run auto-fire wasn't reliably starting the coder — the bridge now waits for RUNNING+endpoint, mints an auth token, and POSTs the payload to /run on the VM endpoint (deterministic, matches the reference run_session; verified by probe that this starts the coder). Needs ingressNetworkConnectors:ALL_INGRESS on the Microvm to reach the endpoint. (2) runtime logs — add Microvm.spec.logging.cloudWatch (logStream runtime-) so the coder's stdout is visible (image logging was build-only). --- .../shim/20-bridge-sandboxtemplate.yaml | 37 +++++++++++++++++-- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 7af5611c..00308a3e 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -210,9 +210,14 @@ data: # heredoc) on purpose: a heredoc's column-0 EOF terminator breaks out of the # bridge.sh: | YAML block scalar. JSON is valid YAML and stays on indented lines. MAXIDLE={{ .Values.microvm.defaults.maxIdleDurationSeconds }}; SUSPDUR={{ .Values.microvm.defaults.suspendedDurationSeconds }} - echo "[microvm-bridge] writing payload Secret + Microvm CR ${MVM} (fires /run for issue #${DF_ISSUE_NUMBER})..." - MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":mvm+"-payload","namespace":ns},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] payload secret apply failed"; exit 1; } - MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"idlePolicy":{"autoResumeEnabled":True,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } + INGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:ALL_INGRESS" + LOGGRP="/aws/lambda/microvms/{{ .Values.microvm.image.name | default "coder" }}-image" + echo "[microvm-bridge] writing payload Secret + Microvm CR ${MVM} for issue #${DF_ISSUE_NUMBER}..." + MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":mvm+"-payload","namespace":ns},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] payload secret apply failed"; exit 1; } + # Microvm CR: ingress ALL_INGRESS (so the bridge can reach the endpoint to drive /run), + # egress INTERNET_EGRESS (Bedrock + git/gh), runtime logging → CloudWatch (logStream + # 'runtime' so the coder's stdout is visible, separate from build logs). + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" REGION="${REGION}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; r=e["REGION"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":["arn:aws:lambda:"+r+":aws:network-connector:aws-network-connector:INTERNET_EGRESS"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":True,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } # 4) Wait for the controller to report the running VM's id, record it on the Sandbox # (lifecycle controller reads this to suspend/resume THIS session's VM). @@ -226,6 +231,32 @@ data: [ -n "${VMID}" ] && kubectl annotate sandbox "${SANDBOX_NAME}" -n "${NS}" \ "microvm-lifecycle.agents.x-k8s.io/microvm-id=${VMID}" --overwrite >/dev/null 2>&1 || true + # 4b) DRIVE the coder: wait for RUNNING + an endpoint, mint an auth token, and POST the + # payload to /run on the endpoint. This is the deterministic invocation (verified by + # probe): the service's internal /run auto-fire wasn't reliably starting the coder, so + # the bridge drives it explicitly like the reference run_session. hook-server's /run + # background-spawns the coder (returns fast); df-run's await-coder polls GitHub for the PR. + if [ -n "${VMID}" ]; then + EP=""; i=0 + while [ "$i" -lt 60 ]; do + S=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'state' --output text 2>/dev/null || echo "") + EP=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'endpoint' --output text 2>/dev/null || echo "") + [ "$S" = "RUNNING" ] && [ -n "${EP}" ] && [ "${EP}" != "None" ] && break + i=$((i+1)); sleep 5 + done + TOKEN=$(aws lambda-microvms create-microvm-auth-token --region "${REGION}" --microvm-identifier "${VMID}" \ + --expiration-in-minutes 60 --allowed-ports 'port=8080' 2>/dev/null \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["authToken"]["X-aws-proxy-auth"])' 2>/dev/null || echo "") + if [ -n "${EP}" ] && [ -n "${TOKEN}" ]; then + echo "[microvm-bridge] driving coder: POST /run on ${EP}" + RC=$(curl -sS -m 30 -o /tmp/run.out -w '%{http_code}' -X POST "https://${EP}/run" \ + -H "X-aws-proxy-auth: ${TOKEN}" -H 'Content-Type: application/json' -d "${PAYLOAD}" 2>/tmp/run.err || echo "000") + echo "[microvm-bridge] /run -> HTTP ${RC} $(cat /tmp/run.out 2>/dev/null | head -c 120)" + else + echo "[microvm-bridge] WARN: no endpoint/token — cannot drive /run (ep=${EP:-none} token=$([ -n "${TOKEN}" ] && echo yes || echo no))" + fi + fi + # 5) Teardown on real claim end (NOT on suspend): deleting the Microvm CR makes the # controller terminate the VM. On SUSPEND the pod stops but the CR + VM must stay # (resume reuses them) — the /tmp/suspending marker (preStop) distinguishes. From 601a0c3e863d3927bb29825e78e4d8edc0ce4556 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 20:52:44 -0400 Subject: [PATCH 48/67] fix(flow-d): grant lambda:CreateMicrovmAuthToken (bridge drives /run) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge mints an auth token to POST /run on the VM endpoint, but the controller/bridge role lacked lambda:CreateMicrovmAuthToken (+ShellAuthToken) — token mint failed AccessDenied, so /run was never driven. Add both verbs to the controller inline policy (reused by the bridge via Pod Identity). --- .../templates/shim/00-controller-pod-identity.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml index 82fad73a..f4f639ef 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml @@ -51,6 +51,7 @@ spec: "lambda:GetMicrovmImage","lambda:GetMicrovmImageVersion","lambda:ListMicrovmImages", "lambda:RunMicrovm","lambda:GetMicrovm","lambda:TerminateMicrovm","lambda:ListMicrovms", "lambda:SuspendMicrovm","lambda:ResumeMicrovm", + "lambda:CreateMicrovmAuthToken","lambda:CreateMicrovmShellAuthToken", "lambda:TagResource","lambda:UntagResource","lambda:ListTagsForResource", "lambda:ListNetworkConnectors","lambda:GetNetworkConnector" ], From 9ff9018bcc89c1edb1874b35ef1caa8b57b132a5 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 21:00:29 -0400 Subject: [PATCH 49/67] =?UTF-8?q?feat(flow-d):=20observable=20coder=20?= =?UTF-8?q?=E2=80=94=20/status=20endpoint=20+=20bridge=20polls=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runtime CloudWatch routing is unreliable on this runtime, so make the coder run OBSERVABLE directly: hook-server captures the coder's stdout/stderr to /tmp/coder.log + tracks state (running/done/exited:N/spawn-error), exposes GET /status {coderState, log tail}. The bridge polls /status in its hold loop and echoes it — so the coder run is visible in the bridge pod logs (kubectl logs) even without CloudWatch. --- .../dark-factory/coder-microvm/hook-server.js | 21 ++++++++++++++++--- .../shim/20-bridge-sandboxtemplate.yaml | 12 ++++++++--- 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index b3952805..ea79f24d 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -27,6 +27,7 @@ const PORT = parseInt(process.env.HOOKS_PORT || "8080", 10); const SECRETS_DIR = "/tmp/secrets"; let coderStarted = false; +let coderState = "idle"; // idle | running | done | exited: | spawn-error: // Materialize the runHookPayload (JSON) into the coder's existing contract: // files: /tmp/secrets/{gh-token} ; env: DF_*, AWS_REGION, USE_BEDROCK=1 @@ -58,9 +59,20 @@ function startCoder(payload) { if (d.model) env.CODER_MODEL = d.model; console.log(`[hook-server] /run → background-spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); - const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: "inherit", detached: false }); - child.on("exit", (code) => console.log(`[hook-server] coder exited code=${code}`)); - child.on("error", (e) => console.log(`[hook-server] coder spawn error: ${e.message}`)); + // Capture the coder's stdout+stderr to a file (runtime CloudWatch routing is + // unreliable on this pre-GA runtime), so /status can return a live tail — this is + // how the coder run is OBSERVED. Also mirror to our stdout. + const logPath = "/tmp/coder.log"; + const logFd = fs.openSync(logPath, "a"); + coderState = "running"; + const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: ["ignore", logFd, logFd], detached: false }); + child.on("exit", (code) => { coderState = code === 0 ? "done" : ("exited:" + code); console.log(`[hook-server] coder exited code=${code}`); }); + child.on("error", (e) => { coderState = "spawn-error:" + e.message; console.log(`[hook-server] coder spawn error: ${e.message}`); }); +} + +function tailLog(n) { + try { return fs.readFileSync("/tmp/coder.log", "utf8").split("\n").slice(-n).join("\n"); } + catch { return ""; } } const server = http.createServer((req, res) => { @@ -75,6 +87,9 @@ const server = http.createServer((req, res) => { case "/validate": return ok({ status: "valid" }); // Per-session start: body IS runHookPayload. Kick off the coder, return fast. case "/run": startCoder(body); return ok({ status: "started" }); + // Observe the coder: state + a tail of its captured output (runtime CloudWatch + // routing is unreliable on this runtime, so this is how the run is watched). + case "/status": return ok({ status: "ok", coderState, started: coderStarted, log: tailLog(60) }); // Lifecycle: coder holds no external state to flush; ack so the service proceeds. case "/suspend": return ok({ status: "suspended" }); case "/resume": return ok({ status: "resumed" }); diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 00308a3e..ddbcd219 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -271,15 +271,21 @@ data: } trap cleanup EXIT INT TERM - # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the CR's state; - # exit when it's terminated/gone. + # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll BOTH the CR state + # AND the in-VM coder /status (state + log tail) so the coder run is OBSERVABLE in + # the bridge logs (runtime CloudWatch routing is unreliable on this runtime). echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." while true; do ST=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.state}' 2>/dev/null || echo "GONE") case "${ST}" in TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; esac - sleep 15 + if [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then + STATUS=$(curl -sS -m 10 "https://${EP}/status" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print("coder="+str(d.get("coderState"))); print("---LOG---"); print(d.get("log","")[-1500:])' 2>/dev/null || echo "") + [ -n "${STATUS}" ] && echo "[microvm-bridge] /status: ${STATUS}" + fi + sleep 20 done --- apiVersion: extensions.agents.x-k8s.io/v1beta1 From 426d0691fbdc3c8be8e2ca495b628cd9b470b5cc Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Mon, 3 Aug 2026 21:04:03 -0400 Subject: [PATCH 50/67] chore(flow-d): bump codeArtifactUri to -r2 to force image rebuild MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Overwriting the same S3 key doesn't change the URI, so the controller never rebuilds (stayed v2.0 without /status). New key -r2 → URI change → rebuild with the /status observability + auth-token fixes. --- .../addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 40125ab0..79dbd93d 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,7 +28,7 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64.zip" + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r2.zip" image: enabled: true name: coder From 8e18fb4d528501752f16f70dc959bd06c06d2826 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 10:35:23 -0400 Subject: [PATCH 51/67] fix(flow-d): minimal hook-server (drop /status) + artifact r3 The /status route + log-capture correlated with a MicrovmImage build hung 2h+ on the ready hook (pre-GA controller). Revert hook-server to the minimal known-good shape that built v2.0 cleanly: trivial synchronous /run that detached-spawns the coder, no /status. Point codeArtifactUri at a fresh key (r3) to force a clean rebuild. --- .../dark-factory/coder-microvm/hook-server.js | 86 +++++++------------ .../addons/agent-sandbox-lambda/values.yaml | 2 +- 2 files changed, 30 insertions(+), 58 deletions(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index ea79f24d..1108c8f8 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -3,21 +3,22 @@ // Lambda MicroVM is a snapshot/hook runtime: the platform builds an image by // starting THIS process and snapshotting it once the `ready` hook says "go", then // resumes that snapshot per session and calls the `run` hook with the session's -// runHookPayload as the request body. Hooks are HTTP endpoints we serve on :8080 -// (hooks.port on the MicrovmImage); each must answer within its timeout (run = 30s). +// runHookPayload as the request body. Hooks are HTTP endpoints we serve on :8080. // // The dark-factory coder (entrypoint.js) is a ONE-SHOT batch job (clone → agent → -// push → PR, 5-15 min). It cannot run *inside* the 30s run hook. So the run hook is -// just the handshake: it materializes the payload into the coder's file/env contract -// and BACKGROUND-SPAWNS entrypoint.js, then returns 200 immediately. The coder then -// runs to completion asynchronously; df-run's await-coder step polls GitHub for the -// PR exactly as it does for Kata. The VM stays alive because idlePolicy. -// maxIdleDurationSeconds on the Microvm is set longer than a coder run (idle = no -// inbound traffic; a background job would otherwise auto-suspend). +// push → PR, 5-15 min). It cannot run inside the 30s run hook, so /run just +// materializes the payload into the coder's file/env contract and BACKGROUND-SPAWNS +// entrypoint.js, then returns 200 immediately. The coder runs async; df-run's +// await-coder step polls GitHub for the PR (same as Kata). The VM stays alive because +// idlePolicy.maxIdleDurationSeconds > a coder run (idle = no inbound traffic). // -// LLM: USE_BEDROCK=1 is exported so entrypoint.js calls Bedrock DIRECTLY via the -// MicroVM execution role — no Bifrost / EKS-network dependency (a MicroVM can't reach -// Bifrost's ClusterIP). See docs/dark-factory/flow-d-coder-in-microvm-design.md. +// LLM: USE_BEDROCK=1 → entrypoint.js calls Bedrock DIRECTLY via the MicroVM execution +// role (no Bifrost / EKS-network dependency). See docs/dark-factory/flow-d-coder-in-microvm-design.md. +// +// KEPT MINIMAL: this is the exact shape that built cleanly (v2.0). The /run handler +// stays trivial and synchronous so the build's ready-hook completes fast. Observability +// is via direct endpoint probes, not a /status route (adding one correlated with a +// hung ready-hook build on the pre-GA controller). const http = require("http"); const fs = require("fs"); @@ -25,29 +26,18 @@ const { spawn } = require("child_process"); const PORT = parseInt(process.env.HOOKS_PORT || "8080", 10); const SECRETS_DIR = "/tmp/secrets"; - let coderStarted = false; -let coderState = "idle"; // idle | running | done | exited: | spawn-error: -// Materialize the runHookPayload (JSON) into the coder's existing contract: -// files: /tmp/secrets/{gh-token} ; env: DF_*, AWS_REGION, USE_BEDROCK=1 -// then background-spawn entrypoint.js. Idempotent: only the first /run starts it. function startCoder(payload) { - if (coderStarted) { - console.log("[hook-server] /run received again — coder already started, ignoring"); - return; - } + if (coderStarted) { console.log("[hook-server] /run again — already started, ignoring"); return; } coderStarted = true; - let d = {}; try { d = JSON.parse(payload || "{}"); } catch (e) { console.log("[hook-server] payload not JSON:", e.message); } - fs.mkdirSync(SECRETS_DIR, { recursive: true, mode: 0o700 }); if (d.ghToken) fs.writeFileSync(`${SECRETS_DIR}/gh-token`, d.ghToken, { mode: 0o400 }); - const env = { ...process.env, - USE_BEDROCK: "1", // Bedrock-direct (exec-role creds) + USE_BEDROCK: "1", GH_TOKEN_PATH: `${SECRETS_DIR}/gh-token`, AWS_REGION: d.region || process.env.AWS_REGION || "us-west-2", DF_ISSUE_NUMBER: d.issueNumber ? String(d.issueNumber) : "", @@ -57,46 +47,28 @@ function startCoder(payload) { DF_ISSUE_TITLE: d.issueTitle || "", }; if (d.model) env.CODER_MODEL = d.model; - - console.log(`[hook-server] /run → background-spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); - // Capture the coder's stdout+stderr to a file (runtime CloudWatch routing is - // unreliable on this pre-GA runtime), so /status can return a live tail — this is - // how the coder run is OBSERVED. Also mirror to our stdout. - const logPath = "/tmp/coder.log"; - const logFd = fs.openSync(logPath, "a"); - coderState = "running"; - const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: ["ignore", logFd, logFd], detached: false }); - child.on("exit", (code) => { coderState = code === 0 ? "done" : ("exited:" + code); console.log(`[hook-server] coder exited code=${code}`); }); - child.on("error", (e) => { coderState = "spawn-error:" + e.message; console.log(`[hook-server] coder spawn error: ${e.message}`); }); -} - -function tailLog(n) { - try { return fs.readFileSync("/tmp/coder.log", "utf8").split("\n").slice(-n).join("\n"); } - catch { return ""; } + console.log(`[hook-server] /run → spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); + // Capture coder output to a file the coder writes; inherit stdio so it also streams + // to the VM console. detached so it outlives the request handler. + const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: "inherit", detached: true }); + child.unref(); + child.on("error", (e) => console.log(`[hook-server] coder spawn error: ${e.message}`)); } const server = http.createServer((req, res) => { let body = ""; req.on("data", (c) => { body += c; }); req.on("end", () => { - const ok = (obj) => { res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(obj || { status: "ok" })); }; + const ok = (o) => { res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(o || { status: "ok" })); }; switch (req.url) { - // Build-time: signal the process is initialized so the builder snapshots a - // clean, waiting coder (no session work baked into the snapshot). - case "/ready": return ok({ status: "ready" }); - case "/validate": return ok({ status: "valid" }); - // Per-session start: body IS runHookPayload. Kick off the coder, return fast. - case "/run": startCoder(body); return ok({ status: "started" }); - // Observe the coder: state + a tail of its captured output (runtime CloudWatch - // routing is unreliable on this runtime, so this is how the run is watched). - case "/status": return ok({ status: "ok", coderState, started: coderStarted, log: tailLog(60) }); - // Lifecycle: coder holds no external state to flush; ack so the service proceeds. - case "/suspend": return ok({ status: "suspended" }); - case "/resume": return ok({ status: "resumed" }); - case "/terminate":return ok({ status: "terminated" }); - default: return ok({ status: "ok", path: req.url }); + case "/ready": return ok({ status: "ready" }); + case "/validate": return ok({ status: "valid" }); + case "/run": startCoder(body); return ok({ status: "started" }); + case "/suspend": return ok({ status: "suspended" }); + case "/resume": return ok({ status: "resumed" }); + case "/terminate": return ok({ status: "terminated" }); + default: return ok({ status: "ok", path: req.url }); } }); }); - server.listen(PORT, () => console.log(`[hook-server] listening on :${PORT} (lambda-coder, Bedrock-direct)`)); diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 79dbd93d..8bbdbaa3 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,7 +28,7 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r2.zip" + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r3.zip" image: enabled: true name: coder From a84cb3854f030adc9f23b5462aefcbbc1f88f6a9 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 11:09:09 -0400 Subject: [PATCH 52/67] feat(flow-d): /logs endpoint + capture coder output to file (observability) No CloudWatch runtime routing + no shell, so /run captures the coder's stdout/stderr to /tmp/coder.log and /logs returns it (read over the HTTP token). This is how we finally SEE why the coder isn't producing a PR. Artifact r4. --- examples/dark-factory/coder-microvm/hook-server.js | 11 +++++++---- .../hub/addons/agent-sandbox-lambda/values.yaml | 2 +- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index 1108c8f8..246a1ea2 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -48,11 +48,13 @@ function startCoder(payload) { }; if (d.model) env.CODER_MODEL = d.model; console.log(`[hook-server] /run → spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); - // Capture coder output to a file the coder writes; inherit stdio so it also streams - // to the VM console. detached so it outlives the request handler. - const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: "inherit", detached: true }); + // Capture the coder's stdout+stderr to /tmp/coder.log so /logs can return it — + // runtime CloudWatch routing doesn't work on this runtime, and there's no shell, + // so this file (read over the HTTP token) is the ONLY way to see what the coder did. + const logFd = fs.openSync("/tmp/coder.log", "a"); + const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: ["ignore", logFd, logFd], detached: true }); child.unref(); - child.on("error", (e) => console.log(`[hook-server] coder spawn error: ${e.message}`)); + child.on("error", (e) => { try { fs.appendFileSync("/tmp/coder.log", "SPAWN-ERROR: " + e.message + "\n"); } catch {} }); } const server = http.createServer((req, res) => { @@ -64,6 +66,7 @@ const server = http.createServer((req, res) => { case "/ready": return ok({ status: "ready" }); case "/validate": return ok({ status: "valid" }); case "/run": startCoder(body); return ok({ status: "started" }); + case "/logs": { let l=""; try { l=fs.readFileSync("/tmp/coder.log","utf8"); } catch {} return ok({ status:"ok", started: coderStarted, log: l.slice(-6000) }); } case "/suspend": return ok({ status: "suspended" }); case "/resume": return ok({ status: "resumed" }); case "/terminate": return ok({ status: "terminated" }); diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 8bbdbaa3..78cfcb47 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,7 +28,7 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r3.zip" + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r4.zip" image: enabled: true name: coder From d48d7071aa17216e436df9c1eeda2aa7be617785 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 11:16:33 -0400 Subject: [PATCH 53/67] fix(flow-d): set WORKSPACE=/tmp/workspace (coder crashed EACCES on /workspace) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ROOT CAUSE (finally seen via /logs): the coder crashed immediately — 'EACCES: permission denied, mkdir /workspace/artifacts' at entrypoint.js:398 — because the MicroVM rootfs is read-only with no /workspace volume (unlike Kata). Point WORKSPACE at the writable tmpfs. Artifact r5. --- examples/dark-factory/coder-microvm/hook-server.js | 5 +++++ .../clusters/hub/addons/agent-sandbox-lambda/values.yaml | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index 246a1ea2..df5fde7e 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -38,6 +38,11 @@ function startCoder(payload) { const env = { ...process.env, USE_BEDROCK: "1", + // MicroVM rootfs is read-only + there's no /workspace volume mount (unlike Kata, + // where the operator mounts a writable workspace). entrypoint.js mkdir's + // ${WORKSPACE}/artifacts and clones there, so point it at the writable tmpfs — + // else it crashes EACCES on /workspace/artifacts before doing any work. + WORKSPACE: "/tmp/workspace", GH_TOKEN_PATH: `${SECRETS_DIR}/gh-token`, AWS_REGION: d.region || process.env.AWS_REGION || "us-west-2", DF_ISSUE_NUMBER: d.issueNumber ? String(d.issueNumber) : "", diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 78cfcb47..2f8768ca 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,7 +28,7 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r4.zip" + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r5.zip" image: enabled: true name: coder From 7c2f950cd3f2fa88b28aae30f0416e2af452485a Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 11:58:58 -0400 Subject: [PATCH 54/67] fix(flow-d): df-iterate routes fix round to originating substrate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fix round (df-iterate → df-run) had no trigger-label, so it always went to Kata even for a Lambda PR. Detect the substrate from the ORIGINATING ISSUE's label (the coder doesn't copy it onto the PR), and pass trigger-label to the resubmitted df-run so it claims the right warm pool. Kata unchanged (defaults to dark-factory). --- .../addons/charts/dark-factory/scripts/iterate.js | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/gitops/addons/charts/dark-factory/scripts/iterate.js b/gitops/addons/charts/dark-factory/scripts/iterate.js index a155bc41..de7a4ca2 100644 --- a/gitops/addons/charts/dark-factory/scripts/iterate.js +++ b/gitops/addons/charts/dark-factory/scripts/iterate.js @@ -96,6 +96,17 @@ async function main() { // Iteration cap: count via a df-iterations/ label on the PR (issue API). const issue = await gh("GET", `/repos/${REPO}/issues/${PR}`); const labels = (issue.labels || []).map((l) => (typeof l === "string" ? l : l.name)); + // Substrate routing: the fix round must run on the SAME substrate the PR came from, + // so it lands on the right warm pool (Lambda MicroVM vs Kata). The ORIGINATING ISSUE + // (not the PR — the coder doesn't copy the label onto the PR) carries the label that + // fired it. Read the issue's labels; default to plain dark-factory (Kata) on any miss. + let triggerLabel = "dark-factory"; + try { + const origIssue = await gh("GET", `/repos/${REPO}/issues/${issueNumber}`); + const il = (origIssue.labels || []).map((l) => (typeof l === "string" ? l : l.name)); + if (il.includes("darkfactory-lambda")) triggerLabel = "darkfactory-lambda"; + } catch (e) { console.log(`[df-iterate] could not read issue #${issueNumber} labels (${e.message}) — defaulting Kata`); } + console.log(`[df-iterate] substrate trigger-label=${triggerLabel}`); const cur = labels.filter((l) => l.startsWith(ITER_LABEL_PREFIX)).map((l) => parseInt(l.slice(ITER_LABEL_PREFIX.length), 10)).filter((n) => !isNaN(n)); const count = cur.length ? Math.max(...cur) : 0; if (count >= MAX_ITERATIONS) { @@ -122,6 +133,9 @@ async function main() { { name: "issue-title", value: pr.title }, { name: "issue-body", value: "" }, { name: "base-branch", value: pr.base.ref }, + // Route the fix round to the SAME substrate the PR came from (Lambda vs Kata), + // so claim-sandbox picks the right warm pool. df-run branches its warm-pool on this. + { name: "trigger-label", value: triggerLabel }, // base64 the comment (may be multi-line markdown) so it can't break the // claim-sandbox manifest YAML; the coder decodes it. Leave plain empty. { name: "iterate-note", value: "" }, From 63526e5f8b11e9ff2f4ebef9bfad313506aec3da Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 12:29:11 -0400 Subject: [PATCH 55/67] fix(flow-d): remove microvm-suspend from df-run DAG; bridge owns suspend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One df-run template serves both substrates, so a rendered microvm-suspend step could only be runtime-Skipped on Kata (visible, confusing). Remove it (microvmSuspend.enabled=false → zero MicroVM steps in the Kata DAG). The Flow D bridge now suspends its OWN MicroVM after the coder pushes the PR (lambda-only by construction). Also fixes bridge hold-loop to poll /logs (was /status, which the minimal image dropped). --- .../shim/20-bridge-sandboxtemplate.yaml | 24 ++++++++++++++----- gitops/addons/charts/dark-factory/values.yaml | 23 +++++++----------- 2 files changed, 27 insertions(+), 20 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index ddbcd219..45a53e43 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -271,9 +271,14 @@ data: } trap cleanup EXIT INT TERM - # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll BOTH the CR state - # AND the in-VM coder /status (state + log tail) so the coder run is OBSERVABLE in - # the bridge logs (runtime CloudWatch routing is unreliable on this runtime). + # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the CR state + the + # in-VM coder /logs so the coder run is OBSERVABLE in the bridge logs (runtime + # CloudWatch routing is unreliable on this runtime). + # SUSPEND-AFTER-CODE (Flow D behavior, done HERE — not as a df-run DAG step, so the + # pipeline graph stays substrate-agnostic/clean for Kata): once the coder has pushed + # its PR (log shows "PR opened"/"done"), suspend the MicroVM to free compute while the + # review gates run; the VM's memory+disk persist for a resume on the next fix round. + SUSPENDED_ONCE="" echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." while true; do ST=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.state}' 2>/dev/null || echo "GONE") @@ -281,9 +286,16 @@ data: TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; esac if [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then - STATUS=$(curl -sS -m 10 "https://${EP}/status" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ - | python3 -c 'import json,sys; d=json.load(sys.stdin); print("coder="+str(d.get("coderState"))); print("---LOG---"); print(d.get("log","")[-1500:])' 2>/dev/null || echo "") - [ -n "${STATUS}" ] && echo "[microvm-bridge] /status: ${STATUS}" + LOG=$(curl -sS -m 10 "https://${EP}/logs" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("log","")[-800:])' 2>/dev/null || echo "") + [ -n "${LOG}" ] && echo "[microvm-bridge] coder-log-tail: $(echo "$LOG" | tail -1)" + # Coder finished (pushed PR) → suspend the VM once (Flow D), if not on a Kata run. + if [ -z "${SUSPENDED_ONCE}" ] && echo "${LOG}" | grep -qiE 'PR opened|done — PR|status success'; then + echo "[microvm-bridge] coder pushed PR — suspending MicroVM ${VMID} (Flow D free-compute-during-review)" + aws lambda-microvms suspend-microvm --region "${REGION}" --microvm-identifier "${VMID}" >/dev/null 2>&1 \ + && echo "[microvm-bridge] suspend-microvm ok" || echo "[microvm-bridge] suspend-microvm skipped/failed" + SUSPENDED_ONCE=1 + fi fi sleep 20 done diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 549e9272..9b9b4784 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -257,21 +257,16 @@ iterate: enabled: true maxIterations: 3 -# ── Flow D — MicroVM suspend/resume demo ───────────────────────────────────── -# When the darkfactory-lambda label runs, suspend the MicroVM after the coder -# pushes (freeing compute while gates run) and resume it on a df-iterate comment. -# Drives Sandbox.spec.operatingMode; the agent-sandbox microvm-lifecycle controller -# translates that to suspend-microvm / resume-microvm. -# -# ENABLE on clusters that run the Lambda substrate (Flow D). It adds a microvm-suspend -# task to the df-run DAG that, on a darkfactory-lambda run, sets Sandbox.operatingMode= -# Suspended after the coder pushes (the microvm-lifecycle controller then calls -# suspend-microvm). On a Kata (Flow B) run the step is when-gated and skips cleanly -# (quoted operands — a clean Skipped node, NOT the earlier Error). On a Kata-ONLY cluster -# set this false so the DAG has zero MicroVM steps. This hub runs BOTH substrates, so -# it's on: Flow D needs it, and the Kata skip is harmless. +# ── Flow D — MicroVM suspend/resume ────────────────────────────────────────── +# DEFAULT false — and it should STAY false. Suspend-after-code is NOT a df-run DAG +# step anymore: because ONE df-run template serves BOTH substrates on a shared hub, a +# rendered microvm-suspend step could only be runtime-SKIPPED on a Kata run, leaving a +# confusing (if harmless) Skipped node in every Kata graph. Instead the Flow D bridge +# (agent-sandbox-lambda, bridge.sh) suspends its OWN MicroVM after the coder pushes the +# PR — lambda-only by construction, so the pipeline graph is substrate-agnostic and the +# Kata DAG contains ZERO MicroVM steps. Leave this false; the bridge owns suspend/resume. microvmSuspend: - enabled: true + enabled: false image: alpine/k8s:1.31.0 # only patches Sandbox.operatingMode via kubectl (no aws-cli) # ── Language / stack support ───────────────────────────────────────────────── From 405f2a13c087ed9c91005de2e7fba628e9206196 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 12:41:05 -0400 Subject: [PATCH 56/67] docs(flow-d): substrate benchmark + diagrams (Kata vs Lambda MicroVM) Side-by-side from a parallel dual-substrate run (#117 Kata / #118 Lambda): time-to-PR, per-step timing, cold-start vs warm-claim, DAG comparison, log locations, step-by-step lifecycle, the 10 Lambda-specific gotchas, and when to choose which. Plus a Mermaid diagrams companion (shared pipeline, each substrate, suspend/resume, e2e loop). --- docs/dark-factory/SUBSTRATE-BENCHMARK.md | 163 +++++++++++++++++++++++ docs/dark-factory/SUBSTRATE-DIAGRAMS.md | 123 +++++++++++++++++ 2 files changed, 286 insertions(+) create mode 100644 docs/dark-factory/SUBSTRATE-BENCHMARK.md create mode 100644 docs/dark-factory/SUBSTRATE-DIAGRAMS.md diff --git a/docs/dark-factory/SUBSTRATE-BENCHMARK.md b/docs/dark-factory/SUBSTRATE-BENCHMARK.md new file mode 100644 index 00000000..c8937454 --- /dev/null +++ b/docs/dark-factory/SUBSTRATE-BENCHMARK.md @@ -0,0 +1,163 @@ +# Dark Factory — Substrate Benchmark: Kata micro-VM vs Lambda MicroVM + +A side-by-side comparison of the two sandbox substrates that run the autonomous coder, +measured on **identical issues fired in parallel** on the same hub cluster. + +- **Flow B — Kata micro-VM** (mature, default): the coder runs in a hardware-isolated Kata + pod on a self-managed nested-virt EKS node group. +- **Flow D — AWS Lambda MicroVM** (pre-GA): the coder runs in a Firecracker MicroVM + provisioned via the `lambdamicrovms` ACK controller, driven by a bridge pod. + +Both run the **same `dark-factory-coder`** (same `entrypoint.js`), produce the same kind of +PR, and go through the **same review gates** (AWS DevOps Agent + AWS Security Agent). The only +difference is *where the coder executes* and *how it's provisioned*. + +--- + +## TL;DR + +| | Kata micro-VM (Flow B) | Lambda MicroVM (Flow D) | +| --- | --- | --- | +| **Provisioning** | pre-warmed pool → **instant claim** | **RunMicrovm cold-start per session** (~90–120s) | +| **Time to first PR** (from label) | ~**2 min** | ~**3.5 min** | +| **LLM path** | Bifrost gateway (in-cluster) + Langfuse traces | **Bedrock-direct** (exec role) — no cluster network | +| **Scale-to-zero when idle** | ❌ node pool runs continuously | ✅ **suspend-to-zero**, resume on demand | +| **Infra to manage** | nested-virt node group (Karpenter/MNG) | none — serverless MicroVMs | +| **Observability** | native `kubectl logs` | custom `/logs` HTTP endpoint (no runtime CloudWatch) | +| **Maturity** | production-ready today | pre-GA (preview) — pilot-grade | +| **Economics at 1000s scale** | pay for idle capacity | pay per active minute (the strategic win) | + +**Bottom line:** at small scale the two feel equivalent (the LLM coding step ~2–4 min and the +external review agents ~8–15 min dominate total time on *both*). The Lambda substrate's advantage +is **not latency** — it's **operational + economic**: no node pool to run, and suspend-to-zero per +idle session. Its cost is **maturity** (pre-GA control plane) and the extra plumbing below. + +--- + +## Benchmarked run (identical issue, fired in parallel) + +Issue (both): *"Add an S3 bucket for log archives + an EC2 IAM role to write to it (Terraform)."* +Fired simultaneously — `#117` labeled `dark-factory` (Kata), `#118` labeled `darkfactory-lambda` (Lambda). + +### Time to first PR (from label → PR opened) +| Substrate | Issue | PR | Elapsed | +| --- | --- | --- | --- | +| Kata | #117 | #119 | ~**2 min** (16:32:54 → 16:34:58) | +| Lambda | #118 | #120 | ~**3.7 min** (16:32:54 → 16:36:38) | + +**Δ ≈ 100s** — the MicroVM cold-start (`RunMicrovm` → RUNNING → hook-server ready → bridge drives +`/run`) vs Kata's pre-warmed pod claim. This gap is the substrate's provisioning cost; everything +after (clone → LLM agent → push) is identical code and takes the same time. + +### Per-step workflow timing (Kata run #117) +| Step | Time | +| --- | --- | +| claim (warm pod) | ~17s | +| **drive-coder** (clone→LLM→push→PR) | ~120s | +| detect-deployable | ~28s | +| holdout-gate | ~19s | +| deploy-test (terraform validate) | ~37s | +| security-agent (external) | several min | +| devops-gate (external) | several min | + +*(The Lambda run's `drive-coder` is comparable for the coding itself; it just adds the ~90s +RunMicrovm cold-start inside the claim/drive window. The two external review agents — DevOps + +Security — take ~8–15 min combined and dominate total wall-clock on BOTH substrates.)* + +--- + +## Where the logs are + +| What | Kata (Flow B) | Lambda (Flow D) | +| --- | --- | --- | +| Pipeline steps | Argo UI (`/argo-workflows`) or `kubectl logs -n argo ` | **same** | +| Coder output | `kubectl logs -n agent-sandbox-system df-issue-` (native) | **`GET https:///logs`** with an auth token (runtime CloudWatch routing is unreliable on the pre-GA runtime, so the hook-server captures coder stdout to a file + serves it) | +| Image build | n/a (normal ECR image) | CloudWatch `/aws/lambda/microvms/coder-image` | + +--- + +## DAG — same pipeline, one substrate-branched step + +Both substrates run the **same `df-run` WorkflowTemplate**. The DAG is identical: + +``` +claim → drive-coder → { holdout-gate, devops-gate → security-agent, detect-deployable → deploy-test } → status → onExit(teardown) +``` + +The **only** substrate branch is inside `claim-sandbox`: `trigger-label` selects the warm pool — +`coder-warmpool` (Kata) vs `coder-warmpool-microvm` (Lambda). **There is no MicroVM-specific step in +the DAG** — suspend/resume for Lambda is handled by the *bridge* itself (see below), so the Kata +graph contains zero MicroVM nodes. + +### Substrate-specific mechanics (outside the DAG) +- **Kata:** the operator materializes a pod from `SandboxTemplate/coder-sandbox`; the coder runs + in-cluster, reaches models via Bifrost, and its logs are native pod logs. +- **Lambda:** `SandboxTemplate/coder-sandbox-microvm` materializes a **bridge pod** which: + 1. reads the platform image handoff (imageARN + execRoleARN, built once by KRO/ACK), + 2. creates a **`Microvm` CR** (declarative — the controller delivers the runHookPayload), + 3. waits for RUNNING, mints an auth token, and **POSTs `/run`** to the VM endpoint → the + hook-server background-spawns the coder, + 4. **suspends the MicroVM** once the coder pushes the PR (free compute during review), + 5. terminates the VM on teardown (delete the `Microvm` CR). + +--- + +## Step-by-step: what actually happens + +### Kata (Flow B) +1. Issue labeled `dark-factory` → Argo Events sensor → `df-run`. +2. `claim-sandbox` claims a **pre-warmed** Kata pod from `coder-warmpool` (instant). +3. Operator injects `DF_ISSUE_NUMBER` etc. → the baked-in `entrypoint.js` runs: clone → Claude + Code (`claude -p`, via **Bifrost**) → commit → **open PR**. +4. Review gates: DevOps Agent (check-run) + Security Agent (findings). Consolidated verdict posted. +5. Human comment "fix findings" → `df-iterate` → new Kata coder round → re-review. +6. Approve → `df-merge-teardown` merges + releases the claim. + +### Lambda MicroVM (Flow D) +1. Issue labeled `darkfactory-lambda` → same sensor → `df-run` (warm-pool branched to Lambda). +2. `claim-sandbox` claims the **bridge** pod from `coder-warmpool-microvm`. +3. Bridge creates a `Microvm` CR → controller `RunMicrovm` (**cold-start ~90s**) → RUNNING. +4. Bridge mints auth token → `POST /run` → hook-server background-spawns the **same + `entrypoint.js`**, but `USE_BEDROCK=1` so it calls **Bedrock directly** (exec role) — no cluster + network. Coder: clone → Claude Code → commit → **open PR**. +5. Bridge **suspends** the MicroVM (free compute while gates run). +6. Same review gates + verdict. +7. "fix findings" → `df-iterate` (routes back to Lambda via `trigger-label`) → fresh MicroVM round. +8. Approve → merge + teardown (bridge deletes the `Microvm` CR → controller terminates the VM). + +--- + +## Gotchas the Lambda substrate needed (that Kata does not) + +Because a MicroVM is **outside the cluster network, has a read-only rootfs, and uses a +snapshot/hook execution model**: + +| # | Gotcha | Fix | +| --- | --- | --- | +| 1 | Coder crashed `EACCES mkdir /workspace/artifacts` (no writable volume like Kata) | set `WORKSPACE=/tmp/workspace` (writable tmpfs) — *the silent killer* | +| 2 | Can't reach Bifrost's ClusterIP from a MicroVM | **Bedrock-direct** via the exec role (`bedrock:InvokeModel`); no Bifrost/NLB/VPC-connector | +| 3 | Runtime logs don't reach CloudWatch | hook-server captures coder stdout → `/logs` HTTP endpoint | +| 4 | Coder is one-shot but the MicroVM `/run` hook has a 30s timeout | `/run` **background-spawns** the coder + returns fast; pipeline polls GitHub for the PR | +| 5 | Env injection needs a `coder` container | bridge container named `coder` (claim contract parity) | +| 6 | aws-cli image lacks `lambda-microvms`; no node | bridge image = `aws-cli:latest` (has the verbs) + python3 for JSON + fetch kubectl at start | +| 7 | Ingress: `ALL_INGRESS` blocks auth-token minting | use **`HTTP_INGRESS`** (+ `SHELL_INGRESS` for debug) | +| 8 | `runHookPayload` is a `SecretKeyReference`; imperative `run-microvm --run-hook-payload` doesn't fire `/run` | deliver via the **declarative `Microvm` CR** | +| 9 | Image rebuild: overwriting the same S3 key doesn't rebuild | use versioned artifact keys; bump `codeArtifactUri` | +| 10 | Pre-GA controller state can wedge (ConflictException / hung build) on delete/recreate | delete the AWS image by ARN or the CR cleanly; keep the hook-server minimal | +| — | IAM for the controller/bridge/exec roles | `iam:PassRole` (ARN-scoped), `lambda:PassNetworkConnector`, `lambda:CreateMicrovmAuthToken`, `bedrock:InvokeModel`, `s3:ListAllMyBuckets` on the capability role | + +Kata needs **none** of these — it's an in-cluster pod with a mounted workspace, native logs, +Bifrost reachability, and a normal ECR image. + +--- + +## When to choose which + +- **Kata (today):** production-ready, mature, standard `kubectl`/IDE access, in-cluster networking. + Choose it now for reliability. Cost: you run + pay for a nested-virt node pool continuously. +- **Lambda MicroVM (strategic):** serverless, suspend-to-zero per idle session, no node pool — + the model that scales economically to thousands of sessions. Choose it as it reaches GA. Cost + today: pre-GA control-plane maturity + the plumbing above. + +Both share the **same coder, same pipeline, same review gates, same UX** — so migrating between +substrates is a label change, invisible to the developer/issue author. diff --git a/docs/dark-factory/SUBSTRATE-DIAGRAMS.md b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md new file mode 100644 index 00000000..4b9651f8 --- /dev/null +++ b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md @@ -0,0 +1,123 @@ +# Dark Factory — Substrate Diagrams (Kata vs Lambda MicroVM) + +Visual companion to [`SUBSTRATE-BENCHMARK.md`](./SUBSTRATE-BENCHMARK.md). All diagrams are +Mermaid (render on GitHub). + +--- + +## 1. Shared pipeline, substrate-branched claim + +Both substrates run the **same `df-run` WorkflowTemplate**. The only branch is which warm pool +`claim-sandbox` claims from — decided by the issue's trigger label. + +```mermaid +flowchart TD + ISSUE["GitHub issue labeled
dark-factory OR darkfactory-lambda"] --> SENSOR["Argo Events sensor"] + SENSOR --> DFRUN["df-run WorkflowTemplate"] + DFRUN --> CLAIM{"claim-sandbox
trigger-label?"} + CLAIM -->|dark-factory| KP["coder-warmpool
(Kata pool)"] + CLAIM -->|darkfactory-lambda| LP["coder-warmpool-microvm
(Lambda bridge pool)"] + KP --> CODE["drive-coder"] + LP --> CODE + CODE --> GATES["holdout-gate · detect→deploy-test
devops-gate → security-agent"] + GATES --> STATUS["status (consolidated verdict)"] + STATUS --> EXIT["onExit: teardown"] +``` + +The DAG has **no MicroVM-specific node** — Lambda suspend/resume lives in the bridge (§4), so the +Kata graph is 100% clean. + +--- + +## 2. Kata micro-VM substrate (Flow B) + +```mermaid +flowchart LR + CLAIM["SandboxClaim"] --> OP["agent-sandbox operator"] + OP --> POD["Kata pod (kata-clh)
on nested-virt node group"] + POD --> ENT["entrypoint.js (baked)"] + ENT -->|models| BIF["Bifrost gateway
(ClusterIP, in-cluster)"] + BIF --> BED["Bedrock"] + ENT -->|git/gh :443| GH["GitHub → PR"] + ENT -->|secrets| SEC["/etc/secrets
(projected tmpfs)"] + POD -.native logs.-> KL["kubectl logs"] + BIF -.traces.-> LF["Langfuse"] +``` + +- Pre-warmed pod → **instant claim**. +- Workspace is a **mounted writable volume**; logs are native; models via Bifrost (with Langfuse + traces). Node pool runs continuously. + +--- + +## 3. Lambda MicroVM substrate (Flow D) + +```mermaid +flowchart LR + CLAIM["SandboxClaim"] --> BR["bridge pod (in-cluster)"] + BR -->|reads handoff| IMG["MicrovmSandbox status
imageARN + execRoleARN
(built once by KRO/ACK)"] + BR -->|creates| MCR["Microvm CR
(runHookPayload = Secret ref)"] + MCR --> CTRL["lambdamicrovms controller"] + CTRL -->|RunMicrovm cold-start| VM["Firecracker MicroVM
hook-server :8080"] + BR -->|mint token, POST /run| VM + VM --> ENT["entrypoint.js (USE_BEDROCK=1)"] + ENT -->|models, direct| BED["Bedrock
(exec role, public egress)"] + ENT -->|git/gh :443| GH["GitHub → PR"] + VM -.coder stdout.-> LOGS["GET /logs (token)"] + BR -->|after PR pushed| SUSP["suspend-microvm
(free compute)"] + BR -->|teardown: delete CR| TERM["controller TerminateMicrovm"] +``` + +- `RunMicrovm` **cold-start per session** (~90s); no node pool. +- No cluster network dependency — **Bedrock-direct**. Logs via `/logs`. Bridge suspends the VM + after the PR, terminates on teardown. + +--- + +## 4. Lambda suspend / resume (bridge-owned, not a DAG step) + +```mermaid +sequenceDiagram + participant B as bridge + participant C as lambdamicrovms controller + participant V as MicroVM + B->>C: create Microvm CR (runHookPayload) + C->>V: RunMicrovm (cold-start) + V-->>B: RUNNING + endpoint + B->>V: POST /run (token) → coder starts + V-->>B: /logs shows "PR opened" + B->>V: suspend-microvm (free compute during review) + Note over V: SUSPENDED (memory+disk preserved) + Note over B,V: on fix round, a fresh Microvm CR is created
(Kata likewise claims a fresh coder per round) + B->>C: delete Microvm CR (on teardown) + C->>V: TerminateMicrovm +``` + +--- + +## 5. End-to-end lifecycle (issue → PR → fix → merge) — both substrates + +```mermaid +flowchart TD + A["Issue labeled"] --> B["df-run: claim + coder → PR"] + B --> C["DevOps Agent + Security Agent review"] + C --> D{"Security findings?"} + D -->|clean| APR["Human approves PR"] + D -->|findings| FIX["Human comments 'fix findings'"] + FIX --> IT["df-iterate → df-run (same substrate via trigger-label)"] + IT --> B + APR --> MERGE["df-merge-teardown:
merge PR + release/terminate sandbox"] +``` + +The loop is identical for both substrates; `df-iterate` reads the originating issue's label to +route the fix round back to the **same** substrate (Kata pool or Lambda pool). + +--- + +## Legend / key facts + +- **Warm pool:** Kata = ready pods (instant); Lambda = bridge pods that RunMicrovm on claim. +- **LLM:** Kata → Bifrost (traced in Langfuse); Lambda → Bedrock-direct (exec role). +- **Workspace:** Kata → mounted volume; Lambda → `/tmp/workspace` (read-only rootfs). +- **Logs:** Kata → `kubectl logs`; Lambda → `/logs` endpoint (+ build logs in CloudWatch). +- **Teardown:** Kata → release claim; Lambda → delete `Microvm` CR → TerminateMicrovm. From 8089bc9fbb0959db7a9bc72c114741892cbb3757 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 13:02:51 -0400 Subject: [PATCH 57/67] =?UTF-8?q?fix(flow-d):=20bump=20bridge=20memory=201?= =?UTF-8?q?28Mi=E2=86=921Gi=20(was=20OOMKilled=20mid=20fix-round)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge runs python3 + curl + fetched kubectl + aws-cli v2 in a poll loop; 128Mi OOMKilled it during the fix round, so it died before the coder's new commit landed and await-coder spun forever ('head still at start sha'). Give it 256Mi req / 1Gi limit. --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 45a53e43..7cea77da 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -375,8 +375,11 @@ spec: capabilities: drop: ["ALL"] resources: - requests: { cpu: 50m, memory: 64Mi } - limits: { cpu: 200m, memory: 128Mi } + # The bridge runs python3 (JSON), curl, a fetched kubectl, and an aws-cli v2 + # (glibc) in a poll loop — 128Mi OOMKilled it mid fix-round (bridge died before + # the coder's new commit landed → await-coder spun forever). Give it real headroom. + requests: { cpu: 100m, memory: 256Mi } + limits: { cpu: "1", memory: 1Gi } volumeMounts: - name: bridge-script mountPath: /scripts From 3ec7fb310e39dc3909918eb4414b27ea33063cc2 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 13:31:04 -0400 Subject: [PATCH 58/67] fix(flow-d): forward df-iterate note into the MicroVM coder A Lambda fix round (df-iterate) re-ran the coder with NO change request: - df-run injects DF_ITERATE_NOTE_B64 into the bridge 'coder' container (Kata parity), but the bridge's runHookPayload dropped it, and hook-server never mapped it into the MicroVM coder's env. - Result: the coder saw the PR already open and reported 'done' on the old sha with zero commits, so findings were never fixed. Fixes: bridge folds DF_ITERATE_NOTE_B64/_NOTE into runHookPayload; hook-server maps them back to the coder env; bump codeArtifactUri r5->r6 to rebuild the MicrovmImage with the fixed hook-server + current entrypoint. --- examples/dark-factory/coder-microvm/hook-server.js | 7 +++++++ .../templates/shim/20-bridge-sandboxtemplate.yaml | 8 +++++++- .../clusters/hub/addons/agent-sandbox-lambda/values.yaml | 6 +++++- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index df5fde7e..78046782 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -51,6 +51,13 @@ function startCoder(payload) { DF_BASE_BRANCH: d.baseBranch || "main", DF_ISSUE_TITLE: d.issueTitle || "", }; + // Fix round (df-iterate): the bridge folds the human's change request into the + // payload as iterateNoteB64. Without this, a Lambda fix round re-runs the coder + // with NO instructions → it sees the PR already open and reports "done" on the + // old sha with zero changes (the Kata path injects DF_ITERATE_NOTE_B64 as claim + // env; the MicroVM has no claim env, so it must ride in on the runHookPayload). + if (d.iterateNoteB64) env.DF_ITERATE_NOTE_B64 = d.iterateNoteB64; + if (d.iterateNote) env.DF_ITERATE_NOTE = d.iterateNote; if (d.model) env.CODER_MODEL = d.model; console.log(`[hook-server] /run → spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); // Capture the coder's stdout+stderr to /tmp/coder.log so /logs can return it — diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 7cea77da..0a008d6c 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -203,7 +203,13 @@ data: [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" # python3 (node is absent in the aws-cli image); json.dumps escapes token/title. One # line so it stays inside the bridge.sh: | YAML block scalar. - PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE","")}))') + # FIX ROUND: df-run injects DF_ITERATE_NOTE_B64 (the human change request) into this + # `coder` container's env (same claim contract as Kata). The MicroVM coder has no + # claim env, so fold the note into the runHookPayload — hook-server maps it back to + # the coder's DF_ITERATE_NOTE_B64. Without this a Lambda fix round runs with NO + # instructions and reports "done" on the OLD sha (zero commits). + [ -n "${DF_ITERATE_NOTE_B64:-}" ] && echo "[microvm-bridge] fix round: forwarding iterate note to coder" + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE",""),"iterateNoteB64":e("DF_ITERATE_NOTE_B64",""),"iterateNote":e("DF_ITERATE_NOTE","")}))') MVM="mvm-${DF_ISSUE_NUMBER}" # Microvm CR + payload Secret name for this session # Build BOTH manifests as JSON with python3 and pipe to kubectl apply. JSON (not a diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 2f8768ca..92968b87 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,7 +28,11 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r5.zip" + # r6: fixed the Lambda fix-round (df-iterate) path — hook-server now maps the + # iterate note (folded into runHookPayload by the bridge) to the coder's + # DF_ITERATE_NOTE_B64, so a fix round actually revises the branch instead of + # re-running with no instructions and reporting "done" on the old sha. + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r6.zip" image: enabled: true name: coder From 4b088b8a0cdf81f0b4d1e1b89eb917a26b98d47e Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 14:28:04 -0400 Subject: [PATCH 59/67] fix(flow-d): fresh MicroVM per fix round (terminate stale suspended VM) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bridge names the Microvm CR mvm- (stable per session). On a fix round the prior VM is still around, SUSPENDED after the first PR. kubectl apply on the same name RESUMES that snapshot, which restores hook-server's one-shot coderStarted=true guard in memory, so the second /run is ignored and the coder never re-runs — the fix round reports 'done' on the OLD sha with no commits, even with the iterate note now forwarded. Delete the stale CR + payload Secret and wait for TerminateMicrovm before recreating, so the fix round gets a clean hook-server that accepts /run. This matches the documented 'fresh Microvm CR per fix round' semantic. No image rebuild needed (r6 already forwards the note). --- .../templates/shim/20-bridge-sandboxtemplate.yaml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 0a008d6c..98c06c01 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -212,6 +212,21 @@ data: PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE",""),"iterateNoteB64":e("DF_ITERATE_NOTE_B64",""),"iterateNote":e("DF_ITERATE_NOTE","")}))') MVM="mvm-${DF_ISSUE_NUMBER}" # Microvm CR + payload Secret name for this session + # FRESH VM PER FIX ROUND: the CR name is stable (mvm-), so on a fix round the + # previous round's VM is still around — SUSPENDED after the first PR. `kubectl apply` + # on the same name RESUMES that snapshot, which restores hook-server's one-shot + # coderStarted=true guard in memory → the second /run is IGNORED and the coder never + # re-runs (observed: fix round reports "done" on the OLD sha, zero commits, even with + # the iterate note forwarded). The documented semantic is "a fresh Microvm CR per fix + # round" — so delete the stale CR + payload Secret and wait for TerminateMicrovm before + # recreating, guaranteeing a clean hook-server that accepts /run. + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + echo "[microvm-bridge] fix round: terminating stale VM ${MVM} to get a fresh hook-server" + kubectl delete microvm "${MVM}" -n "${NS}" --wait=false >/dev/null 2>&1 || true + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + i=0; while [ "$i" -lt 60 ]; do kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1 || break; i=$((i+1)); sleep 5; done + echo "[microvm-bridge] stale VM ${MVM} gone (waited ${i}x5s) — creating fresh" + fi # Build BOTH manifests as JSON with python3 and pipe to kubectl apply. JSON (not a # heredoc) on purpose: a heredoc's column-0 EOF terminator breaks out of the # bridge.sh: | YAML block scalar. JSON is valid YAML and stays on indented lines. From 879347e3c608596764e9243427c6a2b5bbafac92 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 15:13:59 -0400 Subject: [PATCH 60/67] fix(flow-d): real CRD-driven suspend/resume across the fix loop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make Sandbox.operatingMode the single source of truth for MicroVM scale-to-zero (the Flow D highlight), and make resume actually re-run the coder: - idlePolicy.autoResumeEnabled: true -> FALSE. With autoResume on, ANY hit to the VM endpoint auto-resumed it, and the bridge polled /logs every 20s FOREVER, so the VM bounced back to RUNNING seconds after every suspend (console never showed it suspended). Now a suspended VM stays suspended. - Suspend via CRD, not imperatively: after the coder pushes its PR the bridge sets Sandbox.operatingMode=Suspended and STOPS touching the endpoint; the microvm-lifecycle controller (template 30) reconciles that to suspend-microvm. - Resume-on-fix-round: instead of terminating + rebuilding, the bridge flips operatingMode=Running so the controller resume-microvm's the SAME suspended VM (memory+disk preserved) — true warm resume, the whole Flow D value prop. - hook-server /run guard keyed on a per-invocation run-id (issue+note hash) instead of a one-shot boolean frozen in the snapshot, so the resumed VM accepts the fix round's /run and re-runs the coder; also truncates /tmp/coder.log per run so the bridge's 'PR pushed' grep can't match the prior round's line. Bump artifact r6->r7. Verified in-cluster via mini-tests (bare Microvm CR + test Sandbox): create->RUNNING, operatingMode=Suspended->stays SUSPENDED, operatingMode=Running->RUNNING, delete-> TERMINATED; hook-server run-id logic unit-tested (dup ignored, fix-round re-run accepted). --- .../dark-factory/coder-microvm/hook-server.js | 39 +++++++-- .../shim/20-bridge-sandboxtemplate.yaml | 81 ++++++++++++------- .../addons/agent-sandbox-lambda/values.yaml | 12 +-- 3 files changed, 94 insertions(+), 38 deletions(-) diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js index 78046782..0cad32fd 100644 --- a/examples/dark-factory/coder-microvm/hook-server.js +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -26,13 +26,39 @@ const { spawn } = require("child_process"); const PORT = parseInt(process.env.HOOKS_PORT || "8080", 10); const SECRETS_DIR = "/tmp/secrets"; -let coderStarted = false; +// Run-id of the coder invocation currently in flight (or last completed). NOT a plain +// boolean: the VM is SUSPENDED after the first PR and RESUMED for a fix round, and the +// resumed process keeps its in-memory state — a one-shot `coderStarted=true` guard, +// frozen in the snapshot, made the resumed VM ignore the fix round's /run entirely (the +// coder never re-ran; the fix round reported "done" on the old sha). Instead we key on a +// per-invocation run-id (issue + iterate-note hash): a /run whose id differs from the +// one in flight starts a fresh coder (this is a new round after a resume); a /run that +// repeats the current id is a duplicate webhook and is ignored. +let currentRunId = null; +let coderRunning = false; + +function runIdOf(d) { + const note = d.iterateNoteB64 || d.iterateNote || ""; + // Cheap stable hash of issue+note so a fix round (new note) => new id => re-run. + let h = 0; const s = `${d.issueNumber || ""}:${note}`; + for (let i = 0; i < s.length; i++) { h = ((h << 5) - h + s.charCodeAt(i)) | 0; } + return `${d.issueNumber || "?"}#${(h >>> 0).toString(36)}`; +} function startCoder(payload) { - if (coderStarted) { console.log("[hook-server] /run again — already started, ignoring"); return; } - coderStarted = true; let d = {}; try { d = JSON.parse(payload || "{}"); } catch (e) { console.log("[hook-server] payload not JSON:", e.message); } + const rid = runIdOf(d); + if (rid === currentRunId) { console.log(`[hook-server] /run duplicate for ${rid} — ignoring`); return; } + if (coderRunning) { console.log(`[hook-server] /run for ${rid} but ${currentRunId} still running — ignoring`); return; } + const isRerun = currentRunId !== null; // a prior run existed => this is a post-resume fix round + currentRunId = rid; + coderRunning = true; + // Truncate the coder log on each new run. Otherwise the previous round's + // "done — PR opened on " line lingers and the bridge's /logs grep matches it + // instantly, suspending the VM before the fix-round coder has done anything. + try { fs.writeFileSync("/tmp/coder.log", ""); } catch {} + console.log(`[hook-server] /run accepted run-id=${rid}${isRerun ? " (post-resume re-run)" : ""}`); fs.mkdirSync(SECRETS_DIR, { recursive: true, mode: 0o700 }); if (d.ghToken) fs.writeFileSync(`${SECRETS_DIR}/gh-token`, d.ghToken, { mode: 0o400 }); const env = { @@ -66,7 +92,10 @@ function startCoder(payload) { const logFd = fs.openSync("/tmp/coder.log", "a"); const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: ["ignore", logFd, logFd], detached: true }); child.unref(); - child.on("error", (e) => { try { fs.appendFileSync("/tmp/coder.log", "SPAWN-ERROR: " + e.message + "\n"); } catch {} }); + child.on("error", (e) => { coderRunning = false; try { fs.appendFileSync("/tmp/coder.log", "SPAWN-ERROR: " + e.message + "\n"); } catch {} }); + // Clear the in-flight flag when the coder exits so a resumed VM's next /run (fix round) + // is accepted. `unref`'d + detached, but we still get 'exit' while this process lives. + child.on("exit", (code) => { coderRunning = false; console.log(`[hook-server] coder run-id=${currentRunId} exited code=${code}`); }); } const server = http.createServer((req, res) => { @@ -78,7 +107,7 @@ const server = http.createServer((req, res) => { case "/ready": return ok({ status: "ready" }); case "/validate": return ok({ status: "valid" }); case "/run": startCoder(body); return ok({ status: "started" }); - case "/logs": { let l=""; try { l=fs.readFileSync("/tmp/coder.log","utf8"); } catch {} return ok({ status:"ok", started: coderStarted, log: l.slice(-6000) }); } + case "/logs": { let l=""; try { l=fs.readFileSync("/tmp/coder.log","utf8"); } catch {} return ok({ status:"ok", runId: currentRunId, running: coderRunning, log: l.slice(-6000) }); } case "/suspend": return ok({ status: "suspended" }); case "/resume": return ok({ status: "resumed" }); case "/terminate": return ok({ status: "terminated" }); diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index 98c06c01..aae719f4 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -212,20 +212,25 @@ data: PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE",""),"iterateNoteB64":e("DF_ITERATE_NOTE_B64",""),"iterateNote":e("DF_ITERATE_NOTE","")}))') MVM="mvm-${DF_ISSUE_NUMBER}" # Microvm CR + payload Secret name for this session - # FRESH VM PER FIX ROUND: the CR name is stable (mvm-), so on a fix round the - # previous round's VM is still around — SUSPENDED after the first PR. `kubectl apply` - # on the same name RESUMES that snapshot, which restores hook-server's one-shot - # coderStarted=true guard in memory → the second /run is IGNORED and the coder never - # re-runs (observed: fix round reports "done" on the OLD sha, zero commits, even with - # the iterate note forwarded). The documented semantic is "a fresh Microvm CR per fix - # round" — so delete the stale CR + payload Secret and wait for TerminateMicrovm before - # recreating, guaranteeing a clean hook-server that accepts /run. + # RESUME-ON-FIX-ROUND (the suspend/resume highlight): the CR name is stable + # (mvm-). On a fix round the previous round's VM is still around — SUSPENDED + # after the first PR (idlePolicy.autoResumeEnabled=false keeps it down). Rather than + # terminate + rebuild a fresh VM, we RESUME the suspended one via the Sandbox CRD: + # flip operatingMode=Running → the microvm-lifecycle controller calls resume-microvm → + # the SAME VM (memory+disk preserved) comes back, and hook-server accepts a NEW /run + # because its guard is keyed on a per-invocation run-id (issue+note hash), not a + # one-shot boolean (see coder-microvm/hook-server.js). This is the whole Flow D value + # prop: scale-to-zero between rounds, warm-resume for the fix. Fresh VMs are only made + # on the FIRST round (no existing CR). + RESUME_ROUND="" if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then - echo "[microvm-bridge] fix round: terminating stale VM ${MVM} to get a fresh hook-server" - kubectl delete microvm "${MVM}" -n "${NS}" --wait=false >/dev/null 2>&1 || true - kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true - i=0; while [ "$i" -lt 60 ]; do kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1 || break; i=$((i+1)); sleep 5; done - echo "[microvm-bridge] stale VM ${MVM} gone (waited ${i}x5s) — creating fresh" + echo "[microvm-bridge] fix round: resuming suspended VM ${MVM} via Sandbox.operatingMode=Running" + kubectl patch sandbox "${SANDBOX_NAME}" -n "${NS}" --type merge \ + -p '{"spec":{"operatingMode":"Running"}}' >/dev/null 2>&1 \ + && echo "[microvm-bridge] operatingMode=Running set — microvm-lifecycle will resume the VM" \ + || echo "[microvm-bridge] WARN: could not set operatingMode=Running" + # Refresh the payload Secret so the controller/hook sees the NEW iterate note. + RESUME_ROUND=1 fi # Build BOTH manifests as JSON with python3 and pipe to kubectl apply. JSON (not a # heredoc) on purpose: a heredoc's column-0 EOF terminator breaks out of the @@ -238,7 +243,7 @@ data: # Microvm CR: ingress ALL_INGRESS (so the bridge can reach the endpoint to drive /run), # egress INTERNET_EGRESS (Bedrock + git/gh), runtime logging → CloudWatch (logStream # 'runtime' so the coder's stdout is visible, separate from build logs). - MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" REGION="${REGION}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; r=e["REGION"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":["arn:aws:lambda:"+r+":aws:network-connector:aws-network-connector:INTERNET_EGRESS"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":True,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" REGION="${REGION}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; r=e["REGION"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":["arn:aws:lambda:"+r+":aws:network-connector:aws-network-connector:INTERNET_EGRESS"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":False,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } # 4) Wait for the controller to report the running VM's id, record it on the Sandbox # (lifecycle controller reads this to suspend/resume THIS session's VM). @@ -292,33 +297,53 @@ data: } trap cleanup EXIT INT TERM - # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the CR state + the - # in-VM coder /logs so the coder run is OBSERVABLE in the bridge logs (runtime - # CloudWatch routing is unreliable on this runtime). - # SUSPEND-AFTER-CODE (Flow D behavior, done HERE — not as a df-run DAG step, so the - # pipeline graph stays substrate-agnostic/clean for Kata): once the coder has pushed - # its PR (log shows "PR opened"/"done"), suspend the MicroVM to free compute while the - # review gates run; the VM's memory+disk persist for a resume on the next fix round. + # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the in-VM coder /logs + # (OBSERVABILITY — runtime CloudWatch routing is unreliable on this runtime) ONLY + # until the coder pushes its PR. Then suspend and STOP touching the endpoint. + # + # SUSPEND-VIA-CRD (the Flow D highlight): suspend/resume is driven declaratively + # through Sandbox.spec.operatingMode, reconciled by the microvm-lifecycle controller + # (template 30) — NOT by an imperative suspend-microvm call here. Two reasons this + # matters and why the old imperative path FAILED to keep the VM suspended: + # (a) The CR is created with idlePolicy.autoResumeEnabled=FALSE, so a suspended VM + # stays suspended. With autoResume=true (the old value) ANY hit to the VM + # endpoint auto-resumes it — and this loop used to curl /logs every 20s + # FOREVER, so the VM bounced back to RUNNING seconds after every suspend + # (observed in the Lambda console: never actually suspended). + # (b) Routing suspend through operatingMode is the whole point — it shows the + # Agent Sandbox CRD driving MicroVM scale-to-zero via the shim controller. + # So: once the coder pushes, set operatingMode=Suspended (controller suspends the + # VM), then switch to a lightweight CR-existence watch that NEVER touches the + # endpoint again. The VM's memory+disk persist; a fix round flips operatingMode + # back to Running (controller resumes) and the pipeline claims a fresh session. SUSPENDED_ONCE="" echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." while true; do + # CR gone/terminating => real teardown => exit (cleanup trap handles CR delete). ST=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.state}' 2>/dev/null || echo "GONE") case "${ST}" in TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; esac - if [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then + if [ -z "${SUSPENDED_ONCE}" ] && [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then + # Pre-suspend: poll /logs for observability + to detect "coder pushed PR". LOG=$(curl -sS -m 10 "https://${EP}/logs" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("log","")[-800:])' 2>/dev/null || echo "") [ -n "${LOG}" ] && echo "[microvm-bridge] coder-log-tail: $(echo "$LOG" | tail -1)" - # Coder finished (pushed PR) → suspend the VM once (Flow D), if not on a Kata run. - if [ -z "${SUSPENDED_ONCE}" ] && echo "${LOG}" | grep -qiE 'PR opened|done — PR|status success'; then - echo "[microvm-bridge] coder pushed PR — suspending MicroVM ${VMID} (Flow D free-compute-during-review)" - aws lambda-microvms suspend-microvm --region "${REGION}" --microvm-identifier "${VMID}" >/dev/null 2>&1 \ - && echo "[microvm-bridge] suspend-microvm ok" || echo "[microvm-bridge] suspend-microvm skipped/failed" + if echo "${LOG}" | grep -qiE 'PR opened|done — PR|status success'; then + echo "[microvm-bridge] coder pushed PR — requesting SUSPEND via Sandbox.operatingMode (controller reconciles)" + kubectl patch sandbox "${SANDBOX_NAME}" -n "${NS}" --type merge \ + -p '{"spec":{"operatingMode":"Suspended"}}' >/dev/null 2>&1 \ + && echo "[microvm-bridge] operatingMode=Suspended set — microvm-lifecycle will suspend ${VMID}" \ + || echo "[microvm-bridge] WARN: could not set operatingMode=Suspended" SUSPENDED_ONCE=1 fi + sleep 20 + else + # Post-suspend: do NOT touch the endpoint (autoResume=false, but never poke it + # anyway) — just watch for teardown. Longer interval; endpoint stays quiet so the + # VM stays SUSPENDED in the console until a fix round resumes it via operatingMode. + sleep 30 fi - sleep 20 done --- apiVersion: extensions.agents.x-k8s.io/v1beta1 diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml index 92968b87..0abcde06 100644 --- a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -28,11 +28,13 @@ microvm: # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the # artifact zip to this exact bucket/key. - # r6: fixed the Lambda fix-round (df-iterate) path — hook-server now maps the - # iterate note (folded into runHookPayload by the bridge) to the coder's - # DF_ITERATE_NOTE_B64, so a fix round actually revises the branch instead of - # re-running with no instructions and reporting "done" on the old sha. - codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r6.zip" + # r7: hook-server /run guard is now keyed on a per-invocation run-id (issue+note + # hash) instead of a one-shot boolean, so a RESUMED VM (fix round) accepts a fresh + # /run and re-runs the coder — the resume path was previously a no-op because the + # snapshot froze coderStarted=true. Also truncates /tmp/coder.log per run so the + # bridge's "PR pushed" grep doesn't match the previous round's line. + # r6: fix-round iterate note forwarded (bridge payload -> hook-server -> coder env). + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r7.zip" image: enabled: true name: coder From d9bed02d324c40db1f50e5137a742ea72384a05b Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 15:33:49 -0400 Subject: [PATCH 61/67] =?UTF-8?q?fix(flow-d):=20keep=20VM=20on=20suspend?= =?UTF-8?q?=20=E2=80=94=20annotation=20selector=20+=20Sandbox-existence=20?= =?UTF-8?q?teardown=20guard?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two bugs made operatingMode=Suspended TERMINATE the VM instead of suspending it: 1) microvm-lifecycle controller selected Sandboxes by label agent-sandbox.io/substrate=lambda-microvm — but the operator does NOT propagate SandboxTemplate labels onto the Sandbox object, so the selector matched NOTHING and the controller never reconciled any real session (Sandbox went SandboxSuspended, VM never suspended). Select by the microvm-id ANNOTATION the bridge writes instead — only lambda sessions have it. 2) On operatingMode=Suspended the operator DELETES THE POD (keeps the Sandbox alive), firing the bridge cleanup trap. The old trap deleted the Microvm CR unless a racy preStop /tmp marker was set → VM terminated. cleanup now keeps the CR whenever the owning Sandbox STILL EXISTS (suspend), and deletes it only when the Sandbox is GONE (real teardown). Removed the preStop marker. Bridge now exits right after setting operatingMode=Suspended (pod is killed anyway) and stops polling /logs (each hit would auto-resume the VM). --- .../shim/20-bridge-sandboxtemplate.yaml | 64 +++++++++---------- .../templates/shim/30-microvm-lifecycle.yaml | 11 +++- 2 files changed, 39 insertions(+), 36 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml index aae719f4..cc845c6a 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -283,15 +283,25 @@ data: fi fi - # 5) Teardown on real claim end (NOT on suspend): deleting the Microvm CR makes the - # controller terminate the VM. On SUSPEND the pod stops but the CR + VM must stay - # (resume reuses them) — the /tmp/suspending marker (preStop) distinguishes. + # 5) Teardown vs suspend — decided by whether the OWNING SANDBOX STILL EXISTS. + # The agent-sandbox operator handles operatingMode=Suspended by DELETING THE POD + # (verified in the operator log: "Deleting Pod because .Spec.OperatingMode is + # Suspended") while KEEPING the Sandbox object alive in state SandboxSuspended. So + # the bridge pod is torn down on EVERY suspend — and its cleanup trap must NOT delete + # the Microvm CR then, or the VM is terminated instead of suspended (exactly the bug + # we hit: Sandbox survived Suspended, but the VM was gone). + # - Sandbox STILL EXISTS => this is a SUSPEND (or transient pod restart) => KEEP CR. + # - Sandbox GONE => real teardown (df-merge-teardown deleted the claim) => + # delete CR so the controller TerminateMicrovm's the VM. + # (Checking Sandbox existence is more robust than a preStop /tmp marker, which raced + # the SIGTERM and didn't reliably stick.) cleanup() { - if [ -f /tmp/suspending ]; then - echo "[microvm-bridge] pod stopping for SUSPEND — keeping Microvm ${MVM}" + if kubectl get sandbox "${SANDBOX_NAME}" -n "${NS}" >/dev/null 2>&1; then + M=$(kubectl get sandbox "${SANDBOX_NAME}" -n "${NS}" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") + echo "[microvm-bridge] pod stopping but Sandbox ${SANDBOX_NAME} still exists (operatingMode=${M:-?}) — KEEPING Microvm ${MVM} (suspend, not teardown)" return fi - echo "[microvm-bridge] deleting Microvm ${MVM} (controller terminates the VM)" + echo "[microvm-bridge] Sandbox ${SANDBOX_NAME} gone — real teardown: deleting Microvm ${MVM} (controller terminates the VM)" kubectl delete microvm "${MVM}" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true } @@ -316,7 +326,6 @@ data: # VM), then switch to a lightweight CR-existence watch that NEVER touches the # endpoint again. The VM's memory+disk persist; a fix round flips operatingMode # back to Running (controller resumes) and the pipeline claims a fresh session. - SUSPENDED_ONCE="" echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." while true; do # CR gone/terminating => real teardown => exit (cleanup trap handles CR delete). @@ -324,26 +333,27 @@ data: case "${ST}" in TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; esac - if [ -z "${SUSPENDED_ONCE}" ] && [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then - # Pre-suspend: poll /logs for observability + to detect "coder pushed PR". + if [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then + # Poll /logs for observability + to detect "coder pushed PR". LOG=$(curl -sS -m 10 "https://${EP}/logs" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("log","")[-800:])' 2>/dev/null || echo "") [ -n "${LOG}" ] && echo "[microvm-bridge] coder-log-tail: $(echo "$LOG" | tail -1)" if echo "${LOG}" | grep -qiE 'PR opened|done — PR|status success'; then - echo "[microvm-bridge] coder pushed PR — requesting SUSPEND via Sandbox.operatingMode (controller reconciles)" + # Coder is done → request SUSPEND declaratively. The operator then DELETES this + # pod (operatingMode=Suspended), and the microvm-lifecycle controller reconciles + # the same intent into suspend-microvm. We EXIT right after setting it: the pod is + # about to be killed anyway, and exiting cleanly lets the cleanup trap run while + # the Sandbox still exists → it KEEPS the CR (VM suspends, not terminates). We do + # NOT keep polling /logs — every endpoint hit would auto-resume the VM. + echo "[microvm-bridge] coder pushed PR — requesting SUSPEND via Sandbox.operatingMode (controller reconciles suspend-microvm ${VMID})" kubectl patch sandbox "${SANDBOX_NAME}" -n "${NS}" --type merge \ -p '{"spec":{"operatingMode":"Suspended"}}' >/dev/null 2>&1 \ - && echo "[microvm-bridge] operatingMode=Suspended set — microvm-lifecycle will suspend ${VMID}" \ + && echo "[microvm-bridge] operatingMode=Suspended set — exiting bridge (pod will be removed; CR + VM persist)" \ || echo "[microvm-bridge] WARN: could not set operatingMode=Suspended" - SUSPENDED_ONCE=1 + break fi - sleep 20 - else - # Post-suspend: do NOT touch the endpoint (autoResume=false, but never poke it - # anyway) — just watch for teardown. Longer interval; endpoint stays quiet so the - # VM stays SUSPENDED in the console until a fix round resumes it via operatingMode. - sleep 30 fi + sleep 20 done --- apiVersion: extensions.agents.x-k8s.io/v1beta1 @@ -400,21 +410,9 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - # Preflight suspend marker: if this pod is being deleted because the owning - # Sandbox went operatingMode=Suspended, the microvm-lifecycle controller has - # (or will) suspend the MicroVM — the bridge must NOT TerminateMicrovm. The - # controller writes operatingMode; the bridge checks it to set the marker. - lifecycle: - preStop: - exec: - command: - - /bin/sh - - -c - - | - # If the owning Sandbox is Suspended, mark so cleanup() keeps the CR. - M=$(kubectl get sandbox "${SANDBOX_NAME:-df-${DF_ISSUE_NUMBER:-}}" \ - -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") - [ "$M" = "Suspended" ] && touch /tmp/suspending || true + # NOTE: no preStop suspend marker needed anymore — cleanup() decides suspend-vs- + # teardown by whether the owning Sandbox still EXISTS (it survives Suspended, + # is gone on real teardown), which is race-free unlike a preStop /tmp marker. securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml index 6f741a80..bc4e9757 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml @@ -89,10 +89,15 @@ data: INTERVAL="{{ .Values.microvm.lifecycle.intervalSeconds | default 15 }}" echo "[microvm-lifecycle] reconciling every ${INTERVAL}s (ns=${NS} region=${REGION})" while true; do - # Only Sandboxes on the lambda-microvm substrate (bridge sets this label). + # Select Sandboxes on the lambda-microvm substrate by the microvm-id ANNOTATION the + # bridge writes after RunMicrovm — NOT a label. The agent-sandbox operator does NOT + # propagate SandboxTemplate labels onto the Sandbox object, so a label selector + # (agent-sandbox.io/substrate=lambda-microvm) matches NOTHING and the controller + # stays blind to every real session (observed: operatingMode=Suspended set, Sandbox + # went SandboxSuspended, but the VM was never suspended because this loop skipped it). + # Only lambda sessions carry the microvm-id annotation, so it's the reliable signal. for sb in $(kubectl get sandbox -n "$NS" \ - -l agent-sandbox.io/substrate=lambda-microvm \ - -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do + -o jsonpath='{range .items[?(@.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id)]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do MODE=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "Running") LAST=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath="{.metadata.annotations.${ANN}}" 2>/dev/null || echo "") [ "$MODE" = "$LAST" ] && continue # no transition From 81483000d1f1fcfab5545338e18cc7e030d0817d Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 16:19:27 -0400 Subject: [PATCH 62/67] =?UTF-8?q?fix(flow-d):=20one=20Sandbox=20per=20VM?= =?UTF-8?q?=20=E2=80=94=20stable=20claim=20naming=20by=20issue-number?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the suspend/resume flap: the claim/Sandbox was named by issue-id, which DIVERGES between the sensor's first run (github event id, e.g. 5064265549) and df-iterate's fix round (issue number, e.g. 133). So a fix round created a SECOND Sandbox (df-issue-133) while the first run's (df-issue-5064265549) still existed — BOTH annotated with the same microvm-id. The lifecycle controller then saw two Sandboxes with OPPOSITE operatingMode intents for one VM and flapped suspend<->resume until the VM died (Internal service error). Name the claim/Sandbox by issue-NUMBER everywhere (claim, mutex, teardown) — stable across first run + all fix rounds, and already the key for the Microvm CR (mvm-). Exactly one Sandbox per VM. Also: suspendedDurationSeconds 300 -> 86400 was wrong (Lambda max suspend is 8h); set within cap next commit. maxIdle 900 -> 1800. --- .../charts/agent-sandbox-lambda/values.yaml | 15 ++++++++++-- .../templates/20-workflowtemplate-df-run.yaml | 23 +++++++++++++++---- 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml index db733ca5..60009822 100644 --- a/gitops/addons/charts/agent-sandbox-lambda/values.yaml +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -99,8 +99,19 @@ microvm: # coder codes → SUSPEND → agents review → (fix findings) → RESUME same VM → # … loop until cleared → merge/exit → TERMINATE. defaults: - maxIdleDurationSeconds: 900 - suspendedDurationSeconds: 300 + # maxIdleDurationSeconds: how long a RUNNING VM may sit idle (no inbound) before the + # runtime suspends it. Must exceed a coder run (a few min) — the bridge suspends + # explicitly anyway, this is just a backstop. + maxIdleDurationSeconds: 1800 + # suspendedDurationSeconds: how long a SUSPENDED VM is kept before the runtime + # AUTO-TERMINATES it. THIS IS CRITICAL for the review→fix loop: the VM is suspended + # while the external review agents run (~8–15 min) and then waits for a human to post + # "fix findings" (minutes to hours). At the old 300s (5 min) the VM was ALWAYS + # auto-terminated before the fix round, so RESUME hit "has been terminated and its + # state cannot be changed" and the same-VM warm-resume was impossible. Keep it + # suspended long enough to span a realistic review+human cycle (24h); teardown + # terminates it explicitly at merge, so this only bounds abandoned PRs. + suspendedDurationSeconds: 86400 # microvm-lifecycle controller (30-microvm-lifecycle.yaml) — reconciles # Sandbox.operatingMode Running↔Suspended → suspend/resume-microvm. Loop interval. diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index d92f3982..398289b5 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -27,10 +27,12 @@ metadata: spec: serviceAccountName: dark-factory-workflow # Serialize per issue (no double-claim for the same issue) and cap total - # concurrent runs against the kata pool size. + # concurrent runs against the kata pool size. Keyed on issue-NUMBER (stable across + # first run + fix rounds) to match the claim/Sandbox name below — so a fix round is + # serialized against its own issue's first run and can't race a second claim. synchronization: mutex: - name: "df-issue-{{`{{workflow.parameters.issue-id}}`}}" + name: "df-issue-{{`{{workflow.parameters.issue-number}}`}}" entrypoint: main arguments: parameters: @@ -227,8 +229,19 @@ spec: manifest: | apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxClaim + # Name the claim (and therefore the Sandbox) by issue-NUMBER, which is STABLE + # across the first run and every fix round — NOT issue-id. The sensor's first + # run passes issue-id= (e.g. 5064265549) while df-iterate + # passes issue-id= (e.g. 133); keying the claim on issue-id made + # a fix round create a SECOND, differently-named Sandbox (df-issue-133) while + # the first run's Sandbox (df-issue-5064265549) still existed — BOTH annotated + # with the SAME microvm-id. The Lambda lifecycle controller then saw two + # Sandboxes giving OPPOSITE operatingMode intents for one VM and flapped + # suspend↔resume until the VM died with an Internal service error. issue-number + # is unique per issue and already names the Microvm CR (mvm-), so + # this gives exactly one Sandbox per VM across all rounds. metadata: - name: df-issue-{{`{{workflow.parameters.issue-id}}`}} + name: df-issue-{{`{{workflow.parameters.issue-number}}`}} namespace: {{ .Values.warmPool.namespace }} labels: dark-factory.io/issue: "{{`{{workflow.parameters.issue-id}}`}}" @@ -880,7 +893,9 @@ spec: set -eu LABEL="{{`{{workflow.parameters.trigger-label}}`}}" NS="{{ .Values.warmPool.namespace }}" - CLAIM="df-issue-{{`{{workflow.parameters.issue-id}}`}}" + # issue-NUMBER — must match the claim name created in claim-sandbox (stable across + # first run + fix rounds); issue-id diverges between the sensor and df-iterate. + CLAIM="df-issue-{{`{{workflow.parameters.issue-number}}`}}" if [ "$LABEL" = "darkfactory-lambda" ]; then echo "[teardown] lambda substrate — KEEPING ${CLAIM} (VM stays SUSPENDED until merge)." exit 0 From b05e9ffd71da987456c0a8d421dbca03e7c7a7ec Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 16:42:47 -0400 Subject: [PATCH 63/67] refactor(flow-d): separate df-run-lambda; restore df-run to certified Kata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Kata and Lambda no longer share one df-run template. Per the constraint 'don't touch certified Kata': - df-run (Kata): reverted BYTE-IDENTICAL to the certified dark-factory-autonomous-agent-coding-pattern template. Zero MicroVM logic. - df-run-lambda (NEW, 23-...): MicroVM-native pipeline, NO SandboxClaim/bridge/ warm-pool. DAG: provision-microvm → drive-coder → suspend-microvm → [holdout ∥ devops ∥ security ∥ deploy-test] → status; onExit KEEPS the suspended VM. provision-microvm creates the Microvm CR + runHookPayload Secret directly (note folded into payload), waits RUNNING+endpoint, mints token, POSTs /run. On a fix round it RESUMES the same suspended VM (warm resume) or recreates if terminated. Encodes every Flow D learning: autoResume=false + no post-/run endpoint polling (suspend sticks), suspendedDuration=8h (survives review→fix), HTTP_INGRESS, stable CR name mvm- (one VM per issue), image/exec-role from the platform MicrovmSandbox status. - df-merge-teardown: microvm-terminate now deletes the Microvm CR by stable name (was Sandbox-annotation lookup); gated on microvm.enabled. - values: replaced microvmSuspend{} with a microvm{} block (region/namespace/ stepImage/image.name/defaults). suspendedDurationSeconds=28800 (8h cap). Routing (sensor + iterate.js → df-run-lambda) follows next. --- .../templates/20-workflowtemplate-df-run.yaml | 111 +- ...21-workflowtemplate-df-merge-teardown.yaml | 49 +- .../23-workflowtemplate-df-run-lambda.yaml | 945 ++++++++++++++++++ gitops/addons/charts/dark-factory/values.yaml | 30 +- 4 files changed, 1003 insertions(+), 132 deletions(-) create mode 100644 gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml diff --git a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml index 398289b5..97706361 100644 --- a/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml +++ b/gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml @@ -27,12 +27,10 @@ metadata: spec: serviceAccountName: dark-factory-workflow # Serialize per issue (no double-claim for the same issue) and cap total - # concurrent runs against the kata pool size. Keyed on issue-NUMBER (stable across - # first run + fix rounds) to match the claim/Sandbox name below — so a fix round is - # serialized against its own issue's first run and can't race a second claim. + # concurrent runs against the kata pool size. synchronization: mutex: - name: "df-issue-{{`{{workflow.parameters.issue-number}}`}}" + name: "df-issue-{{`{{workflow.parameters.issue-id}}`}}" entrypoint: main arguments: parameters: @@ -64,7 +62,7 @@ spec: # Always release the claimed sandbox, on success OR failure. onExit: teardown ttlStrategy: - secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} + secondsAfterCompletion: 3600 {{- if .Values.metrics.enabled }} # Success metrics (Argo-native Prometheus). Argo exposes these on the workflow # controller's :9090/metrics — scraped into the platform Prometheus. Gives the @@ -95,11 +93,6 @@ spec: tasks: - name: claim template: claim-sandbox - arguments: - parameters: - # Flow D: darkfactory-lambda → the Lambda-MicroVM warm pool; else Kata. - - name: warm-pool - value: "{{`{{=workflow.parameters['trigger-label'] == 'darkfactory-lambda' ? '`}}{{ .Values.warmPool.lambdaName | default "coder-warmpool-microvm" }}{{`' : '`}}{{ .Values.warmPool.name }}{{`'}}`}}" - name: drive-coder template: await-coder dependencies: [claim] @@ -107,26 +100,6 @@ spec: parameters: - name: sandbox value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" -{{- if .Values.microvmSuspend.enabled }} - # Flow D ONLY — after the coder pushes, SUSPEND the Lambda MicroVM - # (operatingMode=Suspended → microvm-lifecycle calls suspend-microvm). The VM - # persists suspended across the review→fix loop; df-iterate resumes the SAME - # VM; terminated only at merge. Gated on the lambda substrate + a PR; no-op for Kata. - - name: microvm-suspend - template: microvm-set-mode - dependencies: [drive-coder] - # Quote BOTH operands: Argo substitutes the label value inline, so an - # unquoted `dark-factory == darkfactory-lambda` is parsed as arithmetic on - # bare identifiers and errors ("Failed to evaluate 'when' expression") — - # the step then shows phase=Error even though it's correctly Skipped on Kata. - when: "\"{{`{{workflow.parameters.trigger-label}}`}}\" == \"darkfactory-lambda\" && \"{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}\" != \"\"" - arguments: - parameters: - - name: sandbox - value: "{{`{{tasks.claim.outputs.parameters.sandbox}}`}}" - - name: mode - value: "Suspended" -{{- end }} {{- if .Values.holdout.enabled }} # P2 — holdout gate: hidden scenarios + a different-family judge. - name: holdout-gate @@ -211,11 +184,6 @@ spec: # ---- 1. Claim a warm sandbox (creates the SandboxClaim with issue env) ---- - name: claim-sandbox - inputs: - parameters: - # Which warm pool to claim from — Kata (default) or Lambda-MicroVM (Flow D). - - name: warm-pool - value: "{{ .Values.warmPool.name }}" outputs: parameters: - name: sandbox @@ -229,19 +197,8 @@ spec: manifest: | apiVersion: extensions.agents.x-k8s.io/v1beta1 kind: SandboxClaim - # Name the claim (and therefore the Sandbox) by issue-NUMBER, which is STABLE - # across the first run and every fix round — NOT issue-id. The sensor's first - # run passes issue-id= (e.g. 5064265549) while df-iterate - # passes issue-id= (e.g. 133); keying the claim on issue-id made - # a fix round create a SECOND, differently-named Sandbox (df-issue-133) while - # the first run's Sandbox (df-issue-5064265549) still existed — BOTH annotated - # with the SAME microvm-id. The Lambda lifecycle controller then saw two - # Sandboxes giving OPPOSITE operatingMode intents for one VM and flapped - # suspend↔resume until the VM died with an Internal service error. issue-number - # is unique per issue and already names the Microvm CR (mvm-), so - # this gives exactly one Sandbox per VM across all rounds. metadata: - name: df-issue-{{`{{workflow.parameters.issue-number}}`}} + name: df-issue-{{`{{workflow.parameters.issue-id}}`}} namespace: {{ .Values.warmPool.namespace }} labels: dark-factory.io/issue: "{{`{{workflow.parameters.issue-id}}`}}" @@ -252,7 +209,7 @@ spec: dark-factory.io/managed-by: df-run spec: warmPoolRef: - name: "{{`{{inputs.parameters.warm-pool}}`}}" + name: {{ .Values.warmPool.name }} lifecycle: ttlSecondsAfterFinished: {{ .Values.claimTtlSeconds }} env: @@ -852,53 +809,13 @@ spec: node /scripts/status.js # ---- onExit: release the claim (operator refills the pool) ---- -{{- if .Values.microvmSuspend.enabled }} - # ---- Flow D: set Sandbox.operatingMode (suspend/resume the Lambda MicroVM) ---- - # Flips spec.operatingMode; the agent-sandbox-lambda microvm-lifecycle controller - # observes it and calls suspend-microvm / resume-microvm by the microvm-id - # annotation the bridge wrote. Advisory verify (never fails the run). Lambda only. - - name: microvm-set-mode - inputs: - parameters: - - name: sandbox - - name: mode # Running | Suspended - script: - image: {{ .Values.microvmSuspend.image | default "alpine/k8s:1.31.0" }} - command: [sh] - source: | - set -eu - SB="{{`{{inputs.parameters.sandbox}}`}}" - MODE="{{`{{inputs.parameters.mode}}`}}" - NS="{{ .Values.warmPool.namespace }}" - echo "[microvm-set-mode] Sandbox/$SB -> operatingMode=$MODE" - kubectl patch sandbox "$SB" -n "$NS" --type merge -p "{\"spec\":{\"operatingMode\":\"$MODE\"}}" - i=0; while [ "$i" -lt 12 ]; do - VMID=$(kubectl get sandbox "$SB" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") - [ -n "$VMID" ] && { echo "[microvm-set-mode] microvmID=$VMID"; break; } - i=$((i+1)); sleep 5 - done - echo "[microvm-set-mode] done (advisory)." - activeDeadlineSeconds: 180 -{{- end }} - - # ---- onExit: release the claim (operator refills the pool) ---- - # SUBSTRATE-AWARE: Kata deletes the claim now (df-run done → free the pool). LAMBDA - # (Flow D) KEEPS the claim + SUSPENDED VM so the same VM survives the review→fix loop - # (df-iterate resumes it); the VM is terminated only at merge (df-merge-teardown). - name: teardown - script: - image: {{ .Values.stepImage }} - command: [sh] - source: | - set -eu - LABEL="{{`{{workflow.parameters.trigger-label}}`}}" - NS="{{ .Values.warmPool.namespace }}" - # issue-NUMBER — must match the claim name created in claim-sandbox (stable across - # first run + fix rounds); issue-id diverges between the sensor and df-iterate. - CLAIM="df-issue-{{`{{workflow.parameters.issue-number}}`}}" - if [ "$LABEL" = "darkfactory-lambda" ]; then - echo "[teardown] lambda substrate — KEEPING ${CLAIM} (VM stays SUSPENDED until merge)." - exit 0 - fi - echo "[teardown] kata substrate — deleting SandboxClaim/${CLAIM}." - kubectl delete sandboxclaim "${CLAIM}" -n "${NS}" --ignore-not-found --wait=false || true + resource: + action: delete + flags: ["--ignore-not-found"] + manifest: | + apiVersion: extensions.agents.x-k8s.io/v1beta1 + kind: SandboxClaim + metadata: + name: df-issue-{{`{{workflow.parameters.issue-id}}`}} + namespace: {{ .Values.warmPool.namespace }} diff --git a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml index 56f297fa..da82dae0 100644 --- a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml +++ b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml @@ -39,18 +39,18 @@ spec: tasks: - name: merge template: merge-pr -{{- if .Values.microvmSuspend.enabled }} - # Flow D: this is the FINAL exit — TERMINATE the Lambda MicroVM that was - # kept suspended across the review→fix loop. Runs before the claim delete so - # the VM is destroyed explicitly (the suspended VM would otherwise be kept by - # the bridge's suspend-marker on pod delete → leak). No-op for Kata. +{{- if .Values.microvm.enabled }} + # Flow D: this is the FINAL exit — TERMINATE the Lambda MicroVM that df-run-lambda + # kept SUSPENDED across the review→fix loop. Deletes the Microvm CR (stable name + # mvm-) → the controller TerminateMicrovm's the VM. No-op for Kata + # (there is no such CR). Advisory — never fails the merge. - name: microvm-terminate template: microvm-terminate dependencies: [merge] {{- end }} - name: teardown template: teardown-claim -{{- if .Values.microvmSuspend.enabled }} +{{- if .Values.microvm.enabled }} dependencies: [microvm-terminate] {{- else }} dependencies: [merge] @@ -104,30 +104,31 @@ spec: echo "[df-merge] human-approved PR #${PR} in ${REPO} — verifying + merging" node /scripts/merge.js -{{- if .Values.microvmSuspend.enabled }} +{{- if .Values.microvm.enabled }} # ---- Flow D: terminate the Lambda MicroVM (final exit) ---- - # Resolve the per-session microvm id from the owning Sandbox annotation (the bridge - # wrote it after RunMicrovm), then TerminateMicrovm. A suspended VM can be - # terminated directly. Advisory (never fails the merge) — the reaper + idlePolicy - # are backstops. Needs lambda-microvms:TerminateMicrovm on the workflow's IRSA role. + # df-run-lambda kept the VM SUSPENDED (CR mvm-) across the review→fix + # loop. At merge we DELETE that CR → the controller TerminateMicrovm's the VM. Stable + # name, so no Sandbox/annotation lookup needed. No-op for Kata (no such CR). Advisory. - name: microvm-terminate script: - image: {{ .Values.microvmSuspend.image | default "alpine/k8s:1.31.0" }} + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} command: [sh] source: | set -eu - NS="{{ .Values.warmPool.namespace }}" - REGION="{{ .Values.securityAgent.region | default "us-west-2" }}" - NUM="{{`{{workflow.parameters.issue-number}}`}}" - # Find the Sandbox for this issue (by the number label the claim/sandbox carry). - SB=$(kubectl get sandbox -n "$NS" -l "dark-factory.io/issue-number=${NUM}" \ - -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "") - [ -z "$SB" ] && { echo "[microvm-terminate] no Sandbox for issue #${NUM} — nothing to terminate"; exit 0; } - VMID=$(kubectl get sandbox "$SB" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") - [ -z "$VMID" ] && { echo "[microvm-terminate] Sandbox/$SB has no microvm-id — nothing to terminate"; exit 0; } - echo "[microvm-terminate] TerminateMicrovm ${VMID} (issue #${NUM}, final exit)" - command -v aws >/dev/null 2>&1 || { echo "[microvm-terminate] aws cli missing on image — skipping (advisory)"; exit 0; } - aws lambda-microvms terminate-microvm --region "$REGION" --microvm-identifier "$VMID" 2>&1 || echo "[microvm-terminate] terminate failed (advisory)" + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + echo "[microvm-terminate] deleting Microvm/${MVM} → controller TerminateMicrovm (final exit)" + kubectl delete microvm "${MVM}" -n "${NS}" --wait=false 2>&1 || echo "[microvm-terminate] delete failed (advisory)" + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + else + echo "[microvm-terminate] no Microvm/${MVM} — nothing to terminate (Kata or already gone)" + fi {{- end }} # ---- Teardown the claim (operator refills the pool) ---- diff --git a/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml b/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml new file mode 100644 index 00000000..692d7dec --- /dev/null +++ b/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml @@ -0,0 +1,945 @@ +{{- /* +df-run — the Flow B P1 pipeline as an Argo WorkflowTemplate. + + trigger → claim warm sandbox → coder implements+tests (in the Kata VM) → + coder opens PR + posts a live sticky status → await human approval. + +Single-cluster on the hub: the claim step creates a SandboxClaim with the issue +injected as env (verified contract: envVarsInjectionPolicy=Allowed), waits for +the operator to bind a warm micro-VM (status Ready), then the coder VM — which +boots the coder image from the SandboxTemplate — reads the env, implements on +df/issue-N, builds+tests, pushes, and opens the PR + sticky comment itself using +the short-TTL GitHub token projected into the VM. The workflow watches the coder +pod to completion, then its onExit handler releases the claim (the operator +refills the pool). P1 stops at "PR open, awaiting human" — merge/teardown + +verification gates are P2-P4. + +Parameters (supplied by the trigger / Sensor): + issue-id, issue-number, repo (owner/name), issue-title, issue-body, base-branch +*/ -}} +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: df-run-lambda + namespace: {{ .Values.argo.namespace }} + labels: + {{- include "dark-factory.labels" . | nindent 4 }} +spec: + serviceAccountName: dark-factory-workflow + # Serialize per issue (no double-claim for the same issue) and cap total + # concurrent runs against the kata pool size. Keyed on issue-NUMBER (stable across + # first run + fix rounds) to match the claim/Sandbox name below — so a fix round is + # serialized against its own issue's first run and can't race a second claim. + synchronization: + mutex: + name: "df-issue-{{`{{workflow.parameters.issue-number}}`}}" + entrypoint: main + arguments: + parameters: + - name: issue-id + - name: issue-number + - name: repo + - name: issue-title + - name: issue-body + value: "" + - name: base-branch + value: main + # Optional revision request (df-iterate passes the PR comment here; empty on + # a first pass). Injected into the coder as DF_ITERATE_NOTE so it revises the + # existing branch to address the feedback. + - name: iterate-note + value: "" + # Base64 of the revision note — used when the note is arbitrary markdown + # (auto-fed agent findings) that can't be injected raw into the claim YAML. + # The coder decodes this first, falling back to the plain iterate-note. + - name: iterate-note-b64 + value: "" + # Which label fired the run selects the coder substrate (Kata default vs + # Lambda MicroVM for darkfactory-lambda). MUST be declared with a default so + # every trigger-label reference resolves even when the submitter (sensor / + # manual) does not pass it. Defaulting to dark-factory keeps the Kata path for + # sensor submits that do not map a label. + - name: trigger-label + value: "dark-factory" + # Always release the claimed sandbox, on success OR failure. + onExit: teardown + ttlStrategy: + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} +{{- if .Values.metrics.enabled }} + # Success metrics (Argo-native Prometheus). Argo exposes these on the workflow + # controller's :9090/metrics — scraped into the platform Prometheus. Gives the + # GitOps-native view of the factory: throughput, outcome mix, and lead time. + metrics: + prometheus: + - name: df_runs_total + help: "Dark Factory df-run workflows by status" + labels: + - { key: status, value: "{{`{{workflow.status}}`}}" } + counter: + value: "1" + - name: df_run_duration_seconds + help: "df-run wall-clock duration (lead time proxy)" + gauge: + value: "{{`{{workflow.duration}}`}}" +{{- end }} + templates: + + # ---- DAG ---- + # claim → coder → [ verify fan: holdout-gate ∥ security-review ∥ devops-review ] + # → status. Every verify step runs OUTSIDE the coder (trusted hub pod), is gated + # on a PR existing, and is advisory in v1 (posts a commit status; never fails the + # run unless its *.blocking / blockSeverity is raised). `status` waits for all + # enabled verify steps via its dependencies list. + # ---- DAG (MicroVM-native, NO SandboxClaim/bridge/warm-pool) ---- + # provision-microvm (create Microvm CR + drive /run) → drive-coder (poll GitHub for + # PR) → suspend-microvm (scale-to-zero during review) → [verify fan] → status. + # On a FIX ROUND the workflow first RESUMES the same suspended VM (resume-microvm), + # re-drives /run with the new note, then re-suspends. Teardown (onExit) deletes the + # Microvm CR → controller TerminateMicrovm. + - name: main + dag: + tasks: + # 1. Provision the MicroVM directly (create CR + Secret, wait RUNNING, POST /run). + # On a fix round (iterate-note set) it RESUMES the existing suspended VM instead. + - name: provision-microvm + template: provision-microvm + # 2. Poll GitHub for the PR the coder opens (substrate-agnostic; reused verbatim). + - name: drive-coder + template: await-coder + dependencies: [provision-microvm] + arguments: + parameters: + - name: sandbox + value: "" + # 3. SUSPEND the VM once the coder pushed its PR — scale-to-zero during review. + # Explicit DAG step (this template is Lambda-only, so no substrate gating). + - name: suspend-microvm + template: set-microvm-power + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: action + value: "suspend" +{{- if .Values.holdout.enabled }} + # P2 — holdout gate: hidden scenarios + a different-family judge. + - name: holdout-gate + template: holdout-gate + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: pr-number + value: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}" +{{- end }} +{{- if .Values.review.enabled }} + # P3 — REAL AWS Frontier Agents, ordered: DevOps FIRST, then Security. + # + # DevOps Agent (release readiness) runs via the Claude Code plugin INSIDE + # the coder step (no headless API), so there is no separate DAG task for + # it here — its verdict is reported by the coder and, on a clear verdict, + # the coder applies the `needs-security-review` label. This `devops-gate` + # step confirms the label is present (DevOps cleared) before Security runs. + - name: devops-gate + template: devops-gate + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: pr-number + value: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}" + # Security Agent — WAITS FOR the real aws-security-agent[bot]'s own review + # and mirrors its verdict into dark-factory/security. The App bot reviews + # autonomously + IN PARALLEL with the DevOps bot, so this runs alongside + # devops-gate (depends on drive-coder, NOT gated behind DevOps clearing — + # otherwise a slow DevOps review would skip the security signal entirely). + - name: security-agent + template: security-agent + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" +{{- end }} +{{- if .Values.deployTest.enabled }} + # P4 — detect whether the PR touches deployable artifacts (cheap grep of + # the diff). Its output gates the (expensive, K8s-touching) deploy-test. + - name: detect-deployable + template: detect-deployable + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + # P4 — deploy-test: THE ONLY step that holds K8s access. Content-aware: + # k8s → ephemeral-namespace apply+probe+teardown; terraform → init+validate. + # Runs only when detect classified the change (kind != none). + - name: deploy-test + template: deploy-test + dependencies: [detect-deployable] + when: "{{`{{tasks.detect-deployable.outputs.parameters.kind}}`}} != none" + arguments: + parameters: + - name: kind + value: "{{`{{tasks.detect-deployable.outputs.parameters.kind}}`}}" +{{- end }} + - name: status + template: sticky-status + # Depend on BOTH agents (devops-gate + security-agent) plus holdout + + # deploy-test so the consolidated verdict is only computed once EVERY + # signal is terminal — required so a ❌ (and the auto-fix decision) is made + # on complete input, and so a clear verdict truly means both agents cleared. + dependencies: + {{- if or .Values.holdout.enabled .Values.review.enabled .Values.deployTest.enabled }} + {{- if .Values.holdout.enabled }} + - holdout-gate + {{- end }} + {{- if .Values.review.enabled }} + - devops-gate + - security-agent + {{- end }} + {{- if .Values.deployTest.enabled }} + - deploy-test + {{- end }} + {{- else }} + - drive-coder + {{- end }} + arguments: + parameters: + - name: phase + value: "pr-open" + + # ---- 1. Provision the Lambda MicroVM directly (NO SandboxClaim/bridge/warm-pool) ---- + # Creates the runHookPayload Secret + the Microvm CR (the ACK lambdamicrovms + # controller does RunMicrovm), waits for RUNNING + an endpoint, mints an auth token, + # and POSTs /run to start the coder. On a FIX ROUND (iterate-note set) it RESUMES the + # same suspended VM if still alive (warm resume — the Flow D value prop), else recreates. + # Every hard-won Flow D fix is encoded here: + # • autoResumeEnabled=false + we never poll the endpoint after /run → suspend STICKS. + # • suspendedDurationSeconds=8h so the VM survives the review→fix window. + # • HTTP_INGRESS (ALL_INGRESS blocks auth-token minting). + # • runHookPayload carries ghToken + DF_* + the iterate note (the coder has no claim env). + # • CR/Secret named mvm- (stable across rounds → one VM per issue). + # • image/exec-role read from the platform MicrovmSandbox status (built once by KRO/ACK). + - name: provision-microvm + outputs: + parameters: + - name: microvm-id + valueFrom: { path: /tmp/vmid } + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: token + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BASE_BRANCH + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: ISSUE_TITLE + value: "{{`{{workflow.parameters.issue-title}}`}}" + - name: ITERATE_NOTE_B64 + value: "{{`{{workflow.parameters.iterate-note-b64}}`}}" + - name: ITERATE_NOTE + value: "{{`{{workflow.parameters.iterate-note}}`}}" + source: | + set -eu + REGION="{{ .Values.microvm.region }}" + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + PLATFORM_IMAGE="{{ .Values.microvm.image.name | default "coder" }}" + MVM="mvm-${ISSUE_NUMBER}" + BRANCH="df/issue-${ISSUE_NUMBER}" + MAXIDLE={{ .Values.microvm.defaults.maxIdleDurationSeconds | default 1800 }} + SUSPDUR={{ .Values.microvm.defaults.suspendedDurationSeconds | default 28800 }} + INGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:HTTP_INGRESS" + EGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:INTERNET_EGRESS" + LOGGRP="/aws/lambda/microvms/${PLATFORM_IMAGE}-image" + # aws-cli image has no kubectl — fetch a static one (same as the old bridge). + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + # 1) Read the platform image handoff (imageARN + execRoleARN — built ONCE by KRO/ACK). + echo "[provision] reading platform image ${PLATFORM_IMAGE} (waiting for build ready)..." + i=0; IMAGE_ARN=""; EXEC_ROLE="" + while [ "$i" -lt 240 ]; do + IST=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageState}' 2>/dev/null || echo "") + if [ "${IST}" = "CREATED" ] || [ "${IST}" = "UPDATED" ]; then + IMAGE_ARN=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageARN}' 2>/dev/null || echo "") + EXEC_ROLE=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.executionRoleARN}' 2>/dev/null || echo "") + [ -n "${IMAGE_ARN}" ] && [ -n "${EXEC_ROLE}" ] && break + fi + i=$((i+1)); sleep 5 + done + [ -z "${IMAGE_ARN}" ] || [ -z "${EXEC_ROLE}" ] && { echo "[provision] ERROR: platform image not ready (imageState=${IST:-none})"; exit 1; } + echo "[provision] image=${IMAGE_ARN} execRole=${EXEC_ROLE}" + # 2) FIX ROUND vs FIRST RUN: if a CR exists, check the VM's ACTUAL AWS state. + # RUNNING/SUSPENDED/PENDING → RESUME the same VM (warm resume). Else (TERMINATED + # past the 8h suspend cap, or a service error) → recreate a fresh VM. + FRESH=1 + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + PRIORVMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + PRIORSTATE="" + [ -n "${PRIORVMID}" ] && PRIORSTATE=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${PRIORVMID}" --query 'state' --output text 2>/dev/null || echo "") + echo "[provision] fix round: prior VM ${PRIORVMID:-none} state=${PRIORSTATE:-unknown}" + case "${PRIORSTATE}" in + RUNNING|SUSPENDED|PENDING) + echo "[provision] RESUMING same VM ${PRIORVMID} (warm resume)" + aws lambda-microvms resume-microvm --region "${REGION}" --microvm-identifier "${PRIORVMID}" 2>/dev/null || true + # Refresh the payload Secret so the resumed hook-server gets the NEW note. + FRESH=0; VMID="${PRIORVMID}" + ;; + *) + echo "[provision] prior VM not resumable (${PRIORSTATE:-gone}) — recreating fresh" + kubectl delete microvm "${MVM}" -n "${NS}" --wait=false >/dev/null 2>&1 || true + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + j=0; while [ "$j" -lt 60 ]; do kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1 || break; j=$((j+1)); sleep 5; done + ;; + esac + fi + # 3) Build the runHookPayload (ghToken + DF_* + iterate note) and (re)write the Secret. + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" IN="${ISSUE_NUMBER}" RP="${REPO}" BR="${BRANCH}" BB="${BASE_BRANCH}" IT="${ISSUE_TITLE}" NB="${ITERATE_NOTE_B64}" NP="${ITERATE_NOTE}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("IN",""),"repo":e("RP",""),"branch":e("BR",""),"baseBranch":e("BB","main"),"issueTitle":e("IT",""),"iterateNoteB64":e("NB",""),"iterateNote":e("NP","")}))') + MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" python3 -c 'import json,os; e=os.environ; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":e["MVM"]+"-payload","namespace":e["NS"]},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[provision] payload secret apply failed"; exit 1; } + # 4) On a FIRST RUN (or recreate), create the Microvm CR (autoResume=false). + if [ "${FRESH}" = "1" ]; then + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" EGRESS="${EGRESS}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":[e["EGRESS"]],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":False,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[provision] Microvm CR apply failed"; exit 1; } + VMID=""; i=0 + while [ "$i" -lt 60 ]; do + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -n "${VMID}" ] && break; i=$((i+1)); sleep 5 + done + fi + echo "${VMID:-}" > /tmp/vmid + [ -z "${VMID:-}" ] && { echo "[provision] ERROR: no microvmID"; exit 1; } + echo "[provision] Microvm ${MVM} -> ${VMID}" + # 5) Wait for RUNNING + endpoint, mint token, POST /run (background-spawns the coder). + EP=""; i=0 + while [ "$i" -lt 72 ]; do + S=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'state' --output text 2>/dev/null || echo "") + EP=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'endpoint' --output text 2>/dev/null || echo "") + [ "$S" = "RUNNING" ] && [ -n "${EP}" ] && [ "${EP}" != "None" ] && break + i=$((i+1)); sleep 5 + done + [ -z "${EP}" ] || [ "${EP}" = "None" ] && { echo "[provision] ERROR: VM never reached RUNNING+endpoint (state=${S:-none})"; exit 1; } + TOKEN=$(aws lambda-microvms create-microvm-auth-token --region "${REGION}" --microvm-identifier "${VMID}" --expiration-in-minutes 60 --allowed-ports 'port=8080' 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["authToken"]["X-aws-proxy-auth"])' 2>/dev/null || echo "") + [ -z "${TOKEN}" ] && { echo "[provision] ERROR: could not mint auth token"; exit 1; } + echo "[provision] driving coder: POST /run on ${EP}" + RC=$(curl -sS -m 30 -o /tmp/run.out -w '%{http_code}' -X POST "https://${EP}/run" -H "X-aws-proxy-auth: ${TOKEN}" -H 'Content-Type: application/json' -d "${PAYLOAD}" 2>/tmp/run.err || echo "000") + echo "[provision] /run -> HTTP ${RC} $(cat /tmp/run.out 2>/dev/null | head -c 120)" + case "${RC}" in 2*) echo "[provision] coder started." ;; *) echo "[provision] ERROR: /run returned ${RC}"; exit 1 ;; esac + + # ---- 2. Wait for the coder VM to finish — GitHub is the completion bus ---- + # The coder image (baked into the SandboxTemplate) auto-runs on VM start, + # reads DF_* env, implements + tests, pushes df/issue-N, and opens the PR + # itself using the short-TTL gh-token projected into the VM. The coder is + # credential-less to the k8s API (no SA token), so it CANNOT signal via a pod + # annotation — it self-reports through GitHub. This step polls the GitHub API + # (with the workflow's own token) for a PR whose head is df/issue-N, and + # treats the head commit's dark-factory/implementation check as the verdict. + - name: await-coder + inputs: + parameters: + - name: sandbox + activeDeadlineSeconds: {{ mul .Values.coder.runTimeoutMinutes 60 }} + script: + image: {{ .Values.stepImage }} + command: [sh] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + source: | + set -eu + command -v curl >/dev/null 2>&1 || apk add --no-cache curl jq >/dev/null 2>&1 + REPO="{{`{{workflow.parameters.repo}}`}}" + BRANCH="df/issue-{{`{{workflow.parameters.issue-number}}`}}" + ITERATE_NOTE="{{`{{workflow.parameters.iterate-note}}`}}" + ITERATE_NOTE_B64="{{`{{workflow.parameters.iterate-note-b64}}`}}" + API="https://api.github.com/repos/${REPO}" + # ROUND-AWARENESS: on a FIX ROUND (iterate-note set) the branch already + # exists and its OLD commit already has dark-factory/implementation=success. + # If we accept that, the verify fan (holdout/security/devops) runs against + # the STALE pre-fix commit before the coder's new push lands (observed 45s + # race). So record the branch's starting SHA and require a DIFFERENT head + # SHA (a genuinely new commit) before we consider the coder done on a fix + # round. First-pass runs (no iterate-note) have no prior commit → accept the + # first commit that reports impl=success. + START_SHA="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/branches/${BRANCH}" 2>/dev/null | jq -r '.commit.sha // empty' || echo "")" + IS_FIX="false"; { [ -n "${ITERATE_NOTE}" ] || [ -n "${ITERATE_NOTE_B64}" ]; } && IS_FIX="true" + echo "[df-run] polling GitHub for PR head=${BRANCH} (fix-round=${IS_FIX}, start-sha=${START_SHA:-none})..." + while true; do + pr="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" \ + -H "Accept: application/vnd.github+json" \ + "${API}/pulls?head=${REPO%%/*}:${BRANCH}&state=open" 2>/dev/null || echo '[]')" + n="$(echo "${pr}" | jq -r '.[0].number // empty')" + if [ -n "${n}" ]; then + sha="$(echo "${pr}" | jq -r '.[0].head.sha')" + # On a fix round, ignore the old commit until the coder pushes a new one. + if [ "${IS_FIX}" = "true" ] && [ -n "${START_SHA}" ] && [ "${sha}" = "${START_SHA}" ]; then + echo "[df-run] fix-round: head still at start sha ${sha} — waiting for the coder's new commit..." + sleep 15; continue + fi + st="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/commits/${sha}/status" 2>/dev/null | jq -r '.statuses[] | select(.context=="dark-factory/implementation") | .state' | head -1)" + echo "[df-run] PR #${n} open (sha=${sha}) impl-status=${st:-pending}" + case "${st}" in + success) echo "[df-run] coder finished — PR #${n} @ ${sha} ready."; echo "${n}" > /tmp/pr; echo "${sha}" > /tmp/sha; exit 0 ;; + failure|error) echo "[df-run] coder reported failure."; exit 1 ;; + esac + else + echo "[df-run] no PR yet — coder still working..." + fi + sleep 15 + done + outputs: + parameters: + - name: pr-number + valueFrom: + path: /tmp/pr + default: "" + - name: head-sha + valueFrom: + path: /tmp/sha + default: "" + +{{- if .Values.holdout.enabled }} + # ---- 2b. Holdout gate (P2) — train/test separation for code ---- + # Runs on the HUB (a trusted step, NOT the Kata VM). Checks out the coder's + # PR branch, diffs it vs base, then runs evaluate.js: each hidden scenario + # gets its executable test run against the built code AND a different-family + # judge (Nova) vote (2-of-3). Gate = passRatio >= threshold. The hidden + # scenarios come from the df-holdout-* ConfigMaps mounted here — they are + # NEVER mounted into the sandbox, so the coder cannot see or edit them. + # v1 is advisory: the gate result is posted as the dark-factory/holdout commit + # status; the workflow only fails on a red gate when holdout.blocking=true. + - name: holdout-gate + inputs: + parameters: + - name: pr-number + activeDeadlineSeconds: 900 + volumes: + - name: holdout-eval + configMap: + name: df-holdout-eval + {{- range $repo := .Values.trigger.argoEvents.repositories }} + {{- range $name := $repo.names }} + - name: holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + configMap: + name: df-holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + {{- end }} + {{- end }} + script: + image: {{ .Values.holdout.evalImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: BIFROST_URL + value: {{ .Values.holdout.bifrostUrl | quote }} + - name: JUDGE_MODEL + value: {{ .Values.holdout.judgeModel | quote }} + - name: JUDGE_RUNS + value: {{ .Values.holdout.judgeRuns | quote }} + - name: JUDGE_QUORUM + value: {{ .Values.holdout.judgeQuorum | quote }} + - name: THRESHOLD + value: {{ .Values.holdout.threshold | quote }} + volumeMounts: + - name: holdout-eval + mountPath: /holdout/evaluate.js + subPath: evaluate.js + {{- range $repo := .Values.trigger.argoEvents.repositories }} + {{- range $name := $repo.names }} + - name: holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + mountPath: /holdout/{{ printf "%s-%s" $repo.owner $name }} + {{- end }} + {{- end }} + source: | + set -eu + REPO="{{`{{workflow.parameters.repo}}`}}" + BRANCH="df/issue-{{`{{workflow.parameters.issue-number}}`}}" + BASE="{{`{{workflow.parameters.base-branch}}`}}" + SLUG="$(echo "${REPO}" | tr '/' '-')" + API="https://api.github.com/repos/${REPO}" + WORK=/tmp/holdout-work + rm -rf "${WORK}"; mkdir -p "${WORK}" + echo "[holdout] cloning ${REPO}@${BRANCH} for evaluation..." + git clone --quiet --branch "${BRANCH}" \ + "https://x-access-token:${GH_TOKEN}@github.com/${REPO}.git" "${WORK}/repo" + cd "${WORK}/repo" + # Diff vs base — use the GitHub COMPARE API (authoritative changed-file + # patch), NOT a local git diff. A branch clone shares no reliable merge-base + # with a shallow base fetch, so `origin/BASE...HEAD` yields the WHOLE file as + # "added" (observed: a sum-only PR's diff contained every pre-existing + # function → every scenario's appliesWhen matched → wrong scenarios graded → + # false holdout failure). The compare API returns only the real per-file + # patch hunks, so appliesWhen keys on the ACTUAL change. (Same fix as + # detect-deployable.) Concatenate every file's patch into /tmp/diff.patch. + PR_NUM="{{`{{inputs.parameters.pr-number}}`}}" + curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/pulls/${PR_NUM}/files?per_page=100" 2>/dev/null \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); files=d if isinstance(d,list) else []; open("/tmp/diff.patch","w").write("\n".join("diff --git a/%s b/%s\n%s"%(f.get("filename",""),f.get("filename",""),f.get("patch","")) for f in files))' 2>/dev/null || echo "" > /tmp/diff.patch + # Fallback: if the API path produced nothing, fall back to a local diff. + [ -s /tmp/diff.patch ] || { git fetch --quiet --depth 1 origin "${BASE}" 2>/dev/null || true; git diff "origin/${BASE}...HEAD" > /tmp/diff.patch 2>/dev/null || echo "" > /tmp/diff.patch; } + echo "[holdout] diff.patch $(wc -l < /tmp/diff.patch) lines (via compare API)" + # Build/install so executable tests can require the module. + if [ -f package.json ]; then npm install --no-audit --no-fund >/dev/null 2>&1 || true; fi + SHA="$(git rev-parse HEAD)" + + SCEN="/holdout/${SLUG}/scenarios.json" + if [ ! -f "${SCEN}" ]; then + echo "[holdout] no hidden scenarios for ${SLUG} — skipping gate (advisory)." + exit 0 + fi + + echo "[holdout] evaluating against hidden scenarios..." + set +e + REPO_DIR="${WORK}/repo" DIFF=/tmp/diff.patch SCENARIOS="${SCEN}" \ + OUT=/tmp/holdout-result.json node /holdout/evaluate.js + GATE=$? + set -e + + RATIO="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(Math.round(r.ratio*100))' 2>/dev/null || echo 0)" + PASSED="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.passed+"/"+r.total)' 2>/dev/null || echo '?/?')" + TOTAL="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.total)' 2>/dev/null || echo 0)" + SKIPPED="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.skipped||0)' 2>/dev/null || echo 0)" + # Distinguish the three outcomes so the PR reads honestly: + # total=0 → no hidden scenario matched this change (e.g. a Terraform PR + # vs JS-only scenarios) → NOT APPLICABLE, green as n/a. + # gate ok → passed. gate fail → below threshold. + if [ "${TOTAL}" = "0" ]; then + STATE=success; DESC="not applicable — no hidden scenarios match this change (${SKIPPED} skipped)" + elif [ "${GATE}" -eq 0 ]; then + STATE=success; DESC="holdout ${PASSED} (${RATIO}%) — gate passed" + else + STATE=failure; DESC="holdout ${PASSED} (${RATIO}%) — below threshold" + fi + + # Post the holdout verdict as a commit status on the PR head SHA. The + # coder image has no curl, so use node's https (always present). Retry + # transient failures so a blip doesn't drop the verdict. + GH_TOKEN="${GH_TOKEN}" REPO="${REPO}" SHA="${SHA}" STATE="${STATE}" DESC="${DESC}" node -e ' + const https=require("https"); + const body=JSON.stringify({state:process.env.STATE,context:"dark-factory/holdout",description:process.env.DESC}); + let n=0; + (function post(){ + const req=https.request({host:"api.github.com",method:"POST",path:"/repos/"+process.env.REPO+"/statuses/"+process.env.SHA, + headers:{"User-Agent":"dark-factory-holdout","Authorization":"Bearer "+process.env.GH_TOKEN,"Accept":"application/vnd.github+json","Content-Type":"application/json","Content-Length":Buffer.byteLength(body)}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{if(r.statusCode>=300&&r.statusCode<500){console.error("status post "+r.statusCode+": "+b.slice(0,120))}else if(r.statusCode>=500&&++n<4){setTimeout(post,500*n);return} console.log("[holdout] posted dark-factory/holdout="+process.env.STATE)});}); + req.on("error",e=>{if(++n<4){setTimeout(post,500*n)}else{console.error("status post error: "+e.message)}}); + req.write(body);req.end(); + })(); + ' || true + + {{- if .Values.holdout.blocking }} + echo "[holdout] blocking mode — gate result gates the workflow." + exit ${GATE} + {{- else }} + echo "[holdout] advisory mode — gate result reported, workflow continues." + exit 0 + {{- end }} + outputs: + parameters: + - name: ratio + valueFrom: + path: /tmp/holdout-result.json + default: "{}" +{{- end }} + +{{- if .Values.review.enabled }} + # ---- 2c. devops-gate (P3, step 1) — wait for the AWS DevOps Agent to clear ---- + # The AWS DevOps Agent release-readiness review is its own GitHub App that + # auto-reviews every PR and posts a check-run / commit status (its native model + # — there is NO headless code-review API). This step polls the PR head for that + # check and emits cleared=true|false, which gates the Security Agent step so + # DevOps reviews FIRST, Security SECOND (docs §6.2). + # + # Mode (devopsAgent.gate): + # check → wait for the DevOps Agent's own check-run (context matches + # devopsAgent.checkContext) to conclude success/neutral. This is the + # real GitHub-App path (needs the one-time console repo connect). + # label → fallback: wait for the coder-applied handoffLabel (used only if + # you drive DevOps via the coding-agent plugin instead of the App). + # If DevOps never reports within the window, cleared=false → Security is skipped + # and the sticky status shows DevOps not-run (NEVER a fake pass). Advisory: this + # step itself never fails the run. + - name: devops-gate + inputs: + parameters: + - name: pr-number + activeDeadlineSeconds: {{ add .Values.devopsAgent.waitSeconds 60 }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: PR + value: "{{`{{inputs.parameters.pr-number}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: GATE_MODE + value: {{ .Values.devopsAgent.gate | quote }} + - name: LABEL + value: {{ .Values.review.handoffLabel | quote }} + - name: CHECK_CONTEXT + value: {{ .Values.devopsAgent.checkContext | quote }} + - name: WAIT_SECONDS + value: {{ .Values.devopsAgent.waitSeconds | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/comment.js + subPath: comment.js + source: | + set -eu + echo "[devops-gate] mode=${GATE_MODE} PR #${PR} — waiting for AWS DevOps Agent..." + DEADLINE=$(( $(date +%s) + WAIT_SECONDS )) + CLEARED=false + while [ "$(date +%s)" -lt "$DEADLINE" ]; do + if [ "$GATE_MODE" = "label" ]; then + # Fallback: coder-applied handoff label present on the PR. + R="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" PR="$PR" LABEL="$LABEL" node -e ' + const https=require("https");const {GH_TOKEN,REPO,PR,LABEL}=process.env; + https.get({host:"api.github.com",path:"/repos/"+REPO+"/issues/"+PR+"/labels",headers:{"User-Agent":"df-devops-gate","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{const ls=JSON.parse(b).map(x=>x.name);console.log(ls.includes(LABEL)?"cleared":"waiting")}catch(e){console.log("waiting")}})}).on("error",()=>console.log("waiting"));' 2>/dev/null || echo waiting)" + else + # Default: the DevOps Agent GitHub App's check-run on the PR head SHA. + # cleared iff a matching context concluded success/neutral; "blocked" + # if it concluded failure (BLOCK) — we stop and leave cleared=false. + R="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" BRANCH="$BRANCH" CHECK_CONTEXT="$CHECK_CONTEXT" node -e ' + const https=require("https");const {GH_TOKEN,REPO,BRANCH,CHECK_CONTEXT}=process.env; + const g=(p)=>new Promise((res)=>{https.get({host:"api.github.com",path:p,headers:{"User-Agent":"df-devops-gate","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}},r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{res(JSON.parse(b))}catch(e){res(null)}})}).on("error",()=>res(null))}); + (async()=>{ + const pr=await g("/repos/"+REPO+"/pulls?head="+REPO.split("/")[0]+":"+BRANCH+"&state=open"); + if(!pr||!pr[0]){return console.log("waiting")} + const sha=pr[0].head.sha; + const re=new RegExp(CHECK_CONTEXT,"i"); + // check-runs API + const cr=await g("/repos/"+REPO+"/commits/"+sha+"/check-runs"); + const runs=(cr&&cr.check_runs)||[]; + const m=runs.filter(x=>re.test(x.name)); + if(m.some(x=>x.status==="completed"&&["success","neutral"].includes(x.conclusion)))return console.log("cleared"); + if(m.some(x=>x.status==="completed"&&["failure","action_required","cancelled","timed_out"].includes(x.conclusion)))return console.log("blocked"); + // legacy commit statuses fallback + const st=await g("/repos/"+REPO+"/commits/"+sha+"/status"); + const ss=((st&&st.statuses)||[]).filter(x=>re.test(x.context)); + if(ss.some(x=>x.state==="success"))return console.log("cleared"); + if(ss.some(x=>["failure","error"].includes(x.state)))return console.log("blocked"); + console.log("waiting"); + })();' 2>/dev/null || echo waiting)" + fi + case "$R" in + cleared) CLEARED=true; echo "[devops-gate] AWS DevOps Agent CLEARED"; break ;; + blocked) CLEARED=false; echo "[devops-gate] AWS DevOps Agent returned BLOCK — Security stays gated"; break ;; + *) echo "[devops-gate] waiting for AWS DevOps Agent review..."; sleep 20 ;; + esac + done + echo "$CLEARED" > /tmp/cleared + echo "[devops-gate] cleared=${CLEARED}" + + # NOTE: the DevOps verdict is NO LONGER relayed as its own PR comment. + # To avoid mixed/duplicate signals, the pipeline posts a SINGLE + # consolidated review (status.js → dark-factory:verdict-review) that + # mirrors the source-of-truth checks (Build/Holdout/Security/DevOps). + # The DevOps Agent's own commit status + the App bot's review remain + # the authoritative DevOps signal; `cleared` still gates the flow below. + outputs: + parameters: + - name: cleared + valueFrom: + path: /tmp/cleared + default: "false" + + # ---- 2d. security-agent (P3, step 2) — REAL AWS Security Agent, headless ---- + # Runs SECOND, only after DevOps cleared (label present). A trusted hub-side + # step that clones df/issue-N read-only, stages {source archive, unified diff} + # SINGLE SECURITY SIGNAL = THE REAL AWS SECURITY AGENT BOT. + # We do NOT run a second headless scan (that redundant path disagreed with the + # bot — reported "no findings" while aws-security-agent[bot] flagged real issues + # — and produced a false LGTM). Instead this step WAITS for the GitHub App bot's + # own review on the PR and MIRRORS its verdict into the dark-factory/security + # commit status (findings -> failure -> merge blocked; clean -> success). Because + # the consolidated `status` step depends on this step, the pipeline now naturally + # waits for the real bot before posting its verdict. Node-only → runs on reviewImage. + - name: security-agent + activeDeadlineSeconds: {{ add .Values.securityAgent.pollTimeoutSeconds 300 }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [node] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + # Findings at/above this severity fail the status (block the merge). The bot + # doesn't expose per-severity counts uniformly, so ANY finding fails unless + # BLOCK_LEVEL=none (then findings are advisory). Default medium. + - name: BLOCK_LEVEL + value: {{ .Values.securityAgent.blockLevel | quote }} + - name: POLL_TIMEOUT + value: {{ .Values.securityAgent.pollTimeoutSeconds | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/security-wait.js + subPath: security-wait.js + source: | + require("/scripts/security-wait.js"); +{{- end }} + +{{- if .Values.deployTest.enabled }} + # ---- 2d. detect-deployable (P4) — does the diff touch deployable artifacts? ---- + # Emits `deployable = true|false`, which gates the expensive deploy-test step. + # Uses the GitHub compare API (authoritative changed-file list) rather than a + # local git diff — a depth-1 clone + depth-1 base fetch share no merge-base, so + # `origin/BASE...HEAD` yields nothing (observed: empty file list → false). + - name: detect-deployable + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: BASE + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: K8S_PATTERNS + value: {{ .Values.deployTest.k8sPatterns | quote }} + - name: TF_PATTERNS + value: {{ .Values.deployTest.terraformPatterns | quote }} + source: | + set -eu + FILES="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" BASE="$BASE" BRANCH="$BRANCH" node -e ' + const https=require("https"); + const {GH_TOKEN,REPO,BASE,BRANCH}=process.env; + https.get({host:"api.github.com",path:"/repos/"+REPO+"/compare/"+BASE+"..."+BRANCH,headers:{"User-Agent":"df-detect","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{const j=JSON.parse(b);(j.files||[]).forEach(f=>console.log(f.filename));}catch(e){process.exit(0);}});}).on("error",()=>process.exit(0)); + ')" + echo "[detect] changed files:"; echo "$FILES" | sed 's/^/ /' + # Classify: k8s takes precedence over terraform when both are present. + if echo "$FILES" | grep -qE "$K8S_PATTERNS"; then KIND=k8s + elif echo "$FILES" | grep -qE "$TF_PATTERNS"; then KIND=terraform + else KIND=none; fi + echo "$KIND" > /tmp/kind + echo "[detect] kind=$KIND" + outputs: + parameters: + - name: kind + valueFrom: + path: /tmp/kind + default: "none" + + # ---- 2e. deploy-test (P4) — content-aware; THE ONLY step with K8s access ---- + # Trusted hub step. Validates the change with the RIGHT tool for its kind: + # kind=k8s → ephemeral namespace apply + wait Ready + teardown (trap). + # kind=terraform → terraform init -backend=false + validate (+ fmt check). + # Validation only — NO AWS creds, NO apply (no real infra). + # Posts the dark-factory/deploy-test commit status AND a marker PR comment with + # the details. Advisory in v1 (deployTest.blocking=false). The untrusted coder + # never has K8s access — it only produces the artifacts; this step runs them. + - name: deploy-test + inputs: + parameters: + - name: kind + activeDeadlineSeconds: {{ .Values.deployTest.timeoutSeconds }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.deployTest.image }} + command: [bash] + volumeMounts: + - name: review-script + mountPath: /scripts/comment.js + subPath: comment.js + - name: review-script + mountPath: /scripts/deploy-test.sh + subPath: deploy-test.sh + env: + - name: WF_NAME + value: "{{`{{workflow.name}}`}}" + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: KIND + value: "{{`{{inputs.parameters.kind}}`}}" + - name: MANIFEST_PATH + value: {{ .Values.deployTest.manifestPath | quote }} + - name: TF_PATH + value: {{ .Values.deployTest.terraformPath | quote }} + - name: READY_TIMEOUT + value: {{ .Values.deployTest.readyTimeoutSeconds | quote }} + - name: BLOCKING + value: {{ .Values.deployTest.blocking | quote }} + source: | + set -eu + # All logic lives in review/deploy-test.sh (ConfigMap file) — keeping it out + # of this YAML block scalar avoids the shell-quoting hazards that broke + # inline multi-line markdown. Kind-driven so it generalizes to new profiles. + bash /scripts/deploy-test.sh +{{- end }} + + # ---- 3. Sticky status — rewrite the PR body from the live verdicts ---- + # Runs AFTER every verify step (its DAG deps). The coder wrote the PR body at + # PR-open time, before verification ran, so its holdout/security/devops lines + # are placeholders ("pending"). This step reads the authoritative + # dark-factory/* commit STATUSES from GitHub (the source of truth the verify + # steps posted) and rewrites the PR body in place with the real verdicts — the + # "one live sticky status" from the design (README §7). Idempotent: the body + # between the markers is fully regenerated each run. + - name: sticky-status + inputs: + parameters: + - name: phase + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + # For the AUTO-FIX loop (status.js): on a ❌ verdict from the real agents, + # collect their findings and submit a bounded df-run revision (iterate-note + # = the findings) so the coder auto-revises — no human paraphrasing needed. + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: BASE_BRANCH + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: TRIGGER_LABEL + value: "{{`{{workflow.parameters.trigger-label}}`}}" + - name: ARGO_NAMESPACE + value: {{ .Values.argo.namespace | quote }} + - name: AUTO_FIX_FINDINGS + value: {{ .Values.review.autoFixFindings | default false | quote }} + - name: MAX_ITERATIONS + value: {{ .Values.iterate.maxIterations | default 3 | quote }} +{{- if .Values.devopsAgent.enabled }} + # Real AWS DevOps Agent posts a check-run (not a commit status) named this; + # status.js renders the DevOps row from the check-run when present. + - name: DEVOPS_CHECK + value: {{ .Values.devopsAgent.checkRunName | quote }} +{{- end }} +{{- if and .Values.securityAgent.app .Values.securityAgent.app.enabled }} + # Real AWS Security Agent GitHub App posts its own check/inline review; + # status.js renders the Security row from it when present (else the + # headless dark-factory/security status). + - name: SECURITY_CHECK + value: {{ .Values.securityAgent.app.checkRunName | quote }} +{{- end }} +{{- if .Values.postVerdictReview }} + # Post ONE consolidated verdict review (Security + DevOps results) on the + # PR so both agents' verdicts are ALWAYS visible in the Reviews section — + # the agent Apps review autonomously + inconsistently and cannot be added + # via the requested_reviewers API (verified no-op). status.js posts it once, + # when verification is terminal (idempotent via a hidden marker). + - name: POST_VERDICT_REVIEW + value: "true" +{{- end }} + # Holdout is a train/test QUALITY signal, advisory by default — it does NOT + # gate the merge verdict unless holdout.blocking=true. status.js reads this + # to decide whether a red holdout flips the consolidated verdict. + - name: HOLDOUT_BLOCKING + value: {{ .Values.holdout.blocking | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/status.js + subPath: status.js + source: | + set -eu + echo "[df-run] phase={{`{{inputs.parameters.phase}}`}} — updating PR body from live verdicts (${REPO} ${BRANCH})" + node /scripts/status.js + + # ---- suspend/resume the Lambda MicroVM directly (scale-to-zero) ---- + # Calls suspend-microvm / resume-microvm on the VM the provision step created (id read + # from the Microvm CR by stable name mvm-). Direct AWS call — NO Sandbox/ + # operatingMode/lifecycle-controller dependency (this template owns the VM outright). + # Advisory: never fails the run (suspend is a cost optimization, not correctness). + - name: set-microvm-power + inputs: + parameters: + - name: action # suspend | resume + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + source: | + set -eu + ACTION="{{`{{inputs.parameters.action}}`}}" + REGION="{{ .Values.microvm.region }}" + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -z "${VMID}" ] && { echo "[power] no microvmID for ${MVM} — skipping (advisory)"; exit 0; } + echo "[power] ${ACTION}-microvm ${VMID}" + aws lambda-microvms "${ACTION}-microvm" --region "${REGION}" --microvm-identifier "${VMID}" 2>&1 || echo "[power] ${ACTION} skipped/failed (advisory)" + echo "[power] done." + activeDeadlineSeconds: 180 + + # ---- onExit: KEEP the suspended VM (terminated only at merge) ---- + # Flow D lifecycle: the VM is SUSPENDED (by suspend-microvm above) and must SURVIVE + # the review→fix loop so a fix round can RESUME the SAME VM. The Microvm CR is + # therefore NOT deleted here — it is deleted at merge by df-merge-teardown, which + # triggers the controller's TerminateMicrovm. This onExit is a no-op guard: if the + # run FAILED before ever creating/suspending a VM, there's nothing to keep, but we + # still never delete on the normal path (the suspended VM is the whole point). + - name: teardown + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + source: | + set -eu + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + echo "[teardown] lambda: KEEPING Microvm/${MVM} (stays SUSPENDED until merge → df-merge-teardown terminates it)." + exit 0 diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 9b9b4784..863f60ed 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -257,17 +257,25 @@ iterate: enabled: true maxIterations: 3 -# ── Flow D — MicroVM suspend/resume ────────────────────────────────────────── -# DEFAULT false — and it should STAY false. Suspend-after-code is NOT a df-run DAG -# step anymore: because ONE df-run template serves BOTH substrates on a shared hub, a -# rendered microvm-suspend step could only be runtime-SKIPPED on a Kata run, leaving a -# confusing (if harmless) Skipped node in every Kata graph. Instead the Flow D bridge -# (agent-sandbox-lambda, bridge.sh) suspends its OWN MicroVM after the coder pushes the -# PR — lambda-only by construction, so the pipeline graph is substrate-agnostic and the -# Kata DAG contains ZERO MicroVM steps. Leave this false; the bridge owns suspend/resume. -microvmSuspend: - enabled: false - image: alpine/k8s:1.31.0 # only patches Sandbox.operatingMode via kubectl (no aws-cli) +# ── Flow D — Lambda MicroVM substrate (df-run-lambda ONLY) ─────────────────── +# The Kata df-run template has ZERO MicroVM logic (it's byte-identical to the certified +# Kata pipeline). Flow D runs in a SEPARATE WorkflowTemplate, df-run-lambda, which +# provisions the Lambda MicroVM directly (Microvm CR + /run), suspends it during review +# (scale-to-zero), resumes the SAME VM on a fix round, and terminates it at merge. These +# values feed that template; they mirror the agent-sandbox-lambda chart's microvm.* keys. +microvm: + enabled: true + region: us-west-2 + namespace: agent-sandbox-system # where the Microvm CR + platform MicrovmSandbox live + # aws-cli v2 (has the lambda-microvms verbs); kubectl is fetched at step start. + stepImage: public.ecr.aws/aws-cli/aws-cli:latest + image: + name: coder # the platform MicrovmSandbox name (image handoff) + defaults: + maxIdleDurationSeconds: 1800 # RUNNING-idle backstop (we drive /run immediately) + # Lambda caps SUSPEND at 8h (28800s). Use the max so the VM survives the full + # review→fix window; df-merge-teardown terminates it explicitly at merge. + suspendedDurationSeconds: 28800 # ── Language / stack support ───────────────────────────────────────────────── # There is NO per-language profile config here — deliberately. Language support is From d73ef79a06d226e1b209adb1c861af13f38fb7e3 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 16:48:15 -0400 Subject: [PATCH 64/67] =?UTF-8?q?feat(flow-d):=20route=20darkfactory-lambd?= =?UTF-8?q?a=20=E2=86=92=20df-run-lambda=20+=20workflow=20IAM/RBAC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Sensor: split issue-labeled into issue-labeled-kata (→ df-run) and issue-labeled-lambda (→ df-run-lambda) with two triggers. Kata path unchanged. - iterate.js: fix rounds submit df-run-lambda for Lambda issues (resumes the same suspended VM), df-run for Kata. Dedup name df-run-lambda-. - RBAC: dark-factory-workflow SA gets microvms + secrets + microvmsandboxes (kro.run) in agent-sandbox-system, gated on microvm.enabled — for the provision/terminate kubectl steps. - IAM: PodIdentityAssociation binds dark-factory-workflow → the existing hub-ack-lambdamicrovms-controller role (USER-APPROVED reuse; additive, no policy change) so the steps can call aws lambda-microvms get/suspend/resume/terminate + create-auth-token. - values: microvm.podIdentity{clusterName:hub, accountId} + workflowServiceAccount. --- .../charts/dark-factory/scripts/iterate.js | 11 +- .../dark-factory/templates/10-rbac.yaml | 37 ++++++ .../dark-factory/templates/42-sensor.yaml | 119 ++++++++++++++---- gitops/addons/charts/dark-factory/values.yaml | 9 ++ 4 files changed, 148 insertions(+), 28 deletions(-) diff --git a/gitops/addons/charts/dark-factory/scripts/iterate.js b/gitops/addons/charts/dark-factory/scripts/iterate.js index de7a4ca2..63d9bfd9 100644 --- a/gitops/addons/charts/dark-factory/scripts/iterate.js +++ b/gitops/addons/charts/dark-factory/scripts/iterate.js @@ -120,12 +120,19 @@ async function main() { await gh("POST", `/repos/${REPO}/issues/${PR}/labels`, { labels: [`${ITER_LABEL_PREFIX}${next}`] }).catch(() => {}); console.log(`[df-iterate] revision ${next}/${MAX_ITERATIONS} for issue #${issueNumber} (PR #${PR})`); + // Substrate-routed template: Lambda fix rounds run the MicroVM-native df-run-lambda + // (resumes the SAME suspended VM); Kata fix rounds run the certified df-run. Keyed on + // the originating issue's label (resolved above as triggerLabel). + const isLambda = triggerLabel === "darkfactory-lambda"; + const wfTemplate = isLambda ? "df-run-lambda" : "df-run"; + const wfName = isLambda ? `df-run-lambda-${issueNumber}-i${next}` : `df-run-${issueNumber}-i${next}`; + console.log(`[df-iterate] substrate=${triggerLabel} → template=${wfTemplate}`); const wf = { apiVersion: "argoproj.io/v1alpha1", kind: "Workflow", // Dedup per issue+round so a duplicate comment webhook is a no-op. - metadata: { name: `df-run-${issueNumber}-i${next}`, namespace: ARGO_NAMESPACE }, + metadata: { name: wfName, namespace: ARGO_NAMESPACE }, spec: { - workflowTemplateRef: { name: "df-run" }, + workflowTemplateRef: { name: wfTemplate }, arguments: { parameters: [ { name: "issue-id", value: `${issueNumber}` }, // no id in this payload; number is unique enough for the mutex/claim { name: "issue-number", value: `${issueNumber}` }, diff --git a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml index 56724217..5d0157bd 100644 --- a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml +++ b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml @@ -84,6 +84,21 @@ rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "create", "update", "patch"] +{{- if .Values.microvm.enabled }} + # Flow D (df-run-lambda): provision-microvm creates the Microvm CR + its runHookPayload + # Secret directly (no SandboxClaim); set-microvm-power reads the CR for the id; + # df-merge-teardown deletes the CR at merge. Least-privilege, this namespace only. + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvms"] + verbs: ["get", "list", "watch", "create", "delete"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update", "patch", "delete"] + # Read the platform image handoff (imageARN + execRoleARN) built once by KRO/ACK. + - apiGroups: ["kro.run"] + resources: ["microvmsandboxes"] + verbs: ["get", "list", "watch"] +{{- end }} --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding @@ -100,6 +115,28 @@ roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: dark-factory-workflow-sandbox +{{- if .Values.microvm.enabled }} +--- +# Flow D: bind the workflow SA to the lambda-microvms role via EKS Pod Identity, so the +# df-run-lambda steps (provision/power/terminate) can call aws lambda-microvms +# (get/suspend/resume/terminate-microvm + create-microvm-auth-token). Reuses the SAME +# role the bridge/lifecycle controllers already use (user-approved) — additive, +# no IAM policy change. The ACK eks controller reconciles this into a real EKS association. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-dark-factory-workflow + namespace: {{ .Values.warmPool.namespace }} + labels: + {{- include "dark-factory.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ .Values.argo.namespace }} + serviceAccount: {{ .Values.microvm.workflowServiceAccount | default "dark-factory-workflow" }} + roleARN: "arn:aws:iam::{{ .Values.microvm.podIdentity.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +{{- end }} {{- if .Values.deployTest.enabled }} --- # P4 deploy-test: the ONLY cluster-scoped grant. Lets the workflow create/delete diff --git a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml index d01202bc..0e569ff8 100644 --- a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml +++ b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml @@ -59,16 +59,16 @@ spec: template: serviceAccountName: dark-factory-sensor dependencies: - - name: issue-labeled + # TWO label dependencies — one per substrate — so each fires a DIFFERENT + # WorkflowTemplate (Argo Events trigger conditions can only reference dependency + # NAMES, not field values, so routing by label ⇒ separate dependencies + triggers): + # issue-labeled-kata → df-run (certified Kata pipeline) + # issue-labeled-lambda → df-run-lambda (Flow D MicroVM-native pipeline) + - name: issue-labeled-kata eventSourceName: dark-factory-github eventName: dark-factory filters: data: - # Fire when EITHER dark-factory label is added: - # dark-factory → Kata micro-VM substrate (Flow B, default) - # darkfactory-lambda → Lambda MicroVM substrate (Flow D) - # The label chosen is passed to df-run as the `substrate` param, which - # branches the claim step. Everything else in the pipeline is identical. - path: headers.X-Github-Event type: string value: ["issues"] @@ -77,7 +77,21 @@ spec: value: ["labeled"] - path: body.label.name type: string - value: ["dark-factory", "darkfactory-lambda"] + value: ["dark-factory"] + - name: issue-labeled-lambda + eventSourceName: dark-factory-github + eventName: dark-factory + filters: + data: + - path: headers.X-Github-Event + type: string + value: ["issues"] + - path: body.action + type: string + value: ["labeled"] + - path: body.label.name + type: string + value: ["darkfactory-lambda"] - name: pr-approved eventSourceName: dark-factory-github eventName: dark-factory @@ -133,10 +147,10 @@ spec: path: body.comment.body {{- end }} triggers: + # ---- KATA: dark-factory label → df-run (certified pipeline) ---- - template: name: submit-df-run - # With multiple dependencies, each trigger must name the one it fires on. - conditions: "issue-labeled" + conditions: "issue-labeled-kata" argoWorkflow: operation: submit source: @@ -150,8 +164,7 @@ spec: # delivery spawned a competing df-run that force-pushed its own # commit and split the commit statuses across SHAs. With a fixed # name, the 2nd+ submit collides (AlreadyExists) and is a harmless - # no-op — one issue = one in-flight run. The name is overwritten by - # the issue-id parameter below; this is a fallback if that is empty. + # no-op — one issue = one in-flight run. name: df-run-pending namespace: {{ .Values.argo.namespace }} spec: @@ -165,45 +178,99 @@ spec: - name: issue-title - name: issue-body - name: base-branch - # The label that fired: "dark-factory" or "darkfactory-lambda". - # df-run maps this to the substrate (kata vs lambda-microvm). - name: trigger-label - # Map GitHub webhook fields → workflow parameters. parameters: - # Deterministic workflow name = df-run- (the dedup key). - # sprig `int64` renders the large JSON number as a plain integer — - # without it Go templating emits scientific notation (4.88e+09), - # which is an invalid RFC-1123 metadata.name. - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataTemplate: "df-run-{{ `{{ .Input.body.issue.id | int64 }}` }}" dest: metadata.name - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata + dataKey: body.issue.id + dest: spec.arguments.parameters.0.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.issue.number + dest: spec.arguments.parameters.1.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.repository.full_name + dest: spec.arguments.parameters.2.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.issue.title + dest: spec.arguments.parameters.3.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.issue.body + dest: spec.arguments.parameters.4.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.repository.default_branch + dest: spec.arguments.parameters.5.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.label.name + dest: spec.arguments.parameters.6.value + + # ---- LAMBDA: darkfactory-lambda label → df-run-lambda (Flow D MicroVM) ---- + - template: + name: submit-df-run-lambda + conditions: "issue-labeled-lambda" + argoWorkflow: + operation: submit + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + name: df-run-lambda-pending + namespace: {{ .Values.argo.namespace }} + spec: + workflowTemplateRef: + name: df-run-lambda + arguments: + parameters: + - name: issue-id + - name: issue-number + - name: repo + - name: issue-title + - name: issue-body + - name: base-branch + - name: trigger-label + parameters: + # Dedup key = df-run-lambda- (stable across first run + fix + # rounds → one workflow name per issue, matching the mvm- CR). + - src: + dependencyName: issue-labeled-lambda + dataTemplate: "df-run-lambda-{{ `{{ .Input.body.issue.number }}` }}" + dest: metadata.name + - src: + dependencyName: issue-labeled-lambda dataKey: body.issue.id dest: spec.arguments.parameters.0.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.issue.number dest: spec.arguments.parameters.1.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.repository.full_name dest: spec.arguments.parameters.2.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.issue.title dest: spec.arguments.parameters.3.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.issue.body dest: spec.arguments.parameters.4.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.repository.default_branch dest: spec.arguments.parameters.5.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-lambda dataKey: body.label.name dest: spec.arguments.parameters.6.value diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 863f60ed..2bd029ac 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -276,6 +276,15 @@ microvm: # Lambda caps SUSPEND at 8h (28800s). Use the max so the VM survives the full # review→fix window; df-merge-teardown terminates it explicitly at merge. suspendedDurationSeconds: 28800 + # Pod Identity: the df-run-lambda steps run as the dark-factory-workflow SA (argo ns) + # and call aws lambda-microvms (get/suspend/resume/terminate/create-auth-token). Bind + # that SA to the SAME lambda-microvms role the bridge/lifecycle use (least-privilege; + # additive; no existing role policy changes). ACK eks PodIdentityAssociation. + podIdentity: + clusterName: hub + accountId: "940019131157" + # role: -ack-lambdamicrovms-controller + workflowServiceAccount: dark-factory-workflow # ── Language / stack support ───────────────────────────────────────────────── # There is NO per-language profile config here — deliberately. Language support is From c69b3acad83247fffb5fc0764ddcc1fd89fef9fc Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 17:21:51 -0400 Subject: [PATCH 65/67] docs(flow-d): update benchmark + diagrams for separate df-run-lambda Reflect the MicroVM-native architecture: two separate WorkflowTemplates (df-run Kata / df-run-lambda), provision-microvm (no bridge/claim/warm-pool), explicit suspend-microvm step, warm-resume with recreate-fallback, one-VM-per-issue naming. Updated timing (native path ~2.5min to PR, faster than old bridge ~3.7min), gotchas table (added suspend-sticks/8h-timeout/note-in-payload/resume-flakiness/ flap fixes; stale CR status note), and all Mermaid diagrams. --- docs/dark-factory/SUBSTRATE-BENCHMARK.md | 169 ++++++++++++----------- docs/dark-factory/SUBSTRATE-DIAGRAMS.md | 79 ++++++----- 2 files changed, 137 insertions(+), 111 deletions(-) diff --git a/docs/dark-factory/SUBSTRATE-BENCHMARK.md b/docs/dark-factory/SUBSTRATE-BENCHMARK.md index c8937454..196f34cb 100644 --- a/docs/dark-factory/SUBSTRATE-BENCHMARK.md +++ b/docs/dark-factory/SUBSTRATE-BENCHMARK.md @@ -18,10 +18,12 @@ difference is *where the coder executes* and *how it's provisioned*. | | Kata micro-VM (Flow B) | Lambda MicroVM (Flow D) | | --- | --- | --- | -| **Provisioning** | pre-warmed pool → **instant claim** | **RunMicrovm cold-start per session** (~90–120s) | -| **Time to first PR** (from label) | ~**2 min** | ~**3.5 min** | +| **Workflow** | `df-run` (certified) | `df-run-lambda` (separate, MicroVM-native) | +| **Provisioning** | pre-warmed pool → **instant claim** | **RunMicrovm cold-start per session** (~90s) | +| **Time to first PR** (from label) | ~**2 min** | ~**2.5 min** | | **LLM path** | Bifrost gateway (in-cluster) + Langfuse traces | **Bedrock-direct** (exec role) — no cluster network | -| **Scale-to-zero when idle** | ❌ node pool runs continuously | ✅ **suspend-to-zero**, resume on demand | +| **Scale-to-zero when idle** | ❌ node pool runs continuously | ✅ **suspend-to-zero** between PR and merge | +| **Fix-round mechanic** | fresh pod each round | **resume the SAME suspended VM** (warm); recreate if the pre-GA resume fails | | **Infra to manage** | nested-virt node group (Karpenter/MNG) | none — serverless MicroVMs | | **Observability** | native `kubectl logs` | custom `/logs` HTTP endpoint (no runtime CloudWatch) | | **Maturity** | production-ready today | pre-GA (preview) — pilot-grade | @@ -29,40 +31,37 @@ difference is *where the coder executes* and *how it's provisioned*. **Bottom line:** at small scale the two feel equivalent (the LLM coding step ~2–4 min and the external review agents ~8–15 min dominate total time on *both*). The Lambda substrate's advantage -is **not latency** — it's **operational + economic**: no node pool to run, and suspend-to-zero per -idle session. Its cost is **maturity** (pre-GA control plane) and the extra plumbing below. +is **not latency** — it's **operational + economic**: no node pool to run, and suspend-to-zero +between the PR and the human's review/merge. Its cost is **maturity** (pre-GA control plane — resume +from suspend is occasionally flaky, mitigated by the recreate-fallback) and the extra plumbing below. --- -## Benchmarked run (identical issue, fired in parallel) - -Issue (both): *"Add an S3 bucket for log archives + an EC2 IAM role to write to it (Terraform)."* -Fired simultaneously — `#117` labeled `dark-factory` (Kata), `#118` labeled `darkfactory-lambda` (Lambda). +## Benchmarked run (identical issue, per substrate) ### Time to first PR (from label → PR opened) -| Substrate | Issue | PR | Elapsed | -| --- | --- | --- | --- | -| Kata | #117 | #119 | ~**2 min** (16:32:54 → 16:34:58) | -| Lambda | #118 | #120 | ~**3.7 min** (16:32:54 → 16:36:38) | - -**Δ ≈ 100s** — the MicroVM cold-start (`RunMicrovm` → RUNNING → hook-server ready → bridge drives -`/run`) vs Kata's pre-warmed pod claim. This gap is the substrate's provisioning cost; everything -after (clone → LLM agent → push) is identical code and takes the same time. - -### Per-step workflow timing (Kata run #117) -| Step | Time | +| Substrate | Issue | PR | Elapsed | Notes | +| --- | --- | --- | --- | --- | +| Kata | #137 | — | ~**2 min** | pre-warmed pod, instant claim | +| Lambda | #135 | #136 | ~**2.5 min** (20:53:03 → 20:55:35) | native `df-run-lambda`, RunMicrovm cold-start | + +**Δ ≈ 30–90s** — the MicroVM cold-start (`RunMicrovm` → RUNNING → `/run`) vs Kata's pre-warmed pod +claim. Note the MicroVM-native `df-run-lambda` is **faster than the old bridge path** (~3.7 min): +removing the SandboxClaim/warm-pool indirection cut ~1 min. Everything after (clone → LLM → push) is +identical code and takes the same time. + +### Lifecycle timing (Lambda #135, native pipeline) +| Phase | Time | | --- | --- | -| claim (warm pod) | ~17s | -| **drive-coder** (clone→LLM→push→PR) | ~120s | -| detect-deployable | ~28s | -| holdout-gate | ~19s | -| deploy-test (terraform validate) | ~37s | -| security-agent (external) | several min | -| devops-gate (external) | several min | - -*(The Lambda run's `drive-coder` is comparable for the coding itself; it just adds the ~90s -RunMicrovm cold-start inside the claim/drive window. The two external review agents — DevOps + -Security — take ~8–15 min combined and dominate total wall-clock on BOTH substrates.)* +| provision-microvm (RunMicrovm → RUNNING → `/run` HTTP 200) | ~90s | +| **drive-coder** (clone → LLM → push → PR) | ~60s | +| suspend-microvm (VM → SUSPENDED, stays down) | ~5s | +| holdout / deploy-test (terraform validate) | ~20–40s each | +| security-agent + devops-gate (external) | ~8–15 min combined (dominates) | +| fix round: resume-or-recreate + coder re-run → new commit | ~2 min | + +*(The external review agents dominate total wall-clock on BOTH substrates. The MicroVM is SUSPENDED +for the entire multi-minute review window — that idle time is free on Lambda, billed on Kata.)* --- @@ -76,54 +75,63 @@ Security — take ~8–15 min combined and dominate total wall-clock on BOTH sub --- -## DAG — same pipeline, one substrate-branched step +## DAG — two SEPARATE WorkflowTemplates (one per substrate) -Both substrates run the **same `df-run` WorkflowTemplate**. The DAG is identical: +The substrates run **different Argo WorkflowTemplates**, so each graph is clean and Kata's certified +pipeline is never touched by Flow D changes. The Argo Events sensor routes by label: +`dark-factory` → `df-run` (Kata), `darkfactory-lambda` → `df-run-lambda` (Flow D). +**Kata — `df-run` (certified, byte-identical to the mature pipeline):** ``` -claim → drive-coder → { holdout-gate, devops-gate → security-agent, detect-deployable → deploy-test } → status → onExit(teardown) +claim(SandboxClaim) → drive-coder → { holdout, devops-gate → security, detect → deploy-test } → status → onExit(teardown: delete claim) ``` -The **only** substrate branch is inside `claim-sandbox`: `trigger-label` selects the warm pool — -`coder-warmpool` (Kata) vs `coder-warmpool-microvm` (Lambda). **There is no MicroVM-specific step in -the DAG** — suspend/resume for Lambda is handled by the *bridge* itself (see below), so the Kata -graph contains zero MicroVM nodes. - -### Substrate-specific mechanics (outside the DAG) -- **Kata:** the operator materializes a pod from `SandboxTemplate/coder-sandbox`; the coder runs - in-cluster, reaches models via Bifrost, and its logs are native pod logs. -- **Lambda:** `SandboxTemplate/coder-sandbox-microvm` materializes a **bridge pod** which: - 1. reads the platform image handoff (imageARN + execRoleARN, built once by KRO/ACK), - 2. creates a **`Microvm` CR** (declarative — the controller delivers the runHookPayload), - 3. waits for RUNNING, mints an auth token, and **POSTs `/run`** to the VM endpoint → the - hook-server background-spawns the coder, - 4. **suspends the MicroVM** once the coder pushes the PR (free compute during review), - 5. terminates the VM on teardown (delete the `Microvm` CR). +**Lambda — `df-run-lambda` (MicroVM-native; NO SandboxClaim / bridge / warm pool):** +``` +provision-microvm → drive-coder → suspend-microvm → { holdout, devops-gate, security, detect → deploy-test } → status → onExit(keep suspended VM) +``` +The one extra node — `suspend-microvm` — is **explicit and lives only in the Lambda graph**, so the +Kata graph still contains zero MicroVM nodes. Suspend/resume is owned by the workflow directly (it +calls `aws lambda-microvms suspend/resume-microvm`), not a bridge or a lifecycle controller. + +### Substrate-specific mechanics +- **Kata:** `claim-sandbox` binds a **pre-warmed** pod from `coder-warmpool`; the operator injects + `DF_*` env; the baked `entrypoint.js` runs in-cluster, reaches models via **Bifrost**, native logs. +- **Lambda:** `provision-microvm` (a single workflow step, running as `dark-factory-workflow` with the + lambda-microvms role via Pod Identity) does it all — no bridge pod, no warm pool: + 1. reads the platform image handoff (imageARN + execRoleARN, built **once** by KRO/ACK), + 2. creates the **`Microvm` CR** (stable name `mvm-`) + a runHookPayload Secret + (the review note is folded in here — the coder has no claim env), + 3. waits RUNNING + endpoint, mints an auth token, **POSTs `/run`** → the hook-server + background-spawns the same `entrypoint.js` with `USE_BEDROCK=1` (Bedrock-direct, no cluster net), + 4. the `suspend-microvm` DAG step suspends the VM once the PR is open (idlePolicy + `autoResumeEnabled=false` + nothing polls the endpoint → it **stays** suspended), + 5. on a fix round `provision-microvm` **resumes the same suspended VM** (warm resume); if the + pre-GA service failed the resume (VM terminated), it **recreates a fresh VM** automatically, + 6. `df-merge-teardown` deletes the `Microvm` CR at merge → controller `TerminateMicrovm`. --- ## Step-by-step: what actually happens -### Kata (Flow B) -1. Issue labeled `dark-factory` → Argo Events sensor → `df-run`. -2. `claim-sandbox` claims a **pre-warmed** Kata pod from `coder-warmpool` (instant). -3. Operator injects `DF_ISSUE_NUMBER` etc. → the baked-in `entrypoint.js` runs: clone → Claude - Code (`claude -p`, via **Bifrost**) → commit → **open PR**. -4. Review gates: DevOps Agent (check-run) + Security Agent (findings). Consolidated verdict posted. -5. Human comment "fix findings" → `df-iterate` → new Kata coder round → re-review. -6. Approve → `df-merge-teardown` merges + releases the claim. - -### Lambda MicroVM (Flow D) -1. Issue labeled `darkfactory-lambda` → same sensor → `df-run` (warm-pool branched to Lambda). -2. `claim-sandbox` claims the **bridge** pod from `coder-warmpool-microvm`. -3. Bridge creates a `Microvm` CR → controller `RunMicrovm` (**cold-start ~90s**) → RUNNING. -4. Bridge mints auth token → `POST /run` → hook-server background-spawns the **same - `entrypoint.js`**, but `USE_BEDROCK=1` so it calls **Bedrock directly** (exec role) — no cluster - network. Coder: clone → Claude Code → commit → **open PR**. -5. Bridge **suspends** the MicroVM (free compute while gates run). -6. Same review gates + verdict. -7. "fix findings" → `df-iterate` (routes back to Lambda via `trigger-label`) → fresh MicroVM round. -8. Approve → merge + teardown (bridge deletes the `Microvm` CR → controller terminates the VM). +### Kata (Flow B) — `df-run` +1. Issue labeled `dark-factory` → sensor dep `issue-labeled-kata` → `df-run`. +2. `claim-sandbox` binds a **pre-warmed** Kata pod from `coder-warmpool` (instant). +3. Operator injects `DF_*` → baked `entrypoint.js`: clone → Claude Code (via **Bifrost**) → **open PR**. +4. Review gates: DevOps Agent + Security Agent → consolidated verdict. +5. "fix findings" → `df-iterate` → **new** Kata coder round → re-review. +6. Approve → `df-merge-teardown` merges + deletes the claim. + +### Lambda MicroVM (Flow D) — `df-run-lambda` +1. Issue labeled `darkfactory-lambda` → sensor dep `issue-labeled-lambda` → `df-run-lambda`. +2. `provision-microvm` creates the `Microvm` CR → controller `RunMicrovm` (**cold-start ~90s**) → + RUNNING; mints token; `POST /run` → hook-server spawns the coder (`USE_BEDROCK=1`, + **Bedrock-direct**). Coder: clone → Claude Code → **open PR**. +3. `suspend-microvm` step suspends the VM → it **stays SUSPENDED** while gates run (scale-to-zero). +4. Same review gates + verdict. +5. "fix findings" → `df-iterate` → `df-run-lambda` fix round: **resume the SAME VM** (warm) or, if the + pre-GA service failed the resume, **recreate fresh**; the coder re-runs with the note → new commit. +6. Approve → `df-merge-teardown` merges + deletes the `Microvm` CR → controller `TerminateMicrovm`. --- @@ -138,16 +146,23 @@ snapshot/hook execution model**: | 2 | Can't reach Bifrost's ClusterIP from a MicroVM | **Bedrock-direct** via the exec role (`bedrock:InvokeModel`); no Bifrost/NLB/VPC-connector | | 3 | Runtime logs don't reach CloudWatch | hook-server captures coder stdout → `/logs` HTTP endpoint | | 4 | Coder is one-shot but the MicroVM `/run` hook has a 30s timeout | `/run` **background-spawns** the coder + returns fast; pipeline polls GitHub for the PR | -| 5 | Env injection needs a `coder` container | bridge container named `coder` (claim contract parity) | -| 6 | aws-cli image lacks `lambda-microvms`; no node | bridge image = `aws-cli:latest` (has the verbs) + python3 for JSON + fetch kubectl at start | -| 7 | Ingress: `ALL_INGRESS` blocks auth-token minting | use **`HTTP_INGRESS`** (+ `SHELL_INGRESS` for debug) | -| 8 | `runHookPayload` is a `SecretKeyReference`; imperative `run-microvm --run-hook-payload` doesn't fire `/run` | deliver via the **declarative `Microvm` CR** | -| 9 | Image rebuild: overwriting the same S3 key doesn't rebuild | use versioned artifact keys; bump `codeArtifactUri` | -| 10 | Pre-GA controller state can wedge (ConflictException / hung build) on delete/recreate | delete the AWS image by ARN or the CR cleanly; keep the hook-server minimal | -| — | IAM for the controller/bridge/exec roles | `iam:PassRole` (ARN-scoped), `lambda:PassNetworkConnector`, `lambda:CreateMicrovmAuthToken`, `bedrock:InvokeModel`, `s3:ListAllMyBuckets` on the capability role | +| 5 | Ingress: `ALL_INGRESS` blocks auth-token minting | use **`HTTP_INGRESS`** | +| 6 | aws-cli image lacks `lambda-microvms`; no node | step image = `aws-cli:latest` (has the verbs) + python3 for JSON + fetch kubectl at start | +| 7 | `runHookPayload` is a `SecretKeyReference`; imperative `run-microvm --run-hook-payload` doesn't fire `/run` | deliver via the **declarative `Microvm` CR** | +| 8 | Image rebuild: overwriting the same S3 key doesn't rebuild | use versioned artifact keys; bump `codeArtifactUri` | +| 9 | **VM would not stay SUSPENDED** (console showed RUNNING) | `idlePolicy.autoResumeEnabled=false` **and** never hit the VM endpoint after `/run` — any request auto-resumes it | +| 10 | **VM auto-terminated before the fix round** (resume hit "already terminated") | `suspendedDurationSeconds=28800` (Lambda's 8h max) so it survives the review→human window; 5 min was far too short | +| 11 | **Fix round re-ran but committed nothing** (coder "done" on old sha) | the review note must ride on the **runHookPayload** (MicroVM has no claim env); hook-server keys its `/run` guard on a **per-invocation run-id** so a resumed VM accepts a fresh run | +| 12 | Pre-GA **resume-from-suspend is intermittently flaky** (Internal service error → VM terminates) | `provision-microvm` checks the VM's real AWS state and **recreates a fresh VM** when resume isn't possible — the fix round self-heals | +| 13 | Two Sandboxes fought over one VM → suspend/resume **flapped** until the VM died | name the CR/workflow by **issue-number** (stable across rounds) so there's exactly **one VM per issue** | +| — | IAM: the workflow SA calls the lambda-microvms verbs | Pod Identity binds `dark-factory-workflow` → the lambda-microvms role (get/suspend/resume/terminate-microvm + create-auth-token); exec role keeps `bedrock:InvokeModel` | + +> **Note on the ACK CR status:** `Microvm.status.state` is **stale** — it does not reflect +> suspend/resume/terminate. Always read AWS truth with `aws lambda-microvms get-microvm --query state`. Kata needs **none** of these — it's an in-cluster pod with a mounted workspace, native logs, -Bifrost reachability, and a normal ECR image. +Bifrost reachability, and a normal ECR image. That's why the two substrates are **separate +WorkflowTemplates** (`df-run` vs `df-run-lambda`): Flow D's plumbing never touches the certified Kata graph. --- diff --git a/docs/dark-factory/SUBSTRATE-DIAGRAMS.md b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md index 4b9651f8..26442e88 100644 --- a/docs/dark-factory/SUBSTRATE-DIAGRAMS.md +++ b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md @@ -5,27 +5,27 @@ Mermaid (render on GitHub). --- -## 1. Shared pipeline, substrate-branched claim +## 1. Label-routed to two SEPARATE WorkflowTemplates -Both substrates run the **same `df-run` WorkflowTemplate**. The only branch is which warm pool -`claim-sandbox` claims from — decided by the issue's trigger label. +The Argo Events sensor routes each label to a **different** WorkflowTemplate, so Kata's certified +pipeline is never touched by Flow D. Kata keeps its SandboxClaim; Lambda provisions a MicroVM directly. ```mermaid flowchart TD - ISSUE["GitHub issue labeled
dark-factory OR darkfactory-lambda"] --> SENSOR["Argo Events sensor"] - SENSOR --> DFRUN["df-run WorkflowTemplate"] - DFRUN --> CLAIM{"claim-sandbox
trigger-label?"} - CLAIM -->|dark-factory| KP["coder-warmpool
(Kata pool)"] - CLAIM -->|darkfactory-lambda| LP["coder-warmpool-microvm
(Lambda bridge pool)"] - KP --> CODE["drive-coder"] - LP --> CODE - CODE --> GATES["holdout-gate · detect→deploy-test
devops-gate → security-agent"] + ISSUE["GitHub issue labeled"] --> SENSOR["Argo Events sensor"] + SENSOR -->|"dark-factory
(issue-labeled-kata)"| DFRUN["df-run
(certified Kata)"] + SENSOR -->|"darkfactory-lambda
(issue-labeled-lambda)"| DFRUNL["df-run-lambda
(MicroVM-native)"] + DFRUN --> KCLAIM["claim-sandbox
(warm Kata pod)"] --> KCODE["drive-coder"] + DFRUNL --> PROV["provision-microvm
(create Microvm CR + POST /run)"] --> LCODE["drive-coder"] + LCODE --> SUSP["suspend-microvm
(scale-to-zero)"] + KCODE --> GATES["holdout · detect→deploy-test
devops-gate · security-agent"] + SUSP --> GATES GATES --> STATUS["status (consolidated verdict)"] - STATUS --> EXIT["onExit: teardown"] + STATUS --> EXIT["onExit: Kata deletes claim ·
Lambda KEEPS suspended VM"] ``` -The DAG has **no MicroVM-specific node** — Lambda suspend/resume lives in the bridge (§4), so the -Kata graph is 100% clean. +The Kata graph has **zero MicroVM nodes**. `suspend-microvm` is explicit and lives only in +`df-run-lambda`; suspend/resume is driven by the workflow itself (§4), not a bridge or controller. --- @@ -50,49 +50,60 @@ flowchart LR --- -## 3. Lambda MicroVM substrate (Flow D) +## 3. Lambda MicroVM substrate (Flow D) — MicroVM-native, no bridge ```mermaid flowchart LR - CLAIM["SandboxClaim"] --> BR["bridge pod (in-cluster)"] - BR -->|reads handoff| IMG["MicrovmSandbox status
imageARN + execRoleARN
(built once by KRO/ACK)"] - BR -->|creates| MCR["Microvm CR
(runHookPayload = Secret ref)"] + PROV["provision-microvm step
(dark-factory-workflow SA
+ lambda-microvms role)"] -->|reads handoff| IMG["MicrovmSandbox status
imageARN + execRoleARN
(built once by KRO/ACK)"] + PROV -->|creates mvm-<issue-number>| MCR["Microvm CR
(runHookPayload = Secret ref,
autoResume=false)"] MCR --> CTRL["lambdamicrovms controller"] CTRL -->|RunMicrovm cold-start| VM["Firecracker MicroVM
hook-server :8080"] - BR -->|mint token, POST /run| VM + PROV -->|mint token, POST /run| VM VM --> ENT["entrypoint.js (USE_BEDROCK=1)"] ENT -->|models, direct| BED["Bedrock
(exec role, public egress)"] ENT -->|git/gh :443| GH["GitHub → PR"] VM -.coder stdout.-> LOGS["GET /logs (token)"] - BR -->|after PR pushed| SUSP["suspend-microvm
(free compute)"] - BR -->|teardown: delete CR| TERM["controller TerminateMicrovm"] + SUSP["suspend-microvm step
(after PR)"] -->|suspend-microvm| VM + MERGE["df-merge-teardown
(at merge)"] -->|delete CR| TERM["controller TerminateMicrovm"] ``` -- `RunMicrovm` **cold-start per session** (~90s); no node pool. -- No cluster network dependency — **Bedrock-direct**. Logs via `/logs`. Bridge suspends the VM - after the PR, terminates on teardown. +- One workflow **step** (`provision-microvm`) does create + drive `/run` — **no bridge pod, no + SandboxClaim, no warm pool**. `RunMicrovm` cold-start per session (~90s); no node pool. +- No cluster network dependency — **Bedrock-direct**. Logs via `/logs`. The explicit + `suspend-microvm` step suspends after the PR; the VM stays suspended (autoResume=false) until a fix + round resumes it or merge terminates it. --- -## 4. Lambda suspend / resume (bridge-owned, not a DAG step) +## 4. Lambda suspend / resume (workflow-driven; warm resume + recreate-fallback) ```mermaid sequenceDiagram - participant B as bridge + participant W as df-run-lambda (workflow) participant C as lambdamicrovms controller participant V as MicroVM - B->>C: create Microvm CR (runHookPayload) + W->>C: provision: create Microvm CR (autoResume=false) C->>V: RunMicrovm (cold-start) - V-->>B: RUNNING + endpoint - B->>V: POST /run (token) → coder starts - V-->>B: /logs shows "PR opened" - B->>V: suspend-microvm (free compute during review) - Note over V: SUSPENDED (memory+disk preserved) - Note over B,V: on fix round, a fresh Microvm CR is created
(Kata likewise claims a fresh coder per round) - B->>C: delete Microvm CR (on teardown) + V-->>W: RUNNING + endpoint + W->>V: POST /run (token) → coder starts → PR + W->>V: suspend-microvm step + Note over V: SUSPENDED — stays down (no endpoint polling) + Note over W,V: review gates run while VM is suspended (free) + Note over W,V: FIX ROUND (df-iterate → df-run-lambda): + W->>V: resume-microvm (warm — SAME VM) + alt resume OK (pre-GA happy path) + V-->>W: RUNNING → POST /run → coder re-runs → new commit + else resume fails (pre-GA flakiness → VM terminated) + W->>C: recreate: fresh Microvm CR + C->>V: RunMicrovm → coder re-runs → new commit + end + W->>C: at merge (df-merge-teardown): delete Microvm CR C->>V: TerminateMicrovm ``` +The CR is named `mvm-` (stable across rounds) → exactly one VM per issue, so +suspend/resume never flap between competing owners. + --- ## 5. End-to-end lifecycle (issue → PR → fix → merge) — both substrates From 809cbae8efff4c43e2b867c4ac83934488435f00 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 17:32:01 -0400 Subject: [PATCH 66/67] fix(df-merge): only count agent findings on the CURRENT head sha MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit merge.js refused to merge PR #136 citing 'Security agent: 1 finding, DevOps: 2 findings' even though every commit status on the head was green. Cause: it took the agents' latest REVIEW regardless of commit — those bodies/inline comments were on the OLD sha (round 1), while the fix-round re-review posts fresh commit STATUSES (green) on the NEW head. So a stale first-round finding permanently blocked any PR that was ever fixed. Filter reviews (commit_id) and inline comments (original_commit_id/commit_id) to the current head sha before counting. Green statuses on the head remain the gate. --- .../addons/charts/dark-factory/scripts/merge.js | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/gitops/addons/charts/dark-factory/scripts/merge.js b/gitops/addons/charts/dark-factory/scripts/merge.js index 28fe432a..273c15cb 100644 --- a/gitops/addons/charts/dark-factory/scripts/merge.js +++ b/gitops/addons/charts/dark-factory/scripts/merge.js @@ -87,8 +87,19 @@ async function main() { // if the Security or DevOps agent BOT posted findings (a review body reporting // "N finding(s)" or change-requesting inline review comments). try { - const reviews = (await api("GET", `/repos/${REPO}/pulls/${PR}/reviews?per_page=100`)) || []; - const comments = (await api("GET", `/repos/${REPO}/pulls/${PR}/comments?per_page=100`)) || []; + const allReviews = (await api("GET", `/repos/${REPO}/pulls/${PR}/reviews?per_page=100`)) || []; + const allComments = (await api("GET", `/repos/${REPO}/pulls/${PR}/comments?per_page=100`)) || []; + // CRITICAL: only count findings on the CURRENT head sha. A fix round pushes a NEW + // commit and the agents re-review it (posting fresh commit STATUSES, already checked + // green above); their earlier REVIEW bodies/inline comments remain attached to the + // OLD (superseded) sha. Without this filter merge.js counts those stale first-round + // findings and refuses to merge every PR that was ever fixed — observed on PR #136: + // security/devops reviews on sha 3b11b497 (round 1) blocked a merge whose head + // 8089fa0a (fix round) was fully green. Match reviews by commit_id and inline + // comments by original_commit_id/commit_id to the head sha. + const onHead = (c) => c === sha; + const reviews = allReviews.filter((r) => onHead(r.commit_id)); + const comments = allComments.filter((c) => onHead(c.commit_id) || onHead(c.original_commit_id)); const isSecBot = (l) => /aws-security-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const isDevBot = (l) => /aws-devops-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const botFindings = (pred) => { From 2457dfaebc2d3b728628cd28ad7d6e48c1fa2c62 Mon Sep 17 00:00:00 2001 From: Elamaran Shanmugam Date: Tue, 4 Aug 2026 17:35:35 -0400 Subject: [PATCH 67/67] fix(df-merge): match inline findings by original_commit_id (exclude carried-forward) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First fix filtered by commit_id, but GitHub AUTO-ADVANCES an inline comment's commit_id to the latest head when the line still exists — so round-1 findings reappeared with commit_id==head and still blocked the merge (PR #136: 3 inline comments carried forward from sha 3b11b497 onto head 8089fa0a). original_commit_id preserves the sha the comment was truly filed against; match on that so only findings genuinely filed against the current head count. Green head statuses (round-2 re-review) remain the authoritative gate. --- gitops/addons/charts/dark-factory/scripts/merge.js | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/gitops/addons/charts/dark-factory/scripts/merge.js b/gitops/addons/charts/dark-factory/scripts/merge.js index 273c15cb..c8144403 100644 --- a/gitops/addons/charts/dark-factory/scripts/merge.js +++ b/gitops/addons/charts/dark-factory/scripts/merge.js @@ -97,9 +97,14 @@ async function main() { // security/devops reviews on sha 3b11b497 (round 1) blocked a merge whose head // 8089fa0a (fix round) was fully green. Match reviews by commit_id and inline // comments by original_commit_id/commit_id to the head sha. - const onHead = (c) => c === sha; - const reviews = allReviews.filter((r) => onHead(r.commit_id)); - const comments = allComments.filter((c) => onHead(c.commit_id) || onHead(c.original_commit_id)); + // A review counts only if it was submitted against the current head. An inline + // comment counts only if it was ORIGINALLY filed against the current head + // (original_commit_id) — GitHub auto-advances an inline comment's commit_id to the + // latest head when the line still exists, so round-1 comments reappear with + // commit_id==head; original_commit_id preserves the sha they were truly filed on. + // Matching on original_commit_id excludes those carried-forward round-1 findings. + const reviews = allReviews.filter((r) => r.commit_id === sha); + const comments = allComments.filter((c) => (c.original_commit_id || c.commit_id) === sha); const isSecBot = (l) => /aws-security-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const isDevBot = (l) => /aws-devops-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const botFindings = (pred) => {