diff --git a/docs/dark-factory/README.md b/docs/dark-factory/README.md index 94a0f66a..cb097cbd 100644 --- a/docs/dark-factory/README.md +++ b/docs/dark-factory/README.md @@ -26,6 +26,7 @@ spokes as normal deployments. 2. [Two flows at a glance](#2-two-flows-at-a-glance) 3. [Flow A — Agent Sandbox capability](#3-flow-a--agent-sandbox-capability-permanent-platform-feature) 4. [Flow B — the Dark Factory pipeline](#4-flow-b--the-dark-factory-pipeline) + - [Flow D — Lambda MicroVM substrate (alternative to Flow A)](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a) 5. [The pluggable coding assistant](#5-the-pluggable-coding-assistant) 6. [Independent verification](#6-independent-verification-the-heart-of-the-pattern) 7. [Live status in the PR](#7-live-status-in-the-pr) @@ -84,6 +85,15 @@ useful on its own and the factory is a consumer of it. > an isolated VM). The Dark Factory is one *consumer* of that capability. Keeping them separate > means the isolation substrate can ship, be tested, and be reused independently of the factory. +> **Flow D — a second substrate.** Flow A's isolation boundary is a **Kata micro-VM pod** on a +> platform-owned nested-virt node group. **Flow D** offers an *alternative* Flow-A substrate — an +> **AWS Lambda MicroVM** (serverless micro-VM, no node group) provisioned via the ACK `lambdamicrovms` +> controller and composed by a single **KRO `ResourceGraphDefinition`**. Flow B is unchanged and can +> target either substrate through the same `SandboxClaim` contract. See +> [§4.5](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a) and +> [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md). *(Flow C is reserved for +> other work.)* + > **Why the hub, not a spoke?** The Dark Factory is a **pre-dev build/author** activity: it *writes* > code and needs GitHub write access. That belongs on the **hub — the control/build plane** — not on > a spoke, which is the **deploy/run plane** hosting real enterprise workloads (putting a @@ -256,6 +266,102 @@ UI — the substrate for scaling across many concurrent issues. --- +## 4.5. Flow D — Lambda MicroVM substrate (alternative to Flow A) + +> 📊 **See the diagrams:** [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md) +> (substrate architecture + platform/app ownership split + the RuntimeClass-shim bridge). + +Flow A's isolation boundary is a **Kata micro-VM pod** on a platform-owned nested-virt node group. +**Flow D is a second Flow-A substrate**: an **AWS Lambda MicroVM** — a *serverless* micro-VM with no +node group to run or pay for while idle, per-claim lifecycle, and sub-second warm starts. Flow B is +unchanged: it still creates a `SandboxClaim`, a pod still shows up, and the **same `dark-factory-coder`** +runs its coding/testing loop — except the coder executes inside a Lambda MicroVM. *(Flow C is reserved +for other work; this substrate is Flow D.)* + +### How it's built — KRO RGD over ACK primitives + +| Layer | Mechanism | Notes | +|---|---|---| +| **Composition** | **Managed KRO** (EKS Capability) + one `MicrovmSandbox` `ResourceGraphDefinition` | One CR expands into the IMAGE primitives below (built once); the running `Microvm` is NOT in the graph — the shim runs it imperatively | +| **GA primitives** | **Managed ACK** (EKS Capability) — `iam` Role, `s3` Bucket | AWS-run; the image store + build/exec roles | +| **Image primitive** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` CRD (`lambdamicrovms.services.k8s.aws/v1alpha1`); the `Microvm` is created via SDK by the shim, not as a graph resource | + +> **Why self-managed for the MicroVM controller?** Managed ACK bundles only controllers whose service +> is **GA upstream** (see the [ACK community services / GA list](https://aws-controllers-k8s.github.io/community/docs/community/services/)). +> `lambdamicrovms` is **pre-GA** (`v1alpha1`, not on that list), so it isn't in Managed ACK yet — it +> runs as its own GitOps addon. **Managed ACK + self-managed lambdamicrovms coexist** (different CRD +> groups → no conflict). When `lambdamicrovms` goes GA, delete the self-managed addon and Managed ACK +> adopts it — **the RGD is unchanged**. This "install both now" posture is deliberate and futuristic. +> +> The **Managed KRO + Managed ACK capabilities themselves** are enabled at the platform layer in the +> **appmod-blueprints** repo (an EKS Capability toggle) — see that repo's +> `docs/EKS-Capabilities-KRO-ACK-Setup.md`. This repo owns only the **self-managed `lambdamicrovms` +> controller + the KRO `MicrovmSandbox` RGD + the sandbox shim** (Flow D). + +### The split: KRO builds the image ONCE; the shim runs the VM per session + +This is the load-bearing design decision (and it matches the ACK controller's own guidance — +image build is slow/declarative, running a VM is fast/imperative): + +- **Platform image — declarative, built ONCE by KRO/ACK.** The `MicrovmSandbox` RGD + (`agent-sandbox-lambda/templates/image/`) composes only the slow-changing infra: `MicrovmImage` + (`baseImageARN`, `buildRoleARN`, `codeArtifact.uri` — an **S3 zip** of the arm64 `dark-factory-coder` + + a Dockerfile) plus its **build role**, **execution role**, and **S3 artifact bucket** (ACK GA + controllers). A **single committed `MicrovmSandbox` instance** (GitOps-applied) is reconciled once; + KRO gates the handoff on a successful build (`readyWhen state == CREATED||UPDATED`). Its status + surfaces `imageARN` + `executionRoleARN`. The RGD **does not** contain a `Microvm`. +- **Per-session VM — imperative, driven by the shim.** Running a MicroVM (`RunMicrovm`), and its + `suspend` / `resume` / `TerminateMicrovm`, are request-time SDK ops the ACK controller does **not** + reconcile. So the shim owns them — never a `Microvm` CR per claim. + +### The RuntimeClass shim (claim → pod → MicroVM) + +A literal K8s `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**; Lambda +MicroVM is a **remote AWS service**, so a true node-level RuntimeClass would require a virtual-kubelet +provider (a large Go runtime — **out of scope**). Flow D instead ships a **`lambda-microvm` +SandboxTemplate variant** (`agent-sandbox-lambda/templates/shim/`) whose pod is a lightweight +**bridge**: on claim it **reads the platform image handoff** (`imageARN` + `executionRoleARN` from the +one built `MicrovmSandbox`) and calls **`RunMicrovm`** (SDK) to launch this session's VM, records the +`microvmID` as an annotation on the owning `Sandbox`, and holds the pod so its lifecycle mirrors the +MicroVM's. On real teardown it calls `TerminateMicrovm`. To Flow B and the user the UX is identical to +Flow A. Interactive exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up. + +### Suspend / resume / terminate — the coder VM persists across the review loop + +Because the substrate is a Lambda MicroVM (not a pod), Flow D uses **suspend/resume through the Agent +Sandbox CRD** to keep the coder's context across the whole review→fix→re-review loop — the payoff of +this substrate over Kata (where each fix round claims a fresh pod): + +1. **Coder finishes the coding task → SUSPEND** (`df-run` flips `Sandbox.operatingMode=Suspended`; the + `microvm-lifecycle` reconcile loop calls `suspend-microvm` by the annotated id). Compute is freed; + the VM's memory/disk are snapshotted. +2. DevOps + Security agents review the PR while the coder is suspended. +3. **Findings + "fix" → RESUME the SAME VM** (df-iterate sets `operatingMode=Running` → `resume-microvm`). + Context intact — no cold re-implement. +4. Coder fixes → SUSPEND again; loop 2–4 until both agents clear. +5. **Final exit (merge) → TERMINATE** (`df-merge-teardown` calls `TerminateMicrovm`, then deletes the + claim). This is the **only** place the VM is destroyed — `df-run`'s onExit is substrate-aware and + **keeps** the suspended Lambda VM (unlike Kata, which frees its pod on df-run exit). + +The ACK `Microvm` has no suspend field, so the `microvm-lifecycle` loop supplies the intent→SDK +translation — pure shim, no virtual-kubelet. See +[`diagrams/flow-d-microvm-sandbox.md` §D.3a](diagrams/flow-d-microvm-sandbox.md). + +### Delivery & status + +Shipped as GitOps in its **own chart** — `gitops/addons/charts/agent-sandbox-lambda/` (separate from +the Kata `agent-sandbox` chart), structured as `templates/image/` (KRO RGD + the one platform +`MicrovmSandbox`) and `templates/shim/` (bridge SandboxTemplate + warm pool + `microvm-lifecycle` +controller). **Disabled by default** (`microvm.enabled=false`); the hub overlay +(`clusters/hub/addons/agent-sandbox-lambda/values.yaml`) carries cluster-specific values, and a gated +`agent-sandbox-lambda` addon entry deploys it hub-only. The platform-capability enablement (Managed ACK ++ Managed KRO) lands separately in the **appmod-blueprints** platform repo (they're EKS Capabilities, +like the Managed ArgoCD the hub already runs). This PR delivers the **design + GitOps scaffold**; the +live end-to-end path (enable capabilities → sync controller → publish the arm64 artifact → run a MicroVM +coder with suspend/resume) is the follow-up. + +--- + ## 5. The pluggable coding assistant The coder is behind a **thin, swappable interface** — a deliberate choice (the industry lesson is diff --git a/docs/dark-factory/SUBSTRATE-BENCHMARK.md b/docs/dark-factory/SUBSTRATE-BENCHMARK.md new file mode 100644 index 00000000..196f34cb --- /dev/null +++ b/docs/dark-factory/SUBSTRATE-BENCHMARK.md @@ -0,0 +1,178 @@ +# Dark Factory — Substrate Benchmark: Kata micro-VM vs Lambda MicroVM + +A side-by-side comparison of the two sandbox substrates that run the autonomous coder, +measured on **identical issues fired in parallel** on the same hub cluster. + +- **Flow B — Kata micro-VM** (mature, default): the coder runs in a hardware-isolated Kata + pod on a self-managed nested-virt EKS node group. +- **Flow D — AWS Lambda MicroVM** (pre-GA): the coder runs in a Firecracker MicroVM + provisioned via the `lambdamicrovms` ACK controller, driven by a bridge pod. + +Both run the **same `dark-factory-coder`** (same `entrypoint.js`), produce the same kind of +PR, and go through the **same review gates** (AWS DevOps Agent + AWS Security Agent). The only +difference is *where the coder executes* and *how it's provisioned*. + +--- + +## TL;DR + +| | Kata micro-VM (Flow B) | Lambda MicroVM (Flow D) | +| --- | --- | --- | +| **Workflow** | `df-run` (certified) | `df-run-lambda` (separate, MicroVM-native) | +| **Provisioning** | pre-warmed pool → **instant claim** | **RunMicrovm cold-start per session** (~90s) | +| **Time to first PR** (from label) | ~**2 min** | ~**2.5 min** | +| **LLM path** | Bifrost gateway (in-cluster) + Langfuse traces | **Bedrock-direct** (exec role) — no cluster network | +| **Scale-to-zero when idle** | ❌ node pool runs continuously | ✅ **suspend-to-zero** between PR and merge | +| **Fix-round mechanic** | fresh pod each round | **resume the SAME suspended VM** (warm); recreate if the pre-GA resume fails | +| **Infra to manage** | nested-virt node group (Karpenter/MNG) | none — serverless MicroVMs | +| **Observability** | native `kubectl logs` | custom `/logs` HTTP endpoint (no runtime CloudWatch) | +| **Maturity** | production-ready today | pre-GA (preview) — pilot-grade | +| **Economics at 1000s scale** | pay for idle capacity | pay per active minute (the strategic win) | + +**Bottom line:** at small scale the two feel equivalent (the LLM coding step ~2–4 min and the +external review agents ~8–15 min dominate total time on *both*). The Lambda substrate's advantage +is **not latency** — it's **operational + economic**: no node pool to run, and suspend-to-zero +between the PR and the human's review/merge. Its cost is **maturity** (pre-GA control plane — resume +from suspend is occasionally flaky, mitigated by the recreate-fallback) and the extra plumbing below. + +--- + +## Benchmarked run (identical issue, per substrate) + +### Time to first PR (from label → PR opened) +| Substrate | Issue | PR | Elapsed | Notes | +| --- | --- | --- | --- | --- | +| Kata | #137 | — | ~**2 min** | pre-warmed pod, instant claim | +| Lambda | #135 | #136 | ~**2.5 min** (20:53:03 → 20:55:35) | native `df-run-lambda`, RunMicrovm cold-start | + +**Δ ≈ 30–90s** — the MicroVM cold-start (`RunMicrovm` → RUNNING → `/run`) vs Kata's pre-warmed pod +claim. Note the MicroVM-native `df-run-lambda` is **faster than the old bridge path** (~3.7 min): +removing the SandboxClaim/warm-pool indirection cut ~1 min. Everything after (clone → LLM → push) is +identical code and takes the same time. + +### Lifecycle timing (Lambda #135, native pipeline) +| Phase | Time | +| --- | --- | +| provision-microvm (RunMicrovm → RUNNING → `/run` HTTP 200) | ~90s | +| **drive-coder** (clone → LLM → push → PR) | ~60s | +| suspend-microvm (VM → SUSPENDED, stays down) | ~5s | +| holdout / deploy-test (terraform validate) | ~20–40s each | +| security-agent + devops-gate (external) | ~8–15 min combined (dominates) | +| fix round: resume-or-recreate + coder re-run → new commit | ~2 min | + +*(The external review agents dominate total wall-clock on BOTH substrates. The MicroVM is SUSPENDED +for the entire multi-minute review window — that idle time is free on Lambda, billed on Kata.)* + +--- + +## Where the logs are + +| What | Kata (Flow B) | Lambda (Flow D) | +| --- | --- | --- | +| Pipeline steps | Argo UI (`/argo-workflows`) or `kubectl logs -n argo ` | **same** | +| Coder output | `kubectl logs -n agent-sandbox-system df-issue-` (native) | **`GET https:///logs`** with an auth token (runtime CloudWatch routing is unreliable on the pre-GA runtime, so the hook-server captures coder stdout to a file + serves it) | +| Image build | n/a (normal ECR image) | CloudWatch `/aws/lambda/microvms/coder-image` | + +--- + +## DAG — two SEPARATE WorkflowTemplates (one per substrate) + +The substrates run **different Argo WorkflowTemplates**, so each graph is clean and Kata's certified +pipeline is never touched by Flow D changes. The Argo Events sensor routes by label: +`dark-factory` → `df-run` (Kata), `darkfactory-lambda` → `df-run-lambda` (Flow D). + +**Kata — `df-run` (certified, byte-identical to the mature pipeline):** +``` +claim(SandboxClaim) → drive-coder → { holdout, devops-gate → security, detect → deploy-test } → status → onExit(teardown: delete claim) +``` + +**Lambda — `df-run-lambda` (MicroVM-native; NO SandboxClaim / bridge / warm pool):** +``` +provision-microvm → drive-coder → suspend-microvm → { holdout, devops-gate, security, detect → deploy-test } → status → onExit(keep suspended VM) +``` +The one extra node — `suspend-microvm` — is **explicit and lives only in the Lambda graph**, so the +Kata graph still contains zero MicroVM nodes. Suspend/resume is owned by the workflow directly (it +calls `aws lambda-microvms suspend/resume-microvm`), not a bridge or a lifecycle controller. + +### Substrate-specific mechanics +- **Kata:** `claim-sandbox` binds a **pre-warmed** pod from `coder-warmpool`; the operator injects + `DF_*` env; the baked `entrypoint.js` runs in-cluster, reaches models via **Bifrost**, native logs. +- **Lambda:** `provision-microvm` (a single workflow step, running as `dark-factory-workflow` with the + lambda-microvms role via Pod Identity) does it all — no bridge pod, no warm pool: + 1. reads the platform image handoff (imageARN + execRoleARN, built **once** by KRO/ACK), + 2. creates the **`Microvm` CR** (stable name `mvm-`) + a runHookPayload Secret + (the review note is folded in here — the coder has no claim env), + 3. waits RUNNING + endpoint, mints an auth token, **POSTs `/run`** → the hook-server + background-spawns the same `entrypoint.js` with `USE_BEDROCK=1` (Bedrock-direct, no cluster net), + 4. the `suspend-microvm` DAG step suspends the VM once the PR is open (idlePolicy + `autoResumeEnabled=false` + nothing polls the endpoint → it **stays** suspended), + 5. on a fix round `provision-microvm` **resumes the same suspended VM** (warm resume); if the + pre-GA service failed the resume (VM terminated), it **recreates a fresh VM** automatically, + 6. `df-merge-teardown` deletes the `Microvm` CR at merge → controller `TerminateMicrovm`. + +--- + +## Step-by-step: what actually happens + +### Kata (Flow B) — `df-run` +1. Issue labeled `dark-factory` → sensor dep `issue-labeled-kata` → `df-run`. +2. `claim-sandbox` binds a **pre-warmed** Kata pod from `coder-warmpool` (instant). +3. Operator injects `DF_*` → baked `entrypoint.js`: clone → Claude Code (via **Bifrost**) → **open PR**. +4. Review gates: DevOps Agent + Security Agent → consolidated verdict. +5. "fix findings" → `df-iterate` → **new** Kata coder round → re-review. +6. Approve → `df-merge-teardown` merges + deletes the claim. + +### Lambda MicroVM (Flow D) — `df-run-lambda` +1. Issue labeled `darkfactory-lambda` → sensor dep `issue-labeled-lambda` → `df-run-lambda`. +2. `provision-microvm` creates the `Microvm` CR → controller `RunMicrovm` (**cold-start ~90s**) → + RUNNING; mints token; `POST /run` → hook-server spawns the coder (`USE_BEDROCK=1`, + **Bedrock-direct**). Coder: clone → Claude Code → **open PR**. +3. `suspend-microvm` step suspends the VM → it **stays SUSPENDED** while gates run (scale-to-zero). +4. Same review gates + verdict. +5. "fix findings" → `df-iterate` → `df-run-lambda` fix round: **resume the SAME VM** (warm) or, if the + pre-GA service failed the resume, **recreate fresh**; the coder re-runs with the note → new commit. +6. Approve → `df-merge-teardown` merges + deletes the `Microvm` CR → controller `TerminateMicrovm`. + +--- + +## Gotchas the Lambda substrate needed (that Kata does not) + +Because a MicroVM is **outside the cluster network, has a read-only rootfs, and uses a +snapshot/hook execution model**: + +| # | Gotcha | Fix | +| --- | --- | --- | +| 1 | Coder crashed `EACCES mkdir /workspace/artifacts` (no writable volume like Kata) | set `WORKSPACE=/tmp/workspace` (writable tmpfs) — *the silent killer* | +| 2 | Can't reach Bifrost's ClusterIP from a MicroVM | **Bedrock-direct** via the exec role (`bedrock:InvokeModel`); no Bifrost/NLB/VPC-connector | +| 3 | Runtime logs don't reach CloudWatch | hook-server captures coder stdout → `/logs` HTTP endpoint | +| 4 | Coder is one-shot but the MicroVM `/run` hook has a 30s timeout | `/run` **background-spawns** the coder + returns fast; pipeline polls GitHub for the PR | +| 5 | Ingress: `ALL_INGRESS` blocks auth-token minting | use **`HTTP_INGRESS`** | +| 6 | aws-cli image lacks `lambda-microvms`; no node | step image = `aws-cli:latest` (has the verbs) + python3 for JSON + fetch kubectl at start | +| 7 | `runHookPayload` is a `SecretKeyReference`; imperative `run-microvm --run-hook-payload` doesn't fire `/run` | deliver via the **declarative `Microvm` CR** | +| 8 | Image rebuild: overwriting the same S3 key doesn't rebuild | use versioned artifact keys; bump `codeArtifactUri` | +| 9 | **VM would not stay SUSPENDED** (console showed RUNNING) | `idlePolicy.autoResumeEnabled=false` **and** never hit the VM endpoint after `/run` — any request auto-resumes it | +| 10 | **VM auto-terminated before the fix round** (resume hit "already terminated") | `suspendedDurationSeconds=28800` (Lambda's 8h max) so it survives the review→human window; 5 min was far too short | +| 11 | **Fix round re-ran but committed nothing** (coder "done" on old sha) | the review note must ride on the **runHookPayload** (MicroVM has no claim env); hook-server keys its `/run` guard on a **per-invocation run-id** so a resumed VM accepts a fresh run | +| 12 | Pre-GA **resume-from-suspend is intermittently flaky** (Internal service error → VM terminates) | `provision-microvm` checks the VM's real AWS state and **recreates a fresh VM** when resume isn't possible — the fix round self-heals | +| 13 | Two Sandboxes fought over one VM → suspend/resume **flapped** until the VM died | name the CR/workflow by **issue-number** (stable across rounds) so there's exactly **one VM per issue** | +| — | IAM: the workflow SA calls the lambda-microvms verbs | Pod Identity binds `dark-factory-workflow` → the lambda-microvms role (get/suspend/resume/terminate-microvm + create-auth-token); exec role keeps `bedrock:InvokeModel` | + +> **Note on the ACK CR status:** `Microvm.status.state` is **stale** — it does not reflect +> suspend/resume/terminate. Always read AWS truth with `aws lambda-microvms get-microvm --query state`. + +Kata needs **none** of these — it's an in-cluster pod with a mounted workspace, native logs, +Bifrost reachability, and a normal ECR image. That's why the two substrates are **separate +WorkflowTemplates** (`df-run` vs `df-run-lambda`): Flow D's plumbing never touches the certified Kata graph. + +--- + +## When to choose which + +- **Kata (today):** production-ready, mature, standard `kubectl`/IDE access, in-cluster networking. + Choose it now for reliability. Cost: you run + pay for a nested-virt node pool continuously. +- **Lambda MicroVM (strategic):** serverless, suspend-to-zero per idle session, no node pool — + the model that scales economically to thousands of sessions. Choose it as it reaches GA. Cost + today: pre-GA control-plane maturity + the plumbing above. + +Both share the **same coder, same pipeline, same review gates, same UX** — so migrating between +substrates is a label change, invisible to the developer/issue author. diff --git a/docs/dark-factory/SUBSTRATE-DIAGRAMS.md b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md new file mode 100644 index 00000000..26442e88 --- /dev/null +++ b/docs/dark-factory/SUBSTRATE-DIAGRAMS.md @@ -0,0 +1,134 @@ +# Dark Factory — Substrate Diagrams (Kata vs Lambda MicroVM) + +Visual companion to [`SUBSTRATE-BENCHMARK.md`](./SUBSTRATE-BENCHMARK.md). All diagrams are +Mermaid (render on GitHub). + +--- + +## 1. Label-routed to two SEPARATE WorkflowTemplates + +The Argo Events sensor routes each label to a **different** WorkflowTemplate, so Kata's certified +pipeline is never touched by Flow D. Kata keeps its SandboxClaim; Lambda provisions a MicroVM directly. + +```mermaid +flowchart TD + ISSUE["GitHub issue labeled"] --> SENSOR["Argo Events sensor"] + SENSOR -->|"dark-factory
(issue-labeled-kata)"| DFRUN["df-run
(certified Kata)"] + SENSOR -->|"darkfactory-lambda
(issue-labeled-lambda)"| DFRUNL["df-run-lambda
(MicroVM-native)"] + DFRUN --> KCLAIM["claim-sandbox
(warm Kata pod)"] --> KCODE["drive-coder"] + DFRUNL --> PROV["provision-microvm
(create Microvm CR + POST /run)"] --> LCODE["drive-coder"] + LCODE --> SUSP["suspend-microvm
(scale-to-zero)"] + KCODE --> GATES["holdout · detect→deploy-test
devops-gate · security-agent"] + SUSP --> GATES + GATES --> STATUS["status (consolidated verdict)"] + STATUS --> EXIT["onExit: Kata deletes claim ·
Lambda KEEPS suspended VM"] +``` + +The Kata graph has **zero MicroVM nodes**. `suspend-microvm` is explicit and lives only in +`df-run-lambda`; suspend/resume is driven by the workflow itself (§4), not a bridge or controller. + +--- + +## 2. Kata micro-VM substrate (Flow B) + +```mermaid +flowchart LR + CLAIM["SandboxClaim"] --> OP["agent-sandbox operator"] + OP --> POD["Kata pod (kata-clh)
on nested-virt node group"] + POD --> ENT["entrypoint.js (baked)"] + ENT -->|models| BIF["Bifrost gateway
(ClusterIP, in-cluster)"] + BIF --> BED["Bedrock"] + ENT -->|git/gh :443| GH["GitHub → PR"] + ENT -->|secrets| SEC["/etc/secrets
(projected tmpfs)"] + POD -.native logs.-> KL["kubectl logs"] + BIF -.traces.-> LF["Langfuse"] +``` + +- Pre-warmed pod → **instant claim**. +- Workspace is a **mounted writable volume**; logs are native; models via Bifrost (with Langfuse + traces). Node pool runs continuously. + +--- + +## 3. Lambda MicroVM substrate (Flow D) — MicroVM-native, no bridge + +```mermaid +flowchart LR + PROV["provision-microvm step
(dark-factory-workflow SA
+ lambda-microvms role)"] -->|reads handoff| IMG["MicrovmSandbox status
imageARN + execRoleARN
(built once by KRO/ACK)"] + PROV -->|creates mvm-<issue-number>| MCR["Microvm CR
(runHookPayload = Secret ref,
autoResume=false)"] + MCR --> CTRL["lambdamicrovms controller"] + CTRL -->|RunMicrovm cold-start| VM["Firecracker MicroVM
hook-server :8080"] + PROV -->|mint token, POST /run| VM + VM --> ENT["entrypoint.js (USE_BEDROCK=1)"] + ENT -->|models, direct| BED["Bedrock
(exec role, public egress)"] + ENT -->|git/gh :443| GH["GitHub → PR"] + VM -.coder stdout.-> LOGS["GET /logs (token)"] + SUSP["suspend-microvm step
(after PR)"] -->|suspend-microvm| VM + MERGE["df-merge-teardown
(at merge)"] -->|delete CR| TERM["controller TerminateMicrovm"] +``` + +- One workflow **step** (`provision-microvm`) does create + drive `/run` — **no bridge pod, no + SandboxClaim, no warm pool**. `RunMicrovm` cold-start per session (~90s); no node pool. +- No cluster network dependency — **Bedrock-direct**. Logs via `/logs`. The explicit + `suspend-microvm` step suspends after the PR; the VM stays suspended (autoResume=false) until a fix + round resumes it or merge terminates it. + +--- + +## 4. Lambda suspend / resume (workflow-driven; warm resume + recreate-fallback) + +```mermaid +sequenceDiagram + participant W as df-run-lambda (workflow) + participant C as lambdamicrovms controller + participant V as MicroVM + W->>C: provision: create Microvm CR (autoResume=false) + C->>V: RunMicrovm (cold-start) + V-->>W: RUNNING + endpoint + W->>V: POST /run (token) → coder starts → PR + W->>V: suspend-microvm step + Note over V: SUSPENDED — stays down (no endpoint polling) + Note over W,V: review gates run while VM is suspended (free) + Note over W,V: FIX ROUND (df-iterate → df-run-lambda): + W->>V: resume-microvm (warm — SAME VM) + alt resume OK (pre-GA happy path) + V-->>W: RUNNING → POST /run → coder re-runs → new commit + else resume fails (pre-GA flakiness → VM terminated) + W->>C: recreate: fresh Microvm CR + C->>V: RunMicrovm → coder re-runs → new commit + end + W->>C: at merge (df-merge-teardown): delete Microvm CR + C->>V: TerminateMicrovm +``` + +The CR is named `mvm-` (stable across rounds) → exactly one VM per issue, so +suspend/resume never flap between competing owners. + +--- + +## 5. End-to-end lifecycle (issue → PR → fix → merge) — both substrates + +```mermaid +flowchart TD + A["Issue labeled"] --> B["df-run: claim + coder → PR"] + B --> C["DevOps Agent + Security Agent review"] + C --> D{"Security findings?"} + D -->|clean| APR["Human approves PR"] + D -->|findings| FIX["Human comments 'fix findings'"] + FIX --> IT["df-iterate → df-run (same substrate via trigger-label)"] + IT --> B + APR --> MERGE["df-merge-teardown:
merge PR + release/terminate sandbox"] +``` + +The loop is identical for both substrates; `df-iterate` reads the originating issue's label to +route the fix round back to the **same** substrate (Kata pool or Lambda pool). + +--- + +## Legend / key facts + +- **Warm pool:** Kata = ready pods (instant); Lambda = bridge pods that RunMicrovm on claim. +- **LLM:** Kata → Bifrost (traced in Langfuse); Lambda → Bedrock-direct (exec role). +- **Workspace:** Kata → mounted volume; Lambda → `/tmp/workspace` (read-only rootfs). +- **Logs:** Kata → `kubectl logs`; Lambda → `/logs` endpoint (+ build logs in CloudWatch). +- **Teardown:** Kata → release claim; Lambda → delete `Microvm` CR → TerminateMicrovm. diff --git a/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md new file mode 100644 index 00000000..0eadcd25 --- /dev/null +++ b/docs/dark-factory/diagrams/flow-d-microvm-sandbox.md @@ -0,0 +1,138 @@ +# Flow D — Lambda MicroVM–backed Agent Sandbox (alternative substrate) + +**Flow D is a second Flow-A substrate.** Where Flow A hands out **Kata micro-VM pods** on a +self-managed nested-virt EKS node group, Flow D hands out **AWS Lambda MicroVMs** provisioned by the +ACK `lambdamicrovms` controller and composed by a single **KRO `ResourceGraphDefinition`**. The Agent +Sandbox UX is unchanged: a consumer (notably **Flow B — Dark Factory**) creates a `SandboxClaim`, a +pod shows up, and the **same `dark-factory-coder`** runs its coding/testing loop — except the coder +executes inside a Lambda MicroVM instead of on the Kata node. + +> **Flow C is reserved for other work** — this substrate is **Flow D**. + +> **Why a second substrate?** Kata (Flow A) needs a dedicated nested-virt node group the platform owns +> and pays for while idle. Lambda MicroVM is a **serverless** micro-VM: no node group to run, per-claim +> lifecycle, sub-second warm starts, and a clean **platform-owns-the-image / app-owns-the-instance** +> split that maps directly onto the two ACK CRDs. Flow B can target either substrate with no pipeline +> change — it only ever sees the Agent Sandbox `SandboxClaim` contract. + +> 🎨 Diagrams are editable draw.io — sources in [`src/`](./src/), rendered PNGs in [`img/`](./img/). +> *(Flow D diagram sources are added alongside the Flow A/B ones; see `src/flow-d-*.drawio`.)* + +--- + +## D.1 — Substrate architecture (KRO RGD over ACK primitives) + +The platform installs two controllers and one composition layer, then exposes **one** custom +resource to consumers: + +- **Managed ACK** (EKS Capability) runs the **GA** controllers — `iam.services.k8s.aws` (Role) and + `s3.services.k8s.aws` (Bucket) — that the MicroVM image + instance depend on. +- **Self-managed ACK** runs **only** the pre-GA `lambdamicrovms.services.k8s.aws` controller (its own + Helm chart / ArgoCD addon), because Managed ACK bundles GA controllers only. +- **Managed KRO** (EKS Capability) runs the `ResourceGraphDefinition` engine. +- A single **`MicrovmSandbox` RGD** ties it all together: one CR expands into `MicrovmImage` + + `Microvm` + IAM `Role`(s) + S3 `Bucket`. + +``` +consumer (Flow B / any agent) + │ creates + ▼ + MicrovmSandbox (kro.run/v1alpha1 — the single abstraction) + │ expands into + ├── MicrovmImage (lambdamicrovms.services.k8s.aws) ── platform-owned inputs + ├── S3 Bucket (s3.services.k8s.aws) ── image codeArtifact store + ├── IAM Role (build) (iam.services.k8s.aws) ── MicrovmImage.buildRoleArn + ├── IAM Role (exec) (iam.services.k8s.aws) ── Microvm.executionRoleArn + └── Microvm (lambdamicrovms.services.k8s.aws) ── app-owned instance lifecycle +``` + +*Edit: `src/flow-d-substrate.drawio` → `img/flow-d-substrate.png`.* + +--- + +## D.2 — Platform-owned vs app-owned split (inside one RGD) + +The two ACK CRDs encode the ownership boundary the platform team and application teams care about; +the RGD schema surfaces each half to the right owner: + +| Layer | Owner | ACK resource | Key fields | +|---|---|---|---| +| **Image / substrate** | Platform | `MicrovmImage` | `baseImageARN`, `buildRoleArn`, `codeArtifact.uri` (S3), egress connectors | +| **Instance / run** | App team | `Microvm` | `imageIdentifier`, `executionRoleArn`, `ingress/egressNetworkConnectors`, `idlePolicy` | + +- **Platform** sets the image once (built **from the existing `dark-factory-coder` image** + its + `entrypoint.js`, published to the S3 `codeArtifact` bucket) — declarative, ACK-managed, GitOps. +- **App teams / Flow B** create per-claim `Microvm` instances referencing that image, and own the + instance lifecycle (`RunMicrovm` / `TerminateMicrovm`, idle policy) via the same claim they use today. + +*Edit: `src/flow-d-ownership.drawio` → `img/flow-d-ownership.png`.* + +--- + +## D.3 — The RuntimeClass shim (claim → pod → MicroVM) + +A literal Kubernetes `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**. +Lambda MicroVM is a **remote AWS service**, so a true node-level RuntimeClass isn't possible without a +virtual-kubelet provider (a large Go runtime component — explicitly **out of scope**). Flow D uses a +**RuntimeClass-marked bridge pod** instead, preserving the exact Agent Sandbox UX: + +``` +SandboxClaim (Flow B injects DF_ISSUE_NUMBER, repo, branch — unchanged) + │ + ▼ +Sandbox → Pod from the `lambda-microvm` SandboxTemplate variant + │ (bridge container; lands on a normal Auto-Mode node, NOT the kata pool) + ▼ +bridge applies a MicrovmSandbox (KRO) CR + │ + ▼ +Microvm RUNNING ── runs the SAME dark-factory-coder entrypoint (node /app/entrypoint.js) + │ + ├── bridge streams MicroVM logs → pod logs (pod Running ⇔ Microvm RUNNING) + └── pod exit / claim teardown → TerminateMicrovm +``` + +To Flow B and the user this is identical to Flow A — "a sandbox pod appeared and ran the coder" — but +the coder actually executed in the Lambda MicroVM. Log streaming is straightforward; interactive +exec/attach passthrough is **best-effort** (full fidelity would need virtual-kubelet). + +*Edit: `src/flow-d-shim.drawio` → `img/flow-d-shim.png`.* + +--- + +## D.3a — Suspend / resume (Sandbox.operatingMode → MicroVM) + +The Agent Sandbox CRD exposes `spec.operatingMode ∈ {Running, Suspended}` — the declarative +suspend/resume intent. But the ACK `Microvm` CR has **no suspend field**: its spec is create-time +only, `State` is status-only, and `suspend-microvm`/`resume-microvm` are **imperative SDK ops the ACK +controller deliberately does not reconcile**. So flipping `operatingMode` does nothing on its own — a +controller must translate intent into the SDK call. + +Flow D closes that gap with a tiny always-on **`microvm-lifecycle`** reconcile loop (a ConfigMap +script on `alpine/k8s`, same pattern as the pool-manager — **no virtual-kubelet, no new image**): + +``` +Sandbox.operatingMode: Running → Suspended : aws lambda-microvms suspend-microvm --microvm-identifier +Sandbox.operatingMode: Suspended → Running : aws lambda-microvms resume-microvm --microvm-identifier +``` + +- `` (the `microvmID`) is resolved from the `MicrovmSandbox` (KRO) status; the loop is idempotent + (stamps a `last-mode` annotation, acts only on transitions). +- The `MicrovmSandbox` is **kept** across suspend (the bridge's `preStop` detects `operatingMode: + Suspended` and skips teardown), so the VM survives suspend/resume; it's deleted only on real claim + teardown → `TerminateMicrovm`. +- Chosen over bridge `preStop` hooks alone because a reconcile loop is **robust to pod/node loss** and + resume needs no live pod. This is the open-source **Sandbox-CRD-driven** suspend/resume you get with + the MicroVM substrate. + +*Edit: `src/flow-d-suspend-resume.drawio` → `img/flow-d-suspend-resume.png`.* + +--- + +## D.4 — Future: when `lambdamicrovms` goes GA + +`lambdamicrovms` is currently **pre-GA** (`v1alpha1`), so its controller is self-managed. When it +graduates to GA upstream, **Managed ACK adopts it automatically** — the self-managed chart is deleted +and the `MicrovmSandbox` RGD is **unchanged** (it references the same `lambdamicrovms.services.k8s.aws` +CRDs regardless of who runs the controller). The design deliberately keeps the RGD independent of the +controller install method so this migration is a one-line addon removal. diff --git a/docs/dark-factory/flow-d-coder-in-microvm-design.md b/docs/dark-factory/flow-d-coder-in-microvm-design.md new file mode 100644 index 00000000..05101b87 --- /dev/null +++ b/docs/dark-factory/flow-d-coder-in-microvm-design.md @@ -0,0 +1,214 @@ +# Flow D — Running the Coder *inside* the Lambda MicroVM (design) + +**Status:** design / spike — NOT implemented. Written after proving the Flow D **substrate** +end-to-end and discovering that running the actual coder in the VM is an application +re-architecture, not a wiring task. + +## TL;DR + +The Flow D **substrate + lifecycle is proven live**: a `darkfactory-lambda` GitHub issue → +Argo sensor → `df-run` claims the Lambda warm pool → the bridge calls `RunMicrovm` → a real +Lambda MicroVM reaches **RUNNING** in AWS → `suspend`/`resume`/`terminate` are wired → the VM +is terminated on teardown (verified, zero orphans). + +What is **NOT** done: the coder does not actually *execute* inside that MicroVM, so no PR is +produced. That is because the one-shot `dark-factory-coder` and the Lambda MicroVM +snapshot/hook execution model are **fundamentally different execution shapes**. Closing the gap +requires re-architecting the coder, plus VPC/Bifrost networking. This doc specifies that work +so it can be decided deliberately. + +## Why it isn't just wiring — the execution-model mismatch + +| | Kata coder (Flow B, works today) | Lambda MicroVM model | +| --- | --- | --- | +| Shape | **one-shot batch process**: `node entrypoint.js` runs clone→agent→push→PR, then exits | **long-lived HTTP service** that is *snapshotted* at build, *resumed* per session | +| Duration | 5–15 min per run | per-request; the `run` lifecycle hook has a **30s timeout** ("keep it short — on the critical path") | +| Trigger | pod start + `DF_ISSUE_NUMBER` env injected by the SandboxClaim | build-time `ready`/`validate` hooks; per-instance `run` hook receives `runHookPayload` as the HTTP request body | +| Secrets/context | files projected into the pod (`/etc/secrets/gh-token`, `bifrost-api-key`) + `DF_*` env | `runHookPayload` — a **Kubernetes `SecretKeyReference`** on the `Microvm` CR, delivered as the `/run` hook body (≤16 KB); image must set `hooks.microvmHooks.run: ENABLED` | +| Network | in-cluster: reaches Bifrost by ClusterIP `172.20.181.17:8080`; git/gh over public :443 | runs **outside the cluster network**; only `INTERNET_EGRESS` by default; cannot reach a ClusterIP; VPC reach needs an egress **network connector** | + +The killer facts (verified against `mmeckes/lambdamicrovms-controller` docs + the live `aws +lambda-microvms`/`lambda-core` CLIs, 2026-08-03): + +1. **`/run` hook = 30s timeout.** A 5–15 min coder run cannot happen *in* the hook. +2. **The intended app model is request/response** (02-developer-handoff: RunMicrovm → mint + auth token → HTTP request → response → terminate) — not a batch job. +3. **`runHookPayload` is delivered as an HTTP body to a `/run` endpoint the app must SERVE** — + NOT an env var and NOT a mounted file. (An earlier attempt at an env/file boot-shim was + wrong and is discarded.) +4. So the coder must be **wrapped in a long-running HTTP server** that starts the coding work + asynchronously — the coder's current `entrypoint.js` is not written that way. + +## Proposed design (async `/run` pattern) + +Keep the coder *logic* (`entrypoint.js`) intact; change how it is *invoked*. + +``` +build: MicrovmImage (FROM arm64 dark-factory-coder + a thin HTTP wrapper) + hooks.port: 8080 + microvmImageHooks.ready: server up → safe to snapshot + microvmHooks.run: ENABLED (30s), suspend/resume/terminate ENABLED + +run: controller delivers runHookPayload (Secret {ghToken, bifrostKey, bifrostUrl(NLB), + issueNumber, repo, branch, baseBranch, title}) as the /run body + → wrapper writes /etc/secrets/{gh-token,bifrost-api-key} + exports DF_*/BIFROST_URL + → wrapper spawns `node entrypoint.js` in the BACKGROUND, returns 200 within 30s + → coder does clone→agent→push→PR async (many minutes) + +observe: df-run's existing `await-coder` step ALREADY polls GitHub for the PR head — reuse it + verbatim; it doesn't care whether the coder ran in Kata or a MicroVM. + +teardown: suspend/resume/terminate hooks best-effort flush; bridge TerminateMicrovm on exit. +``` + +### Components to build + +1. **HTTP wrapper + artifact** (`coder-microvm/`): a small server (`server.js`) exposing + `ready`, `run`, `suspend`, `resume`, `terminate` on :8080; `run` materializes the payload + into the coder's existing file/env contract and background-spawns `entrypoint.js`. Dockerfile + `FROM 940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:-arm64`. Zip + (Dockerfile + server.js) → S3, per the controller's `ci/package-artifact.sh` format. + *(Supersedes the placeholder `microvm-entry.js` listener that only existed to get the image + to CREATED.)* + +2. **MicrovmImage: enable hooks** (RGD `templates/image/10-rgd-and-image.yaml`): add + `hooks.port: 8080`, `microvmImageHooks.ready: ENABLED`, `microvmHooks.run/suspend/resume/ + terminate: ENABLED`. Without `run: ENABLED` the payload is silently never delivered. + +3. **VPC egress connector** (bootstrap Job — honest: *GitOps-provisioned, not continuously + reconciled*; no ACK/Crossplane API exists for `lambda-core` connectors). Committed + find-or-create Job modeled on `06-securityagent-bootstrap.yaml`: + `aws lambda-core get/create-network-connector` with + `VpcEgressConfiguration={SubnetIds:[hub subnets], SecurityGroupIds:[sg], NetworkProtocol:IPv4}` + → writes the connector ARN to a ConfigMap the MicrovmImage `egressNetworkConnectors` reads. + IAM: the bootstrap/capability role needs `lambda-core:*NetworkConnector*` + the EC2 ENI perms + Lambda uses to provision ENIs. **Caveat:** if the connector is deleted out-of-band, nothing + self-heals until the Job re-runs (not a controller). + +4. **Bifrost VPC-reachable** (internal NLB — this part *is* declarative): a `Service + type=LoadBalancer` with `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` + + `nlb-target-type` in the bifrost chart, reconciled by the AWS Load Balancer Controller. The + MicroVM (via the egress connector) reaches Bifrost at the NLB's stable VPC address on :8080. + (Bifrost's pod IP `10.0.x.x` is in-VPC and reachable via the connector, but ephemeral — the + NLB gives a stable target. Its ClusterIP `172.20.x.x` is NOT routable from a VPC ENI.) + **Shared-infra change — needs owner sign-off.** + +5. **runHookPayload Secret + Microvm wiring**: the bridge (or a per-session step) writes a + Secret with the payload key and the `Microvm`/RunMicrovm references it as + `runHookPayload: {name, key}`. Since it's a SecretKeyReference the **controller** delivers it + — confirm whether the imperative `RunMicrovm` path the bridge uses accepts the same, or + whether this session should create a short-lived `Microvm` CR instead. + +6. **Security-group rules**: allow the connector ENIs → Bifrost NLB on :8080. + +## Open questions for review + +- **Async vs request-driven?** Background-spawn (df-run polls for the PR, minimal coder change) + vs. the reference's request/response model (bridge sends an HTTP "code this" request + waits; + needs the auth-token path). Background-spawn reuses `await-coder` and is less invasive. +- **Imperative RunMicrovm vs a `Microvm` CR per session?** `runHookPayload` being a + SecretKeyReference is controller-delivered; the current bridge calls `aws run-microvm` + imperatively. Decide whether per-session VMs become short-lived `Microvm` CRs (declarative + payload delivery) or stay imperative (verify the CLI accepts an inline/secret payload). +- **Cost:** the VPC egress connector provisions ENIs; the internal NLB is an hourly resource. + Both are ongoing while Flow D is enabled. +- **Is in-VM coder even required for the goal?** The substrate is a valid deliverable on its + own (a second sandbox substrate). Running the coder in it is the "make it actually code" step + — worth confirming it's in scope before the re-architecture. + +## 2026-08-03 build attempt — where it got to + the confirmed blocker + +Built and deployed the Bedrock-direct async design end-to-end. Live results: + +- ✅ **lambda-coder artifact + image**: `examples/dark-factory/coder-microvm/` (hook-server.js + + Dockerfile FROM the arm64 coder + the USE_BEDROCK entrypoint branch). MicrovmImage rebuilt + to **UPDATED** with `hooks` enabled; exec role has `bedrock:InvokeModel*`. Verified the + hook-server runs in the VM — CloudWatch shows `[hook-server] listening on :8080 (lambda-coder, + Bedrock-direct)`. +- ✅ **Bridge payload**: builds JSON (issue ctx + GitHub token + region) with python3 (node is + absent in the aws-cli image) and passes `--run-hook-payload` on `run-microvm`. VM launches + RUNNING with the payload; no bridge crash. +- ❌ **BLOCKER: the `/run` hook never fires** → the coder never starts in the VM → no PR. + CloudWatch shows the server `listening` but never logs the `/run` handling / background-spawn. + +**Confirmed root cause:** `runHookPayload` is a **`SecretKeyReference`, "not a literal"** — the +docs + the 02-developer-handoff example deliver it via the **declarative `Microvm` CR** +(`runHookPayload: {name, key}` → the self-managed controller reads the Secret and drives the +`/run` hook). The imperative `run-microvm --run-hook-payload ""` CLI path the bridge uses +does **not** invoke `/run` (VM reaches RUNNING but the hook is silent). Two things also worth +noting from the reference: (a) the intended session model is the CLIENT minting an auth token and +sending an HTTP request to the VM endpoint with `X-aws-proxy-auth` (request/response), and (b) the +`/run` hook is service-internal on VM start. + +**Correct path (next):** make the per-session VM a **`Microvm` CR** (declarative), not a bridge +CLI call: +- bridge (or a per-session step) writes a **Secret** with the payload key, then creates a + `Microvm` CR referencing it (`imageIdentifierRef`, `executionRoleRef`, `runHookPayload:{name,key}`, + `idlePolicy`), and reads back `status.microvmID` for the lifecycle annotation. +- the self-managed lambdamicrovms controller reconciles it and delivers the payload to `/run`, + which background-spawns the coder. +- teardown = delete the `Microvm` CR (controller terminates), replacing the imperative + TerminateMicrovm. +This trades the imperative bridge for the declarative CR path the payload mechanism actually +requires — and it's MORE GitOps-faithful. Est: bridge rewrite (CR create/delete instead of CLI) ++ a per-session Secret; the hook-server/artifact/image/IAM/Bedrock pieces are already done and verified. + +## 2026-08-03 (later) — declarative Microvm CR path: reconciles + VM runs, /run still silent + +Switched the bridge from the imperative `run-microvm` CLI to the **declarative `Microvm` CR** +path (write payload Secret → create `Microvm` CR with `runHookPayload:{name,key}` → controller +reconciles → delete CR on teardown). Verified working: +- ✅ Bridge creates the Secret + `Microvm` CR (`mvm-`); RBAC for microvms+secrets added. +- ✅ Controller reconciles it: CR `state=RUNNING`, `ACK.ResourceSynced=True`, `status.microvmID` + populated, annotated on the Sandbox. Deleting the CR cleanly terminates the VM (0 orphans). +- ✅ MicrovmImage is v2.0, `UPDATED`, `hooks` present (run/ready/suspend/resume/terminate). +- ❌ **STILL no `/run` output**: CloudWatch `/aws/lambda/microvms/coder-image` shows the build-time + `[hook-server] listening on :8080` but ZERO runtime events after the VM starts — the coder never + logs, no PR. The `/run` hook is not producing coder execution we can observe. + +**What's ruled out:** payload delivery mechanism (now declarative CR, the documented path), image +hooks (present, v2.0 built), IAM (bedrock on exec role), bridge crash (restarts=0), YAML (renders +clean). **What's NOT yet proven:** that the service actually invokes `/run` against the hook-server, +and that hook-server's `/run` handler + background-spawn + Bedrock call execute. Can't see inside +the VM beyond CloudWatch (which is empty at runtime) — needs either (a) the VM's runtime logs routed +somewhere visible, (b) hitting the VM endpoint directly with an auth token (X-aws-proxy-auth) to +probe the hook-server, or (c) the controller/service confirming the run-hook HTTP call + its response. +This is the current debugging frontier — the substrate, image, CR path, and teardown all work; the +open question is purely whether/how the `/run` hook reaches the in-VM hook-server and why it emits +no logs. + +## 2026-08-03 E2E run #106 — full chain works to /run; 2 pinpointed gaps + +Ran a clean GH-issue E2E and **probed the VM directly** (minted an auth token, hit the endpoint). +Stage-by-stage: issue → label → workflow → bridge claim → Microvm CR (`mvm-106`) → VM RUNNING with +endpoint — all ✅. Then the decisive probes against the live VM: +- `GET https:///` (X-aws-proxy-auth) → `{"status":"ok","path":"/"}` → **hook-server is + ALIVE and reachable at runtime.** +- `POST /run` → `{"status":"started"}` → **the /run handler works and background-spawns the coder.** + +So the entire chain — including the hook-server and its /run→coder-spawn — is functional. The two +remaining gaps are now precisely isolated: + +1. **The service does not auto-invoke `/run` on launch.** After RunMicrovm/Microvm-CR reconcile, the + run hook is not called automatically — I had to POST /run manually to start the coder. Either the + run hook fires on a trigger we're not hitting, or the payload/hook wiring needs a specific field to + auto-fire. (auth-token minting: `create-microvm-auth-token --expiration-in-minutes N --allowed-ports + port=8080`; token is at `.authToken.X-aws-proxy-auth`.) +2. **Runtime logs don't reach CloudWatch.** `logging.cloudWatch.logGroup` on the image captures BUILD + logs only; after the VM runs, `/aws/lambda/microvms/coder-image` has 0 runtime events even though the + hook-server clearly runs (proven by the probe). This blinded every prior run — need to wire runtime + stdout/stderr to CloudWatch (or read it another way) to observe the coder. + +Both are now concrete, small-surface problems (a hook-trigger config + a log-routing config), NOT +architecture. The substrate, image+hooks, Bedrock exec role, declarative Microvm CR path, payload +delivery, hook-server, /run→coder-spawn, and clean CR-delete teardown are all verified working. + +## What exists today (so nothing is lost) + +- Substrate live: RGD Active, S3 bucket, build/exec roles, **MicrovmImage CREATED (v1.0)**, + bridge launches/terminates a real MicroVM from a `darkfactory-lambda` issue. +- All the substrate + bridge fixes are committed on `flow-d-lambda-microvm-sandbox` (container + named `coder`, aws-cli v2 image, kubectl fetch, API-server + Pod Identity egress, + downward-API SANDBOX_NAME, microvmSuspend on, `darkfactory-lambda` label). +- The **placeholder** code artifact (`microvm-entry.js` listener) is what's in S3 today — it + only proved the image builds; it must be replaced per §1 above. diff --git a/examples/dark-factory/coder-microvm/Dockerfile b/examples/dark-factory/coder-microvm/Dockerfile new file mode 100644 index 00000000..3254805a --- /dev/null +++ b/examples/dark-factory/coder-microvm/Dockerfile @@ -0,0 +1,32 @@ +# Flow D — lambda-coder image: the dark-factory coder wrapped for the Lambda MicroVM +# snapshot/hook runtime. +# +# FROM the arm64 dark-factory-coder (Lambda MicroVM is ARM_64-only) — it carries +# entrypoint.js + the toolchain (git, node, claude-code). We add ONLY the hook server +# that adapts the one-shot coder to the MicroVM lifecycle (see hook-server.js): the +# /run hook background-spawns entrypoint.js with USE_BEDROCK=1 so the coder calls +# Bedrock directly via the MicroVM execution role — no Bifrost / EKS-network path. +# +# The image is built by Lambda from a code-artifact ZIP (this Dockerfile + hook-server.js) +# in S3 — NOT pushed to ECR as a normal image. The base coder image IS pulled from ECR +# during that build (the MicrovmImage build role keeps ecr:Get*). +ARG CODER_IMAGE=940019131157.dkr.ecr.us-west-2.amazonaws.com/dark-factory-coder:v0.2.5-arm64 +FROM ${CODER_IMAGE} + +WORKDIR /app + +# The hook server (serves ready/validate/run/suspend/resume/terminate on :8080). +COPY hook-server.js /app/hook-server.js + +# Ship the UPDATED coder over the one baked into the ECR base. The base image's +# entrypoint.js predates the USE_BEDROCK branch; overlaying it here means the +# MicrovmImage build (which pulls the ECR base) gets the Bedrock-capable coder +# WITHOUT a separate ECR rebuild+push. Keep in sync with examples/dark-factory/coder/entrypoint.js. +COPY entrypoint.js /app/entrypoint.js + +# Lambda MicroVM snapshots the process started here. hooks.port on the MicrovmImage +# must match this (8080). The server stays up (long-running) so the VM isn't idle- +# suspended mid coder-run — the Microvm idlePolicy.maxIdleDurationSeconds is set +# longer than a coder run. +EXPOSE 8080 +CMD ["node", "/app/hook-server.js"] diff --git a/examples/dark-factory/coder-microvm/hook-server.js b/examples/dark-factory/coder-microvm/hook-server.js new file mode 100644 index 00000000..0cad32fd --- /dev/null +++ b/examples/dark-factory/coder-microvm/hook-server.js @@ -0,0 +1,118 @@ +// Flow D — Lambda MicroVM hook server (the lambda-coder wrapper). +// +// Lambda MicroVM is a snapshot/hook runtime: the platform builds an image by +// starting THIS process and snapshotting it once the `ready` hook says "go", then +// resumes that snapshot per session and calls the `run` hook with the session's +// runHookPayload as the request body. Hooks are HTTP endpoints we serve on :8080. +// +// The dark-factory coder (entrypoint.js) is a ONE-SHOT batch job (clone → agent → +// push → PR, 5-15 min). It cannot run inside the 30s run hook, so /run just +// materializes the payload into the coder's file/env contract and BACKGROUND-SPAWNS +// entrypoint.js, then returns 200 immediately. The coder runs async; df-run's +// await-coder step polls GitHub for the PR (same as Kata). The VM stays alive because +// idlePolicy.maxIdleDurationSeconds > a coder run (idle = no inbound traffic). +// +// LLM: USE_BEDROCK=1 → entrypoint.js calls Bedrock DIRECTLY via the MicroVM execution +// role (no Bifrost / EKS-network dependency). See docs/dark-factory/flow-d-coder-in-microvm-design.md. +// +// KEPT MINIMAL: this is the exact shape that built cleanly (v2.0). The /run handler +// stays trivial and synchronous so the build's ready-hook completes fast. Observability +// is via direct endpoint probes, not a /status route (adding one correlated with a +// hung ready-hook build on the pre-GA controller). + +const http = require("http"); +const fs = require("fs"); +const { spawn } = require("child_process"); + +const PORT = parseInt(process.env.HOOKS_PORT || "8080", 10); +const SECRETS_DIR = "/tmp/secrets"; +// Run-id of the coder invocation currently in flight (or last completed). NOT a plain +// boolean: the VM is SUSPENDED after the first PR and RESUMED for a fix round, and the +// resumed process keeps its in-memory state — a one-shot `coderStarted=true` guard, +// frozen in the snapshot, made the resumed VM ignore the fix round's /run entirely (the +// coder never re-ran; the fix round reported "done" on the old sha). Instead we key on a +// per-invocation run-id (issue + iterate-note hash): a /run whose id differs from the +// one in flight starts a fresh coder (this is a new round after a resume); a /run that +// repeats the current id is a duplicate webhook and is ignored. +let currentRunId = null; +let coderRunning = false; + +function runIdOf(d) { + const note = d.iterateNoteB64 || d.iterateNote || ""; + // Cheap stable hash of issue+note so a fix round (new note) => new id => re-run. + let h = 0; const s = `${d.issueNumber || ""}:${note}`; + for (let i = 0; i < s.length; i++) { h = ((h << 5) - h + s.charCodeAt(i)) | 0; } + return `${d.issueNumber || "?"}#${(h >>> 0).toString(36)}`; +} + +function startCoder(payload) { + let d = {}; + try { d = JSON.parse(payload || "{}"); } catch (e) { console.log("[hook-server] payload not JSON:", e.message); } + const rid = runIdOf(d); + if (rid === currentRunId) { console.log(`[hook-server] /run duplicate for ${rid} — ignoring`); return; } + if (coderRunning) { console.log(`[hook-server] /run for ${rid} but ${currentRunId} still running — ignoring`); return; } + const isRerun = currentRunId !== null; // a prior run existed => this is a post-resume fix round + currentRunId = rid; + coderRunning = true; + // Truncate the coder log on each new run. Otherwise the previous round's + // "done — PR opened on " line lingers and the bridge's /logs grep matches it + // instantly, suspending the VM before the fix-round coder has done anything. + try { fs.writeFileSync("/tmp/coder.log", ""); } catch {} + console.log(`[hook-server] /run accepted run-id=${rid}${isRerun ? " (post-resume re-run)" : ""}`); + fs.mkdirSync(SECRETS_DIR, { recursive: true, mode: 0o700 }); + if (d.ghToken) fs.writeFileSync(`${SECRETS_DIR}/gh-token`, d.ghToken, { mode: 0o400 }); + const env = { + ...process.env, + USE_BEDROCK: "1", + // MicroVM rootfs is read-only + there's no /workspace volume mount (unlike Kata, + // where the operator mounts a writable workspace). entrypoint.js mkdir's + // ${WORKSPACE}/artifacts and clones there, so point it at the writable tmpfs — + // else it crashes EACCES on /workspace/artifacts before doing any work. + WORKSPACE: "/tmp/workspace", + GH_TOKEN_PATH: `${SECRETS_DIR}/gh-token`, + AWS_REGION: d.region || process.env.AWS_REGION || "us-west-2", + DF_ISSUE_NUMBER: d.issueNumber ? String(d.issueNumber) : "", + DF_REPO: d.repo || "", + DF_BRANCH: d.branch || (d.issueNumber ? `df/issue-${d.issueNumber}` : ""), + DF_BASE_BRANCH: d.baseBranch || "main", + DF_ISSUE_TITLE: d.issueTitle || "", + }; + // Fix round (df-iterate): the bridge folds the human's change request into the + // payload as iterateNoteB64. Without this, a Lambda fix round re-runs the coder + // with NO instructions → it sees the PR already open and reports "done" on the + // old sha with zero changes (the Kata path injects DF_ITERATE_NOTE_B64 as claim + // env; the MicroVM has no claim env, so it must ride in on the runHookPayload). + if (d.iterateNoteB64) env.DF_ITERATE_NOTE_B64 = d.iterateNoteB64; + if (d.iterateNote) env.DF_ITERATE_NOTE = d.iterateNote; + if (d.model) env.CODER_MODEL = d.model; + console.log(`[hook-server] /run → spawning coder for issue #${env.DF_ISSUE_NUMBER} repo=${env.DF_REPO}`); + // Capture the coder's stdout+stderr to /tmp/coder.log so /logs can return it — + // runtime CloudWatch routing doesn't work on this runtime, and there's no shell, + // so this file (read over the HTTP token) is the ONLY way to see what the coder did. + const logFd = fs.openSync("/tmp/coder.log", "a"); + const child = spawn("node", ["/app/entrypoint.js"], { env, stdio: ["ignore", logFd, logFd], detached: true }); + child.unref(); + child.on("error", (e) => { coderRunning = false; try { fs.appendFileSync("/tmp/coder.log", "SPAWN-ERROR: " + e.message + "\n"); } catch {} }); + // Clear the in-flight flag when the coder exits so a resumed VM's next /run (fix round) + // is accepted. `unref`'d + detached, but we still get 'exit' while this process lives. + child.on("exit", (code) => { coderRunning = false; console.log(`[hook-server] coder run-id=${currentRunId} exited code=${code}`); }); +} + +const server = http.createServer((req, res) => { + let body = ""; + req.on("data", (c) => { body += c; }); + req.on("end", () => { + const ok = (o) => { res.writeHead(200, { "Content-Type": "application/json" }); res.end(JSON.stringify(o || { status: "ok" })); }; + switch (req.url) { + case "/ready": return ok({ status: "ready" }); + case "/validate": return ok({ status: "valid" }); + case "/run": startCoder(body); return ok({ status: "started" }); + case "/logs": { let l=""; try { l=fs.readFileSync("/tmp/coder.log","utf8"); } catch {} return ok({ status:"ok", runId: currentRunId, running: coderRunning, log: l.slice(-6000) }); } + case "/suspend": return ok({ status: "suspended" }); + case "/resume": return ok({ status: "resumed" }); + case "/terminate": return ok({ status: "terminated" }); + default: return ok({ status: "ok", path: req.url }); + } + }); +}); +server.listen(PORT, () => console.log(`[hook-server] listening on :${PORT} (lambda-coder, Bedrock-direct)`)); diff --git a/examples/dark-factory/coder/entrypoint.js b/examples/dark-factory/coder/entrypoint.js index a8184e03..5b43fa31 100644 --- a/examples/dark-factory/coder/entrypoint.js +++ b/examples/dark-factory/coder/entrypoint.js @@ -226,28 +226,21 @@ http.createServer((cReq,cRes)=>{ } function runCoder(repoDir) { - // Bifrost is an Anthropic-compatible gateway. Point Claude Code at its - // /anthropic route via ANTHROPIC_BASE_URL + ANTHROPIC_API_KEY. Do NOT set - // CLAUDE_CODE_USE_BEDROCK — that flag makes Claude Code use the AWS Bedrock - // SDK directly (needs AWS creds in the VM, which we deliberately withhold) - // and ignores ANTHROPIC_BASE_URL. Bifrost auth is optional on this platform, - // so the key may be absent; send a placeholder so the CLI doesn't prompt. - const key = readSecret(BIFROST_KEY_PATH) || "bifrost"; - // Route through the localhost UA-shim (see startBifrostUaShim) so Bifrost - // doesn't apply its broken claude-cli request transform. - const base = startBifrostUaShim(`${BIFROST_URL.replace(/\/+$/, "")}/anthropic`); - const env = { + // Two LLM transports, selected by USE_BEDROCK: + // - Kata (Flow B, default): Bifrost gateway. The Kata VM is credential-less + + // in-cluster, so it reaches models through Bifrost's /anthropic route (which + // also gives centralized Langfuse observability). CLAUDE_CODE_USE_BEDROCK is + // deliberately UNSET here (it would make the CLI use the Bedrock SDK directly + // and ignore ANTHROPIC_BASE_URL). + // - Lambda MicroVM (Flow D): USE_BEDROCK=1. A MicroVM runs OUTSIDE the cluster + // network and can't reach Bifrost's ClusterIP; forcing it back in-cluster + // needed a VPC connector + internal NLB. Instead the MicroVM's EXECUTION ROLE + // grants bedrock:InvokeModel, so Claude Code calls Bedrock directly over public + // egress — no EKS network dependency. Trade-off: these calls bypass Bifrost's + // Langfuse telemetry (documented in flow-d-coder-in-microvm-design.md). + const useBedrock = /^(1|true|yes)$/i.test(process.env.USE_BEDROCK || ""); + const baseEnv = { ...process.env, - ANTHROPIC_BASE_URL: base, - ANTHROPIC_API_KEY: key, - // Bifrost maps model ALIASES → Bedrock model IDs. Claude Code's default - // model name (e.g. claude-sonnet-4) isn't a Bifrost alias and returns - // "provided model identifier is invalid" (400). Use the platform's Bifrost - // alias (verified: 'claude-sonnet' → us.anthropic.claude-sonnet-4-5). Set - // both the primary and the small/fast model so the CLI never falls back to - // an unknown identifier. - ANTHROPIC_MODEL: process.env.CODER_MODEL || "claude-sonnet", - ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_MODEL || "claude-sonnet", // The sandbox runs with readOnlyRootFilesystem, so $HOME (/home/node) is NOT // writable. Claude Code writes its config, session state, and — critically — // per-invocation SHELL SNAPSHOT files that its Bash tool sources before every @@ -262,9 +255,41 @@ function runCoder(repoDir) { // Non-interactive: never open a browser / prompt for login in headless mode. CI: "1", }; - fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); - delete env.CLAUDE_CODE_USE_BEDROCK; - console.log(`[coder] LLM: base=${base} model=${env.ANTHROPIC_MODEL}`); + let env; + if (useBedrock) { + // Bedrock-direct: creds come from the MicroVM execution role (Pod Identity / + // instance creds); the CLI uses the Bedrock SDK. Model must be a real Bedrock + // model ID (NOT a Bifrost alias). AWS_REGION comes from the runHookPayload/env. + env = { + ...baseEnv, + CLAUDE_CODE_USE_BEDROCK: "1", + AWS_REGION: process.env.AWS_REGION || process.env.CODER_REGION || "us-west-2", + ANTHROPIC_MODEL: process.env.CODER_MODEL || "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_SMALL_MODEL || process.env.CODER_MODEL || "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + }; + fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); + console.log(`[coder] LLM: Bedrock-direct region=${env.AWS_REGION} model=${env.ANTHROPIC_MODEL}`); + } else { + // Bifrost is an Anthropic-compatible gateway. Route through the localhost + // UA-shim so Bifrost doesn't apply its broken claude-cli request transform. + // Bifrost auth is optional; send a placeholder so the CLI doesn't prompt. + const key = readSecret(BIFROST_KEY_PATH) || "bifrost"; + const base = startBifrostUaShim(`${BIFROST_URL.replace(/\/+$/, "")}/anthropic`); + env = { + ...baseEnv, + ANTHROPIC_BASE_URL: base, + ANTHROPIC_API_KEY: key, + // Bifrost maps model ALIASES → Bedrock model IDs. Claude Code's default + // model name (e.g. claude-sonnet-4) isn't a Bifrost alias and returns + // "provided model identifier is invalid" (400). Use the platform's Bifrost + // alias (verified: 'claude-sonnet' → us.anthropic.claude-sonnet-4-5). + ANTHROPIC_MODEL: process.env.CODER_MODEL || "claude-sonnet", + ANTHROPIC_SMALL_FAST_MODEL: process.env.CODER_MODEL || "claude-sonnet", + }; + fs.mkdirSync("/tmp/coder-home/.claude", { recursive: true }); + delete env.CLAUDE_CODE_USE_BEDROCK; + console.log(`[coder] LLM: Bifrost base=${base} model=${env.ANTHROPIC_MODEL}`); + } // Inherit stdio so the coder CLI's own output + errors stream into the pod // logs (kubectl logs), instead of being swallowed by execFileSync's exception. const opts = { cwd: repoDir, env, stdio: "inherit", maxBuffer: 64 * 1024 * 1024 }; diff --git a/gitops/addons/bootstrap/default/addons.yaml b/gitops/addons/bootstrap/default/addons.yaml index 282961aa..84369c0b 100644 --- a/gitops/addons/bootstrap/default/addons.yaml +++ b/gitops/addons/bootstrap/default/addons.yaml @@ -411,6 +411,27 @@ agent-sandbox: coderTemplate: bifrostUrl: '{{default "http://bifrost.bifrost.svc.cluster.local:8080" (index .metadata.annotations "bifrost_url")}}' +# agent-sandbox-lambda — Flow D. A SECOND, opt-in Agent-Sandbox substrate: the coder +# runs in an AWS Lambda MicroVM (Firecracker) instead of a Kata pod. KRO builds the +# platform image ONCE (MicrovmImage + build/exec IAM + S3 + logs via ACK GA +# controllers); a shim (bridge pod + lifecycle controller) drives the per-session VM +# (RunMicrovm/suspend/resume/TerminateMicrovm) imperatively. Same namespace + operator +# as agent-sandbox; only ADDS the MicroVM path. DORMANT until the per-cluster overlay +# flips microvm.enabled=true (needs Managed KRO+ACK, the self-managed ack-lambdamicrovms +# controller, and a published arm64 coder artifact). Hub-only, sync-wave 2. +agent-sandbox-lambda: + enabled: true + namespace: agent-sandbox-system + defaultVersion: '0.1.0' + path: 'gitops/addons/charts/agent-sandbox-lambda' + annotationsAppSet: + argocd.argoproj.io/sync-wave: '2' + alwaysSelector: + matchExpressions: + - key: environment + operator: In + values: ['control-plane'] + # kata-deploy — installs the Kata runtime (containerd handlers for kata-clh / # kata-qemu) on the tainted kata MNG nodes. Separate app (not a subchart dep of # agent-sandbox) so ArgoCD pulls the upstream OCI chart directly. Gated by @@ -468,6 +489,49 @@ kata-deploy: runtimeClasses: enabled: false +# ack-lambdamicrovms (Flow D) — SELF-MANAGED ACK controller for the pre-GA +# Lambda MicroVM service (lambdamicrovms.services.k8s.aws: MicrovmImage + Microvm). +# Self-managed because Managed ACK (EKS Capability) only bundles GA-upstream +# controllers, and lambdamicrovms is still v1alpha1. Managed ACK (GA iam/s3) and +# this self-managed controller COEXIST (different CRD groups). When lambdamicrovms +# goes GA, delete this addon and Managed ACK adopts it — the KRO RGD is unchanged. +# +# DISABLED by default: Flow D is dormant until a cluster opts in. Hub-only, like +# the rest of the sandbox substrate. Installs the OCI chart +# oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart. Sync-wave 0 so the +# CRDs + controller are up before the agent-sandbox chart's MicrovmSandbox RGD +# (wave 2) references them. +ack-lambdamicrovms: + enabled: true + namespace: ack-system + chartName: lambdamicrovms-chart + defaultVersion: '0.1.1' + chartRepository: 'public.ecr.aws/aws-controllers-k8s' + annotationsAppSet: + argocd.argoproj.io/sync-wave: '0' + # Hub-only: the MicroVM substrate lives with the rest of the sandbox capability + # on the build plane. alwaysSelector pins generation to environment: + # control-plane regardless of useSelectors — never generated on a spoke. + alwaysSelector: + matchExpressions: + - key: environment + operator: In + values: ['control-plane'] + # ACK chart reads these at the ROOT of values (chart v0.1.1). The controller + # reconciles the Microvm + MicrovmImage CRDs; it uses EKS Pod Identity for AWS + # auth (empty SA annotations — the pod-identity association is created out of + # band / by the platform), so no IRSA role-arn annotation is set here. + valuesObject: + aws: + region: '{{default "us-west-2" (index .metadata.annotations "aws_region")}}' + serviceAccount: + create: true + name: ack-lambdamicrovms-controller + # Cluster-scoped install so the single MicrovmSandbox RGD (any namespace) can + # create Microvm/MicrovmImage CRs the controller reconciles. + installScope: cluster + deletionPolicy: delete + # Dark Factory (Flow B) — Argo Workflows that turn a GitHub issue into a PR by # claiming the Flow A warm pool. Hub-only (co-located with Argo Workflows + the # sandbox pool); sync-wave 3 so it lands after the agent-sandbox capability diff --git a/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml b/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml new file mode 100644 index 00000000..d5da419c --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/Chart.yaml @@ -0,0 +1,14 @@ +apiVersion: v2 +name: agent-sandbox-lambda +description: >- + Flow D — Lambda MicroVM substrate for the Agent Sandbox capability. A second, + opt-in execution substrate alongside the Kata micro-VM chart (agent-sandbox): + the coder runs in an AWS Lambda MicroVM (Firecracker) instead of a Kata node. + KRO builds the platform image ONCE (MicrovmImage + build/exec IAM + S3 artifact + + CloudWatch logs, via ACK GA controllers); a lightweight shim (bridge pod + + lifecycle controller) drives the per-session VM (RunMicrovm / suspend / resume / + TerminateMicrovm) imperatively via the AWS SDK. Same dark-factory-coder (arm64) + image + same Agent Sandbox UX as Flow A. Disabled by default (microvm.enabled). +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl b/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl new file mode 100644 index 00000000..dc54887e --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/_helpers.tpl @@ -0,0 +1,28 @@ +{{/* +Common labels applied to every resource this chart renders. Kept under the +app.kubernetes.io/name "agent-sandbox" (same capability, Lambda substrate) so +Flow D resources associate with the Agent Sandbox capability; the chart name +distinguishes them. +*/}} +{{- define "agent-sandbox.labels" -}} +app.kubernetes.io/name: agent-sandbox +app.kubernetes.io/component: lambda-microvm +app.kubernetes.io/part-of: open-agent-platform +app.kubernetes.io/managed-by: {{ .Release.Service }} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version }} +{{- end -}} + +{{/* +Selector labels (stable subset used by controllers). +*/}} +{{- define "agent-sandbox.selectorLabels" -}} +app.kubernetes.io/name: agent-sandbox +app.kubernetes.io/component: lambda-microvm +{{- end -}} + +{{/* +The namespace the capability runs in (must match the Kata agent-sandbox chart). +*/}} +{{- define "agent-sandbox.namespace" -}} +{{- default "agent-sandbox-system" .Values.namespace -}} +{{- end -}} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml new file mode 100644 index 00000000..55201363 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/image/10-rgd-and-image.yaml @@ -0,0 +1,217 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — MicrovmSandbox ResourceGraphDefinition (KRO). + +ONE composite CRD (kro.run) that ties together ALL the Lambda MicroVM primitives so +a consumer (the lambda-microvm SandboxTemplate bridge, or any agent) creates a single +`MicrovmSandbox` and gets the whole substrate: + + MicrovmSandbox -> S3 Bucket (s3.services.k8s.aws) image codeArtifact store + IAM Role (build) (iam.services.k8s.aws) MicrovmImage.buildRoleARN + IAM Role (exec) (iam.services.k8s.aws) Microvm.executionRoleARN + MicrovmImage (lambdamicrovms.services.k8s.aws) platform-owned image + Microvm (lambdamicrovms.services.k8s.aws) app-owned instance + +OWNERSHIP SPLIT is expressed IN THE SCHEMA: + spec.image.* — PLATFORM-owned (base image, code artifact) — set once per image + spec.run.* — APP-owned (per-claim instance: idle policy) + +NETWORK: Lambda MicroVMs have PUBLIC internet egress by DEFAULT, so no network +connectors are attached here (the coder only needs outbound git/gh/registry, like +Flow A). Ingress connectors (AWS-managed, inbound HTTPS) and VPC egress connectors +(created out-of-band via `aws lambda-core create-network-connector`) are optional +add-ons a future variant can wire in; they are intentionally omitted from v1. + +CONTROLLER SPLIT: the s3/iam resources are reconciled by MANAGED ACK (GA controllers); +MicrovmImage/Microvm by the SELF-MANAGED lambdamicrovms controller (pre-GA). The RGD is +identical regardless of who runs the controllers — when lambdamicrovms goes GA and +Managed ACK adopts it, THIS FILE DOES NOT CHANGE. + +Gated behind microvm.enabled so Flow D stays dormant until a cluster opts in. Requires +the Managed KRO capability (kro.run) + the ACK controllers to be present on the cluster. +*/ -}} +apiVersion: kro.run/v1alpha1 +kind: ResourceGraphDefinition +metadata: + name: microvm-sandbox + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + # Apply the RGD FIRST so KRO generates the MicrovmSandbox CRD before the instance + # (below) is synced — otherwise the instance fails dry-run ("CRD not found") and + # blocks the whole app. + argocd.argoproj.io/sync-wave: "-1" +spec: + schema: + apiVersion: v1alpha1 + kind: MicrovmSandbox + group: {{ .Values.microvm.apiGroup | default "kro.run" | quote }} + # SCOPE: this RGD builds ONLY the platform IMAGE + the slow-changing infra it + # needs (S3 artifact bucket, build role, execution role). It deliberately does + # NOT create a `Microvm` — a MicroVM instance is a per-SESSION, request-time + # resource whose create/suspend/resume/terminate are IMPERATIVE SDK ops that the + # ACK controller does not reconcile (confirmed in the lambdamicrovms-controller + # reference: run/suspend/resume/terminate are SDK calls, not desired state, and + # the 06-kro example likewise leaves the running Microvm out of the graph). The + # shim (SandboxTemplate bridge + microvm-lifecycle controller, templates 51/52) + # owns that lifecycle. So a `MicrovmSandbox` = "an image is built + ready + its + # exec identity", and its status is the HANDOFF the shim consumes to RunMicrovm. + spec: + # ── PLATFORM-owned: the image / substrate (set once per image) ────────── + # ARN of the AWS-published base MicroVM image to build from, e.g. + # arn:aws:lambda::aws:microvm-image:al2023-1 (ARM_64 — the only arch + # Lambda MicroVM supports). + baseImageARN: string + # S3 URI of the coder code artifact zip (app + Dockerfile), e.g. + # s3:///. Lambda MicroVM's codeArtifact.uri is S3-ONLY — it is + # NOT an ECR image reference (the Dockerfile inside the zip MAY pull private + # ECR base layers, which is why buildRole keeps ecr:Get*/BatchGetImage). The + # artifact must be published to the bucket before the first build runs. + codeArtifactUri: string + # AWS region + a name stem for the created resources. + region: string | default="{{ .Values.microvm.region }}" + name: string + status: + # The HANDOFF the shim reads to RunMicrovm (imperatively) per session: + # imageARN → Microvm.imageIdentifier + # executionRoleARN → Microvm.executionRoleARN + # Plus imageState so the shim only launches once the build is CREATED/UPDATED. + imageARN: ${image.status.ackResourceMetadata.arn} + imageState: ${image.status.state} + imageVersion: ${image.status.latestActiveImageVersion} + executionRoleARN: ${execRole.status.ackResourceMetadata.arn} + resources: + # 1) S3 bucket that stores the MicroVM code artifact (GA — Managed ACK). + - id: bucket + template: + apiVersion: s3.services.k8s.aws/v1alpha1 + kind: Bucket + metadata: + name: ${schema.spec.name}-microvm-artifacts + spec: + name: ${schema.spec.name}-microvm-artifacts + # 2) IAM role the image BUILD assumes (GA — Managed ACK). Platform-owned. + # Lambda assumes this during create-microvm-image to pull the code artifact + # (ECR/S3) + write build logs. Trust = lambda.amazonaws.com (verified in the + # Lambda MicroVM getting-started docs), with inline ECR-read + S3-read + logs. + - id: buildRole + template: + apiVersion: iam.services.k8s.aws/v1alpha1 + kind: Role + metadata: + name: ${schema.spec.name}-microvm-build + spec: + name: ${schema.spec.name}-microvm-build + assumeRolePolicyDocument: | + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} + inlinePolicies: + microvm-build: | + {"Version":"2012-10-17","Statement":[ + {"Effect":"Allow","Action":["ecr:GetAuthorizationToken","ecr:BatchGetImage","ecr:GetDownloadUrlForLayer"],"Resource":"*"}, + {"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket"],"Resource":["arn:aws:s3:::${bucket.spec.name}","arn:aws:s3:::${bucket.spec.name}/*"]}, + {"Effect":"Allow","Action":["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:*:*:log-group:/aws/lambda/microvms/*"} + ]} + # 3) IAM role the RUNNING MicroVM assumes (GA — Managed ACK). App-owned exec identity. + # trust = lambda.amazonaws.com. Grants bedrock:InvokeModel so the coder calls + # Bedrock DIRECTLY (Flow D is Bedrock-direct: a MicroVM runs outside the cluster + # and can't reach Bifrost's ClusterIP, so it uses this role's creds over public + # egress instead — see flow-d-coder-in-microvm-design.md). git/gh over public :443. + - id: execRole + template: + apiVersion: iam.services.k8s.aws/v1alpha1 + kind: Role + metadata: + name: ${schema.spec.name}-microvm-exec + spec: + name: ${schema.spec.name}-microvm-exec + assumeRolePolicyDocument: | + {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":["sts:AssumeRole","sts:TagSession"]}]} + inlinePolicies: + bedrock-invoke: | + {"Version":"2012-10-17","Statement":[ + {"Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream","bedrock:Converse","bedrock:ConverseStream"],"Resource":["arn:aws:bedrock:*::foundation-model/*","arn:aws:bedrock:*:*:inference-profile/*"]} + ]} + # 4) MicrovmImage (pre-GA — SELF-MANAGED lambdamicrovms controller). Platform-owned. + # readyWhen gates the Microvm (resource 5) on a genuinely SUCCESSFUL build — + # state CREATED/UPDATED — so the instance never launches from a half-built or + # failed image (KRO holds the Microvm until this predicate is true). + - id: image + readyWhen: + - ${image.status.state == "CREATED" || image.status.state == "UPDATED"} + template: + apiVersion: lambdamicrovms.services.k8s.aws/v1alpha1 + kind: MicrovmImage + metadata: + name: ${schema.spec.name}-image + spec: + name: ${schema.spec.name}-image + baseImageARN: ${schema.spec.baseImageARN} + buildRoleARN: ${buildRole.status.ackResourceMetadata.arn} + codeArtifact: + uri: ${schema.spec.codeArtifactUri} + # Lambda MicroVM is ARM_64-ONLY — the sole supported architecture. The + # code artifact + any bundled binaries must be arm64 (dark-factory-coder + # is built for arm64 for exactly this substrate). + cpuConfigurations: + - architecture: ARM_64 + # Lifecycle hooks — the lambda-coder (hook-server.js) serves these on :8080. + # ready : build waits for the server to be up before snapshotting a + # clean, waiting coder (else the snapshot is taken too early). + # run : per-session start; delivers runHookPayload (issue context + + # GitHub token) as the request body → hook-server background- + # spawns the coder. WITHOUT run:ENABLED the payload is silently + # never delivered. + # suspend/resume/terminate : lifecycle acks (coder holds no external state). + hooks: + port: 8080 + microvmImageHooks: + ready: ENABLED + readyTimeoutInSeconds: 120 + microvmHooks: + run: ENABLED + runTimeoutInSeconds: 30 + suspend: ENABLED + suspendTimeoutInSeconds: 30 + resume: ENABLED + resumeTimeoutInSeconds: 30 + terminate: ENABLED + terminateTimeoutInSeconds: 30 + # CloudWatch build/runtime logs. On a CREATE_FAILED the controller can't + # see the build output — this is where it lands: + # aws logs tail /aws/lambda/microvms/${schema.spec.name}-image + logging: + cloudWatch: + logGroup: /aws/lambda/microvms/${schema.spec.name}-image + # NOTE: there is deliberately NO `Microvm` resource here. The running MicroVM is + # a per-session, request-time resource — the shim (microvm-lifecycle controller, + # template 52) creates it with RunMicrovm and drives suspend/resume/terminate as + # imperative SDK ops (the ACK controller does not reconcile those). This RGD stops + # at "image built + exec role ready", handed off via status above. +{{- if .Values.microvm.image.enabled | default true }} +--- +# The ONE platform MicrovmSandbox INSTANCE — applied by GitOps, reconciled by KRO +# ONCE to build the coder image + its build/exec roles + artifact bucket. This is +# slow-changing platform infra (one image per cluster), NOT per-session — so it lives +# here in the chart, not in the per-claim shim. The shim reads THIS object's status +# (imageARN + executionRoleARN) to RunMicrovm per session. Rebuild the image by +# bumping microvm.codeArtifactUri (a new artifact) and re-syncing. +apiVersion: {{ .Values.microvm.apiGroup | default "kro.run" }}/v1alpha1 +kind: MicrovmSandbox +metadata: + name: {{ .Values.microvm.image.name | default "coder" }} + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + # Sync AFTER the RGD (wave -1) so its generated CRD exists. SkipDryRunOnMissingResource + # lets the first sync proceed even if KRO hasn't registered the CRD in the same pass — + # ArgoCD retries + selfHeal converge once the CRD appears (no whole-app block). + argocd.argoproj.io/sync-wave: "1" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + name: {{ .Values.microvm.image.name | default "coder" }} + region: {{ .Values.microvm.region | quote }} + baseImageARN: {{ .Values.microvm.baseImageARN | quote }} + codeArtifactUri: {{ .Values.microvm.codeArtifactUri | quote }} +{{- end }} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml new file mode 100644 index 00000000..f4f639ef --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/00-controller-pod-identity.yaml @@ -0,0 +1,150 @@ +{{/* + Self-managed ACK lambdamicrovms controller → AWS access via EKS Pod Identity, + declared ALL-ACK (same mechanism as the RGD's build/exec roles): an ACK + iam.services.k8s.aws Role (trusted by the EKS Pod Identity service principal) + + an ACK eks.services.k8s.aws PodIdentityAssociation binding it to the controller's + ServiceAccount (ack-lambdamicrovms-controller in ack-system, created by the ACK + chart). No Crossplane, no Terraform, no CLI. + + This is the ONE bootstrap IAM the KRO RGD can't self-create (the controller needs + creds before it can create anything). It's reconciled by the Managed-ACK iam + eks + controllers (both live on the hub). Everything DOWNSTREAM — the S3 bucket, the build + role, the exec role, the MicrovmImage — is created by the KRO RGD via ACK too (see + templates/image/). So the entire IAM surface is ACK/KRO/GitOps. + + Earlier sync-wave so the role + association exist before the controller pod needs + them; Pod Identity creds are vended on demand + ArgoCD selfHeal converges with no + manual steps. Gated by microvm.enabled. +*/}} +{{- if and .Values.microvm .Values.microvm.enabled }} +apiVersion: iam.services.k8s.aws/v1alpha1 +kind: Role +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller + # Trust the EKS Pod Identity service principal (not IRSA/OIDC). + assumeRolePolicyDocument: | + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": {"Service": "pods.eks.amazonaws.com"}, + "Action": ["sts:AssumeRole", "sts:TagSession"] + }] + } + inlinePolicies: + lambdamicrovms: | + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "LambdaMicrovms", + "Effect": "Allow", + "Action": [ + "lambda:CreateMicrovmImage","lambda:UpdateMicrovmImage","lambda:DeleteMicrovmImage", + "lambda:GetMicrovmImage","lambda:GetMicrovmImageVersion","lambda:ListMicrovmImages", + "lambda:RunMicrovm","lambda:GetMicrovm","lambda:TerminateMicrovm","lambda:ListMicrovms", + "lambda:SuspendMicrovm","lambda:ResumeMicrovm", + "lambda:CreateMicrovmAuthToken","lambda:CreateMicrovmShellAuthToken", + "lambda:TagResource","lambda:UntagResource","lambda:ListTagsForResource", + "lambda:ListNetworkConnectors","lambda:GetNetworkConnector" + ], + "Resource": "*" + }, + {{- /* + CreateMicrovmImage/RunMicrovm attach a network connector to the MicroVM. + With no explicit connectors in the RGD, the service uses the AWS-managed + default INTERNET_EGRESS connector, and passing it needs + lambda:PassNetworkConnector (verified: image build denied + 'lambda:PassNetworkConnector on .../network-connector:aws-network-connector: + INTERNET_EGRESS'). Scope to the AWS-managed connector ARNs in-region. + */ -}} + { + "Sid": "PassNetworkConnectors", + "Effect": "Allow", + "Action": "lambda:PassNetworkConnector", + "Resource": [ + "arn:aws:lambda:{{ .Values.microvm.region }}:aws:network-connector:*", + "arn:aws:lambda:{{ .Values.microvm.region }}:{{ .Values.microvm.accountId }}:network-connector:*" + ] + }, + {{- /* + Scope PassRole by the TARGET role ARN (the build/exec roles KRO creates), + NOT by an iam:PassedToService condition. CreateMicrovmImage/RunMicrovm pass + the role to the Lambda MicroVM sub-service whose principal is NOT plain + lambda.amazonaws.com — a StringEquals on lambda.amazonaws.com fails closed, + so the controller got AccessDenied on iam:PassRole for coder-microvm-build + (verified: sim ALLOWED for lambda.amazonaws.com yet the live API DENIED, i.e. + the real passed-to principal differs; microvms/microvm.lambda.amazonaws.com + also implicitDeny). ARN-scoping to *-microvm-build/-exec is net-TIGHTER than + the previous Resource:* — only these two purpose-built roles can be passed — + and is principal-agnostic so it survives whatever sub-service MicroVM uses. + */ -}} + { + "Sid": "PassBuildExecRoles", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": [ + "arn:aws:iam::{{ .Values.microvm.accountId }}:role/*-microvm-build", + "arn:aws:iam::{{ .Values.microvm.accountId }}:role/*-microvm-exec" + ] + } + ] + } +--- +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ .Values.microvm.podIdentity.controllerNamespace | default "ack-system" }} + serviceAccount: {{ .Values.microvm.podIdentity.controllerServiceAccount | default "ack-lambdamicrovms-controller" }} + # ACK PodIdentityAssociation takes the role ARN (no role-ref selector). The ACK + # Role above has a deterministic name, so the ARN is constructed from the account id. + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +--- +# The BRIDGE SA (agent-sandbox-system) calls RunMicrovm/GetMicrovm/TerminateMicrovm — +# reuse the same lambda-microvms role via its own PodIdentityAssociation. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ include "agent-sandbox.namespace" . }} + serviceAccount: microvm-bridge + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +--- +# The LIFECYCLE controller SA calls suspend-microvm/resume-microvm — same role. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ include "agent-sandbox.namespace" . }} + serviceAccount: microvm-lifecycle + roleARN: "arn:aws:iam::{{ .Values.microvm.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml new file mode 100644 index 00000000..cc845c6a --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/20-bridge-sandboxtemplate.yaml @@ -0,0 +1,469 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — `lambda-microvm` SandboxTemplate + bridge RBAC + bridge script. + +The RuntimeClass "shim": Lambda MicroVM is a REMOTE AWS service, not a node-local +containerd handler, so there is no literal `lambda-microvm` RuntimeClass (that would +need a virtual-kubelet — out of scope). Instead this SandboxTemplate's pod is a thin +BRIDGE that preserves the Agent-Sandbox UX: + + 1. runs on a NORMAL Auto-Mode node (no kata runtimeClass / nodeSelector / taint) + 2. reads the ONE platform-built image (the committed MicrovmSandbox in template 50, + whose status carries imageARN + executionRoleARN — built ONCE by KRO/ACK, NOT + per session) and calls RunMicrovm (imperative AWS SDK) to launch a per-session + MicroVM running the SAME dark-factory-coder entrypoint + 3. records the microvmID on the owning Sandbox (annotation) so the microvm-lifecycle + controller (template 52) can suspend/resume it, and holds the pod so the pod + lifecycle mirrors the MicroVM; on real teardown it calls TerminateMicrovm + +ARCHITECTURE SPLIT (why the bridge no longer creates a MicrovmSandbox per claim): +image build = slow, declarative, ONE per cluster (KRO/ACK, template 50). Running a VM += fast, imperative, per SESSION (RunMicrovm/suspend/resume/terminate are SDK ops the +ACK controller does NOT reconcile). So the bridge drives the RUN side via the SDK and +only READS the platform image handoff — it does not re-run KRO per claim. + +To Flow B and the user this looks identical to a Flow A claim. The bridge needs a +ServiceAccount + AWS creds (Pod Identity) to call the Lambda MicroVM SDK. + +Gated behind microvm.enabled. Rendered as a SEPARATE SandboxTemplate +(`-microvm`) so both substrates can coexist on one cluster; a consumer +selects the substrate by which template its SandboxClaim references. +*/ -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +--- +{{- /* + Bridge API-server egress. + + The bridge pod carries agent-sandbox.io/role=coder (so Flow B's claim contract + + the shared coder-sandbox-egress policy apply to it). But that policy denies RFC-1918 + + the service CIDR on :443 to isolate UNTRUSTED coder code from the control plane — + and the K8s API server lives exactly there (kubernetes svc 172.20.0.1 + apiserver + endpoints in the VPC 10.0.0.0/8). NetworkPolicies are additive, so this ADDS an + egress allow for the API server, selected ONLY on the bridge's distinct + agent-sandbox.io/substrate=lambda-microvm label (Kata coders don't have it, so their + isolation is untouched). The bridge runs TRUSTED platform code (reads the MicrovmSandbox + image handoff + annotates the owning Sandbox with the microvmID) — unlike the Kata coder + it MUST reach the API server, or `kubectl get microvmsandbox` hangs and it never + RunMicrovm's (verified: in-pod kubectl to 172.20.0.1:443 timed out under coder-egress). +*/ -}} +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: microvm-bridge-apiserver-egress + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + agent-sandbox.io/substrate: lambda-microvm + policyTypes: + - Egress + egress: + # K8s API server — service ClusterIP + the in-VPC apiserver endpoints, :443 only. + - to: + - ipBlock: + cidr: {{ .Values.microvm.apiServerCidr | default "172.20.0.1/32" | quote }} + {{- range .Values.microvm.apiServerEndpointCidrs | default (list "10.0.0.0/16") }} + - ipBlock: + cidr: {{ . | quote }} + {{- end }} + ports: + - protocol: TCP + port: 443 + # EKS Pod Identity credential endpoint (link-local 169.254.170.23:80). The bridge + # gets its AWS creds — to call lambda-microvms run/suspend/resume/terminate — from + # the Pod Identity agent here. The shared coder-egress policy denies 169.254.0.0/16 + # (to block IMDS for untrusted coder code), which ALSO blocks Pod Identity, so the + # bridge's run-microvm failed "retrieving credentials from container-role: connect + # timeout http://169.254.170.23/v1/credentials". Allow ONLY the Pod Identity /32 (NOT + # IMDS 169.254.169.254, which stays denied) and ONLY for the bridge selector. + - to: + - ipBlock: + cidr: {{ .Values.microvm.podIdentityEndpoint | default "169.254.170.23/32" | quote }} + ports: + - protocol: TCP + port: 80 +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +rules: + # READS the platform MicrovmSandbox (image handoff: imageARN + executionRoleARN) — + # GitOps-owned platform infra (template 50), built once. + - apiGroups: [{{ .Values.microvm.apiGroup | default "kro.run" | quote }}] + resources: ["microvmsandboxes"] + verbs: ["get", "list", "watch"] + # Per-session Microvm CR: the bridge CREATES it (declarative path — the only way the + # controller fires the /run hook that delivers runHookPayload) and DELETES it on + # teardown (controller terminates the VM). + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvms"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Per-session payload Secret (runHookPayload SecretKeyReference). + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update", "patch", "delete"] + # Read the owning Sandbox (suspend-vs-teardown in preStop) + patch it to record the + # per-session microvmID (annotation) so the microvm-lifecycle controller can + # suspend/resume THIS session's VM. + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["get", "patch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: microvm-bridge +subjects: + - kind: ServiceAccount + name: microvm-bridge + namespace: {{ include "agent-sandbox.namespace" . }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: microvm-bridge-script + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +data: + bridge.sh: | + #!/bin/sh + # Flow D bridge — claim -> RunMicrovm (SDK) against the pre-built platform image -> + # mirror lifecycle. Idles until a SandboxClaim injects DF_ISSUE_NUMBER (Flow B), + # exactly like the Kata coder. Needs AWS creds (Pod Identity) for the SDK calls. + set -eu + # The bridge image is public.ecr.aws/aws-cli/aws-cli:latest — a glibc, always-current + # aws-cli v2 that KNOWS the pre-GA `lambda-microvms` service (the alpine/k8s image's + # aws-cli 1.34 does NOT: `aws lambda-microvms` printed the service list = unrecognized, + # so RunMicrovm never fired). That image has no kubectl, so fetch a static one here + # (same pattern as the security-agent bootstrap). Both are then on PATH. + if ! command -v kubectl >/dev/null 2>&1; then + echo "[microvm-bridge] fetching kubectl..." + ARCH="$(uname -m)"; case "$ARCH" in aarch64|arm64) A=arm64;; *) A=amd64;; esac + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/${A}/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + REGION="{{ .Values.microvm.region }}" + PLATFORM_IMAGE="{{ .Values.microvm.image.name | default "coder" }}" # the ONE committed MicrovmSandbox (template 50) + NS="{{ include "agent-sandbox.namespace" . }}" + echo "[microvm-bridge] idle — waiting for a SandboxClaim to inject DF_ISSUE_NUMBER..." + while [ -z "${DF_ISSUE_NUMBER:-}" ]; do sleep 5; done + # SANDBOX_NAME (downward API = owning Sandbox CR name) lets the microvm-lifecycle + # controller correlate THIS session's MicroVM for suspend/resume. + SANDBOX_NAME="${SANDBOX_NAME:-df-${DF_ISSUE_NUMBER}}" + echo "[microvm-bridge] claim for issue #${DF_ISSUE_NUMBER} (sandbox=${SANDBOX_NAME})" + + # 1) READ the platform image handoff (built ONCE by KRO/ACK — template 50). Wait + # until its image build is terminal (CREATED/UPDATED) before launching. + echo "[microvm-bridge] reading platform image ${PLATFORM_IMAGE} (waiting for build ready)..." + i=0; IMAGE_ARN=""; EXEC_ROLE="" + while [ "$i" -lt 240 ]; do + IST=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageState}' 2>/dev/null || echo "") + if [ "${IST}" = "CREATED" ] || [ "${IST}" = "UPDATED" ]; then + IMAGE_ARN=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageARN}' 2>/dev/null || echo "") + EXEC_ROLE=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.executionRoleARN}' 2>/dev/null || echo "") + [ -n "${IMAGE_ARN}" ] && [ -n "${EXEC_ROLE}" ] && break + fi + i=$((i+1)); sleep 5 + done + if [ -z "${IMAGE_ARN}" ] || [ -z "${EXEC_ROLE}" ]; then + echo "[microvm-bridge] ERROR: platform image not ready (imageState=${IST:-}) — cannot RunMicrovm"; exit 1 + fi + echo "[microvm-bridge] image=${IMAGE_ARN} execRole=${EXEC_ROLE}" + + # 2) Build the runHookPayload — the per-session context the coder needs, delivered + # as the /run hook body (hook-server.js background-spawns the coder from it). + # CRITICAL: runHookPayload is a SecretKeyReference delivered by the DECLARATIVE + # Microvm CR (the controller reads the Secret + drives the /run hook). The + # imperative `run-microvm --run-hook-payload` CLI launches the VM but NEVER fires + # /run (verified: VM RUNNING, hook-server listening, but coder never started). So + # we write a Secret + create a Microvm CR. GitHub token from the mounted secret; + # NO Bifrost key — the coder is Bedrock-direct via the exec role. + GH_TOKEN="$(cat /etc/df/gh-token 2>/dev/null || echo "")" + [ -z "${GH_TOKEN}" ] && echo "[microvm-bridge] WARN: no gh-token mounted — coder cannot open a PR" + # python3 (node is absent in the aws-cli image); json.dumps escapes token/title. One + # line so it stays inside the bridge.sh: | YAML block scalar. + # FIX ROUND: df-run injects DF_ITERATE_NOTE_B64 (the human change request) into this + # `coder` container's env (same claim contract as Kata). The MicroVM coder has no + # claim env, so fold the note into the runHookPayload — hook-server maps it back to + # the coder's DF_ITERATE_NOTE_B64. Without this a Lambda fix round runs with NO + # instructions and reports "done" on the OLD sha (zero commits). + [ -n "${DF_ITERATE_NOTE_B64:-}" ] && echo "[microvm-bridge] fix round: forwarding iterate note to coder" + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("DF_ISSUE_NUMBER",""),"repo":e("DF_REPO",""),"branch":e("DF_BRANCH",""),"baseBranch":e("DF_BASE_BRANCH","main"),"issueTitle":e("DF_ISSUE_TITLE",""),"iterateNoteB64":e("DF_ITERATE_NOTE_B64",""),"iterateNote":e("DF_ITERATE_NOTE","")}))') + + MVM="mvm-${DF_ISSUE_NUMBER}" # Microvm CR + payload Secret name for this session + # RESUME-ON-FIX-ROUND (the suspend/resume highlight): the CR name is stable + # (mvm-). On a fix round the previous round's VM is still around — SUSPENDED + # after the first PR (idlePolicy.autoResumeEnabled=false keeps it down). Rather than + # terminate + rebuild a fresh VM, we RESUME the suspended one via the Sandbox CRD: + # flip operatingMode=Running → the microvm-lifecycle controller calls resume-microvm → + # the SAME VM (memory+disk preserved) comes back, and hook-server accepts a NEW /run + # because its guard is keyed on a per-invocation run-id (issue+note hash), not a + # one-shot boolean (see coder-microvm/hook-server.js). This is the whole Flow D value + # prop: scale-to-zero between rounds, warm-resume for the fix. Fresh VMs are only made + # on the FIRST round (no existing CR). + RESUME_ROUND="" + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + echo "[microvm-bridge] fix round: resuming suspended VM ${MVM} via Sandbox.operatingMode=Running" + kubectl patch sandbox "${SANDBOX_NAME}" -n "${NS}" --type merge \ + -p '{"spec":{"operatingMode":"Running"}}' >/dev/null 2>&1 \ + && echo "[microvm-bridge] operatingMode=Running set — microvm-lifecycle will resume the VM" \ + || echo "[microvm-bridge] WARN: could not set operatingMode=Running" + # Refresh the payload Secret so the controller/hook sees the NEW iterate note. + RESUME_ROUND=1 + fi + # Build BOTH manifests as JSON with python3 and pipe to kubectl apply. JSON (not a + # heredoc) on purpose: a heredoc's column-0 EOF terminator breaks out of the + # bridge.sh: | YAML block scalar. JSON is valid YAML and stays on indented lines. + MAXIDLE={{ .Values.microvm.defaults.maxIdleDurationSeconds }}; SUSPDUR={{ .Values.microvm.defaults.suspendedDurationSeconds }} + INGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:ALL_INGRESS" + LOGGRP="/aws/lambda/microvms/{{ .Values.microvm.image.name | default "coder" }}-image" + echo "[microvm-bridge] writing payload Secret + Microvm CR ${MVM} for issue #${DF_ISSUE_NUMBER}..." + MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":mvm+"-payload","namespace":ns},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] payload secret apply failed"; exit 1; } + # Microvm CR: ingress ALL_INGRESS (so the bridge can reach the endpoint to drive /run), + # egress INTERNET_EGRESS (Bedrock + git/gh), runtime logging → CloudWatch (logStream + # 'runtime' so the coder's stdout is visible, separate from build logs). + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" REGION="${REGION}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; r=e["REGION"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":["arn:aws:lambda:"+r+":aws:network-connector:aws-network-connector:INTERNET_EGRESS"],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":False,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[microvm-bridge] Microvm CR apply failed"; exit 1; } + + # 4) Wait for the controller to report the running VM's id, record it on the Sandbox + # (lifecycle controller reads this to suspend/resume THIS session's VM). + VMID=""; i=0 + while [ "$i" -lt 60 ]; do + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -n "${VMID}" ] && break + i=$((i+1)); sleep 5 + done + echo "[microvm-bridge] Microvm ${MVM} -> ${VMID:-}" + [ -n "${VMID}" ] && kubectl annotate sandbox "${SANDBOX_NAME}" -n "${NS}" \ + "microvm-lifecycle.agents.x-k8s.io/microvm-id=${VMID}" --overwrite >/dev/null 2>&1 || true + + # 4b) DRIVE the coder: wait for RUNNING + an endpoint, mint an auth token, and POST the + # payload to /run on the endpoint. This is the deterministic invocation (verified by + # probe): the service's internal /run auto-fire wasn't reliably starting the coder, so + # the bridge drives it explicitly like the reference run_session. hook-server's /run + # background-spawns the coder (returns fast); df-run's await-coder polls GitHub for the PR. + if [ -n "${VMID}" ]; then + EP=""; i=0 + while [ "$i" -lt 60 ]; do + S=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'state' --output text 2>/dev/null || echo "") + EP=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'endpoint' --output text 2>/dev/null || echo "") + [ "$S" = "RUNNING" ] && [ -n "${EP}" ] && [ "${EP}" != "None" ] && break + i=$((i+1)); sleep 5 + done + TOKEN=$(aws lambda-microvms create-microvm-auth-token --region "${REGION}" --microvm-identifier "${VMID}" \ + --expiration-in-minutes 60 --allowed-ports 'port=8080' 2>/dev/null \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["authToken"]["X-aws-proxy-auth"])' 2>/dev/null || echo "") + if [ -n "${EP}" ] && [ -n "${TOKEN}" ]; then + echo "[microvm-bridge] driving coder: POST /run on ${EP}" + RC=$(curl -sS -m 30 -o /tmp/run.out -w '%{http_code}' -X POST "https://${EP}/run" \ + -H "X-aws-proxy-auth: ${TOKEN}" -H 'Content-Type: application/json' -d "${PAYLOAD}" 2>/tmp/run.err || echo "000") + echo "[microvm-bridge] /run -> HTTP ${RC} $(cat /tmp/run.out 2>/dev/null | head -c 120)" + else + echo "[microvm-bridge] WARN: no endpoint/token — cannot drive /run (ep=${EP:-none} token=$([ -n "${TOKEN}" ] && echo yes || echo no))" + fi + fi + + # 5) Teardown vs suspend — decided by whether the OWNING SANDBOX STILL EXISTS. + # The agent-sandbox operator handles operatingMode=Suspended by DELETING THE POD + # (verified in the operator log: "Deleting Pod because .Spec.OperatingMode is + # Suspended") while KEEPING the Sandbox object alive in state SandboxSuspended. So + # the bridge pod is torn down on EVERY suspend — and its cleanup trap must NOT delete + # the Microvm CR then, or the VM is terminated instead of suspended (exactly the bug + # we hit: Sandbox survived Suspended, but the VM was gone). + # - Sandbox STILL EXISTS => this is a SUSPEND (or transient pod restart) => KEEP CR. + # - Sandbox GONE => real teardown (df-merge-teardown deleted the claim) => + # delete CR so the controller TerminateMicrovm's the VM. + # (Checking Sandbox existence is more robust than a preStop /tmp marker, which raced + # the SIGTERM and didn't reliably stick.) + cleanup() { + if kubectl get sandbox "${SANDBOX_NAME}" -n "${NS}" >/dev/null 2>&1; then + M=$(kubectl get sandbox "${SANDBOX_NAME}" -n "${NS}" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "") + echo "[microvm-bridge] pod stopping but Sandbox ${SANDBOX_NAME} still exists (operatingMode=${M:-?}) — KEEPING Microvm ${MVM} (suspend, not teardown)" + return + fi + echo "[microvm-bridge] Sandbox ${SANDBOX_NAME} gone — real teardown: deleting Microvm ${MVM} (controller terminates the VM)" + kubectl delete microvm "${MVM}" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + } + trap cleanup EXIT INT TERM + + # 6) Hold the pod so Sandbox lifecycle == Microvm lifecycle. Poll the in-VM coder /logs + # (OBSERVABILITY — runtime CloudWatch routing is unreliable on this runtime) ONLY + # until the coder pushes its PR. Then suspend and STOP touching the endpoint. + # + # SUSPEND-VIA-CRD (the Flow D highlight): suspend/resume is driven declaratively + # through Sandbox.spec.operatingMode, reconciled by the microvm-lifecycle controller + # (template 30) — NOT by an imperative suspend-microvm call here. Two reasons this + # matters and why the old imperative path FAILED to keep the VM suspended: + # (a) The CR is created with idlePolicy.autoResumeEnabled=FALSE, so a suspended VM + # stays suspended. With autoResume=true (the old value) ANY hit to the VM + # endpoint auto-resumes it — and this loop used to curl /logs every 20s + # FOREVER, so the VM bounced back to RUNNING seconds after every suspend + # (observed in the Lambda console: never actually suspended). + # (b) Routing suspend through operatingMode is the whole point — it shows the + # Agent Sandbox CRD driving MicroVM scale-to-zero via the shim controller. + # So: once the coder pushes, set operatingMode=Suspended (controller suspends the + # VM), then switch to a lightweight CR-existence watch that NEVER touches the + # endpoint again. The VM's memory+disk persist; a fix round flips operatingMode + # back to Running (controller resumes) and the pipeline claims a fresh session. + echo "[microvm-bridge] Microvm ${MVM} running — pod now mirrors its lifecycle." + while true; do + # CR gone/terminating => real teardown => exit (cleanup trap handles CR delete). + ST=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.state}' 2>/dev/null || echo "GONE") + case "${ST}" in + TERMINATED|TERMINATING|GONE|"") echo "[microvm-bridge] Microvm state=${ST:-gone} — exiting."; break ;; + esac + if [ -n "${EP:-}" ] && [ -n "${TOKEN:-}" ]; then + # Poll /logs for observability + to detect "coder pushed PR". + LOG=$(curl -sS -m 10 "https://${EP}/logs" -H "X-aws-proxy-auth: ${TOKEN}" 2>/dev/null \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("log","")[-800:])' 2>/dev/null || echo "") + [ -n "${LOG}" ] && echo "[microvm-bridge] coder-log-tail: $(echo "$LOG" | tail -1)" + if echo "${LOG}" | grep -qiE 'PR opened|done — PR|status success'; then + # Coder is done → request SUSPEND declaratively. The operator then DELETES this + # pod (operatingMode=Suspended), and the microvm-lifecycle controller reconciles + # the same intent into suspend-microvm. We EXIT right after setting it: the pod is + # about to be killed anyway, and exiting cleanly lets the cleanup trap run while + # the Sandbox still exists → it KEEPS the CR (VM suspends, not terminates). We do + # NOT keep polling /logs — every endpoint hit would auto-resume the VM. + echo "[microvm-bridge] coder pushed PR — requesting SUSPEND via Sandbox.operatingMode (controller reconciles suspend-microvm ${VMID})" + kubectl patch sandbox "${SANDBOX_NAME}" -n "${NS}" --type merge \ + -p '{"spec":{"operatingMode":"Suspended"}}' >/dev/null 2>&1 \ + && echo "[microvm-bridge] operatingMode=Suspended set — exiting bridge (pod will be removed; CR + VM persist)" \ + || echo "[microvm-bridge] WARN: could not set operatingMode=Suspended" + break + fi + fi + sleep 20 + done +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: {{ .Values.warmPool.templateName }}-microvm + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + # Same env-injection contract as the Kata template so Flow B is unchanged. + envVarsInjectionPolicy: {{ .Values.coderTemplate.envVarsInjectionPolicy | default "Allowed" }} + podTemplate: + metadata: + labels: + {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} + agent-sandbox.io/role: coder + agent-sandbox.io/substrate: lambda-microvm + spec: + # NO kata runtimeClass / nodeSelector / toleration — the bridge is a normal + # pod on an Auto-Mode node. The isolation boundary is the remote MicroVM. + serviceAccountName: microvm-bridge + # The bridge needs a k8s token (read platform image, patch Sandbox) AND AWS creds + # (Pod Identity association on this SA) to call the Lambda MicroVM SDK + # (RunMicrovm/GetMicrovm/TerminateMicrovm). The Kata coder is credential-less; the + # bridge is not, because RUN is imperative. + automountServiceAccountToken: true + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + # Named `coder` (NOT `bridge`) on purpose: Flow B's SandboxClaim injects env + # (DF_ISSUE_NUMBER, DF_REPO, …) into a container called `coder` — the SAME claim + # contract as the Kata substrate. The operator REJECTS the claim + # ("target container coder not found") if this name differs, so the substrate + # must expose a `coder` container to stay transparent to df-run. This container + # is still the bridge (runs bridge.sh → RunMicrovm); only the name matches Kata. + - name: coder + image: {{ .Values.microvm.bridgeImage }} + command: ["/bin/sh", "/scripts/bridge.sh"] + # SANDBOX_NAME = this pod's own name via the downward API. agent-sandbox names + # the Sandbox CR and its pod identically, so this IS the owning Sandbox name. + # Without it bridge.sh fell back to df- (e.g. df-9999) which does NOT + # match the real Sandbox (named after the claim, e.g. df-issue-smoke-d), so the + # microvm-id annotation write silently failed → the lifecycle controller couldn't + # find the VM to suspend and the pod teardown TERMINATED it instead of suspending. + env: + - name: SANDBOX_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + # NOTE: no preStop suspend marker needed anymore — cleanup() decides suspend-vs- + # teardown by whether the owning Sandbox still EXISTS (it survives Suspended, + # is gone on real teardown), which is race-free unlike a preStop /tmp marker. + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + # The bridge runs python3 (JSON), curl, a fetched kubectl, and an aws-cli v2 + # (glibc) in a poll loop — 128Mi OOMKilled it mid fix-round (bridge died before + # the coder's new commit landed → await-coder spun forever). Give it real headroom. + requests: { cpu: 100m, memory: 256Mi } + limits: { cpu: "1", memory: 1Gi } + volumeMounts: + - name: bridge-script + mountPath: /scripts + - name: tmp + mountPath: /tmp + # GitHub token (read-only) the bridge folds into the runHookPayload so the + # coder in the MicroVM can push + open the PR. Same secret the Kata coder uses. + - name: gh-token + mountPath: /etc/df + readOnly: true + volumes: + - name: bridge-script + configMap: + name: microvm-bridge-script + defaultMode: 0555 + - name: tmp + emptyDir: {} + - name: gh-token + secret: + secretName: {{ .Values.microvm.githubSecretName | default "dark-factory-github" }} + defaultMode: 0400 + optional: true +--- +{{- /* +Lambda-MicroVM SandboxWarmPool. SandboxClaim.spec.warmPoolRef is REQUIRED (a claim +can't bind a bare template), so Flow D needs its own pool the df-run claim step +targets when the darkfactory-lambda label fires. Kept small (bridge pods are tiny); +each idle member is a bridge waiting to RunMicrovm (from the pre-built platform image) +on claim. +*/ -}} +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: {{ .Values.microvm.warmPool.name | default "coder-warmpool-microvm" }} + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} + agent-sandbox.io/substrate: lambda-microvm +spec: + replicas: {{ .Values.microvm.warmPool.targetIdle | default 1 }} + sandboxTemplateRef: + name: {{ .Values.warmPool.templateName }}-microvm +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml new file mode 100644 index 00000000..bc4e9757 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/30-microvm-lifecycle.yaml @@ -0,0 +1,184 @@ +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- /* +Flow D — microvm-lifecycle controller (suspend/resume via Sandbox.operatingMode). + +WHY THIS EXISTS: the Agent Sandbox CRD exposes `spec.operatingMode ∈ {Running, +Suspended}` (the declarative suspend/resume intent), but the ACK Microvm CR has NO +suspend field — its spec is create-time only; suspend/resume are IMPERATIVE SDK ops +(`suspend-microvm`/`resume-microvm`) that the ACK controller deliberately does NOT +reconcile. So switching operatingMode does nothing on its own. This tiny always-on +reconcile loop closes that gap: + + Sandbox.operatingMode: Running -> Suspended : aws lambda-microvms suspend-microvm + Sandbox.operatingMode: Suspended -> Running : aws lambda-microvms resume-microvm + +It resolves the MicroVM id from the MicrovmSandbox (KRO) status (microvmID). The +MicrovmSandbox is NOT deleted on suspend (only on claim teardown -> terminate), so +the VM survives suspend/resume cycles. This is a reconcile loop (not preStop hooks) +so it is robust to pod/node loss and resume needs no live pod — pure shim, no +virtual-kubelet, no new image (alpine/k8s = kubectl + aws cli), matching the +pool-manager/bridge pattern. Auth via EKS Pod Identity (empty SA annotations; the +pod-identity association granting lambda-microvms Suspend/Resume/Get is created by +the platform). +*/ -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +rules: + # Read Sandboxes (watch operatingMode + read the bridge-written microvm-id annotation) + # and patch them (stamp the last-acted mode to detect transitions). No microvmsandboxes + # access needed — the per-session VM id lives on the Sandbox, not a KRO status. + - apiGroups: ["agents.x-k8s.io"] + resources: ["sandboxes"] + verbs: ["get", "list", "watch", "patch", "update"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: microvm-lifecycle +subjects: + - kind: ServiceAccount + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: microvm-lifecycle-script + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +data: + reconcile.sh: | + #!/bin/sh + # Reconcile Sandbox.operatingMode -> Lambda MicroVM suspend/resume. + # Idempotent: we stamp the last-acted mode on an annotation and only act on change. + set -eu + # bridgeImage is aws-cli v2 (has lambda-microvms; alpine/k8s's aws-cli 1.34 does not) + # but has no kubectl — fetch a static one (same as bridge.sh / security-agent). + if ! command -v kubectl >/dev/null 2>&1; then + echo "[microvm-lifecycle] fetching kubectl..." + ARCH="$(uname -m)"; case "$ARCH" in aarch64|arm64) A=arm64;; *) A=amd64;; esac + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/${A}/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + NS="{{ include "agent-sandbox.namespace" . }}" + REGION="{{ .Values.microvm.region }}" + APIGROUP="{{ .Values.microvm.apiGroup | default "kro.run" }}" + ANN="microvm-lifecycle.agents.x-k8s.io/last-mode" + INTERVAL="{{ .Values.microvm.lifecycle.intervalSeconds | default 15 }}" + echo "[microvm-lifecycle] reconciling every ${INTERVAL}s (ns=${NS} region=${REGION})" + while true; do + # Select Sandboxes on the lambda-microvm substrate by the microvm-id ANNOTATION the + # bridge writes after RunMicrovm — NOT a label. The agent-sandbox operator does NOT + # propagate SandboxTemplate labels onto the Sandbox object, so a label selector + # (agent-sandbox.io/substrate=lambda-microvm) matches NOTHING and the controller + # stays blind to every real session (observed: operatingMode=Suspended set, Sandbox + # went SandboxSuspended, but the VM was never suspended because this loop skipped it). + # Only lambda sessions carry the microvm-id annotation, so it's the reliable signal. + for sb in $(kubectl get sandbox -n "$NS" \ + -o jsonpath='{range .items[?(@.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id)]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do + MODE=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.spec.operatingMode}' 2>/dev/null || echo "Running") + LAST=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath="{.metadata.annotations.${ANN}}" 2>/dev/null || echo "") + [ "$MODE" = "$LAST" ] && continue # no transition + # Resolve THIS session's MicroVM id from the annotation the BRIDGE writes on the + # Sandbox after RunMicrovm (microvm-lifecycle.agents.x-k8s.io/microvm-id). There + # is no per-session MicrovmSandbox anymore — the VM is created imperatively by the + # bridge (SDK), so the id lives on the Sandbox, not in a KRO status. + VMID=$(kubectl get sandbox "$sb" -n "$NS" -o jsonpath='{.metadata.annotations.microvm-lifecycle\.agents\.x-k8s\.io/microvm-id}' 2>/dev/null || echo "") + if [ -z "$VMID" ]; then + echo "[microvm-lifecycle] $sb: mode=$MODE but no microvm-id annotation yet — will retry" + continue + fi + case "$MODE" in + Suspended) + echo "[microvm-lifecycle] $sb: Running->Suspended -> suspend-microvm $VMID" + aws lambda-microvms suspend-microvm --microvm-identifier "$VMID" --region "$REGION" 2>&1 || true + ;; + Running) + echo "[microvm-lifecycle] $sb: Suspended->Running -> resume-microvm $VMID" + aws lambda-microvms resume-microvm --microvm-identifier "$VMID" --region "$REGION" 2>&1 || true + ;; + *) + echo "[microvm-lifecycle] $sb: unknown operatingMode '$MODE' — skipping"; continue ;; + esac + # Stamp the mode we acted on so we don't repeat the call. + kubectl annotate sandbox "$sb" -n "$NS" "${ANN}=${MODE}" --overwrite >/dev/null 2>&1 || true + done + sleep "$INTERVAL" + done +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: microvm-lifecycle + namespace: {{ include "agent-sandbox.namespace" . }} + labels: + {{- include "agent-sandbox.labels" . | nindent 4 }} +spec: + replicas: 1 + selector: + matchLabels: + {{- include "agent-sandbox.selectorLabels" . | nindent 6 }} + app.kubernetes.io/component: microvm-lifecycle + template: + metadata: + labels: + {{- include "agent-sandbox.selectorLabels" . | nindent 8 }} + app.kubernetes.io/component: microvm-lifecycle + spec: + serviceAccountName: microvm-lifecycle + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: lifecycle + image: {{ .Values.microvm.bridgeImage }} + command: ["/bin/sh", "/scripts/reconcile.sh"] + env: + - name: AWS_REGION + value: {{ .Values.microvm.region | quote }} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: { cpu: 25m, memory: 64Mi } + limits: { cpu: 100m, memory: 128Mi } + volumeMounts: + - name: script + mountPath: /scripts + - name: tmp + mountPath: /tmp + volumes: + - name: script + configMap: + name: microvm-lifecycle-script + defaultMode: 0555 + - name: tmp + emptyDir: {} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml new file mode 100644 index 00000000..7d13f124 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/templates/shim/40-kro-graph-rbac.yaml @@ -0,0 +1,74 @@ +{{/* + KRO graph child-resource RBAC. + + The MicrovmSandbox ResourceGraphDefinition (templates/image/) is reconciled by EKS + Managed KRO, whose controller authenticates to the API server as the cluster's KRO + capability role (EKS access entry, session name "KRO"). The AWS-managed + AmazonEKSKROPolicy attached to that access entry grants KRO its own kro.run perms and + discovery, but NOT create/update/delete on the ACK kinds this graph composes. So when + KRO tries to materialize the graph it fails: + + resource reconciliation failed: buckets.s3.services.k8s.aws "coder-microvm-artifacts" + is forbidden: User ".../hub-KROCapabilityRole/KRO" cannot get resource "buckets" ... + + This ClusterRole grants KRO CRUD on EXACTLY the three ACK groups the MicrovmSandbox + graph creates as children — s3 Buckets, iam Roles, and the self-managed lambdamicrovms + MicrovmImages/Microvms — and nothing else. K8s RBAC cannot scope list/watch/create by + resource NAME, so the grant is per-kind; it deliberately does NOT include core/*, apps, + rbac, secrets, or any other ACK service. delete is required so KRO can garbage-collect + the graph's children when a MicrovmSandbox is removed. + + Gated by microvm.enabled AND microvm.podIdentity.kroCapability.enabled. Rendered as + ArgoCD sync-wave -2 (with the controller bootstrap IAM) so KRO can watch/CRUD the + children before the platform MicrovmSandbox instance (wave 1) reconciles. +*/}} +{{- if and .Values.microvm .Values.microvm.enabled }} +{{- with .Values.microvm.kroCapability }} +{{- if .enabled }} +{{- $cluster := $.Values.microvm.podIdentity.clusterName }} +{{- $account := $.Values.microvm.accountId | toString }} +{{- $role := .roleName | default (printf "%s-KROCapabilityRole" $cluster) }} +{{- $session := .sessionName | default "KRO" }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ $cluster }}-kro-microvm-graph + labels: + {{- include "agent-sandbox.labels" $ | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +rules: + # S3 artifact bucket (graph resource `bucket`). + - apiGroups: ["s3.services.k8s.aws"] + resources: ["buckets", "buckets/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Build + exec IAM roles (graph resources `buildRole`, `execRole`). + - apiGroups: ["iam.services.k8s.aws"] + resources: ["roles", "roles/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + # Platform image + (future) per-session instance (graph resource `image`; Microvm is + # driven imperatively by the shim but kept here so KRO can read/GC if ever graphed). + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvmimages", "microvmimages/status", "microvms", "microvms/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ $cluster }}-kro-microvm-graph + labels: + {{- include "agent-sandbox.labels" $ | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ $cluster }}-kro-microvm-graph +subjects: + # The EKS access-entry username Managed KRO's controller presents. + - kind: User + name: "arn:aws:sts::{{ $account }}:assumed-role/{{ $role }}/{{ $session }}" + apiGroup: rbac.authorization.k8s.io +{{- end }} +{{- end }} +{{- end }} diff --git a/gitops/addons/charts/agent-sandbox-lambda/values.yaml b/gitops/addons/charts/agent-sandbox-lambda/values.yaml new file mode 100644 index 00000000..60009822 --- /dev/null +++ b/gitops/addons/charts/agent-sandbox-lambda/values.yaml @@ -0,0 +1,125 @@ +# agent-sandbox-lambda — Flow D (Lambda MicroVM substrate) values. +# +# Opt-in second substrate for the Agent Sandbox capability. Disabled by default; +# set microvm.enabled=true (per-cluster overlay) to render it. Runs in the SAME +# namespace as the Kata agent-sandbox chart and reuses the same operator + CRDs — +# it only ADDS the Lambda-MicroVM image build (KRO) + the bridge/lifecycle shim. + +# Namespace the capability runs in (must match the agent-sandbox chart). +namespace: agent-sandbox-system + +# Warm-pool template name stem (the bridge SandboxTemplate is -microvm, +# matching the Kata chart's convention so df-run's claim step resolves it). +warmPool: + templateName: coder-sandbox + +# SandboxTemplate env-injection policy (same contract as the Kata template so Flow B +# is unchanged). +coderTemplate: + envVarsInjectionPolicy: Allowed + +# ── Lambda MicroVM (Flow D) ────────────────────────────────────────────────── +microvm: + # Master gate — Flow D stays dormant until a cluster opts in. + enabled: false + region: us-west-2 + # API group the generated MicrovmSandbox CRD is served under (KRO schema.group). + # MUST be "kro.run": EKS Managed KRO's controller only watches the kro.run group — + # an RGD whose schema.group is anything else (even a *.kro.run subdomain) never leaves + # state=Inactive ("cache sync timeout ... Resource=microvmsandboxes"), because the + # capability's controller identity (hub-KROCapabilityRole via AmazonEKSKROPolicy) has + # no list/watch on other groups. The generated CRD is microvmsandboxes.kro.run; the + # kind (MicrovmSandbox) is unchanged. Verified by group-probe on the hub 2026-08-03. + apiGroup: kro.run + # Bridge/lifecycle pod image: needs the AWS CLI that KNOWS the pre-GA lambda-microvms + # service (RunMicrovm/suspend/resume/terminate) AND kubectl. alpine/k8s bundles kubectl + # but its aws-cli 1.34 does NOT have lambda-microvms (verified: prints the service list + # = unrecognized). So use the glibc, always-current aws-cli v2 (has lambda-microvms) and + # fetch kubectl at start (bridge.sh / lifecycle loop do this, same as the security-agent). + bridgeImage: public.ecr.aws/aws-cli/aws-cli:latest + + # K8s API-server egress for the bridge (microvm-bridge-apiserver-egress NetworkPolicy). + # The bridge needs :443 to the API server to read the MicrovmSandbox handoff + annotate + # the Sandbox, but the shared coder-egress policy denies the service+VPC CIDRs. These + # scope the additive allow to the API server only. apiServerCidr = the `kubernetes` + # service ClusterIP /32; apiServerEndpointCidrs = the VPC range holding the apiserver + # endpoint IPs. Override per cluster (kubectl get svc kubernetes; get endpoints kubernetes). + apiServerCidr: "172.20.0.1/32" + apiServerEndpointCidrs: + - "10.0.0.0/16" + + # AWS account id — used to construct the controller role ARN for the ACK + # PodIdentityAssociation (which takes an ARN, not a role-ref). Overlay per cluster. + accountId: "" + + # Controller bootstrap IAM (ALL-ACK: iam.services.k8s.aws Role + eks.services.k8s.aws + # PodIdentityAssociation — templates/shim/00-controller-pod-identity.yaml). This is + # the ONE IAM the KRO RGD can't self-create (creds-before-create). Managed by the + # Managed-ACK iam+eks controllers on the hub. The bridge/lifecycle SAs reuse this + # same role (they also call lambda-microvms). + podIdentity: + clusterName: hub + controllerNamespace: ack-system + controllerServiceAccount: ack-lambdamicrovms-controller + + # EKS Managed KRO runs its controller as the cluster's KRO capability role. On this + # cluster its k8s identity is the EKS access-entry username + # arn:aws:sts:::assumed-role/-KROCapabilityRole/KRO + # (session name "KRO"). AmazonEKSKROPolicy grants KRO its own kro.run perms but NOT + # CRUD on the ACK children the MicrovmSandbox RGD graph creates (s3 buckets, iam + # roles, lambdamicrovms images/instances) — so KRO's instance reconcile hits + # "forbidden: ... cannot get resource buckets". templates/shim/40-kro-graph-rbac.yaml + # grants exactly those child kinds to this identity. Override roleName/sessionName if + # your cluster's capability wiring differs (confirm via + # aws eks describe-access-entry --principal-arn .../KROCapabilityRole). + kroCapability: + enabled: true + roleName: "" # defaults to "-KROCapabilityRole" + sessionName: KRO + + # The ONE platform image built by KRO/ACK (10-rgd-microvm-image.yaml). Built ONCE + # per cluster; the shim reads its status (imageARN + executionRoleARN) to RunMicrovm + # per session. Lambda MicroVM is ARM_64-ONLY. codeArtifactUri is an S3 URI + # (s3://bucket/key) of a zip containing the coder app + a Dockerfile — NOT an ECR + # image ref (the Dockerfile inside MAY pull private ECR base layers; the build role + # keeps ecr:Get*/BatchGetImage). Publish the arm64 coder artifact before enabling. + # baseImageARN: arn:aws:lambda::aws:microvm-image:al2023-1 + # codeArtifactUri: s3:///dark-factory-coder--arm64.zip + baseImageARN: "" + codeArtifactUri: "" + image: + # Render the single committed MicrovmSandbox instance that triggers the build. + enabled: true + # Name of that platform image object; the bridge reads its status by this name. + name: coder + + # Idle policy for RunMicrovm (auto-suspend/resume). EXPLICIT suspend/resume across + # the review→fix loop is driven by the microvm-lifecycle controller off + # Sandbox.operatingMode — see lifecycle below and project_flow_d_lifecycle memory: + # coder codes → SUSPEND → agents review → (fix findings) → RESUME same VM → + # … loop until cleared → merge/exit → TERMINATE. + defaults: + # maxIdleDurationSeconds: how long a RUNNING VM may sit idle (no inbound) before the + # runtime suspends it. Must exceed a coder run (a few min) — the bridge suspends + # explicitly anyway, this is just a backstop. + maxIdleDurationSeconds: 1800 + # suspendedDurationSeconds: how long a SUSPENDED VM is kept before the runtime + # AUTO-TERMINATES it. THIS IS CRITICAL for the review→fix loop: the VM is suspended + # while the external review agents run (~8–15 min) and then waits for a human to post + # "fix findings" (minutes to hours). At the old 300s (5 min) the VM was ALWAYS + # auto-terminated before the fix round, so RESUME hit "has been terminated and its + # state cannot be changed" and the same-VM warm-resume was impossible. Keep it + # suspended long enough to span a realistic review+human cycle (24h); teardown + # terminates it explicitly at merge, so this only bounds abandoned PRs. + suspendedDurationSeconds: 86400 + + # microvm-lifecycle controller (30-microvm-lifecycle.yaml) — reconciles + # Sandbox.operatingMode Running↔Suspended → suspend/resume-microvm. Loop interval. + lifecycle: + intervalSeconds: 15 + + # Lambda-MicroVM warm pool the df-run claim binds when the darkfactory-lambda label + # fires (Flow D). Small — bridge pods are tiny (the real coder runs in the MicroVM). + warmPool: + name: coder-warmpool-microvm + targetIdle: 1 diff --git a/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml b/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml index 183cb77e..545c3c2d 100644 --- a/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml +++ b/gitops/addons/charts/agent-sandbox/templates/30-networkpolicy.yaml @@ -41,6 +41,24 @@ spec: port: 53 - protocol: TCP port: 53 + # DNS to the PUBLIC resolvers the Kata guest VM actually uses (pod + # dnsConfig.nameservers). The in-cluster :53 rule above can't cover these — + # they're external IPs — and Auto Mode has no kube-dns Service, so without + # this the coder can't resolve ANY name (crashes EAI_AGAIN api.github.com even + # though :443 egress works). Scoped to the resolver /32s (from values), so this + # opens :53 to nothing but the DNS servers themselves. + {{- if .Values.networkPolicy.dnsResolvers }} + - to: + {{- range .Values.networkPolicy.dnsResolvers }} + - ipBlock: + cidr: {{ . | quote }} + {{- end }} + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + {{- end }} # Bifrost LLM gateway (in-cluster) — the ONLY control-plane-side service the # coder may reach, and only on :8080. - to: diff --git a/gitops/addons/charts/agent-sandbox/values.yaml b/gitops/addons/charts/agent-sandbox/values.yaml index 1c896db1..37ee7831 100644 --- a/gitops/addons/charts/agent-sandbox/values.yaml +++ b/gitops/addons/charts/agent-sandbox/values.yaml @@ -63,6 +63,17 @@ networkPolicy: - 172.16.0.0/12 # RFC-1918 - 192.168.0.0/16 # RFC-1918 - 169.254.0.0/16 # link-local (blocks IMDS 169.254.169.254) + # Public DNS resolvers the coder Kata VM uses (its pod dnsConfig.nameservers). + # A Kata guest VM can't use in-cluster CoreDNS (and EKS Auto Mode doesn't run a + # kube-dns Service here), so the coder resolves via PUBLIC DNS. The egress policy + # must therefore allow :53 to these resolver IPs — the in-cluster :53 rule alone + # does NOT (public resolver IPs aren't in-cluster), which silently broke name + # resolution (coder crashed with EAI_AGAIN api.github.com though :443 worked). + # Scoped to the exact resolver /32s — tighter than the :443 public allow. Keep in + # sync with the SandboxTemplate podTemplate dnsConfig.nameservers. + dnsResolvers: + - 8.8.8.8/32 # Google Public DNS + - 1.1.1.1/32 # Cloudflare DNS # Admin-tier ClusterNetworkPolicy (31-clusternetworkpolicy.yaml) — control-plane # isolation that applies regardless of pod ownership (the standard NetworkPolicy diff --git a/gitops/addons/charts/dark-factory/scripts/iterate.js b/gitops/addons/charts/dark-factory/scripts/iterate.js index a155bc41..63d9bfd9 100644 --- a/gitops/addons/charts/dark-factory/scripts/iterate.js +++ b/gitops/addons/charts/dark-factory/scripts/iterate.js @@ -96,6 +96,17 @@ async function main() { // Iteration cap: count via a df-iterations/ label on the PR (issue API). const issue = await gh("GET", `/repos/${REPO}/issues/${PR}`); const labels = (issue.labels || []).map((l) => (typeof l === "string" ? l : l.name)); + // Substrate routing: the fix round must run on the SAME substrate the PR came from, + // so it lands on the right warm pool (Lambda MicroVM vs Kata). The ORIGINATING ISSUE + // (not the PR — the coder doesn't copy the label onto the PR) carries the label that + // fired it. Read the issue's labels; default to plain dark-factory (Kata) on any miss. + let triggerLabel = "dark-factory"; + try { + const origIssue = await gh("GET", `/repos/${REPO}/issues/${issueNumber}`); + const il = (origIssue.labels || []).map((l) => (typeof l === "string" ? l : l.name)); + if (il.includes("darkfactory-lambda")) triggerLabel = "darkfactory-lambda"; + } catch (e) { console.log(`[df-iterate] could not read issue #${issueNumber} labels (${e.message}) — defaulting Kata`); } + console.log(`[df-iterate] substrate trigger-label=${triggerLabel}`); const cur = labels.filter((l) => l.startsWith(ITER_LABEL_PREFIX)).map((l) => parseInt(l.slice(ITER_LABEL_PREFIX.length), 10)).filter((n) => !isNaN(n)); const count = cur.length ? Math.max(...cur) : 0; if (count >= MAX_ITERATIONS) { @@ -109,12 +120,19 @@ async function main() { await gh("POST", `/repos/${REPO}/issues/${PR}/labels`, { labels: [`${ITER_LABEL_PREFIX}${next}`] }).catch(() => {}); console.log(`[df-iterate] revision ${next}/${MAX_ITERATIONS} for issue #${issueNumber} (PR #${PR})`); + // Substrate-routed template: Lambda fix rounds run the MicroVM-native df-run-lambda + // (resumes the SAME suspended VM); Kata fix rounds run the certified df-run. Keyed on + // the originating issue's label (resolved above as triggerLabel). + const isLambda = triggerLabel === "darkfactory-lambda"; + const wfTemplate = isLambda ? "df-run-lambda" : "df-run"; + const wfName = isLambda ? `df-run-lambda-${issueNumber}-i${next}` : `df-run-${issueNumber}-i${next}`; + console.log(`[df-iterate] substrate=${triggerLabel} → template=${wfTemplate}`); const wf = { apiVersion: "argoproj.io/v1alpha1", kind: "Workflow", // Dedup per issue+round so a duplicate comment webhook is a no-op. - metadata: { name: `df-run-${issueNumber}-i${next}`, namespace: ARGO_NAMESPACE }, + metadata: { name: wfName, namespace: ARGO_NAMESPACE }, spec: { - workflowTemplateRef: { name: "df-run" }, + workflowTemplateRef: { name: wfTemplate }, arguments: { parameters: [ { name: "issue-id", value: `${issueNumber}` }, // no id in this payload; number is unique enough for the mutex/claim { name: "issue-number", value: `${issueNumber}` }, @@ -122,6 +140,9 @@ async function main() { { name: "issue-title", value: pr.title }, { name: "issue-body", value: "" }, { name: "base-branch", value: pr.base.ref }, + // Route the fix round to the SAME substrate the PR came from (Lambda vs Kata), + // so claim-sandbox picks the right warm pool. df-run branches its warm-pool on this. + { name: "trigger-label", value: triggerLabel }, // base64 the comment (may be multi-line markdown) so it can't break the // claim-sandbox manifest YAML; the coder decodes it. Leave plain empty. { name: "iterate-note", value: "" }, diff --git a/gitops/addons/charts/dark-factory/scripts/merge.js b/gitops/addons/charts/dark-factory/scripts/merge.js index 28fe432a..c8144403 100644 --- a/gitops/addons/charts/dark-factory/scripts/merge.js +++ b/gitops/addons/charts/dark-factory/scripts/merge.js @@ -87,8 +87,24 @@ async function main() { // if the Security or DevOps agent BOT posted findings (a review body reporting // "N finding(s)" or change-requesting inline review comments). try { - const reviews = (await api("GET", `/repos/${REPO}/pulls/${PR}/reviews?per_page=100`)) || []; - const comments = (await api("GET", `/repos/${REPO}/pulls/${PR}/comments?per_page=100`)) || []; + const allReviews = (await api("GET", `/repos/${REPO}/pulls/${PR}/reviews?per_page=100`)) || []; + const allComments = (await api("GET", `/repos/${REPO}/pulls/${PR}/comments?per_page=100`)) || []; + // CRITICAL: only count findings on the CURRENT head sha. A fix round pushes a NEW + // commit and the agents re-review it (posting fresh commit STATUSES, already checked + // green above); their earlier REVIEW bodies/inline comments remain attached to the + // OLD (superseded) sha. Without this filter merge.js counts those stale first-round + // findings and refuses to merge every PR that was ever fixed — observed on PR #136: + // security/devops reviews on sha 3b11b497 (round 1) blocked a merge whose head + // 8089fa0a (fix round) was fully green. Match reviews by commit_id and inline + // comments by original_commit_id/commit_id to the head sha. + // A review counts only if it was submitted against the current head. An inline + // comment counts only if it was ORIGINALLY filed against the current head + // (original_commit_id) — GitHub auto-advances an inline comment's commit_id to the + // latest head when the line still exists, so round-1 comments reappear with + // commit_id==head; original_commit_id preserves the sha they were truly filed on. + // Matching on original_commit_id excludes those carried-forward round-1 findings. + const reviews = allReviews.filter((r) => r.commit_id === sha); + const comments = allComments.filter((c) => (c.original_commit_id || c.commit_id) === sha); const isSecBot = (l) => /aws-security-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const isDevBot = (l) => /aws-devops-agent/i.test(l || "") && /\[bot\]/i.test(l || ""); const botFindings = (pred) => { diff --git a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml index 4b94ec11..5d0157bd 100644 --- a/gitops/addons/charts/dark-factory/templates/10-rbac.yaml +++ b/gitops/addons/charts/dark-factory/templates/10-rbac.yaml @@ -76,13 +76,29 @@ rules: verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: ["agents.x-k8s.io"] resources: ["sandboxes"] - verbs: ["get", "list", "watch"] + # patch: Flow D suspend/resume flips Sandbox.spec.operatingMode from the workflow. + verbs: ["get", "list", "watch", "patch"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list", "create", "update", "patch"] +{{- if .Values.microvm.enabled }} + # Flow D (df-run-lambda): provision-microvm creates the Microvm CR + its runHookPayload + # Secret directly (no SandboxClaim); set-microvm-power reads the CR for the id; + # df-merge-teardown deletes the CR at merge. Least-privilege, this namespace only. + - apiGroups: ["lambdamicrovms.services.k8s.aws"] + resources: ["microvms"] + verbs: ["get", "list", "watch", "create", "delete"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "create", "update", "patch", "delete"] + # Read the platform image handoff (imageARN + execRoleARN) built once by KRO/ACK. + - apiGroups: ["kro.run"] + resources: ["microvmsandboxes"] + verbs: ["get", "list", "watch"] +{{- end }} --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding @@ -99,6 +115,28 @@ roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: dark-factory-workflow-sandbox +{{- if .Values.microvm.enabled }} +--- +# Flow D: bind the workflow SA to the lambda-microvms role via EKS Pod Identity, so the +# df-run-lambda steps (provision/power/terminate) can call aws lambda-microvms +# (get/suspend/resume/terminate-microvm + create-microvm-auth-token). Reuses the SAME +# role the bridge/lifecycle controllers already use (user-approved) — additive, +# no IAM policy change. The ACK eks controller reconciles this into a real EKS association. +apiVersion: eks.services.k8s.aws/v1alpha1 +kind: PodIdentityAssociation +metadata: + name: {{ .Values.microvm.podIdentity.clusterName }}-dark-factory-workflow + namespace: {{ .Values.warmPool.namespace }} + labels: + {{- include "dark-factory.labels" . | nindent 4 }} + annotations: + argocd.argoproj.io/sync-wave: "-2" +spec: + clusterName: {{ .Values.microvm.podIdentity.clusterName }} + namespace: {{ .Values.argo.namespace }} + serviceAccount: {{ .Values.microvm.workflowServiceAccount | default "dark-factory-workflow" }} + roleARN: "arn:aws:iam::{{ .Values.microvm.podIdentity.accountId }}:role/{{ .Values.microvm.podIdentity.clusterName }}-ack-lambdamicrovms-controller" +{{- end }} {{- if .Values.deployTest.enabled }} --- # P4 deploy-test: the ONLY cluster-scoped grant. Lets the workflow create/delete diff --git a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml index fc7cae97..da82dae0 100644 --- a/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml +++ b/gitops/addons/charts/dark-factory/templates/21-workflowtemplate-df-merge-teardown.yaml @@ -32,16 +32,29 @@ spec: - name: repo - name: pr-number ttlStrategy: - secondsAfterCompletion: 3600 + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} templates: - name: main dag: tasks: - name: merge template: merge-pr +{{- if .Values.microvm.enabled }} + # Flow D: this is the FINAL exit — TERMINATE the Lambda MicroVM that df-run-lambda + # kept SUSPENDED across the review→fix loop. Deletes the Microvm CR (stable name + # mvm-) → the controller TerminateMicrovm's the VM. No-op for Kata + # (there is no such CR). Advisory — never fails the merge. + - name: microvm-terminate + template: microvm-terminate + dependencies: [merge] +{{- end }} - name: teardown template: teardown-claim +{{- if .Values.microvm.enabled }} + dependencies: [microvm-terminate] +{{- else }} dependencies: [merge] +{{- end }} # ---- Merge the PR — only if every dark-factory/* check is green ---- - name: merge-pr @@ -91,8 +104,36 @@ spec: echo "[df-merge] human-approved PR #${PR} in ${REPO} — verifying + merging" node /scripts/merge.js +{{- if .Values.microvm.enabled }} + # ---- Flow D: terminate the Lambda MicroVM (final exit) ---- + # df-run-lambda kept the VM SUSPENDED (CR mvm-) across the review→fix + # loop. At merge we DELETE that CR → the controller TerminateMicrovm's the VM. Stable + # name, so no Sandbox/annotation lookup needed. No-op for Kata (no such CR). Advisory. + - name: microvm-terminate + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + source: | + set -eu + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + echo "[microvm-terminate] deleting Microvm/${MVM} → controller TerminateMicrovm (final exit)" + kubectl delete microvm "${MVM}" -n "${NS}" --wait=false 2>&1 || echo "[microvm-terminate] delete failed (advisory)" + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + else + echo "[microvm-terminate] no Microvm/${MVM} — nothing to terminate (Kata or already gone)" + fi +{{- end }} + # ---- Teardown the claim (operator refills the pool) ---- - # Delete by the issue-number label (the PR-review event has no issue id). + # Delete by the issue-number label (the PR-review event has no issue id). For Lambda + # this also removes the (idle) bridge pod; the VM is already terminated above. - name: teardown-claim resource: action: delete diff --git a/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml b/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml index 379dd7d7..a02da4b5 100644 --- a/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml +++ b/gitops/addons/charts/dark-factory/templates/22-workflowtemplate-df-iterate.yaml @@ -35,7 +35,7 @@ spec: - name: comment-author value: "" ttlStrategy: - secondsAfterCompletion: 3600 + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} templates: - name: main dag: diff --git a/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml b/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml new file mode 100644 index 00000000..692d7dec --- /dev/null +++ b/gitops/addons/charts/dark-factory/templates/23-workflowtemplate-df-run-lambda.yaml @@ -0,0 +1,945 @@ +{{- /* +df-run — the Flow B P1 pipeline as an Argo WorkflowTemplate. + + trigger → claim warm sandbox → coder implements+tests (in the Kata VM) → + coder opens PR + posts a live sticky status → await human approval. + +Single-cluster on the hub: the claim step creates a SandboxClaim with the issue +injected as env (verified contract: envVarsInjectionPolicy=Allowed), waits for +the operator to bind a warm micro-VM (status Ready), then the coder VM — which +boots the coder image from the SandboxTemplate — reads the env, implements on +df/issue-N, builds+tests, pushes, and opens the PR + sticky comment itself using +the short-TTL GitHub token projected into the VM. The workflow watches the coder +pod to completion, then its onExit handler releases the claim (the operator +refills the pool). P1 stops at "PR open, awaiting human" — merge/teardown + +verification gates are P2-P4. + +Parameters (supplied by the trigger / Sensor): + issue-id, issue-number, repo (owner/name), issue-title, issue-body, base-branch +*/ -}} +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: df-run-lambda + namespace: {{ .Values.argo.namespace }} + labels: + {{- include "dark-factory.labels" . | nindent 4 }} +spec: + serviceAccountName: dark-factory-workflow + # Serialize per issue (no double-claim for the same issue) and cap total + # concurrent runs against the kata pool size. Keyed on issue-NUMBER (stable across + # first run + fix rounds) to match the claim/Sandbox name below — so a fix round is + # serialized against its own issue's first run and can't race a second claim. + synchronization: + mutex: + name: "df-issue-{{`{{workflow.parameters.issue-number}}`}}" + entrypoint: main + arguments: + parameters: + - name: issue-id + - name: issue-number + - name: repo + - name: issue-title + - name: issue-body + value: "" + - name: base-branch + value: main + # Optional revision request (df-iterate passes the PR comment here; empty on + # a first pass). Injected into the coder as DF_ITERATE_NOTE so it revises the + # existing branch to address the feedback. + - name: iterate-note + value: "" + # Base64 of the revision note — used when the note is arbitrary markdown + # (auto-fed agent findings) that can't be injected raw into the claim YAML. + # The coder decodes this first, falling back to the plain iterate-note. + - name: iterate-note-b64 + value: "" + # Which label fired the run selects the coder substrate (Kata default vs + # Lambda MicroVM for darkfactory-lambda). MUST be declared with a default so + # every trigger-label reference resolves even when the submitter (sensor / + # manual) does not pass it. Defaulting to dark-factory keeps the Kata path for + # sensor submits that do not map a label. + - name: trigger-label + value: "dark-factory" + # Always release the claimed sandbox, on success OR failure. + onExit: teardown + ttlStrategy: + secondsAfterCompletion: {{ .Values.argo.workflowTtlSecondsAfterCompletion | default 604800 }} +{{- if .Values.metrics.enabled }} + # Success metrics (Argo-native Prometheus). Argo exposes these on the workflow + # controller's :9090/metrics — scraped into the platform Prometheus. Gives the + # GitOps-native view of the factory: throughput, outcome mix, and lead time. + metrics: + prometheus: + - name: df_runs_total + help: "Dark Factory df-run workflows by status" + labels: + - { key: status, value: "{{`{{workflow.status}}`}}" } + counter: + value: "1" + - name: df_run_duration_seconds + help: "df-run wall-clock duration (lead time proxy)" + gauge: + value: "{{`{{workflow.duration}}`}}" +{{- end }} + templates: + + # ---- DAG ---- + # claim → coder → [ verify fan: holdout-gate ∥ security-review ∥ devops-review ] + # → status. Every verify step runs OUTSIDE the coder (trusted hub pod), is gated + # on a PR existing, and is advisory in v1 (posts a commit status; never fails the + # run unless its *.blocking / blockSeverity is raised). `status` waits for all + # enabled verify steps via its dependencies list. + # ---- DAG (MicroVM-native, NO SandboxClaim/bridge/warm-pool) ---- + # provision-microvm (create Microvm CR + drive /run) → drive-coder (poll GitHub for + # PR) → suspend-microvm (scale-to-zero during review) → [verify fan] → status. + # On a FIX ROUND the workflow first RESUMES the same suspended VM (resume-microvm), + # re-drives /run with the new note, then re-suspends. Teardown (onExit) deletes the + # Microvm CR → controller TerminateMicrovm. + - name: main + dag: + tasks: + # 1. Provision the MicroVM directly (create CR + Secret, wait RUNNING, POST /run). + # On a fix round (iterate-note set) it RESUMES the existing suspended VM instead. + - name: provision-microvm + template: provision-microvm + # 2. Poll GitHub for the PR the coder opens (substrate-agnostic; reused verbatim). + - name: drive-coder + template: await-coder + dependencies: [provision-microvm] + arguments: + parameters: + - name: sandbox + value: "" + # 3. SUSPEND the VM once the coder pushed its PR — scale-to-zero during review. + # Explicit DAG step (this template is Lambda-only, so no substrate gating). + - name: suspend-microvm + template: set-microvm-power + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: action + value: "suspend" +{{- if .Values.holdout.enabled }} + # P2 — holdout gate: hidden scenarios + a different-family judge. + - name: holdout-gate + template: holdout-gate + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: pr-number + value: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}" +{{- end }} +{{- if .Values.review.enabled }} + # P3 — REAL AWS Frontier Agents, ordered: DevOps FIRST, then Security. + # + # DevOps Agent (release readiness) runs via the Claude Code plugin INSIDE + # the coder step (no headless API), so there is no separate DAG task for + # it here — its verdict is reported by the coder and, on a clear verdict, + # the coder applies the `needs-security-review` label. This `devops-gate` + # step confirms the label is present (DevOps cleared) before Security runs. + - name: devops-gate + template: devops-gate + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + arguments: + parameters: + - name: pr-number + value: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}}" + # Security Agent — WAITS FOR the real aws-security-agent[bot]'s own review + # and mirrors its verdict into dark-factory/security. The App bot reviews + # autonomously + IN PARALLEL with the DevOps bot, so this runs alongside + # devops-gate (depends on drive-coder, NOT gated behind DevOps clearing — + # otherwise a slow DevOps review would skip the security signal entirely). + - name: security-agent + template: security-agent + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" +{{- end }} +{{- if .Values.deployTest.enabled }} + # P4 — detect whether the PR touches deployable artifacts (cheap grep of + # the diff). Its output gates the (expensive, K8s-touching) deploy-test. + - name: detect-deployable + template: detect-deployable + dependencies: [drive-coder] + when: "{{`{{tasks.drive-coder.outputs.parameters.pr-number}}`}} != \"\"" + # P4 — deploy-test: THE ONLY step that holds K8s access. Content-aware: + # k8s → ephemeral-namespace apply+probe+teardown; terraform → init+validate. + # Runs only when detect classified the change (kind != none). + - name: deploy-test + template: deploy-test + dependencies: [detect-deployable] + when: "{{`{{tasks.detect-deployable.outputs.parameters.kind}}`}} != none" + arguments: + parameters: + - name: kind + value: "{{`{{tasks.detect-deployable.outputs.parameters.kind}}`}}" +{{- end }} + - name: status + template: sticky-status + # Depend on BOTH agents (devops-gate + security-agent) plus holdout + + # deploy-test so the consolidated verdict is only computed once EVERY + # signal is terminal — required so a ❌ (and the auto-fix decision) is made + # on complete input, and so a clear verdict truly means both agents cleared. + dependencies: + {{- if or .Values.holdout.enabled .Values.review.enabled .Values.deployTest.enabled }} + {{- if .Values.holdout.enabled }} + - holdout-gate + {{- end }} + {{- if .Values.review.enabled }} + - devops-gate + - security-agent + {{- end }} + {{- if .Values.deployTest.enabled }} + - deploy-test + {{- end }} + {{- else }} + - drive-coder + {{- end }} + arguments: + parameters: + - name: phase + value: "pr-open" + + # ---- 1. Provision the Lambda MicroVM directly (NO SandboxClaim/bridge/warm-pool) ---- + # Creates the runHookPayload Secret + the Microvm CR (the ACK lambdamicrovms + # controller does RunMicrovm), waits for RUNNING + an endpoint, mints an auth token, + # and POSTs /run to start the coder. On a FIX ROUND (iterate-note set) it RESUMES the + # same suspended VM if still alive (warm resume — the Flow D value prop), else recreates. + # Every hard-won Flow D fix is encoded here: + # • autoResumeEnabled=false + we never poll the endpoint after /run → suspend STICKS. + # • suspendedDurationSeconds=8h so the VM survives the review→fix window. + # • HTTP_INGRESS (ALL_INGRESS blocks auth-token minting). + # • runHookPayload carries ghToken + DF_* + the iterate note (the coder has no claim env). + # • CR/Secret named mvm- (stable across rounds → one VM per issue). + # • image/exec-role read from the platform MicrovmSandbox status (built once by KRO/ACK). + - name: provision-microvm + outputs: + parameters: + - name: microvm-id + valueFrom: { path: /tmp/vmid } + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: token + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BASE_BRANCH + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: ISSUE_TITLE + value: "{{`{{workflow.parameters.issue-title}}`}}" + - name: ITERATE_NOTE_B64 + value: "{{`{{workflow.parameters.iterate-note-b64}}`}}" + - name: ITERATE_NOTE + value: "{{`{{workflow.parameters.iterate-note}}`}}" + source: | + set -eu + REGION="{{ .Values.microvm.region }}" + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + PLATFORM_IMAGE="{{ .Values.microvm.image.name | default "coder" }}" + MVM="mvm-${ISSUE_NUMBER}" + BRANCH="df/issue-${ISSUE_NUMBER}" + MAXIDLE={{ .Values.microvm.defaults.maxIdleDurationSeconds | default 1800 }} + SUSPDUR={{ .Values.microvm.defaults.suspendedDurationSeconds | default 28800 }} + INGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:HTTP_INGRESS" + EGRESS="arn:aws:lambda:${REGION}:aws:network-connector:aws-network-connector:INTERNET_EGRESS" + LOGGRP="/aws/lambda/microvms/${PLATFORM_IMAGE}-image" + # aws-cli image has no kubectl — fetch a static one (same as the old bridge). + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + # 1) Read the platform image handoff (imageARN + execRoleARN — built ONCE by KRO/ACK). + echo "[provision] reading platform image ${PLATFORM_IMAGE} (waiting for build ready)..." + i=0; IMAGE_ARN=""; EXEC_ROLE="" + while [ "$i" -lt 240 ]; do + IST=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageState}' 2>/dev/null || echo "") + if [ "${IST}" = "CREATED" ] || [ "${IST}" = "UPDATED" ]; then + IMAGE_ARN=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.imageARN}' 2>/dev/null || echo "") + EXEC_ROLE=$(kubectl get microvmsandbox "${PLATFORM_IMAGE}" -n "${NS}" -o jsonpath='{.status.executionRoleARN}' 2>/dev/null || echo "") + [ -n "${IMAGE_ARN}" ] && [ -n "${EXEC_ROLE}" ] && break + fi + i=$((i+1)); sleep 5 + done + [ -z "${IMAGE_ARN}" ] || [ -z "${EXEC_ROLE}" ] && { echo "[provision] ERROR: platform image not ready (imageState=${IST:-none})"; exit 1; } + echo "[provision] image=${IMAGE_ARN} execRole=${EXEC_ROLE}" + # 2) FIX ROUND vs FIRST RUN: if a CR exists, check the VM's ACTUAL AWS state. + # RUNNING/SUSPENDED/PENDING → RESUME the same VM (warm resume). Else (TERMINATED + # past the 8h suspend cap, or a service error) → recreate a fresh VM. + FRESH=1 + if kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1; then + PRIORVMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + PRIORSTATE="" + [ -n "${PRIORVMID}" ] && PRIORSTATE=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${PRIORVMID}" --query 'state' --output text 2>/dev/null || echo "") + echo "[provision] fix round: prior VM ${PRIORVMID:-none} state=${PRIORSTATE:-unknown}" + case "${PRIORSTATE}" in + RUNNING|SUSPENDED|PENDING) + echo "[provision] RESUMING same VM ${PRIORVMID} (warm resume)" + aws lambda-microvms resume-microvm --region "${REGION}" --microvm-identifier "${PRIORVMID}" 2>/dev/null || true + # Refresh the payload Secret so the resumed hook-server gets the NEW note. + FRESH=0; VMID="${PRIORVMID}" + ;; + *) + echo "[provision] prior VM not resumable (${PRIORSTATE:-gone}) — recreating fresh" + kubectl delete microvm "${MVM}" -n "${NS}" --wait=false >/dev/null 2>&1 || true + kubectl delete secret "${MVM}-payload" -n "${NS}" --ignore-not-found >/dev/null 2>&1 || true + j=0; while [ "$j" -lt 60 ]; do kubectl get microvm "${MVM}" -n "${NS}" >/dev/null 2>&1 || break; j=$((j+1)); sleep 5; done + ;; + esac + fi + # 3) Build the runHookPayload (ghToken + DF_* + iterate note) and (re)write the Secret. + PAYLOAD=$(GH="${GH_TOKEN}" REGION="${REGION}" IN="${ISSUE_NUMBER}" RP="${REPO}" BR="${BRANCH}" BB="${BASE_BRANCH}" IT="${ISSUE_TITLE}" NB="${ITERATE_NOTE_B64}" NP="${ITERATE_NOTE}" python3 -c 'import json,os; e=os.environ.get; print(json.dumps({"ghToken":e("GH",""),"region":e("REGION","us-west-2"),"issueNumber":e("IN",""),"repo":e("RP",""),"branch":e("BR",""),"baseBranch":e("BB","main"),"issueTitle":e("IT",""),"iterateNoteB64":e("NB",""),"iterateNote":e("NP","")}))') + MVM="${MVM}" NS="${NS}" PAYLOAD="${PAYLOAD}" python3 -c 'import json,os; e=os.environ; print(json.dumps({"apiVersion":"v1","kind":"Secret","metadata":{"name":e["MVM"]+"-payload","namespace":e["NS"]},"type":"Opaque","stringData":{"payload":e["PAYLOAD"]}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[provision] payload secret apply failed"; exit 1; } + # 4) On a FIRST RUN (or recreate), create the Microvm CR (autoResume=false). + if [ "${FRESH}" = "1" ]; then + MVM="${MVM}" NS="${NS}" IMG="${IMAGE_ARN}" EXECROLE="${EXEC_ROLE}" MAXIDLE="${MAXIDLE}" SUSPDUR="${SUSPDUR}" INGRESS="${INGRESS}" EGRESS="${EGRESS}" LOGGRP="${LOGGRP}" python3 -c 'import json,os; e=os.environ; mvm=e["MVM"]; ns=e["NS"]; print(json.dumps({"apiVersion":"lambdamicrovms.services.k8s.aws/v1alpha1","kind":"Microvm","metadata":{"name":mvm,"namespace":ns},"spec":{"imageIdentifier":e["IMG"],"executionRoleARN":e["EXECROLE"],"ingressNetworkConnectors":[e["INGRESS"]],"egressNetworkConnectors":[e["EGRESS"]],"runHookPayload":{"name":mvm+"-payload","key":"payload","namespace":ns},"logging":{"cloudWatch":{"logGroup":e["LOGGRP"],"logStream":"runtime-"+mvm}},"idlePolicy":{"autoResumeEnabled":False,"maxIdleDurationSeconds":int(e["MAXIDLE"]),"suspendedDurationSeconds":int(e["SUSPDUR"])}}}))' | kubectl apply -f - >/dev/null 2>&1 || { echo "[provision] Microvm CR apply failed"; exit 1; } + VMID=""; i=0 + while [ "$i" -lt 60 ]; do + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -n "${VMID}" ] && break; i=$((i+1)); sleep 5 + done + fi + echo "${VMID:-}" > /tmp/vmid + [ -z "${VMID:-}" ] && { echo "[provision] ERROR: no microvmID"; exit 1; } + echo "[provision] Microvm ${MVM} -> ${VMID}" + # 5) Wait for RUNNING + endpoint, mint token, POST /run (background-spawns the coder). + EP=""; i=0 + while [ "$i" -lt 72 ]; do + S=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'state' --output text 2>/dev/null || echo "") + EP=$(aws lambda-microvms get-microvm --region "${REGION}" --microvm-identifier "${VMID}" --query 'endpoint' --output text 2>/dev/null || echo "") + [ "$S" = "RUNNING" ] && [ -n "${EP}" ] && [ "${EP}" != "None" ] && break + i=$((i+1)); sleep 5 + done + [ -z "${EP}" ] || [ "${EP}" = "None" ] && { echo "[provision] ERROR: VM never reached RUNNING+endpoint (state=${S:-none})"; exit 1; } + TOKEN=$(aws lambda-microvms create-microvm-auth-token --region "${REGION}" --microvm-identifier "${VMID}" --expiration-in-minutes 60 --allowed-ports 'port=8080' 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin)["authToken"]["X-aws-proxy-auth"])' 2>/dev/null || echo "") + [ -z "${TOKEN}" ] && { echo "[provision] ERROR: could not mint auth token"; exit 1; } + echo "[provision] driving coder: POST /run on ${EP}" + RC=$(curl -sS -m 30 -o /tmp/run.out -w '%{http_code}' -X POST "https://${EP}/run" -H "X-aws-proxy-auth: ${TOKEN}" -H 'Content-Type: application/json' -d "${PAYLOAD}" 2>/tmp/run.err || echo "000") + echo "[provision] /run -> HTTP ${RC} $(cat /tmp/run.out 2>/dev/null | head -c 120)" + case "${RC}" in 2*) echo "[provision] coder started." ;; *) echo "[provision] ERROR: /run returned ${RC}"; exit 1 ;; esac + + # ---- 2. Wait for the coder VM to finish — GitHub is the completion bus ---- + # The coder image (baked into the SandboxTemplate) auto-runs on VM start, + # reads DF_* env, implements + tests, pushes df/issue-N, and opens the PR + # itself using the short-TTL gh-token projected into the VM. The coder is + # credential-less to the k8s API (no SA token), so it CANNOT signal via a pod + # annotation — it self-reports through GitHub. This step polls the GitHub API + # (with the workflow's own token) for a PR whose head is df/issue-N, and + # treats the head commit's dark-factory/implementation check as the verdict. + - name: await-coder + inputs: + parameters: + - name: sandbox + activeDeadlineSeconds: {{ mul .Values.coder.runTimeoutMinutes 60 }} + script: + image: {{ .Values.stepImage }} + command: [sh] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + source: | + set -eu + command -v curl >/dev/null 2>&1 || apk add --no-cache curl jq >/dev/null 2>&1 + REPO="{{`{{workflow.parameters.repo}}`}}" + BRANCH="df/issue-{{`{{workflow.parameters.issue-number}}`}}" + ITERATE_NOTE="{{`{{workflow.parameters.iterate-note}}`}}" + ITERATE_NOTE_B64="{{`{{workflow.parameters.iterate-note-b64}}`}}" + API="https://api.github.com/repos/${REPO}" + # ROUND-AWARENESS: on a FIX ROUND (iterate-note set) the branch already + # exists and its OLD commit already has dark-factory/implementation=success. + # If we accept that, the verify fan (holdout/security/devops) runs against + # the STALE pre-fix commit before the coder's new push lands (observed 45s + # race). So record the branch's starting SHA and require a DIFFERENT head + # SHA (a genuinely new commit) before we consider the coder done on a fix + # round. First-pass runs (no iterate-note) have no prior commit → accept the + # first commit that reports impl=success. + START_SHA="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/branches/${BRANCH}" 2>/dev/null | jq -r '.commit.sha // empty' || echo "")" + IS_FIX="false"; { [ -n "${ITERATE_NOTE}" ] || [ -n "${ITERATE_NOTE_B64}" ]; } && IS_FIX="true" + echo "[df-run] polling GitHub for PR head=${BRANCH} (fix-round=${IS_FIX}, start-sha=${START_SHA:-none})..." + while true; do + pr="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" \ + -H "Accept: application/vnd.github+json" \ + "${API}/pulls?head=${REPO%%/*}:${BRANCH}&state=open" 2>/dev/null || echo '[]')" + n="$(echo "${pr}" | jq -r '.[0].number // empty')" + if [ -n "${n}" ]; then + sha="$(echo "${pr}" | jq -r '.[0].head.sha')" + # On a fix round, ignore the old commit until the coder pushes a new one. + if [ "${IS_FIX}" = "true" ] && [ -n "${START_SHA}" ] && [ "${sha}" = "${START_SHA}" ]; then + echo "[df-run] fix-round: head still at start sha ${sha} — waiting for the coder's new commit..." + sleep 15; continue + fi + st="$(curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/commits/${sha}/status" 2>/dev/null | jq -r '.statuses[] | select(.context=="dark-factory/implementation") | .state' | head -1)" + echo "[df-run] PR #${n} open (sha=${sha}) impl-status=${st:-pending}" + case "${st}" in + success) echo "[df-run] coder finished — PR #${n} @ ${sha} ready."; echo "${n}" > /tmp/pr; echo "${sha}" > /tmp/sha; exit 0 ;; + failure|error) echo "[df-run] coder reported failure."; exit 1 ;; + esac + else + echo "[df-run] no PR yet — coder still working..." + fi + sleep 15 + done + outputs: + parameters: + - name: pr-number + valueFrom: + path: /tmp/pr + default: "" + - name: head-sha + valueFrom: + path: /tmp/sha + default: "" + +{{- if .Values.holdout.enabled }} + # ---- 2b. Holdout gate (P2) — train/test separation for code ---- + # Runs on the HUB (a trusted step, NOT the Kata VM). Checks out the coder's + # PR branch, diffs it vs base, then runs evaluate.js: each hidden scenario + # gets its executable test run against the built code AND a different-family + # judge (Nova) vote (2-of-3). Gate = passRatio >= threshold. The hidden + # scenarios come from the df-holdout-* ConfigMaps mounted here — they are + # NEVER mounted into the sandbox, so the coder cannot see or edit them. + # v1 is advisory: the gate result is posted as the dark-factory/holdout commit + # status; the workflow only fails on a red gate when holdout.blocking=true. + - name: holdout-gate + inputs: + parameters: + - name: pr-number + activeDeadlineSeconds: 900 + volumes: + - name: holdout-eval + configMap: + name: df-holdout-eval + {{- range $repo := .Values.trigger.argoEvents.repositories }} + {{- range $name := $repo.names }} + - name: holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + configMap: + name: df-holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + {{- end }} + {{- end }} + script: + image: {{ .Values.holdout.evalImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: BIFROST_URL + value: {{ .Values.holdout.bifrostUrl | quote }} + - name: JUDGE_MODEL + value: {{ .Values.holdout.judgeModel | quote }} + - name: JUDGE_RUNS + value: {{ .Values.holdout.judgeRuns | quote }} + - name: JUDGE_QUORUM + value: {{ .Values.holdout.judgeQuorum | quote }} + - name: THRESHOLD + value: {{ .Values.holdout.threshold | quote }} + volumeMounts: + - name: holdout-eval + mountPath: /holdout/evaluate.js + subPath: evaluate.js + {{- range $repo := .Values.trigger.argoEvents.repositories }} + {{- range $name := $repo.names }} + - name: holdout-{{ printf "%s-%s" $repo.owner $name | lower }} + mountPath: /holdout/{{ printf "%s-%s" $repo.owner $name }} + {{- end }} + {{- end }} + source: | + set -eu + REPO="{{`{{workflow.parameters.repo}}`}}" + BRANCH="df/issue-{{`{{workflow.parameters.issue-number}}`}}" + BASE="{{`{{workflow.parameters.base-branch}}`}}" + SLUG="$(echo "${REPO}" | tr '/' '-')" + API="https://api.github.com/repos/${REPO}" + WORK=/tmp/holdout-work + rm -rf "${WORK}"; mkdir -p "${WORK}" + echo "[holdout] cloning ${REPO}@${BRANCH} for evaluation..." + git clone --quiet --branch "${BRANCH}" \ + "https://x-access-token:${GH_TOKEN}@github.com/${REPO}.git" "${WORK}/repo" + cd "${WORK}/repo" + # Diff vs base — use the GitHub COMPARE API (authoritative changed-file + # patch), NOT a local git diff. A branch clone shares no reliable merge-base + # with a shallow base fetch, so `origin/BASE...HEAD` yields the WHOLE file as + # "added" (observed: a sum-only PR's diff contained every pre-existing + # function → every scenario's appliesWhen matched → wrong scenarios graded → + # false holdout failure). The compare API returns only the real per-file + # patch hunks, so appliesWhen keys on the ACTUAL change. (Same fix as + # detect-deployable.) Concatenate every file's patch into /tmp/diff.patch. + PR_NUM="{{`{{inputs.parameters.pr-number}}`}}" + curl -fsS -H "Authorization: Bearer ${GH_TOKEN}" -H "Accept: application/vnd.github+json" \ + "${API}/pulls/${PR_NUM}/files?per_page=100" 2>/dev/null \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); files=d if isinstance(d,list) else []; open("/tmp/diff.patch","w").write("\n".join("diff --git a/%s b/%s\n%s"%(f.get("filename",""),f.get("filename",""),f.get("patch","")) for f in files))' 2>/dev/null || echo "" > /tmp/diff.patch + # Fallback: if the API path produced nothing, fall back to a local diff. + [ -s /tmp/diff.patch ] || { git fetch --quiet --depth 1 origin "${BASE}" 2>/dev/null || true; git diff "origin/${BASE}...HEAD" > /tmp/diff.patch 2>/dev/null || echo "" > /tmp/diff.patch; } + echo "[holdout] diff.patch $(wc -l < /tmp/diff.patch) lines (via compare API)" + # Build/install so executable tests can require the module. + if [ -f package.json ]; then npm install --no-audit --no-fund >/dev/null 2>&1 || true; fi + SHA="$(git rev-parse HEAD)" + + SCEN="/holdout/${SLUG}/scenarios.json" + if [ ! -f "${SCEN}" ]; then + echo "[holdout] no hidden scenarios for ${SLUG} — skipping gate (advisory)." + exit 0 + fi + + echo "[holdout] evaluating against hidden scenarios..." + set +e + REPO_DIR="${WORK}/repo" DIFF=/tmp/diff.patch SCENARIOS="${SCEN}" \ + OUT=/tmp/holdout-result.json node /holdout/evaluate.js + GATE=$? + set -e + + RATIO="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(Math.round(r.ratio*100))' 2>/dev/null || echo 0)" + PASSED="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.passed+"/"+r.total)' 2>/dev/null || echo '?/?')" + TOTAL="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.total)' 2>/dev/null || echo 0)" + SKIPPED="$(node -e 'const r=require("/tmp/holdout-result.json");console.log(r.skipped||0)' 2>/dev/null || echo 0)" + # Distinguish the three outcomes so the PR reads honestly: + # total=0 → no hidden scenario matched this change (e.g. a Terraform PR + # vs JS-only scenarios) → NOT APPLICABLE, green as n/a. + # gate ok → passed. gate fail → below threshold. + if [ "${TOTAL}" = "0" ]; then + STATE=success; DESC="not applicable — no hidden scenarios match this change (${SKIPPED} skipped)" + elif [ "${GATE}" -eq 0 ]; then + STATE=success; DESC="holdout ${PASSED} (${RATIO}%) — gate passed" + else + STATE=failure; DESC="holdout ${PASSED} (${RATIO}%) — below threshold" + fi + + # Post the holdout verdict as a commit status on the PR head SHA. The + # coder image has no curl, so use node's https (always present). Retry + # transient failures so a blip doesn't drop the verdict. + GH_TOKEN="${GH_TOKEN}" REPO="${REPO}" SHA="${SHA}" STATE="${STATE}" DESC="${DESC}" node -e ' + const https=require("https"); + const body=JSON.stringify({state:process.env.STATE,context:"dark-factory/holdout",description:process.env.DESC}); + let n=0; + (function post(){ + const req=https.request({host:"api.github.com",method:"POST",path:"/repos/"+process.env.REPO+"/statuses/"+process.env.SHA, + headers:{"User-Agent":"dark-factory-holdout","Authorization":"Bearer "+process.env.GH_TOKEN,"Accept":"application/vnd.github+json","Content-Type":"application/json","Content-Length":Buffer.byteLength(body)}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{if(r.statusCode>=300&&r.statusCode<500){console.error("status post "+r.statusCode+": "+b.slice(0,120))}else if(r.statusCode>=500&&++n<4){setTimeout(post,500*n);return} console.log("[holdout] posted dark-factory/holdout="+process.env.STATE)});}); + req.on("error",e=>{if(++n<4){setTimeout(post,500*n)}else{console.error("status post error: "+e.message)}}); + req.write(body);req.end(); + })(); + ' || true + + {{- if .Values.holdout.blocking }} + echo "[holdout] blocking mode — gate result gates the workflow." + exit ${GATE} + {{- else }} + echo "[holdout] advisory mode — gate result reported, workflow continues." + exit 0 + {{- end }} + outputs: + parameters: + - name: ratio + valueFrom: + path: /tmp/holdout-result.json + default: "{}" +{{- end }} + +{{- if .Values.review.enabled }} + # ---- 2c. devops-gate (P3, step 1) — wait for the AWS DevOps Agent to clear ---- + # The AWS DevOps Agent release-readiness review is its own GitHub App that + # auto-reviews every PR and posts a check-run / commit status (its native model + # — there is NO headless code-review API). This step polls the PR head for that + # check and emits cleared=true|false, which gates the Security Agent step so + # DevOps reviews FIRST, Security SECOND (docs §6.2). + # + # Mode (devopsAgent.gate): + # check → wait for the DevOps Agent's own check-run (context matches + # devopsAgent.checkContext) to conclude success/neutral. This is the + # real GitHub-App path (needs the one-time console repo connect). + # label → fallback: wait for the coder-applied handoffLabel (used only if + # you drive DevOps via the coding-agent plugin instead of the App). + # If DevOps never reports within the window, cleared=false → Security is skipped + # and the sticky status shows DevOps not-run (NEVER a fake pass). Advisory: this + # step itself never fails the run. + - name: devops-gate + inputs: + parameters: + - name: pr-number + activeDeadlineSeconds: {{ add .Values.devopsAgent.waitSeconds 60 }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: PR + value: "{{`{{inputs.parameters.pr-number}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: GATE_MODE + value: {{ .Values.devopsAgent.gate | quote }} + - name: LABEL + value: {{ .Values.review.handoffLabel | quote }} + - name: CHECK_CONTEXT + value: {{ .Values.devopsAgent.checkContext | quote }} + - name: WAIT_SECONDS + value: {{ .Values.devopsAgent.waitSeconds | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/comment.js + subPath: comment.js + source: | + set -eu + echo "[devops-gate] mode=${GATE_MODE} PR #${PR} — waiting for AWS DevOps Agent..." + DEADLINE=$(( $(date +%s) + WAIT_SECONDS )) + CLEARED=false + while [ "$(date +%s)" -lt "$DEADLINE" ]; do + if [ "$GATE_MODE" = "label" ]; then + # Fallback: coder-applied handoff label present on the PR. + R="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" PR="$PR" LABEL="$LABEL" node -e ' + const https=require("https");const {GH_TOKEN,REPO,PR,LABEL}=process.env; + https.get({host:"api.github.com",path:"/repos/"+REPO+"/issues/"+PR+"/labels",headers:{"User-Agent":"df-devops-gate","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{const ls=JSON.parse(b).map(x=>x.name);console.log(ls.includes(LABEL)?"cleared":"waiting")}catch(e){console.log("waiting")}})}).on("error",()=>console.log("waiting"));' 2>/dev/null || echo waiting)" + else + # Default: the DevOps Agent GitHub App's check-run on the PR head SHA. + # cleared iff a matching context concluded success/neutral; "blocked" + # if it concluded failure (BLOCK) — we stop and leave cleared=false. + R="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" BRANCH="$BRANCH" CHECK_CONTEXT="$CHECK_CONTEXT" node -e ' + const https=require("https");const {GH_TOKEN,REPO,BRANCH,CHECK_CONTEXT}=process.env; + const g=(p)=>new Promise((res)=>{https.get({host:"api.github.com",path:p,headers:{"User-Agent":"df-devops-gate","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}},r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{res(JSON.parse(b))}catch(e){res(null)}})}).on("error",()=>res(null))}); + (async()=>{ + const pr=await g("/repos/"+REPO+"/pulls?head="+REPO.split("/")[0]+":"+BRANCH+"&state=open"); + if(!pr||!pr[0]){return console.log("waiting")} + const sha=pr[0].head.sha; + const re=new RegExp(CHECK_CONTEXT,"i"); + // check-runs API + const cr=await g("/repos/"+REPO+"/commits/"+sha+"/check-runs"); + const runs=(cr&&cr.check_runs)||[]; + const m=runs.filter(x=>re.test(x.name)); + if(m.some(x=>x.status==="completed"&&["success","neutral"].includes(x.conclusion)))return console.log("cleared"); + if(m.some(x=>x.status==="completed"&&["failure","action_required","cancelled","timed_out"].includes(x.conclusion)))return console.log("blocked"); + // legacy commit statuses fallback + const st=await g("/repos/"+REPO+"/commits/"+sha+"/status"); + const ss=((st&&st.statuses)||[]).filter(x=>re.test(x.context)); + if(ss.some(x=>x.state==="success"))return console.log("cleared"); + if(ss.some(x=>["failure","error"].includes(x.state)))return console.log("blocked"); + console.log("waiting"); + })();' 2>/dev/null || echo waiting)" + fi + case "$R" in + cleared) CLEARED=true; echo "[devops-gate] AWS DevOps Agent CLEARED"; break ;; + blocked) CLEARED=false; echo "[devops-gate] AWS DevOps Agent returned BLOCK — Security stays gated"; break ;; + *) echo "[devops-gate] waiting for AWS DevOps Agent review..."; sleep 20 ;; + esac + done + echo "$CLEARED" > /tmp/cleared + echo "[devops-gate] cleared=${CLEARED}" + + # NOTE: the DevOps verdict is NO LONGER relayed as its own PR comment. + # To avoid mixed/duplicate signals, the pipeline posts a SINGLE + # consolidated review (status.js → dark-factory:verdict-review) that + # mirrors the source-of-truth checks (Build/Holdout/Security/DevOps). + # The DevOps Agent's own commit status + the App bot's review remain + # the authoritative DevOps signal; `cleared` still gates the flow below. + outputs: + parameters: + - name: cleared + valueFrom: + path: /tmp/cleared + default: "false" + + # ---- 2d. security-agent (P3, step 2) — REAL AWS Security Agent, headless ---- + # Runs SECOND, only after DevOps cleared (label present). A trusted hub-side + # step that clones df/issue-N read-only, stages {source archive, unified diff} + # SINGLE SECURITY SIGNAL = THE REAL AWS SECURITY AGENT BOT. + # We do NOT run a second headless scan (that redundant path disagreed with the + # bot — reported "no findings" while aws-security-agent[bot] flagged real issues + # — and produced a false LGTM). Instead this step WAITS for the GitHub App bot's + # own review on the PR and MIRRORS its verdict into the dark-factory/security + # commit status (findings -> failure -> merge blocked; clean -> success). Because + # the consolidated `status` step depends on this step, the pipeline now naturally + # waits for the real bot before posting its verdict. Node-only → runs on reviewImage. + - name: security-agent + activeDeadlineSeconds: {{ add .Values.securityAgent.pollTimeoutSeconds 300 }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [node] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + # Findings at/above this severity fail the status (block the merge). The bot + # doesn't expose per-severity counts uniformly, so ANY finding fails unless + # BLOCK_LEVEL=none (then findings are advisory). Default medium. + - name: BLOCK_LEVEL + value: {{ .Values.securityAgent.blockLevel | quote }} + - name: POLL_TIMEOUT + value: {{ .Values.securityAgent.pollTimeoutSeconds | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/security-wait.js + subPath: security-wait.js + source: | + require("/scripts/security-wait.js"); +{{- end }} + +{{- if .Values.deployTest.enabled }} + # ---- 2d. detect-deployable (P4) — does the diff touch deployable artifacts? ---- + # Emits `deployable = true|false`, which gates the expensive deploy-test step. + # Uses the GitHub compare API (authoritative changed-file list) rather than a + # local git diff — a depth-1 clone + depth-1 base fetch share no merge-base, so + # `origin/BASE...HEAD` yields nothing (observed: empty file list → false). + - name: detect-deployable + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: BASE + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: K8S_PATTERNS + value: {{ .Values.deployTest.k8sPatterns | quote }} + - name: TF_PATTERNS + value: {{ .Values.deployTest.terraformPatterns | quote }} + source: | + set -eu + FILES="$(GH_TOKEN="$GH_TOKEN" REPO="$REPO" BASE="$BASE" BRANCH="$BRANCH" node -e ' + const https=require("https"); + const {GH_TOKEN,REPO,BASE,BRANCH}=process.env; + https.get({host:"api.github.com",path:"/repos/"+REPO+"/compare/"+BASE+"..."+BRANCH,headers:{"User-Agent":"df-detect","Authorization":"Bearer "+GH_TOKEN,"Accept":"application/vnd.github+json"}}, + r=>{let b="";r.on("data",c=>b+=c);r.on("end",()=>{try{const j=JSON.parse(b);(j.files||[]).forEach(f=>console.log(f.filename));}catch(e){process.exit(0);}});}).on("error",()=>process.exit(0)); + ')" + echo "[detect] changed files:"; echo "$FILES" | sed 's/^/ /' + # Classify: k8s takes precedence over terraform when both are present. + if echo "$FILES" | grep -qE "$K8S_PATTERNS"; then KIND=k8s + elif echo "$FILES" | grep -qE "$TF_PATTERNS"; then KIND=terraform + else KIND=none; fi + echo "$KIND" > /tmp/kind + echo "[detect] kind=$KIND" + outputs: + parameters: + - name: kind + valueFrom: + path: /tmp/kind + default: "none" + + # ---- 2e. deploy-test (P4) — content-aware; THE ONLY step with K8s access ---- + # Trusted hub step. Validates the change with the RIGHT tool for its kind: + # kind=k8s → ephemeral namespace apply + wait Ready + teardown (trap). + # kind=terraform → terraform init -backend=false + validate (+ fmt check). + # Validation only — NO AWS creds, NO apply (no real infra). + # Posts the dark-factory/deploy-test commit status AND a marker PR comment with + # the details. Advisory in v1 (deployTest.blocking=false). The untrusted coder + # never has K8s access — it only produces the artifacts; this step runs them. + - name: deploy-test + inputs: + parameters: + - name: kind + activeDeadlineSeconds: {{ .Values.deployTest.timeoutSeconds }} + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.deployTest.image }} + command: [bash] + volumeMounts: + - name: review-script + mountPath: /scripts/comment.js + subPath: comment.js + - name: review-script + mountPath: /scripts/deploy-test.sh + subPath: deploy-test.sh + env: + - name: WF_NAME + value: "{{`{{workflow.name}}`}}" + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: KIND + value: "{{`{{inputs.parameters.kind}}`}}" + - name: MANIFEST_PATH + value: {{ .Values.deployTest.manifestPath | quote }} + - name: TF_PATH + value: {{ .Values.deployTest.terraformPath | quote }} + - name: READY_TIMEOUT + value: {{ .Values.deployTest.readyTimeoutSeconds | quote }} + - name: BLOCKING + value: {{ .Values.deployTest.blocking | quote }} + source: | + set -eu + # All logic lives in review/deploy-test.sh (ConfigMap file) — keeping it out + # of this YAML block scalar avoids the shell-quoting hazards that broke + # inline multi-line markdown. Kind-driven so it generalizes to new profiles. + bash /scripts/deploy-test.sh +{{- end }} + + # ---- 3. Sticky status — rewrite the PR body from the live verdicts ---- + # Runs AFTER every verify step (its DAG deps). The coder wrote the PR body at + # PR-open time, before verification ran, so its holdout/security/devops lines + # are placeholders ("pending"). This step reads the authoritative + # dark-factory/* commit STATUSES from GitHub (the source of truth the verify + # steps posted) and rewrites the PR body in place with the real verdicts — the + # "one live sticky status" from the design (README §7). Idempotent: the body + # between the markers is fully regenerated each run. + - name: sticky-status + inputs: + parameters: + - name: phase + volumes: + - name: review-script + configMap: + name: df-review + script: + image: {{ .Values.reviewImage }} + command: [bash] + env: + - name: GH_TOKEN + valueFrom: + secretKeyRef: + name: {{ .Values.github.tokenSecret }} + key: {{ .Values.github.tokenKey }} + - name: REPO + value: "{{`{{workflow.parameters.repo}}`}}" + - name: BRANCH + value: "df/issue-{{`{{workflow.parameters.issue-number}}`}}" + # For the AUTO-FIX loop (status.js): on a ❌ verdict from the real agents, + # collect their findings and submit a bounded df-run revision (iterate-note + # = the findings) so the coder auto-revises — no human paraphrasing needed. + - name: ISSUE_NUMBER + value: "{{`{{workflow.parameters.issue-number}}`}}" + - name: BASE_BRANCH + value: "{{`{{workflow.parameters.base-branch}}`}}" + - name: TRIGGER_LABEL + value: "{{`{{workflow.parameters.trigger-label}}`}}" + - name: ARGO_NAMESPACE + value: {{ .Values.argo.namespace | quote }} + - name: AUTO_FIX_FINDINGS + value: {{ .Values.review.autoFixFindings | default false | quote }} + - name: MAX_ITERATIONS + value: {{ .Values.iterate.maxIterations | default 3 | quote }} +{{- if .Values.devopsAgent.enabled }} + # Real AWS DevOps Agent posts a check-run (not a commit status) named this; + # status.js renders the DevOps row from the check-run when present. + - name: DEVOPS_CHECK + value: {{ .Values.devopsAgent.checkRunName | quote }} +{{- end }} +{{- if and .Values.securityAgent.app .Values.securityAgent.app.enabled }} + # Real AWS Security Agent GitHub App posts its own check/inline review; + # status.js renders the Security row from it when present (else the + # headless dark-factory/security status). + - name: SECURITY_CHECK + value: {{ .Values.securityAgent.app.checkRunName | quote }} +{{- end }} +{{- if .Values.postVerdictReview }} + # Post ONE consolidated verdict review (Security + DevOps results) on the + # PR so both agents' verdicts are ALWAYS visible in the Reviews section — + # the agent Apps review autonomously + inconsistently and cannot be added + # via the requested_reviewers API (verified no-op). status.js posts it once, + # when verification is terminal (idempotent via a hidden marker). + - name: POST_VERDICT_REVIEW + value: "true" +{{- end }} + # Holdout is a train/test QUALITY signal, advisory by default — it does NOT + # gate the merge verdict unless holdout.blocking=true. status.js reads this + # to decide whether a red holdout flips the consolidated verdict. + - name: HOLDOUT_BLOCKING + value: {{ .Values.holdout.blocking | quote }} + volumeMounts: + - name: review-script + mountPath: /scripts/status.js + subPath: status.js + source: | + set -eu + echo "[df-run] phase={{`{{inputs.parameters.phase}}`}} — updating PR body from live verdicts (${REPO} ${BRANCH})" + node /scripts/status.js + + # ---- suspend/resume the Lambda MicroVM directly (scale-to-zero) ---- + # Calls suspend-microvm / resume-microvm on the VM the provision step created (id read + # from the Microvm CR by stable name mvm-). Direct AWS call — NO Sandbox/ + # operatingMode/lifecycle-controller dependency (this template owns the VM outright). + # Advisory: never fails the run (suspend is a cost optimization, not correctness). + - name: set-microvm-power + inputs: + parameters: + - name: action # suspend | resume + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + source: | + set -eu + ACTION="{{`{{inputs.parameters.action}}`}}" + REGION="{{ .Values.microvm.region }}" + NS="{{ .Values.microvm.namespace | default "agent-sandbox-system" }}" + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + if ! command -v kubectl >/dev/null 2>&1; then + KV="$(curl -fsSL https://dl.k8s.io/release/stable.txt)" + curl -fsSL "https://dl.k8s.io/release/${KV}/bin/linux/amd64/kubectl" -o /tmp/kubectl + chmod +x /tmp/kubectl; export PATH="/tmp:$PATH" + fi + VMID=$(kubectl get microvm "${MVM}" -n "${NS}" -o jsonpath='{.status.microvmID}' 2>/dev/null || echo "") + [ -z "${VMID}" ] && { echo "[power] no microvmID for ${MVM} — skipping (advisory)"; exit 0; } + echo "[power] ${ACTION}-microvm ${VMID}" + aws lambda-microvms "${ACTION}-microvm" --region "${REGION}" --microvm-identifier "${VMID}" 2>&1 || echo "[power] ${ACTION} skipped/failed (advisory)" + echo "[power] done." + activeDeadlineSeconds: 180 + + # ---- onExit: KEEP the suspended VM (terminated only at merge) ---- + # Flow D lifecycle: the VM is SUSPENDED (by suspend-microvm above) and must SURVIVE + # the review→fix loop so a fix round can RESUME the SAME VM. The Microvm CR is + # therefore NOT deleted here — it is deleted at merge by df-merge-teardown, which + # triggers the controller's TerminateMicrovm. This onExit is a no-op guard: if the + # run FAILED before ever creating/suspending a VM, there's nothing to keep, but we + # still never delete on the normal path (the suspended VM is the whole point). + - name: teardown + script: + image: {{ .Values.microvm.stepImage | default "public.ecr.aws/aws-cli/aws-cli:latest" }} + command: [sh] + source: | + set -eu + MVM="mvm-{{`{{workflow.parameters.issue-number}}`}}" + echo "[teardown] lambda: KEEPING Microvm/${MVM} (stays SUSPENDED until merge → df-merge-teardown terminates it)." + exit 0 diff --git a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml index fa4c524e..0e569ff8 100644 --- a/gitops/addons/charts/dark-factory/templates/42-sensor.yaml +++ b/gitops/addons/charts/dark-factory/templates/42-sensor.yaml @@ -59,12 +59,16 @@ spec: template: serviceAccountName: dark-factory-sensor dependencies: - - name: issue-labeled + # TWO label dependencies — one per substrate — so each fires a DIFFERENT + # WorkflowTemplate (Argo Events trigger conditions can only reference dependency + # NAMES, not field values, so routing by label ⇒ separate dependencies + triggers): + # issue-labeled-kata → df-run (certified Kata pipeline) + # issue-labeled-lambda → df-run-lambda (Flow D MicroVM-native pipeline) + - name: issue-labeled-kata eventSourceName: dark-factory-github eventName: dark-factory filters: data: - # Only fire when the `dark-factory` label was ADDED to an issue. - path: headers.X-Github-Event type: string value: ["issues"] @@ -74,6 +78,20 @@ spec: - path: body.label.name type: string value: ["dark-factory"] + - name: issue-labeled-lambda + eventSourceName: dark-factory-github + eventName: dark-factory + filters: + data: + - path: headers.X-Github-Event + type: string + value: ["issues"] + - path: body.action + type: string + value: ["labeled"] + - path: body.label.name + type: string + value: ["darkfactory-lambda"] - name: pr-approved eventSourceName: dark-factory-github eventName: dark-factory @@ -129,10 +147,10 @@ spec: path: body.comment.body {{- end }} triggers: + # ---- KATA: dark-factory label → df-run (certified pipeline) ---- - template: name: submit-df-run - # With multiple dependencies, each trigger must name the one it fires on. - conditions: "issue-labeled" + conditions: "issue-labeled-kata" argoWorkflow: operation: submit source: @@ -146,8 +164,7 @@ spec: # delivery spawned a competing df-run that force-pushed its own # commit and split the commit statuses across SHAs. With a fixed # name, the 2nd+ submit collides (AlreadyExists) and is a harmless - # no-op — one issue = one in-flight run. The name is overwritten by - # the issue-id parameter below; this is a fallback if that is empty. + # no-op — one issue = one in-flight run. name: df-run-pending namespace: {{ .Values.argo.namespace }} spec: @@ -161,40 +178,101 @@ spec: - name: issue-title - name: issue-body - name: base-branch - # Map GitHub webhook fields → workflow parameters. + - name: trigger-label parameters: - # Deterministic workflow name = df-run- (the dedup key). - # sprig `int64` renders the large JSON number as a plain integer — - # without it Go templating emits scientific notation (4.88e+09), - # which is an invalid RFC-1123 metadata.name. - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataTemplate: "df-run-{{ `{{ .Input.body.issue.id | int64 }}` }}" dest: metadata.name - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.issue.id dest: spec.arguments.parameters.0.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.issue.number dest: spec.arguments.parameters.1.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.repository.full_name dest: spec.arguments.parameters.2.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.issue.title dest: spec.arguments.parameters.3.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.issue.body dest: spec.arguments.parameters.4.value - src: - dependencyName: issue-labeled + dependencyName: issue-labeled-kata dataKey: body.repository.default_branch dest: spec.arguments.parameters.5.value + - src: + dependencyName: issue-labeled-kata + dataKey: body.label.name + dest: spec.arguments.parameters.6.value + + # ---- LAMBDA: darkfactory-lambda label → df-run-lambda (Flow D MicroVM) ---- + - template: + name: submit-df-run-lambda + conditions: "issue-labeled-lambda" + argoWorkflow: + operation: submit + source: + resource: + apiVersion: argoproj.io/v1alpha1 + kind: Workflow + metadata: + name: df-run-lambda-pending + namespace: {{ .Values.argo.namespace }} + spec: + workflowTemplateRef: + name: df-run-lambda + arguments: + parameters: + - name: issue-id + - name: issue-number + - name: repo + - name: issue-title + - name: issue-body + - name: base-branch + - name: trigger-label + parameters: + # Dedup key = df-run-lambda- (stable across first run + fix + # rounds → one workflow name per issue, matching the mvm- CR). + - src: + dependencyName: issue-labeled-lambda + dataTemplate: "df-run-lambda-{{ `{{ .Input.body.issue.number }}` }}" + dest: metadata.name + - src: + dependencyName: issue-labeled-lambda + dataKey: body.issue.id + dest: spec.arguments.parameters.0.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.issue.number + dest: spec.arguments.parameters.1.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.repository.full_name + dest: spec.arguments.parameters.2.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.issue.title + dest: spec.arguments.parameters.3.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.issue.body + dest: spec.arguments.parameters.4.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.repository.default_branch + dest: spec.arguments.parameters.5.value + - src: + dependencyName: issue-labeled-lambda + dataKey: body.label.name + dest: spec.arguments.parameters.6.value # ---- PR review approved → df-merge-teardown (the ONLY merge path) ---- - template: diff --git a/gitops/addons/charts/dark-factory/values.yaml b/gitops/addons/charts/dark-factory/values.yaml index 26f7e78c..2bd029ac 100644 --- a/gitops/addons/charts/dark-factory/values.yaml +++ b/gitops/addons/charts/dark-factory/values.yaml @@ -9,10 +9,21 @@ namespace: agent-sandbox-system # Argo Workflows install (already on the hub in the `argo` namespace). argo: namespace: argo + # How long a COMPLETED workflow (success or failure) is retained before Argo's + # controller garbage-collects it. There is NO workflow archive/DB persistence on + # this cluster (workflow-controller-configmap has no `persistence:` block), so once + # TTL fires the run is gone for good — not archived. 7 days keeps demo/debug history + # visible in the Argo UI across a work week. Was 3600 (1h), which silently dropped + # runs an hour after they finished. + workflowTtlSecondsAfterCompletion: 604800 warmPool: - # The Flow A SandboxWarmPool the factory claims from. + # The Flow A SandboxWarmPool the factory claims from (Kata substrate, default). name: coder-warmpool + # Flow D: the Lambda-MicroVM SandboxWarmPool the factory claims from when the + # darkfactory-lambda label fires (coder runs in a MicroVM). Must match + # agent-sandbox microvm.warmPool.name. + lambdaName: coder-warmpool-microvm # Namespace of the pool / where claims + coder pods are created. namespace: agent-sandbox-system @@ -246,6 +257,35 @@ iterate: enabled: true maxIterations: 3 +# ── Flow D — Lambda MicroVM substrate (df-run-lambda ONLY) ─────────────────── +# The Kata df-run template has ZERO MicroVM logic (it's byte-identical to the certified +# Kata pipeline). Flow D runs in a SEPARATE WorkflowTemplate, df-run-lambda, which +# provisions the Lambda MicroVM directly (Microvm CR + /run), suspends it during review +# (scale-to-zero), resumes the SAME VM on a fix round, and terminates it at merge. These +# values feed that template; they mirror the agent-sandbox-lambda chart's microvm.* keys. +microvm: + enabled: true + region: us-west-2 + namespace: agent-sandbox-system # where the Microvm CR + platform MicrovmSandbox live + # aws-cli v2 (has the lambda-microvms verbs); kubectl is fetched at step start. + stepImage: public.ecr.aws/aws-cli/aws-cli:latest + image: + name: coder # the platform MicrovmSandbox name (image handoff) + defaults: + maxIdleDurationSeconds: 1800 # RUNNING-idle backstop (we drive /run immediately) + # Lambda caps SUSPEND at 8h (28800s). Use the max so the VM survives the full + # review→fix window; df-merge-teardown terminates it explicitly at merge. + suspendedDurationSeconds: 28800 + # Pod Identity: the df-run-lambda steps run as the dark-factory-workflow SA (argo ns) + # and call aws lambda-microvms (get/suspend/resume/terminate/create-auth-token). Bind + # that SA to the SAME lambda-microvms role the bridge/lifecycle use (least-privilege; + # additive; no existing role policy changes). ACK eks PodIdentityAssociation. + podIdentity: + clusterName: hub + accountId: "940019131157" + # role: -ack-lambdamicrovms-controller + workflowServiceAccount: dark-factory-workflow + # ── Language / stack support ───────────────────────────────────────────────── # There is NO per-language profile config here — deliberately. Language support is # decoupled two ways: diff --git a/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml new file mode 100644 index 00000000..0abcde06 --- /dev/null +++ b/gitops/addons/clusters/hub/addons/agent-sandbox-lambda/values.yaml @@ -0,0 +1,48 @@ +# Hub (openclaw-eks / control-plane) overlay for the agent-sandbox-lambda chart +# (Flow D — Lambda MicroVM substrate). +# +# DORMANT (microvm.enabled=false) until the self-managed ack-lambdamicrovms +# controller is synced (gitops/addons/bootstrap/default/addons.yaml) and the arm64 +# coder artifact is published to S3. Managed KRO + Managed ACK capabilities are +# ACTIVE on the hub. The ARNs below are public identifiers (not secrets). +microvm: + enabled: true + region: us-west-2 + # MUST be kro.run — EKS Managed KRO only watches the kro.run group (see chart values.yaml). + apiGroup: kro.run + # aws-cli v2 (has the pre-GA lambda-microvms service; alpine/k8s's aws-cli 1.34 does NOT, + # so RunMicrovm never fired). bridge.sh/reconcile.sh fetch kubectl at start. See chart values. + bridgeImage: public.ecr.aws/aws-cli/aws-cli:latest + # Hub account (for the ACK PodIdentityAssociation role ARN). + accountId: "940019131157" + podIdentity: + clusterName: hub + controllerNamespace: ack-system + controllerServiceAccount: ack-lambdamicrovms-controller + # AWS-published base MicroVM image (ARM_64 — the only arch Lambda MicroVM supports). + baseImageARN: "arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1" + # Lambda MicroVM codeArtifact.uri is S3-ONLY (a zip with the coder app + a Dockerfile); + # it is NOT an ECR image reference. The Dockerfile inside MAY pull the arm64 + # dark-factory-coder from ECR as a base layer (the build role keeps ecr:Get*). + # MUST live in the bucket the RGD creates: ${image.name}-microvm-artifacts (here + # `coder-microvm-artifacts`). Previously pointed at a hand-named bucket that the RGD + # never provisioned, so the image build hit NoSuchBucket/CREATE_FAILED. Publish the + # artifact zip to this exact bucket/key. + # r7: hook-server /run guard is now keyed on a per-invocation run-id (issue+note + # hash) instead of a one-shot boolean, so a RESUMED VM (fix round) accepts a fresh + # /run and re-runs the coder — the resume path was previously a no-op because the + # snapshot froze coderStarted=true. Also truncates /tmp/coder.log per run so the + # bridge's "PR pushed" grep doesn't match the previous round's line. + # r6: fix-round iterate note forwarded (bridge payload -> hook-server -> coder env). + codeArtifactUri: "s3://coder-microvm-artifacts/coder-v0.2.5-arm64-r7.zip" + image: + enabled: true + name: coder + defaults: + maxIdleDurationSeconds: 900 + suspendedDurationSeconds: 300 + lifecycle: + intervalSeconds: 15 + warmPool: + name: coder-warmpool-microvm + targetIdle: 1