Flow D — Lambda MicroVM Agent Sandbox (separate df-run-lambda; MicroVM-native suspend/resume) - #41
Merged
elamaran11 merged 68 commits intoAug 4, 2026
Conversation
Flow D is a second Flow-A substrate: AWS Lambda MicroVM via the ACK lambdamicrovms controller, composed by a single KRO ResourceGraphDefinition. Mirrors the flow-a-sandbox-capability.md style. Covers substrate architecture (KRO RGD over ACK primitives), the platform/app ownership split inside one RGD, the RuntimeClass-marked bridge shim (claim -> pod -> MicroVM), and the GA migration path (self-managed controller -> Managed ACK adopts it, RGD unchanged). Flow C is reserved for other work; this substrate is Flow D.
…ctory README New §4.5 'Flow D — Lambda MicroVM substrate (alternative to Flow A)': KRO RGD over ACK primitives (Managed KRO + Managed ACK for GA iam/s3 + self-managed lambdamicrovms pre-GA controller), the platform-owned/app-owned split encoded in the MicrovmImage/Microvm CRDs, the RuntimeClass-marked bridge shim, and disabled-by-default GitOps delivery. Adds the TOC entry and a 'second substrate' note to the Two-flows-at-a-glance section. Reuses dark-factory-coder image.
…isabled) New addons.yaml entry installing the pre-GA Lambda MicroVM ACK controller (oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart v0.1.1) that reconciles the MicrovmImage + Microvm CRDs. Self-managed because Managed ACK only bundles GA controllers; coexists with Managed ACK (GA iam/s3) since CRD groups differ. enabled:false (Flow D dormant), hub-only via alwaysSelector, sync-wave 0 so CRDs+controller precede the MicrovmSandbox RGD (wave 2). Pod-identity auth (no IRSA annotation), cluster install scope.
templates/50-rgd-microvm-sandbox.yaml: one kro.run/v1alpha1 ResourceGraphDefinition that expands a single MicrovmSandbox CR into S3 Bucket + build/exec IAM Roles (GA -> Managed ACK) + MicrovmImage + Microvm (pre-GA -> self-managed ACK). The platform/app ownership split is encoded in the schema (spec.image.* platform-owned, spec.run.* app-owned); status surfaces microvmID/state for the bridge to mirror. RGD is controller-install-agnostic (unchanged when lambdamicrovms goes GA). values.yaml: new microvm: block (enabled:false, region, apiGroup, bridgeImage, idle/connector defaults; cluster-specific ARNs left blank for the overlay). Verified: helm template renders 0 RGD when disabled, 1 valid RGD (5 resources: bucket/buildRole/execRole/image/microvm) when enabled.
…eClass shim) templates/51-sandboxtemplate-microvm.yaml: the shim that keeps the Agent-Sandbox UX while the coder runs in a remote Lambda MicroVM. A SandboxTemplate (<templateName>-microvm) whose pod is a thin BRIDGE on a normal Auto-Mode node (no kata runtimeClass/nodeSelector) — it idles until a claim injects DF_ISSUE_NUMBER (same as Flow A), applies a MicrovmSandbox (KRO) CR, waits for Microvm RUNNING, mirrors lifecycle (pod Running <-> Microvm RUNNING), and deletes the CR on exit (-> TerminateMicrovm). Ships its ServiceAccount + least-priv Role/RoleBinding (microvmsandboxes only) + the bridge.sh ConfigMap. Both substrates coexist; a claim picks one by which template it references. Verified: helm template renders 5 objects when enabled, 0 when disabled; bridge.sh passes sh -n.
Hub per-cluster overlay gets a microvm: block (enabled:false) alongside the kata nodepool block. Cluster/account-specific ARNs (baseImageARN, codeArtifactUri, ingress/egress connectors) left blank with guidance to fill before enabling. Flow D stays fully dormant on the hub until the Managed KRO + Managed ACK capabilities and the self-managed lambdamicrovms controller are in place. Verified full-chart render with the overlay: 0 Flow D objects when dormant (existing Kata objects intact), clean inventory when enabled (1 RGD + 2 SandboxTemplates + bridge SA/Role/RoleBinding/ConfigMap), valid YAML in both modes.
…into Flow D The ACK GA-services-list reference and the pre-GA-vs-GA / self-managed rationale belong with the consumer (Flow D), not the platform capability-enablement doc. Add the authoritative ACK community services (GA list) link backing the 'GA upstream' claim, and clarify the ownership split: the Managed KRO + Managed ACK CAPABILITIES are enabled in appmod-blueprints (EKS Capability toggle, see its docs/EKS-Capabilities-KRO-ACK-Setup.md); THIS repo owns the self-managed lambdamicrovms controller + the KRO MicrovmSandbox RGD + the sandbox shim.
Align the MicrovmSandbox RGD with the actual lambda-microvms API (verified against the live service + AWS docs in us-west-2): - IAM trust principal: lambda.amazonaws.com (+ sts:TagSession) — NOT the guessed lambdamicrovms.amazonaws.com. buildRole gets inline ECR-read + S3-read + logs (Lambda pulls the coder image from private ECR at build). - Drop network connectors entirely: Lambda MicroVMs have PUBLIC egress by default, which is all the coder needs (git/gh/registry). Ingress/VPC-egress connectors are optional add-ons (the latter created via a different service, aws lambda-core) — omitted from v1. Removes ingress/egressConnectorARN from schema + both resources. - codeArtifactUri: document it accepts an ECR image URI or S3 path (Flow D uses the arm64 dark-factory-coder ECR image directly). - base image ARN format corrected to arn:aws:lambda:<region>:aws:microvm-image:al2023-1 (ARM_64-only) in values guidance. Verified: helm template renders 5 resources, 0 connector refs, correct principal.
Set the hub Flow D overlay to the confirmed live values: - baseImageARN: arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1 (verified available; ARM_64-only). - codeArtifactUri: dark-factory-coder:v0.2.3-arm64 ECR URI (accepts ECR directly; arm64 tag to be built — MicroVM is ARM_64-only, existing amd64 tags untouched). - Drop the connector fields (MicroVMs default to public egress). Still enabled:false (dormant) until the ack-lambdamicrovms controller is synced.
…ifecycle)
Implements the REQUIRED declarative suspend/resume: Sandbox.spec.operatingMode has
{Running,Suspended} but the ACK Microvm CR has no suspend field (suspend/resume are
imperative SDK ops the controller doesn't reconcile). Close the gap with a tiny
always-on reconcile loop (templates/52) — pure shim, no virtual-kubelet, no new image:
- microvm-lifecycle Deployment (alpine/k8s, ConfigMap reconcile.sh, matches
pool-manager pattern): watches Sandboxes labeled substrate=lambda-microvm; on
operatingMode Running->Suspended calls suspend-microvm, Suspended->Running calls
resume-microvm, by microvmID resolved from the MicrovmSandbox status. Idempotent
via a last-mode annotation. + SA/Role/RoleBinding (pod-identity auth).
- Bridge (51) made suspend-aware: labels its MicrovmSandbox agent-sandbox.io/sandbox
for correlation; preStop checks operatingMode and, if Suspended, writes a marker so
cleanup() KEEPS the MicrovmSandbox (VM survives suspend); real teardown only on
claim end. Create is idempotent (resume-or-create). +sandboxes get RBAC.
- values: microvm.lifecycle.intervalSeconds (15s).
Robust vs preStop-only: survives pod/node loss; resume needs no live pod. Verified:
0 objects disabled; 10 when enabled; reconcile.sh + bridge.sh pass sh -n.
Add §D.3a (diagram doc) + a Suspend/resume subsection (README §4.5) describing the microvm-lifecycle reconcile loop that translates Sandbox.operatingMode -> Lambda MicroVM suspend/resume-microvm, why it's needed (ACK Microvm has no suspend field), and why a reconcile loop beats preStop hooks (robust to pod/node loss). Matches the templates/52 implementation.
Per reviewer request: the coder now writes a concise, reviewer-facing description of WHAT it changed (and why) to artifacts/description.md as part of its run, and the PR body renders it as a '### 📝 Changes' section ABOVE the existing verification block. Falls back to a neutral line if absent so the body is never empty. - claude path: description instruction appended to the prompt. - kiro path: same instruction appended to SPEC.md (kiro reads --spec). - PR body: new Changes section placed before the <!-- dark-factory:status --> marker, so status.js (which rewrites from the marker down) preserves it on every update. node --check passes.
Two fixes for the 'security review waits ~15 min while the PR already shows a result' problem: 1. Early-exit: the code-review job's status field flips to COMPLETED long AFTER the AWS Security Agent App bot has already posted its findings on the PR. So in addition to polling status, each iteration now probes list-findings; the moment it returns a well-formed result (findingsSummaries present) we proceed — no waiting for the status flip. Reuses those findings for the report (no re-fetch). 2. Graceful timeout: this step is advisory (the App bot posts the authoritative result regardless), so exceeding the poll timeout now posts a neutral 'pending' status + a pointer to the bot comment, instead of a misleading red 'error'. This also fixes the stale sticky board: the sticky-status step depends on security-agent (runs once, after it), so a fast security step means the board is rewritten with accurate rows within seconds instead of ~15 min later. sh -n passes.
… substrate One pipeline, substrate chosen by label (per design decision): - sensor: accept BOTH dark-factory (Kata) + darkfactory-lambda (MicroVM) labels; pass the firing label to df-run as the trigger-label param (index 6). - df-run: new trigger-label param (default dark-factory); claim-sandbox takes a warm-pool input; the DAG computes it via an Argo expression (darkfactory-lambda -> coder-warmpool-microvm, else coder-warmpool). Everything downstream (coder, holdout, devops, security, PR, merge, teardown) is identical. - agent-sandbox: add the Lambda-MicroVM SandboxWarmPool (claim.warmPoolRef is required, so Flow D needs its own pool) referencing the -microvm SandboxTemplate; microvm.warmPool values (name, targetIdle). dark-factory: warmPool.lambdaName. Verified: df-run + sensor render to valid YAML; warm-pool expression resolves correctly; lambda warmpool renders only when microvm.enabled (0 when disabled).
Flow D visible suspend/resume via Sandbox.operatingMode (per design decision):
- New microvm-suspend DAG step (gated microvmSuspend.enabled + lambda substrate +
PR exists): after drive-coder, flips Sandbox.spec.operatingMode=Suspended so the
agent-sandbox microvm-lifecycle controller calls suspend-microvm — compute freed
while the gates run (coder is idle during review). No-op for Kata.
- microvm-set-mode script template: patches operatingMode, best-effort verifies the
MicrovmSandbox microvmID (advisory). RBAC: workflow SA gets sandboxes 'patch'.
- Resume: happens naturally — df-iterate (PR comment) needs the VM, and the MicroVM
idlePolicy.autoResumeEnabled resumes on next request; the lifecycle controller
also resumes on operatingMode=Running.
- values: microvmSuspend {enabled, image}.
Verified: full chart renders valid YAML; suspend step present when enabled, absent
when disabled.
elamaran11
changed the title
Applied field-level corrections from the lambdamicrovms-controller reference
(mmeckes/lambdamicrovms-controller, 06-kro example + CRD source):
- codeArtifact.uri is S3-ONLY (s3://bucket/key of a zip w/ Dockerfile), NOT an
ECR image URI. Fixed the misleading 'ECR or S3' comments in the RGD schema +
values (the Dockerfile inside MAY pull ECR base layers — build role keeps ecr:Get*).
- image.readyWhen: gate on status.state == CREATED||UPDATED so the Microvm never
launches from a half-built/failed image (KRO holds the instance until ready).
- MicrovmImage.cpuConfigurations: [{architecture: ARM_64}] — the sole supported
arch; make it explicit rather than relying on a default.
- MicrovmImage.logging.cloudWatch.logGroup: /aws/lambda/microvms/<name>-image so
CREATE_FAILED build output is retrievable (aws logs tail ...).
- status.microvmState now reads Microvm.status.state (PENDING/RUNNING/SUSPENDED/
...) not conditions[0].status; added imageState passthrough.
- build role S3 policy scoped to the actual bucket via CEL ref (bucket.spec.name)
+ ListBucket, and logs scoped to the microvms log-group prefix.
Design unchanged: Sandbox CRDs → ACK Lambda MicroVM for create/delete + the
microvm-lifecycle shim for suspend/resume (imperative ops, not controller-managed
— confirmed by the reference). Still gated microvm.enabled=false; render-verified.
…bda chart
Clean folder separation — Flow D (Lambda MicroVM) is now a standalone chart instead
of files dumped into the Kata agent-sandbox chart:
gitops/addons/charts/agent-sandbox-lambda/
Chart.yaml, values.yaml, templates/_helpers.tpl
templates/image/10-rgd-and-image.yaml — KRO RGD (image-only) + the ONE platform
MicrovmSandbox instance (built ONCE)
templates/shim/20-bridge-sandboxtemplate.yaml — bridge: reads image handoff,
RunMicrovm (SDK), TerminateMicrovm
templates/shim/30-microvm-lifecycle.yaml — suspend/resume controller (SDK)
- Moved the 3 microvm templates out of agent-sandbox (Kata chart) → now Kata-only.
- Moved the microvm: values block out of agent-sandbox/values.yaml + the hub overlay
into agent-sandbox-lambda/{values.yaml, clusters/hub overlay}. Corrected stale docs
(codeArtifact is S3-only, not ECR).
- New gated addon entry agent-sandbox-lambda (hub-only, sync-wave 2); dormant until
microvm.enabled=true.
ARCHITECTURE (per design): KRO builds the platform image ONCE; the shim runs/suspends/
resumes/terminates the per-session VM imperatively (ACK doesn't reconcile those). RGD no
longer contains a Microvm resource. Renders: Kata chart microvm-free; Lambda chart 0
disabled / 13 enabled. Flow D only — dark-factory (PR #32) untouched.
… across review loop) Implements the required Lambda coder lifecycle: coder codes → SUSPEND; findings + fix → RESUME the SAME VM (keeps context); terminate ONLY at final exit (merge). - df-run onExit teardown is now SUBSTRATE-AWARE (script, not a blind resource delete): Kata deletes the claim (frees pool); LAMBDA KEEPS the claim + suspended VM so the same VM survives the whole review→fix→re-review loop (df-iterate resumes it). Without this, df-run's exit deleted the claim → orphaned/terminated the suspended VM. - df-merge-teardown adds a microvm-terminate step (gated microvmSuspend.enabled) that resolves the per-session VM id from the Sandbox annotation and TerminateMicrovm on merge — the ONLY place the Lambda VM is destroyed. Ordered before the claim delete so the bridge's suspend-marker can't keep a leaked VM. - microvm-set-mode verify reads the VM id from the Sandbox annotation (post image-only RGD refactor — no per-session MicrovmSandbox status anymore). Flow-D-branch changes to the dark-factory template (base of PR #41); gated by microvmSuspend.enabled so Flow B (Kata, PR #32) is unaffected. Renders clean both ways.
…d/resume/terminate lifecycle Aligns the Flow D docs with the refactor: KRO builds the image ONCE (RGD has no Microvm); the shim runs the per-session VM imperatively (RunMicrovm/suspend/resume/ TerminateMicrovm); the coder VM PERSISTS suspended across the review→fix loop and is terminated only at merge. Updated the composition table, platform/app split, shim description, the suspend/resume/terminate lifecycle section, and the delivery section (now the standalone agent-sandbox-lambda chart with image/ + shim/ subfolders). NOTE: diagrams/flow-d-microvm-sandbox.md ASCII still shows the old per-claim model — follow-up.
Per design: everything via Managed ACK + KRO, self-managed ACK only for the pre-GA lambdamicrovms controller — NO Crossplane, NO Terraform, NO CLI. - Controller bootstrap IAM is now an ACK iam.services.k8s.aws Role + ACK eks.services.k8s.aws PodIdentityAssociation (templates/shim/00-controller-pod-identity.yaml) — same Managed-ACK mechanism as the RGD's build/exec roles. Role trusts pods.eks.amazonaws.com; inline policy = lambda:*Microvm* + iam:PassRole (lambda). Bridge + lifecycle SAs get their own PodIdentityAssociations reusing the role. All 4 objects server-dry-run VERIFIED against the live ACK CRDs on the hub. - Enabled ack-lambdamicrovms controller addon (enabled:true) + microvm.enabled=true in the hub overlay (accountId + podIdentity set). Downstream (S3, build/exec roles, MicrovmImage) stays in the KRO RGD via ACK. Flow D only (PR #41); dark-factory (PR #32) untouched.
…ambda-microvm-sandbox # Conflicts: # examples/dark-factory/coder/entrypoint.js # gitops/addons/charts/dark-factory/scripts/security-agent.sh # gitops/addons/charts/dark-factory/templates/20-workflowtemplate-df-run.yaml
…rking structure) After #45 was reverted (PR back to old flat-addons + working ACK structure), reset flow-d to the pre-#45-merge state and re-merged the CURRENT #32 (47b8908, contains all the colleague's still-valid Flow B work minus the reverted hub-sandbox commit). Then re-applied the Flow D df-run orchestration (dropped when I took their df-run in the merge): substrate-branched claim (lambda warm pool on darkfactory-lambda), microvm-suspend DAG step + microvm-set-mode template (suspend after coder pushes), substrate-aware teardown (Kata deletes claim; Lambda keeps suspended VM until merge). df-merge-teardown already has microvm-terminate. Renders clean (default + microvmSuspend).
elamaran11
force-pushed
the
flow-d-lambda-microvm-sandbox
branch
from
August 1, 2026 23:24
fabe855 to
ba514ac
Compare
added 5 commits
August 2, 2026 21:02
The MicrovmSandbox instance failed dry-run ('CRD not found') and blocked the whole
app sync (nothing applied, incl the RGD that GENERATES that CRD — chicken-and-egg
within the app). RGD → sync-wave -1 (KRO generates the CRD first); instance → wave 1 +
SkipDryRunOnMissingResource so the first pass doesn't block. ArgoCD retries/selfHeal
converge.
The coder Kata VM resolves via public DNS (8.8.8.8/1.1.1.1) since a Kata
guest can't use in-cluster CoreDNS and EKS Auto Mode runs no kube-dns
Service. coder-sandbox-egress only permitted :53 to namespaceSelector{}
(in-cluster), so name resolution silently failed — the coder crashed with
EAI_AGAIN api.github.com even though :443 egress worked, and no PR was ever
pushed. Add a :53 egress rule scoped to the resolver /32s (networkPolicy.
dnsResolvers), tighter than the existing :443 public allow.
ttlStrategy.secondsAfterCompletion was 3600, so runs were GC'd an hour after finishing — and there's no workflow DB archive on this cluster, so they were lost for good (Friday's runs gone). Bump to 604800 (7d) via new argo.workflowTtlSecondsAfterCompletion value across df-run/df-iterate/ df-merge-teardown so demo+debug history survives a work week.
Two fixes so the MicrovmSandbox RGD actually reconciles on the hub:
1. apiGroup kro.run (was sandbox.agents.x-k8s.io): EKS Managed KRO's
controller only watches the kro.run group — an RGD on any other group
stays state=Inactive ('cache sync timeout ... microvmsandboxes').
Verified by group-probe on the hub. Generated CRD is microvmsandboxes.
kro.run; kind (MicrovmSandbox) unchanged.
2. New 40-kro-graph-rbac.yaml: KRO runs as the cluster's KRO capability
role (EKS access entry, session KRO). AmazonEKSKROPolicy grants kro.run
but NOT CRUD on the ACK children the graph creates, so reconcile hit
'forbidden: cannot get buckets'. Grant a ClusterRole scoped to exactly
the 3 child groups (s3 buckets, iam roles, lambdamicrovms images/vms)
bound to that identity (microvm.kroCapability values).
The Kata substrate showed microvm-suspend as phase=Error (type=Skipped):
Argo substitutes the trigger-label value inline, so the unquoted
'dark-factory == darkfactory-lambda' parsed as arithmetic on bare
identifiers and errored ('Failed to evaluate when expression'). Harmless
(the step is correctly skipped on Kata and the run still completes) but it
surfaces a spurious Error node. Quote both operands so it evaluates as a
clean string comparison and skips silently on non-lambda runs.
added 26 commits
August 3, 2026 19:10
The multi-line python at column 0 broke out of the bridge.sh: | block scalar → ArgoCD ComparisonError 'could not find expected :'. Collapse to one line so all script content stays indented under the block scalar.
Bedrock-direct async design built + deployed: hook-server runs in the VM (verified via CloudWatch 'listening on :8080'), bridge passes runHookPayload, VM launches RUNNING — but the /run hook never fires so the coder doesn't start. Confirmed cause: runHookPayload is a SecretKeyReference delivered via the declarative Microvm CR (controller-driven), NOT the imperative run-microvm --run-hook-payload CLI the bridge uses. Correct path documented: per-session Microvm CR + Secret. hook-server/artifact/image/IAM/Bedrock are done + verified.
Switch the bridge from the imperative run-microvm CLI (which never fires /run) to the
declarative path the runHookPayload SecretKeyReference actually requires: write a payload
Secret + create a Microvm CR (imageIdentifier, executionRoleARN, runHookPayload{name,key},
idlePolicy); the self-managed controller reconciles it and delivers the payload to the /run
hook → hook-server background-spawns the Bedrock-direct coder. Teardown deletes the CR
(controller terminates the VM); suspend keeps it. Manifests built as JSON via python3 (no
heredoc — a column-0 EOF breaks the bridge.sh block scalar). Bridge RBAC += microvms +
secrets CRUD.
…ilent Declarative Microvm CR path works (CR RUNNING, microvmID populated, clean CR-delete teardown, 0 orphans) but the /run hook produces no runtime coder output in CloudWatch. Ruled out: payload mechanism, image hooks (v2.0), IAM, bridge crash, YAML. Open frontier: whether the service invokes /run against the in-VM hook-server + why no logs.
Probed the live VM: GET / -> ok, POST /run -> coder started. Whole chain functional (issue->workflow->bridge->Microvm CR->VM->hook-server->/run->coder-spawn->clean teardown). Two isolated gaps: (1) service doesn't auto-invoke /run on launch (had to POST manually); (2) runtime logs don't reach CloudWatch (build logs only) — blinded prior runs.
…connector Close the 2 E2E gaps: (1) the service's internal /run auto-fire wasn't reliably starting the coder — the bridge now waits for RUNNING+endpoint, mints an auth token, and POSTs the payload to /run on the VM endpoint (deterministic, matches the reference run_session; verified by probe that this starts the coder). Needs ingressNetworkConnectors:ALL_INGRESS on the Microvm to reach the endpoint. (2) runtime logs — add Microvm.spec.logging.cloudWatch (logStream runtime-<mvm>) so the coder's stdout is visible (image logging was build-only).
The bridge mints an auth token to POST /run on the VM endpoint, but the controller/bridge role lacked lambda:CreateMicrovmAuthToken (+ShellAuthToken) — token mint failed AccessDenied, so /run was never driven. Add both verbs to the controller inline policy (reused by the bridge via Pod Identity).
Runtime CloudWatch routing is unreliable on this runtime, so make the coder run OBSERVABLE
directly: hook-server captures the coder's stdout/stderr to /tmp/coder.log + tracks state
(running/done/exited:N/spawn-error), exposes GET /status {coderState, log tail}. The bridge
polls /status in its hold loop and echoes it — so the coder run is visible in the bridge pod
logs (kubectl logs) even without CloudWatch.
Overwriting the same S3 key doesn't change the URI, so the controller never rebuilds (stayed v2.0 without /status). New key -r2 → URI change → rebuild with the /status observability + auth-token fixes.
The /status route + log-capture correlated with a MicrovmImage build hung 2h+ on the ready hook (pre-GA controller). Revert hook-server to the minimal known-good shape that built v2.0 cleanly: trivial synchronous /run that detached-spawns the coder, no /status. Point codeArtifactUri at a fresh key (r3) to force a clean rebuild.
…ility) No CloudWatch runtime routing + no shell, so /run captures the coder's stdout/stderr to /tmp/coder.log and /logs returns it (read over the HTTP token). This is how we finally SEE why the coder isn't producing a PR. Artifact r4.
…orkspace) ROOT CAUSE (finally seen via /logs): the coder crashed immediately — 'EACCES: permission denied, mkdir /workspace/artifacts' at entrypoint.js:398 — because the MicroVM rootfs is read-only with no /workspace volume (unlike Kata). Point WORKSPACE at the writable tmpfs. Artifact r5.
The fix round (df-iterate → df-run) had no trigger-label, so it always went to Kata even for a Lambda PR. Detect the substrate from the ORIGINATING ISSUE's label (the coder doesn't copy it onto the PR), and pass trigger-label to the resubmitted df-run so it claims the right warm pool. Kata unchanged (defaults to dark-factory).
One df-run template serves both substrates, so a rendered microvm-suspend step could only be runtime-Skipped on Kata (visible, confusing). Remove it (microvmSuspend.enabled=false → zero MicroVM steps in the Kata DAG). The Flow D bridge now suspends its OWN MicroVM after the coder pushes the PR (lambda-only by construction). Also fixes bridge hold-loop to poll /logs (was /status, which the minimal image dropped).
Side-by-side from a parallel dual-substrate run (#117 Kata / #118 Lambda): time-to-PR, per-step timing, cold-start vs warm-claim, DAG comparison, log locations, step-by-step lifecycle, the 10 Lambda-specific gotchas, and when to choose which. Plus a Mermaid diagrams companion (shared pipeline, each substrate, suspend/resume, e2e loop).
The bridge runs python3 + curl + fetched kubectl + aws-cli v2 in a poll loop; 128Mi
OOMKilled it during the fix round, so it died before the coder's new commit landed and
await-coder spun forever ('head still at start sha'). Give it 256Mi req / 1Gi limit.
A Lambda fix round (df-iterate) re-ran the coder with NO change request: - df-run injects DF_ITERATE_NOTE_B64 into the bridge 'coder' container (Kata parity), but the bridge's runHookPayload dropped it, and hook-server never mapped it into the MicroVM coder's env. - Result: the coder saw the PR already open and reported 'done' on the old sha with zero commits, so findings were never fixed. Fixes: bridge folds DF_ITERATE_NOTE_B64/_NOTE into runHookPayload; hook-server maps them back to the coder env; bump codeArtifactUri r5->r6 to rebuild the MicrovmImage with the fixed hook-server + current entrypoint.
The bridge names the Microvm CR mvm-<issue> (stable per session). On a fix round the prior VM is still around, SUSPENDED after the first PR. kubectl apply on the same name RESUMES that snapshot, which restores hook-server's one-shot coderStarted=true guard in memory, so the second /run is ignored and the coder never re-runs — the fix round reports 'done' on the OLD sha with no commits, even with the iterate note now forwarded. Delete the stale CR + payload Secret and wait for TerminateMicrovm before recreating, so the fix round gets a clean hook-server that accepts /run. This matches the documented 'fresh Microvm CR per fix round' semantic. No image rebuild needed (r6 already forwards the note).
Make Sandbox.operatingMode the single source of truth for MicroVM scale-to-zero (the Flow D highlight), and make resume actually re-run the coder: - idlePolicy.autoResumeEnabled: true -> FALSE. With autoResume on, ANY hit to the VM endpoint auto-resumed it, and the bridge polled /logs every 20s FOREVER, so the VM bounced back to RUNNING seconds after every suspend (console never showed it suspended). Now a suspended VM stays suspended. - Suspend via CRD, not imperatively: after the coder pushes its PR the bridge sets Sandbox.operatingMode=Suspended and STOPS touching the endpoint; the microvm-lifecycle controller (template 30) reconciles that to suspend-microvm. - Resume-on-fix-round: instead of terminating + rebuilding, the bridge flips operatingMode=Running so the controller resume-microvm's the SAME suspended VM (memory+disk preserved) — true warm resume, the whole Flow D value prop. - hook-server /run guard keyed on a per-invocation run-id (issue+note hash) instead of a one-shot boolean frozen in the snapshot, so the resumed VM accepts the fix round's /run and re-runs the coder; also truncates /tmp/coder.log per run so the bridge's 'PR pushed' grep can't match the prior round's line. Bump artifact r6->r7. Verified in-cluster via mini-tests (bare Microvm CR + test Sandbox): create->RUNNING, operatingMode=Suspended->stays SUSPENDED, operatingMode=Running->RUNNING, delete-> TERMINATED; hook-server run-id logic unit-tested (dup ignored, fix-round re-run accepted).
…ence teardown guard Two bugs made operatingMode=Suspended TERMINATE the VM instead of suspending it: 1) microvm-lifecycle controller selected Sandboxes by label agent-sandbox.io/substrate=lambda-microvm — but the operator does NOT propagate SandboxTemplate labels onto the Sandbox object, so the selector matched NOTHING and the controller never reconciled any real session (Sandbox went SandboxSuspended, VM never suspended). Select by the microvm-id ANNOTATION the bridge writes instead — only lambda sessions have it. 2) On operatingMode=Suspended the operator DELETES THE POD (keeps the Sandbox alive), firing the bridge cleanup trap. The old trap deleted the Microvm CR unless a racy preStop /tmp marker was set → VM terminated. cleanup now keeps the CR whenever the owning Sandbox STILL EXISTS (suspend), and deletes it only when the Sandbox is GONE (real teardown). Removed the preStop marker. Bridge now exits right after setting operatingMode=Suspended (pod is killed anyway) and stops polling /logs (each hit would auto-resume the VM).
Root cause of the suspend/resume flap: the claim/Sandbox was named by issue-id, which DIVERGES between the sensor's first run (github event id, e.g. 5064265549) and df-iterate's fix round (issue number, e.g. 133). So a fix round created a SECOND Sandbox (df-issue-133) while the first run's (df-issue-5064265549) still existed — BOTH annotated with the same microvm-id. The lifecycle controller then saw two Sandboxes with OPPOSITE operatingMode intents for one VM and flapped suspend<->resume until the VM died (Internal service error). Name the claim/Sandbox by issue-NUMBER everywhere (claim, mutex, teardown) — stable across first run + all fix rounds, and already the key for the Microvm CR (mvm-<issue-number>). Exactly one Sandbox per VM. Also: suspendedDurationSeconds 300 -> 86400 was wrong (Lambda max suspend is 8h); set within cap next commit. maxIdle 900 -> 1800.
… Kata
Kata and Lambda no longer share one df-run template. Per the constraint 'don't
touch certified Kata':
- df-run (Kata): reverted BYTE-IDENTICAL to the certified
dark-factory-autonomous-agent-coding-pattern template. Zero MicroVM logic.
- df-run-lambda (NEW, 23-...): MicroVM-native pipeline, NO SandboxClaim/bridge/
warm-pool. DAG: provision-microvm → drive-coder → suspend-microvm → [holdout ∥
devops ∥ security ∥ deploy-test] → status; onExit KEEPS the suspended VM.
provision-microvm creates the Microvm CR + runHookPayload Secret directly (note
folded into payload), waits RUNNING+endpoint, mints token, POSTs /run. On a fix
round it RESUMES the same suspended VM (warm resume) or recreates if terminated.
Encodes every Flow D learning: autoResume=false + no post-/run endpoint polling
(suspend sticks), suspendedDuration=8h (survives review→fix), HTTP_INGRESS,
stable CR name mvm-<issue-number> (one VM per issue), image/exec-role from the
platform MicrovmSandbox status.
- df-merge-teardown: microvm-terminate now deletes the Microvm CR by stable name
(was Sandbox-annotation lookup); gated on microvm.enabled.
- values: replaced microvmSuspend{} with a microvm{} block (region/namespace/
stepImage/image.name/defaults). suspendedDurationSeconds=28800 (8h cap).
Routing (sensor + iterate.js → df-run-lambda) follows next.
…/RBAC
- Sensor: split issue-labeled into issue-labeled-kata (→ df-run) and
issue-labeled-lambda (→ df-run-lambda) with two triggers. Kata path unchanged.
- iterate.js: fix rounds submit df-run-lambda for Lambda issues (resumes the same
suspended VM), df-run for Kata. Dedup name df-run-lambda-<issue-number>.
- RBAC: dark-factory-workflow SA gets microvms + secrets + microvmsandboxes
(kro.run) in agent-sandbox-system, gated on microvm.enabled — for the
provision/terminate kubectl steps.
- IAM: PodIdentityAssociation binds dark-factory-workflow → the existing
hub-ack-lambdamicrovms-controller role (USER-APPROVED reuse; additive, no policy
change) so the steps can call aws lambda-microvms get/suspend/resume/terminate +
create-auth-token.
- values: microvm.podIdentity{clusterName:hub, accountId} + workflowServiceAccount.
Reflect the MicroVM-native architecture: two separate WorkflowTemplates (df-run Kata / df-run-lambda), provision-microvm (no bridge/claim/warm-pool), explicit suspend-microvm step, warm-resume with recreate-fallback, one-VM-per-issue naming. Updated timing (native path ~2.5min to PR, faster than old bridge ~3.7min), gotchas table (added suspend-sticks/8h-timeout/note-in-payload/resume-flakiness/ flap fixes; stale CR status note), and all Mermaid diagrams.
merge.js refused to merge PR #136 citing 'Security agent: 1 finding, DevOps: 2 findings' even though every commit status on the head was green. Cause: it took the agents' latest REVIEW regardless of commit — those bodies/inline comments were on the OLD sha (round 1), while the fix-round re-review posts fresh commit STATUSES (green) on the NEW head. So a stale first-round finding permanently blocked any PR that was ever fixed. Filter reviews (commit_id) and inline comments (original_commit_id/commit_id) to the current head sha before counting. Green statuses on the head remain the gate.
…arried-forward) First fix filtered by commit_id, but GitHub AUTO-ADVANCES an inline comment's commit_id to the latest head when the line still exists — so round-1 findings reappeared with commit_id==head and still blocked the merge (PR #136: 3 inline comments carried forward from sha 3b11b497 onto head 8089fa0a). original_commit_id preserves the sha the comment was truly filed against; match on that so only findings genuinely filed against the current head count. Green head statuses (round-2 re-review) remain the authoritative gate.
elamaran11
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flow D — Lambda MicroVM Agent Sandbox (second substrate for the Dark Factory)
Adds Flow D: a second Agent-Sandbox substrate where the autonomous coder runs in an AWS Lambda MicroVM instead of a Kata pod. Base branch:
dark-factory-autonomous-agent-coding-pattern.The two substrates run as separate Argo WorkflowTemplates so the certified Kata pipeline is never touched by Flow D:
df-run— restored byte-identical to the certified pipeline (zero MicroVM logic).df-run-lambda(new) — MicroVM-native: no SandboxClaim, no bridge pod, no warm pool, no lifecycle controller.The Argo Events sensor routes by label (
dark-factory→df-run,darkfactory-lambda→df-run-lambda);df-iterateroutes fix rounds the same way. To the issue author the UX is identical — only a label differs.df-run-lambdaDAG (MicroVM-native)dark-factory-workflowwith the lambda-microvms role via Pod Identity): reads the platform image handoff (imageARN+execRoleARN, built once by KRO/ACK), creates theMicrovmCR (mvm-<issue-number>) + a runHookPayload Secret (review note folded in), waits RUNNING+endpoint, mints an auth token,POST /run→ the hook-server spawns the coder (USE_BEDROCK=1, Bedrock-direct, no cluster network).idlePolicy.autoResumeEnabled=falseand nothing polls the endpoint.provision-microvmresumes the SAME suspended VM (warm); if the pre-GA service failed the resume, it recreates a fresh VM automatically.df-merge-teardowndeletes the CR → controllerTerminateMicrovm.Platform layer (KRO + ACK)
MicrovmSandboxRGD ties MicrovmImage + IAM roles + S3 (one CR);schema.group=kro.run(Managed KRO only watches that group). ARM_64-only; S3 codeArtifact zip.ack-lambdamicrovmscontroller addon (pre-GA — Managed ACK can't bundle it); disabled by default.dark-factory-workflowSA reuses the existing lambda-microvms role (additive, least-privilege) for get/suspend/resume/terminate + create-auth-token.Verified end-to-end (live on the hub)
df-run).Hard-won Flow D fixes (all encoded; see
docs/dark-factory/SUBSTRATE-BENCHMARK.md§Gotchas)WORKSPACE=/tmp/workspace(read-only rootfs) · Bedrock-direct (no Bifrost reachability) ·/logsendpoint (no runtime CloudWatch) ·/runbackground-spawn (30s hook timeout) · HTTP_INGRESS (ALL_INGRESS blocks token mint) · note-via-runHookPayload · autoResume=false + never poll endpoint → suspend sticks · suspendedDurationSeconds=8h (survives review→fix) · hook-server per-run-id guard (resume re-runs coder) · one-VM-per-issue naming (issue-number) · resume-flakiness → recreate-fallback · merge-gate counts findings byoriginal_commit_id(excludes carried-forward round-1 comments).Docs
docs/dark-factory/SUBSTRATE-BENCHMARK.md— Kata vs Lambda comparison, timing, DAGs, gotchas.docs/dark-factory/SUBSTRATE-DIAGRAMS.md— Mermaid diagrams (routing, both substrates, suspend/resume sequence, e2e loop).Known follow-up
Pre-GA resume-from-suspend is occasionally flaky (Internal service error → VM terminates); the recreate-fallback recovers on the next run. Future: in-workflow retry so a single flaky resume self-heals within the same run.