refactor(husk): bring the tap up while the pod is dormant, not while a tenant waits#883
Merged
Conversation
…solve off the warm-claim path
The engine's warm-claim activate is 60 ms P50, but nothing measured what surrounded
it. Adding claim-path stage timing (logClaimStage, the analog of the fork path's
logForkOrchStage) showed the Kubernetes round-trips cost about as much as the microVM
restore, and one of them was a 300 ms cold spike paid by whichever user created the
first sandbox after a controller start.
Three changes, in decreasing size:
* The mTLS dial config is pre-warmed once the manager cache is up. The first claim
after a controller start or a leader handover used to pay 301 ms in dial_tls (the
lazy Secret informer cache fill), against ~1 ms for every claim after it. Best
effort: a pre-warm failure is not fatal, the reconciler still resolves on demand.
* The Restoring status write is now in memory only. It was a full API round-trip on
the critical path that no observer could ever see: control only reaches that line
once a dormant pod has been SELECTED, so the phase was Restoring for exactly the
duration of the activate. A claim that cannot place returns Pending far above it.
The quota live-counter counts Pending and Restoring alike, so concurrency
accounting is unchanged, and every failure path writes its own terminal phase.
* The per-sandbox token Secret is written CONCURRENTLY with the activate RPC. Both
inputs (the minted token, the endpoint derived from the pod IP) are known before
the RPC, so it never needed to be serial. The ordering contract holds: the write is
joined before the Ready status write, so the Secret is durable before the claim is
servable. A failed activate leaves a Secret owner-ref'd to the claim, garbage
collected with it and overwritten by the next pass.
Measured on prod, mean of 9 sequential warm claims:
stage before after
status_write_restoring 10.40 gone
dial_tls 34.42 1.04 (max 301.69 -> 2.37)
token_secret (blocking) 7.42 0.00 (overlapped, span 115 ms)
mark_pod_claimed 16.39 14.86 (the concurrency gate; unchanged)
status_write_ready 9.44 8.39
TOTAL 190.34 139.45 (max 471.04 -> 174.53)
token_secret is reported as two stages on purpose. token_secret_wait is what the
critical path pays (0 ms, the activate outlasts the write); token_secret_span is the
overlapped wall time. Reporting only the span would read as a 150 ms regression when
the write in fact left the critical path entirely.
Verified: internal/controller envtest suite green, both golangci-lint invocations
clean, hosted create/run_code/fork/child-run_code and the #870 restart signal all
still pass against the built controller on prod.
Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
…et write its own copy Two review findings, both real. Claim stage timings were recorded as claim_* labels on mitos_fork_stage_duration_seconds, whose help text documents a fixed fork-stage vocabulary. Any aggregation over "all fork stages" would have quietly included warm-claim work. They now go to mitos_claim_stage_duration_seconds, and a test asserts a claim observation increments that histogram and leaves the fork one untouched. The concurrent token Secret write was handed the LIVE claim object. The activate-failure path mutates claim.Status and returns without joining the goroutine, so a still-running writer shared mutable state with the reconcile path under no synchronization. It now gets claim.DeepCopy() and a precomputed Secret name, so nothing it reads can be written underneath it. The owner reference is unchanged: the copy carries the same name, namespace, and UID, so the Secret is still garbage-collected with the claim. Verified: go test -race ./internal/controller/ passes, and golangci-lint is clean for both GOOS=darwin and GOOS=linux. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
…ducible peer table TTI is create() to a command having actually run inside the sandbox and returned, the definition the public ComputeSDK benchmark uses. Paced 20 iterations from a same-region client against api.mitos.run: P50 495 -> 341 ms, min 273 ms, 20/20. Both runs hit the same server build, so the whole difference is the client-side connection reuse and template caching in the previous commit. The peer numbers are computed, not copied. The vendor leaderboard renders client side and the upstream README shows only chart placeholders, so bench/peer-tti.py fetches the raw per-iteration JSON the harness commits and computes percentiles itself, per the no-unverified-claims rule. Doing that caught an error in the figure I had been carrying: Northflank is 95.9 ms P50, not 269 ms. 269 was Upstash, at 258.3. The result page states plainly what the table cannot support. Their probe is node -v against our print(1) through a Python interpreter, their run is 100 un-paced iterations against our 13 s pacing under the free tier's rate limit, and their client and region are not ours. The single clean statement is the ordering against E2B, the one peer documenting the same isolation primitive: 340.7 ms vs 365.6 ms P50. The sub-40 ms entries are below the cost of restoring a microVM at all on this hardware, so they are almost certainly not booting one per request. ComputeSDK's snapshot-fork benchmark measures object storage (S3, R2, Azure Blob, Tigris), not VM fork, so there is still no public peer number for the primitive mitos is built on. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
…claim Review found a data-integrity bug in the concurrent Secret write this branch added, and it is a real one. The warm-claim path writes the per-sandbox token Secret concurrently with the activate RPC. The activate-failure branch returned WITHOUT joining that write. The writer is still holding THIS pass's token; the next pass mints a NEW token, activates the husk with it, and rewrites the SAME Secret. Whichever write lands last wins. A stale writer landing last leaves a Ready claim whose Secret holds a token the husk never received, and every tenant call made with it is rejected. Nothing would have pointed at the cause. The write now runs under a cancellable context and EVERY path out of the function joins it. The failure branch cancels first so the join is prompt, then drains, then persists Pending. A cancelled write is not an error worth logging; any other failure is logged without the token value, and the next pass re-mints and overwrites it anyway. Proven by TestHuskClaimActivateFailureJoinsTheTokenSecretWrite, through a new ensureTokenSecret seam: the fake writer blocks and deliberately ignores its context, so the reconcile can only proceed by joining it. Without the join the test fails with "claim reached phase Pending while the token Secret write was still in flight". It also asserts the writer's context was cancelled, so a slow write can always be cut short. Verified: go test -race ./internal/controller/ passes (three consecutive runs), and golangci-lint is clean for both GOOS=darwin and GOOS=linux. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
… into perf/claim-control-plane
# Conflicts: # bench/results/2026-07-10-tti-hosted.md
stubbi
changed the title
…a tenant waits
A warm claim pays the whole microVM restore while a tenant waits. Measured on prod, one
matched claim: the husk stub's own activate is 112.8 ms, of which egress_filter is 29.6.
That stage is two things: bringing the tap up (enable forwarding, delete any leftover
link, one ip -batch) and installing the netfilter policy (one atomic nft transaction).
Only the policy depends on the claim. The tap does not: its name derives from the guest
IP, and the in-pod guest IP is a fixed constant.
applyEgressFilter is split into ensureEgressLink and applyEgressPolicy. The composed call
still runs exactly the same commands in the same order, pinned by a test that records
both and compares them command for command, so a pod that does not opt in is byte-for-
byte what it was.
Behind --husk-prepare-egress-link (controller) and --prepare-egress-link (stub), both
DEFAULT OFF, a warm multi-VM pod brings its default VM's tap up while DORMANT under a
default-deny policy, and the claim installs the tenant's policy on it. The hot path
becomes exactly one command.
Fail-closed at every step, which is most of the work:
- Prepare creates the tap and its default-deny policy together and refuses to go
dormant if either half fails, so the pool never offers a pod with a half-built
datapath and a tap never exists without a policy.
- The claim REPLACES the policy in one nft transaction, so there is no window in which
a VM is unfiltered.
- A rejected claim policy tears the tap down (as it always did) AND clears the prepared
marker, so the retry rebuilds the link instead of installing a policy on a tap that no
longer exists, which would poison the warm pod permanently.
- A malformed allowlist is still rejected before any command runs; splitting the call
must not start creating taps for a policy that can never be installed.
Nothing is loaded behind a dormant tap today: the snapshot restore stays on the activate
path. That is slice 2, and it needs this because Firecracker requires the tap to exist at
restore time. The design, the measured budget, the guards, and the rollout are written
down in docs/superpowers/plans/2026-07-10-prepare-time-restore.md, and the threat model
records the new dormant-tap surface.
Verified: go test -race ./internal/husk/, go test ./internal/controller/ ./cmd/husk-stub/
./internal/firecracker/, and golangci-lint for both GOOS=darwin and GOOS=linux. Not yet
canaried on prod; the flag is off, so merging changes nothing until an operator sets it.
Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
stubbi
force-pushed
the
perf/husk-prepare-egress-link
branch
from
e89cec8 to
c4fc27d
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds an opt-in prepare-time dormant egress tap and default-deny policy for multi-VM husks, replaces that policy during activation, instruments warm-claim stages, overlaps token Secret creation with activation, and adds controller validation, tests, and design documentation. ChangesPrepare-time egress and warm-claim flow
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant Controller
participant SandboxPoolReconciler
participant HuskStub
participant Netfilter
Controller->>SandboxPoolReconciler: enable PrepareEgressLink
SandboxPoolReconciler->>HuskStub: pass prepare-egress-link arguments
HuskStub->>Netfilter: create dormant tap and default-deny policy
Controller->>HuskStub: send Activate request
HuskStub->>Netfilter: atomically replace policy with tenant rules
HuskStub-->>Controller: return activation result
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@bench/peer-tti.py`:
- Line 37: Replace all three percent-format calls in the benchmark script with
equivalent f-strings, including the call printing data["timestamp"] and
data["config"], to resolve Ruff UP031 diagnostics without changing the output.
- Around line 40-41: Update the `tti` comprehension in the iteration percentile
calculation to first require each iteration to be a dictionary, then accept only
numeric `ttiMs` values that are finite and non-negative; explicitly exclude
booleans, NaN, and infinities before computing percentiles.
In `@cmd/controller/main.go`:
- Line 240: Validate the resolved run-mode prerequisites after resolveRunMode:
reject configurations where PrepareEgressLink (including
--husk-prepare-egress-link) is enabled while MultiVMFork is disabled, or derive
a single effective value that is passed consistently to both the pool builder
and claim reconciler wiring at PrepareEgressLink and MultiVMFork.
In `@docs/threat-model.md`:
- Line 1222: Update the Husk egress enforcement row to distinguish the
activation-time path, which remains IMPLEMENTED and KVM-VERIFIED, from the
experimental prepare-time path. In the prepare-time section referencing
“--husk-prepare-egress-link” and “--prepare-egress-link,” explicitly state that
it is default-off and pending KVM/canary validation, removing any implication
that this path is already verified.
In `@internal/controller/huskpod_test.go`:
- Around line 1510-1538: Extend TestBuildHuskPodThreadsPrepareEgressLink with a
PrepareEgressLink: true and MultiVM: false case. Assert the generated Husk pod
omits --prepare-egress-link, --in-pod-guest-ip, and --in-pod-gateway-ip,
preserving the contract that preparation is only supported for MultiVM.
In `@internal/controller/huskpod.go`:
- Around line 203-207: Enforce the MultiVM prerequisite for PrepareEgressLink in
both pod argument construction and reconciliation. Update the relevant checks
around the PrepareEgressLink argument block and the reconciliation forwarding of
r.PrepareEgressLink so the option is enabled only when both PrepareEgressLink
and MultiVM are true, preventing unsupported pod shapes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 206f6a09-867f-442f-b036-2c5f7edc74e5
📒 Files selected for processing (19)
bench/peer-tti.pycmd/controller/main.gocmd/husk-stub/main.godocs/superpowers/plans/2026-07-10-prepare-time-restore.mddocs/threat-model.mdinternal/controller/husk_activation_test.gointernal/controller/huskpod.gointernal/controller/huskpod_test.gointernal/controller/metrics.gointernal/controller/metrics_internal_test.gointernal/controller/sandboxclaim_controller.gointernal/controller/sandboxpool_controller.gointernal/controller/suite_test.gointernal/controller/testsupport_test.gointernal/husk/multivm.gointernal/husk/multivm_network_test.gointernal/husk/netfilter.gointernal/husk/netfilter_test.gointernal/husk/stub.go
…me egress link
Review found that PrepareEgressLink was documented as requiring MultiVM and then trusted
to the caller. With --husk-prepare-egress-link set and --multi-vm-fork unset, the pool
builder received MultiVM=false while the claim reconciler still received
PrepareEgressLink=true, and the stub got flags it can never act on: only the multi-VM
Prepare path reaches prepareInstance, so a single-VM stub never brings its tap up, and
its Activate does not consult preparedLinkTap either. The operator asked for a latency
optimization and would have got a pod shape that cannot deliver one, silently.
Enforced at all three layers, because each is a boundary someone can reach on its own:
- cmd/controller refuses the combination at startup rather than running degraded;
- the pod builder emits the flags only when MultiVM is also set;
- the stub states the !multiVM guard instead of relying on the fact that only the
multi-VM Prepare currently reaches it.
Also from review, the threat model: the husk egress row is marked IMPLEMENTED and
KVM-VERIFIED end to end, and the prepare-time paragraph sat inside it, so an experimental
default-off path inherited a verification it has not had. It now says plainly that the
KVM verification covered the activation-time filter, and that the prepare-time path needs
a production canary and its own KVM gate before it may be called verified.
TestBuildHuskPodOmitsPrepareEgressLinkWithoutMultiVM pins the pod-builder gate.
Verified: go test -race ./internal/husk/, go test ./internal/controller/ ./cmd/husk-stub/,
and golangci-lint clean for both GOOS=darwin and GOOS=linux.
Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
Contributor
Author
|
All addressed in Validate the prepare-egress prerequisites (Major). Correct, and worse than inconsistent: it was silently useless. Only the multi-VM Enforced at all three layers, because each is a boundary someone can reach independently:
Cover the unsupported MultiVM combination (Trivial). Added Do not label the experimental path KVM-verified (Minor). Well caught, and this is the kind of thing that quietly becomes false documentation. The row is titled "IMPLEMENTED and KVM-VERIFIED end to end" and my paragraph sat inside it, so a default-off experimental path inherited a verification it has not had. It now says plainly that the KVM verification covered the activation-time filter, and that the prepare-time link needs a production canary and its own KVM gate before it may be called verified. The plan document says the same.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cmd/controller/main.go`:
- Around line 143-151: Reject --husk-prepare-egress-link when raw-forkd or mock
mode is enabled, since resolveRunMode disables husk pods and the option
otherwise has no effect. Extend the validation around the existing
prepareEgressLink and multiVMFork check in main, using the resolved run-mode
values (or equivalent raw-forkd/mock flags), and fail fast with a clear
logger.Error message before continuing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: afb9f305-ca5e-4be2-b5ea-ace0bdfd6d0b
📒 Files selected for processing (6)
cmd/controller/main.godocs/superpowers/plans/2026-07-10-prepare-time-restore.mddocs/threat-model.mdinternal/controller/huskpod.gointernal/controller/huskpod_test.gointernal/husk/multivm.go
…ess-link # Conflicts: # bench/peer-tti.py # internal/controller/sandboxclaim_controller.go # internal/controller/testsupport_test.go
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
internal/controller/sandboxclaim_controller.go (1)
993-1001: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
activate_rpcstage is recorded even when the TLS dial fails and Activate is skipped.When
huskDialTLSreturns an error,activateis not invoked, yetlogClaimStage(..., "activate_rpc", ...)still fires with a near-zero duration. During a husk outage (mass dial failures) this floods theactivate_rpchistogram with ~0 samples, making activation latency look healthy exactly when it is broken. Only observe/log the stage when the RPC actually ran.📊 Only record activate_rpc when the RPC executes
var res husk.ActivateResult - mark = time.Now() if err == nil { + mark = time.Now() res, err = activate(ctx, addr, tlsConf, req) + logClaimStage(logger, claim.Name, "activate_rpc", time.Since(mark)) } - logClaimStage(logger, claim.Name, "activate_rpc", time.Since(mark))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/sandboxclaim_controller.go` around lines 993 - 1001, Only call logClaimStage for the "activate_rpc" stage when activate actually executes successfully past the TLS error check. Update the flow around huskDialTLS and activate so the timing mark and activate_rpc observation are conditional on err == nil, while preserving the existing error propagation when TLS dialing fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@internal/controller/sandboxclaim_controller.go`:
- Around line 993-1001: Only call logClaimStage for the "activate_rpc" stage
when activate actually executes successfully past the TLS error check. Update
the flow around huskDialTLS and activate so the timing mark and activate_rpc
observation are conditional on err == nil, while preserving the existing error
propagation when TLS dialing fails.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: d6eab41a-cfc6-49fd-a81d-a63be86c60cf
📒 Files selected for processing (1)
internal/controller/sandboxclaim_controller.go
14 tasks
Review: the startup guard checked --multi-vm-fork but not the resolved run path, so --husk-prepare-egress-link --multi-vm-fork --enable-raw-forkd (or --mock) was accepted. On the raw-forkd path resolveRunMode sets enableHuskPods false, so no husk pod is ever created and the requested optimization is silently inert. The check now runs against the RESOLVED enableHuskPods and requires it, extracted into validatePrepareFlags so it is unit-tested directly (five cases, including the raw-forkd combinations) rather than only reachable through os.Exit. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
Contributor
Author
|
Fixed in It now runs against the RESOLVED func validatePrepareFlags(prepareEgressLink, multiVMFork, enableHuskPods bool) error {
if prepareEgressLink && (!multiVMFork || !enableHuskPods) {
return fmt.Errorf("--husk-prepare-egress-link requires --multi-vm-fork AND the husk-pods run path ...")
}
return nil
}Extracted into |
stubbi
pushed a commit
that referenced
this pull request
Jul 10, 2026
… a tenant waits Slice 2 of docs/superpowers/plans/2026-07-10-prepare-time-restore.md, building on the tap-at-prepare slice (#883). Together they move almost the whole warm-claim activate off the hot path. Measured on prod, one matched claim: the husk activate is 112.8 ms, of which the restore (vmstate_restore 22.9), the resume (2.4), and the guest-ready wait (40.6, largely demand fault-in under the lazy UFFD restore) are ~66 ms. All of it is a function of the pod and its template, not of the claim. Only the fork-correctness handshake (fresh entropy, the clock step, the per-VM network re-addressing, and the tenant secrets) and serving the sandbox API with the claim token actually need the claim. Behind --husk-prepare-restore (which requires --husk-prepare-egress-link, because Firecracker binds the baked NIC to the tap at snapshot load), a warm pod now loads its default VM's snapshot and RESUMES its guest while dormant. Activate takes a fast path when the instance is pre-restored: it skips the restore, the rootfs rebind, the resume, and the guest-ready wait, re-dials the already-running guest (sub-millisecond, the vsock listener has been up since the prepare-time resume), and pays only the handshake. The handshake-and-serve tail is extracted into activateHandshakeAndServe so both paths share it byte-for-byte. Fail-closed, which is the load-bearing part: - Prepare refuses to go dormant if the restore, rootfs rebind, resume, or guest-ready fails, so the pool never offers a half-restored pod. - Activate refuses a claim whose snapshot dir differs from the one pre-restored, with a named error, because a resumed guest cannot be reloaded and must never be served against the wrong image. - Only the DEFAULT VM is pre-restored. A co-located fork child (its own snapshot, childUFFD backend, per-fork tap) and the pre-warm generic child are excluded, so the fork paths are unchanged. - The ready conn proven at prepare is CLOSED, not held across dormancy where it could go stale. The dormant guest is a new surface and the threat model records it: it serves no tenant (the sandbox API is bound and the token delivered only at the claim), it is reseeded fail-closed at the claim's NotifyForked exactly as a restore-at-activate guest is (the dormancy window only enlarges the absolute clock step), and it holds its working set resident (~72 MiB/sandbox) versus ~0 for a never-loaded VMM. DEFAULT OFF at every layer: cmd/controller rejects --husk-prepare-restore without --husk-prepare-egress-link (and therefore without --multi-vm-fork), the pod builder emits --prepare-restore only inside the prepared-link block, and the stub no-ops unless the tap was prepared. Off, the restore path is byte-for-byte the activate-time one. NOT KVM-verified and NOT canaried on prod: this needs a firecracker-test gate and a one-pod canary before it may be called verified, and it is the highest-risk change in the latency effort (it resumes a live guest during dormancy on the tenant-VM boot path). Verified with the mock VMM (the state machine is engine-independent): TestPrepareRestoresAndResumesTheGuestWhileDormant (load+resume at prepare, none at activate), TestPrepareRestoreIsOptIn (default off is byte-for-byte activate-time), TestActivateFailsClosedOnASnapshotMismatchAfterPrepareRestore, TestBuildHuskPodThreadsPrepareRestore. Each fails on the unfixed code. go test -race ./internal/husk/, go test ./internal/controller/ ./cmd/husk-stub/, and golangci-lint clean for both GOOS=darwin and GOOS=linux. internal/husk is a security-sensitive path per CLAUDE.md and needs a named human reviewer. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
stubbi
added a commit
that referenced
this pull request
Jul 10, 2026
… a tenant waits (#892) * refactor(husk): restore and resume the guest while dormant, not while a tenant waits Slice 2 of docs/superpowers/plans/2026-07-10-prepare-time-restore.md, building on the tap-at-prepare slice (#883). Together they move almost the whole warm-claim activate off the hot path. Measured on prod, one matched claim: the husk activate is 112.8 ms, of which the restore (vmstate_restore 22.9), the resume (2.4), and the guest-ready wait (40.6, largely demand fault-in under the lazy UFFD restore) are ~66 ms. All of it is a function of the pod and its template, not of the claim. Only the fork-correctness handshake (fresh entropy, the clock step, the per-VM network re-addressing, and the tenant secrets) and serving the sandbox API with the claim token actually need the claim. Behind --husk-prepare-restore (which requires --husk-prepare-egress-link, because Firecracker binds the baked NIC to the tap at snapshot load), a warm pod now loads its default VM's snapshot and RESUMES its guest while dormant. Activate takes a fast path when the instance is pre-restored: it skips the restore, the rootfs rebind, the resume, and the guest-ready wait, re-dials the already-running guest (sub-millisecond, the vsock listener has been up since the prepare-time resume), and pays only the handshake. The handshake-and-serve tail is extracted into activateHandshakeAndServe so both paths share it byte-for-byte. Fail-closed, which is the load-bearing part: - Prepare refuses to go dormant if the restore, rootfs rebind, resume, or guest-ready fails, so the pool never offers a half-restored pod. - Activate refuses a claim whose snapshot dir differs from the one pre-restored, with a named error, because a resumed guest cannot be reloaded and must never be served against the wrong image. - Only the DEFAULT VM is pre-restored. A co-located fork child (its own snapshot, childUFFD backend, per-fork tap) and the pre-warm generic child are excluded, so the fork paths are unchanged. - The ready conn proven at prepare is CLOSED, not held across dormancy where it could go stale. The dormant guest is a new surface and the threat model records it: it serves no tenant (the sandbox API is bound and the token delivered only at the claim), it is reseeded fail-closed at the claim's NotifyForked exactly as a restore-at-activate guest is (the dormancy window only enlarges the absolute clock step), and it holds its working set resident (~72 MiB/sandbox) versus ~0 for a never-loaded VMM. DEFAULT OFF at every layer: cmd/controller rejects --husk-prepare-restore without --husk-prepare-egress-link (and therefore without --multi-vm-fork), the pod builder emits --prepare-restore only inside the prepared-link block, and the stub no-ops unless the tap was prepared. Off, the restore path is byte-for-byte the activate-time one. NOT KVM-verified and NOT canaried on prod: this needs a firecracker-test gate and a one-pod canary before it may be called verified, and it is the highest-risk change in the latency effort (it resumes a live guest during dormancy on the tenant-VM boot path). Verified with the mock VMM (the state machine is engine-independent): TestPrepareRestoresAndResumesTheGuestWhileDormant (load+resume at prepare, none at activate), TestPrepareRestoreIsOptIn (default off is byte-for-byte activate-time), TestActivateFailsClosedOnASnapshotMismatchAfterPrepareRestore, TestBuildHuskPodThreadsPrepareRestore. Each fails on the unfixed code. go test -race ./internal/husk/, go test ./internal/controller/ ./cmd/husk-stub/, and golangci-lint clean for both GOOS=darwin and GOOS=linux. internal/husk is a security-sensitive path per CLAUDE.md and needs a named human reviewer. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks> * fix(husk): refuse a mismatched pre-restored snapshot before any side effect Review found that the snapshot-identity gate fired too late. It sat at the pre-restored fast path, AFTER the verify-on-activate re-hash and AFTER the per-VM egress filter, so a claim naming a different snapshot than the one this pod pre-restored had already had the tenant's real egress policy installed on the tap fronting the wrong guest (and a re-verify run against the mismatched dir) before the "refusing to serve the wrong image" rejection fired. The VM never went active and never got secrets or a reseed, but the network mutation happened before the gate, which is the opposite of fail-closed. The identity check now runs FIRST, right after the empty-snapshot-dir guard, before any verify or egress mutation. The fast path keeps only a comment noting the invariant now holds. TestActivateFailsClosedOnASnapshotMismatchAfterPrepareRestore now records the net command count and asserts a mismatch adds NONE; with the gate moved back to its old position the test fails with "mutated the egress datapath (1 new net commands)". Also from review: two docstrings named a nonexistent opts.preWarmBoot; the actual guard is opts.skipRootfsClone || opts.reuseVM != nil, and the docstrings now say so. Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks> --------- Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks> Co-authored-by: Jannes Stubbemann <jannes@openclaw.rocks>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thinking Path
Linked Issues or Issue Description
No issue existed. Latency attribution, and the first slice of
docs/superpowers/plans/2026-07-10-prepare-time-restore.md(added here).Related: #871 (the gap between the published warm-claim activate and what the hardware does).
Time to Interactive on the hosted API is 307.7 ms P50 (
bench/results/2026-07-10-tti-hosted.md), of which create is ~196 ms. The husk activate is the largest single component the engine controls.What Changed
applyEgressFilteris split intoensureEgressLink(forwarding, link delete,ip -batch) andapplyEgressPolicy(one atomicnft -f -). The composed call is unchanged:TestApplyEgressFilterIsTheTwoHalvesrecords both paths and compares them command for command, argv and stdin.Options.PrepareEgressLinkplusInPodGuestIP/InPodGatewayIPon the stub;--prepare-egress-link,--in-pod-guest-ip,--in-pod-gateway-iponcmd/husk-stub;--husk-prepare-egress-linkoncmd/controller, threaded throughHuskPodOptions. All DEFAULT OFF.prepareInstancebrings the default VM's tap up with a default-deny policy when opted in, and records it onactiveTap(soClosetears it down even if the pod is never claimed) and on a newpreparedLinkTap.activateInstanceinstalls only the tenant policy whenpreparedLinkTapmatches the tap the claim resolves to, and takes the full path otherwise (a fork child, or a pod that did not opt in).docs/superpowers/plans/2026-07-10-prepare-time-restore.md: the measured budget, what is actually claim-specific, the target, the guards, and the three slices.docs/threat-model.md: the dormant-tap surface.Fail-closed, which is most of the work
nfttransaction, so there is no window in which a VM is unfiltered.EBUSY(husk activation: egress-setup step fails on stock k3s (tap teardown now deterministic via #429) #428).Nothing is loaded behind a dormant tap today: the snapshot restore stays on the activate path. That is slice 2.
Verification
go test -race ./internal/husk/ -count=1go test ./internal/controller/ ./cmd/husk-stub/ ./internal/firecracker/ -count=1go build ./...,gofmt -lclean on the touched treesgolangci-lint run --timeout=8mclean for./internal/husk/... ./cmd/husk-stub/... ./cmd/controller/..., and again withGOOS=linuxNew tests:
TestApplyEgressFilterIsTheTwoHalves,TestApplyEgressPolicyTouchesOnlyNft,TestApplyEgressPolicyFailsClosed,TestEnsureEgressLinkInstallsNoPolicy,TestApplyEgressFilterRejectsABadAllowlistBeforeTouchingTheLink,TestPrepareBringsTheTapUpDormantAndActivateOnlyInstallsThePolicy,TestPrepareTimeLinkIsOptIn,TestAFailedClaimPolicyRebuildsTheLinkOnRetry,TestBuildHuskPodThreadsPrepareEgressLink.The controller test asserts the stub receives the fixed in-pod
/30alongside the flag: a mismatch would bring up a tap the claim never resolves to, and the skip would silently not apply.Not yet canaried on prod. The flag is off, so merging changes nothing until an operator sets it. The rollout in the plan is: canary ONE dormant husk pod, check
restartCount=0, claim it, compare theegress_filterstage, then recycle the pool.Risks
The security surface moves, and the threat model is updated in this commit: a dormant husk pod may now hold a tap in its own netns before any tenant is attached. It carries a default-deny policy from the moment it exists, Prepare fails closed if it cannot, nothing is loaded behind it, and the pod netns is shared with no tenant.
The subtle failure mode is a stale prepared marker: if a claim tore the tap down but the marker survived, the retry would install a policy on a missing tap and the pod would be poisoned.
TestAFailedClaimPolicyRebuildsTheLinkOnRetryexists for exactly that, and the marker is cleared before either filter call so no error path can skip it.Off (the default, and every deployment today) the filter path is byte-for-byte the activate-time one, which the two-halves test pins.
Model Used
claude-opus-4-8), 1M context, extended thinking.Checklist
#NNN/ mitos-run/mitos URLs)Signed-off-bytrailer (git commit -s)Summary by CodeRabbit