Skip to content

prepare-time restore: move the microVM snapshot restore off the warm-claim hot path (slices 2-3) #889

Description

@stubbi

Tracks the remaining slices of docs/superpowers/plans/2026-07-10-prepare-time-restore.md.

Why

A warm claim pays the whole microVM restore while a tenant waits. Measured on prod (mitos-kvm-1), one matched claim: husk activate 112.8 ms, of which guest_ready 40.6, egress_filter 29.6, vmstate_restore 22.9, handshake 17.2, resume 2.4. Everything except the handshake is work a pod could do while dormant. guest_ready is largely demand fault-in under the lazy UFFD restore, which is also why the first run_code costs ~105 ms against ~35 ms warm: the ipykernel is in the snapshot (warmKernel: true) but its pages are not resident.

Time to Interactive is ~310 ms P50 (~195 create + ~115 first exec). This is the single largest lever on both halves and the path to beating Daytona (136 ms) and Northflank (95.9 ms).

Slices

  1. Tap at Prepare (PR refactor(husk): bring the tap up while the pod is dormant, not while a tenant waits #883, open): split applyEgressFilter so a dormant pod brings its tap up under a default-deny policy and a claim pays only the atomic nft transaction. ~20 ms, and a prerequisite because Firecracker needs the tap to exist at restore time.
  2. Restore at Prepare: move setLiveCowMemSource / LoadSnapshot / PatchDrive / Resume / guest-ready into prepareInstance, behind the same flag, with the guards in the plan (snapshot identity, fork children, atomic policy swap, dormant reseed, teardown, ~72 MiB/sandbox resident memory). Activate becomes policy + handshake + serve_api. Removes vmstate_restore + most of guest_ready from the claim, ~55 ms.
  3. Prefault the kernel: run the inert warm cell at Prepare so the first tenant run_code does not fault the ipykernel pages in, ~70 ms off first exec.

Each slice ships with its own KVM gate in firecracker-test and a one-pod prod canary before pool recycle. Default OFF until measured.

Related

Model Used

Claude Opus 4.8 (claude-opus-4-8).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions