gocracker

One binary. One command. Real VM isolation.

What is gocracker?

I fell in love with Firecracker -- the idea of booting a real Linux VM in milliseconds with minimal overhead is brilliant. But every time I wanted to just run a container image as a microVM, I had to: pull the image manually, extract the rootfs, build an ext4 disk, generate an initrd, write a JSON config, set up a TAP interface, configure iptables... and only then call the Firecracker API. I kept thinking: why can't I just type one command?

So I built gocracker. It is a micro-VMM written in pure Go that does everything Firecracker does at the KVM level -- virtio devices, jailer, seccomp -- but adds the developer experience on top: pull any OCI image, build any Dockerfile, clone any git repo, orchestrate Docker Compose stacks. Each service becomes a real Linux VM.

gocracker run --image ubuntu:22.04 --kernel artifacts/kernels/gocracker-guest-standard-vmlinux --wait

No Docker daemon. No containerd. No runc. Just KVM.

This started as a hobby project to understand KVM and Go systems programming. It grew into something that boots 378 real-world projects, runs Flask + PostgreSQL compose stacks on ARM64 bare metal, and supports snapshots, live migration, and a Firecracker-compatible REST API. All in a single static binary.

Quick Start

# One-time: join the kvm group so gocracker works without sudo
sudo usermod -aG kvm $USER && newgrp kvm

# Build gocracker + unpack the prebuilt guest kernel that ships with the repo
make build kernel-unpack

# Run your first microVM — no sudo needed
./gocracker run \
  --image alpine:latest \
  --kernel ./artifacts/kernels/gocracker-guest-standard-vmlinux \
  --cmd "echo hello from a real VM" \
  --wait

That's it — alpine boots in ~80 ms (warm OCI cache), runs the command, and exits. No kernel build required: the repo ships gzipped guest kernels under artifacts/kernels/ for both x86_64 and arm64. make kernel-unpack decompresses them in place. To build a custom kernel from source, run make kernel-guest (x86_64) or make kernel-guest-arm64 (arm64) instead — that takes ~10 min the first time.

Cassette Deploys — one VM, infinite versions

The base VM never changes. Only the cassette does.

Every platform that runs user code faces the same problem: how do you deploy a new version without rebooting every running instance? The standard answer is containers (rebuild the image, restart the process) or blue/green deploys (spin up a new fleet, cut traffic over). Both require teardown and cold boot.

gocracker solves this differently. Application code lives on a separate ext4 disk — not baked into the rootfs. The base VM (kernel + runtime + libraries) is a frozen snapshot. At boot, gocracker attaches the code disk and mounts it inside the guest. To run a different version, attach a different disk. The VM never reboots.

┌─────────────────────────────────┐
│  base snapshot (frozen, shared) │  node:20-alpine · Python 3 · Go 1.22
│  kernel + runtime + libraries   │  built once, restored in ~40 ms
└────────────────┬────────────────┘
                 │ attach at boot
        ┌────────┴────────┐
        │   code disk     │  your app — ext4, read-only, swappable
        │  /app/v1  v2  … │  swap the disk, not the VM
        └─────────────────┘

Next.js API route — deploy without Lambda cold starts. This is what a cassette-based deploy handler looks like compared to the Lambda model:

// Lambda / Vercel serverless — cold boot on every deploy
// AWS rebuilds your container image, distributes it, cold-starts on first request
export async function POST(req: Request) {          // ~500 ms cold start after deploy
  const { code } = await req.json();
  // ... runs in shared-kernel container, no hardware isolation
}

// app/api/run/route.ts — gocracker cassette model
// The runtime VM (node:20-alpine + all deps) is a frozen snapshot, always warm.
// Only the cassette (your code as ext4) changes on each deploy.
import { Client } from '@gocracker/sdk';

const gc = new Client('http://127.0.0.1:9091');

export async function POST(req: Request) {
  const { version } = await req.json();             // e.g. "v41" or "v42"

  // VM boots from warm snapshot in ~40 ms — runtime was never torn down
  const sb = await gc.createSandbox({
    image: 'node:20-alpine',                        // frozen snapshot, shared
    network_mode: 'none',
    code_disks: [
      { host_path: `/cassettes/app-${version}.ext4`, mount: '/app', read_only: true },
    ],
  });

  try {
    // exec runs inside a real KVM-isolated VM, not a shared container
    const tb = sb.toolbox();
    const result = await tb.exec(['node', '/app/server.js', '--once']);
    return Response.json({ output: result.stdout });
  } finally {
    await sb.stop();   // VM destroyed — next request gets a fresh one from the pool
  }
}

Lambda deploy:  rebuild image (2 min) → push to ECR → distribute → cold boot (~500 ms)
Cassette deploy: pack code into ext4 (5 s) → drop file → next request warm (~40 ms)

This is the same architecture Vercel uses for preview deployments — code is a separate artifact from the runtime — but with real KVM hardware isolation instead of shared-kernel containers. To our knowledge, no other open-source microVM project exposes cassette-style deploys as a first-class API.

See real working examples: examples/code-disk/ and the smoke test: tests/manual-smoke/code-disk-apps/run.sh.

Building a platform on top of gocracker

Every existing platform (Fly.io, Railway, Render, AWS Lambda) bakes code and runtime into the same container image. Deploying a new version means rebuilding the entire image, pushing it to a registry, pulling it on every node, and cold-booting a new container — even if only one .js file changed.

With gocracker's code-disk model the runtime and the code are permanently separate:

┌─────────────────────────────────────────────────────────────────┐
│  DEPLOY PIPELINE                                                │
│                                                                 │
│  git push → CI packages only your code into an ext4 disk       │
│             (no docker build, no image pull, takes seconds)     │
│                                                                 │
│  gocracker swaps the disk on the warm VM snapshot              │
│  → new version live in < 100 ms                                 │
└─────────────────────────────────────────────────────────────────┘

┌──────────────────────────────┐     ┌──────────────────────────┐
│  base snapshot  (never       │     │  code disk  (changes     │
│  changes across deploys)     │     │  on every deploy)        │
│                              │     │                          │
│  node:20-alpine              │ ←── │  app-v42.ext4            │
│  + your framework            │     │  ├── index.js            │
│  + all npm deps              │     │  ├── routes/             │
│  + runtime config            │     │  └── package.json        │
└──────────────────────────────┘     └──────────────────────────┘

What this unlocks that no other open-source platform does today:

	Traditional containers	gocracker code-disk
Deploy a new version	Rebuild image + pull + cold boot	Package code into ext4, swap disk
Deploy time	Minutes (build + push + pull)	Seconds (code only, no runtime)
Rollback	Pull previous image tag	Point to previous ext4
Preview environments	Full container per PR	One ext4 per PR, shared base VM
A/B testing	Two container fleets	Two VMs, same snapshot, different cassette
AI code execution	Docker per session (heavy)	One cassette per conversation, warm VM
Isolation	Shared kernel (containers)	KVM hardware per tenant

The base snapshot is built once when you define your stack (e.g. node:20-alpine + Express + your deps). Every deploy after that only touches the code disk. Runtime cold-start is eliminated because the snapshot is already warm in the pool.

Features

Container Sources -- OCI images from any registry, local Dockerfiles (BuildKit AST), git repos with auto-detected Dockerfiles, local directories.

Orchestration -- Docker Compose stacks as microVMs, healthchecks (CMD/CMD-SHELL executed in-guest), depends_on with conditions (service_healthy, service_completed_successfully), .env and interpolation.

Networking -- --net auto (CLI) and network_mode=auto (REST) for single-command TAP + IPv4 + NAT, per-stack network namespaces for Compose, manual TAP for advanced setups, deterministic per-VM MAC addresses, port publishing.

Isolation -- KVM hardware virtualization, Firecracker-style jailer (gocracker-jailer), seccomp filters (per-arch), private mount namespaces, pivot_root isolation for builds.

Operations -- Snapshot / restore / pause / resume (RAM + vCPU + device state), in-place /clone (snapshot + restore as a new VM on the same server, fresh tap + guest re-IP), live migration (stop-and-copy), Firecracker-compatible REST API with extensions, structured logging, event streaming (SSE). Stale /tmp/gocracker-* from SIGKILL'd prior runs is pruned on serve startup.

Devices -- virtio-net, virtio-blk, virtio-rng, virtio-vsock, virtio-balloon (manual + auto reclaim), virtio-fs, UART 16550A serial console, memory hotplug.

Sandbox control plane -- gocracker-sandboxd — HTTP daemon that wraps the runtime with warm pools (~1.5 ms p95 lease), content-addressed templates (~80 µs cache hit), HMAC-signed preview URLs, and Python / Go / JS SDKs with typed errors and context-manager sandbox lifecycle. Per-sandbox Firecracker-style UDS at <state-dir>/sandboxes/<id>.sock speaks directly to a baked-in toolbox agent on vsock 10023 (framed exec + files + git + secrets + SetNetwork re-IP). See the Sandboxd overview.

Cassette deploys -- Separate ext4 disk carries application code, mounted inside the guest at boot via gc.code_disk= kernel cmdline. The base VM snapshot never changes; swap the cassette to deploy a new version. Attach multiple cassettes to one running VM for side-by-side version comparison without rebooting. See Cassette Deploys.

Examples

Run Alpine, print a message
Interactive Ubuntu session
Build from Dockerfile
Clone and boot a git repo
Docker Compose (Flask + PostgreSQL)
Exec into a running Compose service
Networking (auto NAT)
Multi-vCPU
Sandbox pool from template (REST API)
Sandbox control plane — raw HTTP — gocracker-sandboxd HTTP daemon + per-sandbox UDS + toolbox agent (exec / files / git / re-IP)

The snippets below use $KERNEL for the guest kernel path. Set it once after running make build kernel-unpack:

export KERNEL=$PWD/artifacts/kernels/gocracker-guest-standard-vmlinux

1. Run Alpine, print a message

sudo ./gocracker run --image alpine:latest --kernel "$KERNEL" --cmd "echo hello from a real VM" --wait

2. Interactive Ubuntu session

sudo ./gocracker run --image ubuntu:22.04 --kernel "$KERNEL"

Drops you into a shell inside the VM over the serial console.

3. Build from Dockerfile

sudo ./gocracker run --dockerfile tests/examples/python-api/Dockerfile \
  --context tests/examples/python-api --kernel "$KERNEL" --wait

Parses the Dockerfile, builds layers, creates an ext4 disk, and boots the result.

4. Clone and boot a git repo

sudo ./gocracker repo --url https://github.com/user/myapp --kernel "$KERNEL" --wait

Clones the repo, auto-detects the Dockerfile, builds, and boots.

5. Docker Compose (Flask + PostgreSQL)

sudo ./gocracker compose \
  --file tests/manual-smoke/fixtures/compose-todo-postgres/docker-compose.yml \
  --kernel "$KERNEL" --wait

Each service runs in its own VM. PostgreSQL waits for healthcheck, then the app starts. Port 18081 is published to the host.

6. Exec into a running Compose service

# Start the API server + compose stack
sudo ./gocracker compose --file docker-compose.yml --kernel "$KERNEL" --server http://127.0.0.1:8080

# In another terminal, exec into a service
sudo ./gocracker compose exec --server http://127.0.0.1:8080 --file docker-compose.yml app

Opens an interactive shell inside the running VM.

7. Networking (auto NAT)

sudo ./gocracker run --image nginx:alpine --kernel "$KERNEL" --net auto --wait

Creates a TAP device, assigns IPv4 addresses, sets up NAT. The guest gets internet access automatically.

8. Multi-vCPU

sudo ./gocracker run --image alpine:latest --kernel "$KERNEL" --cpus 4 --mem 512 \
  --cmd "nproc && free -m" --wait

9. Sandbox pool from template (REST API)

Boot a template VM, install tools once, then clone it per sandbox. Each clone gets a fresh TAP + subnet and the guest is re-IP'd on the fly via the exec agent so outbound networking works without the caller pre-allocating any host resources. network_mode=auto creates TAP devices so the server process must have root or CAP_NET_ADMIN (sudo or setcap cap_net_admin+ep on the binary); /run and /clone return 403 with an actionable message if it's missing.

sudo ./gocracker serve --addr 127.0.0.1:8080 --trusted-kernel-dir ./artifacts/kernels &

# 1. Boot template (network_mode=auto + exec + wait=true)
TEMPLATE=$(curl -sS http://127.0.0.1:8080/run -X POST \
  -H 'Content-Type: application/json' \
  -d '{"image":"alpine:3.20","kernel_path":"./artifacts/kernels/gocracker-guest-minimal-vmlinux",
       "mem_mb":256,"network_mode":"auto","exec_enabled":true,"wait":true,
       "cmd":["/bin/sh","-lc","sleep infinity"]}' | jq -r .id)

# 2. Install tools on the template (persists in the ext4 disk)
curl -sS http://127.0.0.1:8080/vms/$TEMPLATE/exec -X POST \
  -d '{"command":["/bin/sh","-lc","apk add --no-cache bc && echo TEMPLATE-READY > /tmp/marker"]}'

# 3. Clone it. Source keeps running. Clone gets tclone-<N> tap + fresh /30.
CLONE=$(curl -sS http://127.0.0.1:8080/vms/$TEMPLATE/clone -X POST \
  -d '{"exec_enabled":true,"network_mode":"auto"}' | jq -r .id)

# 4. Clone has bc (disk inherited) AND working outbound net (re-IP'd eth0).
curl -sS http://127.0.0.1:8080/vms/$CLONE/exec -X POST \
  -d '{"command":["/bin/sh","-lc","echo 2*21 | bc && apk add --no-cache file"]}'

# 5. Optionally pause idle clones; resume on next request.
curl -sS -X POST http://127.0.0.1:8080/vms/$CLONE/pause -d '{}'
curl -sS -X POST http://127.0.0.1:8080/vms/$CLONE/resume -d '{}'

See docs/SNAPSHOTS.md#sandbox-template-flow for the full walkthrough and the pkg/warmcache / pkg/warmpool primitives (content-addressable snapshot cache + pre-spawned VMM pool) that make this pattern sub-100 ms on the hot path.

10. Sandbox control plane — raw HTTP

gocracker-sandboxd is a small HTTP service that wraps the runtime with sandbox-specific lifecycle (see the Sandboxd overview below for the high-level description and perf numbers). Each POST /sandboxes cold-boots a VM in ~80 ms (or ~40 ms on warm restore), auto-generates a per-sandbox Firecracker-style UDS, and exposes the toolbox agent for framed exec / files / git / re-IP RPCs. Most users will reach it through the Python/Go/JS SDKs rather than curl.

sudo ./gocracker-sandboxd serve --addr 127.0.0.1:9091 \
  --state-dir /var/lib/gocracker-sandboxd &

# Create a sandbox — cold boot ~80 ms on cached image
SB=$(curl -sS http://127.0.0.1:9091/sandboxes -X POST \
  -H 'Content-Type: application/json' \
  -d '{"image":"alpine:3.20","kernel_path":"./artifacts/kernels/gocracker-guest-standard-vmlinux",
       "network_mode":"auto","cmd":["sleep","3600"]}' | jq -r .sandbox.id)
UDS=$(curl -s http://127.0.0.1:9091/sandboxes/$SB | jq -r .uds_path)

# Dial the baked toolbox agent on vsock 10023 via the UDS bridge (Firecracker-style)
./toolbox-cli health -uds $UDS                      # → ok=true version=0.1.0
./toolbox-cli exec   -uds $UDS -- sh -c 'echo hi'   # → hi

# Re-IP the guest's eth0 in one RPC (used by the warm-pool slice to hand out a
# restored VM with a fresh lease subnet without the caller doing DHCP gymnastics)
./toolbox-cli setnetwork -uds $UDS \
  -ip 10.100.42.2/30 -gw 10.100.42.1 -mac 02:42:00:00:2a:02   # → 11-13 ms

curl -sS -X DELETE http://127.0.0.1:9091/sandboxes/$SB        # → 204

The toolbox agent binary is baked into every disk gocracker builds (/opt/gocracker/toolbox/toolboxguest, via go:embed in internal/toolbox/embed/) and spawned by internal/guest/init.go post-switch_root before the user's CMD, so there is no post-boot install race — the v2 architectural cul-de-sac of runtime.Exec → base64 upload → spawn → EnsureToolbox-on-lease doesn't apply here. The framed /exec data plane ([channel][len][payload], stdin/stdout/stderr/exit/signal) lives on vsock 10023, coexisting with the existing internal/guestexec JSON agent on 10022 — nothing in the current /vms/{id}/exec surface regressed.

Platform Support

Platform	Status	Tested On
x86-64 Linux	Full support	Ubuntu 22.04, 24.04
ARM64 Linux	Full support	AWS a1.metal (Graviton 1), Ubuntu 24.04
macOS	Planned	Hypervisor.framework

Networking

One-command networking

sudo ./gocracker run --net auto --image nginx:alpine --kernel "$KERNEL" --wait

Automatic TAP creation, IPv4 assignment, and NAT. The guest gets internet access with no manual setup.

Compose networking

Each Compose stack gets an isolated Linux network namespace. Services within a stack communicate by service name. Port publishing maps host ports to guest ports:

ports:
  - "18081:8080"   # host:18081 -> guest:8080

How it works

Guest VM
  |
virtio-net
  |
TAP device
  |
bridge / NAT (iptables)
  |
Host network

How It Works

Pull -- Fetch an OCI image from any registry (or build a Dockerfile, or clone a repo)
Extract -- Unpack layers into an ext4 disk image (pure Go, no mkfs.ext4)
Initrd -- Generate an initrd with an embedded init binary (pure Go, no shell tools)
Create VM -- Open /dev/kvm, configure vCPU, memory, and virtio MMIO devices
Boot -- Load the kernel, attach the disk and network, start the vCPU run loop
Guest init -- The init process mounts the disk, sets up the environment, and runs the user command

The device model uses virtio MMIO transport, the same approach Firecracker uses. Each device (net, blk, rng, vsock, balloon, fs) is memory-mapped and interrupt-driven.

gocracker vs Firecracker

	Firecracker	gocracker
Language	Rust	Go
OCI image support	No	Yes
Dockerfile support	No	Yes
Docker Compose	No	Yes
ARM64	Yes	Yes
API	REST	REST (Firecracker-compatible + extensions)
Jailer	Yes	Yes
Seccomp	Yes	Yes
virtio devices	net, blk, balloon, vsock	net, blk, rng, vsock, balloon, fs
Snapshots	Yes	Yes
Live migration	No	Yes

gocracker builds on Firecracker's proven security model and adds the developer experience layer: pull an image, boot a VM, one command.

Boot-time benchmark

End-to-end wall clock for gocracker run --image alpine:3.20 --wait --cmd "echo OK", measured from process start to the first stdout byte of the user CMD. Warm OCI artifact cache (so the run pays only VMM setup + guest boot + first output, not OCI pull or ext4 build).

Host: AMD Ryzen AI 9 HX 370, Linux 6.17, /dev/kvm available. Guest: 1 vCPU, 128 MiB RAM, alpine 3.20, jailer=off. 10 samples per cell after 1 warmup.

kernel	network	p50	p90	max
standard	`--net none`	77 ms	86 ms	108 ms
standard	`--net auto` (tap)	89 ms	90 ms	91 ms
minimal	`--net none`	77 ms	83 ms	93 ms
minimal	`--net auto` (tap)	79 ms	81 ms	84 ms

standard is the default guest kernel (artifacts/kernels/gocracker-guest-standard-vmlinux) — works against any OCI image the user might throw at it. minimal (artifacts/kernels/gocracker-guest-minimal-vmlinux) trims subsystems the microVM path never uses (ACPI NUMA, USB, hibernation, bzip2/lzma/lzo decompressors, etc.); virtio + vsock + DNS + IPv6 + TLS still work — apk update inside Alpine boots green.

Reproduce

make build kernel-unpack
KERNEL=$PWD/artifacts/kernels/gocracker-guest-standard-vmlinux
for i in {1..10}; do
  sudo ./gocracker run \
    --image alpine:3.20 --kernel "$KERNEL" \
    --mem 128 --cpus 1 --jailer off --net none \
    --wait --cmd "echo OK" 2>&1 | grep -oP 'duration=\K[0-9]+ms'
done

The duration=… line in [container:INFO] started … is the end-to-end measurement the runtime reports. It breaks down into orchestration_ms (rootfs/initrd/disk reuse), vmm_setup_ms (KVM create + kernel load), and guest_first_output_ms (KVM_RUN → first serial byte from the guest CMD).

Time-to-Interactive benchmark

The Firecracker head-to-head above measures at the VMM level. Time-to-Interactive (TTI) is the higher-level number most sandbox callers actually care about: the wall-clock time from sandbox.create() returning a handle to the first successful stdout byte of runCommand("node -v"), against a pre-built sandbox image.

tools/bench-node-tti.sh in this repo runs that test: a node:20-alpine Dockerfile with CMD ["node","-v"], warm artifact cache, and gocracker run ... --wait timed to the first stdout byte starting with v.

Setup

Host: AMD Ryzen AI 9 HX 370 (24 threads), Linux 6.17, /dev/kvm available
Guest kernel: artifacts/kernels/gocracker-guest-minimal-vmlinux (Linux 6.1.102, trimmed initcalls, loglevel=4 default)
Image: node:20-alpine, CMD ["node","-v"]
Warm artifact cache: first iteration pulls/extracts once, subsequent timed runs boot from the cached ext4
Rootfs mode: default (read-only rootfs with tmpfs overlay) — matches Docker's ephemeral-writable-layer semantics and enables the hardlink fast-path in pkg/container/container.go

Two paths the bench exposes

The same script runs in two modes — they answer different questions:

sudo ./tools/bench-node-tti.sh 10              # cold-CLI (--dockerfile)
sudo WARM=1 ./tools/bench-node-tti.sh 10       # snapshot-pool (--image, --warm)

Mode	What it measures	median	p95	min
cold-CLI	true cold create: OCI artifact cache hit, fresh kernel boot, fresh init, exec `node -v`	231 ms	239 ms	227 ms
WARM=1 snapshot-pool	what Daytona / Vercel / E2B actually publish: pre-warmed snapshot of the same image, MAP_PRIVATE COW restore, exec `node -v` against the restored guest	92 ms	100 ms	88 ms

The WARM mode auto-captures a snapshot on the first iteration of a fresh image (~9 s warmup) and restores from it on every subsequent invocation (~2 ms restore primitive + ~50 ms node -v startup in the guest + ~12 ms host process startup + ~28 ms in misc dial / framing / log / scheduler overhead).

Both modes pin gocracker to CPU 0 with taskset -c 0 and SCHED_FIFO priority 50 via chrt (override with TTI_PIN_CPUS="0-3" or ""). The VMM process itself calls mlockall(MCL_CURRENT) in cmd/gocracker-vmm/main.go so its working set stays resident.

The 100 ms WARM p95 sits at or below Daytona's published 100 ms median on the same node -v methodology — see www.computesdk.com/benchmarks. Hardware is not directly comparable (we measure on a Ryzen laptop, ComputeSDK's leaderboard runs on its own infra); ranking is meaningful, exact margin would shift on a level-playing-field rerun.

The cold-CLI 231 ms is what users see on a first-of-its-kind image with no warm pool. That's the path that's hardware-floored: ~12 ms host process startup, ~3 ms KVM/memfd setup, ~41 ms guest kernel boot to first printk, ~50 ms guest init → exec → node -v first byte, plus ~125 ms of structural overhead the kernel boot path can't skip without dropping initramfs / using virtio-pci hotplug / similar architectural changes.

Sandboxd (sandbox control plane)

gocracker-sandboxd is a separate daemon that turns the runtime into a hosted sandbox platform. It exposes a small HTTP API on top of gocracker that does four things the raw runtime doesn't:

Warm pools — register a template, and sandboxd keeps N paused VMs ready. A POST /sandboxes/lease returns a handle in single-digit milliseconds because the VM is already booted, the IP is already baked into the snapshot, and the exec agent is already listening on vsock.
Content-addressed templates — register a Dockerfile or image once, sandboxd cold-boots it, takes a warm snapshot, and indexes it by SpecHash. Subsequent CreateTemplate calls with the same spec are a no-op (microsecond cache hit). LeaseSandbox against a pool backed by that template restores from the snapshot.
Signed preview URLs — sb.preview_url(8080) mints an HMAC-signed token that lets callers hit a guest-side port via a dedicated proxy path (/previews/<token>/...) or one-label subdomain (<id>.<preview-host>/...).
Namespaced SDKs in Python / Go / JS — with client.create_sandbox(template="base-python") as sb: sb.process.exec("python -c '…'"); sb.fs.read_file("/tmp/x"); sb.preview_url(8080). Context managers, typed errors (ProcessExitError, PoolExhausted, TemplateNotFound, RuntimeUnreachable, SandboxTimeout), and zero third-party runtime deps in each SDK.

The runtime does VM lifecycle. The sandbox daemon does pool lifecycle, template caching, preview auth, and per-sandbox UDS routing. They're separate processes talking over the runtime's HTTP API, so either one can be swapped out.

Reproduce the numbers below with the scripts in sandboxes/examples/python/bench/ (Python), sandboxes/examples/go/bench/ (Go), and sandboxes/examples/js/bench/ (Node). All were measured on a Ryzen AI 9 HX 370 laptop, Linux 6.17, /dev/kvm available, feat/sandboxd-v3-arm64-and-cleanup head.

Warm-lease latency per SDK

lease → process.exec("echo hi") → delete against a pool of 8 paused VMs, 8 sequential samples per SDK (all times in ms):

SDK	phase	min	p50	p95	max
Python	`lease_sandbox`	0.72	1.09	40.17	40.17
Python	`process.exec("echo")`	12.52	18.86	21.53	21.53
Python	`delete`	0.70	1.06	2.42	2.42
Go	`LeaseSandbox`	0.45	0.93	103.94¹	103.94
Go	`Process().Exec`	12.59	20.48	22.76	22.76
Go	`Delete`	0.40	0.55	0.61	0.61
Node	`leaseSandbox`	3.46	5.14	16.78	16.78
Node	`process.exec`	24.88	41.03	84.94	84.94
Node	`delete`	1.09	2.70	5.80	5.80

¹ One Go p95 outlier of 103 ms while the refiller was concurrently catching up after the previous sample drained the pool. The other 7/8 samples were ≤ 2 ms. Node's exec is slower than Python/Go because the JS SDK currently doesn't connection-pool the UDS bridge — every call pays the CONNECT handshake from scratch.

Time-to-Interactive (via SDK)

Wall-clock from sandbox.create() returning a handle → first stdout byte of runCommand("node", "-v"). The pre-built sandbox image is our base-node template (auto-registered when sandboxd starts with -kernel-path).

10 timed runs after one warmup, against a pool of 8 paused base-node VMs:

median = 277 ms   mean = 274 ms   p95 = 314 ms   min = 206 ms   max = 314 ms

Per-workload breakdown (same path, different command):

workload	p50	what dominates
`sb.process.exec(['/bin/true'])`	~20 ms	UDS + CONNECT round trip + agent fork
`sb.process.exec(['echo','hi'])`	~19 ms	same as `/bin/true` within noise
`sb.process.exec(['node','-v'])`	277 ms	node startup on alpine/musl (~258 ms of the total)

The 258 ms node-startup floor is inside the guest process — our boot path and agent overhead account for ~19 ms of the total. A post-ready snapshot (node already parsed and resident in the page cache) would drop the outer total under 50 ms; the wiring is in tree (ReadinessProbe in the template spec) but has a known issue at restore.

Cookbook + cold-boot bursts (Python SDK)

Numbers from the 10-example cookbook sweep (sandboxes/examples/python/cookbook/):

Workload	p95
`concurrent_cold.py` N=3 (cold-boot each)	142 ms
`pool_burst.py` burst=3 (3 concurrent warm leases)	3.4 ms
`pool_burst.py` burst=5 (5 concurrent warm leases)	6.0 ms
`template_pool.py` (lease from base-python pool)	0.9–2.9 ms

Validated images sweep (50 images)

End-to-end smoke against 50 docker-hub images covering web servers, databases, runtimes, proxies, CLIs, language runtimes, and observability tools. Each image is cold-booted via client.create_sandbox(image=..., entrypoint=['sleep'], cmd=['infinity']) (so PID 1 outlives the agent dial), then sb.process.exec runs a version command and verifies the output. create_ms includes the OCI pull on first run; later runs hit the artifact cache and create drops to ~120–500 ms regardless of image size.

Result: 43/50 PASS (86 %). All 7 failures are caller-side / external (image renamed on DockerHub, port-80 privilege, etc.) — none is a runtime bug. Reproduce with sandboxes/examples/python/bench/sweep_validated.py.

Pass — language runtimes

image	create	exec	delete	output
`alpine:3.20`	126 ms	202 ms	3 ms	`3.20.10`
`debian:12-slim`	4.7 s	217 ms	3 ms	`12.x`
`ubuntu:24.04`	6.3 s	224 ms	3 ms	`24.04`
`amazonlinux:2023`	9.1 s	268 ms	3 ms	`Amazon Linux release 2023`
`rockylinux:9-minimal`	8.4 s	233 ms	3 ms	`Rocky Linux release 9`
`python:3.12-alpine`	156 ms	210 ms	1 ms	`Python 3.12.13`
`node:22-alpine`	2.6 s	112 ms	1 ms	`v22.22.2`
`golang:1.23-alpine`	154 ms	223 ms	2 ms	`go1.23.12 linux/amd64`
`oven/bun:1` (`/usr/local/bin/bun`)	206 ms	323 ms	3 ms	`1.3.13`
`ruby:3-alpine`	11.3 s	201 ms	3 ms	`ruby 3.4.9`
`php:8-cli-alpine`	12.6 s	366 ms	3 ms	`PHP 8.5.5 (cli)`
`elixir:alpine`	11.3 s	1.4 s	3 ms	`Erlang/OTP 28`
`eclipse-temurin:21-jre-alpine`	4.1 s	434 ms	3 ms	`OpenJDK Runtime`
`busybox:latest`	4.6 s	266 ms	3 ms	`BusyBox v1.37.0`
`alpine:3.20 + apk add git`	105 ms	1.7 s	2 ms	`git version 2.45.4`

Pass — web servers / proxies / CLIs

image	create	exec	delete	output
`nginx:alpine`	9.4 s	150 ms	3 ms	`nginx/1.29.8`
`caddy:2-alpine`	8.2 s	303 ms	3 ms	`v2.11.2`
`traefik:latest`	8.3 s	457 ms	3 ms	`Version: 3.6.14`
`httpd:alpine` (`/usr/local/apache2/bin/httpd`)	10.0 s	250 ms	3 ms	`Server version: Apache`
`haproxy:lts-alpine`	8.9 s	293 ms	3 ms	`HAProxy version 3.2.15`
`traefik/whoami:latest`	1.4 s	28 ms	2 ms	`whoami help`
`mailhog/mailhog:latest`	2.0 s	134 ms	3 ms	`MailHog`

Pass — databases / caches / messaging

image	create	exec	delete	output
`redis:alpine`	10.1 s	164 ms	3 ms	`Redis server v=8.6.2`
`postgres:16-alpine`	5.9 s	51 ms	1 ms	`postgres (PostgreSQL) 16.13`
`mariadb:lts`	14.9 s	137 ms	3 ms	`mariadbd Ver 11.8.6`
`cockroachdb/cockroach:latest`	5.8 s	54 s	3 ms	`Build Tag` (cockroach version is heavy)
`influxdb:2-alpine`	15.6 s	409 ms	3 ms	`InfluxDB v2.8.0`
`nats:alpine`	7.2 s	173 ms	6 ms	`nats-server v2.12.7`
`eclipse-mosquitto:2`	6.4 s	133 ms	3 ms	`mosquitto version 2.1.2`

Pass — observability + Hashicorp + dev tools

image	create	exec	delete	output
`prom/prometheus:latest`	5.7 s	469 ms	3 ms	`prometheus, version`
`prom/alertmanager:latest`	3.0 s	297 ms	3 ms	`alertmanager, version`
`prom/node-exporter:latest`	2.0 s	274 ms	3 ms	`node_exporter`
`prom/blackbox-exporter:latest`	3.0 s	290 ms	4 ms	`blackbox_exporter`
`jaegertracing/all-in-one:latest`	2.6 s	361 ms	3 ms	`jaeger help`
`linuxserver/syslog-ng:latest`	3.8 s	64 ms	2 ms	`syslog-ng 4.7.1`
`hashicorp/vault:latest`	22.8 s	679 ms	3 ms	`Vault v2.0.0`
`hashicorp/consul:latest`	12.9 s	729 ms	3 ms	`Consul v1.22.6`
`gitea/gitea:latest`	3.4 s	813 ms	3 ms	`gitea version 1.26.0`
`jenkins/jenkins:lts-jdk21`	9.0 s	227 ms	3 ms	`OpenJDK Runtime`

Fail (7 images — none is a gocracker runtime bug)

image	failure	category
`pocketbase/pocketbase:latest`	404 on docker hub	image renamed
`etcd:latest`	404 on docker hub	image renamed (now `gcr.io/etcd-development/etcd` or `quay.io/coreos/etcd`)
`vault:latest` (legacy)	404 on docker hub	image renamed (now `hashicorp/vault`)
`hashicorp/http-echo:latest`	timed out connecting to agent	minimal scratch-based image — toolbox can't supervise
`coredns/coredns:latest`	timed out connecting to agent	minimal scratch-based image — toolbox can't supervise
`gotify/server:latest`	exit=1 — `listen tcp :80: permission denied`	guest tries to bind privileged port without root
`memcached:alpine`	`KVM_CREATE_VM` failed	host KVM exhaustion (76+ leaked VMs from earlier sessions on the same host)

The runtime handles 43 distinct images, 13 language runtimes (incl. bun), 5 databases, 4 proxies/load balancers, 6 messaging / observability tools, and 5 OS bases without code changes — all via a single client.create_sandbox(image=...) call from the SDK.

Calling-convention tip: for service images that ship their own ENTRYPOINT (most non-base images: nginx, prom/, hashicorp/, jenkins, bun, etc.), pass entrypoint=['sleep'], cmd=['infinity'] so init runs a benign keep-alive instead of the image's daemon. Skipping this is the main cause of the "agent timed out" / "exit=-1" symptoms in the few-failure column above.

Snapshot / restore

POST /restore {resume:true} maps the memory snapshot MAP_PRIVATE (lazy COW — no up-front read or copy), re-wires virtio devices, restores vCPU state, and resumes. Because memory is page-faulted in lazily rather than eagerly copied, restore latency is O(1) in memory size rather than O(memory size).

Key design points that keep the restore fast (bench-rtt numbers below):

MAP_PRIVATE COW memory restore — internal/kvm/kvm.go. The snapshot file is mapped lazily; pages fault in as the guest touches them rather than up-front.
Single-call /restore {resume:true} — internal/vmmserver/server.go. Snapshot-load and vCPU-resume in one HTTP round-trip, not two.
SkipDiscardProbe on restore — internal/virtio/blk.go. Skips the FALLOC_FL_PUNCH_HOLE probe because the guest already negotiated VIRTIO_BLK_F_DISCARD against the pre-snapshot DeviceFeatures.
postCreateVCPUs in restore path — pkg/vmm/vmm.go. Wires per-device KVM_IRQFD against freshly-created vCPU GSIs so queue notifies don't take a reconfiguration VMexit on first use.
MADV_HUGEPAGE + MADV_WILLNEED on the first 8 MiB of the memory mapping — serves hot early pages from 2 MiB TLB entries without page-fault stalls.

Hot-path numbers (sandboxd)

Measurements from the end-to-end sandboxd path on the same host:

scenario	latency	notes
Cold boot (`gocracker run` to first `node -v` stdout)	253 ms p50	+35 ms over the pre-toolbox baseline — cost of init spawning the baked toolbox agent before the user CMD.
Snapshot resume (runtime call only)	1.67 ms p50 / 3.08 ms p90	`RestoreFromSnapshotWithOptions → vm.Start`. See bench-rtt below for full distribution.
Warm-pool lease (`POST /sandboxes/lease`)	1.5 ms p95	Pool of 8, sequential. The pool reuses the IP baked into each VM's cold-boot snapshot and skips SetNetwork on the hot path.
Sandboxd E2E `create → exec echo → delete`	~35 ms p95	Full round trip from the Python/Go/Node SDK. Async delete lands the HTTP response in 1.9 ms; the ~22 ms residual is UDS+vsock CONNECT + agent fork/exec for `echo`.
Sandboxd warm TTI (`node -v` against `base-node`)	277 ms p50	Warm pool of 8, lease handle then time `sb.process.exec(['node','-v'])` to first byte. ~258 ms is node's own startup on alpine/musl; agent + UDS overhead is ~20 ms.

The bench runs with the VMM pinned to a dedicated CPU (taskset) and boosted to SCHED_FIFO (chrt -r 50), with mlockall(MCL_CURRENT) locking the VMM's working set. The snapshot path deliberately does NOT set MCL_FUTURE because that would eager-fault the MAP_PRIVATE memory snapshot and turn a ~2 ms lazy-COW restore into ~30 ms.

ARM64 warm-cache benchmark

gocracker run ... --warm captures a snapshot on first run and restores from it on every subsequent run with identical parameters. The ARM64 port round-trips the full vCPU register set (including SP_EL1, ELR_EL1, SPSR[0], V0-V31 FPSIMD, FPSR/FPCR, MPState, KVM_REG_ARM_TIMER_), plus the entire VGICv3 state (distributor + per-vCPU redistributor + ICC_ CPU sysregs, with IC-then-IS ordered writes), so multi-vCPU guests resume with IRQ delivery and scheduler state intact — not just vCPU 0.

Measured on a1.metal Graviton, Ubuntu 24.04, Alpine 3.20 guest, --net auto --cmd nproc, wall-clock end-to-end and the pure restoreFromSnapshot → vm.Start() phase (from the [container] restored duration=... log):

vCPUs	cold boot	warm restore (total)	pure restore
1	5.28 s	293 ms	87 ms
2	5.31 s	288 ms	82 ms
4	5.24 s	310 ms	105 ms
8	5.28 s	290 ms	96 ms
16	5.41 s	330 ms	117 ms

Cold-boot time here is dominated by OCI pull + ext4 build (~3 s for the alpine rootfs); the actual kernel-to-init boot is ~150 ms. Warm restore stays ~linear in vCPU count because VGIC state is O(vCPU) (one redistributor frame + 9 ICC sysregs each).

Per-primitive RTT benchmark (tools/bench-rtt)

Unlike the end-to-end TTI and snapshot-resume numbers above, bench-rtt isolates each primitive on the warm-cache hot path so a regression can be attributed to a specific code path. Numbers below are from a fresh run on a Ryzen AI 9 HX 370 laptop, Linux 6.17, N=30 iterations after 3 warmups, against the branch head:

primitive	p50	p90	max	notes
pause (`vm.Pause()`)	10.33 ms	10.36 ms	10.44 ms	floor is the 10 ms `time.Sleep` polling loop inside `(*VM).Pause()` — an architectural cost, not measurement noise.
resume (`vm.Resume()`)	~1 µs	~2 µs	~2 µs	near-instant — no vCPU state round-trip.
snapshot capture (`vm.TakeSnapshot()`)	170 ms	268 ms	359 ms	O(memory size) — the cost is writing the 256 MB memory image to disk, bounded by page-cache + device speed.
snapshot restore (`vmm.RestoreFromSnapshotWithOptions → vm.Start`)	1.67 ms	3.08 ms	3.23 ms	`MAP_PRIVATE` COW of the memory snapshot + vCPU state round-trip. O(1) in memory size — no eager page-in.
warmcache lookup hit	9 µs	11 µs	18 µs	`pkg/warmcache.Lookup()` — hashes the spec + stats the snapshot dir.
warmcache lookup miss	4 µs	4 µs	7 µs	early-out when the directory does not exist.
UDS handshake (`CONNECT → OK`)	~0.1 ms	~1 ms	~2 ms	Firecracker-style Unix socket: dial + `CONNECT <port>\n` + read `OK\n`. Measures the host→guest→host virtio-vsock round-trip the sandbox orchestrator pays per exec. Populated when bench-rtt is run with `-uds <path>`.
Toolbox framed exec stdout throughput	50 MB/s sustained	—	—	`POST /exec` on vsock 10023 with `[channel][len][payload]` framing. Validated end-to-end 4 MB round-trips (64 KB 22 ms, 512 KB 31 ms, 1 MB 33 ms, 4 MB 83 ms).
SetNetwork RPC (re-IP post-restore)	~12 ms	~15 ms	~20 ms	Toolbox `POST /internal/setnetwork`: `LinkSetHardwareAddr → LinkSetUp → flush stale v4 addrs → AddrReplace → RouteReplace → arping -U` (async best-effort). Now bypassed on the warm-pool hot path because the IP is baked into the cold-boot snapshot.

Reproduce:

sudo go run ./tools/bench-rtt \
  -kernel ./artifacts/kernels/gocracker-guest-standard-vmlinux \
  -iter 30 -warmups 3

Documentation

Getting Started -- build, install, first VM in 60 seconds
Networking -- auto NAT, compose bridges, manual TAP
Architecture -- boot flow, device model, security layers
Docker Compose -- multi-service stacks, healthchecks, ports
API Reference -- Firecracker-compatible + extended endpoints
CLI Reference -- every command and flag
Snapshots and Migration -- save, restore, live migrate
Examples -- 16 example apps across 10 languages
Validated Projects -- 378 real-world repos tested
Troubleshooting -- common issues and fixes
Security Policy -- isolation model, vulnerability reporting

CLI Reference

gocracker <command> [flags]

Commands:
  run        Build and boot a microVM (image, Dockerfile, or local path)
  repo       Clone a git repo and boot its Dockerfile
  compose    Boot a docker-compose.yml stack as microVMs
  build      Build a disk image without booting
  restore    Restore and boot a VM from a snapshot
  migrate    Live-migrate a VM between gocracker API servers
  serve      Start the REST API server
  snapshot   Take a snapshot of a running VM via the API
  version    Print build version, commit, date, and Go runtime

Common flags (run):
  --image       OCI image ref (e.g. ubuntu:22.04)
  --dockerfile  Path to Dockerfile
  --kernel      Kernel image path [required]
  --mem         RAM in MiB (default: 256)
  --cpus        vCPU count (default: 1)
  --disk        Disk size MiB (default: 2048)
  --net         Network mode: none or auto
  --cmd         Override CMD
  --wait        Block until VM stops
  --tty         Console mode: auto, off, or force
  --jailer      Privilege model: on or off (default: on)

Tested with 378 Real-World Projects

gocracker has been validated against 378 open-source projects across 16 languages, booting each from its own Dockerfile as a microVM:

Category	Projects
Infrastructure	Traefik, Caddy, Envoy, HAProxy, NGINX, CoreDNS
Databases	PostgreSQL, Redis, Valkey, CockroachDB, DragonflyDB, Memcached
Search	Meilisearch, Qdrant, Typesense, Quickwit
Monitoring	Prometheus, Grafana, Loki, Tempo, Telegraf, VictoriaMetrics
ML/AI	MLflow, BentoML, Gradio, Streamlit, Ollama
CI/CD	Woodpecker, Drone, Kestra, Dagster, Prefect
Messaging	RabbitMQ, NATS, EMQX, Mosquitto
Storage	MinIO, SeaweedFS, Distribution (registry)
Forges	Gitea, Gogs
Languages	Go (123), Node.js (62), Python (50), PHP (24), Rust (21), Ruby (12), Java (12), Elixir (6), C (6), C++ (4), and more

Full manifest: tests/external-repos/manifest.tsv (378 entries)

See docs/EXAMPLES.md for included example applications.

Requirements

Linux kernel 5.10+ with KVM (/dev/kvm accessible)
Go 1.22+ (build from source)
Network tools: ip, iptables (for --net auto)
sudo for KVM access and networking

Verify your host is ready:

make hostcheck

Roadmap

#	Idea	Notes
1	Code-disk from object storage (S3 / R2)	Accept `s3://bucket/app-v42.ext4` or `r2://bucket/app-v42.ext4` as a `--code-disk` source. sandboxd downloads and caches the ext4 locally before attaching, keyed by ETag/checksum so the same version is never downloaded twice. Enables a deploy pipeline where CI uploads the code disk to object storage and gocracker pulls it on demand — no shared filesystem required.
2	Shared-kernel mode	One guest VM hosts multiple isolated user workloads via Linux namespaces + seccomp per process. Shared runtime pages (CoW), ~1 ms fork instead of ~40 ms restore, ~5–20 MB per tenant vs ~128 MB today. Tradeoff: namespace isolation instead of KVM hardware isolation — good fit for trusted tenants or high-density deployments.
3	Edge layer / public ingress	Anycast-style routing so sandboxd is reachable from the internet without a manual reverse proxy. TLS termination, per-tenant auth tokens, rate limiting.
4	Hot disk swap	virtio-blk hot-plug so a running VM can swap its code disk without rebooting. Currently requires attaching all disks at boot.
5	ARM64 warm snapshots	Warm pool and snapshot/restore tested on x86-64; needs validation + tuning on Graviton / Apple Silicon (nested KVM).
6	macOS support	Hypervisor.framework backend for local development without Linux.

License

Apache License 2.0

Name		Name	Last commit message	Last commit date
Latest commit History 54 Commits
.github/workflows		.github/workflows
artifacts/kernels		artifacts/kernels
assets/demos		assets/demos
cmd		cmd
docs		docs
examples/code-disk		examples/code-disk
internal		internal
pkg		pkg
sandboxes		sandboxes
site		site
tests		tests
tools		tools
.gitignore		.gitignore
AGENT.md		AGENT.md
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
coverage.out		coverage.out
go.mod		go.mod
go.sum		go.sum
wrangler.jsonc		wrangler.jsonc

Folders and files

Latest commit

History

Repository files navigation