feat(isolate): V8-isolate runtime (workerd) — Phases 1–3, off by default

[sumansaurabh-slice]

Summary

• Adds the V8-isolate runtime (runtime: "isolate"), the 5th AerolVM runtime, backed by Cloudflare workerd (Workers model). Host-mediated like WASM: no IP/TAP/iptables — one workerd OS process per isolate group, sandboxes loaded dynamically by id via a controller worker + workerLoader.
• Ships Phases 1–3: enum + tenant_id schema + jail spec (P1); content-addressed JS/TS bundle store + POST /v1/js-bundles upload catalogue + cold create path (P2); group router, inbound HTTP fetch-proxy, exec=invoke, idle-TTL reaper, warm pool, bundle GC, egress boundary (P3). Plus the isolate constant + tenant_id across all 5 SDKs and a docs page.
• Off by default (SB_ENABLE_ISOLATE=false; Terraform/install.sh WITH_ISOLATE=false) — merging activates nothing until an operator opts in.

Review fixes in this PR (pre-merge /review pass)

A full review + real-workerd integration run found and fixed several issues, including one that made the feature non-functional as committed:

• CRITICAL — every isolate 500'd (was committed red): the per-sandbox egress shim wired globalOutbound: <dynamically-loaded worker stub>, which real workerd rejects (not a Fetcher; getEntrypoint() → "entrypoints of dynamically-loaded workers cannot be transferred"). Fixed to globalOutbound: env.EGRESS (static service Fetcher). Consequence: egress is fail-closed deny-all (see Known limitations).
• Tenant isolation: jail was computed but never applied while SB_ISOLATE_USE_JAIL=true claimed confinement → now fail-closed (refuse to spawn when a required jail can't be realized) + honest jail_realizable boot log + Linux realization seam (priv-drop/chroot). Empty tenant_id collapsed all scoped tenants into one "default" process → now a stable per-owner group key.
• Concurrency/correctness: group join↔last-member-teardown race (zombie sandbox) → router member-set under lock; idempotent duplicate create-with-id; host lifecycle data race; egress SSRF IP-range block (loopback/link-local/RFC1918/metadata) at dial time; expose_port listener leak on caddy failure; bundle-GC TOCTOU; cross-tenant blob delete now ref-counted (404 on unowned); file:// refs operator-only; isolate health surfaced in /health; warm-pool refill on daemon ctx; 4 non-compiling SDK doc examples.

Code-path diagram

flowchart TB
  C["POST /v1/sandboxes runtime=isolate"] --> R{"file:// ref & scoped caller?"}
  R -->|yes| Rej["reject: operator-only ref"]
  R -->|no| Res["resolve + stage bundle (Put, GC-pinned)"]
  Res --> Auth["authorizeIsolateTenantID<br/>empty+scoped -> per-owner key"]
  Auth --> Adm["admit"] --> Drv["driver.Create"]
  Drv --> AG["acquireGroup: add id to member set UNDER groupsMu"]
  AG -->|new key| Spawn["spawn workerd (jail: apply or FAIL CLOSED)"]
  AG -->|existing| Join["join group"]
  Spawn --> Load["host.Load (outside lock)"]
  Join --> Load
  Load -->|ok| Store["store.Create"]
  Load -->|fail| Rel["releaseFromGroup: member-set--, teardown iff empty"]
  Store -->|ErrSandboxExists| Idem["idempotent: re-Get, NO rollback of winner"]
  Store -->|other err| Unwind["Destroy + releaseAdmission"]
  Store -->|ok| OK["started"]
  subgraph egress["isolate outbound fetch"]
    EG["globalOutbound = static EGRESS service"] --> Proxy["Go egress proxy"]
    Proxy -->|no x-sb-id| Deny["403 deny-all (Phase-2 posture)"]
    Proxy -->|blocked IP range| Deny
  end

Loading

Sandbox boot impact

Isolate-only path (does not touch docker/fc/wasm boot). Per-create work, all called out:

• One owner-scoped bundle resolve + a Put that is a fast no-op when the digest is already staged (the common case); first stage of a digest writes the index once. No network.
• Group routing is a map lookup + single-flight spawn only on a tenant's first create (workerd spawn ~28ms cold / ~0.7ms warm inject, measured offline). Non-isolate runtimes unaffected.

Idempotency

• Create: ret/concurrent same-id → store.Create PK conflict returns ErrSandboxExists, handled as idempotent success (re-Get; never rolls back the winner's shared driver state/admission). Group membership, host.Load, and admission are all keyed by id and idempotent.
• POST /v1/js-bundles: content-addressed; re-uploading identical bytes → same digest, no error; whole Put under one lock.
• expose_port (HTTP): EnsureHTTPListener returns the existing loopback listener; caddy upsert + store.UpsertPort are idempotent; never walks the TCP host-port pool (isolate rejects tcp/tls).
• Group spawn: per-key single-flight — N concurrent first-creates → one workerd process.

Failure-path consistency

• Create unwind is LIFO: host.Load fail → releaseFromGroup (member-set decremented under lock, group torn down iff it was the last member); store.Create fail (non-conflict) → driver.Destroy + releaseAdmission.
• expose_port HTTP: caddy-upsert failure now releases the loopback listener before returning (no orphan listener/goroutine).
• Bundle GC cannot reap a digest an in-flight create staged (staging pin), nor a digest a live sandbox pins.

L4 / host-port-pool changes

N/A — neither touched. Isolate is HTTP-only and host-mediated; it rejects tcp/tls and never calls TryReserveHostPort / allocateHostPort / EnsureLayer4. The store change is an additive tenant_id TEXT NOT NULL DEFAULT '' column (idempotent migration, provably balanced INSERT/SELECT/scan at 56 columns — verified), not the host-port region.

Known limitations (deliberate, safe-by-default)

• Egress is deny-all. Per-sandbox allowlist attribution needs a workerd-supported mechanism (the loaded-shim globalOutbound is rejected by workerd) — tracked as a §4 follow-up. Until then every isolate outbound fetch is denied (safe).
• Jail is fail-closed, not fully realized. Priv-drop + chroot land on Linux; seccomp BPF + cgroup limits + chroot population + a real-Linux-host integration test are follow-ups. SB_ISOLATE_USE_JAIL=true refuses to spawn where it can't realize the jail, so there is no silent-unconfined state.
• Feature is off by default; not yet a hardened multi-tenant boundary for untrusted code.

Test plan

• Full offline make test green (62 packages); gofmt clean.
• New unit tests: egress matcher + SSRF IP-block, fail-closed jail, deterministic group teardown/join-race regression + -race concurrent churn, owner-key derivation, bundle ref-count delete.
• All isolate integration tests pass against a real workerd binary (TestHostEndToEnd, TestDriverCreateEndToEnd, TestP0HostileIsolateContainment, all TestPhase3*) — including the ones that were red at HEAD before this pass.
• Offline in-process create-latency benchmark harness (pkg/daemon/isolate_bench_integration_test.go): warm p50 ~0.7ms / cold ~28ms on darwin (not t3-comparable; live t3 bench pending).

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	microvm	918cf14	Commit Preview URL Branch Preview URL	Jul 18 2026, 03:24 AM

[codecov]

Codecov Report

❌ Patch coverage is 75.25355% with 244 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/service/isolate.go	41.50%	53 Missing and 9 partials ⚠️
pkg/isolate/host.go	58.10%	57 Missing and 5 partials ⚠️
pkg/jsbundle/store.go	75.19%	17 Missing and 15 partials ⚠️
internal/service/isolate_bundles.go	81.11%	9 Missing and 8 partials ⚠️
internal/runtime/isolate/grouprouter.go	76.36%	8 Missing and 5 partials ⚠️
pkg/jsbundle/build.go	66.66%	6 Missing and 5 partials ⚠️
internal/config/config.go	72.41%	4 Missing and 4 partials ⚠️
internal/runtime/isolate/create.go	71.42%	4 Missing and 4 partials ⚠️
pkg/api/v1/js_bundle.go	77.77%	4 Missing and 2 partials ⚠️
pkg/daemon/daemon.go	28.57%	3 Missing and 2 partials ⚠️
... and 6 more

📢 Thoughts on this report? Let us know!