feat(isolate): per-sandbox egress attribution via per-slot service pool

[sumansaurabh]

Summary

• Flips the isolate runtime's egress from fail-closed deny-all (the state PR feat(isolate): V8-isolate runtime (workerd) — Phases 1–3, off by default #302 shipped) to enforced per-sandbox allowlists — the §4 follow-up.
• Root constraint (spike-proven vs real workerd): workerd only accepts a static service binding as a dynamically-loaded worker's globalOutbound. An inline JS-object Fetcher is rejected (not of type 'Fetcher') and a dynamically-loaded worker stub is rejected (Entrypoints to dynamically-loaded workers cannot be transferred). So attribution can't be a controller-stamped header — it must come from which static service the sandbox binds.
• Mechanism: the group config pre-declares a pool of K per-slot egress services (EGRESS_0..), each on its own host UDS. The host assigns each egressing sandbox a slot and returns egress_slot in the bundle spec; the controller sets globalOutbound: env["EGRESS_"+slot]. The loaded worker is given no other binding, so it can reach only its own slot → attribution is structural and unforgeable. Block-all / pool-exhausted sandboxes bind a fail-closed EGRESS_DENY (spill is logged, never silent). workerd dials external services lazily, so the pool is cheap config and the host binds a slot socket only on assignment (runtime cost ∝ egressing sandboxes, not K). SB_ISOLATE_EGRESS_POOL_SIZE (default 64).
• Off by default — isolate runtime is still SB_ENABLE_ISOLATE=false; this changes nothing until an operator opts in.

Code-path diagram

flowchart TB
  subgraph create["driver.Create (isolate)"]
    L["host.Load(id, bundle)"] --> SEP["SetEgressPolicy(id, policy)"]
    SEP --> BA{"policy.BlockAll?"}
    BA -->|yes| NoSlot["free any slot; id binds EGRESS_DENY"]
    BA -->|no| Free{"free slot in pool?"}
    Free -->|yes| Assign["assign slot; bind egress-<slot>.sock listener (lazy); slotByID[id]=slot"]
    Free -->|no| Spill["log pool-exhausted; id binds EGRESS_DENY (deny-all fallback)"]
  end
  subgraph outbound["isolate fetch() → globalOutbound"]
    Spec["controller reads egress_slot from bundle spec"] --> Bind{"slot present?"}
    Bind -->|yes| G0["globalOutbound = env[EGRESS_slot] → egress-<slot>.sock"]
    Bind -->|no| GD["globalOutbound = EGRESS_DENY → 403"]
    G0 --> Handler["serveEgressSlot(slot): id = idBySlot[slot] (SOCKET is the identity)"]
    Handler --> Pol{"egressAllowed(policy, host)?"}
    Pol -->|no| D1["403 denied by policy"]
    Pol -->|yes| SSRF{"host IP in blocked range? (loopback/link-local/RFC1918/metadata)"}
    SSRF -->|yes| D2["403 blocked address"]
    SSRF -->|no| RP["reconstruct URL from Host header, force https, RoundTrip upstream"]
  end
  subgraph teardown["host.Unload(id)"]
    U["free slot: stop listener, remove socket, idBySlot[slot]='' "]
  end

Loading

Before: every isolate's globalOutbound was the single static EGRESS service; the Go proxy saw no attribution and 403'd everything (deny-all). After: per-slot service + socket-scoped handler enforces the sandbox's real allowlist.

Sandbox boot impact

Isolate-only; does not touch the docker/firecracker/wasm/containerd boot paths. Added to driver.Create (isolate) per-create, all called out:

• SetEgressPolicy (already called after Load) now also does an O(K) scan for a free slot and, on the first egress-policy set for a non-block-all sandbox, one net.Listen("unix", …) to bind the slot socket. Both are local, in-memory / local-UDS — no DB, no network round-trip, no lock beyond the host's existing mu. Bounded by K (default 64). Block-all sandboxes skip slot assignment entirely.
• No change to non-isolate runtimes.

Idempotency

• SetEgressPolicy(id, p) is idempotent: re-setting replaces the stored policy; if the sandbox already holds a slot it is kept (no churn), so retried/duplicate creates don't leak slots or listeners. Switching to block-all releases the slot; switching back re-acquires one.
• The controller's provider is keyed by sandbox id and re-reads egress_slot on each (cache-miss) load, so an idle-evict → reload re-binds the same slot the sandbox still holds.
• Unload(id) frees the slot and removes its socket; a second Unload is a no-op. Create/Destroy idempotency is otherwise unchanged from PR feat(isolate): V8-isolate runtime (workerd) — Phases 1–3, off by default #302.
• Bundle upload / create idempotency: unchanged.

Failure-path consistency

• No new multi-step caddy+store write — egress is entirely host-side (workerd process + Go proxy), not persisted.
• startSlotServerLocked (bind the slot listener) failing → the sandbox is not assigned a slot and falls back to EGRESS_DENY (deny-all), logged; no partial slot state is left (the maps are only updated after a successful bind).
• Pool exhaustion → same fail-closed deny + log. Unload and stopServers close every per-slot listener and remove sockets; stopServers takes mu so it can't race a concurrent slot assign.
• Intra-tenant slot-reuse-on-destroy is a documented narrow minor (a freed slot's socket is unbound until reassigned, so a lingering outbound from a destroyed sandbox hits a closed socket = denied); cross-tenant isolation is the group-process boundary, unaffected.

L4 / host-port-pool changes

N/A — neither touched. Isolate is HTTP-only and host-mediated; it never calls TryReserveHostPort / allocateHostPort / EnsureLayer4 / the l4Ready latch. The egress "pool" here is a set of per-group unix sockets, wholly separate from the TCP host-port pool.

Test plan

• Offline make test: green (full tree).
• Spikes vs real workerd (2026-07-17 binary): confirmed JS-object globalOutbound rejected; per-slot static-service pool works + isolates by socket; workerd dials external services lazily.
• New/updated unit tests (pkg/isolate): slot-lifecycle (assign / block-all-no-slot / pool-exhaust-no-share / Unload-frees-and-reuses), proxyEgress policy-deny + SSRF-literal block + no-host, serveEgressSlot fail-closed, pooled capnpConfig + EGRESS_DENY wiring, bundleWireJSON.egress_slot.
• Integration vs real workerd (-tags=integration): TestPhase3PerSandboxEgressEndToEnd (allow-listed sandbox reaches allowed host, refused non-allowed; block-all sandbox in the SAME group refused) + existing P0 / ingress / exec / idle-reaper suite — all green.
• Live on AWS (fresh isolated t3.medium, torn down — 36 resources destroyed, no leak): per-sandbox proof allow→example.com 200, allow→not-allowed 403, block→example.com 403; SSRF proof on real EC2 allow-all → 169.254.169.254 IAM creds 403, → loopback sandboxd API 403, → example.com 200.

Known limitations (deliberate)

These are conscious trade-offs, not bugs. Each is safe-by-default and has a clear path forward.

1. There's a ceiling on how many sandboxes can do egress at once (per tenant)

In plain terms: For the host to know whose outbound traffic is whose, every sandbox that's allowed to make network calls gets its own private "lane" — a numbered egress slot backed by a dedicated unix socket. There is a fixed number of these lanes per tenant's workerd process, set by SB_ISOLATE_EGRESS_POOL_SIZE (default 64).

Why it exists: workerd requires these lanes (globalOutbound service bindings) to be declared up front, when the tenant's workerd process starts. We can't mint a new one on demand for a sandbox that appears later, so the count is fixed at process start.

What happens at the ceiling: If more than 64 sandboxes in the same tenant are simultaneously allowed egress, the next one can't get a lane and falls back to no egress at all (deny-all) — the safe direction. This never happens silently: it's logged as isolate egress pool exhausted; sandbox falls back to deny-all egress. Sandboxes that are block-all (no egress) cost zero lanes, so only egress-enabled sandboxes count against the ceiling.

What to do: 64 concurrent egress-enabled isolates in one tenant is already a lot; raise SB_ISOLATE_EGRESS_POOL_SIZE if a tenant legitimately needs more. The cost is tiny — the lanes are just config until used, and the host only opens a socket for a lane once a sandbox actually claims it. A future Phase-4 change can raise the effective ceiling further by letting sandboxes that share the identical allowlist share one lane.

2. Outbound calls are always made over HTTPS

In plain terms: When a sandbox calls fetch("http://…") or fetch("https://…"), our host proxy always connects to the destination over HTTPS, regardless of which the sandbox wrote.

Why it exists: When workerd forwards a sandbox's outbound request to our proxy, it drops the http vs https part — an http:// and an https:// request arrive looking identical (confirmed by a spike against real workerd). Since we can't tell which was intended, we pick the safe one.

What this means for users: Egress to normal HTTPS endpoints (APIs, webhooks, cloud services — essentially all modern egress) works exactly as expected. A destination that is plaintext-HTTP only won't be reachable, because we upgrade the connection to HTTPS. For a security-oriented egress proxy, defaulting to HTTPS is the desirable behavior anyway.

3. This PR controls where a sandbox can connect, not how much it can transfer

In plain terms: This change enforces the allowlist — which hosts a sandbox may reach. It does not yet meter or cap the number of bytes a sandbox sends or receives.

Why it exists: Scope. This PR is specifically the "turn egress from fully-off (deny-all) to on-with-per-sandbox-allowlists" flip. Byte accounting and per-sandbox bandwidth quotas are separate plumbing.

What this means: You can restrict destinations today; you cannot yet cap bandwidth per sandbox for the isolate runtime. (The Sandbox model already carries network_bytes_* fields, but they aren't wired to isolate egress yet.) This is a tracked Phase-4 follow-up.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	microvm	da6ed5f	Commit Preview URL Branch Preview URL	Jul 18 2026, 07:21 AM

[codecov]

Codecov Report

❌ Patch coverage is 85.07463% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/isolate/egress.go	77.46%	12 Missing and 4 partials ⚠️
pkg/isolate/host.go	90.47%	2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!