Skip to content

fix(docker): ready-socket race fallback + cluster-gated gVisor --host-uds#272

Open
sumansaurabh-slice wants to merge 1 commit into
mainfrom
fix/docker-ready-socket-race-and-host-uds-optin
Open

fix(docker): ready-socket race fallback + cluster-gated gVisor --host-uds#272
sumansaurabh-slice wants to merge 1 commit into
mainfrom
fix/docker-ready-socket-race-and-host-uds-optin

Conversation

@sumansaurabh-slice

@sumansaurabh-slice sumansaurabh-slice commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two follow-ups to the #271 Docker push-readiness socket — a real race-fallback
bug, plus correctly scoping the gVisor isolation relaxation #271 introduced.

  • Race robustness (pkg/docker/client.go). waitForToolboxReady took the
    first race result including errors, so a socket-path error — e.g. an
    in-container process tripping maxInvalidReadyAttempts, or any listener
    error — won the race and failed an otherwise-healthy create. It now takes the
    first path that succeeds, so the health poll is a true backstop for a
    silent or erroring socket; the health error (the actual readiness
    probe) is surfaced when both fail.
  • gVisor --host-uds=open is now scoped to cluster nodes. feat(docker): unix-socket push readiness for cluster creates #271 added it to
    every --with-gvisor install, including standalone / non-cluster hosts
    where the cluster-only ready socket never runs (DockerReadySocketEffective() = enabled && EnableCluster) — relaxing gVisor host-UDS isolation fleet-wide
    with zero benefit there. install.sh runs before the node is in a
    cluster and can't self-gate (SB_ENABLE_CLUSTER is set later by
    cluster-init.sh / cluster-join.sh), so the relaxation moves into those
    scripts via ensure_gvisor_host_uds — the one place that authoritatively
    knows the host is a cluster node. install.sh reverts to plain runsc.

Why cluster-init/join, not an install.sh flag

The only signal that "this host is in a cluster" is SB_ENABLE_CLUSTER=true,
which is written by cluster-init.sh / cluster-join.sh, after install.sh
has already registered the runsc runtime. The Terraform bootstrap always runs
install.sh → then cluster-init/cluster-join, so every cluster node passes
through one of those scripts. ensure_gvisor_host_uds:

  • appends --host-uds=open to the registered runsc runtimeArgs,
    preserving path and any existing args (e.g. --platform=systrap);
  • is a no-op when gVisor isn't registered (firecracker/wasm-only nodes) and
    idempotent — it checks for the arg semantically (not by byte-diffing
    reformatted JSON), so a re-run never bounces Docker;
  • is best-effort: a jq/python3-absent host or a failed docker restart logs a
    warning and degrades the socket to health polling (UC-97) rather than failing
    the cluster join.

Net effect: host-UDS isolation is relaxed iff (cluster mode) and
(gVisor present) — exactly where the readiness socket runs. Standalone gVisor
hosts keep runsc at the safe --host-uds=none. No new Terraform variable is
needed (the bootstrap already runs cluster-init/join on every node).

Sandbox boot impact

Touches the create path (waitForToolboxReady), but adds no new DB query,
HTTP round-trip, file I/O, or lock acquisition.

  • Normal case — no change. The first success still wins immediately
    (socket push, or socket-silent + health). Same latency as before.
  • Pathological case — bounded increase, intentional. When the socket path
    errors, the create previously failed instantly; it now waits for the health
    poll to resolve (success or the existing toolboxWaitTimeout, default 30s —
    the same bound the health poll already used). This is the fix: a
    socket-channel error must not fail a healthy sandbox.
  • cluster-init.sh / cluster-join.sh changes are provisioning-time only, not
    boot path.

Idempotency

  • The race fix doesn't alter the per-create nonce-keyed socket path or its
    retry/cleanup semantics; it only changes which race result is authoritative.
  • ensure_gvisor_host_uds is idempotent by a semantic presence check
    (--host-uds=open already in runtimeArgs → no write, no docker restart), so
    re-running cluster-init/join is safe and won't kick running containers.

Failure-path consistency

N/A — no multi-step caddy+store write path changed. The only added create-path
teardown is the ready-listener Close(), already covered by the existing
defer on every create error path. ensure_gvisor_host_uds leaves
daemon.json untouched on any error.

Cluster-correctness impact

ensure_gvisor_host_uds lives in the cluster bootstrap scripts but is a host
docker-config concern, not the Raft/gossip/placement path — no FSM, replication,
owner-watcher, or capacity-lease behavior changes. It runs once at init/join,
before sandboxd is (re)started into cluster mode. No split-brain / replay /
leader-change surface. Single-node (EnableCluster=false) is unaffected:
install.sh leaves runsc at --host-uds=none and the socket path stays off.

L4 / host-port-pool changes

N/A — neither touched.

Test plan

  • Added TestCreate_SocketErrorFallsBackToHealth: spams invalid frames to trip
    the listener's error path while the container is healthy-but-slow, asserts the
    create still succeeds via health. Verified it fails against the pre-fix
    (feat(docker): unix-socket push readiness for cluster creates #271) client.go
    with ready socket: too many invalid attempts, and
    passes post-fix.
  • pkg/docker socket suite (incl. the new test) green on real Linux under
    -race (-count repeated); deterministic, no flake. go build ./...,
    gofmt, full TestCreate* run clean on darwin.
  • ensure_gvisor_host_uds logic exercised against six daemon.json fixtures
    (no runsc / runsc-no-args / runsc-with-other-arg / already-has-host-uds /
    no-runtimes-key / extra-top-keys) on both the jq and python3 paths —
    identical, correct output; path + existing args preserved; idempotent no-op
    when already present or gVisor absent. bash -n clean on all three scripts.

🤖 Generated with Claude Code

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 30, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
microvm cd01a29 Commit Preview URL

Branch Preview URL
Jun 30 2026, 02:24 PM

@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.19048% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/docker/client.go 76.19% 4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

…-uds

Two follow-ups to the #271 push-readiness socket.

1. Race robustness (pkg/docker/client.go). waitForToolboxReady took the
   *first* race result including errors, so a socket-path error (e.g. an
   in-container process tripping maxInvalidReadyAttempts, or any listener
   error) won the race and failed an otherwise-healthy create. It now takes
   the first path that SUCCEEDS, making the health poll a true backstop for a
   silent OR erroring socket; the health error is surfaced when both fail.
   Regression test TestCreate_SocketErrorFallsBackToHealth fails pre-fix,
   passes post-fix.

2. gVisor --host-uds=open is now scoped to cluster nodes. #271 added it to
   every --with-gvisor install, including standalone hosts where the
   cluster-only ready socket never runs — relaxing gVisor host-UDS isolation
   fleet-wide with zero benefit there. install.sh runs before the node is in a
   cluster and cannot self-gate (SB_ENABLE_CLUSTER is set later by
   cluster-init/join), so the relaxation moves to ensure_gvisor_host_uds in
   cluster-init.sh and cluster-join.sh — the one place that knows the node is
   a cluster node. It appends --host-uds=open to the registered runsc
   runtimeArgs idempotently (no-op without gVisor or when already present;
   never fails the join). install.sh reverts to plain runsc, and the
   speculative Terraform with_gvisor_host_uds plumbing is dropped (the
   Terraform bootstrap always runs cluster-init/join, so cluster gVisor nodes
   get it automatically).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sumansaurabh-slice sumansaurabh-slice force-pushed the fix/docker-ready-socket-race-and-host-uds-optin branch from d27485c to cd01a29 Compare June 30, 2026 14:23
@sumansaurabh-slice sumansaurabh-slice changed the title fix(docker): ready-socket race fallback + opt-in gVisor --host-uds fix(docker): ready-socket race fallback + cluster-gated gVisor --host-uds Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants