fix(daemon): the isolation between servers is measured, not assumed - #70
Merged
Conversation
Servers on a node are kept apart by one Docker network option, `com.docker.network.bridge.enable_icc=false`, and it was set only on the creation path. `ensureNetwork` returned the moment it saw a network by the right name, without ever reading its options. So a `hopper0` created by a bare `docker network create`, by an operator following half a runbook, by a Hopper old enough to predate the option, or restored with a machine image, left inter-container traffic **on** — and every server on that node could reach every other server's unpublished ports. `docs/security.md` listed network isolation among the things an operator has nothing to configure because it is simply the default. On such a node it was not true, and nothing said so. The option is now read on both paths, so what the daemon reports is always something Docker was asked rather than something the code assumes because it sent the option once. An absent option reads as open, not as neutral: Docker's default is to allow the traffic, so silence is the dangerous answer. The value is parsed the way Docker's own bridge driver parses it rather than compared to the string this file writes. **A bad network is reported, never repaired, and never fatal.** Both cures are worse than the fault. `enable_icc` cannot be changed on an existing network, so repairing means delete-and-rebuild, which disconnects every container on the machine — and Docker refuses to remove a network while anything is attached, so an unprompted startup repair would fail halfway on exactly the busy node where it matters. Refusing to start protects the isolation and takes the node down to do it: no console, no backups, nothing startable, and a daemon that has exited cannot tell anyone why — the panel shows it as offline, indistinguishable from a dead machine. The loudest possible action producing the quietest possible message. The deciding asymmetry is that the fault's blast radius is between the servers on one node, while either cure's is every server on that node, immediately and unasked. So it is said instead: at every start, again every thirty minutes while it lasts, on `/api/system`, and as a failing `hopper doctor` check — which said nothing about the network at all before this. The repair advice branches on `docker.network.autoCreate`, and that is not a detail. With it false — a documented setting, and the one an operator who hand-provisions networks is likeliest to have chosen, which is exactly the operator this check exists to catch — "remove it and restart hopperd" rebuilds nothing: the daemon finds the network gone and refuses to start. Following the advice would have produced the permanent outage the design refused to cause. `hopper doctor` prints no command for this, because the panel cannot see that setting and the wrong half of the guess is the expensive half; it points at the node's log, where the daemon writes the sequence for its own configuration. Two neighbouring claims were checked rather than assumed, and both already held: containers are attached to the dedicated network on every path, and what Docker publishes is exactly the server's allocations on their own addresses. Both now have a test. The documentation's example of an unpublished port was wrong, though — RCON is modelled as an allocation, so it is published like any other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QL3QL3ReEa9Sk68mxJW6Fu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Servers on a node are kept apart by exactly one thing —
com.docker.network.bridge.enable_icc=falseon the shared bridge — and it was set only on the creation path:A
hopper0created by a baredocker network create, by an operator following half a runbook, by a Hopper old enough to predate the option, or restored with a machine image, left inter-container traffic on. Every server on that node could then reach every other server's unpublished ports.docs/security.mdlisted network isolation among the things an operator has nothing to configure, because it is the default behaviour. On such a node it was not true, and nothing anywhere said so.Measured on both paths
The option is now read after creating as well as after finding, so the verdict is always something Docker was asked rather than something the code assumes because it sent the option once.
An absent option reads as
open, not as neutral — Docker's default is to allow the traffic, so silence is the dangerous answer. The value is parsed the way Docker's own bridge driver parses it (Go'sstrconv.ParseBool) rather than compared to the string this file happens to write. A Docker that cannot be asked answersunknown, never an accusation.Reported, never repaired, never fatal
Both cures are worse than the fault, and the reasoning is written where the next reader will find it.
Recreating is impossible in place —
enable_icccannot be changed on an existing network — so it means delete-and-rebuild, which disconnects every container on the machine. Docker also refuses to remove a network while anything is attached, so an unprompted startup repair would fail halfway on precisely the busy node where it matters.Refusing to start protects the isolation and takes the node down to do it: no console, no backups, nothing startable. Worse, a daemon that has exited cannot tell anyone why — the panel shows it as offline, indistinguishable from a dead machine. The loudest possible action producing the quietest possible message.
The deciding asymmetry: the fault's blast radius is between the servers on one node; either cure's is every server on that node, immediately and unasked.
So it is said instead — at every start, again every thirty minutes while it lasts, on
/api/system, and as a failinghopper doctorcheck. Doctor said nothing about the network before this.The repair advice branches on
autoCreate, and that is not a detailThe review caught this and it is the sharp edge of the whole change. With
docker.network.autoCreate: false— a documented setting, and the one an operator who hand-provisions networks is likeliest to have chosen, which is exactly the population this check exists to catch — "remove it and restart hopperd" rebuilds nothing. The daemon finds the network gone and refuses to start.Following the advice would have produced the permanent outage the design explicitly refused to cause.
The daemon now prints the sequence for its own configuration, including the full
docker network createwhen it will not rebuild one itself.hopper doctorprints no command: it runs in the panel, which cannot see that setting, and the wrong half of the guess is the expensive half — so it names the fault and points at the node's log.Checked rather than assumed
Two neighbouring halves of the same guarantee, both already sound, and the review says so rather than inventing work: containers are attached to the dedicated network on every path (
NetworkModecomes from config, nothing callsnetwork connect), and what Docker publishes is exactly the server's allocations, TCP and UDP, on their own addresses. Both now have a test —enable_icc=falseprotects nothing if the container is not on that network, and Docker's default when the field is dropped isbridge.One documentation error found: RCON was given as an example of a port no other server can reach. Hopper models RCON as an allocation, so it is published like any other and reachable from outside the node.
Verification
hopper0without the option is detected, is not recreated (same network Id), and produces exactly one error line. A correct network, anenable_icc=0network and Docker 29's extra options do not misfire. The reviewer also proved the underlying claim directly — container-to-container traffic is dropped on an ICC-off bridge and flows on a bare one./api/systemwiring between detection andhopper doctorhad no test: deleting the field left all 1 214 tests green, and doctor would have reported "this daemon does not check" for a daemon that does. The thirty-minute de-duplication had none either — pinned now in both directions, so neither "log on every measurement" nor "never repeat" passes.lint,typecheck,test,format:checkgreen — 1 487 tests.🤖 Generated with Claude Code
https://claude.ai/code/session_01QL3QL3ReEa9Sk68mxJW6Fu