Skip to content

fix(daemon): the isolation between servers is measured, not assumed - #70

Merged
aaldersondev merged 1 commit into
mainfrom
fix/network-isolation
Aug 8, 2026
Merged

fix(daemon): the isolation between servers is measured, not assumed#70
aaldersondev merged 1 commit into
mainfrom
fix/network-isolation

Conversation

@aaldersondev

Copy link
Copy Markdown
Contributor

Servers on a node are kept apart by exactly one thing — com.docker.network.bridge.enable_icc=false on the shared bridge — and it was set only on the creation path:

const networks = await this.docker.listNetworks({ filters: { name: [name] } });
if (networks.some((network) => network.Name === name)) {
  this.logger.debug({ network: name }, 'Docker network already present');
  return;                                   // ← never reads its Options
}

A hopper0 created by a bare docker network create, by an operator following half a runbook, by a Hopper old enough to predate the option, or restored with a machine image, left inter-container traffic on. Every server on that node could then reach every other server's unpublished ports.

docs/security.md listed network isolation among the things an operator has nothing to configure, because it is the default behaviour. On such a node it was not true, and nothing anywhere said so.

Measured on both paths

The option is now read after creating as well as after finding, so the verdict is always something Docker was asked rather than something the code assumes because it sent the option once.

An absent option reads as open, not as neutral — Docker's default is to allow the traffic, so silence is the dangerous answer. The value is parsed the way Docker's own bridge driver parses it (Go's strconv.ParseBool) rather than compared to the string this file happens to write. A Docker that cannot be asked answers unknown, never an accusation.

Reported, never repaired, never fatal

Both cures are worse than the fault, and the reasoning is written where the next reader will find it.

Recreating is impossible in place — enable_icc cannot be changed on an existing network — so it means delete-and-rebuild, which disconnects every container on the machine. Docker also refuses to remove a network while anything is attached, so an unprompted startup repair would fail halfway on precisely the busy node where it matters.

Refusing to start protects the isolation and takes the node down to do it: no console, no backups, nothing startable. Worse, a daemon that has exited cannot tell anyone why — the panel shows it as offline, indistinguishable from a dead machine. The loudest possible action producing the quietest possible message.

The deciding asymmetry: the fault's blast radius is between the servers on one node; either cure's is every server on that node, immediately and unasked.

So it is said instead — at every start, again every thirty minutes while it lasts, on /api/system, and as a failing hopper doctor check. Doctor said nothing about the network before this.

The repair advice branches on autoCreate, and that is not a detail

The review caught this and it is the sharp edge of the whole change. With docker.network.autoCreate: false — a documented setting, and the one an operator who hand-provisions networks is likeliest to have chosen, which is exactly the population this check exists to catch — "remove it and restart hopperd" rebuilds nothing. The daemon finds the network gone and refuses to start.

Following the advice would have produced the permanent outage the design explicitly refused to cause.

The daemon now prints the sequence for its own configuration, including the full docker network create when it will not rebuild one itself. hopper doctor prints no command: it runs in the panel, which cannot see that setting, and the wrong half of the guess is the expensive half — so it names the fault and points at the node's log.

Checked rather than assumed

Two neighbouring halves of the same guarantee, both already sound, and the review says so rather than inventing work: containers are attached to the dedicated network on every path (NetworkMode comes from config, nothing calls network connect), and what Docker publishes is exactly the server's allocations, TCP and UDP, on their own addresses. Both now have a test — enable_icc=false protects nothing if the container is not on that network, and Docker's default when the field is dropped is bridge.

One documentation error found: RCON was given as an example of a port no other server can reach. Hopper models RCON as an allocation, so it is published like any other and reachable from outside the node.

Verification

  • Confirmed against a real Docker 29.3.1 engine, not a stub: a hand-made hopper0 without the option is detected, is not recreated (same network Id), and produces exactly one error line. A correct network, an enable_icc=0 network and Docker 29's extra options do not misfire. The reviewer also proved the underlying claim directly — container-to-container traffic is dropped on an ICC-off bridge and flows on a bare one.
  • Two mutation survivors closed. The /api/system wiring between detection and hopper doctor had no test: deleting the field left all 1 214 tests green, and doctor would have reported "this daemon does not check" for a daemon that does. The thirty-minute de-duplication had none either — pinned now in both directions, so neither "log on every measurement" nor "never repeat" passes.
  • lint, typecheck, test, format:check green — 1 487 tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QL3QL3ReEa9Sk68mxJW6Fu

Servers on a node are kept apart by one Docker network option,
`com.docker.network.bridge.enable_icc=false`, and it was set only on the
creation path. `ensureNetwork` returned the moment it saw a network by the
right name, without ever reading its options.

So a `hopper0` created by a bare `docker network create`, by an operator
following half a runbook, by a Hopper old enough to predate the option, or
restored with a machine image, left inter-container traffic **on** — and
every server on that node could reach every other server's unpublished
ports. `docs/security.md` listed network isolation among the things an
operator has nothing to configure because it is simply the default. On
such a node it was not true, and nothing said so.

The option is now read on both paths, so what the daemon reports is always
something Docker was asked rather than something the code assumes because
it sent the option once. An absent option reads as open, not as neutral:
Docker's default is to allow the traffic, so silence is the dangerous
answer. The value is parsed the way Docker's own bridge driver parses it
rather than compared to the string this file writes.

**A bad network is reported, never repaired, and never fatal.** Both cures
are worse than the fault. `enable_icc` cannot be changed on an existing
network, so repairing means delete-and-rebuild, which disconnects every
container on the machine — and Docker refuses to remove a network while
anything is attached, so an unprompted startup repair would fail halfway
on exactly the busy node where it matters. Refusing to start protects the
isolation and takes the node down to do it: no console, no backups,
nothing startable, and a daemon that has exited cannot tell anyone why —
the panel shows it as offline, indistinguishable from a dead machine. The
loudest possible action producing the quietest possible message.

The deciding asymmetry is that the fault's blast radius is between the
servers on one node, while either cure's is every server on that node,
immediately and unasked.

So it is said instead: at every start, again every thirty minutes while it
lasts, on `/api/system`, and as a failing `hopper doctor` check — which
said nothing about the network at all before this.

The repair advice branches on `docker.network.autoCreate`, and that is not
a detail. With it false — a documented setting, and the one an operator
who hand-provisions networks is likeliest to have chosen, which is exactly
the operator this check exists to catch — "remove it and restart hopperd"
rebuilds nothing: the daemon finds the network gone and refuses to start.
Following the advice would have produced the permanent outage the design
refused to cause. `hopper doctor` prints no command for this, because the
panel cannot see that setting and the wrong half of the guess is the
expensive half; it points at the node's log, where the daemon writes the
sequence for its own configuration.

Two neighbouring claims were checked rather than assumed, and both already
held: containers are attached to the dedicated network on every path, and
what Docker publishes is exactly the server's allocations on their own
addresses. Both now have a test. The documentation's example of an
unpublished port was wrong, though — RCON is modelled as an allocation, so
it is published like any other.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QL3QL3ReEa9Sk68mxJW6Fu
@aaldersondev
aaldersondev merged commit c8b4ee3 into main Aug 8, 2026
3 checks passed
@aaldersondev
aaldersondev deleted the fix/network-isolation branch August 8, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant