Skip to content

'Support & document both DooD and DinD docker-isolation workflows (DooD = zero-copy/zero-disk image reuse for disk-constrained hosts)' - #1970

Open
konard wants to merge 6 commits into
mainfrom
issue-1962-3089d8dcbbe8
Open

'Support & document both DooD and DinD docker-isolation workflows (DooD = zero-copy/zero-disk image reuse for disk-constrained hosts)'#1970
konard wants to merge 6 commits into
mainfrom
issue-1962-3089d8dcbbe8

Conversation

@konard

@konard konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements #1962: support and document both Docker isolation workflows — DinD (Docker-in-Docker, nested daemon, the existing behavior) and DooD (Docker-out-of-Docker, share the host daemon) — across the isolation runner, startup preflights, and docs (en/zh/hi/ru).

DooD lets disk-constrained hosts reuse the host's copy of the multi-GB isolation image with zero copy / zero pull / zero extra disk, where the DinD passthrough model would otherwise duplicate the image into a nested store.

What changed

Explicit isolation mode

  • The isolation runner resolves an explicit mode from HIVE_MIND_DOCKER_ISOLATION_MODE (dind/dood) and infers it from the DIND_SKIP_DAEMON / DOCKER_HOST signals, defaulting to dind so existing deployments are unchanged.
  • The startup preflight and verbose launch diagnostics adapt their wording per mode. In DooD they describe the host daemon and concrete-tag remediation (docker pull <exact tag> on the host, pin HIVE_MIND_DOCKER_ISOLATION_IMAGE_TAG) and no longer false-warn about a nested daemon or host-image passthrough that doesn't exist in DooD.

DooD credential-mount fix (the host-daemon mount-source trap)

In DooD an isolated task runs on the host daemon, which resolves bind-mount sources against the host filesystem. The bot's home credential sources (~/.config/gh, ~/.gitconfig, ~/.claude[.json], ~/.codex) don't exist there, so Docker auto-creates each missing source as an empty directory, breaking the task two ways:

  1. File mounts (.claude.json, .gitconfig) fail with "Are you trying to mount a directory onto a file?" — the task dies before it starts.
  2. The git identity is empty (fatal: empty ident name) because the mounted ~/.gitconfig is an empty dir.

Fix: in DooD, resolve the conventional ~/.x mount sources against HIVE_MIND_HOST_CONFIG_DIR (the host-side config root) so the host daemon binds the real files. Because the bot can't stat host-daemon paths, relocated sources bypass the bot-FS existence gate and trust the operator's host layout. DinD is unchanged (it shares the bot filesystem, so the sources are already the real files).

A new startup preflight preflightDockerIsolationAuthMounts detects DooD and warns — before the first task — when mount sources are still the bot's home paths and HIVE_MIND_HOST_CONFIG_DIR is unset, turning a raw mid-run Docker mount failure into an actionable message. It's wired into telegram-bot.mjs alongside the existing image and git-identity preflights (best-effort; never throws).

~/.gitconfig write trap — git identity mounted read-only (:ro)

~/.gitconfig is the one credential that cannot be a writable single-file bind mount. git config --global (run by gh-setup-git-identity --repair, which the startup git-identity preflight invokes) writes a temp file and rename()s it over the target — and a rename over a mountpoint fails with error: could not write config file …: Device or resource busy (git config exits 4). By contrast ~/.claude.json is rewritten in place, so a single-file mount there is fine.

Fix: the git identity (~/.gitconfig, ~/.config/git) is now mounted read-only (:ro) into each task — getDockerIsolationAuthMounts flags it and buildDockerIsolationStartArgs emits the :ro suffix. The task only reads the identity to commit, so :ro makes the contract explicit and turns any stray write-through-the-mount into an immediate, legible error. The writable credentials (gh/claude/codex) are unchanged; verbose diagnostics annotate read-only mounts with (ro).

Docs

  • New docs/DOCKER-ISOLATION.md (+ zh/hi/ru): the DinD-vs-DooD trade-off, both run recipes, the --group-add <host-docker-gid> socket requirement, the DooD credential-mount requirement (HIVE_MIND_HOST_CONFIG_DIR / matching host paths), the ~/.gitconfig write trap (write-then-copy or a mounted directory with GIT_CONFIG_GLOBAL; never populate the identity through the mounted file), the concrete-tag requirement, and how to verify image reuse.
  • docs/DOCKER.md (+ zh/hi/ru): a DinD-vs-DooD deploy subsection recommending DooD where disk is tight.

How to reproduce / verify

The trap reproduces by running the bot in DooD (-v /var/run/docker.sock:/var/run/docker.sock -e DIND_SKIP_DAEMON=1) without exposing host config: the first --isolation docker task fails with the "directory onto a file" mount error / empty git identity. With HIVE_MIND_HOST_CONFIG_DIR set (or matching host symlinks), the host daemon binds the real files and the task runs.

Tests

tests/test-issue-1962-dood-dind-isolation.mjs66 passing assertions covering:

  • mode resolution and per-mode preflight wording (DinD unchanged, DooD host-daemon wording);
  • resolveDockerIsolationConfigSourceHome (DinD keeps bot home; DooD relocates only when HIVE_MIND_HOST_CONFIG_DIR is set and non-blank);
  • getDockerIsolationAuthMounts (DinD existence-gated; DooD relocation bypasses the gate, preserves per-tool scoping, keeps container targets);
  • git identity mounted read-only — the :ro flag is set on ~/.gitconfig/~/.config/git and reaches the --volume spec, while gh/claude stay writable;
  • preflightDockerIsolationAuthMounts (silent in DinD, one actionable warning in unconfigured DooD, cleared once the host config dir is set).

npm run lint is clean on the changed sources.

Closes #1962.

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #1962
@konard konard self-assigned this Jun 22, 2026
Resolve an explicit isolation mode (HIVE_MIND_DOCKER_ISOLATION_MODE),
inferred from DIND_SKIP_DAEMON / DOCKER_HOST, defaulting to dind so
existing deployments are unchanged. Verbose launch logging and the
startup preflight adapt their wording per mode: DooD describes the host
daemon and concrete-tag remediation and no longer false-warns about a
nested daemon or host-image passthrough that does not exist in DooD.

Add docs/DOCKER-ISOLATION.md (+ zh/hi/ru) covering the DinD-vs-DooD
trade-off, both run recipes, the --group-add socket requirement, the
concrete-tag requirement, and image-reuse verification; add a DinD-vs-DooD
deploy subsection to docs/DOCKER.md (+ zh/hi/ru) and link both pages from
the READMEs. Cover the new behavior with tests/test-issue-1962-dood-dind-isolation.mjs.
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Working session summary

I'll await the notification from the monitor when the suite completes.


This summary was automatically extracted from the AI working session output.

@konard konard changed the title [WIP] Support & document both DooD and DinD docker-isolation workflows (DooD = zero-copy/zero-disk image reuse for disk-constrained hosts) 'Support & document both DooD and DinD docker-isolation workflows (DooD = zero-copy/zero-disk image reuse for disk-constrained hosts)' Jun 22, 2026
@konard
konard marked this pull request as ready for review June 22, 2026 06:45
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Public pricing estimate: $11.847261
  • Calculated by Anthropic: $11.856310
  • Difference: $0.009049 (+0.08%)

📊 Context and tokens usage:

Claude Opus 4.8: (3 sub-sessions)

  1. 111.7K / 1M (11%) input tokens, 34.9K / 128K (27%) output tokens
  2. 115.7K / 1M (12%) input tokens, 38.5K / 128K (30%) output tokens
  3. 96.2K / 1M (10%) input tokens, 28.6K / 128K (22%) output tokens

Total: (43.9K new + 287.9K cache writes + 11.7M cache reads) input tokens, 116.9K output tokens, $11.847261 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus (claude-opus-4-8)
  • Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (6765KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🔄 Auto-restart 1/5

Detected uncommitted changes from previous run. Starting new session to review and commit or discard them.

Uncommitted files:

M .changeset/dood-dind-isolation-modes.md
 M docs/DOCKER-ISOLATION.hi.md
 M docs/DOCKER-ISOLATION.md
 M docs/DOCKER-ISOLATION.ru.md
 M docs/DOCKER-ISOLATION.zh.md
 M src/isolation-runner.lib.mjs
 M src/telegram-bot.mjs
 M tests/test-issue-1962-dood-dind-isolation.mjs

Auto-restart will stop after changes are committed or discarded, or after 4 more iterations. Please wait until working session will end and give your feedback.

konard added 2 commits June 22, 2026 06:47
In DooD the isolated task runs on the HOST daemon, so the bot-home
credential mount sources (~/.config/gh, ~/.gitconfig, ~/.claude[.json],
~/.codex) don't exist on the host and Docker auto-creates them as empty
directories — breaking file mounts (.claude.json/.gitconfig 'directory
onto a file') and yielding an empty git identity ('fatal: empty ident
name'). Resolve the conventional ~/.x sources against
HIVE_MIND_HOST_CONFIG_DIR (host-side config root) in DooD so the host
daemon binds the real files, bypassing the bot-FS existence gate.

Add preflightDockerIsolationAuthMounts: detect DooD and warn, before the
first task, when sources are still bot-home paths and the host config dir
is unset — turning a raw Docker mount failure into an actionable message.
Wire it into telegram-bot startup preflights. Docs (en/zh/hi/ru) and tests
updated.
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Working session summary

The last background poller has finished. The work is complete and verified — no further action needed.

Final state:

The issue #1962 solution (explicit DinD/DooD isolation modes plus the DooD credential-mount fix via HIVE_MIND_HOST_CONFIG_DIR, the preflightDockerIsolationAuthMounts preflight, 59 passing tests, and docs in en/zh/hi/ru) is fully committed and pushed.


This summary was automatically extracted from the AI working session output.

@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🔄 Auto-restart 1/5 Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Public pricing estimate: $1.693755
  • Calculated by Anthropic: $13.550065
  • Difference: $11.856310 (+700.00%)

📊 Context and tokens usage:

  • 65.2K / 1M (7%) input tokens, 11.4K / 128K (9%) output tokens

Total: (4.2K new + 52.6K cache writes + 1.7M cache reads) input tokens, 11.4K output tokens, $1.693755 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus (claude-opus-4-8)
  • Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (7906KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

⚠️ .gitconfig-as-mountable-file collides with git's atomic-rename write (Device or resource busy)

This PR fixes the DooD mount-source resolution (great), but the ~/.gitconfig part carries an unexamined assumption that bit us in production. Flagging it because the recommended setup can fail with a confusing error.

The chain

  1. The startup preflight still calls ensureHostGitIdentityForIsolationrepairGitIdentity()gh-setup-git-identity --repairgit config --global user.name/email.
  2. git config does not write in place — it writes a temp file and rename()s it over ~/.gitconfig.
  3. This PR (and getDockerIsolationAuthMounts) treats ~/.gitconfig as a single-file bind-mount source, and the docs recommend exposing it as a mountable file (sudo ln -s /srv/hive-config/.gitconfig /home/box/.gitconfig).

If ~/.gitconfig is a single-file bind mount (or a symlink to one) and anything runs git config --global against it, the rename-over-a-mountpoint fails:

error: could not write config file /home/box/.gitconfig: Device or resource busy

(git config exits 4.) Reproduced on a real DooD deploy (Docker 29.5, containerd snapshotter): the bot's startup git-identity step couldn't write the mounted ~/.gitconfig.

Why .claude.json survives a file mount but .gitconfig doesn't

.claude.json is rewritten in place by the claude tooling, so a single-file bind mount is fine. git config uses temp-file + atomic rename, which cannot rename over a mountpoint. So .gitconfig is the one credential that cannot be a writable single-file bind mount — even though the PR groups it with the others.

Suggested fix / doc change

The PR's "expose ~/.gitconfig at the same host path (symlinks work)" advice is incomplete for .gitconfig specifically, because the thing that populates it (gh-setup-git-identity/git config --global) can't write through a file mount. Options:

  1. Write-then-copy (what we do in our deploy): let gh-setup-git-identity write ~/.gitconfig to the container's own FS (no mount → rename works), then copy it out to the host path the task mounts. The task only reads it, so a read-only file mount there is fine.
  2. Mount a directory, not a file: set GIT_CONFIG_GLOBAL to a file inside a mounted directory (e.g. /home/box/.gitcfg/config where .gitcfg/ is the bind mount). Files inside a mounted dir can be renamed freely. This also needs the isolation task to honor the same GIT_CONFIG_GLOBAL.
  3. At minimum, document that ~/.gitconfig must be a read-only file mount for the task and must NOT be the path gh-setup-git-identity writes through, and mount the task's ~/.gitconfig :ro.

Happy to share the deploy-side write-then-copy implementation. Context: this is the same investigation behind issues box#110 / command-stream#172 and the deploy scripts in the #1962 thread.

@konard
konard marked this pull request as draft June 22, 2026 08:12
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 AI Work Session Started

Starting automated work session at 2026-06-22T08:12:31.957Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

… trap (#1962)

The isolated task only reads the git identity to commit, and ~/.gitconfig
is the one credential that breaks as a writable single-file bind mount:
git config --global (e.g. gh-setup-git-identity --repair) writes a temp
file and rename()s it over the target, and a rename over a mountpoint
fails with 'Device or resource busy' (git config exits 4).

- getDockerIsolationAuthMounts now marks the git identity mounts
  (~/.gitconfig, ~/.config/git) read-only; buildDockerIsolationStartArgs
  emits the :ro suffix on those --volume specs. Writable credentials
  (gh/claude/codex) are unchanged.
- Verbose launch diagnostics annotate read-only mounts with '(ro)'.
- docs/DOCKER-ISOLATION.md (+ zh/hi/ru): new section on the .gitconfig
  write trap with write-then-copy and mounted-directory (GIT_CONFIG_GLOBAL)
  remedies, and the rule to never populate identity through the mount.
- Tests: assert git-identity mounts are read-only and the :ro suffix
  reaches the --volume spec, while gh/claude stay writable (66 passing).
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — addressed the .gitconfig-as-mountable-file / atomic-rename trap in commit 2cf866c.

Code: getDockerIsolationAuthMounts now marks the git-identity mounts (~/.gitconfig, ~/.config/git) read-only, and buildDockerIsolationStartArgs emits the :ro suffix on those --volume specs. The isolated task only reads the identity to commit, so :ro makes the read-only contract explicit and turns any stray write-through-the-mount into an immediate, legible error instead of a confusing mid-run Device or resource busy. The writable credentials (~/.config/gh, ~/.claude[.json], ~/.codex) are unchanged — .claude.json is rewritten in place, so a single-file mount stays fine. Verbose launch diagnostics now annotate read-only mounts with (ro).

Docs (en/zh/hi/ru): new "The ~/.gitconfig write trap (Device or resource busy)" section in docs/DOCKER-ISOLATION.md explaining why .gitconfig is the one credential that cannot be a writable single-file bind mount (temp-file + rename() over a mountpoint), and the two remedies you outlined:

  • Write-then-copy (recommended): gh-setup-git-identity writes ~/.gitconfig on the container's own FS (rename succeeds), then it's copied out to the host path the task mounts :ro.
  • Mount a directory, not a file: point GIT_CONFIG_GLOBAL at a file inside a mounted directory (renames inside a mounted dir work); both bot and task must honor the same GIT_CONFIG_GLOBAL.

It also explicitly warns that the path gh-setup-git-identity / git config --global writes must not be the bind mount (or a symlink to one), and corrects the Option-A symlink advice accordingly.

Tests: +7 assertions (66 passing) covering that the git-identity mounts are read-only, the :ro suffix reaches the --volume spec, and that gh/claude stay writable. npm run lint is clean.

@konard
konard marked this pull request as ready for review June 22, 2026 08:21
@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $4.781401

📊 Context and tokens usage:

  • 113.9K / 1M (11%) input tokens, 41.0K / 128K (32%) output tokens

Total: (16.1K new + 145.6K cache writes + 4.8M cache reads) input tokens, 41.0K output tokens, $4.781401 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus (claude-opus-4-8)
  • Model: Claude Opus 4.8 (claude-opus-4-8)

📎 Log file uploaded as Gist (2645KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard

konard commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support & document both DooD and DinD docker-isolation workflows (DooD = zero-copy/zero-disk image reuse for disk-constrained hosts)

1 participant