Skip to content

fix(husk): template artifacts stay restorable by a non-root husk (mandatory CoW + group-read contract)#612

Open
stubbi wants to merge 7 commits into
mainfrom
fix/template-artifacts-husk-readable
Open

fix(husk): template artifacts stay restorable by a non-root husk (mandatory CoW + group-read contract)#612
stubbi wants to merge 7 commits into
mainfrom
fix/template-artifacts-husk-readable

Conversation

@stubbi

@stubbi stubbi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Thinking Path

The verified failure is a jailed build leaving template artifacts owned by the
jailer build uid (64000), so a husk that is neither that uid nor privileged
fails EACCES and the warm pool crashloops while still reporting Ready. #587's
normalize chowned to the builder euid at 0o644, which only works because forkd
and the husk both run uid 0 today. I took the smallest correct path: make
per-activation rootfs CoW mandatory so Firecracker never in-place restores the
shared template, and change the ownership contract to root:SharedKVMGID with a
group-readable, not-world-writable mode so a future non-root husk (#585) reads
through the group class while the current uid-0 husk keeps working as the file
owner. I introduced named constants for the jailer uid and the shared kvm gid
(there was none) and added the gid as a husk-pod supplemental group. I stayed
out of #578's pool-status wiring (left a TODO) and stated honestly that the
baked rootfs load is O_RDWR, so the non-root husk (#585) will still need the
load leg resolved on top of this group-read contract.

Linked Issues or Issue Description

Closes #597
Related #585, #587, #578

What Changed

  • Mandatory per-activation rootfs CoW (internal/husk/stub.go): Stub.Activate
    now fails closed when a shared template rootfs is configured but no
    per-activation reflink clone was prepared, instead of silently loading the
    shared template in place O_RDWR. Firecracker only ever binds to the husk-owned
    clone; the shared rootfs.ext4 is never opened in place.
  • Group-read ownership contract:
    • internal/firecracker/template.go normalizeTemplateArtifacts now hands the
      artifacts back to root and the shared kvm group (SharedKVMGID), files
      0o640, dirs 0o750 (was world-readable 0o644/0o755).
    • internal/fork/template_invariants.go checkTemplateArtifactInvariants (the
      read-side reuse gate) now verifies euid:SharedKVMGID and mode 0o640.
    • Named constants firecracker.JailerBuildUID (64000) and
      firecracker.SharedKVMGID (65000, chosen OUTSIDE the per-VM jailer range so
      it can never collide with a per-VM jailer gid) replace the bare literal.
  • internal/controller/huskpod.go: the husk pod carries SharedKVMGID as a
    SupplementalGroups entry so a future non-root husk (Harden husk pod: run the VMM as a non-root uid (kvm group) + evaluate userns remap #585) can read the
    group-owned template artifacts; a no-op for the current uid-0 husk (file owner).
  • docs/threat-model.md: documented the new file permission posture (group-owned
    to a named kvm gid, group-readable, not world-writable, still a read-only
    hostPath mount for the husk) and the honest O_RDWR-at-load caveat for Harden husk pod: run the VMM as a non-root uid (kvm group) + evaluate userns remap #585.
  • Pool honesty deferred to Surface template-build failures (e.g. ENOSPC) on SandboxPool status, not just controller logs #578: a TODO in internal/controller/pool_rebuild.go
    driveTemplateHealth. The existing crashloop-based templateRestoreFailing
    already reflects a Prepare-time failure on the pool; surfacing per-activation
    failures on SandboxPool status is Surface template-build failures (e.g. ENOSPC) on SandboxPool status, not just controller logs #578's scope.

Verification

  • go build ./...
  • go test ./internal/firecracker/... ./internal/fork/... ./internal/husk/...
    (non-root; root-gated ownership tests skip)
  • go test ./internal/controller/... with envtest
  • Root-gated suites run as root in a container (proving them, since the go-test
    CI job is non-root):
    • TestNormalizeTemplateArtifactsRestorableByNonRootGroupMember DROPS
      privileges to a non-root uid whose primary + supplemental group is
      SharedKVMGID and reads every normalized artifact without EACCES, and
      confirms a non-member uid is DENIED (proving the group is load-bearing, not
      world-read). This is the in-process (dropped-privilege subprocess) non-root
      restore proof.
    • TestNormalizeTemplateArtifactsSetsGroupReadableContract,
      TestNormalizeTemplateArtifactsRestoresJailFlippedOwner, and the
      internal/fork invariant + reuse tests assert the new root:SharedKVMGID,
      0o640 class.
  • internal/husk TestActivateFailsClosedWhenTemplateWithoutClone: Activate
    refuses without a clone and never loads the shared template
    (loadCalls == 0); TestActivateRebindsRootfsDriveToClone still proves the
    CoW path binds Firecracker to the per-vm clone, never the shared template.
  • Both lints clean: golangci-lint run --timeout=5m <pkgs> and
    GOOS=linux golangci-lint run --timeout=5m <pkgs>.
  • Dash scan clean over the diff.

Risks

  • The baked rootfs drive is is_read_only=false, so /snapshot/load opens the
    shared template O_RDWR BEFORE the PatchDrive rebind. The current uid-0 husk is
    the file OWNER (owner rw on 0o640), so the load succeeds; this PR keeps prod
    correct. A FUTURE non-root husk (Harden husk pod: run the VMM as a non-root uid (kvm group) + evaluate userns remap #585) reads through the GROUP class (read
    only), so the O_RDWR load leg still needs resolving there (group-write on the
    template, or a read-only baked rootfs drive). Called out in the threat model
    and the deferred-work note; not regressed by this PR.
  • SharedKVMGID = 65000 is a fixed gid the forkd chown and the husk supplemental
    group agree on; it is outside the jailer [64000, 64999] range to avoid any
    per-VM jailer gid collision.

Security

This touches security-sensitive paths (internal/fork, internal/firecracker,
internal/husk, plus the husk pod securityContext in internal/controller). Per
repo policy it needs a named human review before merge. Do NOT auto-merge. No
auto-merge is enabled.

Model Used

Claude Fable 5 (claude-fable-5), via Claude Code.

Checklist

  • Tests added/updated in the same change (root-gated non-root restore proof,
    husk fail-closed unit test, invariant + normalize gid/mode tests, husk pod
    supplemental-group test)
  • Docs updated in the same PR (threat-model file permission posture + caveat)
  • Threat-model delta for the moved security surface
  • No em/en dashes anywhere
  • Conventional commit, DCO sign-off, explicit path staging
  • Both lints and the dash scan clean
  • Named human review by @stubbi before merge (security-sensitive path)

Summary by CodeRabbit

  • New Features
    • Template artifacts are now normalized for safer shared reuse via group-readable permissions and consistent shared-group ownership.
    • Husk pods are provisioned with supplemental group access needed to read shared template artifacts.
  • Bug Fixes
    • Template reuse/invariant checks are stricter and reject unnormalized or contract-violating templates.
    • Activation now fails closed when a required per-activation clone is missing.
    • UID range validation now rejects ranges that would include the reserved shared KVM group.
  • Documentation
    • Updated the threat model for the new template artifact permission posture.
  • Tests
    • Added/updated coverage for shared-group invariants, reuse behavior, and fail-closed activation.

…datory CoW + group-read contract)

The jailed template build leaves /var/lib/mitos/templates/<id>/{rootfs.ext4,
snapshot/mem,snapshot/vmstate} owned by the jailer build uid, and #587's
normalize chowned them to the builder euid at 0o644, which only worked because
forkd and the husk both run uid 0 today. A husk that is neither that uid nor
privileged fails EACCES and the warm pool crashloops while the pool still
reports Ready (#597).

Close two failure modes:

1. Mandatory per-activation rootfs CoW. internal/husk Stub.Activate now fails
   closed when a shared template rootfs is configured but no per-activation
   reflink clone was prepared, rather than opening the shared template in place
   O_RDWR. Firecracker only ever binds to the husk-owned clone.

2. Group-read contract that survives a non-root husk. normalizeTemplateArtifacts
   (internal/firecracker) and the read-side reuse gate
   checkTemplateArtifactInvariants (internal/fork) now target root:SharedKVMGID
   with group-readable, not world-writable modes (files 0o640, dirs 0o750).
   Named constants JailerBuildUID and SharedKVMGID replace the bare 64000. The
   husk pod carries SharedKVMGID as a supplemental group so a future non-root
   husk (#585) reads the shared template through the group class; the current
   uid-0 husk reads as owner and is unaffected.

Pool honesty (#578/#579): the existing crashloop-based driveTemplateHealth
already reflects a Prepare-time failure; surfacing per-activation failures on the
SandboxPool status is deferred to #578 with a TODO.

Threat model updated with the new file permission posture and the honest O_RDWR
-at-load caveat for the future non-root husk.

Tests: a root-gated in-process proof drops privileges to a non-root uid in the
shared kvm group and reads the normalized artifacts without EACCES (and confirms
a non-member is denied); root-gated normalize/invariant tests assert the new
gid+mode class; a husk unit test proves Activate fails closed without a clone and
never loads the shared template.

Closes #597
Related #585, #587, #578, #579

Security-sensitive paths (internal/fork, internal/firecracker, internal/husk):
needs a named human reviewer before merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
@stubbi

stubbi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Orchestrator review before handing to @stubbi. Local verification: builds clean, both lints clean, dash scan clean; internal/husk + internal/firecracker + internal/fork tests pass (root-gated ownership tests skip non-root as designed). Holding for human review + a KVM run per the security-sensitive rule.

One unresolved question the reviewer should settle on KVM, because it bears on whether this actually stops the prod recurrence:

I checked prod (v1.13.1). forkd DOES have CAP_CHOWN + DAC_OVERRIDE + FOWNER and there are zero normalize/chown errors in its logs, yet a FRESH template build on v1.13.0 forkd still produced 64000:64000 artifacts (I had to chown 0:0 by hand twice today). Since #587's normalizeTemplateArtifacts chowns to the builder euid (0) and forkd can chown, normalize is either not the last writer or is being undone. The reuse gate rebuilds on a failed invariant, and snapshot/ is under workDir so it is covered, so those are not the gap.

Most likely mechanism: the jailer hardlinks the artifacts between the chroot and the canonical templates/ tree (shared inode), and chownIntoJail flips the shared inode to 64000. If any post-normalize jailer op (a later fork or warm activation reusing the chroot) re-chowns that shared inode, the canonical path reads 64000 again AFTER normalize ran. #612 changes normalize's TARGET (root:SharedKVMGID, 0640) but does not change this ordering, so if the re-chown is the cause, the recurrence persists.

The robust fix that survives a re-chown: make the husk's baked template rootfs drive is_read_only=true so /snapshot/load never opens the shared rootfs.ext4 O_RDWR (the PR's own Risks note flags that it currently does, is_read_only=false, and only the uid-0-as-owner case saves it). With a read-only baked drive + mandatory CoW clone + group/other READ, restore succeeds regardless of whether the file is owned 0 or 64000, as long as it stays readable. That closes both the current uid-0 case and the future non-root #585 case, and is immune to the shared-inode re-chown.

Recommend: on KVM, (1) confirm whether template ownership survives a fork/second-activation after build (reproduce the 64000-after-normalize), and (2) consider flipping the baked husk template drive to read-only in this PR or an immediate follow-up. The mandatory-CoW + group-read changes here are correct and worth landing regardless; I just do not want us to believe #597 is closed until the re-chatown-or-O_RDWR leg is confirmed dead on real KVM.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds a shared KVM group contract for jailed template artifacts, updates normalization and reuse checks to enforce it, wires husk pods into the group, and makes activation fail closed when a per-activation CoW clone is missing.

Changes

Template artifact permission model

Layer / File(s) Summary
Shared KVM contract and pod wiring
internal/firecracker/jailer.go, cmd/forkd/jailer.go, cmd/forkd/jailer_test.go, internal/controller/huskpod.go, internal/controller/huskpod_test.go
Adds JailerBuildUID and SharedKVMGID, reserves the shared group in UID range validation, sets husk pod supplemental groups to include SharedKVMGID, and asserts that pod security context in tests.
Template normalization and artifact reads
internal/firecracker/template.go, internal/firecracker/template_owner_test.go
Normalizes template artifacts to the daemon UID and SharedKVMGID with group-readable modes, and updates ownership/readability tests for jailed-owner, root-gated, and non-root group-member cases.
Read-side reuse invariants
internal/fork/template_invariants.go, internal/fork/template_reuse_test.go, internal/fork/template_reuse_engine_test.go
Requires SharedKVMGID ownership and 0640 mode in the reuse gate, and updates reuse tests for compliant, wrong-mode, foreign-owner, wrong-group, and root-gated flows.
Husk activation fail-closed
internal/husk/stub.go, internal/husk/stub_test.go
Rejects activation when a shared template rootfs is configured but no per-activation CoW clone exists, with a test covering the no-load/no-patch path.
Threat-model and health notes
docs/threat-model.md, internal/controller/pool_rebuild.go
Documents the permission posture and invariant, and adds a controller comment about the current activation-failure detection gap.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

  • mitos-run/mitos#611 — shares the rootfsClone/activation path context around template-backed boots and writable filesystem state.
  • mitos-run/mitos#772 — also changes internal/husk/stub.go’s (*Stub).Activate flow.
  • mitos-run/mitos#799 — also touches internal/husk/stub.go activation logic, though for in-pod policy assembly.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR fixes artifact ownership and clone enforcement for #597, but it defers pool-status honesty instead of surfacing activation permission failures. Implement the SandboxPool status update so activation filesystem permission failures are surfaced and Ready is withheld when restores fail, or retarget the deferred work to #578.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is conventional, specific, and accurately summarizes the main change: template artifacts are made restorable for a non-root husk.
Description check ✅ Passed The description matches the repo template and fills the required sections with concrete reasoning, changes, verification, risks, and model info.
Out of Scope Changes check ✅ Passed The docs, tests, constants, and comments all support the same template-readability fix; no unrelated feature work stands out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/template-artifacts-husk-readable

Comment @coderabbitai help to get the list of available commands.

@stubbi stubbi enabled auto-merge (squash) July 3, 2026 06:26

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/husk/stub.go (1)

700-815: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Move the rootfs-CoW guard before network setup.
This check runs after applyEgressFilter, so a failed Activate can leave the tap, nftables rules, and optional DNS proxy alive until pod shutdown while the stub stays dormant. Hoist the guard above the egress-filter block, and add a test that sets req.Network to cover this path.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/husk/stub.go` around lines 700 - 815, Move the rootfs CoW validation
in Activate ahead of the network setup block so a missing clone fails before
applyEgressFilter, tap creation, nftables changes, or DNS proxy startup can
occur. Locate the guard in the Activate flow in husk/stub.go and hoist it above
the section that builds NetfilterConfig and calls applyEgressFilter, keeping the
same fail-closed error behavior. Add or update a test for Activate that sets
req.Network and verifies the rootfs-CoW failure returns before any network side
effects are started.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/firecracker/jailer.go`:
- Around line 40-49: SharedKVMGID can collide with a custom per-VM jailer gid
because parseUIDRange currently allows any non-root range, so update the
UID-range validation in parseUIDRange (and any caller that accepts --uid-range)
to reject ranges that include SharedKVMGID 65000, or otherwise constrain the
accepted range to avoid that value. Keep the SharedKVMGID constant and related
jailer setup in internal/firecracker/jailer.go aligned with the validation so
the supplemental-group/read-path for the husk remains unique.

In `@internal/firecracker/template_owner_test.go`:
- Around line 158-166: The ancestor-permission widening loop in the test
currently mutates directory modes via os.Chmod and leaves them changed, so
update the logic around the filepath.Dir traversal to capture each directory’s
original permissions before widening and register a t.Cleanup to restore them
afterward. Keep the fix localized to the test helper in template_owner_test.go
so the root-only setup still works without permanently altering ancestor
directories.
- Around line 4-8: The child-process failure handling in template_owner_test.go
is converting the error to a string, which drops the original cause. Update the
error path in the test helper/assertion logic (the code that handles runErr) to
wrap the returned exec error with fmt.Errorf using %w instead of calling
runErr.Error(), so callers retain the underlying error details. Apply the same
change in the related duplicate logic referenced by the review, keeping the
existing context message but preserving the original error value.

In `@internal/firecracker/template.go`:
- Around line 582-590: The ownership normalization in WalkDir should fail closed
on symlinks before calling os.Chown or os.Chmod, since both follow links and can
affect targets outside the template tree. Update the WalkDir callback in
internal/firecracker/template.go to detect symlink entries via os.DirEntry
before any privileged mutation, and return an error (or otherwise stop
processing) when a symlink is encountered. Keep the existing normalization flow
in place for regular files and directories only.

In `@internal/fork/template_invariants.go`:
- Around line 30-39: The reuse invariant in checkTemplateArtifactInvariants only
validates snapshot files, but it must also enforce the directory ownership and
permissions contract for the template and snapshot directories. Extend the
read-side checks in checkTemplateArtifactInvariants to include template
directories (including snapshot) and verify they remain owned by the effective
uid, group-owned by firecracker.SharedKVMGID, and have group-executable mode
0o750 so reuse fails early instead of later with EACCES.

---

Outside diff comments:
In `@internal/husk/stub.go`:
- Around line 700-815: Move the rootfs CoW validation in Activate ahead of the
network setup block so a missing clone fails before applyEgressFilter, tap
creation, nftables changes, or DNS proxy startup can occur. Locate the guard in
the Activate flow in husk/stub.go and hoist it above the section that builds
NetfilterConfig and calls applyEgressFilter, keeping the same fail-closed error
behavior. Add or update a test for Activate that sets req.Network and verifies
the rootfs-CoW failure returns before any network side effects are started.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9f194d62-7cea-495a-8607-c0fc972b639c

📥 Commits

Reviewing files that changed from the base of the PR and between 38d4357 and 96b0eee.

📒 Files selected for processing (12)
  • docs/threat-model.md
  • internal/controller/huskpod.go
  • internal/controller/huskpod_test.go
  • internal/controller/pool_rebuild.go
  • internal/firecracker/jailer.go
  • internal/firecracker/template.go
  • internal/firecracker/template_owner_test.go
  • internal/fork/template_invariants.go
  • internal/fork/template_reuse_engine_test.go
  • internal/fork/template_reuse_test.go
  • internal/husk/stub.go
  • internal/husk/stub_test.go

Comment thread internal/firecracker/jailer.go
Comment thread internal/firecracker/template_owner_test.go Outdated
Comment thread internal/firecracker/template_owner_test.go
Comment thread internal/firecracker/template.go Outdated
Comment thread internal/fork/template_invariants.go
@stubbi

stubbi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Merge-sweep note: this PR has a deterministic firecracker-test failure, reproduced across two consecutive runs after a branch update to current main (runs 28642727338 first attempt and its rerun). Signature, repeating several times per run in the "SDK direct-mode example executed end to end against a real KVM sandbox-server" step:

ERROR sandbox_agent::init: mount /dev: Resource busy (os error 16)

The guest agent fails init mounting /dev, then the step exits 1. The box itself is healthy (other PRs' firecracker-test runs pass on the same runner before and after, including #611 which just merged), so this looks specific to this branch's change interacting with guest boot, possibly the ownership normalization changing what the jailer pre-mounts or a devtmpfs double-mount.

Leaving this PR out of the merge sweep; auto-merge stays armed so it lands once the failure is fixed. Happy to help debug if useful.

@stubbi

stubbi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Heads-up before this auto-merges: firecracker-test is failing deterministically (not the usual external flake), so the merge is correctly blocked. Root cause, from the direct-mode sandbox-server step:

AgentRunError: create template: normalize template artifact ownership:
chown .../templates/python-312: operation not permitted

The new ownership-normalization chown (internal/firecracker/template.go) runs unconditionally, but a chown to a different owner needs privilege (root or CAP_CHOWN). In direct-mode sandbox-server the process runs as the invoking (non-root) user, so it EPERMs; the same would bite any self-hoster running direct mode as a normal user. #611 and #627 firecracker-test both pass, so this is specific to #612's chown.

Suggested direction (for you / the implementer, not pushed): make the normalization best-effort in a non-privileged context. The husk-readability contract only matters on the jailed husk restore path; direct mode has no husk. Options: skip normalization when there is no jailer/husk, or attempt the chown and fall back to the owner-safe chmod g+rX (group-read) path on EPERM instead of failing template creation. Holding this PR (not merging, not fixing it myself) since it is security-sensitive and yours to review.

@stubbi

stubbi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Held for maintainer review (security-sensitive: internal/firecracker + husk restore). Auto-merge is armed but correctly blocked: firecracker-test fails, and this run is the REAL bug, not the external-asset flake.

Why the check is red

The direct-mode leg (SDK direct-mode example ... against a real KVM sandbox-server) fails at template CREATE, before any restore:

create template: normalize template artifact ownership:
chown /home/runner/work/_temp/.../templates/python-312: operation not permitted

normalizeTemplateArtifacts still calls:

uid := os.Geteuid()
os.Chown(path, uid, SharedKVMGID)

Setting the owner to your own euid is a no-op, but changing the GROUP to SharedKVMGID is a privileged operation for a non-root process: POSIX lets a regular user chgrp a file they own only to a group they are a MEMBER of. The jailed/husk daemon runs in the KVM group so it succeeds there; the direct-mode sandbox-server runs as a plain non-privileged user (the CI runner, not in SharedKVMGID), so the chgrp EPERMs. This is the same non-root ownership-normalization EPERM #597 is about; the group-read restore contract is the right target, but the mechanism to reach it (an unconditional group chown) reintroduces the privilege requirement on the create side.

Fix direction (maintainer call, security path)

Make the group normalization best-effort against EPERM rather than fatal, so the non-privileged create path still produces a usable template:

  • Attempt the chgrp to SharedKVMGID; on EPERM, fall back to chmod that makes the artifact group-and-other readable under its EXISTING group (0o644/0o755), so restore still works for a same-user or group-member reader; or
  • Only run the group chown when the process can actually change groups (jailed/privileged), and in direct mode leave the artifact owned by the running user (already readable by that same user on restore).

Either keeps the husk group-read contract intact while not hard-failing the unprivileged direct-mode path that firecracker-test exercises. Not merging or patching this myself (security path); flagging the precise blocker for the implementer.

@stubbi

stubbi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my earlier comment: the mount /dev EBUSY signature has now also appeared on an unrelated billing-only branch (PR #666) while main's push runs with identical engine content pass, so it is a flaky guest-init race, not something deterministic to this branch; filed as #668 with the cross-branch evidence and a fix shape (tolerate EBUSY as already-mounted in guest/agent-rs init). Your three consecutive hits may still mean this change raises the race probability, worth a look while fixing, but the earlier deterministic framing was too strong. Rerunning checks here is reasonable once #668 is understood.

@stubbi

stubbi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Found the branch-specific failure, distinct from the flakes. The rerun (job 84973802316) shows two independent failures:

  1. THIS PR's real bug, deterministic: the tmpl-smoke phase fails with

    FAIL: create template from image: normalize template artifact ownership: chown /tmp/mitos-test/smoke-data/templates/smoke-tmpl: operation not permitted
    

    The new ownership contract chowns artifacts to root:SharedKVMGID, but the CI smoke runs UNPRIVILEGED, and so does any self-host standalone sandbox-server run as a regular user: a non-root euid cannot chown to root (EPERM), so template creation fails outright on every unprivileged engine. The normalize step needs a privilege-aware posture: when running as root, apply the full root:SharedKVMGID 0o640 contract; when unprivileged, keep the builder euid (chgrp to SharedKVMGID only if the euid is a member, else skip with a logged note) and still enforce the mode bits; the invariant check has to accept whichever contract matches the engine's privilege. Earlier my comments blamed the guest-init mount noise; that was wrong, this chown EPERM is the deterministic failure, it was just buried under the noise (Guest init logs a false ERROR for the kernel-premounted /dev on every boot (misattributed as a firecracker-test flake) #668).

  2. Independent of this PR: the PTY stdin EOF race also fired in the same run; that is the Hosted interactive PTY connects but the write/onData data round-trip never delivers #535 guest-agent race (evidence there), being fixed separately, and will keep intermittently failing reruns until it lands.

So: fix the unprivileged chown path here, rebase, and the branch should go green modulo the #535 flake.

stubbi and others added 4 commits July 4, 2026 17:31
… follow symlinks

Two fixes to normalizeTemplateArtifacts, surfaced by the tmpl-smoke KVM
job (which runs the standalone build path as the non-root runner user)
and by CodeRabbit review.

1. Non-root builder support. The normalize step chgrped every artifact to
   the shared kvm gid (65000) so a future non-root husk (#585) could read
   the template through the group class. Only a root builder can chgrp to
   a group it does not belong to, so the standalone sandbox-server and any
   non-root self-host aborted with EPERM: 'operation not permitted'. A
   non-root builder both builds AND restores the template as the same uid,
   so it needs no shared-group contract. Attempt the shared gid and fall
   back to the builder's own gid on the first permission error. The reuse
   invariant (internal/fork.checkTemplateArtifactInvariants) now accepts
   either the shared kvm gid or the process's own gid, keeping the write
   and read sides in lockstep so a non-root deployment reuses its
   templates instead of rebuilding every one. The root path is unchanged:
   euid 0 still tags root:SharedKVMGID at 0o640/0o750.

2. Symlink safety. os.Chown and os.Chmod dereference symlinks, so a link
   planted in the build output could redirect a privileged ownership or
   mode change onto a target outside the template tree. The walk now skips
   symlinks and uses Lchown, so a link inode is never followed.

Tests: two new cases run in the ordinary non-root environment (the
fallback succeeds and tags the builder's own uid:gid; a symlink out of
the tree is left untouched), plus the CodeRabbit test-hygiene nits
(wrap the child-process error, restore widened ancestor perms via
t.Cleanup). Updated TestCheckTemplateArtifactInvariantsRejectsUnnormalized
to assert the mode citation, since own-gid is now accepted.

Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
…plate dirs

Two more CodeRabbit review findings on the husk-readable template work:

- parseUIDRange now rejects a --uid-range that covers the shared kvm gid
  (65000). A per-VM jailer uid/gid drawn from an overlapping range would
  collide with the group that owns the group-readable template artifacts,
  so a jailed build VM could carry the template-read group. The shipped
  default 64000-64999 is unaffected.

- checkTemplateArtifactInvariants now also checks the template and snapshot
  directories, not just the files. normalizeTemplateArtifacts sets the dirs
  to 0o750, but a husk cannot traverse to the files (EACCES) if a dir loses
  group execute even when the files are compliant; the read-side gate must
  catch that. The uid/gid/mode check is factored into one helper shared by
  the dir (0o750) and file (0o640) checks, and the root-gated compliant test
  helper normalizes the dirs too.

Signed-off-by: Jannes Stubbemann <jannes@openclaw.rocks>
@stubbi

stubbi commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Root cause of the firecracker-test failure + review pass

The tmpl-smoke KVM step failed with normalize template artifact ownership: chown ...: operation not permitted. Root cause: normalizeTemplateArtifacts chgrps every artifact to the shared kvm gid (65000), but tmpl-smoke runs the standalone build path as the non-root runner user, and that user is neither root nor a member of gid 65000, so the chgrp fails EPERM. The normalize comment assumed only a root forkd calls this, but cmd/sandbox-server (and any non-root self-host) also does.

Fixes pushed

  1. Non-root builder support (7c16e9a): normalize attempts the shared kvm gid and falls back to the builder's own gid on the first permission error. A non-root builder both builds and restores as the same uid, so it needs no shared-group contract. The reuse invariant now accepts the shared kvm gid or the process's own gid, keeping write and read sides in lockstep. Root path is byte-identical (euid 0 still tags root:SharedKVMGID 0o640/0o750).
  2. Symlink safety (7c16e9a): the walk skips symlinks and uses Lchown, so a planted link can't redirect a privileged chown/chmod out of the template tree.
  3. uid-range guard (85f2302): parseUIDRange rejects a range covering gid 65000.
  4. Directory invariant (85f2302): the reuse gate now also checks the template and snapshot dirs are group-traversable (0o750), not just the files.
  5. Plus the two test-hygiene nits (error wrap, restore widened ancestor perms).

Verification

  • Local (darwin, non-root): go build/vet/gofmt clean; GOOS=linux golangci-lint clean on all four packages; new non-root tests pass (TestNormalizeTemplateArtifactsFallsBackForNonRootBuilder, TestNormalizeTemplateArtifactsSkipsSymlinks, TestParseUIDRange overlap cases); existing fork/firecracker suites green. The root-gated tests (SharedKVMGID setup) still skip locally and run on the KVM runner.
  • The tmpl-smoke firecracker-test itself verifies end to end in CI on the KVM runner.

This touches security-sensitive paths (internal/firecracker, internal/fork), so per CLAUDE.md it needs a named human reviewer before merge; I am not merging it.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/fork/template_invariants.go`:
- Line 49: The reuse gate is following symlinks when validating template
artifacts, allowing a symlinked rootfs.ext4, mem, or vmstate to pass as if it
were a regular file. Update normalizeTemplateArtifacts and the validation path
around os.Stat to use os.Lstat or explicitly reject ModeSymlink so only real
files are accepted and symlinked inodes fail fast.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: dc0ef7ff-bbca-44dc-935d-e626d659be5d

📥 Commits

Reviewing files that changed from the base of the PR and between 7c16e9a and 85f2302.

📒 Files selected for processing (4)
  • cmd/forkd/jailer.go
  • cmd/forkd/jailer_test.go
  • internal/fork/template_invariants.go
  • internal/fork/template_reuse_test.go

// "artifact" or "dir" for the message; wantMode is 0o640 for files and
// 0o750 for directories.
checkOwnership := func(kind, path string, wantMode os.FileMode) error {
info, err := os.Stat(path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Look for any symlink creation in build output / template artifact paths.
rg -nP '(os\.Symlink|Symlink\()' internal/firecracker internal/fork
# Confirm what templateSnapshotFiles resolves to.
ast-grep run --pattern 'func templateSnapshotFiles($$$) $_ { $$$ }' --lang go internal/fork

Repository: mitos-run/mitos

Length of output: 1259


🏁 Script executed:

#!/bin/bash
sed -n '1,220p' internal/fork/template_invariants.go
printf '\n---\n'
sed -n '1,140p' internal/fork/verify.go
printf '\n---\n'
sed -n '1640,1705p' internal/fork/engine.go

Repository: mitos-run/mitos

Length of output: 15076


🏁 Script executed:

#!/bin/bash
rg -n "normalizeTemplateArtifacts|checkTemplateArtifactInvariants|templateSnapshotFiles|writeVerifiedMarker|writeDigestFile|recordTemplateDigest|verifyTemplate|shouldReuseTemplate" internal/fork
printf '\n---\n'
sed -n '1,220p' internal/fork/template_owner.go
printf '\n---\n'
sed -n '1,220p' internal/fork/verify.go

Repository: mitos-run/mitos

Length of output: 18833


🏁 Script executed:

#!/bin/bash
sed -n '2180,2275p' internal/fork/engine.go
printf '\n---\n'
sed -n '2388,2455p' internal/fork/engine.go
printf '\n---\n'
sed -n '1,240p' internal/fork/template_reuse_test.go

Repository: mitos-run/mitos

Length of output: 17123


🏁 Script executed:

#!/bin/bash
rg -n "normalizeTemplateArtifacts|Lchown|Lstat|Symlink|templateSnapshotFiles|checkTemplateArtifactInvariants" internal/fork
printf '\n---\n'
sed -n '1,220p' internal/fork/* | sed -n '1,220p'

Repository: mitos-run/mitos

Length of output: 11611


🏁 Script executed:

#!/bin/bash
rg -n "func normalizeTemplateArtifacts|normalizeTemplateArtifacts\(" internal/fork
printf '\n---\n'
fd -a "template*go" internal/fork

Repository: mitos-run/mitos

Length of output: 158


🏁 Script executed:

#!/bin/bash
rg -n "normalizeTemplateArtifacts|Lchown|Lstat|os\.Symlink|Symlink\(" internal/fork internal/firecracker
printf '\n---\n'
sed -n '1,220p' internal/fork/engine.go | sed -n '2280,2365p'

Repository: mitos-run/mitos

Length of output: 3194


🏁 Script executed:

#!/bin/bash
rg -n "normalizeTemplateArtifacts|Lchown|Lstat|os\.Symlink|Symlink\(" internal/fork internal/firecracker
printf '\n---\n'
sed -n '2280,2365p' internal/fork/engine.go

Repository: mitos-run/mitos

Length of output: 7310


🏁 Script executed:

#!/bin/bash
sed -n '554,630p' internal/firecracker/template.go
printf '\n---\n'
sed -n '250,290p' internal/firecracker/template_owner_test.go
printf '\n---\n'
sed -n '1658,1692p' internal/fork/engine.go

Repository: mitos-run/mitos

Length of output: 7092


Reject symlinks in the reuse gate. normalizeTemplateArtifacts already skips symlinks, but os.Stat here follows them, so a planted rootfs.ext4/mem/vmstate symlink is validated against its target. Use os.Lstat (or reject ModeSymlink) so regular templates fail fast instead of passing via a symlinked inode.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/fork/template_invariants.go` at line 49, The reuse gate is following
symlinks when validating template artifacts, allowing a symlinked rootfs.ext4,
mem, or vmstate to pass as if it were a regular file. Update
normalizeTemplateArtifacts and the validation path around os.Stat to use
os.Lstat or explicitly reject ModeSymlink so only real files are accepted and
symlinked inodes fail fast.

@stubbi

stubbi commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Re-assessed against current main (v1.38 era). Still relevant, not superseded: issue #597 is open, nothing on main touched template_invariants.go or the jailer legs since, and a merge against main is textually conflict-free. Caveat: the branch predates the live-cow lazy-UFFD restore work, so a fresh firecracker-test and cluster-husk-e2e run after update-branch is mandatory before this merges.

Two small code fixes still owed from the CodeRabbit round: (1) the read-side reuse gate checkTemplateArtifactInvariants still uses os.Stat (follows symlinks) while the write side got the symlink-safe Lchown treatment; make it Lstat plus a ModeSymlink reject for symmetry. (2) the fail-closed CoW guard in Stub.Activate sits after applyEgressFilter, so a fail-closed activation leaves tap/nftables/DNS-proxy alive until pod shutdown; hoist the guard above the network block.

One open question to settle before treating this as "Closes #597": if the prod recurrence mechanism is a post-normalize chownIntoJail re-flipping hardlinked inodes, this PR changes normalize's target but not that ordering, so the recurrence could survive. The mandatory-CoW and group-read contract are worth landing regardless; just keep the close provisional pending a KVM repro.

Held for the named human security review per the standing instruction on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jailed template build leaves artifacts unreadable by husk restore; pool crash-loops but still reports Ready

1 participant