fix(guest): init skips kernel-premounted targets, ending the false /dev ERROR

[stubbi]

Thinking Path

• Mitos boots Firecracker microVMs and forks them via copy-on-write snapshots, exposed through CRDs; the guest agent is PID 1 inside every VM.
• Issue Guest init logs a false ERROR for the kernel-premounted /dev on every boot (misattributed as a firecracker-test flake) #668 reported a flaky guest-agent init: mount /dev intermittently fails EBUSY, blamed for failing the required firecracker-test job on unrelated PRs.
• Investigation of the actual run logs shows the EBUSY is neither flaky nor a failure: the guest kernel automounts devtmpfs on /dev (CONFIG_DEVTMPFS_MOUNT) before init runs, so init's own devtmpfs mount ALWAYS returns EBUSY. The line appears 14 times in a fully green main run (28646803309); the runs cited by Guest init logs a false ERROR for the kernel-premounted /dev on every boot (misattributed as a firecracker-test flake) #668 actually failed on the PTY stdin race (fixed by fix(guest): interactive PTY works, devpts mounted and StdinClose delivered #670, gated by ci: grpc-exec-smoke gates firecracker-test; PTY regressions can no longer hide #677), a live-fork egress socket collision (filed separately), and PR fix(husk): template artifacts stay restorable by a non-root husk (mandatory CoW + group-read contract) #612's own template-ownership bug.
• The genuine defect is the false ERROR itself: boot-log noise at ERROR level in every VM boot is exactly how real failures get misattributed, which is what happened to Guest init logs a false ERROR for the kernel-premounted /dev on every boot (misattributed as a firecracker-test flake) #668.
• This pull request makes init consult /proc/mounts before each mount-table entry: already-mounted-as-expected is skipped at info level, an unexpected fstype is left in place and logged at ERROR, and the firecracker-test boot smoke gains a regression gate asserting the false ERROR is gone from the serial log.
• The benefit is a clean, truthful guest boot log: the next real firecracker-test failure will not have a red herring ERROR line pointing at the guest.

Linked Issues or Issue Description

Closes #668.

What Changed

• guest/agent-rs/src/sys/mount.rs: new mounted_fstype() returns the fstype mounted at a path from /proc/mounts; the parser (fstype_from_mounts) is a pure function, unit-tested on any host, and resolves overmounts to the LAST matching line (the visible mount). is_mounted() is refactored onto the same parser, semantics unchanged.
• guest/agent-rs/src/init/mod.rs: init decides per mount-table entry via premount_action(): Mount when nothing is mounted, SkipAlreadyMounted (info log) when the expected fstype is already there, SkipUnexpectedFstype (error log, mount left in place) when something else is. Decision logic is a pure function with unit tests.
• .github/workflows/kvm-test.yaml: the boot smoke now fails if the serial log contains the false mount /dev: Resource busy ERROR, so the fix cannot silently regress.

Verification

• cargo test --lib in rust:1-bookworm (privileged): 134 passed, and the SAME 10 environment-dependent failures (netlink dummy-link, in-container python kernel tests) as an origin/main baseline run in the identical container; zero regressions, 8 new tests pass.
• The privileged run includes the fork::volumes mount-namespace tests, which exercise the refactored is_mounted against real mounts; they pass.
• rustfmt --check: the only diff in touched files within this change's regions is resolved; pre-existing hand-aligned regions (MOUNT_TABLE) left untouched.
• cargo clippy --lib: no new warnings introduced (checked for all new symbols).
• The firecracker-test job on this PR is the end-to-end proof: the new serial-log gate passes only if the ERROR is gone on a real KVM boot.

Risks

Low risk, but this is the PID-1 boot path of every guest (security-sensitive path, named human reviewer required). Behavior change: init no longer attempts to mount over an already-mounted target. For /dev this is strictly more correct (the EBUSY attempt was a no-op); for a hypothetical target premounted with a DIFFERENT fstype init now leaves it in place with an ERROR instead of stacking a mount over it, which is the more conservative and more visible behavior. No new interface, no secret handling, no threat-model surface move.

Model Used

• Claude Fable 5 (Anthropic), model id claude-fable-5, extended thinking enabled, via Claude Code.

Checklist

• PR title is a conventional commit (feat, fix, docs, ci, chore, refactor, test)
• Thinking Path traces from project context to this change
• Model Used is filled in (with version and capability details)
• Tests added for behavior changes, in the same commit (TDD)
• Docs updated in the same PR (none needed: no user-facing surface; boot-log behavior documented in code)
• Threat-model delta (docs/threat-model.md) included if the security surface moved (surface did not move)
• Benchmark run (bench/) included if the hot path was touched (not touched: one /proc/mounts read per mount-table entry at boot, off the fork hot path)
• No em or en dashes introduced anywhere
• Secret values never logged, in errors, in condition messages, or on host paths
• No internal/instance-local references (only public #NNN / mitos-run/mitos URLs)
• Every commit carries a Signed-off-by trailer (git commit -s)

Summary by CodeRabbit

• Bug Fixes

  • Improved startup mounting behavior so existing mounts are handled safely and consistently, reducing noisy errors and avoiding unnecessary remount attempts.
  • Fixed mount detection to use exact mount targets, improving reliability when paths overlap or are overmounted.

• New Features

  • Added a regression check to catch a known false boot error during automated guest testing, helping prevent bad test results from slipping through.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 507c1b0b-37be-4887-a7d4-0e48c8e4f3c0

📥 Commits

Reviewing files that changed from the base of the PR and between de48b3b and dbc5bf2.

📒 Files selected for processing (3)

• .github/workflows/kvm-test.yaml
• guest/agent-rs/src/init/mod.rs
• guest/agent-rs/src/sys/mount.rs

---

📝 Walkthrough

Walkthrough

This PR makes guest init mount handling idempotent by adding a mounted_fstype helper (built on /proc/mounts parsing) and a PremountAction decision function to skip or log instead of blindly mounting already-mounted targets. A CI workflow check is added to detect the known false-positive "mount /dev: Resource busy" boot error.

Changes

Idempotent Mount Handling

Layer / File(s)	Summary
Mount detection via /proc/mounts guest/agent-rs/src/sys/mount.rs	Replaces is_mounted_linux's boolean scan with a new public mounted_fstype function that parses /proc/mounts, extracts filesystem type with exact target matching, resolves overmounts to the last match, and adds unit tests covering matches, mismatches, malformed input, and overmounts.
Premount decision logic and init mount loop guest/agent-rs/src/init/mod.rs	Adds PremountAction enum and premount_action helper deciding to mount, skip silently, or skip with error based on existing fstype; updates init_system's mount loop to query mounted_fstype before mounting instead of always attempting mount; adds unit tests for the three decision outcomes.
CI regression guard for false ERROR .github/workflows/kvm-test.yaml	Adds a serial-log scan after the vsock test-driver run that fails the step if the known false-positive "mount /dev: Resource busy" error (issue #668) appears, otherwise prints an OK message.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant InitSystem as init_system
  participant MountSys as mounted_fstype
  participant PremountAction as premount_action
  participant Kernel

  InitSystem->>MountSys: mounted_fstype(m.target)
  MountSys-->>InitSystem: existing fstype option
  InitSystem->>PremountAction: premount_action(existing, want_fstype)
  PremountAction-->>InitSystem: Mount / SkipSilently / SkipWithError
  alt Mount decision
    InitSystem->>Kernel: mount(m.source, m.target, m.fstype, 0)
  else Skip decision
    InitSystem->>InitSystem: log info or error
  end

Loading

Related issues: #668 (guest init false ERROR for kernel-premounted /dev)

Suggested labels: guest-agent, ci, bug-fix

Suggested reviewers: Security-sensitive path reviewer for guest/agent-rs mount logic

Poem

A rabbit checked the mounts with care,
No blind remount, just checks laid bare.
devtmpfs beat us to the door,
so skip, don't panic, log no more.
CI now watches for the old false cry —
hop on, the boot logs pass us by. 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is a clear conventional fix message that matches the guest init /dev mount error change.
Description check	✅ Passed	The description includes all required sections and gives a detailed thinking path, change summary, verification, risks, and model info.
Linked Issues check	✅ Passed	The changes address #668 by checking mounted state first, skipping already-mounted targets, and preserving an error for unexpected filesystems.
Out of Scope Changes check	✅ Passed	The workflow gate, mount parser refactor, and init mount decision logic all support the same /dev false-error fix with no obvious unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/guest-init-skip-premounted

---

_{Comment @coderabbitai help to get the list of available commands.}