Skip to content

feat(release): ship minvmd to Linux as a static musl binary - #1104

Merged
norrietaylor merged 2 commits into
feat/static-musl-libkrunfrom
feat/ship-minvmd-linux
Jul 31, 2026
Merged

feat(release): ship minvmd to Linux as a static musl binary#1104
norrietaylor merged 2 commits into
feat/static-musl-libkrunfrom
feat/ship-minvmd-linux

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 30, 2026

Copy link
Copy Markdown
Member

Completes the Linux half of #980.

Stacked on #1070 (static musl libkrun), and based on its branch rather than main, so the diff here is only the wiring — #1070 is the mechanism. GitHub retargets this to main automatically when that one merges. Review it first.

minvmd was the only one of the four binaries not staged for Linux, and the only one that could not be: a native-glibc build linking libkrun.so and dlopening libkrunfw.so.5. Shipping it as-is meant ~29 MB of libraries plus RUNPATH machinery, a bin/lib sibling constraint, and a glibc floor on users' disks. #1070 removed the reason; this ships it.

release.yml

Both Linux jobs build a static libkrun.a from the vendored pin and link minvmd against it as musl, alongside the three binaries already built that way.

arm64 gains a minvmd artifact for the first time. The dynamic build needed a native glibc host per arch and only amd64 had one — a constraint a static build simply doesn't have. The two Linux arches finally ship the same set of binaries.

The libkrun build rides a new build-libkrun-static-linux composite modelled on setup-libkrun-macos: pin-keyed cache (commit + script hash + patches hash + target), build-on-miss, and a verify step that re-asserts both the API floor and the symbol surface on a possibly cache-restored archive.

Each job asserts the binary is statically linked and actually contains the KVM backend. Neither is implied by a successful build — when build.rs finds no libkrun it emits a stub that compiles and links perfectly well, and would ship as a binary that bails at VM boot. That failure mode is silent without an explicit check, so both are explicit.

stage-release.sh

Linux gains bin/minvmd plus the guest payload (data/{vmlinuz,rootfs.img,initramfs.cpio}) for both arches. All six guest artifacts were already produced for both arches — only the rows were missing.

No lib/ component and no RUNPATH rewrite. That asymmetry with darwin is the entire point:

Linux macOS
bin/minvmd
lib/libkrun.* (linked in)
RUNPATH rewrite
glibc floor n/a

nightly.yml

smoke-linux-kvm no longer materializes a libkrun prefix or sets LD_LIBRARY_PATH, because the shipped binary needs neither. It asserts that instead, so a regression to a dynamic build fails with a clear message rather than a confusing loader error mid-boot — the smoke's whole point is running the artifact exactly as a user receives it.

The userns sysctl stays. Its comment justified it by the removed mip materialize step, but the sandboxed build the e2e drives needs userns independently, so only the comment changed.

setup-libkrun-linux is untouched and not orphaned — ci-linux-kvm and nightly-tests still use it to build against the dynamic upstream package.

Verification

  • Workflow + composite YAML parse
  • shellcheck clean
  • Dry-run stage against stub artifacts emits all eight new Linux rows with correct dests, and the arm64 guest artifacts dedupe with darwin's as intended (one upload, two rows)

Not verified locally: the builds themselves. x86_64 in particular has never been exercised — the static-musl proof in #1070 is aarch64-only, and this is the lane that settles it. Worth a dry_run dispatch of Release before merging.

Rebased onto #1063

#1063 (Linux gvproxy-min rows) landed while this was open and conflicted in the two files it predicted. Resolved additively — Linux now stages bin/gvproxy-min and bin/minvmd + the guest payload — and re-verified with a dry-run stage that emits all ten Linux rows. The stale "minvmd is not staged for Linux yet" note is gone from both the component table and the pipeline doc, and macOS keeps bin/gvproxy-min (this branch predated that rename fix).

Refs #980, #1065

🤖 Generated with Claude Code

Note

Ship minvmd to Linux as a static musl binary linked against vendored libkrun

  • The release pipeline now builds minvmd for linux-amd64 and linux-arm64 as statically linked musl binaries using MINVMD_REQUIRE_LIBKRUN=static, replacing the previous glibc + dynamic libkrun approach.
  • A new reusable composite action build-libkrun-static-linux builds and caches libkrun.a for a given musl target, verifying the krun_add_disk3 symbol is present and no non-krun_* symbols are exported.
  • The nightly smoke test in nightly.yml no longer sets up a dynamic libkrun prefix; it instead asserts the downloaded minvmd artifact is statically linked.
  • scripts/stage-release.sh now stages minvmd and guest payload artifacts (initramfs.cpio, rootfs.img, vmlinuz) for Linux amd64 and arm64.
  • Risk: builds fail if the static libkrun build is missing the KVM backend or if the resulting binary is not fully statically linked.

Changes since #1104 opened

  • Switched minvmd and libkrun to static musl compilation [c0fed12]
  • Removed dynamic linking infrastructure [c0fed12]
  • Added static linking verification [c0fed12]
  • Updated workflow triggers and composite action references [c0fed12]

Macroscope summarized 4004389.

@norrietaylor
norrietaylor requested a review from a team as a code owner July 30, 2026 19:09
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The changes add a cached static-musl libkrun.a build, enable static linking in minvmd, verify Linux artifacts for both architectures, and update release staging, nightly checks, and pipeline documentation.

Changes

Static-musl libkrun build

Layer / File(s) Summary
Build and sanitize libkrun
scripts/build-libkrun-linux.sh, scripts/build-libkrun-macos.sh
The Linux script builds the pinned and patched libkrun commit, sanitizes archive symbols, validates the API, and stages libkrun.a; the macOS build now runs Cargo from the fetched worktree.
Cache and expose static archive
.github/actions/build-libkrun-static-linux/action.yml
The composite action caches the archive using target, pin, script, and patch hashes, verifies symbols, and exports LIBKRUN_PREFIX.

minvmd linkage and release packaging

Layer / File(s) Summary
Select static minvmd linkage
crates/minvmd/build.rs, crates/minvmd/src/krun/raw.rs
minvmd detects libkrun.a, disables rpaths, and conditionally links the FFI library statically.
Build and package Linux artifacts
.github/workflows/release.yml, .github/workflows/nightly.yml, scripts/stage-release.sh, docs/internal/release-pipeline.md
Release jobs build and verify static amd64 and arm64 minvmd binaries, nightly checks static linkage, and Linux staging includes minvmd and guest payloads.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseWorkflow
  participant LibkrunAction
  participant BuildScript
  participant Cargo
  participant MinvmdVerifier
  ReleaseWorkflow->>LibkrunAction: request musl target
  LibkrunAction->>BuildScript: build pinned libkrun
  BuildScript->>Cargo: compile staticlib with musl target
  BuildScript-->>LibkrunAction: staged libkrun.a
  ReleaseWorkflow->>Cargo: build static minvmd with LIBKRUN_PREFIX
  Cargo-->>MinvmdVerifier: minvmd binary
  MinvmdVerifier-->>ReleaseWorkflow: static linkage and kvm_run verification
Loading

Possibly related issues

Possibly related PRs

Suggested reviewers: bryan-minimal

Poem

A rabbit builds a library bright,
With musl beneath and symbols tight.
No hosty lib, no runtime snare,
Two arches hop through release air.
KVM hums—static and neat! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is clear, conventional, and accurately summarizes the main change: shipping Linux minvmd as a static musl binary.
Description check ✅ Passed The description is detailed and covers the summary and verification, though it does not follow the template headings exactly or include a checklist.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ship-minvmd-linux

Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor force-pushed the feat/ship-minvmd-linux branch from 3d98212 to b2920f5 Compare July 30, 2026 19:15

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/workflows/release.yml (1)

203-222: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the duplicated static/KVM verification logic into a shared script.

The "Build static minvmd binary" + "Verify minvmd is static and links libkrun" steps are near-identical between the amd64 and arm64 jobs. The repo already has precedent for this exact pattern (scripts/rewrite-macos-linkage.sh, used in this same file at line 364) — extracting to e.g. scripts/verify-minvmd-static.sh <bin> would remove the duplication and prevent the two jobs' verification logic from silently drifting apart over time.

Also applies to: 312-329

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/release.yml around lines 203 - 222, The static and libkrun
verification logic is duplicated between the amd64 and arm64 release jobs.
Extract the checks currently in the “Verify minvmd is static and links libkrun”
steps into a shared executable script such as scripts/verify-minvmd-static.sh
accepting the binary path as its argument, then replace both workflow blocks
with calls to that script while preserving the existing failure messages and
success output.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/release.yml:
- Around line 203-216: Replace the strings-based kvm_run check in the minvmd
verification step with an assertion based on linkage or another
strip-independent libkrun artifact. Apply the same linkage-based validation to
the later copy verification, preserving the existing static-binary check and
failure behavior.

---

Nitpick comments:
In @.github/workflows/release.yml:
- Around line 203-222: The static and libkrun verification logic is duplicated
between the amd64 and arm64 release jobs. Extract the checks currently in the
“Verify minvmd is static and links libkrun” steps into a shared executable
script such as scripts/verify-minvmd-static.sh accepting the binary path as its
argument, then replace both workflow blocks with calls to that script while
preserving the existing failure messages and success output.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 13bad9d6-9590-416d-ba9d-c9f7616bb1a9

📥 Commits

Reviewing files that changed from the base of the PR and between 063d478 and 3d98212.

📒 Files selected for processing (9)
  • .github/actions/build-libkrun-static-linux/action.yml
  • .github/workflows/nightly.yml
  • .github/workflows/release.yml
  • crates/minvmd/build.rs
  • crates/minvmd/src/krun/raw.rs
  • docs/internal/release-pipeline.md
  • scripts/build-libkrun-linux.sh
  • scripts/build-libkrun-macos.sh
  • scripts/stage-release.sh

Comment thread .github/workflows/release.yml
@norrietaylor
norrietaylor changed the base branch from main to feat/static-musl-libkrun July 30, 2026 19:25
norrietaylor and others added 2 commits July 30, 2026 14:04
Completes the Linux half of #980. minvmd was the only one of the four
binaries not staged for Linux, and the only one that could not be: it
was a native-glibc build linking libkrun.so and dlopening libkrunfw.so.5,
so shipping it meant ~29 MB of libraries plus RUNPATH machinery, a
bin/lib sibling constraint and a glibc floor on users' disks.

Built on #1070, which made minvmd linkable against a static libkrun.

release.yml: both Linux jobs now build a static libkrun.a from the
vendored pin and link minvmd against it as musl, alongside the three
binaries that were already musl. arm64 gains a minvmd artifact for the
first time — the dynamic build needed a native glibc host per arch and
only amd64 had one, a constraint a static build does not have. The
libkrun build rides a new build-libkrun-static-linux composite, modelled
on setup-libkrun-macos: pin-keyed cache, build-on-miss, and a verify
step that re-asserts the API floor and the symbol surface on a
cache-restored archive.

Both jobs assert the built binary is `statically linked` and actually
contains the KVM backend. Neither is implied by a successful build: when
build.rs finds no libkrun it emits a stub that compiles and links
perfectly well, and would ship as a binary that bails at VM boot.

stage-release.sh: Linux gains bin/minvmd plus the guest payload
(data/{vmlinuz,rootfs.img,initramfs.cpio}) for both arches. All six
guest artifacts were already produced for both arches; only the rows
were missing. No lib/ component and no RUNPATH rewrite — that asymmetry
with darwin is the entire point.

nightly.yml: smoke-linux-kvm no longer materializes a libkrun prefix or
sets LD_LIBRARY_PATH, because the shipped binary needs neither. It
asserts that instead, so a regression to a dynamic build fails with a
clear message rather than a loader error mid-boot. The userns sysctl
stays — its justification cited the removed materialize step, but the
sandboxed build the e2e drives needs it independently.

setup-libkrun-linux is untouched and still used by ci-linux-kvm and
nightly-tests, which build against the dynamic upstream package.

Verified: workflow and composite YAML parse; shellcheck clean; a
dry-run stage against stub artifacts emits all eight new Linux rows with
correct dests, and the arm64 guest artifacts dedupe with darwin's as
intended.

Not verified locally: the builds themselves. x86_64 in particular has
never been exercised — the static-musl proof in #1070 is aarch64-only,
and this is the lane that settles it.

Refs #980, #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ci-linux-kvm is the only pre-merge lane that boots a real microVM, and
it was booting the wrong binary. It built minvmd against the dynamic
upstream libkrun package, shipped that .so in the testbed, and resolved
it at runtime via LD_LIBRARY_PATH. Meanwhile Linux now ships a
static-musl minvmd.

That left the shipped configuration first exercised by nightly's
smoke-linux-kvm — after merge. Nothing pre-merge compiled
build-libkrun-linux.sh at all, and nothing anywhere had ever BOOTED a
statically linked minvmd: release.yml only asserts `file` says static
and `strings` finds kvm_run, which proves the archive is well-formed and
the backend is linked, not that libkrun still works.

That gap matters here more than usual. libkrun.a is produced by merging
a whole Rust dependency closure with `ld -r` and then objcopy-localizing
everything outside the krun_* API — precisely the kind of transform that
links clean and misbehaves at runtime (two libstd copies, TLS, the panic
runtime). A symbol check cannot catch that; a boot can.

So this lane now builds libkrun statically from the vendored pin, builds
minvmd for x86_64-unknown-linux-musl, and boots that. The nextest
archive moves to the same target — the harnesses link libkrun too, and a
native-gnu archive would test a different binary than the one shipped.

Falls out of the change: no libkrun-prefix.tgz in the testbed, no
$HOME/.krun unpack, no LIBKRUN_PREFIX or LD_LIBRARY_PATH in the test
job. The binary carries its own libkrun.

Three assertions rather than assumptions, because a silent regression to
a dynamic link would quietly restore the gap this closes:
MINVMD_REQUIRE_LIBKRUN=static on both cargo invocations (a build.rs that
cannot find libkrun.a is now an error, not a stub), and a `file ... |
grep statically linked` check at each end of the artifact handoff.

Two path-filter fixes: vendor/libkrun/** was never listed, so a pin or
patch bump could not re-run the lane that now builds from it; and the
composite entry follows the lane onto build-libkrun-static-linux.

The dynamic path keeps its coverage in nightly-tests.yml, which still
builds against the upstream package — it remains the dev/`just up-kvm`
configuration, just no longer the one CI proves before merge.

Refs #980, #1065

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@norrietaylor
norrietaylor force-pushed the feat/ship-minvmd-linux branch from c0fed12 to 4004389 Compare July 30, 2026 21:04
@norrietaylor
norrietaylor merged commit bc6b20e into feat/static-musl-libkrun Jul 31, 2026
13 of 14 checks passed
@norrietaylor
norrietaylor deleted the feat/ship-minvmd-linux branch July 31, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants