Skip to content

Add GitHub workflow for cross-compilation and fix build issues for macOS and FreeBSD #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nohajc wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add GitHub workflow for cross-compilation and fix build issues for macOS and FreeBSD #636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4f8adc4
Makefile: Add Linux -> FreeBSD init cross-compilation support
nohajc Apr 15, 2026
39e063a
CI: Add init cross-compilation workflow
nohajc Apr 15, 2026
9e9c626
Makefile fixes for FreeBSD
nohajc Apr 14, 2026
15c0c08
vmm: fix unit tests on macOS
nohajc Apr 16, 2026
45659e1
vmm: fixed linking of crate against Hypervisor when running separately
nohajc Apr 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cargo/config.toml
View file
Open in desktop
Copy link
Copy Markdown
Contributor Author

@nohajc nohajc Apr 14, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runner configuration lets me run unit tests from VS Code editor.

Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[target.aarch64-apple-darwin]
runner = ".cargo/macos-test-runner.sh"
14 changes: 14 additions & 0 deletions .cargo/macos-test-runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/sh
# Cargo target runner for macOS (aarch64-apple-darwin).
# Codesigns the test/run binary with the Hypervisor.framework entitlement
# before executing it, so that HVF-based tests can run without a developer
# certificate.
Comment thread
mtjhrc marked this conversation as resolved.
set -eu

BINARY="$1"
shift

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
codesign --entitlements "$REPO_ROOT/hvf-entitlements.plist" --force -s - "$BINARY"
exec "$BINARY" "$@"
41 changes: 41 additions & 0 deletions .github/workflows/cross-compilation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Cross-compilation
on: [pull_request]

jobs:
cross-compile-macos:
name: Cross-compilation (macOS)
runs-on: macos-latest
env:
DYLD_LIBRARY_PATH: /Library/Developer/CommandLineTools/usr/lib/
Copy link
Copy Markdown
Contributor Author

@nohajc nohajc Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slp I had issues building the macOS version which looked like this:
slp/homebrew-krun#14

Setting DYLD_LIBRARY_PATH fixed it for the GitHub action. Maybe the krun_display patch for homebrew is not needed.

steps:
- uses: actions/checkout@v4

- name: Setup build environment
uses: ./.github/actions/setup-build-env

- name: Install cross-compilation dependencies
run: |
brew install lld xz

- name: Build libkrun on macOS (with Linux init)
run: make

- name: Build FreeBSD init on macOS
run: make BUILD_BSD_INIT=1 -- init/init-freebsd

cross-compile-linux:
name: Cross-compilation (Linux x86_64 -> FreeBSD)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Setup build environment
uses: ./.github/actions/setup-build-env

- name: Install cross-compilation dependencies
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends lld

- name: Build FreeBSD init on Linux
run: make BUILD_BSD_INIT=1 -- init/init-freebsd
22 changes: 18 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ AWS_NITRO_INIT_SRC = \

AWS_NITRO_INIT_LD_FLAGS = -larchive -lnsm

INIT_SRC = init/init.c

ifeq ($(SEV),1)
VARIANT = -sev
FEATURE_FLAGS := --features amd-sev
Expand Down Expand Up @@ -120,7 +122,7 @@ $(AWS_NITRO_INIT_BINARY): $(AWS_NITRO_INIT_SRC)
$(CC) -O2 -static -s -Wall $(AWS_NITRO_INIT_LD_FLAGS) -o $@ $(AWS_NITRO_INIT_SRC) $(AWS_NITRO_INIT_LD_FLAGS)

ifeq ($(OS),Darwin)
# If SYSROOT_BSD is not set and we're on macOS, generate sysroot automatically
# macOS -> FreeBSD cross-compilation
ifeq ($(SYSROOT_BSD),)
SYSROOT_BSD = $(FREEBSD_ROOTFS_DIR)
SYSROOT_BSD_TARGET = $(FREEBSD_ROOTFS_DIR)/.sysroot_ready
Expand All @@ -129,6 +131,16 @@ else
endif
# Cross-compile on macOS with the LLVM linker (brew install lld)
CC_BSD=$(CLANG) -target $(ARCH)-unknown-freebsd -fuse-ld=lld -stdlib=libc++ -Wl,-strip-debug --sysroot $(SYSROOT_BSD)
else ifeq ($(OS),Linux)
# Linux -> FreeBSD cross-compilation
ifeq ($(SYSROOT_BSD),)
SYSROOT_BSD = $(FREEBSD_ROOTFS_DIR)
SYSROOT_BSD_TARGET = $(FREEBSD_ROOTFS_DIR)/.sysroot_ready
else
SYSROOT_BSD_TARGET =
endif
# Cross-compile on Linux with clang
CC_BSD=$(CLANG) -target $(ARCH)-unknown-freebsd -fuse-ld=lld -Wl,-strip-debug --sysroot $(SYSROOT_BSD)
else
# Build on FreeBSD host
CC_BSD=$(CC)
Expand All @@ -138,7 +150,7 @@ endif
ifeq ($(BUILD_BSD_INIT),1)
INIT_BINARY_BSD = init/init-freebsd
$(INIT_BINARY_BSD): $(INIT_SRC) $(SYSROOT_BSD_TARGET)
$(CC_BSD) -std=c23 -O2 -static -Wall $(INIT_DEFS) -lutil -o $@ $(INIT_SRC) $(INIT_DEFS)
$(CC_BSD) -std=c23 -O2 -static -Wall -lutil -o $@ $(INIT_SRC)
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In C linking, libraries should generally follow the source files that depend on them to ensure symbols are resolved correctly by the linker. Moving -lutil to the end of the command is a safer practice.

        $(CC_BSD) -std=c23 -O2 -static -Wall -o $@ $(INIT_SRC) -lutil

endif

# Sysroot preparation rules for cross-compilation on macOS
Expand Down Expand Up @@ -179,10 +191,12 @@ $(FREEBSD_ROOTFS_DIR)/.sysroot_ready: $(FREEBSD_BASE_TXZ)
@cd $(FREEBSD_ROOTFS_DIR) && tar xJf base.txz 2>/dev/null || true
@touch $@

BSD_ARCH=$(subst x86_64,amd64,$(ARCH))

$(FREEBSD_BASE_TXZ):
@echo "Downloading FreeBSD $(FREEBSD_VERSION) base for $(ARCH)..."
@echo "Downloading FreeBSD $(FREEBSD_VERSION) base for $(BSD_ARCH)..."
@mkdir -p $(FREEBSD_ROOTFS_DIR)
@curl -fL -o $@ https://download.freebsd.org/releases/$(ARCH)/$(FREEBSD_VERSION)/base.txz
@curl -fL -o $@ https://download.freebsd.org/releases/$(BSD_ARCH)/$(FREEBSD_VERSION)/base.txz

clean-sysroot:
rm -rf $(ROOTFS_DIR)
Expand Down
8 changes: 8 additions & 0 deletions hvf-entitlements.plist
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.hypervisor</key>
<true/>
</dict>
</plist>
2 changes: 0 additions & 2 deletions src/libkrun/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,4 @@ fn main() {
std::env::var("CARGO_PKG_VERSION_MAJOR").unwrap(), std::env::var("CARGO_PKG_VERSION_MAJOR").unwrap(),
std::env::var("CARGO_PKG_VERSION_MAJOR").unwrap(), std::env::var("CARGO_PKG_VERSION_MINOR").unwrap()
);
#[cfg(target_os = "macos")]
println!("cargo:rustc-link-lib=framework=Hypervisor");
Comment thread
nohajc marked this conversation as resolved.
}
4 changes: 4 additions & 0 deletions src/vmm/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,8 @@ fn main() {
println!("cargo:rustc-env=KRUN_EDK2_BINARY_PATH={edk2_binary_path}");
println!("cargo:rerun-if-env-changed=KRUN_EDK2_BINARY_PATH");
}

if std::env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("macos") {
println!("cargo:rustc-link-lib=framework=Hypervisor");
}
Comment on lines +13 to +15
Copy link
Copy Markdown
Collaborator

@mtjhrc mtjhrc Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libkrun crate already links this. Do we need this in both places?
Possibly this is the correct place and it can be removed from libkrun crate?

Copy link
Copy Markdown
Contributor Author

@nohajc nohajc Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I will test this. I had the linker error when trying to run cargo test on the vmm crate so you're probably right.

Copy link
Copy Markdown
Contributor Author

@nohajc nohajc Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still works after removing from libkrun crate.

}
1 change: 1 addition & 0 deletions src/vmm/src/builder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2343,6 +2343,7 @@ pub mod tests {
use super::*;
use crate::vmm_config::kernel_bundle::KernelBundle;

#[allow(unused)]
fn default_guest_memory(
mem_size_mib: usize,
) -> std::result::Result<
Expand Down
21 changes: 3 additions & 18 deletions src/vmm/src/device_manager/hvf/mmio.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,6 @@ impl DeviceInfoForFDT for MMIODeviceInfo {

#[cfg(test)]
mod tests {
use super::super::super::builder;
use super::*;
use arch;
use devices::legacy::DummyIrqChip;
Expand All @@ -339,7 +338,6 @@ mod tests {
impl MMIODeviceManager {
fn register_virtio_device(
&mut self,
_vm: &Vm,
guest_mem: GuestMemoryMmap,
device: Arc<Mutex<dyn devices::virtio::VirtioDevice>>,
_cmdline: &mut kernel_cmdline::Cmdline,
Expand Down Expand Up @@ -423,19 +421,14 @@ mod tests {
let start_addr2 = GuestAddress(0x1000);
let guest_mem =
GuestMemoryMmap::from_ranges(&[(start_addr1, 0x1000), (start_addr2, 0x1000)]).unwrap();
let mut vm = builder::setup_kvm_vm(&guest_mem).unwrap();
let mut device_manager =
MMIODeviceManager::new(&mut 0xd000_0000, (arch::IRQ_BASE, arch::IRQ_MAX));

let mut cmdline = kernel_cmdline::Cmdline::new(4096);
let dummy = Arc::new(Mutex::new(DummyDevice::new()));
#[cfg(target_arch = "x86_64")]
assert!(builder::setup_interrupt_controller(&mut vm).is_ok());
#[cfg(target_arch = "aarch64")]
assert!(builder::setup_interrupt_controller(&mut vm, 1).is_ok());

assert!(device_manager
.register_virtio_device(&vm, guest_mem, dummy, &mut cmdline, 0, "dummy")
.register_virtio_device(guest_mem, dummy, &mut cmdline, 0, "dummy")
.is_ok());
}

Expand All @@ -445,20 +438,14 @@ mod tests {
let start_addr2 = GuestAddress(0x1000);
let guest_mem =
GuestMemoryMmap::from_ranges(&[(start_addr1, 0x1000), (start_addr2, 0x1000)]).unwrap();
let mut vm = builder::setup_kvm_vm(&guest_mem).unwrap();
let mut device_manager =
MMIODeviceManager::new(&mut 0xd000_0000, (arch::IRQ_BASE, arch::IRQ_MAX));

let mut cmdline = kernel_cmdline::Cmdline::new(4096);
#[cfg(target_arch = "x86_64")]
assert!(builder::setup_interrupt_controller(&mut vm).is_ok());
#[cfg(target_arch = "aarch64")]
assert!(builder::setup_interrupt_controller(&mut vm, 1).is_ok());

for _i in arch::IRQ_BASE..=arch::IRQ_MAX {
device_manager
.register_virtio_device(
&vm,
guest_mem.clone(),
Arc::new(Mutex::new(DummyDevice::new())),
&mut cmdline,
Expand All @@ -472,7 +459,6 @@ mod tests {
"{}",
device_manager
.register_virtio_device(
&vm,
guest_mem,
Arc::new(Mutex::new(DummyDevice::new())),
&mut cmdline,
Expand All @@ -489,7 +475,7 @@ mod tests {
fn test_dummy_device() {
let dummy = DummyDevice::new();
assert_eq!(dummy.device_type(), 0);
assert_eq!(dummy.queue_config().len(), QUEUE_CONFIG.len());
assert_eq!(dummy.queue_config().len(), QUEUE_SIZES.len());
}

#[test]
Expand Down Expand Up @@ -548,7 +534,6 @@ mod tests {
let start_addr2 = GuestAddress(0x1000);
let guest_mem =
GuestMemoryMmap::from_ranges(&[(start_addr1, 0x1000), (start_addr2, 0x1000)]).unwrap();
let vm = builder::setup_kvm_vm(&guest_mem).unwrap();
let mut device_manager =
MMIODeviceManager::new(&mut 0xd000_0000, (arch::IRQ_BASE, arch::IRQ_MAX));
let mut cmdline = kernel_cmdline::Cmdline::new(4096);
Expand All @@ -557,7 +542,7 @@ mod tests {
let type_id = 0;
let id = String::from("foo");
if let Ok(addr) =
device_manager.register_virtio_device(&vm, guest_mem, dummy, &mut cmdline, type_id, &id)
device_manager.register_virtio_device(guest_mem, dummy, &mut cmdline, type_id, &id)
{
assert!(device_manager
.get_device(DeviceType::Virtio(type_id), &id)
Expand Down
Loading
Loading