From 7547cc846c00c198be73f0d1c91e0ad9c73b95c9 Mon Sep 17 00:00:00 2001 From: Mlanawo MBECHEZI Date: Sat, 25 Jul 2026 09:12:01 +0300 Subject: [PATCH] feat(microvm): apply outbound NAT and host-memory admission on both VM runtimes --- documentation/concepts/runtimes.md | 3 ++- documentation/runtimes/cloud-hypervisor.md | 10 +++++++++- documentation/runtimes/firecracker.md | 2 ++ src/runtime/cloud_hypervisor/lifecycle.rs | 21 +++++++++++++++------ src/runtime/firecracker/lifecycle.rs | 7 +++++++ 5 files changed, 35 insertions(+), 8 deletions(-) diff --git a/documentation/concepts/runtimes.md b/documentation/concepts/runtimes.md index 41a505f..7d1b070 100644 --- a/documentation/concepts/runtimes.md +++ b/documentation/concepts/runtimes.md @@ -24,7 +24,8 @@ Enable just the runtimes a host actually runs: Docker-only, Podman-only, contain | Boot time | ~1 s | ~1 s | ~1 s | ~3–5 s (cloud-init, kernel boot) | ~1 s (minimal microVM) | | Memory overhead per workload | ~10 MB | ~10 MB | ~10 MB | ~80–150 MB (kernel + guest userland) | ~5–50 MB (minimal device model) | | Image format | Docker image (`nginx:1.25`) | Docker/OCI image | OCI image (`nginx:1.25`) | Raw disk image on the host filesystem | Kernel (`vmlinux`) + ext4 rootfs on the host filesystem | -| Networking | Per-namespace bridge, DNS aliases | Per-namespace bridge, DNS aliases | CNI (bridge + host-local IPAM) | Per-VM /30 subnet, host-port forwarding via `socat` | Per-VM /30 subnet, host-port forwarding via `socat` (Ring-owned host TAP) | +| Networking | Per-namespace bridge, DNS aliases | Per-namespace bridge, DNS aliases | CNI (bridge + host-local IPAM) | Per-VM /30 subnet, host-port forwarding via `socat`, Ring-managed outbound NAT | Per-VM /30 subnet, host-port forwarding via `socat` (Ring-owned host TAP), Ring-managed outbound NAT | +| Host-memory admission | N/A (cgroup limits) | N/A (cgroup limits) | ✓ before create | ✓ before VM boot | ✓ before rootfs copy and VM boot | | Crash detection | ✓ event-driven (sub-second) | ✓ reconcile-based (per scheduler tick) | ✓ reconcile-based (per scheduler tick) | ✓ reconcile-based (per scheduler tick) | ✓ reconcile-based (per scheduler tick) | | `command` health checks | `docker exec` | `podman exec` (same API) | `Tasks.Exec` (gRPC) | In-guest `ring-agent` over AF_VSOCK | In-guest `ring-agent` over vsock (host Unix socket) | | `kind: job` | Exit code visible | Exit code visible | Exit code visible | Clean shutdown = success (no exit code from host) | Clean shutdown (guest reboot) = success (no exit code from host) | diff --git a/documentation/runtimes/cloud-hypervisor.md b/documentation/runtimes/cloud-hypervisor.md index 2807882..0c265fe 100644 --- a/documentation/runtimes/cloud-hypervisor.md +++ b/documentation/runtimes/cloud-hypervisor.md @@ -196,6 +196,14 @@ If `published` is already taken on the host, Ring **refuses to start the VM** an Both `tcp` (default) and `udp` are supported via the port's `protocol` field; `socat` uses a `UDP4-LISTEN`/`UDP4` pair for UDP. +### Outbound access + +The per-VM /30 lets the guest reach the host, but reaching *external* networks (package installs, `git clone`, …) needs the host to masquerade the guest subnet. Ring installs that itself — a single idempotent rule set covering `10.42.0.0/16`, so it is set up once and re-asserted for free on later boots. The operator never touches `iptables`. + +This requires `CAP_NET_ADMIN` on the Ring binary (`setcap cap_net_admin+ep $(command -v ring)`). Without it the rules are skipped: the VM still boots and serves inbound traffic, but outbound calls fail to resolve. The failure is logged at warn on the first VM start. + +Note that the guest network is only allocated when the deployment publishes at least one port; a VM with no `ports:` has no network at all. + ## Health checks `tcp`, `http`, `command` all work. `tcp` and `http` probe from the host against the guest IP (no agent required). `command` goes through the in-guest `ring-agent` over AF_VSOCK port 2375, so install the agent in the guest image. If the agent isn't reachable (missing from the image, or not started yet), the `command` probe fails with an explicit message naming ring-agent rather than a bare connection error. @@ -247,7 +255,7 @@ This is the canonical parity table. Other pages link here rather than restate it | `labels:` | **Stored and usable.** Not applied to the VM (no container-label equivalent), but persisted as Ring metadata (shown in `inspect` and filterable with `ring deployment list --label key=value`), same as Docker | | `resources.limits.cpu` | Honored as **allocation, not cap**: rounded down to whole vCPU, floor 1 (`"500m"` → 1 vCPU) | | `resources.limits.memory` | Honored as **allocation, not cap**: VM RAM size, minimum 128 MiB | -| `resources.requests.*` | Ignored (VM is sized from `limits`); a **warning event** is recorded at create so it isn't silent | +| `resources.requests.*` | Not used to **size** the VM (that comes from `limits`); a **warning event** is recorded at create so it isn't silent. `requests.memory` is still read for host-memory admission control, where it takes precedence over `limits.memory` | | `config.image_pull_policy` / `server` / `username` / `password` | Ignored (no image to pull); a **warning event** is recorded at create | | `config.user` (privileged / id / group) | Ignored; a **warning event** is recorded at create | | `kind: job` | **Supported, coarser signal.** Clean guest shutdown → `completed`. CH does not expose the workload's exit code, so Ring sees VM state only. | diff --git a/documentation/runtimes/firecracker.md b/documentation/runtimes/firecracker.md index 33f08ca..bea3247 100644 --- a/documentation/runtimes/firecracker.md +++ b/documentation/runtimes/firecracker.md @@ -55,6 +55,8 @@ ring apply -f app.yaml What Ring does: copies the rootfs per instance, spawns a `firecracker` process, drives its REST API to set the kernel / rootfs / network / machine config, then boots. Networking is a Ring-owned TAP (a /30 subnet per VM) with `socat` host-port forwarding; outbound NAT lets guests reach external networks. +Before any of that, the deployment's memory ask is admitted against the host's available memory. A microVM reserves its whole RAM at boot, so an over-ask would otherwise die on an opaque allocation failure — and only after a full rootfs copy. The check reads `resources.requests.memory`, falling back to `resources.limits.memory`; a deployment declaring neither is not gated. Refusal is terminal (`insufficient_resources`) rather than a crash loop, since a retry won't free memory. + ## Logs The guest serial console (kernel, init, and anything the workload writes to the console) is persisted per instance and readable with the standard commands, same as every other runtime: diff --git a/src/runtime/cloud_hypervisor/lifecycle.rs b/src/runtime/cloud_hypervisor/lifecycle.rs index 808ce76..54e13f6 100644 --- a/src/runtime/cloud_hypervisor/lifecycle.rs +++ b/src/runtime/cloud_hypervisor/lifecycle.rs @@ -639,12 +639,21 @@ impl CloudHypervisorLifecycle { } let net_config = match &net_alloc { - Some(n) => NetConfig { - tap: Some(n.tap_name.clone()), - ip: Some(n.host_ip.clone()), - mask: Some(n.netmask.clone()), - mac: Some(n.mac.clone()), - }, + Some(n) => { + // The tap CH is about to create lets the guest reach the host; + // outbound NAT is what lets it reach the Internet (apt, git + // clone, …). Same call Firecracker makes after creating its own + // tap: idempotent, scoped to the whole guest supernet, so it is + // a no-op after the first VM on this host. Without it a CH guest + // resolved nothing, while an equivalent Firecracker guest did. + crate::hypervisor::host_nat::ensure_outbound_nat(); + NetConfig { + tap: Some(n.tap_name.clone()), + ip: Some(n.host_ip.clone()), + mask: Some(n.netmask.clone()), + mac: Some(n.mac.clone()), + } + } None => NetConfig { tap: None, ip: None, diff --git a/src/runtime/firecracker/lifecycle.rs b/src/runtime/firecracker/lifecycle.rs index b343274..48cd664 100644 --- a/src/runtime/firecracker/lifecycle.rs +++ b/src/runtime/firecracker/lifecycle.rs @@ -393,6 +393,13 @@ impl FirecrackerLifecycle { ))); } + // Admission control before the rootfs copy and the VM boot. A microVM + // reserves its whole memory at boot, so an over-ask dies on an opaque + // allocation failure — and here it would do so only *after* copying a + // full rootfs image. Same check Cloud Hypervisor already ran; without it + // Firecracker had no memory gate at all. + crate::hypervisor::resources::check_host_memory(deployment)?; + std::fs::create_dir_all(&self.config.socket_dir).map_err(|e| { RuntimeError::VmStartFailed(format!( "could not create socket_dir '{}': {}",