diff --git a/crates/ring-agent/src/main.rs b/crates/ring-agent/src/main.rs index 4984c71b..5f71ec87 100644 --- a/crates/ring-agent/src/main.rs +++ b/crates/ring-agent/src/main.rs @@ -1,9 +1,15 @@ -//! ring-agent — in-guest companion to the Cloud Hypervisor runtime. +//! ring-agent — in-guest companion to Ring's micro-VM runtimes. //! //! Listens on AF_VSOCK port 2375 (well-known to the host-side client) and //! services length-prefixed JSON requests. Today the only supported request -//! is `Exec`, used to back `health_checks: [{ type: command, ... }]` on CH -//! deployments where Ring has no `docker exec` equivalent. +//! is `Exec`, used to back `health_checks: [{ type: command, ... }]` on +//! Cloud Hypervisor and Firecracker deployments, where Ring has no +//! `docker exec` equivalent. +//! +//! The same binary serves both: the guest side is plain AF_VSOCK either way. +//! Only the host differs — Cloud Hypervisor connects over kernel AF_VSOCK, +//! Firecracker multiplexes through a Unix socket and a `CONNECT ` +//! handshake before speaking this protocol. //! //! Wire format: //! request: [u32 BE length][JSON ExecRequest] diff --git a/documentation/concepts/runtimes.md b/documentation/concepts/runtimes.md index 7d1b0703..067c48ff 100644 --- a/documentation/concepts/runtimes.md +++ b/documentation/concepts/runtimes.md @@ -142,7 +142,7 @@ deployments: Ring copies the rootfs per instance (so replicas and reboots don't share guest state), spawns one `firecracker` process per VM bound to a private API socket, then drives Firecracker's REST API (`boot-source`, `drives`, `machine-config`, `actions`) to configure and boot the microVM. -**Networking.** A deployment that publishes `ports` gets a per-VM `/30` subnet (the same `10.42.x.y` scheme as Cloud Hypervisor), with host-port forwarding via `socat`. Unlike Cloud Hypervisor (which creates its own TAP), Firecracker expects the host TAP to already exist, so **Ring owns the TAP's whole lifecycle**: it creates the interface (via direct `ioctl`s, so the `CAP_NET_ADMIN` capability stays in-process), assigns the host side an IP, hands the device name to Firecracker, and deletes it on teardown. The guest IP is configured by cloud-init from a NoCloud datasource. Running `ring-server` therefore needs `CAP_NET_ADMIN` (or root) for any deployment with ports; grant it with `setcap cap_net_admin+ep $(command -v ring)`. +**Networking.** A deployment that publishes `ports` gets a per-VM `/30` subnet (the same `10.42.x.y` scheme as Cloud Hypervisor), with host-port forwarding via `socat`. Unlike Cloud Hypervisor (which creates its own TAP), Firecracker expects the host TAP to already exist, so **Ring owns the TAP's whole lifecycle**: it creates the interface (via direct `ioctl`s, so the `CAP_NET_ADMIN` capability stays in-process), assigns the host side an IP, hands the device name to Firecracker, and deletes it on teardown. The guest IP is configured by cloud-init from a NoCloud datasource. Running `ring-server` therefore needs `CAP_NET_ADMIN` (or root) for *every* Firecracker deployment — the TAP is allocated per microVM, whether or not it publishes ports, unlike Cloud Hypervisor which only creates one when ports are declared. Grant it with `setcap cap_net_admin+ep $(command -v ring)`. **Good fit when:** - You want VM-grade isolation with a smaller footprint and faster boot than Cloud Hypervisor diff --git a/documentation/runtimes/cloud-hypervisor.md b/documentation/runtimes/cloud-hypervisor.md index 0c265fe8..78d09fc0 100644 --- a/documentation/runtimes/cloud-hypervisor.md +++ b/documentation/runtimes/cloud-hypervisor.md @@ -54,6 +54,8 @@ You need all of these on the host: If you'd rather build it yourself: `cargo build -p ring-agent --release --target x86_64-unknown-linux-musl`. Either way, install it at `/usr/local/bin/ring-agent` in the guest and run it at boot via a systemd unit. It listens on AF_VSOCK port 2375. + The [Firecracker page](/documentation/runtimes/firecracker#command-health-checks) has a step-by-step recipe for installing the binary and its systemd unit into a disk image; it applies unchanged here, since the guest side is plain AF_VSOCK on both runtimes. + Run `ring doctor` to verify everything is in place; it checks each item and prints the missing pieces. ## Configure Ring diff --git a/documentation/runtimes/firecracker.md b/documentation/runtimes/firecracker.md index bea32475..62b92a19 100644 --- a/documentation/runtimes/firecracker.md +++ b/documentation/runtimes/firecracker.md @@ -21,6 +21,16 @@ A minimal KVM-backed **micro-VM**, the technology behind AWS Lambda. A tiny devi curl -sSL -o /var/lib/ring/firecracker/rootfs.ext4 "$BASE/ubuntu-22.04.ext4" ``` +3. **`CAP_NET_ADMIN` on `ring`.** Unlike Cloud Hypervisor, Firecracker expects the host TAP to already exist, so Ring creates and deletes it itself: + + ```bash + sudo setcap cap_net_admin+ep $(command -v ring) + ``` + + The capability goes on the **Ring binary**, not on `firecracker`. This runtime allocates a TAP for *every* microVM, not just those publishing `ports` (Cloud Hypervisor only creates one when ports are declared), so without the capability no Firecracker deployment boots at all — each fails with `could not create tap '…': operation not permitted`. `setcap` does not survive a rebuild or upgrade, so re-run it after replacing the binary. + +4. **`ring-agent` inside the guest image** (only if you use `health_checks: [{ type: command, ... }]`). See [Command health checks](#command-health-checks) below. + ## Enable it ```toml @@ -73,6 +83,78 @@ Console logs are rotated once they cross `max_console_log_bytes` (10 MiB by defa Per-instance CPU, memory, network, disk I/O, and thread counts are exposed at `GET /deployments/{id}/metrics`, the same as every other runtime. Ring reads them host-side from the `firecracker` process (`/proc//{stat,status,io}`) and the per-VM tap counters, with no in-guest agent required. Memory `usage_percent` is reported against the deployment's memory limit; network counters read zero for deployments that publish no ports (no tap is created). +## Health checks + +`tcp` and `http` probe from the host against the guest IP, with nothing needed inside the VM. + +### Command health checks + +`command` checks have no `docker exec` equivalent on a VM, so they run through **`ring-agent`**, a small static binary you install in the guest image. It listens on AF_VSOCK port 2375; Ring reaches it through the VM's vsock device and runs the command via `/bin/sh -c`. + +Each Ring release attaches a static musl build: + +```bash +TAG=$(curl -s https://api.github.com/repos/kemeter/ring/releases/latest | grep -oP '"tag_name": "\K[^"]+') +curl -L "https://github.com/kemeter/ring/releases/download/${TAG}/ring-agent-${TAG}-x86_64-unknown-linux-musl.tar.gz" \ + | tar -xz +``` + +Or build it yourself: `cargo build -p ring-agent --release --target x86_64-unknown-linux-musl`. + +Install it into the rootfs at `/usr/local/bin/ring-agent` and start it at boot. Mounting the image on the host is the simplest route: + +```bash +sudo mount -o loop rootfs.ext4 /mnt +sudo install -m 0755 ring-agent /mnt/usr/local/bin/ring-agent +sudo tee /mnt/etc/systemd/system/ring-agent.service > /dev/null <<'EOF' +[Unit] +Description=Ring in-guest agent +After=network.target + +[Service] +ExecStart=/usr/local/bin/ring-agent +Restart=always + +[Install] +WantedBy=multi-user.target +EOF +sudo ln -sf /etc/systemd/system/ring-agent.service \ + /mnt/etc/systemd/system/multi-user.target.wants/ring-agent.service +sudo umount /mnt +``` + +The symlink is what enables the unit. `sudo systemctl --root=/mnt enable ring-agent.service` does the same thing if the host's systemd is recent enough; the explicit symlink avoids depending on that. + +Without root, `debugfs` from `e2fsprogs` writes into the ext4 image directly: + +```bash +cat > ring-agent.service <<'EOF' +[Unit] +Description=Ring in-guest agent +After=network.target + +[Service] +ExecStart=/usr/local/bin/ring-agent +Restart=always + +[Install] +WantedBy=multi-user.target +EOF + +debugfs -w rootfs.ext4 <<'EOF' +write ring-agent /usr/local/bin/ring-agent +sif /usr/local/bin/ring-agent mode 0100755 +write ring-agent.service /etc/systemd/system/ring-agent.service +symlink /etc/systemd/system/multi-user.target.wants/ring-agent.service /etc/systemd/system/ring-agent.service +EOF +``` + +`write` doesn't preserve the source's mode, hence the explicit `sif`. Check the result with `debugfs -R "ls -l /usr/local/bin" rootfs.ext4`. + +The same binary and unit work on Cloud Hypervisor: the guest side is plain AF_VSOCK on both runtimes, and only the host transport differs. + +> **A `command` check added to a running deployment needs a VM restart.** The vsock device is attached at boot, and only when the deployment already declares such a check — Firecracker has no hot-plug path for it. Until the VM restarts, the probe cannot reach the guest. Whether it restarts on its own depends on the check's `on_failure`: `restart` heals itself once the failure threshold is reached, while `alert` and `stop` never reboot the VM. Declaring the check before the first boot avoids the situation entirely. + ## Jobs (`kind: job`) A `kind: job` deployment boots a single microVM (replicas are ignored) and is marked **`completed`** once the guest finishes. Firecracker exposes no VM-state API, so completion is signalled by the **guest rebooting**: with the default `reboot=k` kernel cmdline, a guest `reboot` is trapped by Firecracker and exits the VMM cleanly. Ring's next scheduler tick sees the process gone and finalizes the deployment. @@ -103,6 +185,8 @@ Bind and config/secret images are ephemeral and reaped when the instance stops; ## Known gaps (experimental) - `image:` must be a host rootfs file, with no registry pull. +- `command` health checks need `ring-agent` installed in the guest image, and the vsock device that carries them is attached at boot only (see [Command health checks](#command-health-checks)). +- `labels:` are stored and filterable as Ring metadata, but not applied to the VM. ## See also