feat(vm): add openshell-vm microVM gateway backend (opt-in via NEMOCLAW_GATEWAY_BACKEND=vm)#1791
Closed
ericksoa wants to merge 47 commits into
Closed
feat(vm): add openshell-vm microVM gateway backend (opt-in via NEMOCLAW_GATEWAY_BACKEND=vm)#1791ericksoa wants to merge 47 commits into
ericksoa wants to merge 47 commits into
Conversation
OpenShell v0.0.26 ships openshell-vm, a standalone binary that boots a hardware-isolated microVM via libkrun. This commit lays the groundwork for NemoClaw to support both the existing Docker/k3s gateway and the new microVM gateway as selectable backends. Phase 1 changes: - Bump min_openshell_version from 0.0.24 to 0.0.26 across blueprint, install script, onboard preflight, CI scripts, E2E tests, and docs - Add gateway_backends field to blueprint.yaml schema (docker, vm) - Add isOpenshellVmAvailable() and getInstalledOpenshellVmVersion() to openshell.ts for detecting the openshell-vm binary - Add detectGatewayBackend() to platform.ts with NEMOCLAW_GATEWAY_BACKEND env var override, auto-detection preferring VM when available and falling back to Docker, and mandatory Docker for GPU workloads - Add gatewayBackend field to onboard session schema for persisting the selected backend across resume cycles - Add tests for all new functions The VM backend requires no Docker daemon and provides faster boot, but has no NVIDIA GPU passthrough (libkrun lacks PCI/VFIO support), so the Docker backend remains mandatory for local inference on GPU workstations. Refs: NVIDIA/OpenShell#611 Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
🚀 Docs preview ready! |
Phase 4 of the dual-backend feature (PR #1791): - New test/e2e/test-vm-backend-e2e.sh: full E2E journey for the VM backend — install openshell-vm from release assets, onboard with NEMOCLAW_GATEWAY_BACKEND=vm, verify sandbox creation, live inference through the microVM gateway, resume after openshell-vm kill, and reset to clean slate. - New vm-e2e job in nightly-e2e.yaml: runs on ubuntu-latest (has /dev/kvm), installs openshell-vm, executes the VM backend E2E test. - New vm-backend test suite in brev-e2e.test.ts: allows running the VM backend E2E on ephemeral Brev instances via TEST_SUITE=vm-backend. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The openshell-vm binary is published on the vm-dev pre-release tag (not v0.0.26) with gnu libc (not musl). Also downloads the VM runtime (kernel + rootfs) needed by libkrun, and installs zstd for decompression. Asset corrections: - Tag: vm-dev (not v0.0.26) - Binary: openshell-vm-*-unknown-linux-gnu.tar.gz (not musl) - Checksums: vm-binary-checksums-sha256.txt - Runtime: vm-runtime-linux-*.tar.zst → ~/.local/share/openshell-vm/ Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
When detectGatewayBackend() returns "vm", the onboard and runtime recovery flows now use openshell-vm instead of Docker: - startGatewayWithOptions() detects the backend and delegates to startVmGatewayProcess() for the VM path, which spawns openshell-vm as a detached background process with PID tracking - VM lifecycle helpers: writeVmPidFile, readVmPid, isVmProcessAlive, stopVmGateway (SIGTERM→SIGKILL), isVmGatewayHealthy - destroyGateway() checks session.gatewayBackend and stops the VM process instead of Docker volume cleanup when backend is "vm" - recoverGatewayRuntime() reads session.gatewayBackend to choose VM vs Docker recovery path - recoverNamedGatewayRuntime() in nemoclaw.ts skips Docker-specific gateway select commands for VM backend - cleanupGatewayAfterLastSandbox() stops VM process instead of Docker cleanup when backend is "vm" - Gateway backend is saved to onboard session on step completion so resume cycles know which backend to use - Resume flow checks VM health via isVmGatewayHealthy() instead of Docker gateway state when session records VM backend Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Reading the openshell-vm source (NVIDIA/OpenShell crates/openshell-vm)
revealed three bugs in the Phase 2 implementation:
1. openshell-vm was spawned with no arguments. It needs `--name nemoclaw`
so it extracts the rootfs to the correct instance directory and
registers gateway metadata under the right identity.
2. openshell-vm prefixes instance names: gateway_name("nemoclaw")
produces "openshell-vm-nemoclaw". All OPENSHELL_GATEWAY env vars
and `openshell gateway select` calls for the VM path now use
VM_GATEWAY_NAME ("openshell-vm-nemoclaw") instead of GATEWAY_NAME.
3. Health poll was too short (15 × 2s = 30s). The VM boots k3s inside
a microVM with its own 90s internal health check. Increased to
60 × 3s = 180s to avoid racing the inner bootstrap. Also log the
last 10 lines from openshell-vm.log on failure for diagnostics.
Additionally: the VM gateway listens on port 30051 (NodePort), not
8080. The openshell-vm binary handles the port mapping internally
(gvproxy host:30051 → VM:30051 → kube-proxy → pod:8080).
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The GitHub-hosted ubuntu runner has /dev/kvm (crw-rw---- root:kvm) but the runner user is not in the kvm group. openshell-vm opens /dev/kvm directly via libkrun and fails with EACCES. Fix by chmod 666 /dev/kvm in the KVM verification step. Also add the user to the kvm group for completeness, though the chmod is sufficient for the current process. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…tics Three changes to address gvproxy crash on GitHub Actions: 1. Pass --mem 4096 to openshell-vm. The default 8GB is half the 16GB runner memory. With k3s pulling container images inside the VM, host processes (gvproxy, gvisor netstack) get starved. 4GB is enough for a lightweight gateway without GPU workloads. 2. Detect and use the E2E-downloaded VM runtime via OPENSHELL_VM_RUNTIME_DIR. The test script downloads gvproxy/libkrun/libkrunfw from the vm-dev release to ~/.local/share/openshell-vm/ but never tells openshell-vm to use it. The downloaded runtime may contain fixes not in the binary's embedded copy. When the downloaded runtime exists (has gvproxy), set OPENSHELL_VM_RUNTIME_DIR to prefer it. 3. Add VM diagnostics step to CI: openshell-vm log, gvproxy log, dmesg OOM check, memory stats, and VM console log. This will show the actual root cause if gvproxy crashes again. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The openshell-vm kernel has CONFIG_POSIX_MQUEUE=y but the init script
(openshell-vm-init.sh) never mounts the mqueue filesystem. When k3s
creates a pod, runc tries to mount mqueue at /dev/mqueue inside the
container namespace and gets ENODEV ("no such device") because the
host mount point doesn't exist.
Fix: run `openshell-vm prepare-rootfs` to extract the rootfs, then
patch the init script to mkdir + mount mqueue alongside the existing
devpts/shm mounts. The patch is idempotent — skipped if the init
script already contains /dev/mqueue.
Root cause found by tracing the VM console log:
runc create failed: error mounting "mqueue" to rootfs at
"/dev/mqueue": no such device
This should be fixed upstream in the OpenShell init script.
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The previous mqueue patch ran silently when prepare-rootfs failed or the init script wasn't found. Add verbose logging at each step so CI output shows exactly what happened: rootfs path, init script location, whether the string replacement matched, and any errors. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The vm-dev kernel (built 2026-04-09) predates the CONFIG_POSIX_MQUEUE addition to the kconfig (2026-04-10 d8cf7951). runc fails with ENODEV when trying to mount -t mqueue inside container namespaces. Previous approach (mounting mqueue in init script) didn't work because the kernel itself lacks the filesystem type — mount returns ENODEV. New approach: install a runc wrapper shim that strips mqueue mount entries from the OCI config.json before calling the real runc binary. The shim is only installed when the kernel actually lacks mqueue support (tested by attempting mount -t mqueue). When a future kernel rebuild includes CONFIG_POSIX_MQUEUE, the shim is not installed. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…pt breakage The previous approach embedded a heredoc in the init script patch, which broke the init script (VM died before exec agent started). New approach: write runc-wrapper.sh as a separate file in /opt/nemoclaw/ of the rootfs, then patch the init script with a simple test-and-swap block. The wrapper is a standalone script that strips mqueue entries from config.json via sed before calling the real runc binary. The init script patch is minimal — just an if/fi block that copies the wrapper over runc when the kernel lacks mqueue. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…shim Previous approach of wrapping /usr/bin/runc didn't work because k3s bundles its own runc in /var/lib/rancher/k3s/data/<hash>/bin/, extracted fresh at each startup. Our host-side wrapper was never called. New approach: write /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl in the init script, which k3s reads before generating its containerd config. The template sets BinaryName to /opt/nemoclaw/runc-shim, which strips mqueue mount entries from config.json before calling the k3s-bundled runc. This routes ALL runc invocations through the shim, correctly handling the k3s data directory extraction timing. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
When the session's gatewayBackend is "vm", the standard `openshell sandbox create --from Dockerfile` path fails because the openshell CLI internally uses Docker put_archive to push the image into the gateway container — which doesn't exist for VM gateways. Fix: detect VM backend during sandbox creation and: 1. Build the image locally with Docker (same Dockerfile) 2. Export with docker save to the VM's rootfs via virtio-fs (host can write directly to the rootfs/tmp/ directory) 3. Import into VM containerd via openshell-vm exec + ctr images import 4. Pass --from <image-ref> instead of --from <Dockerfile> so the openshell CLI treats it as a pre-existing image (skips Docker push) Falls back to the standard Dockerfile path if any step fails, so Docker backend behavior is unchanged. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…erlay) Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…g issues Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Two post-sandbox fixes for VM backend: 1. DNS proxy: setup-dns-proxy.sh uses docker exec to reach kubectl inside the gateway container. VM backend has no Docker container. Skip the DNS proxy — gvproxy provides NAT networking with working DNS through the gateway IP (192.168.127.1). 2. Sandbox readiness: the sandbox pod may briefly flip Ready→NotReady during init container restarts in the VM. Add a 30s wait loop in setupOpenclaw before attempting sandbox connect, preventing the FailedPrecondition "sandbox is not ready" error. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
The vm-dev openshell-sandbox binary was built with cargo-zigbuild on a host with glibc 2.39, but sandbox containers use Ubuntu 22.04 (glibc 2.35). The binary crashes at startup: GLIBC_2.38 not found (required by /opt/openshell/bin/openshell-sandbox) Fix: extract the openshell-sandbox binary from the Docker gateway image (ghcr.io/nvidia/openshell/cluster:<version>), which was built with rust:1.88-slim (Debian bookworm, glibc 2.36 — compatible with Ubuntu 22.04's glibc 2.35). Replace the VM rootfs copy before boot. The supervisor is side-loaded into sandbox pods via hostPath volume from /opt/openshell/bin on the k3s node, so this fix propagates to all sandbox pods automatically. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
waitForSandboxReady() used `openshell doctor exec -- kubectl` which runs docker exec inside the gateway container. This is Docker-specific and fails silently for VM gateways (no Docker container exists). For VM backend, use `openshell sandbox list` + isSandboxReady() which goes through the gRPC API and works for both Docker and VM gateways. Docker backend behavior is unchanged. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…ateway When openshell-vm is killed after a successful onboard, the session is marked complete (resumable=false). Running `nemoclaw onboard --resume` correctly reports "No resumable session found" — but the E2E test expects resume to restart the gateway and reconnect the sandbox. Fix: when --resume finds a completed session with gatewayBackend=vm and the VM gateway is dead, restart openshell-vm and check if the sandbox is still alive. If the sandbox reconnects after gateway restart, exit 0 (recovery successful). If not, mark the session resumable and fall through to the normal re-onboard path. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
…ng resume Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
GPU inference is routed through inference.local (OpenShell L7 proxy → host inference server). The sandbox and gateway never need direct GPU access — the GPU is used by the host-side Ollama/vLLM/NIM process. The VM backend works for all scenarios including local GPU inference. Remove the gpuRequested parameter and the conditional that forced Docker when GPU was detected. VM is now preferred whenever openshell-vm is available, regardless of GPU presence. Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Merged
3 tasks
ericksoa
added a commit
that referenced
this pull request
Apr 16, 2026
… version field Refactor test/e2e/brev-e2e.test.ts for issue #1390: - Extract named helpers from the ~350-line monolithic beforeAll: cleanupLeftoverInstance(), createBrevInstance(), refreshAndWaitForSsh(), bootstrapLaunchable(), pollForSandboxReady(), writeManualRegistry(). The beforeAll now reads as a high-level orchestration (~50 lines). - Deduplicate brev create + error recovery: the deploy-cli and launchable paths shared duplicated brev refresh + waitForSsh patterns, now consolidated into refreshAndWaitForSsh(). - Remove phantom version: 1 from the manual registry write. The SandboxRegistry interface in src/lib/registry.ts has no version field; registerSandbox() doesn't write one either. - bootstrapLaunchable() returns { remoteDir, needsOnboard } instead of mutating module-level state as a hidden side-effect. - instanceCreated is set at call sites in beforeAll, not hidden inside createBrevInstance(). - Remove dead sleep() helper (defined but never called). Pure refactoring — no behavior changes. // @ts-nocheck pragma preserved. Note: PR #1791 (openshell-vm microVM) also touches this file — whoever merges second will need a rebase.
ericksoa
added a commit
that referenced
this pull request
Apr 16, 2026
…nstaller detection (#1888) ## Summary Addresses all three items from #1390, plus a bonus installer fix discovered while running the pre-push hooks. ### Issue #1390 — Brev E2E cleanup - **Extract helpers from the monolithic beforeAll**: The ~350-line `beforeAll` block is now ~50 lines of high-level orchestration calling named helpers: `cleanupLeftoverInstance()`, `createBrevInstance()`, `refreshAndWaitForSsh()`, `bootstrapLaunchable()`, `pollForSandboxReady()`, `writeManualRegistry()`. - **Deduplicate brev create + error recovery**: The deploy-cli and launchable paths shared duplicated `brev refresh` + `waitForSsh` patterns, now consolidated into `refreshAndWaitForSsh()`. - **Remove phantom `version: 1`** from the manual registry write. The `SandboxRegistry` interface in `src/lib/registry.ts` has no version field; `registerSandbox()` doesn't write one either. Additional cleanup: - `bootstrapLaunchable()` returns `{ remoteDir, needsOnboard }` instead of mutating module-level state as a hidden side-effect. - `instanceCreated` is set at call sites in `beforeAll`, not hidden inside `createBrevInstance()`. - Dead `sleep()` helper removed. Pure refactoring — no behavior changes. `// @ts-nocheck` pragma preserved. ### Installer worktree fix `scripts/install.sh` used `-d "${repo_root}/.git"` to detect source checkouts, but in a git worktree `.git` is a file, not a directory. This caused `is_source_checkout()` to return false, falling through to the GitHub clone path. Fixed by using `-e` (exists) instead of `-d` (is directory). This resolved 12 pre-existing test failures in `test/install-preflight.test.ts`. ## Related Issue Closes #1390 ## Note PR #1791 also touches `test/e2e/brev-e2e.test.ts` (appends a new `vm-backend` test case). Clean merge — whoever merges second rebases trivially. ## Type of Change - [x] Code change for a new feature, bug fix, or refactor. ## Testing - [x] `npx prek run --all-files` passes (all pre-commit and pre-push hooks green). - [x] `npx vitest run --project cli` — 1213 passed, 0 failed (was 12 failed before installer fix). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Enhanced end-to-end test infrastructure and setup orchestration for improved test reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
Merge main into feat/openshell-vm-backend, keeping both: - VM backend features (openshell-vm lifecycle, K8s manifest, E2E job) - Main branch additions (sandbox-operations, inference-routing, snapshot, shields-config, rebuild, upgrade-stale-sandbox, hermes rebuild E2E jobs, stale gateway container detection, LOCAL_INFERENCE_PROVIDERS, version pinning, pruneKnownHostsEntries)
The VM backend K8s manifest used privileged: true and curl|sh, which fails the security-configuration-hardening test. Restore main's Docker-in-Docker manifest with proper security context and add a comment pointing to VM backend docs.
Revert the Docker-based manifest restoration — the point of this PR is the microVM backend. Restore the VM manifest with pod-level hardening from main (automountServiceAccountToken, enableServiceLinks, secretKeyRef for COMPATIBLE_API_KEY, suggested policy mode). Update the security-configuration-hardening test to branch on backend type: VM backend validates privileged: true for KVM, Docker backend validates the full container-level lockdown and secure download pattern.
Swap detectGatewayBackend priority so Docker is preferred when both runtimes are available. Users can opt into the VM backend by setting NEMOCLAW_GATEWAY_BACKEND=vm — the env override is already wired through detectGatewayBackend and the E2E job sets it. Restore the Docker-based K8s manifest and security test since Docker is the default path. All VM backend code remains intact and exercised by the vm-e2e nightly job.
ericksoa
changed the title
OpenShell v0.0.28+ includes CONFIG_POSIX_MQUEUE=y in the VM kernel (commit d8cf7951), so the runc shim that replaced mqueue with tmpfs is no longer needed. Remove ~80 lines of workaround code. The glibc supervisor workaround stays — upstream still builds openshell-sandbox against glibc 2.39 (Ubuntu 24.04) but sandbox containers use Ubuntu 22.04 (glibc 2.35). Version pin changes: - max_openshell_version: 0.0.26 → 0.0.32 - install-openshell.sh MAX_VERSION: 0.0.26 → 0.0.32 - brev-launchable-ci-cpu.sh default: v0.0.26 → v0.0.32 - test-vm-backend-e2e.sh MIN_OPENSHELL: 0.0.26 → 0.0.32 - glibc fallback image tag: 0.0.26 → 0.0.32
The vm-dev runtime artifacts still ship without CONFIG_POSIX_MQUEUE despite the source fix (d8cf7951). Every container inside the VM fails with "error mounting mqueue: no such device" without the shim. Restore the self-disabling workaround. The version bump to 0.0.32 is kept — it brings security improvements and the shim will no-op once the vm-dev kernel is actually rebuilt.
A cancelled worktree agent extracted provider code into a new onboard-providers.ts module and rewrote imports in onboard.ts. This introduced a duplicate `getSandboxInferenceConfig` identifier (imported from the new module AND defined locally) causing a runtime SyntaxError that broke `nemoclaw --version` and all E2E jobs. Fix: restore onboard.ts from the clean merge commit (878ef15) and re-apply only the glibc fallback version bump (0.0.26 → 0.0.32). The mqueue shim and all VM lifecycle code are intact.
The inference route can take a few seconds to come back after the VM gateway restarts. Add a 3-attempt retry with 10s pauses to avoid flaky failures on the post-resume PONG check.
When the VM gateway is killed and --resume recreates it, the new gateway has no provider/route state from the old one. But the resume logic was checking isInferenceRouteReady() which could return stale metadata, causing the inference step to be skipped. The sandbox then has no working inference.local route. Track when the gateway is recreated during resume and force inference re-registration in that case. Also add diagnostics to the E2E test (SSH check, raw response, route/provider state on failure).
When the VM gateway dies and --resume recreates it, the old sandbox pod state persists on the VM's disk. openshell sandbox list reports it as "ready" (from recovered k8s metadata) but the pod isn't actually functional — SSH fails with "sandbox is not ready". Extend the gatewayRecreatedDuringResume guard to also skip the sandbox reuse check, forcing full sandbox recreation alongside the inference provider re-registration.
k3s bootstrap inside the microVM can fail due to internal race conditions (e.g. cloud-controller-manager starting before required configmaps exist). When this happens the VM process dies and gvproxy loses its virtio-net socket. Instead of failing immediately, retry up to 3 times (configurable via NEMOCLAW_VM_START_ATTEMPTS). Each retry stops the dead VM process and spawns a fresh one. Log tail is printed between attempts for diagnostics.
setupInference hardcodes `gateway select nemoclaw` which resets the active gateway away from `openshell-vm-nemoclaw` when the VM backend is in use. This causes provider creation to fail with "No gateway metadata found for 'nemoclaw'" and leaves the sandbox with no inference route, no providers, and broken SSH. Add getEffectiveGatewayName() that returns the correct gateway name based on the active backend. Use it in setupInference instead of the hardcoded GATEWAY_NAME constant.
The VM resume had a fast-path that short-circuited the entire resume flow: restart VM, check if sandbox metadata says "ready", wait for inference route, exit 0. This assumed k3s state survives an unclean VM shutdown — but it doesn't reliably. Providers, routes, and sandbox pod readiness are all stale after VM kill. Replace with: mark all session steps as pending and fall through to the normal resume path, which re-runs gateway startup, inference provider registration, and sandbox creation from scratch. This is the same path used for Docker gateway recovery and is known to work.
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
Closing as superseded by #3001. |
wscurran
added
area: packaging
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds openshell-vm (libkrun microVM) as an opt-in alternative gateway backend. Docker remains the default. Users enable the VM backend by setting
NEMOCLAW_GATEWAY_BACKEND=vm— no documented flags or CLI changes.Also bumps the OpenShell version pin from 0.0.26 to 0.0.32, picking up seccomp hardening, Landlock fixes, SSRF protection, deny rules in the policy schema, and standalone binary publishing.
All VM backend code is exercised nightly by the
vm-e2eCI job (30/30 passing on ubuntu-latest with KVM). The Docker path is unchanged and all existing E2E jobs continue to pass.How it works
detectGatewayBackend()checks, in order:NEMOCLAW_GATEWAY_BACKENDenv var —"vm"or"docker"overrides everythingopenshell-vmbinary in PATH → use VM"unknown"(onboard fails with guidance)When VM is selected,
onboard.tsspawnsopenshell-vm --name nemoclaw --mem 4096as a detached process, tracks its PID, polls gRPC health, and manages the full lifecycle (start, health check, resume, cleanup). The sandbox image is built withdocker buildon the host, exported viadocker save, written into the VM rootfs via virtio-fs, and imported into the VM's containerd viactr images import.Benefits of the VM backend
NEMOCLAW_GATEWAY_BACKEND=vm)/dev/kvmand theopenshell-vmbinary. No Docker daemon for the gateway--mem)inference.local(OpenShell L7 proxy → host), so the gateway never touches the GPU directlyTradeoffs and limitations
/dev/kvm, which means bare-metal or nested virtualization. Not available in all cloud VMs or container runtimes.docker build+docker saverun on the host to create the sandbox image. This is standard CI usage, not Docker-in-Docker, but it means Docker isn't fully eliminated.OpenShell version bump: 0.0.26 → 0.0.32
This PR bumps
max_openshell_versionand the install pin from 0.0.26 to 0.0.32. Key improvements in the new range:CONFIG_POSIX_MQUEUE(d8cf7951) — runtime not yet rebuilt, shim still neededopenshell-sandboxbinaries published, system CA cert supportWorkarounds (both self-disabling)
1. mqueue runc shim (still needed)
The vm-dev kernel runtime artifacts have not been rebuilt since
CONFIG_POSIX_MQUEUE=ywas added to the source (d8cf7951, 2026-04-10). Without mqueue support, every container inside the VM fails witherror mounting "mqueue" to rootfs: no such device.Fix: Write a containerd config that routes runc through a shim script. The shim edits each container's config.json to use tmpfs instead of mqueue.
Self-disables: The init script tests
mount -t mqueueat boot. If the kernel supports it, the shim is never installed. Once the vm-dev runtime is rebuilt with the kernel fix, this becomes a no-op.2. Supervisor glibc extraction (still needed)
The
openshell-sandboxsupervisor binary is built against glibc 2.39 (Ubuntu 24.04) but sandbox containers use Ubuntu 22.04 (glibc 2.35). It crashes withGLIBC_2.38 not found.Fix: Extract the compatible binary from the Docker gateway image at onboard time (~7s, once).
Self-disables: Only runs if Docker is available. Overwrites with an equivalent binary if upstream fixes the build target.
What changed (17 files, ~+1500/-40)
src/lib/platform.tsdetectGatewayBackend()— Docker preferred, VM via env overridesrc/lib/onboard.tssrc/lib/openshell.tsisOpenshellVmAvailable(),getInstalledOpenshellVmVersion()src/lib/onboard-session.tsgatewayBackendfield in session (persists choice across resume)src/nemoclaw.tsstopVmGateway),pruneKnownHostsEntriesimportnemoclaw-blueprint/blueprint.yamlgateway_backends: [docker, vm], max version bump to 0.0.32schemas/blueprint.schema.jsongateway_backendsscripts/install-openshell.shscripts/brev-launchable-ci-cpu.sh.github/workflows/nightly-e2e.yamlvm-e2ejob (KVM,NEMOCLAW_GATEWAY_BACKEND=vm, diagnostics)test/e2e/test-vm-backend-e2e.shtest/openshell-vm.test.tstest/platform.test.tsdetectGatewayBackend()priority and overridesE2E test phases (30/30 passing)
NEMOCLAW_GATEWAY_BACKEND=vmTest plan
npm test— all tests passvm-e2e: 30/30 on GitHub Actions ubuntu-latest (KVM)cloud-e2e: passes (Docker path not regressed)sandbox-survival-e2e: passeshermes-e2e: passesmessaging-providers-e2e: passesskip-permissions-e2e: passessecurity-configuration-hardeningtest: passes (Docker manifest unchanged)Refs: NVIDIA/OpenShell#611