Findings from a tenant-escape-focused security audit. The items below remain after the second hardening pass, which addressed vsock removal, cgroup limits, bandwidth reduction, iptables chain isolation, connlimit, IPv6 disablement, agent auth hardening, cert permissions, and guest image hardening.

• Hardcoded API key — A Fireworks AI API key is committed in src/config/defaults.ts. Every tenant receives it via secret injection. Move to an environment variable or external secret store before any production use.

These are inherent to Firecracker/KVM and cannot be resolved in application code.

• KVM is the trust boundary. A KVM guest-to-host escape bypasses all other isolation (jailer, iptables, seccomp). Google's kvmCTF program offers $250K for a full escape, indicating it's rare but not impossible. Keep the host kernel patched aggressively.
• Spectre-PHT remains exploitable across Firecracker VMs even with AWS-recommended mitigations (arXiv:2311.15999). A Medusa "cache indexing / block write" variant is uniquely exploitable inside Firecracker VMs (works in the VM but not on bare metal). The nosmt host boot parameter mitigates but does not fully resolve.
• Virtio device emulation is the primary code-level attack surface. Firecracker's Rust codebase and minimal device model (now 4 devices — vsock removed) makes exploitation difficult but not impossible. The jailer contains blast radius.

These are operational changes on the host OS, not lobsterd code changes.

• Disable SMT — Add nosmt to the host kernel boot parameters. Required for meaningful side-channel protection in multi-tenant deployments.
• Disable KSM — echo 0 > /sys/kernel/mm/ksm/run. Kernel Samepage Merging enables page deduplication side-channel attacks.
• Constrain kvm-pit threads — The jailer creates cgroups for the Firecracker process, but kvm-pit kernel threads spawned by KVM live outside them. These can be abused to consume ~68% of host CPU (USENIX Security 2023). Move them into per-VM cgroups and lower min_timer_period_us.
• Tune nf_conntrack — Even with per-tenant connlimit rules (1024 connections), tune sysctl net.netfilter.nf_conntrack_max appropriately for the expected tenant count.
• Pin Firecracker version — CVE-2026-1386 (jailer symlink arbitrary file overwrite) affected <= v1.13.1. Ensure v1.13.2+ or v1.14.1+.
• Use ECC RAM and DDR4 with TRR — Rowhammer protection for physical hosts.

Nice-to-have hardening that could be added to lobsterd itself.

• Encrypt snapshots — The snap command copies overlay files containing /root/.openclaw/openclaw.json (with API keys) in cleartext. Encrypt at rest or wipe sensitive pages before snapshot.
• Audit logging — No logging of sensitive operations (spawn, evict, secret injection). Add structured logs for forensic analysis if a tenant is compromised.
• Caddy admin API access control — Caddy listens on localhost:2019. Any host process can reconfigure all tenant routes. Consider binding to a Unix socket with restricted permissions.
• Caddy-to-guest TLS — Caddy reverse-proxies to guest_ip:9000 over plain HTTP. Acceptable on a point-to-point /30 link but would matter on a shared network.
• Agent token exposure — The per-tenant agent_token is passed via /proc/cmdline inside the guest, readable by any process in the VM. Acceptable for single-user VMs but would be a concern for multi-user guests.

• Jailer integration — Per-tenant UID/GID, chroot, PID/mount/network namespaces, default seccomp BPF (~35 syscalls)
• Cgroup resource limits — Memory capped at 1.5x VM size, CPU quota proportional to vCPU count
• Network isolation — Dedicated LOBSTER-INPUT/LOBSTER-FORWARD iptables chains (inserted at position 1), blocking guest→host, RFC1918, and link-local
• DNAT source restriction — PREROUTING DNAT excludes 10.0.0.0/8 source IPs to prevent cross-tenant access
• Per-tenant connlimit — 1024 concurrent connections per tenant in FORWARD chain to prevent conntrack exhaustion
• IPv6 disabled — disable_ipv6=1 set on each TAP device at creation
• Rate limiting — Network: 10 Mbps RX/TX, 1K ops/s. Disk: 50 MB/s, 5K ops/s
• Agent auth — Per-tenant UUID token, timing-safe comparison, fail-closed (no token = deny all), 1MB message size limit
• Guest hardening — apk, curl, git, wget removed; busybox recon applets (nc, telnet, ftp, traceroute, nslookup) removed; root locked; serial console getty removed; 8250 UARTs disabled
• Vsock device removed — Eliminates virtio-vsock emulation as guest-reachable attack surface
• Read-only rootfs — Overlayfs with per-tenant writable upper layer
• TLS termination — Caddy with ACME or bundled Cloudflare origin certs (key: 0640)

• Firecracker Production Host Setup
• Microarchitectural Security of Firecracker VMM (arXiv:2311.15999)
• Attacks are Forwarded — USENIX Security 2023
• Attacking Firecracker — Grapl Security
• Google kvmCTF Bounty Program
• HyperHammer — ASPLOS 2025