Skip to content

cli: drop snapshot/restore in favor of fork as the CoW primitive #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aljoscha merged 1 commit into from
May 5, 2026
Merged

cli: drop snapshot/restore in favor of fork as the CoW primitive #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 3 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ ember vm create myvm --image ubuntu-dev
ember ssh myvm
```

No `sudo` needed. State is stored in `~/Library/Application Support/ember/`. Storage uses instant APFS copy-on-write clones — creating VMs and snapshots takes milliseconds regardless of disk size.
No `sudo` needed. State is stored in `~/Library/Application Support/ember/`. Storage uses instant APFS copy-on-write clones — creating and forking VMs takes milliseconds regardless of disk size.

The kernel build requires Docker or Podman and enables Docker networking inside your VMs. It only needs to run once — the kernel is cached for all future VMs. If you don't need Docker inside VMs, skip the `kernel build` step and the stock kernel will be auto-downloaded on first use.

Expand Down Expand Up @@ -201,23 +201,6 @@ Forks can grow the disk but not shrink it below the source size. Use `--no-start
ember vm fork base template --no-start
```

## Snapshots

Snapshots capture point-in-time state of a VM's disk. Useful for checkpointing before risky changes.

```bash
ember snapshot create myvm before-upgrade
ember snapshot list myvm

# Something went wrong? Roll back (VM must be stopped):
ember vm stop myvm
ember snapshot restore myvm before-upgrade
ember vm start myvm

# Clean up:
ember snapshot delete myvm before-upgrade
```

## Guest access

SSH keys are auto-injected at image build and VM creation time. The SSH user is auto-detected (`ubuntu` if `/home/ubuntu` exists, otherwise `root`).
Expand All @@ -237,7 +220,7 @@ ember cp myvm:/var/log/syslog ./syslog.txt

## Storage efficiency

Both platforms use copy-on-write storage, so VMs and snapshots share disk blocks with their parent image. Check actual disk usage:
Both platforms use copy-on-write storage, so VMs and forks share disk blocks with their parent image. Check actual disk usage:

```bash
ember debug storage-efficiency
Expand Down Expand Up @@ -306,7 +289,7 @@ The CLI is identical on both platforms. Under the hood:
| | Linux | macOS |
|---|---|---|
| Hypervisor | Firecracker (KVM) | Apple Virtualization Framework |
| Storage | ZFS zvols + snapshots | APFS clones (`cp -c`) |
| Storage | ZFS zvols + clones | APFS clones (`cp -c`) |
| Networking | TAP devices + iptables NAT | vmnet shared mode |
| Root required | Yes | No |
| State directory | `/var/lib/ember/` | `~/Library/Application Support/ember/` |
Expand Down
57 changes: 6 additions & 51 deletions crates/ember-core/src/backend.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,8 @@ pub struct StartedVm {
pub network: NetworkInfo,
}

/// Platform-agnostic snapshot information.
///
/// On Linux/ZFS this is backed by `zfs list -t snapshot`.
/// On macOS/APFS this is backed by APFS clone files in the VM's
/// `snapshots/` directory. On Linux/dm-thin this is backed by entries
/// stored on `VmMetadata::snapshots`.
pub struct SnapshotInfo {
/// Snapshot name (e.g., "snap1"). Does not include dataset path or directory prefix.
pub name: String,
/// Creation timestamp (Unix epoch seconds).
pub created_at: u64,
/// Size in bytes.
///
/// - Linux/ZFS: `referenced` property (bytes the snapshot points to).
/// - macOS/APFS: logical file size via `stat`.
/// - Linux/dm-thin: virtual volume size at snapshot time.
pub size: u64,
}

/// A storage volume returned by the [`StorageBackend`] when a fresh
/// volume is created (image base, VM clone, fork, restore).
/// volume is created (image base, VM clone, fork).
///
/// `disk_path` is what gets recorded on `VmMetadata::disk_path` /
/// `ImageEntry::disk_path` and passed to Firecracker as
Expand Down Expand Up @@ -151,10 +132,10 @@ pub trait VmBackend {
fn is_running(pid: u32) -> bool;
}

/// Storage backend: manages disk images, clones, and snapshots.
/// Storage backend: manages disk images, clones, and forks.
///
/// - **Linux/ZFS**: ZFS zvols with snapshots and `zfs clone`.
/// - **Linux/dm-thin**: device-mapper thin volumes with kernel snapshots.
/// - **Linux/ZFS**: ZFS zvols with `zfs clone`.
/// - **Linux/dm-thin**: device-mapper thin volumes with kernel `create_snap`.
/// - **macOS/APFS**: raw `.img` files with APFS CoW clones (`cp -c`).
///
/// Methods take `&VmMetadata` / `&ImageEntry` rather than bare names
Expand Down Expand Up @@ -208,38 +189,12 @@ pub trait StorageBackend {
/// macOS/APFS: `cp -c <image>.img <vm>/rootfs.img`.
fn clone_for_vm(&self, image: &ImageEntry, vm_name: &str) -> Result<VolumeHandle>;

/// Create a named snapshot of a VM's current disk state.
///
/// Returns `Some(SnapshotEntry)` when the backend persists snapshot
/// metadata in user-space state (dm-thin). Returns `None` when the
/// backend tracks snapshots itself (ZFS in the kernel, APFS as
/// files on disk).
fn snapshot(
&self,
vm: &VmMetadata,
snap_name: &str,
) -> Result<Option<crate::state::vm::SnapshotEntry>>;

/// Restore a VM's disk to a previously created snapshot.
///
/// Returns a fresh `VolumeHandle` because some backends generate a
/// new identifier on restore (dm-thin's `delete` + `create_snap`
/// produces a new `thin_id`). For backends that mutate the volume
/// in place (ZFS rollback) or replace the file atomically (APFS),
/// the handle's `thin_id` is `None`.
fn restore_snapshot(&self, vm: &VmMetadata, snap_name: &str) -> Result<VolumeHandle>;

/// Delete a snapshot.
fn delete_snapshot(&self, vm: &VmMetadata, snap_name: &str) -> Result<()>;

/// List all snapshots for a VM.
fn list_snapshots(&self, vm: &VmMetadata) -> Result<Vec<SnapshotInfo>>;

/// Resize a VM's disk to `new_size`. Caller is responsible for
/// stopping the VM first.
fn resize(&self, vm: &VmMetadata, new_size: ByteSize) -> Result<()>;

/// Destroy all storage for a VM (disk image, snapshots).
/// Destroy all storage for a VM (disk image and any internal fork
/// snapshots beneath it).
fn destroy_vm_storage(&self, vm: &VmMetadata) -> Result<()>;

/// Destroy storage for a base image.
Expand Down
30 changes: 0 additions & 30 deletions crates/ember-core/src/state/vm.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,28 +63,6 @@ pub struct NetworkInfo {
pub wan_iface: Option<String>,
}

/// A snapshot tracked by a backend that doesn't have a native list
/// query.
///
/// ZFS records snapshots in the kernel and lists them via `zfs list -t
/// snapshot`, so [`VmMetadata::snapshots`] stays empty for ZFS. dm-thin
/// addresses snapshots by numeric thin id with no name attached at the
/// kernel level, so it persists names + ids in `vm.json`. macOS APFS
/// uses on-disk filenames, so it also doesn't need this list.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SnapshotEntry {
/// User-visible snapshot name.
pub name: String,
/// Backend-specific thin id. Only meaningful for the dm-thin backend.
pub thin_id: u64,
/// Creation time as Unix epoch seconds — same shape as
/// [`crate::backend::SnapshotInfo::created_at`] so the backend's
/// `list_snapshots` can copy this through without reparsing.
pub created_at: u64,
/// Volume size in 512-byte sectors.
pub size_sectors: u64,
}

/// SSH connection configuration for a VM.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct SshConfig {
Expand Down Expand Up @@ -158,12 +136,6 @@ pub struct VmMetadata {
/// volume identity in [`disk_path`](Self::disk_path) instead.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub thin_id: Option<u64>,
/// Snapshots maintained by the storage backend in user-space state.
///
/// dm-thin populates this; ZFS and macOS leave it empty and surface
/// snapshots through their native APIs.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub snapshots: Vec<SnapshotEntry>,
}

impl VmMetadata {
Expand Down Expand Up @@ -195,7 +167,6 @@ impl VmMetadata {
},
parent_vm: None,
thin_id: None,
snapshots: Vec::new(),
}
}
}
Expand Down Expand Up @@ -394,7 +365,6 @@ mod tests {
ssh: SshConfig::default(),
parent_vm: None,
thin_id: None,
snapshots: Vec::new(),
}
}

Expand Down
120 changes: 5 additions & 115 deletions crates/ember-linux/src/dm_thin_storage.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command as ProcessCommand;

use ember_core::backend::{InitConfig, SnapshotInfo, StorageBackend, VolumeHandle};
use ember_core::backend::{InitConfig, StorageBackend, VolumeHandle};
use ember_core::config::size::ByteSize;
use ember_core::config::{DmThinMode, GlobalConfig};
use ember_core::error::{Error, Result};
use ember_core::image::registry::ImageEntry;
use ember_core::state::vm::{SnapshotEntry, VmMetadata};
use ember_core::state::vm::VmMetadata;

use crate::dm_thin::{dm_device_exists, loop_device, pool, thin, tools, SECTOR_SIZE};
use crate::zvol;
Expand All @@ -42,9 +42,9 @@ const MAX_METADATA_SIZE_BYTES: u64 = 16 * 1024 * 1024 * 1024;
/// dm-thin storage backend.
///
/// Holds the configured backing path and pool block size; thin id state
/// lives on `VmMetadata`/`ImageEntry`/`SnapshotEntry`. Concurrent
/// invocations are race-free thanks to the kernel's atomic id rejection
/// in `create_thin`/`create_snap`.
/// lives on `VmMetadata`/`ImageEntry`. Concurrent invocations are
/// race-free thanks to the kernel's atomic id rejection in
/// `create_thin`/`create_snap`.
#[derive(Clone)]
pub struct DmThinStorage {
/// Backing path. Either a directory holding `metadata.img` and
Expand Down Expand Up @@ -429,109 +429,6 @@ impl StorageBackend for DmThinStorage {
}
}

fn snapshot(&self, vm: &VmMetadata, snap_name: &str) -> Result<Option<SnapshotEntry>> {
self.ensure_pool_active()?;
self.assert_pool_healthy()?;
let vm_id = Self::require_vm_thin_id(vm)?;
let dm_name = thin::vm_dm_name(&vm.name);
let size_sectors = Self::vm_size_sectors(vm);

// Suspend so create_snap sees a metadata-coherent volume.
// Some operations (e.g. snapshotting a never-activated volume)
// can run without an active device, but suspending an inactive
// device errors. Activate first if needed.
self.ensure_thin_active(&dm_name, vm_id, size_sectors)?;

thin::suspend(&dm_name)?;
let snap_result = thin::allocate_snap(pool::POOL_NAME, vm_id);
let _ = thin::resume(&dm_name);
let snap_id = snap_result?;

Ok(Some(SnapshotEntry {
name: snap_name.to_string(),
thin_id: snap_id,
created_at: ember_core::state::vm::now_epoch_secs(),
size_sectors,
}))
}

fn restore_snapshot(&self, vm: &VmMetadata, snap_name: &str) -> Result<VolumeHandle> {
self.ensure_pool_active()?;
self.assert_pool_healthy()?;
let vm_id = Self::require_vm_thin_id(vm)?;
let snap = vm
.snapshots
.iter()
.find(|s| s.name == snap_name)
.ok_or_else(|| {
Error::Vm(format!(
"snapshot '{snap_name}' not found on vm '{}'",
vm.name
))
})?;
let snap_id = snap.thin_id;

let dm_name = thin::vm_dm_name(&vm.name);
let size_sectors = Self::vm_size_sectors(vm);

// Allocate the replacement thin id from the snapshot up-front so
// a failure here leaves `vm.thin_id` and the kernel pool
// unchanged. The old order (deactivate -> delete -> allocate)
// would orphan `vm.thin_id` on any allocate hiccup.
let new_id = thin::allocate_snap(pool::POOL_NAME, snap_id)?;

// Once new_id exists, swap the dm-mapper slot over to it. Any
// failure from here on must release new_id so we don't leak
// kernel state.
let result = (|| -> Result<PathBuf> {
if dm_device_exists(&dm_name)? {
thin::deactivate(&dm_name)?;
}
thin::delete(pool::POOL_NAME, vm_id)?;
thin::activate(&dm_name, pool::POOL_NAME, new_id, size_sectors)
})();

match result {
Ok(disk_path) => Ok(VolumeHandle {
disk_path,
thin_id: Some(new_id),
}),
Err(e) => {
let _ = thin::delete(pool::POOL_NAME, new_id);
Err(e)
}
}
}

fn delete_snapshot(&self, vm: &VmMetadata, snap_name: &str) -> Result<()> {
self.ensure_pool_active()?;
let snap = vm
.snapshots
.iter()
.find(|s| s.name == snap_name)
.ok_or_else(|| {
Error::Vm(format!(
"snapshot '{snap_name}' not found on vm '{}'",
vm.name
))
})?;
thin::delete(pool::POOL_NAME, snap.thin_id)
}

fn list_snapshots(&self, vm: &VmMetadata) -> Result<Vec<SnapshotInfo>> {
// dm-thin tracks snapshots via the persisted `vm.snapshots`
// list; the kernel knows nothing about names.
Ok(vm
.snapshots
.iter()
.map(|s| SnapshotInfo {
name: s.name.clone(),
created_at: s.created_at,
size: s.size_sectors * SECTOR_SIZE,
})
.collect())
}

fn resize(&self, vm: &VmMetadata, new_size: ByteSize) -> Result<()> {
self.ensure_pool_active()?;
self.assert_pool_healthy()?;
Expand All @@ -558,13 +455,6 @@ impl StorageBackend for DmThinStorage {
if let Ok(true) = dm_device_exists(&dm_name) {
let _ = thin::deactivate(&dm_name);
}
// Snapshots only live in the kernel pool; the user-level
// handle is `vm.json`, which is about to disappear. Free their
// thin ids before the VM's own id, otherwise they'd remain
// pinned in pool metadata with no way for ember to reach them.
for snap in &vm.snapshots {
let _ = thin::delete(pool::POOL_NAME, snap.thin_id);
}
if let Some(id) = vm.thin_id {
let _ = thin::delete(pool::POOL_NAME, id);
}
Expand Down
Loading
Loading