Skip to content

fix(restore): materialize OCI-ref base images before the restore clone#33

Open
tonicmuroq wants to merge 1 commit into
mainfrom
fix/restore-materialize-base
Open

fix(restore): materialize OCI-ref base images before the restore clone#33
tonicmuroq wants to merge 1 commit into
mainfrom
fix/restore-materialize-base

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

#30 (862c5be) added base-image materialization for fresh clones only. The restore path — resolveWakeSourcecloneFromHibernate, used by both same-node wake and the operator's cross-node migration restore — pulls just the :hibernate snapshot, and the hole covers both base ref shapes:

  • bare OCI refs (e.g. simular/win11:25h2-20260705-1, the current AR-era bakes): cocoon core refuses to fetch these at all, and nothing on the restore path materializes them;
  • http(s) refs (e.g. https://epoch.simular.cloud/dl/simular/win11/25h2-20260625-1, the legacy-era bakes that the existing VM stock still carries): core can fetch these, but only under --pull, which the restore clone never passed.

The snapshot artifact carries no base layer (cocoon.json / config.json / memory-ranges / overlay.qcow2 / state.json) and its overlay backs onto the base cloudimg by digest (/var/lib/cocoon/cloudimg/blobs/<digest>.qcow2) — so restoring on a node whose local store lacks the base fails on the backing file. Every internal-cocoon node happens to carry the fleet cloudimgs today, which is the only reason migrate/wake currently works; a new node, a cross-region cluster, or a wake of an old snapshot after image turnover + gc hits this immediately.

Fix

  • resolveWakeSource returns the snapshot metadata it already looks up (and inspects the import after a registry pull, mirroring ensureSnapshot).
  • cloneFromHibernate applies the same digest-deduped ensureRunImage guard as the fresh-clone path for OCI-ref bases, and passes Pull so core fetches missing http(s) bases — one choke point covering wake and migration restore.

Tests

  • TestBringUpVMRestoreEnsuresOCIRefBaseImage — bare OCI-ref base absent locally → ensured before clone.
  • TestBringUpVMRestoreSkipsEnsureWhenDigestPresent — digest already local under any name → no re-download.
  • TestBringUpVMRestorePullsHTTPBase — restore clone passes --pull; no EnsureImage for http(s) bases.
  • Existing resolveWakeSource tests updated for the new return value.

Live verification (staging internal-cocoon, 2026-07-13)

Deployed this branch (a020dfe) fleet-wide as v0.3.3-dev-pr33b, then created a VM from legacy SAI snapshot sai-staging-snapshot:win11-20260614-1 (base = https://epoch-us.simular.cloud/dl/simular/win11/25h2-20260608) and drove the operator's cross-node migration to a node verified to lack that base (ref and blob both absent):

  • restore pulled the :hibernate snapshot (~13 GB from AR) plus the 12.65 GiB base from epoch-us before the clone (download observed in-flight in cloudimg/temp), then the VM came up Ready with its qcow2 chain resolving to the freshly pulled blob;
  • end-to-end migrate wall-clock ~7.5 min vs ~5 min with a cached base — the delta is the base download;
  • on the pre-fix binary this exact scenario fails on the missing backing file (nothing on the restore path repopulates the blob).

The create-time counterpart (#30 guard + --pull) was also exercised in the same run: the fresh clone from the legacy snapshot cold-pulled both artifacts on its create node.

@tonicmuroq tonicmuroq force-pushed the fix/restore-materialize-base branch from 3420e43 to a7c382b Compare July 13, 2026 09:48
…clone

The guard added for fresh clones (862c5be,
#30) covers only the default clone branch of bringUpVM. The restore path
(resolveWakeSource -> cloneFromHibernate, used by both same-node wake and
the operator's cross-node migration restore) pulls just the :hibernate
snapshot and invokes the clone without --pull, so restoring on a node
whose local store lacks the snapshot's base cloudimg fails resolving the
overlay's backing file — for BOTH base ref shapes: http(s) refs never get
core's auto-pull (no --pull flag), and bare OCI refs (e.g.
simular/win11:25h2-20260705-1) are ones core refuses to fetch at all.

Today every internal-cocoon node happens to carry the fleet cloudimgs,
which is the only reason migrate/wake works — verified on staging
2026-07-13 while testing cross-node migration: the restored VM's qcow2
chains directly onto /var/lib/cocoon/cloudimg/blobs/<digest>.qcow2 and
nothing on the restore path would repopulate it. New or cross-region
nodes hit this immediately.

resolveWakeSource now returns the snapshot metadata it already looks up
(inspecting the import after a registry pull, mirroring ensureSnapshot);
cloneFromHibernate applies the same digest-deduped ensureRunImage guard
as the fresh-clone path for OCI-ref bases and passes Pull so core fetches
missing http(s) bases.
@tonicmuroq tonicmuroq force-pushed the fix/restore-materialize-base branch from a7c382b to a020dfe Compare July 13, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant