fix(restore): materialize OCI-ref base images before the restore clone#33
Open
tonicmuroq wants to merge 1 commit into
Open
fix(restore): materialize OCI-ref base images before the restore clone#33tonicmuroq wants to merge 1 commit into
tonicmuroq wants to merge 1 commit into
Conversation
3420e43 to
a7c382b
Compare
…clone The guard added for fresh clones (862c5be, #30) covers only the default clone branch of bringUpVM. The restore path (resolveWakeSource -> cloneFromHibernate, used by both same-node wake and the operator's cross-node migration restore) pulls just the :hibernate snapshot and invokes the clone without --pull, so restoring on a node whose local store lacks the snapshot's base cloudimg fails resolving the overlay's backing file — for BOTH base ref shapes: http(s) refs never get core's auto-pull (no --pull flag), and bare OCI refs (e.g. simular/win11:25h2-20260705-1) are ones core refuses to fetch at all. Today every internal-cocoon node happens to carry the fleet cloudimgs, which is the only reason migrate/wake works — verified on staging 2026-07-13 while testing cross-node migration: the restored VM's qcow2 chains directly onto /var/lib/cocoon/cloudimg/blobs/<digest>.qcow2 and nothing on the restore path would repopulate it. New or cross-region nodes hit this immediately. resolveWakeSource now returns the snapshot metadata it already looks up (inspecting the import after a registry pull, mirroring ensureSnapshot); cloneFromHibernate applies the same digest-deduped ensureRunImage guard as the fresh-clone path for OCI-ref bases and passes Pull so core fetches missing http(s) bases.
a7c382b to
a020dfe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#30 (862c5be) added base-image materialization for fresh clones only. The restore path —
resolveWakeSource→cloneFromHibernate, used by both same-node wake and the operator's cross-node migration restore — pulls just the:hibernatesnapshot, and the hole covers both base ref shapes:simular/win11:25h2-20260705-1, the current AR-era bakes): cocoon core refuses to fetch these at all, and nothing on the restore path materializes them;https://epoch.simular.cloud/dl/simular/win11/25h2-20260625-1, the legacy-era bakes that the existing VM stock still carries): core can fetch these, but only under--pull, which the restore clone never passed.The snapshot artifact carries no base layer (cocoon.json / config.json / memory-ranges / overlay.qcow2 / state.json) and its overlay backs onto the base cloudimg by digest (
/var/lib/cocoon/cloudimg/blobs/<digest>.qcow2) — so restoring on a node whose local store lacks the base fails on the backing file. Every internal-cocoon node happens to carry the fleet cloudimgs today, which is the only reason migrate/wake currently works; a new node, a cross-region cluster, or a wake of an old snapshot after image turnover + gc hits this immediately.Fix
resolveWakeSourcereturns the snapshot metadata it already looks up (and inspects the import after a registry pull, mirroringensureSnapshot).cloneFromHibernateapplies the same digest-dedupedensureRunImageguard as the fresh-clone path for OCI-ref bases, and passesPullso core fetches missing http(s) bases — one choke point covering wake and migration restore.Tests
TestBringUpVMRestoreEnsuresOCIRefBaseImage— bare OCI-ref base absent locally → ensured before clone.TestBringUpVMRestoreSkipsEnsureWhenDigestPresent— digest already local under any name → no re-download.TestBringUpVMRestorePullsHTTPBase— restore clone passes--pull; noEnsureImagefor http(s) bases.resolveWakeSourcetests updated for the new return value.Live verification (staging internal-cocoon, 2026-07-13)
Deployed this branch (a020dfe) fleet-wide as
v0.3.3-dev-pr33b, then created a VM from legacy SAI snapshotsai-staging-snapshot:win11-20260614-1(base =https://epoch-us.simular.cloud/dl/simular/win11/25h2-20260608) and drove the operator's cross-node migration to a node verified to lack that base (ref and blob both absent)::hibernatesnapshot (~13 GB from AR) plus the 12.65 GiB base from epoch-us before the clone (download observed in-flight incloudimg/temp), then the VM came up Ready with its qcow2 chain resolving to the freshly pulled blob;The create-time counterpart (#30 guard +
--pull) was also exercised in the same run: the fresh clone from the legacy snapshot cold-pulled both artifacts on its create node.