feat(hibernate): release the DHCP lease pre-snapshot; renew-nudge stalled wakes in-window#39
Merged
Merged
Conversation
tonicmuroq
force-pushed
the
feat/hibernate-dhcp-release
branch
4 times, most recently
from
July 18, 2026 03:28
2c76ae2 to
fb69a13
Compare
tonicmuroq
changed the title
…lled wakes in-window
Hibernate snapshots freeze the guest with its DHCP lease still bound while
the server-side lease (24h) keeps squatting under the old MAC. The restored
clone re-DHCPs on a fresh NIC/MAC, so any wake within the lease lifetime
REQUESTs the cached IP and the server must NAK it ("leased to another
client" — the VM's own previous MAC). win11's DHCP client then wedges on
APIPA ~3% of the time (event 1002, then no re-DISCOVER, no background
retry — measured fleet-wide: 66 NAKs/64 recovered/2 wedged in one week)
and the 45s lease wait times out into lifecycle=failed.
Two changes, both CH+Windows-gated and best-effort, mirroring VMware
Tools' default power scripts (release on suspend, renew on resume):
- hibernate: exec `ipconfig /release` over the agent channel before the
NetResize(0) NIC drop. Frees the server lease immediately (no 24h zombie
pool slots) and freezes a guest with no cached lease, so restored clones
DISCOVER instead of triggering the NAK path. rollbackHibernateNIC renews
after re-adding the NIC since the lease was already released.
- waitForFreshIP: one-shot `ipconfig /renew` nudge at 15s (tunable) still
lease-less, inside the same 45s budget — the wake either completes in
time or fails for real; the deadline stays the external contract.
Unsticks the APIPA wedge (4/4 in incident response) and covers
pre-existing snapshots that still carry cached leases. Never fires once
a lease is observed, never fires for Linux guests.
New metrics: hibernate_total gains phase=dhcp_release; new
wake_renew_nudge_total{result}. docs/metrics.md updated.
tonicmuroq
force-pushed
the
feat/hibernate-dhcp-release
branch
from
July 18, 2026 03:35
fb69a13 to
7d254cf
Compare
The release-then-drop sequence compensated every later failure through rollbackHibernateNIC, except the drop itself failing: that branch returned with the guest lease-less and its NIC up, and Windows never re-DISCOVERs after an explicit release. Renew before surfacing the error; extracting dropNICForHibernate keeps the branch flat.
…ten comments waitForFreshIP re-derived the guest OS through a lock and map lookup that could race pod tracking mid-wait; both callers already hold the pod, so take it as the parameter and drop isWindowsGuest. Fold the nudge tests onto newDropNICWakeFixture and trim multi-line comments to their one load-bearing constraint.
…IP waiter to its VMID An exec error on the release does not prove the guest skipped it — a vsock timeout can land after the command ran — so gating the compensation renew on a confirmed release could leave a live VM lease-less. Renew regardless; renewing a still-bound adapter is benign. waitForFreshIP resolved the VM by pod key each round, so a same-name pod recreate mid-wait handed the waiter the successor VM: the renew nudge and the caller's refresh/notify all sit outside the VMID lifecycle guard. The waiter now carries the VMID it was armed for and bails on mismatch.
… the lease write-back by VMID The drop can fail precisely because ctx died, and the compensation ran on that same dead context — exec.CommandContext then refuses it outright. Detach it with context.WithoutCancel, bounded by the exec's own timeout; the fake runtime now honors ctx like the real one so the case is testable. resolveVMIP looked the lease up for v's MAC but setVMIP wrote it back by pod key unconditionally, so a same-name recreate landing inside the lookup put the old MAC's IP on the successor VM with nothing to ever correct it. setVMIP now writes only when the tracked VMID still matches, and the lease is returned only after the write lands.
SnapshotSave, push and remove can fail precisely because ctx died, and rollbackHibernateNIC then ran its NetResize(1) and renew on that dead context, leaving an online VM NIC-less. The whole rollback now runs under one cancel-detached, bounded lifetime; the fake runtime's NetResize honors ctx like the real CommandContext so the case stays pinned.
Rooted in context.Background, lifecycle goroutines outlived SIGTERM whenever Close was bypassed; buildProvider already holds the signal context, so derive from it and let cancellation propagate.
…-NIC comments The unconditional-renew rationale lived twice (godoc and call site); it now lives once at the call it explains. newLeaseParser replaces the leases.json boilerplate duplicated across two tests. The swap test's nudge delay and the cancelled-ctx test's netResizeErr never influenced their runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure class this kills
Hibernate freezes the guest with its DHCP lease still bound, and never tells cocoon-net — the server-side lease (24h) keeps squatting under the old MAC. Wake restores onto a fresh NIC with a new MAC, so the resumed Windows REQUESTs its cached IP and cocoon-net must NAK it (
leased to another client— the VM's own previous MAC). From there:lifecycle=failed.Four production machines hit this in one morning (updater's short-cycle wake sweeps roll the 3% dice constantly); all were revived by a single manual
ipconfig /renew, which is what motivated the design.What this PR does (two things, CH+Windows-gated, best-effort)
Mirrors VMware Tools' default power scripts — release on suspend, renew on resume (
@%SYSTEMROOT%\system32\ipconfig /release//renew, shipped as defaults for decades):hibernate():ipconfig /releasebefore theNetResize(0)NIC drop. Server lease frees instantly (also ends the 24h zombie-lease pool depletion), and the snapshot freezes a guest with no cached lease → restored clones DISCOVER cleanly; the NAK precondition is gone.rollbackHibernateNICrenews after re-adding the NIC since the lease is already released. A sick guest (dead agent) still hibernates — release failure is logged and skipped.waitForFreshIP(): one-shotipconfig /renewnudge at 30s (tunable) still lease-less, inside the unchanged 45s budget. The deadline stays the external contract: the wake either completes in time or fails for real — no post-deadline recovery machinery. The nudge unsticks the APIPA wedge and covers every pre-existing:hibernatesnapshot that still carries a cached lease (each gets one more risky wake during the transition). Guarded three ways: never once a lease is observed, never for Linux guests (markReadyAfterIPshares this wait), never more than once.Observability
cocoon_vk_hibernate_totalgainsphase=dhcp_releasecocoon_vk_wake_renew_nudge_total{result}— should trend to zero as the fleet's snapshots roll over to post-release generationdocs/metrics.mdupdatedTests
8 cases pin the behavior: release exactly once before NIC drop; release failure doesn't block hibernate; Linux never gets ipconfig; rollback renews; nudge fires exactly once when lease-less; nudge never fires when a lease exists (even with a 1ns nudge window); nudge never fires for Linux. Full suite green.
Verification plan
Canary on one node: hibernate a test VM (expect
RELEASEin cocoon-net journal + lease gone from leases.json), wake it (first wake of a legacy snapshot may still NAK→self-heal; second-generation wakes should be NAK-free DISCOVERs).