fix(gc): capture the protect set after the mark phase so live blobs are not swept#240
Open
cbenhagen wants to merge 1 commit into
Open
fix(gc): capture the protect set after the mark phase so live blobs are not swept#240cbenhagen wants to merge 1 commit into
cbenhagen wants to merge 1 commit into
Conversation
…re not swept GC could delete a blob that is still referenced by a consumer (e.g. an iroh-docs document entry), leaving a dangling reference. `run_gc` captured the `add_protected` set before `gc_run_once` ran `clear_protected` and the mark phase, so a write that imported a blob, committed its reference, and dropped its import temp tag in that window escaped all three protections (the protected set, the temp-tag scan, and the external snapshot) and was swept. Invoke the protect callback inside `gc_run_once`, after the mark phase and immediately before the sweep. A reference committed after the snapshot must still have had its import temp tag held during the mark phase (the temp tag spans the commit), and anything imported after `clear_protected` is held by the protected set, so no gap remains. Also populate the mem store's `protected` set on import, matching the fs store, so both backends honor the same protection across the mark/sweep window. `gc_run_once` and `run_gc` are internal (the `gc` module is private), so the signature change is not a public API break.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Garbage collection could delete a blob that is still referenced by a consumer (for example an iroh-docs document entry), leaving a dangling reference (
entry -> hash, but the blob is gone).The cause is the ordering of GC's protections. A referenced blob is kept alive by three things:
add_protectedcallback, which reports the hashes of its committed references;protectedset, which holds every imported hash and is honored by the sweep, until it is wiped once per cycle byclear_protected.run_gcinvoked theadd_protectedcallback beforegc_run_once, which then ranclear_protected→ mark → sweep. A single consumer write (import blob → commit reference → drop import temp tag) that interleaved with this could fall through all three: its import precededclear_protected, so its entry in theprotectedset was wiped; its reference was committed after the callback snapshot, so it was absent fromadd_protected; and its temp tag was dropped before the mark phase, so it was absent from the temp-tag scan. The blob was then swept while still referenced. The window is small, but a frequently rewritten key against a slow GC interval lands in it eventually (observed in production with a ~30s key and a 600s interval).The fix invokes the protect callback inside
gc_run_once, after the mark phase and immediately before the sweep. With the callback captured this late, a reference committed after the snapshot must still have had its import temp tag held during the mark phase (the temp tag spans the commit), and anything imported afterclear_protectedis held by theprotectedset. No gap remains.The memory store also never populated its
protectedset on import (it only checked and cleared it), unlike the fs store, so it relied on temp tags alone for in-flight blobs. It now inserts on import, giving both backends the same protection across the mark→sweep window.Changes
store/gc.rs:gc_run_oncetakes the protect callback and invokes it after the mark phase, before the sweep;run_gcpassesconfig.add_protectedthrough. OnProtectOutcome::Abortthe cycle returns before the sweep (after the mark phase) — as before, no blobs are deleted on abort.store/mem.rs: insert imported hashes into theprotectedset, matching the fs store.gc_protect_race_mem,gc_protect_race_fs) and a concurrent stress test (gc_protect_race_stress_mem). The deterministic tests fail on the old ordering and pass with the fix.Breaking Changes
None to the public API.
gc_run_onceandrun_gcare internal — thegcmodule is private and onlyGcConfig,ProtectCb, andProtectOutcomeare re-exported — so thegc_run_oncesignature change is not observable downstream.Notes & open questions
protectedset now grows until the nextclear_protected, matching existing fs-store behavior; for a store with GC disabled it is never cleared. Could be skipped when GC is disabled as a follow-up.run_gcbreaks its loop permanently on anygc_run_onceerror, so a single transient error (e.g. duringclear_protectedor the sweep) silently disables GC for the process lifetime. Worth a separate follow-up.Change checklist