Skip to content

fix(gc): capture the protect set after the mark phase so live blobs are not swept#240

Open
cbenhagen wants to merge 1 commit into
n0-computer:mainfrom
cbenhagen:fix/gc-protect-set-race
Open

fix(gc): capture the protect set after the mark phase so live blobs are not swept#240
cbenhagen wants to merge 1 commit into
n0-computer:mainfrom
cbenhagen:fix/gc-protect-set-race

Conversation

@cbenhagen

Copy link
Copy Markdown
Contributor

Description

Garbage collection could delete a blob that is still referenced by a consumer (for example an iroh-docs document entry), leaving a dangling reference (entry -> hash, but the blob is gone).

The cause is the ordering of GC's protections. A referenced blob is kept alive by three things:

  1. the consumer's add_protected callback, which reports the hashes of its committed references;
  2. the import temp tag, which the consumer holds from import until just after it commits the reference, and which the mark phase scans;
  3. the store's in-memory protected set, which holds every imported hash and is honored by the sweep, until it is wiped once per cycle by clear_protected.

run_gc invoked the add_protected callback before gc_run_once, which then ran clear_protected → mark → sweep. A single consumer write (import blob → commit reference → drop import temp tag) that interleaved with this could fall through all three: its import preceded clear_protected, so its entry in the protected set was wiped; its reference was committed after the callback snapshot, so it was absent from add_protected; and its temp tag was dropped before the mark phase, so it was absent from the temp-tag scan. The blob was then swept while still referenced. The window is small, but a frequently rewritten key against a slow GC interval lands in it eventually (observed in production with a ~30s key and a 600s interval).

The fix invokes the protect callback inside gc_run_once, after the mark phase and immediately before the sweep. With the callback captured this late, a reference committed after the snapshot must still have had its import temp tag held during the mark phase (the temp tag spans the commit), and anything imported after clear_protected is held by the protected set. No gap remains.

The memory store also never populated its protected set on import (it only checked and cleared it), unlike the fs store, so it relied on temp tags alone for in-flight blobs. It now inserts on import, giving both backends the same protection across the mark→sweep window.

Changes

  • store/gc.rs: gc_run_once takes the protect callback and invokes it after the mark phase, before the sweep; run_gc passes config.add_protected through. On ProtectOutcome::Abort the cycle returns before the sweep (after the mark phase) — as before, no blobs are deleted on abort.
  • store/mem.rs: insert imported hashes into the protected set, matching the fs store.
  • Tests: deterministic regression tests (gc_protect_race_mem, gc_protect_race_fs) and a concurrent stress test (gc_protect_race_stress_mem). The deterministic tests fail on the old ordering and pass with the fix.

Breaking Changes

None to the public API. gc_run_once and run_gc are internal — the gc module is private and only GcConfig, ProtectCb, and ProtectOutcome are re-exported — so the gc_run_once signature change is not observable downstream.

Notes & open questions

  • Related to Calling garbage collection manually #235 / docs: add temp_tag pattern for protecting long-running downloads #236: temp tags protect in-flight blobs, but this race could collect a blob even when the consumer uses temp tags correctly across the commit. It is a GC-internal ordering issue, not a consumer-side or docs issue.
  • The mem-store protected set now grows until the next clear_protected, matching existing fs-store behavior; for a store with GC disabled it is never cleared. Could be skipped when GC is disabled as a follow-up.
  • Pre-existing and out of scope: run_gc breaks its loop permanently on any gc_run_once error, so a single transient error (e.g. during clear_protected or the sweep) silently disables GC for the process lifetime. Worth a separate follow-up.

Change checklist

  • Self-review.
  • Documentation updates following the style guide, if relevant.
  • Tests if relevant.
  • All breaking changes documented.

…re not swept

GC could delete a blob that is still referenced by a consumer (e.g. an
iroh-docs document entry), leaving a dangling reference. `run_gc` captured
the `add_protected` set before `gc_run_once` ran `clear_protected` and the
mark phase, so a write that imported a blob, committed its reference, and
dropped its import temp tag in that window escaped all three protections
(the protected set, the temp-tag scan, and the external snapshot) and was
swept.

Invoke the protect callback inside `gc_run_once`, after the mark phase and
immediately before the sweep. A reference committed after the snapshot must
still have had its import temp tag held during the mark phase (the temp tag
spans the commit), and anything imported after `clear_protected` is held by
the protected set, so no gap remains.

Also populate the mem store's `protected` set on import, matching the fs
store, so both backends honor the same protection across the mark/sweep
window.

`gc_run_once` and `run_gc` are internal (the `gc` module is private), so the
signature change is not a public API break.
@n0bot n0bot Bot added this to iroh Jun 16, 2026
@github-project-automation github-project-automation Bot moved this to 🚑 Needs Triage in iroh Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🚑 Needs Triage

Development

Successfully merging this pull request may close these issues.

1 participant