Make the durable fold a pluggable SnapshotStore trait#2
Merged
Conversation
Turn the snapshot layer into a trait with swappable backends so consumers that can't hold their fold in RAM (e.g. routing at ~1B keys) can fold to a queryable on-disk store instead. slipstream owns the durable-fold primitive; every large-bucket consumer gets it for free. - SnapshotStore trait: atomic apply(batch, cursor), load (resume), get, range. Preserves the existing invariants — pure function of the log, cursor-after- apply, gap-free integrity, "snapshot is a cache." - AppendLogSnapshot (default, pure-Rust): the existing append-only CRC log, refactored behind the trait + an in-RAM fold for get/range. v2 on-disk format, CRC, and FDB-versionstamp-safe cursor encoding unchanged; old files still load. SnapshotWriter/load stay public and untouched. - FjallSnapshot (feature = "fjall"): on-disk fjall LSM backend. One atomic batch per apply commits data + cursor together; NO_SYNC configurable; cheap Slice-backed get/range. Core stays pure-Rust; the C-free dep is feature-gated. - watch_applied is now generic over S: SnapshotStore — a consumer picks the backend (or None). Flush offloads store.apply to a blocking task; per-flush compaction policy moved into the backend. - Backend-agnostic conformance suite (tests/snapshot_store.rs) run for each backend: the pure-fold property test (rm store mid-stream -> reopen+replay -> byte-identical), round-trip, cursor-resume, get/range, fjall NO_SYNC crash-tail + idempotency, and AppendLog legacy-file load. - README + ARCHITECTURE document the trait and backends. BREAKING CHANGE: watch_applied's 4th arg is now Option<S: SnapshotStore> instead of Option<SnapshotWriter>. Pass AppendLogSnapshot::open(path, t).1. Bumps to 0.3.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Turns the snapshot layer into a trait with swappable backends, so consumers that can't hold their fold in RAM (e.g. routing at ~1B keys) can fold to a queryable on-disk store instead. slipstream owns the durable-fold primitive; every large-bucket consumer gets it for free. The trait stops at durable fold + cursor + query — serving structures (routing rings, etc.) stay in consumers.
Changes
SnapshotStoretrait — atomicapply(batch, cursor),load(resume),get,range. Preserves the existing invariants: pure-function-of-log, cursor-after-apply, gap-free integrity, "snapshot is a cache."AppendLogSnapshot(default, pure-Rust) — the existing append-only CRC log behind the trait + an in-RAM fold forget/range. v2 on-disk format, CRC, and FDB-versionstamp-safe cursor encoding unchanged; old files still load.SnapshotWriter/loadstay public and untouched.FjallSnapshot(feature = "fjall") — on-disk fjall LSM backend. One atomic batch perapplycommits data + cursor together; NO_SYNC configurable; cheapSlice-backedget/range. Core stays pure-Rust; the dep is feature-gated.watch_appliedis now generic overS: SnapshotStore— the consumer picks the backend (orNone). Flush offloadsstore.applyto a blocking task; per-flush compaction policy moved into the backend.tests/snapshot_store.rs) run for each backend: the pure-fold property test (rm store mid-stream → reopen+replay → byte-identical), round-trip, cursor-resume, get/range, fjall NO_SYNC crash-tail + idempotency, and AppendLog legacy-file load.watch_applied's 4th arg is nowOption<S: SnapshotStore>instead ofOption<SnapshotWriter>. Migrate call sites toAppendLogSnapshot::open(path, threshold)(returns(cursor, store)), passSome(store). On-disk format is unchanged, so existing snapshots keep loading. Version bumped to 0.3.0. (edge/tunnel migrations staged separately, pending the 0.3.0 publish.)Verification
cargo test --lib --tests✅ (lib 52, integration 38, snapshot_store 9)cargo clippy --all-targets -- -D warnings✅cargo test --features fjall✅ (adds the fjall backend through the same harness)🤖 Generated with Claude Code