Skip to content

Port HelixDB fork changes to SlateDB v0.15 - #8

Draft
matthewsanetra wants to merge 64 commits into
mainfrom
codex/port-v0.15
Draft

Port HelixDB fork changes to SlateDB v0.15#8
matthewsanetra wants to merge 64 commits into
mainfrom
codex/port-v0.15

Conversation

@matthewsanetra

@matthewsanetra matthewsanetra commented Aug 7, 2026

Copy link
Copy Markdown

Important

This has yet to be tested with helix-proper.

Summary

  • Port the HelixDB fork changes onto the SlateDB v0.15.0 codebase while preserving v0.15 reader modes, manifest polling, WAL behavior, and cache policy support.
  • Reconcile reader snapshots with ManagedCheckpoint, Checkpoint, and FollowLatest modes, including protected generations and mode-specific behavior.
  • Restore multi-get across databases, readers, snapshots, transactions, and shared read operations.
  • Integrate query-scoped storage metrics, synchronous cache usage snapshots, the typed DatabaseMissing error, binding expectations, and the macOS cross-test matrix.

Validation

  • Workspace and all-feature checks
  • SlateDB unit, integration, and documentation tests
  • DST test suite
  • Node, Python, and UniFFI binding tests
  • Clippy and reader benchmark compile checks

rodesai and others added 30 commits July 8, 2026 18:43
@rodesai
[k/N] wal refactor: reorganize wal buffer so its owned by batch writer (
076834d 
slatedb#1882)
@rodesai
[k/n] wal refactor: only update last seen wal id with flushed wals (s…
5cdc57d 
…latedb#1885)
@hachikuji
Blog post on segmentation (slatedb#1912)
8729820 
Blog post on segment-oriented compaction from rfc 24.
@geeknarrator
[1908] Wire RetryingObjectStore into Admin and Clone paths (slatedb#1909
016b676 
)
@geeknarrator
1707 RFC-0029 GC Safe SST Ulid Allocator (slatedb#1822)
0789090
@KrishnaSindhur
Did small refactoring for db_state.rs triple-duplicates first/last ke…
323ed1b 
…y logic (slatedb#1916)
@criccomini
Optimize DST binary (slatedb#1918)
c676f11
@nomiero
RFC 0027 impl (4/N) - wiring object store cache for compaction and gc (…
9937c2d 
…slatedb#1899)
@nomiero
extract gc deletion into maybe_delete methods (slatedb#1919)
d304e71
@criccomini
Add a DST for DB rescaling (slatedb#1920)
62047c3
@criccomini
Allow users to disable boundary files in GC (slatedb#1917)
44295b0
@owen-d
adds schema evolution docs wrt PrefixExtractor (slatedb#1921)
6b4857b
@criccomini
Regenerate Go bindings (slatedb#1922)
593c62e
@criccomini
Add DbReaderMode and latest manifest polling (slatedb#1915)
4ff907a
@criccomini
Clarify impact of disabled boundary files (slatedb#1925)
8f4eada
@rodesai
[k/N] wal refactor: reorganize WalBufferManager into separate handles…
e7d6b4f 
… that share inner (slatedb#1886)
@agavra
fix accounting for max_concurrent_compactions (slatedb#1926)
cc69461
@rodesai
k/N wal refactor: drop last seq tracking from the wal (slatedb#1927)
844f724
@rodesai
switch fence fault injection over to fail channel from fail-parallel (s…
7e5108d 
…latedb#1928)
@geeknarrator
[1923] Add configurable object-store retry bound (slatedb#1924)
6a28896
@ryandielhenn
Eliminate unnecessary object store requests made on each `CommitCompa…
c8e62bc 
…cted` tick and increase worker default heartbeat_interval (slatedb#1931)
@1996fanrui
[1941] Fix stale nightly benchmark description in slatedb-bencher REA…
5ddac5b 
…DME (slatedb#1942)
@geeknarrator
[1923] Expose object_store_max_retries on UniFFI (slatedb#1933)
3f8e301
@criccomini
Fix flaky local filesystem boundary test (slatedb#1940)
5f7ac90
@nomiero
remove fsync from cached object store (slatedb#1945)
5beefe1
@1996fanrui
[1932] Expose manifest structural counts as DbStats gauges (slatedb#1934
4df9dcf 
)
@rodesai
wal refactor k/N: simplify backpressure to use flushed events only (s…
5604c9b 
…latedb#1930)
@tysont
Fix stale external SSTs after manifest merge (slatedb#1936)
5ea9afb
@1996fanrui
[1938] Count active memtable in total_mem_size_bytes (fix 0 with WAL …
57c0965 
…disabled) (slatedb#1939)
@rodesai
rename wait_until_wal_released->wait_until_wal_flushed (slatedb#1946)
0f39324
nomiero and others added 30 commits July 21, 2026 11:56
@nomiero
Update README with database split/merge status (slatedb#1959)
ca6067b
@criccomini
Freeze large memtables on WAL replay (slatedb#1955)
75c1394
@geeknarrator
(RFC-0029) Move L0 SST id allocation to dispatch (slatedb#1957)
f363a39
@nomiero
RFC 0031 - pluggable block cache policy (slatedb#1913)
76a1324
@nomiero
Apply ruff check --fix with ruff 0.16 for python bindings (slatedb#1965)
7574f29
@criccomini
Account for WAL bytes in write backpressure (slatedb#1967)
5b494cc
@nomiero
RFC - 0031 impl: Block cache policy - writes (slatedb#1962)
ac19c5c
@nomiero
Make object store cache pluggable and external (slatedb#1947)
bb3c30e
@rodesai
[rfc-30 1/N]: refactor writer WAL write path to use new traits (slate…
40ef029 
…db#1961)
@uttom-akash
Fix panic when probing a zero-bit prefix Bloom filter (slatedb#1969)
779c905
@criccomini
fix flaky test_size_based_flush_triggering (slatedb#1979)
54b2f33
@1996fanrui
[1977] Fix flaky test_backpressure_waiter_exits_when_db_is_fenced (sl…
71a4963 
…atedb#1978)
@criccomini
Fix test_compactor_applies_output_cache_policy (slatedb#1981)
42a9f15
@nomiero
convert block prefetch panic to error (slatedb#1983)
402a3cf
@criccomini
Add simple trivial moves to compactor scheduler (slatedb#1982)
69892b3
@JayJamieson
Add DbIterator.next_batch to the UniFFI binding (slatedb#1986)
4a9a2e4
@criccomini
Probe latest sequenced objects before listing (slatedb#1968)
81acc3c
@criccomini
Bump version to 0.15.0
7db4911
@xav-db @matthewsanetra
Add request-scoped query storage metrics
8742c84
@matthewsanetra
Add snapshot isolation to DbReader
7869213
@matthewsanetra
Harden reader snapshot WAL replay test
b313ac2
@xav-db @matthewsanetra
Add multi-get functionality across database components
d05328d 
- Introduced `multi_get` and `multi_get_with_options` methods in `Db`, `DbReader`, `DbSnapshot`, and `DbTransaction` to retrieve multiple values efficiently.
- Implemented a new `WriteBatchLookup` enum to facilitate lookup operations in `WriteBatch`.
- Enhanced the `lookup_latest_for_key` method to support the new multi-get functionality.
- Added tests to ensure correctness of multi-get operations, preserving input order and handling duplicates.

This update improves the performance and usability of batch retrieval operations in the database.
@matthewsanetra
ci: drop Windows test runner
4e3c5df
@xav-db @matthewsanetra
Add SlateDB cache usage snapshots
d439a32
@xav-db @matthewsanetra
Update SlateDB dependency lockfile
ec429d7
@xav-db @matthewsanetra
Keep unbounded cache accounting synchronous
2b99b7b
@xav-db @matthewsanetra
Add typed database-missing reader error
eaff69b
@xav-db @matthewsanetra
Update missing database binding tests
567d683
@matthewsanetra
Finish v0.15 reader API adaptation
ce9c7d2
@matthewsanetra
Merge origin/main after v0.15 port
25101ca 
All origin/main changes were reapplied as v0.15-compatible commits on top of v0.15.0. Record origin/main as merged without reintroducing the pre-v0.15 implementations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.