feat: expose staged index segment transactions by ragnorc · Pull Request #3 · ModernRelay/lance

ragnorc · 2026-05-16T02:36:50Z

Summary

Add DatasetIndexExt::build_existing_index_segments_transaction to prepare an Operation::CreateIndex transaction for existing index segments without committing.
Refactor commit_existing_index_segments to reuse the staged transaction path.
Add coverage that verifies staging does not publish the index and the returned transaction can be committed via CommitBuilder.

Tests

cargo fmt --all
cargo test -p lance existing_index_segments -- --nocapture
cargo test -p lance test_vector_execute_uncommitted_segments_commit_without_staging -- --nocapture
External crate cargo check against the new public API

Fixes lance-format#6666

Summary by cubic

Adds a staged transaction builder to publish existing index segments without auto-committing, enabling two-phase index creation; refactors the existing commit path to use it. Also pulls in recent search and tokenizer improvements from main.

New Features
- DatasetIndexExt::build_existing_index_segments_transaction(index_name, column, segments) -> Transaction for staging, then commit via CommitBuilder (and commit_existing_index_segments now uses this path).
- Search upgrades: batch KNN queries with a query_index column; fast_search for scalar indexes to scan only indexed fragments; ICU-based FTS tokenizer via base_tokenizer="icu".
Bug Fixes
- Correctness: apply LIMIT after refine filters in filtered reads; fix stale reads in mem_wal via dedup-on-scan and deletion vectors; stable-row-id deletion masks honor fragment bitmaps; coerce expressions before simplification to avoid UDF panics.
- Performance/maintenance: avoid extra stat calls on OpenDAL ranged reads; clean up IVF shuffler temp dirs.

^{Written for commit 53aa593. Summary will update on new commits.}

Review in cubic

github-actions · 2026-06-23T16:31:51Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

…pdate (lance-format#7410) With stable row ids, a full-row `merge_insert` moves updated rows to a new fragment while preserving their row ids. The RewriteRows index maintenance (`register_pure_rewrite_rows_update_frags_in_indices`) then extends a scalar index's fragment bitmap to the new fragment unless the index covers a modified field. `merge_insert` built that modified-field set from the top-level `schema().fields`, so a **nested**-field index's leaf id was absent. The index was therefore wrongly extended over the rewritten fragment, which was then treated as indexed and never re-scanned — so updated rows were silently dropped from queries on that index (false negatives). The fix builds the set with `fields_pre_order()` so nested leaf field ids are included. Flat schemas are unaffected (`fields_pre_order()` equals the top-level list when there are no children). Adds two regression tests in `dataset_merge_update.rs`: a flat-index control and a nested-field guard that asserts the rewritten fragment is still data-scanned (via the `fragments_scanned` metric) so the updated value is found.

@dantasse

…e-format#7415) This PR picks up on the work in lance-format#7403 by @dantasse. `create_branch` on managed tables calls `create_table_version` before `_refs/branches/<branch>.json` exists, and `resolve_branch_location` requires that ref. Bootstrap commit fails. `create_table_version` now goes through `resolve_branch_for_commit` instead check the ref first, and if it's missing only allow the commit when the branch chain has no versions yet. Versions without a ref still get rejected (zombie). `create_branch("exp")` used to fail on the first commit because `create_table_version` looked for `_refs/branches/exp.json` before phase 2 wrote it. ``` # branching isn't atomic so: phase 1: manifest → create_table_version (ref doesn't exist yet ← was failing here) phase 2: write _refs/branches/exp.json ``` ### Testing - Added `test_create_branch_on_managed_dataset_succeeds`. - Zombie case is already in `test_branch_ops_reject_zombie_branch`. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>

…at#7311) ## Summary Adds a generic `SpillStore` — reclaimable RAII scratch storage for intermediate state that overflows memory (e.g. index-build posting lists, shuffle runs, BTree pages). Mechanism only; consumer migration (IVF shuffler) is a follow-up. - `SpillStore` / `SpillFile` (lance-io): `create_spill_file()` vends a write-once RAII handle whose `writer()` / `reader()` hand back `Box<dyn Writer>` / `Box<dyn Reader>`, so callers feed spill files straight into `FileWriter::try_new` and a v2 `FileReader` without leaking an `ObjectStore` + path. Dropping the handle deletes the file and releases its bytes back to the store's budget. - `LocalSpillStore`: writes to an OS temp directory; `with_cap` enforces an optional byte budget shared across all handles, returning a typed `Error::DiskCapExceeded` instead of silently filling the disk. Enforcement lives entirely in the spill store — the spill file decorates the writer with a `QuotaWriter` (reserve-on-write, release-on-drop-by-stat) rather than threading a field through `ObjectStore` and every provider, so it works for any backend the store opens. - `From<io::Error>` recovers a wrapped lance `Error`, so typed errors such as `DiskCapExceeded` survive the `AsyncWrite` boundary. - `ScanScheduler::open_reader` builds a `FileScheduler` over an already-open `Reader` (no path/size lookup), bridging a bare spill reader into the v2 reader path. - `Session` gains a `spill_store` field (default: uncapped `LocalSpillStore`), a `with_spill_store()` builder for injection, and a `spill_store()` accessor. Closes lance-format#7300 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… LanceFileSession (lance-format#7426) ## Summary Adds three primitives to `LanceFileSession` (`lance.file`), each a binding-exposure of an existing `object_store` capability (Rust → PyO3 → Python). They let callers route listing, deletion, and ranged reads through the dataset's `object_store` session instead of PyArrow's `AzureFileSystem`. ## APIs added ### 1. `list_with_delimiter(path=None) -> ListResult` Non-recursive, path-segment-delimited list of a single directory level. Unlike the existing recursive `list()`, it returns only the immediate children of `path`: child "directories" as `common_prefixes` and direct child files as `objects` (a `ListResult` dataclass), all session-relative. ```python result = session.list_with_delimiter("subdir") result.common_prefixes # immediate child "directories" result.objects # immediate child files ``` - `lance-io`: add `ObjectStore::list_with_delimiter(prefix) -> Result<ListResult>`. ### 2. `delete_file(path) -> None` Delete an object by session-relative path. Deleting a missing path is a **no-op (idempotent)** rather than an error, so it is safe on best-effort cleanup paths. ```python session.delete_file("subdir/checkpoint.lance") # removes it session.delete_file("already_gone.lance") # no-op, no error ``` ### 3. `read_range(path, offset, length) -> bytes` Single ranged read of a byte slice from a file. Reading a missing object raises `OSError`, consistent with `download_file`. ```python data = session.read_range("data/file-0.lance", offset=1024, length=256) # -> bytes ``` ## Why this matters These remove the remaining PyArrow `AzureFileSystem` / `filesystem_from_uri` dependencies from geneva's checkpoint and range-blob paths: - The recursive-only `list()` forced the flat-namespace bf-dir enumeration fallback to scan the whole checkpoint subtree; `list_with_delimiter` makes that fallback bounded again. - `LanceFileSession` had no delete, so checkpoint purge fell back to PyArrow and silently became a no-op on flat / unreachable-DFS accounts; `delete_file` makes purge functional and blob-only. - The range-blob reader's last PyArrow handle is replaced by `read_range`. ## Testing - `test_session_list_with_delimiter` — non-recursive semantics (base level returns immediate files + child dirs but not their contents; subdir descends one level; trailing-slash equivalence; empty for a missing prefix). - `test_session_delete_file` — top-level delete, nested delete, idempotent no-op on missing. - `test_session_read_range` — mid-file range, start-of-file, up-to-end, zero-length read, and `OSError` on a missing object. - All existing session tests pass. - `cargo fmt`; `cargo clippy -- -D warnings` clean (default features). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…at#7405) It was timing out on every merge to main. Move it from python.yml to the nightly workflow with a 360-minute timeout; it now builds the reader wheel in-job since needs: can't span workflows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e-format#7429) A fragment's RowIdSequence can hold segments whose ranges overlap but whose selected ids are disjoint -- compaction concatenates the surviving and rewritten halves of an updated fragment this way. From applied each segment's bitmap/hole removals to one shared map, so a later segment's removals cleared ids an earlier overlapping segment owned, dropping live rows from the result. Build each segment in isolation and union it in. Surfaced via COUNT(*) pushdown over stable row ids returning 0 after update + compact + filter. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DataFusion, `object_store`, and `prost-derive` already resolved to itertools 0.14, while our workspace pin held a second 0.13 copy in the dependency graph. Bumping the pin to 0.14 collapses the duplicate; only `criterion` retains 0.13, and that is dev/bench-only so it does not affect shipped crates. No source changes were needed — we use no items removed between 0.13 and 0.14. Verified with `cargo check --workspace --tests --benches` and `cargo clippy --all --tests --benches -- -D warnings`. --- Context: this came out of a dependency-graph audit. A `rand` 0.9 → 0.10 bump was also considered, but dropped — `rand` 0.9 is held alive by DataFusion (and goosefs-sdk/hf-hub/jsonb), so bumping our code to 0.10 would not eliminate the duplicate, only move which copy our code uses. Worth revisiting once DataFusion moves to `rand` 0.10. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

## Summary Adds row-level **DELETE** to the in-memory WAL (`mem_wal`), expressed as **tombstone (sentinel) rows**: a delete appends the primary key plus `_tombstone = true` (null elsewhere). A tombstone is the newest value for its PK — it wins newest-per-PK resolution (suppressing the older real row) and is then silently dropped from query results. This is the lance (OSS) half of the WAL delete feature. The sophon half (compaction hard-delete + serving) consumes this API separately. ## What's here - **`_tombstone` column** — lance-owned, non-nullable `Boolean` appended to the memtable/generation schema only (never the base table). `ShardWriter::open` extends the caller's base schema internally; callers keep passing the base schema and never name the column. - **Write path** — `ShardWriter::put` injects `_tombstone = false`; new **`ShardWriter::delete(keys)`** builds the tombstone batch from a key-only input. Legacy WAL entries (written before this change) are back-filled on replay. - **HNSW null-vector fix** — mem_wal's HNSW rejected null vectors outright, so it couldn't handle a nullable vector column at all (a pre-existing bug, independent of deletes). `append_batch` now skips null rows like the base-table index, compacting survivors while preserving their original offsets in `row_ids` so search still resolves to the right row. - **Read paths** - *Filtered read*: fold `NOT _tombstone` into the predicate per WAL arm (only when the source carries the column), so it runs before the per-source pushdown limit and post-dedup in the active arm. - *Point lookup*: carry `_tombstone` through and filter **after** `CoalesceFirstExec` (per-source filtering would let the coalesce fall through and resurrect the row); the plan-free BTree fast path short-circuits a tombstone hit to not-found. - *Vector / FTS*: no filter needed — a null-payload tombstone is never an index candidate; the deleted real row is suppressed by the existing block-list (cross-gen) / `NewestPkFilter` (within active). ## Notable design points - Fold/carry `_tombstone` **only when the source schema has the column** — back-compat for legacy generations and existing tests for free. - `_tombstone` is non-nullable so the point-lookup base arm can synthesize a matching `Literal(false)` for `CoalesceFirstExec`'s exact-schema check. - Delete requires non-PK columns to be nullable (a tombstone nulls them); surfaced as a clear error otherwise. - Caveat documented in code: a delete-heavy burst between compactions can under-deliver a `LIMIT`/top-k query (uncompensated block-list drops) — recall degradation, not wrong content; self-limiting as compaction drains tombstones. ## Testing 19 new tests; full `mem_wal` suite green (417 passed, 0 failed). `cargo fmt` + `cargo clippy -p lance --tests` clean. Coverage: filtered-read masking + filter + limit, point-lookup resurrection guards (fast + plan paths), delete-then-reinsert, delete-of-absent, HNSW null-vector handling (mid-batch + all-null), write-path injection/round-trip and validation. Draft — opening for early review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…nce-format#7433) lance-format#7373 made a DataReplacement conflict with any concurrent Update/Delete that touched a target fragment. That is too conservative: a delete that only adds a deletion vector leaves the positional column file aligned, and build_manifest rebases the replacement onto the current fragment, preserving the deletion vector. A DataReplacement vs a concurrent Delete/Update now conflicts only when: - the target fragment is removed outright -- returns a non-retryable IncompatibleTransaction so the caller can drop it and commit the rest; - a row-rewriting (RewriteRows) update moves the matched rows out to new fragments the positional file does not cover; or - the update rewrote one of the replaced fields in place (fields_modified). Deletion-vector-only deletes/updates and non-overlapping column rewrites are now allowed.

## Performance Improvement ### What is the performance issue or bottleneck? Recent FTS changes added AND/fuzzy bookkeeping to shared OR query paths. Ordinary non-fuzzy OR searches were still allocating and checking grouped expansion state, maintaining matched-position tracking in posting-list loading, and recording AND-only WAND metrics even though those values are only meaningful for conjunction/grouped expansion paths. ### How does this PR improve performance? This PR keeps ordinary OR on the lightweight path: - only allocates and updates required/matched position tracking for AND or phrase queries - skips grouped-position `HashSet` creation and per-frequency grouped checks unless grouped fuzzy expansions are present - records AND-only WAND metrics only for `Operator::And` Grouped fuzzy AND rescoring remains on its existing grouped path so matched-expansion scoring stays correct. ### Benchmark or measurement results No benchmark was run in this PR. The change is intentionally scoped to hot-path bookkeeping removal; benchmark validation can run later on the existing benchmark branch/VM if needed. ## Bug Fix ### What is the bug? The ordinary OR path was paying for bookkeeping intended for AND/fuzzy/grouped expansion correctness. ### What issues or incorrect behavior does the bug cause? It caused avoidable overhead in OR query execution and showed up as a small regression in OR benchmark configurations after the recent FTS changes. ### How does this PR fix the problem? The OR path now bypasses the AND/fuzzy-only bookkeeping while preserving the grouped expansion branch used by fuzzy AND. ## Validation - `cargo fmt --all` - `cargo fmt --all --check` - `git diff --check` - `CARGO_TARGET_DIR=/tmp/lance-target-fts-or-hotpath cargo clippy -p lance-index --tests -- -D warnings` - `CARGO_TARGET_DIR=/tmp/lance-target-fts-or-hotpath cargo test -p lance-index scalar::inverted::wand::tests -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-fts-or-hotpath cargo test -p lance-index scalar::inverted::index::tests::test_bm25_search_uses_global_idf -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-fts-or-hotpath cargo test -p lance-index scalar::inverted::index::tests::test_and_query_returns_empty_when_exact_term_missing -- --nocapture` - `CARGO_TARGET_DIR=/tmp/lance-target-fts-or-hotpath cargo test -p lance-index scalar::inverted::index::tests::test_fuzzy_and -- --nocapture`

…ce-format#7375) ## Summary This PR adds a lightweight file metadata index and a data-reader path that can load column metadata on demand for supported narrow projections. Dataset fragment reads now use this path for sparse top-level V2.1+ physical column projections and continue to fall back to full metadata when the projection is unsupported or the caller needs file-wide metadata. The problem is that the current full-metadata path decodes every column's metadata before reading data. On ultra-wide files, a one-column scan or point lookup pays metadata cost proportional to the total column count. ## Performance Benchmarked on EC2 instance `xuanwo-lance-lazy-metadata-bench` with a 65,000-column Lance file and random single-column point lookups. The full-metadata reader is the `origin/main` behavior. Results below are mean latency. | Storage | origin/main full metadata cold | this PR lazy metadata cold | Speedup | Metadata read bytes | IOPS | | --- | ---: | ---: | ---: | ---: | ---: | | Local EBS | 159.1 ms | 19.3 ms | 8.24x | 10.388 MiB -> 0.998 MiB | 4 -> 5 | | S3 us-east-2 | 368.4 ms | 144.4 ms | 2.55x | 10.388 MiB -> 1.057 MiB | 4 -> 5 | Hot reopen reads are effectively unchanged because metadata/data page reads are already cached: Local EBS was 10.4 ms -> 9.6 ms, and S3 was 36.3 ms -> 35.7 ms. The S3 cold path now issues only one additional request versus full metadata, not two, because dataset-known schema and row count are passed into the metadata-index reader instead of re-reading the schema global buffer. ## Validation Validated with targeted lazy metadata tests, the dropped-column statistics regression, `cargo fmt --all`, and full Rust clippy with tests and benches enabled.

## Summary Add Java and PyLance APIs for cleanup explain. - Add PyLance `Dataset.explain_cleanup_old_versions(...)` with cleanup explanation result types - Export PyLance cleanup explanation types from the top-level `lance` package - Add Java `dataset.cleanup(policy).explain()` / `.execute()` operation API - Keep Java `cleanupWithPolicy(policy)` as a compatibility wrapper for execute - Add JNI conversions and tests for cleanup explanations ## Testing - `cargo fmt --all` - `cargo fmt --manifest-path ./java/lance-jni/Cargo.toml --all` - `cargo check --manifest-path python/Cargo.toml` - `cargo check --manifest-path ./java/lance-jni/Cargo.toml` - `./mvnw compile` - `./mvnw -Dtest=CleanupTest test` - `uv run --frozen --python 3.12 pytest python/tests/ test_dataset.py::test_explain_cleanup_old_versions`

Pins goosefs-sdk to 0.1.5 because 0.1.6 fails to compile on Linux due to a missing std::ffi::CString import. This pin can be removed once upstream publishes a fixed release. ``` error[E0433]: cannot find type `CString` in this scope --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/goosefs-sdk-0.1.6/src/cache/store/uring/store.rs:314:23 | 314 | let cstring = CString::new(path) | ^^^^^^^ use of undeclared type `CString` ```  ## Summary by CodeRabbit * **Bug Fixes** * Updated GooseFS integration to use a compatible SDK version, improving reliability for supported storage operations.  --------- Co-authored-by: zhangyue19921010 <zhangyue.1010@bytedance.com>

@200M

…nce-format#7624) Built on the MAXSCORE work merged in lance-format#7603; this is the next PR in the Lucene-parity series. This PR now includes the 2/3-clause SIMD and frequency-bound work previously stacked as lance-format#7625. ## Performance issue Top-k AND and phrase queries previously leapfrogged doc-at-a-time through boxed `PostingIterator::next` calls. Phrase checks additionally decoded a whole 256-doc position block per candidate and allocated cursor vectors per candidate. ## What changed - Add a bulk conjunction path for compressed top-k AND and phrase queries. It keeps the classic block-max window pruning semantics while intersecting decompressed posting slices in batches. - Specialize the 2- and 3-clause merge kernels. x86_64 uses runtime-dispatched AVX2 catch-up scans, with scalar kernels on other CPUs. - Add a frequency-bucketed score-bound LUT so lead docs that cannot beat the threshold are rejected before follower advances. - Keep a generic merge kernel for 4+ clauses when bulk mode is explicitly forced. - Score candidates in two passes so document-length loads are issued back-to-back. - Decode only the PackedDelta position groups needed by the current phrase candidate instead of the whole position block. - Use an allocation-free exact-phrase check and recycled owned position buffers. - Return contextual errors for malformed packed-position data instead of panicking. ## Runtime selection `LANCE_FTS_BULK_AND` accepts: - `auto` (default): use bulk for 2/3 effective posting clauses and classic for all other widths. - `on` or legacy `1`: force bulk for every eligible compressed AND/phrase query; 4+ clauses use the generic kernel. - `off` or legacy `0`: always use the classic conjunction loop. The clause count is measured after tokenization, deduplication, and expansion. Invalid values warn and fall back to `auto`. The 4/5-clause experiments did not show a consistent specialized-kernel win, and generic bulk regressed against classic in several tiers. Therefore no 4/5-clause specialized kernels are retained and `auto` remains limited to 2/3 clauses. ## Measured performance Per-branch-tip wheels, 1000 queries × 8 concurrent. AND used the warm, fully prewarmed 200M-doc V3/256 index; phrase used the 50M-doc V3/256 positions index. | query | previous stack base | combined PR | |---|---:|---:| | AND 3w k10 @200M | 70 qps | **172 qps (2.46×)** | | AND 3w k100 @200M | 33 qps | **86 qps (2.61×)** | | phrase 3w k10 @50m | 19 qps | **35 qps (1.84×)** | | phrase 2w k10 @50m | 59 qps | **102 qps (1.73×)** | The AWS 4/5-clause follow-up used an `r7i.24xlarge`, the 200M English-only MMLB dataset, a V3/256 index with 338 partitions, k=100, c32, a 600 GB cache, and synchronous full prewarm. ## Compatibility This is a query-time implementation change. It does not change the FTS index format or index version. ## Verification - Bulk/classic/auto parity and dispatch coverage for 2 through 6 clauses, including nonzero phrase slop. - PackedDelta per-doc seek parity across group boundaries and tails. - PackedDelta and VarintDocDelta cursor-recycling coverage. - Malformed packed-position data returns a recoverable search error. - Final WAND suite: 80 passed. - `cargo test -p lance-index`, `cargo clippy --all --tests --benches -- -D warnings`, and `cargo fmt --all -- --check` passed across the reviewed stack.  ## Summary by CodeRabbit * **Performance Improvements** * Added a faster bulk AND/phrase execution path for compressed postings with block-level skipping and slice intersection; availability controlled via `LANCE_FTS_BULK_AND` (auto by default). * Improved phrase verification to decode only required positions per matching document. * Optimized packed-delta position seeking using cached group state and efficient tail handling. * **Bug Fixes** * Ensured bulk AND/phrase results match classic behavior, including filtering and pruning semantics. * Improved rejection of malformed packed-position data. * **Tests** * Added seek accuracy coverage across group boundaries and tail cases, plus malformed-group rejection and cursor independence checks.  --------- Co-authored-by: Yang Cen <yang@lancedb.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

## Summary Creates a new `lance-index-core` crate containing the abstract trait layer for scalar indices. The goal is to allow index plugin authors to implement `ScalarIndex` without depending on the full `lance-index` crate (~17 dependencies vs 75+). ### What moved The following types were extracted from `lance-index` into `lance-index-core` and re-exported from their original locations for backward compatibility: - **Traits**: `Index`, `IndexParams`, `ScalarIndex`, `AnyQuery`, `IndexStore`, `IndexReader`, `IndexWriter`, `RowIdRemapper`, `MetricsCollector` - **Supporting types**: `IndexType`, `SearchResult`, `CreatedIndex`, `UpdateCriteria`, `OldIndexDataFilter`, `ScalarIndexParams`, `BuiltinIndexType`, `TrainingCriteria`, `TrainingOrdering`, `IndexFile` The concrete query type implementations (`SargableQuery`, `LabelListQuery`, `TextQuery`, `BloomFilterQuery`, `FullTextSearchQuery`, `TokenQuery`, `GeoQuery`) stay in `lance-index`. ### Dependency graph ``` lance-index-core (new, ~17 deps — traits only) ↑ lance-index (concrete impls, re-exports everything for compat) ↑ lance (unchanged relationship) lance-table (no new dependency on lance-index-core) ``` --- ## Non-trivial changes ### `TrainingCriteria`/`TrainingOrdering` lifted out of the plugin registry These two types were defined in `lance-index::scalar::registry` alongside the plugin infrastructure. Because `UpdateCriteria` — a field on the `ScalarIndex` trait — references `TrainingCriteria`, both types need to live in `lance-index-core`. They are re-exported from their original path for backward compatibility: ```rust // lance-index/src/scalar/registry.rs pub use crate::scalar::{TrainingCriteria, TrainingOrdering}; ``` ### Newtype wrappers for `IndexReader` in `lance_format.rs` `IndexReader` was previously implemented directly on `lance_file::previous::reader::FileReader` and `lance_file::reader::FileReader`. With `IndexReader` now in `lance-index-core`, both the trait and the target types are foreign to `lance-index`, violating the orphan rule. Two private newtype wrappers resolve this: ```rust struct PreviousIndexReader(PreviousFileReader); struct CurrentIndexReader(CurrentFileReader); ``` These implement `IndexReader` locally and are boxed as `Arc<dyn IndexReader>` before being returned from `open_index_file`. No public API changed. The existing blanket `impl IndexWriter for PreviousFileWriter<M>` was also removed for the same reason (foreign trait over a foreign generic type). The one call site that used it was updated to call `PreviousFileWriter::write` and `finish_with_metadata` directly. ### Newtype wrappers for system-index types (`FragReuseIndex`, `MemWalIndex`) `FragReuseIndex` and `MemWalIndex` are defined in `lance-table`. Previously, `lance-index` implemented `Index` and `RowIdRemapper` directly on those types — valid because both trait and impl were in `lance-index`. With the traits now in `lance-index-core`, the orphan rule requires the impl to live in the trait crate or the type crate. To avoid adding a `lance-table → lance-index-core` dependency, `lance-index` instead defines two newtype wrappers: ```rust // lance-index/src/frag_reuse.rs pub struct FragReuseIndexHandle(pub Arc<FragReuseIndex>); impl Index for FragReuseIndexHandle { ... } impl RowIdRemapper for FragReuseIndexHandle { ... } // lance-index/src/mem_wal.rs pub struct MemWalIndexHandle(pub Arc<MemWalIndex>); impl Index for MemWalIndexHandle { ... } ``` Call sites in `lance` that previously cast `Arc<FragReuseIndex>` directly to `Arc<dyn Index>` now wrap it in the handle first. `lance-table` retains zero knowledge of `lance-index-core`. ### Dual `IndexFile` types with explicit conversion functions `IndexFile` (`path: String`, `size_bytes: u64`) exists in both `lance-index-core::scalar` (for plugin use) and `lance-table::format` (for protobuf serialization and manifest tracking). The two are kept as independent structs — a `From` impl is not possible due to the orphan rule, and having `lance-table` re-export from `lance-index-core` would introduce the unwanted dependency. Two free functions in `lance-index` bridge them at the boundary: ```rust pub fn index_files_to_table(files: Vec<lance_index_core::scalar::IndexFile>) -> Vec<lance_table::format::IndexFile> pub fn table_files_to_index(files: Vec<lance_table::format::IndexFile>) -> Vec<lance_index_core::scalar::IndexFile> ```  ## Summary by CodeRabbit * **New Features** * Added a shared core API for scalar/vector index implementations, including unified index typing, search results, storage interfaces, and row-id remapping. * Introduced a standardized metrics collection interface for indexing/query operations. * **Bug Fixes** * Improved index build/optimization write paths and related test write flows for more reliable batch persistence. * Normalized index file metadata handling across creation, merging, remapping, and manifest generation. * **Refactor** * Consolidated public index and metrics APIs via core re-exports, using handle wrappers for better consistency. * **Tests** * Updated and expanded index-creation/optimization tests to align with the revised persistence and wrapping behavior.  Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

## Summary - fail RLE miniblock encoding when chunk selection cannot advance instead of returning partial buffers with the original value count - assert that raw encoded run lengths match each declared chunk count across U8, U16, and U32 run-length widths - retain the exact-prefix implementation introduced in lance-format#7376; this does not change the file format ## Context `LANCE_MINIBLOCK_MAX_VALUES=1` is currently accepted, but a multi-value RLE page cannot represent a non-final one-value chunk because `log_num_values == 0` identifies the final chunk. The encoder previously broke out of the loop and returned incomplete data while retaining the original `num_values`. ## Test plan - [x] `cargo test -p lance-encoding` (430 passed, 5 ignored) - [x] `cargo fmt --all -- --check` - [x] `cargo clippy --all --tests --benches -- -D warnings` Made with [Cursor](https://cursor.com)  ## Summary by CodeRabbit * **Bug Fixes** * Prevented incomplete or invalid compressed data from being returned when encoding cannot make progress. * Improved error reporting with contextual details for encoding failures. * Fixed run-length handling across the 2,048-value boundary. * **Tests** * Added coverage for multiple run-length widths and zero-progress encoding scenarios.  Co-authored-by: Cursor <cursoragent@cursor.com>

) ## What & why The write-path reserved-column-name guard only rejected three of the five system column names (`_rowid`, `_rowaddr`, `_rowoffset`), missing `_row_created_at_version` and `_row_last_updated_at_version`. Those two were introduced by the row-tracking feature and the guard was never extended. System columns are virtual — they are injected into scan results at read time and never stored in the physical data. `Projection::to_schema` appends `_rowid`, `_rowaddr`, and the two row-version columns to a projection, and the scanner's `filterable_schema` sets all of those flags on every filtered scan. So a user data column literally named `_row_created_at_version` passed ingest, then collided with the appended system field the next time the dataset was scanned with a filter, hitting `to_schema`'s `extend(...).unwrap()` and panicking. Rare (the column name has to match exactly) but reachable on well-formed, in-memory data — not a corrupt-file case. ## How & why it is correct Replace the hardcoded three-name check with `is_system_column`, which covers all five names. This rejects the collision at the write boundary with a clear error instead of letting it surface as a later panic, so `to_schema`'s existing contract genuinely holds. `is_system_column` is a strict superset of the old three names, so the previously-rejected names still error exactly as before. ## Compatibility No behavior change for legitimate writes. The row-version columns are virtual and never part of a stored schema, so no valid write payload contains them; the only schema that reaches this guard is the user-provided data schema in `InsertBuilder`. Internal writers (merge-insert, update) build their schemas from `dataset.schema()` and bypass this guard entirely. The only newly-rejected input is a user explicitly naming a physical column after a system column — which is precisely the case that used to panic on read. ## Test plan - New parameterized test `rejects_reserved_system_column_names` covering all five system column names; the two version-column cases fail against the old three-name guard and pass with the fix. - `cargo test -p lance --lib`, `cargo clippy -p lance --tests -- -D warnings`, `cargo fmt --all -- --check` are green.  ## Summary by CodeRabbit * **Bug Fixes** * Dataset ingestion now rejects user-provided field names that conflict with reserved system columns. * Validation covers the complete set of system column names, including row-version fields. * Added coverage to ensure conflicting field names consistently fail ingestion.

`geodatafusion 0.5.0` supports DataFusion 54, which was the last remaining blocker. This bumps DataFusion 53 → 54, and transitively `geodatafusion` 0.4 → 0.5, `sqlparser` 0.61 → 0.62, and `substrait` 0.62 → 0.63. Arrow stays at 58. ### Migration for DataFusion 54 breaking changes - `as_any` was removed from the `ExecutionPlan`, `PhysicalExpr`, `ScalarUDFImpl`, `TableProvider`, `SchemaProvider`, `CatalogProvider`, and `CatalogProviderList` traits. The impls are dropped and downcasting now uses the inherent `dyn Trait::downcast_ref`/`is` (e.g. `plan.downcast_ref::<T>()` instead of `plan.as_any().downcast_ref::<T>()`). - `ExecutionPlan::partition_statistics` now returns `Arc<Statistics>`. - `Expr::Cast`/`TryCast` hold a `FieldRef` instead of a `DataType`. - `AnalyzeExec::new` gained a `metric_categories` argument and `MetricType::SUMMARY` was renamed to `MetricType::Summary`. - sqlparser 0.62 wraps the LIKE escape character in `ValueWithSpan`. - substrait 0.63 adds `RexType::Lambda`/`LambdaInvocation` variants and replaces extension URIs with URNs. - Migrated the deprecated `SimplifyContext::default()` builder to `SimplifyContext::builder().build()`. `create_aggregate_expr_and_maybe_filter` is retained under `#[allow(deprecated)]`; migrating to `LoweredAggregateBuilder` is left as follow-up. ### Verification - `cargo clippy --all --tests --benches -- -D warnings` clean, plus `-p lance-datafusion --features substrait`. - `lance-datafusion` substrait tests pass (exercising the URN and `RexType` changes). - Python (`pylance`) and Java (`lance-jni`) bindings compile; all three lockfiles refreshed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added support for DataFusion 54 and GeoDataFusion 0.5. * Enabled higher-order functions in SQL planning. * **Bug Fixes** * Improved compatibility with updated DataFusion casting and execution-plan/statistics behavior. * Updated Substrait conversion to use the current extension identifier format. * Added validation to reject unsupported lambda expressions in filter conditions. * **Tests** * Refreshed planning/explain and count-pushdown assertions to match the latest behavior.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…nce-format#7536) > **Supersedes lance-format#7409.** Relocated into `lance-format/lance` and **properly stacked on the OSS-1322 branch** so the diff is now exactly this PR's change (no spec/1322/1323 commits to scroll past). Resolves the `take` random-access path against data overlay files, replacing the temporary "overlays not supported" error from OSS-1322. Implements OSS-1324. Because `take` and scan share `FragmentReader`'s read path, the merge is wired there once: each row is addressed by its physical offset (from `ReadBatchParams::to_offsets_total`) and resolved against the overlays that cover its field. This also enables the scan-path merge that the OSS-1322 PR stubbed out. ### How it works - **Lazy, rank-pushed reads.** Each contributing overlay file is opened once (projected to the covered ∩ requested fields) and **no value bytes are read up front**. Requested offsets are routed to the newest covering overlay at their coverage rank, and only the touched ranks are fetched — via the existing `take` primitive — so overlay reads are O(requested rows), not O(coverage). - **Concurrent IO.** Base and overlay reads are issued together, then the column is assembled with `interleave` (rank → fetch-position remap). - The merge runs on physical rows in read order, **before** deletion filtering, so: - deletions take precedence (an overlay value computed for a deleted row is dropped with the row), - NULL overrides apply (a covered offset with a NULL value resolves to NULL, distinct from fall-through), - fields resolve independently. - Sparse per-field overlays read each field's value column independently, so unequal-length value columns (OSS-1323) need no rectangular batch. Rank-based addressing only (rank on the coverage bitmap + a value fetch; no offset key column, no binary search). Overlays on nested (non-top-level) fields are not yet matched and are left for follow-up. There's a `TODO(overlay perf)` on reader priority to settle with a benchmark. ### Tests take covered/uncovered offsets; multiple overlays (newest wins); per-field coverage with unequal-length columns; NULL override; overlay on a deleted row (inert); multi-fragment scan — each over v2.0 and v2.1. Plus unit tests for the routing/assembly core, and an IO guard (`test_take_reads_only_needed_overlay_ranks`) asserting a 2-row take over a fully-covering 100k-row overlay reads ~one miniblock, not the whole value column. ### Stacking Stacked on the **OSS-1322** branch (`will/oss-1322-write-data-overlay-and-scan-data`). Merge that first; this PR's base retargets to `main` automatically when it lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added support for reading dataset fragments that include overlay files. * Overlay values are resolved and merged into results during both scans and targeted row reads, honoring newest-wins semantics across nested fields. * Optimizes reads by fetching only overlay data needed for the requested rows and projections. * **Bug Fixes** * Fragments with overlays are no longer rejected, and overlay resolution remains correct when overlays overlap deleted rows. * **Tests** * Added end-to-end overlay read coverage for scan/take, including multi-batch slicing and edge cases.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Unreleased version after creating v9.0.0-rc.1

Wrap the `MatchQuery` query argument in brackets so metric suffixes do not look like part of the query term. Example verbose metrics output changes from: ```text MatchQuery: column=content_text, query=government: elapsed=342ms ``` to: ```text MatchQuery: column=content_text, query=[government]: elapsed=342ms ```  ## Summary by CodeRabbit * **Style** * Updated Full Text Search (FTS) execution-plan display for `MatchQuery` to show the query value in square brackets (e.g., `query=[...]`) for the default/verbose formats. * **Tests** * Updated FTS-related execution-plan expectation strings and snapshots to match the new `MatchQuery` bracketed formatting across legacy and non-legacy scenarios.

…back (lance-format#7807) ## Summary When no preset base scorer is injected, the FTS exec nodes (`MatchQueryExec` / `BooleanQueryExec` / `PhraseQueryExec`) build one via `build_global_bm25_scorer` over every segment they search — per-segment corpus stats plus per-term doc-frequency reads. That cost was folded invisibly into the node's `elapsed`, which makes it impossible to tell "search was slow" from "scorer construction was slow" in EXPLAIN ANALYZE. Adds a `scorer_build_ms` gauge to `FtsIndexMetrics` timing exactly that fallback at all three call sites. It reads 0 when a caller supplies a preset scorer (e.g. a distributed engine shipping corpus-wide stats via `with_base_scorer`), so the gauge also doubles as a visible marker of which scoring path a node took. ## Test plan - `cargo check -p lance` / `cargo fmt` clean - Verified end-to-end in a distributed setup (100M-row dataset, 4 index segments over 4 workers): with no preset scorer each worker's `MatchQuery` line reports its own `scorer_build_ms` (non-zero on cold caches, ~0 warm); with a preset scorer the gauge stays 0 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added performance metrics to track the time required to build BM25 scorers during full-text search. * **Monitoring** * Search execution metrics now report scorer construction duration when a fallback scorer is created.  Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…e-format#7772) A minimal, first slice of overlay-aware compaction (OSS-1326), stacked on lance-format#7536 (OSS-1324, `will/oss-1324-take-can-read-overlays`) — review against that base. ## What it does Adds a `max_overlays_per_fragment: Option<usize>` option to `CompactionOptions`. When a fragment carries **more than** this many data overlay files, the planner marks it `CompactItself`, so `compact_files` fully rewrites it into a fresh fragment with its overlays **and** deletions materialized into the base data. A fragment at or below the limit is not a candidate on this basis. The read side needs no new machinery: compaction already scans each fragment through the normal read path, which (per lance-format#7536) resolves overlays and applies deletions, so the merged post-image is what gets written to the new base file. ### Index correctness A full rewrite removes the overlays, and with them the staleness signal the query path uses to mask an index older than an overlay. So the `Rewrite` commit now drops the rewritten fragment from the coverage of any index left stale by those overlays — **field-aware** (only indices covering a field an overlay actually supplied) and **version-gated** (`overlay.committed_version > index.dataset_version`). Those rows fall back to a flat scan until reindex; an index is never left serving stale values. Non-stale indices (built at/after the overlay, or on un-overlaid fields) are remapped normally. This reuses the existing `Operation::Rewrite` path — no new persisted operation or conflict-matrix change. ## Scope / follow-ups Only the "fully compact the fragment" strategy is implemented. Overlay→overlay merge, in-place overlay→base folds (column rewrite preserving row addresses), and rebuild-in-place index reconciliation are left as follow-ups. ## Tests - Planner/e2e (`optimize.rs`): overlay count over the limit triggers a full compaction (fresh single-file fragment, overlays cleared, values materialized); at-or-below limit is a no-op; deletions are materialized alongside overlays; a stale scalar index is dropped from the compacted fragment's coverage and the indexed query stays correct. - Unit (`transaction.rs`): `prune_overlay_stale_fields_from_indices` is field-aware and version-gated (stale index dropped; index at/after the overlay kept; index on an un-overlaid field untouched). 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added a configurable overlay-count trigger for compaction; fragments exceeding the limit are compacted into standalone form. * Default limit is 10 overlays per fragment, configurable via `lance.compaction.max_overlays_per_fragment` (set to `none` to disable). * **Bug Fixes** * Prevented stale index coverage from returning outdated values after overlay/materialization compaction. * Improved accuracy by pruning rewritten fragment coverage from older index versions, causing queries to fall back to scans where needed. * **Tests** * Added coverage for the overlay trigger and index-staleness pruning behavior.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…nce-format#7816) ## Summary Reading a segment's tokenizer configuration previously required a full `InvertedIndex` open — partition construction, token dictionaries loaded into memory, and an index-cache entry — even when the caller only needs the params, e.g. to tokenize query text identically to the index or to inspect how an index was built. Measured on one segment of a large text dataset: **7.7 MB resident and a full open, for a few hundred bytes of config**. The manifest's `InvertedIndexDetails` is not a substitute: it is a lossy copy of the params that cannot carry `custom_stop_words` (unbounded user data that doesn't belong in the manifest) nor the json doc type, so a tokenizer rebuilt from it can silently diverge from the index's real tokenizer. This PR adds a params-only read path: - `InvertedIndex::load_params(store)` — reads the params JSON from the metadata file's schema metadata; falls back to the legacy tokens-file location, mirroring `load_legacy_index`. No partitions, no dictionaries, nothing cached. - `load_segment_params(dataset, segment)` — dataset-level helper via `LanceIndexStore::from_dataset_for_existing`, re-exported from `index::scalar`. The returned params are the complete serialized struct, byte-faithful — `custom_stop_words` and `lance_tokenizer` included. The metadata file is read through the session file-metadata cache the store wires up, so repeated calls are memory hits and a later full open of the same segment reuses the read. Measured (one segment, local FS, cold process): | | latency | resident | |---|---|---| | `load_segment_params` | 0.76 ms | 293 B | | full index open | 17.7 ms (with the file cache already warm) | 7.68 MB pinned in the index cache | ## Test plan - New test `test_load_segment_params_full_fidelity`: builds an FTS index with `custom_stop_words` (exactly the field `InvertedIndexDetails` cannot carry) and asserts `load_segment_params` equals the fully opened segment's `params()` field-for-field - `cargo check -p lance -p lance-index` / fmt clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Exposes the `DataOverlay` commit operation to Python so overlays can be created and committed from Python (needed to benchmark and use data overlay files without dropping into Rust). Mirrors the existing `DataReplacement` binding. ## What's here - `LanceOperation.DataOverlay` with `DataOverlayFile` and `DataOverlayGroup`. A `DataOverlayFile` carries the value `DataFile` plus **exactly one** of `shared_offsets` (dense coverage shared by every field) or `field_offsets` (sparse, one offset set per field). The commit stamps `committed_version`; passing both/neither coverage is rejected with a clear error. - PyO3 conversions (both directions) in `python/src/transaction.rs`. - Fills in the `overlays` field on the Python `FragmentMetadata -> Fragment` conversion, which OSS-1322 left unset (Python metadata doesn't carry overlays — they're committed via `DataOverlay`). ## Tests `test_data_overlay_*` in `test_dataset.py`: dense round-trip resolves on read; newest overlay wins; sparse per-field coverage resolves fields independently; and the exactly-one-coverage validation rejects both/neither. ## Stacking Stacked on **lance-format#7536** (OSS-1324 read path) — base retargets automatically when it lands. Next PR (benchmarks) stacks on this. > Note: the `python/src/fragment.rs` one-liner arguably belongs in the OSS-1322 PR (lance-format#7535), which added `Fragment.overlays` without updating the binding; included here so the stack compiles. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit - **New Features** - Added cell-level `DataOverlay` operations to append fragment overlays without rewriting entire fragments. - Supports dense and sparse overlays, with newest overlapping values taking precedence. - Persists overlay metadata on fragments and carries optional committed version stamps. - **Bug Fixes** - Preserves overlay metadata correctly through JSON serialization and Python↔Rust round-trips. - **Tests** - Added end-to-end coverage for dense/sparse overlays, precedence behavior, metadata round-tripping, and offset validation. - **Documentation/Chores** - Updated fragment metadata `repr` expectations and improved default `max_bytes_per_file` for fragment writing.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…-format#7824)

…n caches (lance-format#7817) ## Problem `read_transaction_by_version` delegated to `checkout_version` + `read_transaction`. A caller requesting one transaction does not need a historical `Dataset`, and the checkout path has session-cache side effects: the historical manifest is inserted into the metadata cache, manifest loading opportunistically decodes and caches the `IndexSection` into the index cache, and the transaction is inserted into the metadata cache. A long-lived process scanning many historical versions fills the shared caches with entries it never reuses (more visible after lance-format#7661). Fixes lance-format#7801. ## Change - `read_version_transaction(version) -> VersionTransaction { version, timestamp, transaction }` (new): resolves the version through the dataset's current branch and `CommitHandler`, decodes the manifest transiently (no cache read or write, no `IndexSection` decode), and reads the inline or external transaction. No historical `Dataset` is constructed. The compact record carries the manifest timestamp so callers don't have to check out for it. - The resolved manifest is validated to belong to the dataset's branch (matching `checkout_by_ref`), so a branch-insensitive commit handler errors instead of returning another branch's transaction. - `read_transaction_by_version` now delegates to it; signature and error semantics unchanged (missing/cleaned-up version is still an error). - `read_transaction` (current version) is unchanged apart from factoring the storage read into a shared helper; it still checks/populates the transaction cache. - `checkout_version` caching behavior is untouched. Known trade-off: an inline transaction costs one extra ranged read vs the old path (the manifest tail is read for the timestamp/offsets, then the transaction message separately). Callers scanning history are expected to memoize per-version results; a combined read is a possible follow-up. ## Tests - `test_read_version_transaction_does_not_populate_caches` — 20-version dataset with a BTree index; reads every version via the new API on a fresh session, asserts results and timestamps equal `checkout_version(v)`, that the session's index and metadata cache entries/bytes do not grow, and that a missing version errors with `Error::NotFound` (the version resolves to a manifest path that does not exist). - `test_read_version_transaction_v1_manifest_naming` — V1-named manifests (asserts the naming premise) resolve and match a checkout. - `test_read_version_transaction_on_branch` — versions resolve against the branch chain and match a full branch checkout. - `test_inline_transaction` (existing) extended: the external-transaction-file fallback is asserted for the direct read using the manifest that test already constructs. `read_version_transaction` carries a compiling doc example. Missing-version behavior of `read_transaction_by_version` itself is already covered by the existing `test_read_transaction_properties`, which now runs through the new implementation.  ## Summary by CodeRabbit * **New Features** * Added an API to retrieve transaction details for a specific dataset version, including the UTC commit timestamp and an optional transaction payload. * **Bug Fixes** * Historical transaction reads avoid unnecessary cache/index updates when no transaction is present. * **Tests** * Added coverage for inline-versus-external transaction fallback, correct behavior without cache pollution, and accurate handling of historical manifest formats across versions and branches.

@50m

…ance-format#7629) Built on lance-format#7624, now merged into main; this is the last PR in the series and contains only the norm-cache and slim-heap changes. ## What Two hot-loop changes, both score-identical (only tie ordering among equal scores can shift with the slimmer heap tuple): - **Per-search norm cache** (Lucene's norm cache): `Scorer::doc_norm` exposes the BM25 doc-length denominator addend, and quantized V3 searches bake the 256 possible addends once per partition-search. The OR streaming loop, OR drain/single-essential completion paths, and bulk conjunction scoring pass then use one byte-norm load plus a cached addend instead of recomputing `k1*(1-b+b*dl/avgdl)` per doc. The factored expressions are evaluated identically, so scores are bit-equal to the uncached path. - **Slim top-k heap**: heap entries stay compact while `(term, freq)` pairs live in at most `k` reusable side slots. Replacements clear and refill the evicted entry's slot while retaining its `Vec` capacity, and final candidates take their surviving slots without copying. This keeps frequency storage bounded at `O(k × clauses)` while avoiding Vec-carrying heap entries. ## Measured vs lance-format#7624 Measured at `aa084bd` before the bounded-slot review follow-up in `5ce91ee`: per-branch-tip wheels, 1000 queries × 8 concurrent. OR and AND used the warm, fully prewarmed 200M-doc V3/256 index; phrase used the 50M-doc V3/256 positions index. The scoring path is unchanged, but the bounded collector still needs a full 200M rerun. | query | lance-format#7624 | this PR | |---|---:|---:| | OR 3w k10 | 230 qps | **316 qps (1.37×)** | | OR 3w k100 | 127 qps | **169 qps (1.34×)** | | AND 3w k10 | 172 qps | 177 qps (+3%) | | AND 3w k100 | 86 qps | 89 qps (+3.5%) | | phrase 3w k10 @50m | 0.2256s | 0.2187s (+3%) | The OR win comes from the streaming loop, where recomputing the BM25 denominator per `clause × doc` was the largest single cost. ## Compatibility The norm cache consumes the existing V3 quantized byte norms and falls back to the existing scorer path otherwise. It does not change the FTS index format or index version. ## Verification - A/B against lance-format#7624: `score_diff=0` across AND (bulk and classic) and phrase query sets. - Equal-score tie ordering may change because of the slimmer heap tuple; 34 such tie reorders were observed in the benchmark set. - Final WAND suite: 82 passed. - `cargo test -p lance-index`: 817 passed, 2 ignored. - `cargo clippy --all --tests --benches -- -D warnings` and `cargo fmt --all -- --check` passed across the reviewed stack.  ## Summary by CodeRabbit * **Performance** * Improved full-text search performance by reducing temporary memory usage during ranked searches. * Added caching to speed up BM25 relevance calculations, especially for top-result and bulk searches. * **Search Quality** * Improved consistency of BM25 scoring across different search paths while preserving document-length considerations.  --------- Co-authored-by: Yang Cen <yang@lancedb.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

## What is the bug? `test_vector_index_distance_range` requires exact equality between brute-force and indexed `float32` distances. The two paths can use different scalar and runtime-dispatched SIMD reduction orders, producing a few ULPs of rounding difference even when the results are equivalent. ## What issues does this cause? The test fails intermittently on x86 CI across unrelated changes, while the returned IDs, ordering, and distance-range checks all pass. ## How does this PR fix the problem? Use a relative tolerance of `1e-5` for the distance comparison while retaining `atol=0.0`. This matches the expected precision of the alternative `float32` kernels without weakening the range or result checks. ## Validation - `uv run pytest python/tests/test_vector_index.py::test_vector_index_distance_range -q` - `uv run make lint` Co-authored-by: Yang Cen <yang@lancedb.com>

…7820) Make the full and indexed file metadata lookup methods public so callers can reuse the fragment's dataset-level metadata cache instead of loading metadata independently through FileReader. I ran into this while trying to inspect the file footer of a fragment that had already been opened. The obvious option was to call FileReader::read_all_metadata, but that API reads directly from storage and does not reuse the dataset’s file metadata cache. This can result in another metadata I/O even though the fragment has already loaded the same information. FileFragment already has cache-aware helpers for loading the full metadata or metadata index. This change makes get_file_metadata and get_file_metadata_index public so other fragment-level operations can reuse the existing cached metadata instead of reading it again through FileReader.  ## Summary by CodeRabbit * **New Features** * Exposed file metadata and metadata index access for broader integration and tooling.

## Why FTS corpus stats already read and cache each partition's Arrow document-length column and compute its total token count. The first WAND match then copied that entire column into a new Vec and summed it again while building the num-tokens-only DocSet. On large partitions this introduced O(rows) anonymous-page work for every PE/index partition even when no index parts were loaded. ## What Make the num-tokens-only DocSet retain a shared ScalarBuffer view and accept the previously cached total. Full DocSet materialization remains owned for masks, fragment reuse, remapping, and row-id semantics. Total initialization is single-flight, and retained-memory accounting deduplicates the shared Arrow storage. This removes the confirmed first-touch copy and repeated sum without changing WAND behavior or broadening prewarm or cache scope.

…ance-format#7769) Closes lance-format#7770 ## Summary - propagate final WAL flush failures from both MemTable and WAL-only close paths; - propagate frozen MemTable/L0 flush failures; - preserve the first causal error while continuing close-time cleanup; - always shut down background tasks before returning; - document the `ShardWriter::close()` error contract. ## Root cause `ShardWriter::close()` awaited three close-time completion channels but discarded their values. It also ignored failures to send the final WAL flush requests. As a result, close could return `Ok(())` after WAL persistence failure, peer fencing, L0 flush failure, handler shutdown, or completion-channel closure. The WAL-only case was especially unsafe: an unsuccessful append remained only in the in-memory pending queue, which was dropped when the consuming close returned success. ## Control-flow comparison ```mermaid flowchart TD M["MemTable final WAL"] --> C["Close-time completion"] L["Frozen MemTable / L0 flush"] --> C W["WAL-only final WAL"] --> C C --> OLD["Before: discard completion value"] OLD --> OLD_SHUTDOWN["Shutdown tasks"] OLD_SHUTDOWN --> FALSE_OK["Return Ok even after persistence failure"] C --> NEW["After: convert outcome to Result"] NEW --> FIRST["Preserve the first causal error"] FIRST --> DRAIN["Continue draining remaining watchers"] DRAIN --> NEW_SHUTDOWN["Always shut down tasks"] NEW_SHUTDOWN --> RETURN["Return first error, otherwise Ok"] ``` ## Fix The close path now maintains a first-error accumulator and executes the complete shutdown sequence in operational order: 1. request and await the final WAL flush; 2. freeze the active MemTable; 3. await every frozen MemTable flush; 4. shut down all background tasks; 5. return the first error encountered. `WalFlushFailure::into_error()` preserves typed fence reasons. Frozen MemTable results continue to use the existing `DurabilityResult` conversion. A private close-specific helper centralizes WAL completion handling for both writer modes. ## Tests Added failure-focused coverage for: - WAL-only final WAL persistence failure and task shutdown; - MemTable final WAL persistence failure and first-error preservation; - frozen MemTable/L0 fencing failure. The stricter close behavior also exposed three indexed-flush fixtures using `memory://` even though generation flushing reopens the dataset through an independent object-store registry. Those fixtures now use isolated `shared-memory://` authorities so the reopened generation observes the same backing bytes. ## Validation - `cargo test -p lance --lib dataset::mem_wal::write::tests` - 54 passed - `cargo test -p lance --lib dataset::mem_wal` - 501 passed, 1 ignored, 0 failed - `cargo fmt --all -- --check` - `pre-commit run --files rust/lance/src/dataset/mem_wal/write.rs` - `cargo clippy --all --tests --benches -- -D warnings` - `cargo clippy --all-features --tests --benches -- -D warnings` ## Compatibility - no public signature changes; - no Python or Java binding changes; - no persistent-format changes; - no dependency or lockfile changes. ## Implementation notes Beyond propagating the three close-time failures, the close path also: - drains remaining frozen MemTable watchers after a freeze failure so a successor failure is logged without replacing the first causal error (the accompanying comment documents this first-error-preserving drain behavior); - sends the final WAL flush directly on the channel rather than via `WalFlusher::trigger_flush`, which silently returns `Ok` when the flusher's `flush_tx` is unset and would let close acknowledge durability it never achieved — a closed send channel must surface as an error here; - logs the final close error at WARN so a single-stage failure is observable (`merge_close_stage` only logs the secondary error when both stages fail). Maintainers: please consider applying the `critical-fix` label because the old behavior could acknowledge a graceful close after final persistence failed.  ## Summary by CodeRabbit * **Bug Fixes** * Improved dataset close reliability by preserving the first failure across multi-stage shutdown instead of discarding later outcomes. * Ensured final WAL flush failures are surfaced, including WAL-only closes that drain pending batches before returning. * Propagated memtable freeze/flush watcher results (including fenced/failing flush scenarios) and treated missing watcher completion as an error. * **Tests** * Added/expanded coverage for close error propagation in both MemTable and WAL-only modes, including background task join timing. * Expanded shared in-memory scenarios using unique authorities to improve consistency across reopen/flush behavior.

) ## Summary - make the Python `PackedBlobWriter` and `DedicatedBlobWriter` wrappers safe to release on a thread other than the one that created them - replace PyO3's `unsendable` restriction with mutex-protected writer ownership - add a production-shaped regression test covering both writer types ## Motivation Geneva's Blob v2 checkpoint path prepares packed blob sidecars on a dedicated checkpoint-writer thread. During a large Azure benchmark, Azure returned `503 ServerBusy` from a multipart block `PUT`. The checkpoint thread propagated that object-store exception to the actor's owner thread. The original exception retained its Python traceback. That traceback retained the checkpoint frame and its local `PackedBlobWriter`, so the last reference to the writer was eventually released by the owner thread rather than the checkpoint thread. Because the binding declared the writer as `#[pyclass(unsendable)]`, PyO3 emitted a second unraisable exception: ```text RuntimeError: lance::blob::PyPackedBlobWriter is unsendable, but is being dropped on another thread ``` This secondary error appeared at unrelated Python stack locations, obscured the original Azure failure, and prevented normal RAII cleanup of the writer on that path. The same ownership problem applies to `DedicatedBlobWriter`, which used the same binding policy. ## Root cause The core Rust blob writers are `Send`, so transferring ownership for destruction is safe. They are not `Sync`, however, because the underlying `dyn Writer` is not shareable for concurrent access. PyO3 0.28 requires ordinary Python classes to be both `Send` and `Sync`; simply removing `unsendable` therefore does not compile. ## Fix Store each core writer as `Mutex<Option<Writer>>` and remove the `unsendable` annotation. The mutex makes the Python wrapper `Send + Sync` while retaining exclusive access to the non-`Sync` writer. Existing mutation and finish semantics remain unchanged: - completed writers still raise the existing `ValueError` - writer ownership is still consumed by `finish` - failed bulk writes still drop their active writer through RAII - a poisoned lock surfaces as a descriptive Python `RuntimeError` There is no public Python API change. ## Regression coverage The new parameterized test models the production lifetime directly: 1. create a packed or dedicated writer on a worker thread 2. raise and hand an exception containing that thread's traceback to the owner thread 3. release the exception and force garbage collection on the owner thread 4. assert that `sys.unraisablehook` receives no cross-thread destruction error The test fails on `main` with the same `PyPackedBlobWriter is unsendable` / `PyDedicatedBlobWriter is unsendable` messages and passes with this change. ## Validation - `uv run make build` - `uv run pytest -q python/tests/test_blob.py -k 'packed_blob_writer or failed_blob_writer'` — 32 passed - `uv run pytest -q python/tests/test_blob.py` — 139 passed - `uv run make lint` — Ruff, Pyright (0 errors), Rust formatting, and Clippy passed - commit hooks — Ruff, Ruff format, Rust formatting, and typos passed

…-format#7821) ## Why Implements the new "Lance Docs" design as the real mkdocs site, so `make serve` / `make build` produce it directly. The design is delivered by a dedicated custom MkDocs theme (`docs/theme/`) whose markup and CSS are ported from the design implementation: header with section tabs, GitHub star count and Discord link, grouped side navigation with ink rules, right-hand scroll-spy table of contents, light/dark mode, and the hero/stats/features homepage with real highlighted code blocks. MkDocs keeps providing what fits the design — the markdown pipeline (admonitions, content tabs, snippets, autolinked URLs), awesome-pages navigation, and the search plugin's index — while the theme owns everything user-facing, including a lightweight search overlay (`/` or `Cmd/Ctrl+K`) and on-demand mermaid rendering. `mkdocs-material` is no longer a dependency. Preview: ```bash cd docs make serve ```  ## Summary by CodeRabbit * **New Features** * Rolled out a custom documentation theme with a refreshed homepage, improved typography, navigation, and responsive layout. * Added light/dark mode switching with saved preference. * Introduced a search overlay with keyboard shortcuts, ranked results, and highlighted matches. * Enhanced docs rendering with copy-to-clipboard code controls, on-demand Mermaid diagrams, horizontal table scrolling, and TOC scroll tracking. * Added a dedicated 404 page and improved GitHub star display with caching/offline fallback. * **Documentation** * Updated the MkDocs configuration and theme setup to use the new theme design.  --------- Co-authored-by: prrao87 <35005448+prrao87@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…x/column drops Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- reject staged singleton (frag-reuse/MemWAL) creation over a concurrent removal-only transaction, with a resolver unit test - assert conflict message content in the new staged-transaction tests - reword the staging doc: a deferred-remap rewrite leaves stale fragment coverage until the remap catches up, not duplicate entries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ame conflicts - validate CreateIndex at manifest build time: indexed fields must still exist in the schema (closes the alter_columns type-cast gap that the pairwise Project check could not see, since a cast commits Merge with a new field id) and non-system removed segments must still be present (closes the mirror race where a drop racing a same-name replacement silently no-ops) - collapse the frag-reuse/MemWAL/regular-name conflict booleans into one symmetric name-intersection rule covering both directions - regression tests: staged commit after an alter_columns cast fails as incompatible; drop_index racing a replacement fails retryably and succeeds on retry; resolver unit test covers all conflict directions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cursor Bot mentioned this pull request May 16, 2026

Superseded: feat: expose staged index segment transactions #2

Open

ragnorc force-pushed the ragnorc/two-phase-index-segments branch from 53aa593 to 5cfc088 Compare June 23, 2026 16:31

github-actions Bot added enhancement New feature or request A-java A-namespace A-index A-encoding A-format A-python A-ci A-deps and removed enhancement New feature or request labels Jun 23, 2026

ragnorc and others added 16 commits June 23, 2026 20:53

Merge branch 'main' into ragnorc/two-phase-index-segments

75c5c79

chore: release beta version 9.0.0-beta.3

e88655d

chore: release beta version 9.0.0-beta.4

8eed312

chore: release beta version 9.0.0-beta.5

6ae7dba

zhangyue19921010 and others added 30 commits July 15, 2026 07:05

chore: bump main to 9.1.0-beta.0

9effdb1

Unreleased version after creating v9.0.0-rc.1

chore: release beta version 9.1.0-beta.1

e4529e6

chore: release beta version 9.1.0-beta.2

9139486

fix(table): route tos:// scheme to ConditionalPutCommitHandler (lance…

3bece4e

…-format#7824)

chore: reduce CodeRabbit review noise (lance-format#7822)

0179b59

fix(index): conflict-check staged CreateIndex against concurrent inde…

18effc9

…x/column drops Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'main' into ragnorc/two-phase-index-segments

47c621f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: expose staged index segment transactions#3

feat: expose staged index segment transactions#3
ragnorc wants to merge 428 commits into
mainfrom
ragnorc/two-phase-index-segments

ragnorc commented May 16, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

ragnorc commented May 16, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Summary by cubic

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

ragnorc commented May 16, 2026 •

edited by cubic-dev-ai Bot

Loading