Skip to content

perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858) #861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mosuka merged 1 commit into from
Jul 12, 2026
Merged

perf(vector/index): read + write LVS v3 field-dictionary records in HNSW/Flat/IVF (#858) #861

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/ja/src/concepts/indexing/vector_indexing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,11 @@ let opt = HnswOption {
は int8 SIMD multiply-accumulate 1 回 + scalar 補正 3 回に縮約され、
検索時の per-element dequantize は不要。
- segment ファイルは `LVS1` magic + 16 byte header で始まり、reader
はロード時にフォーマットを判定。
はロード時にフォーマットを判定。header はバージョン（機能のはしご:
v2 = HNSW グラフブロックの ordinal エンコード、v3 = セグメントごとの
フィールド名辞書）を持ち、v3 以降は辞書そのものも header に載る —
各レコードはフィールド名をインラインで繰り返す代わりに 16 ビット ID
で参照する。

### Two-stage rerank（Issue #481 Stage 2）

Expand Down
2 changes: 1 addition & 1 deletion docs/ja/src/laurus/persistence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ sequenceDiagram
4. **クラッシュセーフティ**: 書き込みとコミットの間にプロセスがクラッシュした場合、次回起動時にWALがリプレイされます
5. **アトミックなファイル書き込み**: セグメントファイル（HNSW の `.hnsw` グラフ・そのメタデータ・削除ビットマップなど）は一時ファイルへ書き込んでからアトミックにリネームして配置されるため、書き込み途中のクラッシュでも切り詰められたファイルではなく直前にコミット済みのファイルがそのまま残ります
6. **チェックサム検証**: これらのファイルは CRC-32（`.hnsw` と `.hnsw.f32` rerank sidecar は footer、`metadata.json` と削除ビットマップは framing）を持ち、ロード時に検証されるため、ディスク上の静かな破損を正常データとして読まずに検出できます。チェックサム導入前に書かれたファイルもそのままロードできます（ファイル単位で任意）。また、ローダーはヘッダーを信頼する前にバッファ確保サイズを実ファイルサイズで上限を縛るため、サイズフィールドが破損していても巨大なメモリ確保（OOM）を引き起こさずに破損として拒否します
7. **バージョン付きセグメントヘッダー**: ベクトルセグメントは共有ヘッダーにフォーマットバージョンを持ちます。HNSW セグメントはバージョン 2 で書き込まれ、そのグラフブロックは 64 ビットのドキュメント ID の代わりにセグメントローカルな 32 ビット ordinal を格納します（ディスク上のグラフブロックはおよそ半分になります）。古いビルドが書いたバージョン 1 のセグメントもそのままロードでき、次の書き直し（コンパクションやマージ）でバージョン 2 に更新されます。Flat と IVF のセグメントは引き続きバージョン 1 で書き込まれるため、古いビルドでも読み込めます
7. **バージョン付きセグメントヘッダー**: ベクトルセグメントは共有ヘッダーにフォーマットバージョンを持ち、機能のはしごを形成します。バージョン 2（HNSW のみ）はグラフブロックを 64 ビットのドキュメント ID の代わりにセグメントローカルな 32 ビット ordinal で格納します（ディスク上のグラフブロックはおよそ半分になります）。バージョン 3（全ベクトルインデックス型）はセグメントごとのフィールド名辞書を追加し、各レコードはフィールド名をインラインで繰り返す代わりに 16 ビットの ID で参照します（レコードごとに名前の長さ + 2 バイト分縮小されます）。新しいセグメントはバージョン 3 で書き込まれ、古いビルドが書いたセグメント（バージョン 1・2）もそのままロードでき、次の書き直し（コンパクションやマージ）で更新されます

## Write-Ahead Log（WAL）

Expand Down
8 changes: 6 additions & 2 deletions docs/src/concepts/indexing/vector_indexing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,11 @@ let opt = HnswOption {
collapses to one int8 SIMD multiply-accumulate plus three scalar
corrections — no per-element dequantization at search time.
- Segment files start with the `LVS1` magic + a 16-byte header so the
reader can detect the format at load time.
reader can detect the format at load time. The header carries a
version (a feature ladder: v2 = ordinal-encoded HNSW graph block,
v3 = per-segment field-name dictionary) and, from v3 on, the
dictionary itself — records then reference field names by a 16-bit
id instead of repeating the full name inline.

### Two-stage rerank (Issue #481 Stage 2)

Expand Down Expand Up @@ -327,7 +331,7 @@ skipped.

Every vector segment header carries element counts (`num_vectors`,
`n_clusters`, the HNSW graph's `node_count` / `layer_count` /
`neighbor_count`) and per-record byte lengths (`field_name_len`, PQ
`neighbor_count`) and per-record byte lengths (the v1/v2 inline `field_name_len`, PQ
`codes`) that the reader uses to size `Vec` / `HashMap` capacities and
read buffers. The HNSW CRC footer is verified before the structural
parse, but legacy footer-less segments — and the writer reload paths,
Expand Down
2 changes: 1 addition & 1 deletion docs/src/laurus/persistence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ sequenceDiagram
4. **Crash safety**: If the process crashes between writes and commit, the WAL is replayed on the next startup
5. **Atomic file writes**: Segment files (e.g. the HNSW `.hnsw` graph, its metadata, and the deletion bitmap) are written to a temporary file and atomically renamed into place, so a crash mid-write leaves the previously committed file intact rather than a truncated one
6. **Checksum verification**: Those files carry a CRC-32 (a footer on `.hnsw` and the `.hnsw.f32` rerank sidecar, framing on `metadata.json` and the deletion bitmap) that is verified on load, so silent on-disk corruption is detected instead of being read as valid data. Files written before checksums were added still load (the checksum is optional per file). Loaders also bound buffer allocations against the real file size before trusting a header, so a corrupt size field is rejected as corruption rather than triggering a huge out-of-memory allocation
7. **Versioned segment headers**: Vector segments carry a format version in their shared header. HNSW segments are written with version 2, whose graph block stores segment-local 32-bit ordinals instead of 64-bit document ids (roughly halving the graph block on disk); version 1 segments written by older builds still load and are upgraded to version 2 on the next rewrite (compaction or merge). Flat and IVF segments keep writing version 1, so older builds can still read them
7. **Versioned segment headers**: Vector segments carry a format version in their shared header, forming a feature ladder. Version 2 (HNSW only) stores the graph block as segment-local 32-bit ordinals instead of 64-bit document ids (roughly halving the graph block on disk). Version 3 (all vector index types) adds a per-segment field-name dictionary: records reference field names by a 16-bit id instead of repeating the full name inline, shrinking every record by the name's length plus two bytes. New segments are written at version 3; segments written by older builds (versions 1 and 2) still load and are upgraded on the next rewrite (compaction or merge)

## Write-Ahead Log (WAL)

Expand Down
72 changes: 34 additions & 38 deletions laurus/src/vector/index/flat/reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use crate::storage::Storage;
use crate::vector::core::distance::DistanceMetric;
use crate::vector::core::quantization::QuantizedVectorMeta;
use crate::vector::core::vector::Vector;
use crate::vector::index::format::{QuantHeader, VectorSegmentHeader};
use crate::vector::index::format::{QuantHeader, VectorSegmentHeader, record_prefix_size};
use crate::vector::index::quantized_io::quantized_record_payload_size;
use crate::vector::index::quantized_storage::QuantizedVectorPool;
use crate::vector::reader::{ValidationReport, VectorIndexMetadata, VectorStats};
Expand Down Expand Up @@ -78,7 +78,7 @@ impl FlatVectorIndexReader {
path: &str,
distance_metric: DistanceMetric,
) -> Result<Self> {
use crate::vector::index::alloc_bounds::{checked_capacity, checked_len};
use crate::vector::index::alloc_bounds::checked_capacity;
use std::io::{Read, Seek};

// Open the index file
Expand All @@ -102,9 +102,11 @@ impl FlatVectorIndexReader {

// Read the Issue #481 Stage 1 vector segment header (LVS1).
// Pre-Stage-1 segments are rejected with IncompatibleFormat.
// Matched by reference so `header` (version + field dictionary,
// Issue #633) stays alive for the record parse below.
let header = VectorSegmentHeader::read_from(&mut input)?;
let params = match header.quant {
QuantHeader::Scalar8Bit(p) => p,
let params = match &header.quant {
QuantHeader::Scalar8Bit(p) => *p,
QuantHeader::ProductQuantization { .. } => {
return Err(crate::error::LaurusError::NotImplemented(
"Product quantization (Issue #481 Stage 3) is HNSW-only; \
Expand All @@ -123,12 +125,14 @@ impl FlatVectorIndexReader {
};

// Bytes left for the per-vector records section, captured once at its
// start (Issue #806). Each record is at least doc_id (8) +
// field_name_len (4) + the fixed quantized payload (dim int8 + 8 meta),
// so this stride also bounds the per-record `dimension`-sized int8 read.
// start (Issue #806). Each record is at least the version-dependent
// prefix (doc_id + field reference, Issue #633) + the fixed quantized
// payload (dim int8 + 8 meta), so this stride also bounds the
// per-record `dimension`-sized int8 read.
let records_remaining =
file_size.saturating_sub(input.stream_position().map_err(LaurusError::Io)?);
let record_stride = 12 + quantized_record_payload_size(dimension) as u64;
let record_stride =
record_prefix_size(header.version) + quantized_record_payload_size(dimension) as u64;

let (vectors, vector_ids) = match storage.loading_mode() {
crate::storage::LoadingMode::Eager => {
Expand All @@ -149,17 +153,12 @@ impl FlatVectorIndexReader {
input.read_exact(&mut doc_id_buf)?;
let doc_id = u64::from_le_bytes(doc_id_buf);

// Read field name
let mut field_name_len_buf = [0u8; 4];
input.read_exact(&mut field_name_len_buf)?;
let field_name_len = u32::from_le_bytes(field_name_len_buf) as usize;
checked_len(field_name_len, records_remaining, "flat field_name_len")?;

let mut field_name_buf = vec![0u8; field_name_len];
input.read_exact(&mut field_name_buf)?;
let field_name = String::from_utf8(field_name_buf).map_err(|e| {
LaurusError::InvalidOperation(format!("Invalid UTF-8 in field name: {}", e))
})?;
// Field reference: dictionary id (v3+) or inline name.
let field_name = header.read_record_field(
&mut input,
records_remaining,
"flat field_name_len",
)?;

// Read int8 + meta directly (no dequantize).
let mut int8 = vec![0u8; dimension];
Expand Down Expand Up @@ -189,36 +188,33 @@ impl FlatVectorIndexReader {
let mut offsets = HashMap::with_capacity(num_vectors);
let mut vector_ids = Vec::with_capacity(num_vectors);

// Seek to start of per-vector entries: Flat preamble
// (count u32 + dim u32 = 8 bytes) + VectorSegmentHeader
// (Stage-1 Scalar8Bit = 24 bytes) = 32 bytes.
let start_pos =
8u64 + VectorSegmentHeader::scalar_8bit(params).serialized_size() as u64;
// Seek to the start of the per-vector entries: Flat preamble
// (count u32 + dim u32 = 8 bytes) + the parsed header's real
// size (which includes the v3 field dictionary, Issue #633 —
// reconstructing a fresh header here would omit it).
let start_pos = 8u64 + header.serialized_size() as u64;
input
.seek(std::io::SeekFrom::Start(start_pos))
.map_err(LaurusError::Io)?;

let quant_payload_size = quantized_record_payload_size(dimension) as i64;

for _ in 0..num_vectors {
let start_offset = input.stream_position().map_err(LaurusError::Io)?;

let mut doc_id_buf = [0u8; 8];
input.read_exact(&mut doc_id_buf)?;
let doc_id = u64::from_le_bytes(doc_id_buf);

let mut field_name_len_buf = [0u8; 4];
input.read_exact(&mut field_name_len_buf)?;
let field_name_len = u32::from_le_bytes(field_name_len_buf) as usize;
checked_len(field_name_len, records_remaining, "flat field_name_len")?;

let mut field_name_buf = vec![0u8; field_name_len];
input.read_exact(&mut field_name_buf)?;
let field_name = String::from_utf8(field_name_buf).map_err(|e| {
LaurusError::InvalidOperation(format!("Invalid UTF-8 in field name: {}", e))
})?;

offsets.insert((doc_id, field_name.clone()), start_offset);
let field_name = header.read_record_field(
&mut input,
records_remaining,
"flat field_name_len",
)?;

// Offsets point at the payload start (right after the
// record prefix), so `VectorStorage::get` seeks straight
// to the int8 data without re-parsing the prefix.
let payload_offset = input.stream_position().map_err(LaurusError::Io)?;
offsets.insert((doc_id, field_name.clone()), payload_offset);
vector_ids.push((doc_id, field_name));

// Skip int8 payload + per-vector meta.
Expand Down
48 changes: 24 additions & 24 deletions laurus/src/vector/index/flat/writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ use crate::storage::Storage;
use crate::vector::core::quantization::ScalarQuantParams;
use crate::vector::core::vector::Vector;
use crate::vector::index::FlatIndexConfig;
use crate::vector::index::alloc_bounds::{checked_capacity, checked_len};
use crate::vector::index::alloc_bounds::checked_capacity;
use crate::vector::index::field::LegacyVectorFieldWriter;
use crate::vector::index::format::{QuantHeader, VectorSegmentHeader};
use crate::vector::index::format::{
QuantHeader, VERSION_FIELD_DICT, VectorSegmentHeader, build_field_dict, record_prefix_size,
};
use crate::vector::index::quantized_io::{
quantize_segment, quantized_record_payload_size, read_dequantized_vector,
write_quantized_record,
Expand Down Expand Up @@ -122,9 +124,11 @@ impl FlatIndexWriter {

// Read the Issue #481 Stage 1 vector segment header (LVS1).
// Pre-Stage-1 segments are rejected with IncompatibleFormat.
// Matched by reference so `header` (version + field dictionary,
// Issue #633) stays alive for the record parse below.
let header = VectorSegmentHeader::read_from(&mut input)?;
let params = match header.quant {
QuantHeader::Scalar8Bit(p) => p,
let params = match &header.quant {
QuantHeader::Scalar8Bit(p) => *p,
QuantHeader::ProductQuantization { .. } => {
return Err(crate::error::LaurusError::NotImplemented(
"Product quantization (Issue #481 Stage 3) is HNSW-only; \
Expand All @@ -150,7 +154,8 @@ impl FlatIndexWriter {
// quantized payload (dim int8 + 8 meta).
let records_remaining =
file_size.saturating_sub(input.stream_position().map_err(LaurusError::Io)?);
let record_stride = 12 + quantized_record_payload_size(dimension) as u64;
let record_stride =
record_prefix_size(header.version) + quantized_record_payload_size(dimension) as u64;
checked_capacity(
num_vectors,
record_stride,
Expand All @@ -163,17 +168,9 @@ impl FlatIndexWriter {
input.read_exact(&mut doc_id_buf)?;
let doc_id = u64::from_le_bytes(doc_id_buf);

// Read field name
let mut field_name_len_buf = [0u8; 4];
input.read_exact(&mut field_name_len_buf)?;
let field_name_len = u32::from_le_bytes(field_name_len_buf) as usize;
checked_len(field_name_len, records_remaining, "flat field_name_len")?;

let mut field_name_buf = vec![0u8; field_name_len];
input.read_exact(&mut field_name_buf)?;
let field_name = String::from_utf8(field_name_buf).map_err(|e| {
LaurusError::InvalidOperation(format!("Invalid UTF-8 in field name: {}", e))
})?;
// Field reference: dictionary id (v3+) or inline name.
let field_name =
header.read_record_field(&mut input, records_remaining, "flat field_name_len")?;

// Read quantized payload + dequantize.
let values = read_dequantized_vector(&mut input, dimension, &params)?;
Expand Down Expand Up @@ -457,16 +454,19 @@ impl VectorIndexWriter for FlatIndexWriter {
} else {
quantize_segment(&f32_vectors, self.index_config.dimension)?
};
VectorSegmentHeader::scalar_8bit(params).write_to(&mut output)?;

// Write vectors with field names and quantized records.
// Per-segment field-name dictionary (Issue #633): ids assigned in
// first-appearance order over the exact emission order below.
let (field_dict, field_ids) =
build_field_dict(self.vectors.iter().map(|(_, f, _)| f.as_str()))?;
VectorSegmentHeader::scalar_8bit(params)
.with_version(VERSION_FIELD_DICT)
.with_field_dict(field_dict)
.write_to(&mut output)?;

// Write vectors with dictionary field ids and quantized records.
for ((doc_id, field_name, _), (int8, meta)) in self.vectors.iter().zip(records.iter()) {
output.write_all(&doc_id.to_le_bytes())?;

// Write field name length and field name
let field_name_bytes = field_name.as_bytes();
output.write_all(&(field_name_bytes.len() as u32).to_le_bytes())?;
output.write_all(field_name_bytes)?;
output.write_all(&field_ids[field_name.as_str()].to_le_bytes())?;

// Write quantized payload (dim int8 + sum_q + norm_q).
write_quantized_record(&mut output, int8, *meta)?;
Expand Down
7 changes: 0 additions & 7 deletions laurus/src/vector/index/format.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,9 +591,6 @@ impl VectorSegmentHeader {
///
/// The record's field name, or an error when the id is out of
/// dictionary range / the inline name is oversized or not UTF-8.
// Consumed by the reader/writer migration (sub-issue #858); unused
// while this format layer ships alone.
#[allow(dead_code)]
pub(crate) fn read_record_field<R: Read>(
&self,
reader: &mut R,
Expand Down Expand Up @@ -644,8 +641,6 @@ impl VectorSegmentHeader {
/// # Returns
///
/// The fixed prefix size in bytes.
// Consumed by the reader/writer migration (sub-issue #858).
#[allow(dead_code)]
pub(crate) fn record_prefix_size(version: u16) -> u64 {
if version >= VERSION_FIELD_DICT {
10
Expand All @@ -666,8 +661,6 @@ pub(crate) fn record_prefix_size(version: u16) -> u64 {
///
/// `(dictionary, name → field_id map)` for the writer's emit loop, or
/// an error if more than `u16::MAX` distinct names are present.
// Consumed by the writer migration (sub-issue #858).
#[allow(dead_code)]
pub(crate) fn build_field_dict<'a>(
names: impl Iterator<Item = &'a str>,
) -> Result<(Vec<String>, std::collections::HashMap<String, u16>)> {
Expand Down
Loading