Skip to content

Opt-in dual-write segment files with mmap reads (storage v2 P1) - #32

Draft
SferaDev wants to merge 2 commits into
mainfrom
feat/segment-file-storage
Draft

Opt-in dual-write segment files with mmap reads (storage v2 P1)#32
SferaDev wants to merge 2 commits into
mainfrom
feat/segment-file-storage

Conversation

@SferaDev

@SferaDev SferaDev commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What

Opt-in storage v2 P1/P1b: when pg_deltax.blob_storage = 'dual', every partition compression additionally writes its compressed segment blobs into one immutable .dxs segment file per partition under $PGDATA/pg_deltax/<db_oid>/, and the scan path serves blob bytes from that file via mmap instead of detoasting from the _blobs companion table.

  • Default is unchanged (blob_storage = 'toast'); dual is strictly opt-in.
  • All data remains in regular PostgreSQL tables. The .dxs file is a redundant, read-optimized copy of the TOAST blobs — replication, pg_dump, and backups are unaffected.
  • Fail-open everywhere: any file problem (missing, renamed, truncated, bad magic/CRC) falls back to the TOAST path, which dual mode guarantees is fully populated. Fallback is logged (LOG on open failure, WARNING on checksum mismatch) but never an error.

Why

Cold reads through TOAST pay B-tree descent + tuple reassembly + decompression-unfriendly random I/O per blob. An append-only file with an index-at-footer gives the kernel one sequential mapping to fault in, and blob "fetches" become pointer arithmetic into the mapping.

Measured (EC2 c6a.4xlarge, 100M-row ClickBench)

Partition hits_p20130702 recompressed under blob_storage=dual; Q28-shape sort query confined to that partition; fully cold per run (PG restart + echo 3 > /proc/sys/vm/drop_caches):

read path run 1 run 2
mmap'd .dxs file 894 ms 892 ms
TOAST (files renamed away) 3,966 ms 3,958 ms

4.4× cold-read win. The hidden-file runs double as a live validation of the fail-open path — results stayed correct with the file gone.

Design

Full design + measured verdict in dev/docs/STORAGE_V2.md. Digest:

  • File format (src/segment_file.rs): header magic/version, blob payloads, (col_idx, segment_id, offset, len, crc32c) index at the footer, footer CRC. Index-at-footer enables incremental appends across COPY blob drains.
  • Write protocol: tmp file + append + index/footer + fsync + rename + directory fsync, all before the compressing transaction commits and records the path in the new deltax.deltax_partition.blob_file column. Generation-suffixed names make recompressions collision-free; a committed blob_file always points at a durable, complete file.
  • One write path: both producers — the SPI compress path (src/compress.rs, one shot) and the COPY direct-backfill path (src/copy.rs, incremental across BLOB_BUFFER_THRESHOLD drains; P1b) — go through the same SegmentFileWriter, so the durable-write protocol exists once. A test-only one-shot encoder pins the writer's output byte-for-byte. On the COPY path file-side errors warn once, abandon the writer, and never fail the COPY.
  • Read path (src/scan/exec/segments.rs): when the catalog points at a file, blob slots get BlobBytes::Cached raw-pointer views into an Arc<Mmap> (MappedSegmentFile), with the Arc held in each segment's blob_file_backing so views never outlive the mapping. Per-blob CRC verification is on in debug builds and behind pg_deltax.verify_file_checksums in release.
  • Lifecycle: decompress unlinks the file and clears blob_file; retention drop unlinks best-effort (orphans are left for the planned P2 GC sweep).

Testing

  • make build — clean.
  • make clippy — zero warnings.
  • make test (PG 17) — 577 unit tests passed, including the new segment_file round-trip/corruption/incremental-writer tests.
  • make integration-test PG_VERSIONS=17 — full suite green, including the new tests/test_blob_file.py (catalog/file creation, result equivalence vs toast mode, fallback after file deletion and after corruption/truncation, file removal on decompress, COPY-path dual-write).

Extracted from a longer performance session (perf/clickhouse-gap-session); this PR carries only the storage-v2 feature.

SferaDev added 2 commits June 12, 2026 14:21
@SferaDev
feat: opt-in dual-write segment files with mmap reads (storage v2 P1/…
4438d46 
…P1b)

With pg_deltax.blob_storage = 'dual' (default: 'toast', unchanged), each
partition compression additionally writes its compressed segment blobs to
one immutable .dxs file under $PGDATA/pg_deltax/<db_oid>/ (tmp + fsync +
rename, index-at-footer, crc32c, generation-suffixed names), recorded in
the new deltax.deltax_partition.blob_file column. The scan path mmaps the
file and serves blob slices as zero-copy views instead of detoasting,
failing open to the TOAST blobs table on any file problem (missing,
renamed, truncated, bad magic/CRC) — dual mode keeps TOAST fully
populated, so all data stays in regular PostgreSQL tables and
replication/pg_dump/backups are unaffected.

- src/segment_file.rs: file format, one-shot writer, incremental
  SegmentFileWriter (COPY-path P1b, appends across blob drains,
  footer-last), MappedSegmentFile mmap reader, unlink helpers.
- src/compress.rs: SPI compress path dual-write before commit; decompress
  unlinks the file and clears blob_file.
- src/copy.rs: direct-backfill dual-write with warn-once fail-open (file
  errors never fail the COPY).
- src/scan/exec/segments.rs: file-backed blob load in load_segments_heap
  and fetch_segment_blobs; Arc<Mmap> held per segment (blob_file_backing)
  so views never outlive the mapping.
- src/partition.rs: retention drop unlinks the file best-effort.
- GUCs: pg_deltax.blob_storage (toast|dual), pg_deltax.verify_file_checksums.

Measured (EC2 c6a.4xlarge, 100M ClickBench, hits_p20130702, fully cold):
Q28-shape sort 894/892 ms via mmap'd .dxs vs 3966/3958 ms via TOAST —
4.4x cold-read win. Design + verdict: dev/docs/STORAGE_V2.md.
@SferaDev
Cleanup pass: single durable write path, dedup, corrupt-file e2e test
e4f1b63 
- write_partition_blob_file now goes through SegmentFileWriter, so the
  crash-critical tmp+fsync+rename+dir-fsync protocol exists exactly once
  (the one-shot encoder stays as a #[cfg(test)] oracle pinning the
  writer's byte-for-byte output); SPI path also gains .tmp cleanup on
  append error.
- Drop the local PG17/PG18 tupdesc_get_attr copy in segment_file.rs;
  widen scan::exec::datum_utils::tupdesc_get_attr to pub(crate) instead.
- Extend the BlobBytes Send/Sync SAFETY comment to cover the mmap'd
  segment-file backing alongside the DSA cache pin.
- Add an end-to-end corrupt/truncated-file fallback test (unit tests
  covered detection; only the missing-file wiring was covered e2e).
- STORAGE_V2.md: status line no longer claims nothing is implemented;
  path scheme corrected to <partition_id>_<generation>.dxs (matching
  the implementation and the doc's own decision log).
- Remove unused pytest import in tests/test_blob_file.py.
@SferaDev
SferaDev marked this pull request as ready for review June 12, 2026 12:39
This was referenced Jun 12, 2026
Draft
Closed
SferaDev added a commit that referenced this pull request Jun 12, 2026
@SferaDev
docs: scope sentinel/blob-file comments to features actually in this …
e577088 
…stack

Partition bloom sentinels (#35) and dual-mode segment files (#32) are
separate PRs; the new decompose comments referenced them as if present.
Rephrase as forward-looking interaction notes (matching the existing
'PERF #47, not yet on this branch' convention) or drop the reference.
SferaDev added a commit that referenced this pull request Jun 12, 2026
@SferaDev
docs: scope sentinel/blob-file comments to features actually in this …
ed4d4f7 
…stack

Partition bloom sentinels (#35) and dual-mode segment files (#32) are
separate PRs; the new decompose comments referenced them as if present.
Rephrase as forward-looking interaction notes (matching the existing
'PERF #47, not yet on this branch' convention) or drop the reference.
@tsg

tsg commented Jun 12, 2026

Copy link
Copy Markdown
Member

I was avoiding this to keep everything into Postgres tables, so replication and crash recovery work out of the box. The opt-in dual mode idea is interesting, especially if it tells us how much we can win with it.

But I'm worried about the extra maintenance, will give it a think.

SferaDev added a commit that referenced this pull request Jun 12, 2026
@SferaDev
docs: scope sentinel/blob-file comments to features actually in this …
17c8ec4 
…stack

Partition bloom sentinels (#35) and dual-mode segment files (#32) are
separate PRs; the new decompose comments referenced them as if present.
Rephrase as forward-looking interaction notes (matching the existing
'PERF #47, not yet on this branch' convention) or drop the reference.
SferaDev added a commit that referenced this pull request Jun 12, 2026
@SferaDev
docs: scope sentinel/blob-file comments to features actually in this …
405c7fa 
…stack

Partition bloom sentinels (#35) and dual-mode segment files (#32) are
separate PRs; the new decompose comments referenced them as if present.
Rephrase as forward-looking interaction notes (matching the existing
'PERF #47, not yet on this branch' convention) or drop the reference.
SferaDev added a commit that referenced this pull request Jun 12, 2026
@SferaDev
docs: scope sentinel/blob-file comments to features actually in this …
44f4aed 
…stack

Partition bloom sentinels (#35) and dual-mode segment files (#32) are
separate PRs; the new decompose comments referenced them as if present.
Rephrase as forward-looking interaction notes (matching the existing
'PERF #47, not yet on this branch' convention) or drop the reference.
@SferaDev
SferaDev marked this pull request as draft June 12, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SferaDev @tsg