fix(compress): cherry-pick upstream #27 — read jsonb SPI datums as JsonB, not text#1
Merged
Merged
Conversation
* fix(compress): read jsonb SPI datums as JsonB, not text Post-INSERT compression via SPI returns native jsonb Datums. The classic path assumed jsonb_out text and failed with DatumError(IncompatibleTypes). Read JsonB and extract the binary varlena payload directly; keep text fallback for COPY-style callers. ValorBrain usage_transactions.reason_metadata compresses at 3.7–4.1x. * replace jsonb/serde roundtrip --------- Co-authored-by: Reviewer Agent <agent@valor.digital> Co-authored-by: Tudor Golubenco <tudor@xata.io>
* fix: valbitmap correctness — fail-safe pruning, retention leak, cross-schema companion lookup Three pre-existing correctness bugs found during review of xataio#31/xataio#34/xataio#35: 1. Text valbitmap wrong-empty-results on segment overflow. A segment with more than VALBITMAP_MAX_DISTINCT (32) distinct text values writes no valbitmap row and contributes nothing to the partition-level column_valmap. When the union of the remaining segments stayed <= 32, querying a value unique to the overflowed segment missed the valmap and took the 'prune every segment without reading the bitmap table' shortcut — returning zero rows instead of the overflowed segment's rows. Pruning now always goes through the per-segment bitmap rows: a valmap miss (empty wanted_bits) prunes exactly the segments that wrote a bitmap row; segments without one are never prunable by valmap logic. The now-redundant ValbitmapCheck::prune_all field is removed. 2. Retention drop leaked _valbitmap companions. auto_drop_partitions dropped blobs/blooms/text_lengths/colstats/meta but not valbitmap, leaving orphaned tables in _deltax_compressed forever. (decompress and the empty-partition cleanup already covered all six.) 3. check_compressed_partition matched companions by partition NAME only. Companion tables live in the single shared _deltax_compressed schema and embed only the table name, so a same-named partition in another schema (or any same-named plain table) was treated as compressed and served the other partition's data. The lookup now confirms via deltax.deltax_partition that the exact (schema, table) pair is the compressed one — at most one partition of a given name can be compressed, so is_compressed disambiguates. Regression tests: tests/test_valbitmap.py (one-segment-overflow shape), tests/test_worker.py (no orphaned companions after retention drop), tests/test_compression.py (two-schema same-name partition). * test: make overflow regression probe robust to minmax/dictionary segment skipping The plan-shape assertions (vb_skipped=1, segments=1) now ride on the present-value 'ov07' query — where only the valbitmap can skip the bitmap-covered segment and the overflowed segment must be scanned. The absent-value probe keeps only the row-count assertion: an absent constant can legitimately eliminate the overflowed segment through exact per-segment evidence (text minmax, dictionary value check) regardless of the valbitmap. * rustfmt: format worker.rs launcher fan-out (inherited unformatted from xataio#38 merge) (cherry picked from commit 1649c98)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The pg_deltax maintenance worker on
postgres-ovh-east(dot_token DB) crash-loops every 60s tick since thepingbackstable (3 jsonb columns: jvm_info/geo_data/extra) got a compression policy:The worker dying kills ALL deltax maintenance for the database — telemetry_request_cost compression, its 90-day retention, and premade-window advancement. (Live cluster is mitigated: the pingbacks policy is cleared until this ships.)
What
Clean cherry-pick of upstream xataio#27 (
a52e67b) ontodotcms-0.2.0: the classic post-INSERT compression path read jsonb SPI datums as text viavalue::<String>(); upstream reads the raw binary varlena payload (JsonbRaw) — no jsonb_out/serde/jsonb_in round-trip, no precision loss. Includes upstream's test_compression.py jsonb coverage. Plus the0.2.0-dotcms.4version bump (packaging.mk requires Cargo.toml == tag).Rollout after merge
v0.2.0-dotcms.4→ release.yml builds the .debPG_DELTAX_VERSIONin postgres-cnpg/Dockerfile → image rebuild🤖 Generated with Claude Code