Transparent DML on compressed partitions (INSERT / UPDATE / DELETE) - #49
Merged
Conversation
Two Unix-epoch vs PostgreSQL-epoch (offset 946,684,800 s) confusions in src/scan/exec/segments.rs: 1. Bloom probe encoding (live bug): the per-segment bloom BUILD side (compress.rs) hashes TypedColumn values, which store timestamps/dates as Unix-epoch microseconds — but the probe side hashed the raw PG-epoch datum of the qual constant. The hashes never matched, every segment was falsely bloom-rejected, and `ts = const` / `ts IN (...)` on timestamp/date columns returned ZERO rows on compressed partitions. New bloom_probe_encode() converts the constant into the build-side domain (also normalizing the f32 bit-pattern case the Eq path previously special-cased and the InList path missed). 2. Meta-table min/max encoding (latent): load_segments_heap stored the time column's meta `_min_`/`_max_` identity-encoded (PG-epoch datum) in ColMinMax, while every consumer (decode_encoded_to_pg_i64, segment_all_rows_pass, constant_extract_key_for_segment) decodes the colstats encoding (Unix-epoch us) — timestamps 30 years early. On current main all reachable consumers either pass load_minmax=false (correct block below) or get the value overwritten by the colstats load, so no wrong results are reachable today — but the contract violation bites any new consumer of the map. Convert at load, matching the colstats encoding contract. Regression tests: bloom_probe_encode_matches_build_domain unit test + TestEpochEncoding integration class (timestamp sort column with year-2000-adjacent constants: equality, IN-list, absent-value, DATE payload equality, MIN/MAX aggregate shapes, boundary range predicates). The equality/IN tests return zero rows without fix #1. Extracted from the perf/clickhouse-gap-session branch (commit 7ea1b97 and the bloom-probe portion of fcfb1dc).
…epoch bug in in-list minmax filter The colstats min/max in-list filter identity-encoded raw PG-epoch datums for timestamp/date IN-lists on non-time columns while comparing against Unix-epoch-microsecond colstats bounds - falsely pruning every segment (zero rows). Route all constant encoding (eq, range, in-list, bloom probe) through encode_datum_to_i64, and skip minmax pruning entirely when an element is unencodable rather than risk a wrong prune.
Remove the INSERT-side read-only limitation on compressed partitions. INSERTs (direct or parent-routed) land in the partition heap at native heap speed — no decompression, no segment rewrite. Every read path unions the compressed segments with the uncompressed heap tail: - DeltaXDecompress/DeltaXAppend gain a Phase 3 heap-tail scan (leader- only in parallel plans), with the full plan qual re-enabled when a tail exists (heap rows are never batch-filtered). - DeltaXCount/DeltaXMinMax fold loose heap rows into their metadata- derived results at exec time, so those pushdowns stay enabled. - DeltaXAgg bails to plain Agg over heap-tail-aware scans when any involved compressed partition has loose rows (its columnar accumulators cannot ingest row-form tuples); a begin-time guard errors on the stale-cached-plan race instead of dropping rows. - Top-N pushdown and pathkey claims are disabled at plan time when a tail exists; a [-4, flag] custom-private marker + relcache invalidation on the heap's empty->non-empty transition (sent by the leaf trigger's INSERT branch) catch stale cached plans. - INSERT ... ON CONFLICT stays rejected (conflict inference cannot see rows inside segments). UPDATE/DELETE stay rejected unchanged. deltax_compact_partition() folds loose rows into new segments appended to the existing companion tables (colstats/blooms/text_lengths/ valbitmap kept exact or dropped, never under-covering; catalog counters, column_minmax, HLL/ndistinct refreshed) and truncates the loose region; the background worker compacts automatically each cycle. Extracted from a longer perf session (DML P1, validated against a plain-PostgreSQL twin table); UPDATE/DELETE (P2) and tombstones (P2.5) follow in a separate PR. dev/docs/COMPRESSED_DML.md carries the full program design.
Independent correctness/cleanup pass over the P1 transparent-INSERT work: - Close the data-modifying-CTE bypass of the INSERT ... ON CONFLICT rejection: a wCTE hides the ModifyTable in PlannedStmt.subplans under a top-level CMD_SELECT, so the ExecutorStart hook never saw it. Conflict inference against empty leaf indexes would silently miss conflicts with segment rows (duplicate inserts under ON CONFLICT DO NOTHING). New integration test covers it. - Guard the leader-only Phase 3 heap tail against parallel_leader_participation=off: Gather never drives the leader's plan copy when workers launch, so the tail would be silently dropped. The begin-time guard errors with a remedy instead. - Re-validate the partition catalog row under the AccessExclusive lock in deltax_compact_partition(): a concurrent decompress committing between the unlocked is_compressed check and the lock grant would have made compaction treat the fully-restored heap as loose rows and truncate real data. - Self-heal dead loose-row pages in the loose_rows == 0 compaction branch: a REPEATABLE READ compaction deletes (not truncates) and autovacuum is off on compressed partitions, so the heap would otherwise stay at nonzero blocks forever — pinning scans on the heap-tail path and re-taking the AEL every worker cycle. - Reject json_extract tables in compaction with a clear remedy (synthetic columns have no heap presence — the cursor would fail with a raw SQL error) and skip them in worker auto-compaction to avoid an error loop. - Disambiguate partition_oid_for_companion() by is_compressed: two deltatables with the same table name in different schemas both register same-named partitions in the catalog; the reverse lookup could return the wrong schema's heap and union foreign rows into the scan. - Use the executor snapshot (es_snapshot) instead of GetActiveSnapshot() for the DeltaXCount/DeltaXMinMax heap-tail fold, matching the DeltaXDecompress/Append Phase 3 scan. - Tighten the Phase 3 deform guard to also check the scan-slot tupdesc: for DeltaXAppend the slot is the parent's layout, and a dropped column on either side would silently misalign the positional deform.
The pre-existing correctness test asserted the P0 contract (parent-routed INSERT into a compressed partition raises). DML P1 intentionally accepts plain INSERTs — rows land in the partition heap and are unioned with segment data at scan time. Update the test to insert into both the plain and deltax tables, verify the loose row is visible through the deltax read path (count + total parity), and keep the join-equality check. UPDATE/DELETE rejection coverage is unchanged.
P2 decompose-on-write: the ExecutorStart interceptor walks every ModifyTable node (UPDATE/DELETE/MERGE, incl. data-modifying CTEs), locates candidate segments via the read path's pruning pipeline (dml_candidate_segments), and decomposes them back into ordinary heap rows (decompose_segments_for_dml) inside the user's transaction before the planned DML runs — plain WAL-logged heap DML, so MVCC/rollback/ crash recovery need no extension-side machinery. Whole-segment DELETE drops segments without restoring rows (guards: no RETURNING, no user row DELETE triggers, all quals batch-provable AllPass), with an ExecutorRun hook folding dropped rows into es_processed. P2.5 tombstones-as-rows (DELETE-only fast layer): qualifying DELETEs append (segment_id, row_offset) tombstone rows to a per-partition _tombstones companion instead of decomposing — near-native DELETE latency. Scans AND tombstones into the selection vector on every read path (incl. parallel workers, which load the map under the shared snapshot); DeltaXCount counts live rows; DeltaXMinMax/DeltaXAgg carry stale-plan guards that error when tombstones appeared after planning. Compaction folds tombstoned segments in; the catalog max_segment_id high-water mark is raised before any meta delete so segment ids are never reused (shared caches are keyed by (companion_oid, segment_id)). UPDATE stays on decompose-on-write (correct, transactional, slower). Extracted from a longer perf session; stacks on DML P1 (feat/compressed-insert).
…ETEs Review fixes for the P2/P2.5 fast paths (whole-segment drop + tombstone), which must be disabled whenever something observes the deleted rows: - RETURNING is now checked per ModifyTable (returningLists), not via planned_stmt.hasReturning, which only reflects the top-level query: a data-modifying CTE's DELETE ... RETURNING previously took the fast path and returned no rows to the outer query. - AFTER DELETE triggers with OLD TABLE transition relations (REFERENCING OLD TABLE) now disable the fast paths too — checked on the leaf and on the statement's named target, since transition capture on a partitioned parent collects rows from every leaf. - es_processed credit is only applied when the ModifyTable is the statement's top plan node: CTE-deleted rows never count toward the outer command tag in PostgreSQL. - PENDING_DML_EXTRA_ROWS is a Vec now so nested statements (trigger or function-body DML on another compressed partition) can't drop the outer statement's pending credit. Also: reuse ddl_if_not_exists in copy.rs, doc update, and three new integration tests covering CTE RETURNING, CTE tag integrity, and transition-table observation.
…s succeeds The correctness suite still asserted the P0/P1 contract (UPDATE/DELETE on compressed partitions raises), which this PR obsoletes. Rewrite both tests to assert the new transparent-DML contract, mirroring the accepts_parent_routed_insert pattern: run the same statement against the plain twin and the deltax table, commit, and require parity through the deltax read path. - test_partition_edge_compressed_partition_rejects_dml -> test_partition_edge_compressed_partition_applies_dml: direct partition-targeted UPDATE/DELETE on a compressed partition; asserts rowcount parity, post-UPDATE value visibility, post-DELETE row absence with the partition still compressed (tombstone-aware read path), total count parity, and full ordered result-set parity. - test_rtabench_compressed_partition_rejects_parent_routed_update_delete -> test_rtabench_compressed_partition_accepts_parent_routed_update_delete: parent-routed UPDATE/DELETE across all four COPY layouts; asserts rowcount parity, per-order row parity (update) / absence on both sides with the touched partition still compressed (delete), total count parity, and join-case equality. INSERT ... ON CONFLICT rejection coverage is unchanged.
The new correctness DML tests exposed two jsonb bugs: - restore_segment_rows (shared by decompress_partition, decompose-on-write UPDATE/DELETE, and compaction) decoded jsonb blobs through the UTF-8 text codec path. jsonb blobs hold binary jsonb varlena payloads, so any restore of a jsonb-bearing segment panicked with 'invalid UTF-8 in LZ4 data'. Decode them byte-safe (decompress_column_byte_values) and convert back to JSON text via jsonb_out (jsonb_binary_to_text, the inverse of jsonb_text_to_binary) for the INSERT literal. - The SPI SELECT builders in compress_partition_impl and the compaction path did not cast jsonb columns to ::text, but the accumulate path reads jsonb as String (canonical JSON text) — pgrx rejects the oid mismatch. Include ColumnKind::Jsonb in the ::text cast. Adds a pg_test covering compress -> UPDATE (decompose) -> compact -> decompress with jsonb payloads incl. NULLs.
Combines PR #34 (feat/compressed-insert) and PR #36 (feat/compressed-update-delete) — already stacked, so this branch is #36's head — and merges origin/main (epoch fixes were on the stack's base; main gained flatten-partitions, the worker launcher refactor, and the Q19/Q25 perf work from #47). Conflict resolutions beyond the textual ones: - flatten_eligible_companions: refuse to flatten a parent when any compressed child holds loose heap rows (P1) or tombstones (P2.5). Flattening removes children from simple_rte_array, which is where every DML-aware plan-time gate looks; a flattened plan would sail past the gates and trip the exec-time stale-plan errors instead of falling back. - deltax_planner: never flatten under DML_BYPASS — set_rel_pathlist early-returns under the bypass, so a flattened (dummy) rel would be left path-less and scan empty during internal maintenance. - DeltaXCount fallback: merged main's partial-window segment folding with P2.5 tombstone subtraction (live_row_count for fully-contained segments, time-mask ∧ not-tombstoned for straddling ones). - Q19 point-scan bloom pruning (new on main) probed with the raw PG-epoch datum for timestamp/date constants — the exact epoch bug class #33 fixed. Routed it through bloom_probe_encode. - maintain_one_table (main's worker refactor): re-added the P1/P2.5 auto_compact_partitions step between auto-compress and retention.
run.sh filtered psql output through grep 'Time|psql: error', which drops server-side ERROR lines while \timing still prints a duration — a failing query gets recorded as a fast success. This produced a bogus result set when a json_extract metadata mismatch errored every order_events query (all three runs recorded the time-to-error, some 100x 'faster'). Port the ON_ERROR_STOP=on + QUERY_ERROR marker guard from clickbench/run.sh, and make parse-results.sh map QUERY_ERROR to null explicitly (it previously only degraded to null via flush padding).
The heap-loose-rows and tombstone gates (relation_heap_is_empty + companion_may_have_tombstones) run at several plan-time sites × every compressed partition on every planning: heap-tail Top-N gating, collect_compressed_children, flatten eligibility, the DeltaXAgg/MinMax disables. Each probe is a table_open + nblocks lseek (+ a syscache name lookup for the tombstone sibling); measured +3-4ms/query on ClickBench (18 partitions) and +5-35ms on RTABench point queries (127 partitions). Cache both answers per backend, keyed by partition/companion OID, in the existing hook-cache family: cleared wholesale by invalidate_compressed_cache on any relcache invalidation. Correctness rides the invalidation contract the writers already provide for cached plans — deltax_note_compressed_insert fires on the empty→non-empty heap transition, the tombstone writer / decompose / compaction invalidate the partition explicitly, and compaction's TRUNCATE swaps relfilenodes. Scope guards: - Only COMPRESSED partitions are cached; plain INSERTs into uncompressed children fire no invalidation, so those emptiness checks (DeltaXAppend would silently drop their rows) keep the physical probe. - Exec-time consumers (begin_deltax_append heap-tail collection, attach_tombstones, count/minmax folds, companion_has_live_tombstones stale-plan guards) stay uncached as the last line of defense.
The per-backend probe cache (6de2acd) only amortized within a single planning; a fresh backend still paid ~40ms on the first wide point query (RTABench Q7 46ms cold vs 3.2ms warm) because both the plan-time gates AND begin_deltax_append / the count-minmax heap fold / attach_tombstones / the DeltaXAgg+MinMax stale guards physically opened every one of ~127 partition heaps + _tombstones tables — relations a segment-only scan never otherwise touches — forcing cold relcache + smgr builds. Replace the physical probes with two catalog flags on deltax_partition: has_loose_rows, has_tombstones maintained transactionally by the writers: - INSERT trigger (deltax_note_compressed_insert): idempotent UPDATE + per-txn memo (xact/subxact callbacks clear it so a rolled-back first insert re-runs); skipped under DML_BYPASS. - decompose_segments_for_dml: sets has_loose_rows when rows restored. - insert_dml_tombstones: sets has_tombstones. - compaction / mark_partition_compressed: clear both. Every plan/exec gate now reads a single bulk-loaded per-backend flag map (hook::DML_FLAGS, one SPI joining deltax_partition->pg_class), invalidated by the same relcache-callback path as COMPRESSED_NAMESET. Opens nothing beyond the deltax_partition catalog. Correctness: flags are MVCC-atomic with the rows (a snapshot that sees the rows sees the flag) and refreshed at command start via the writer's relcache invalidation — the same contract COMPRESSED_NAMESET already relies on. The outer statement snapshot precedes the flag-map SPI snapshot, so a stale-false flag cannot coexist with a post-state read; a stale-true flag is merely conservative (slow path, still correct). Behavior change: analytic readers no longer block behind a decompose UPDATE's AccessExclusive partition lock (the gates no longer AccessShare the partition heap) — they read a consistent MVCC snapshot instead (standard PG readers-don't-block-writers). Verified non-torn; test_concurrent_reader_sees_consistent_data updated to assert the pre-or-post invariant rather than the old lock-blocking mechanism. New TestDmlGateFlags covers the flag lifecycle + rollback.
The standalone DML_FLAGS SPI (470eadc) still left the tombstone gate and the exec-time heap-tail collectors calling partition_oid_for_companion — a branch-only SPI cached *per companion*, so a fresh backend running a wide point query (RTABench Q7/Q9-Q13, 127 partitions, no time filter) paid ~127 of them ON TOP of the flag SPI. Net: point queries stayed at 17-44ms vs main's ~10ms (+37% geomean). Build the flag map from ONE double-join SPI that resolves both the partition heap OID (user schema) and the _meta companion OID (_deltax_compressed) alongside the flags, and key the map by BOTH oids (same flag tuple). Every gate is then a single hash lookup: - plan-time loose/tombstone gates key by partition oid, - exec-time heap-tail collectors + DeltaXAgg/MinMax guards + attach_tombstones key by companion oid, - partition_oid_for_companion is resolved ONLY when a partition actually has loose rows (rare) — off the steady-state path entirely. RTABench (clean main baseline, same data, fresh-backend-per-try): point queries Q7/Q9-Q13: 17-44ms -> 11-12ms (main ~10ms) hot geomean: main 0.1095, probes 0.1501 (+37%), this 0.1134 (+3.6%) hot sum: main 11.52s, this 11.56s (+0.3%) Residual +1ms on the smallest queries is the second per-backend SPI (COMPRESSED_NAMESET + DML_FLAGS); folding them into one load is a possible follow-up but is <=1ms on 11ms fresh-backend timings. 607 unit (PG17+18), 426 integration green.
RTABench and ClickBench are read-only; nothing tracked the write side that DML-on-compressed adds. bench_dml.py measures it, self-contained (synthetic 300k-row time-series, no external data), against a plain-PostgreSQL twin so every op is an absolute warm latency AND a ratio to native — the ratio is the product requirement. Ops (each measured warm, rollback-isolated so runs don't accumulate): single/batch/COPY INSERT (loose region), point DELETE (tombstone), retention DELETE (whole-segment drop), DELETE...RETURNING + point/range UPDATE (decompose-on-write). Plus the read-after-write cliff: a GROUP BY aggregate pristine (DeltaXAgg pushdown) vs after one INSERT (pushdown disabled) vs after compaction — the 'does DML slow the happy path' number. Results + timestamped history archive via save_bench_results; a pinned dml_baseline.json drives a soft regression gate (DML_BLESS=1 to record). Targets: make bench-dml / bench-dml-keep / bench-dml-bless. Initial run flagged two issues to fix next (json_extract+heap-tail layout guard; retention DELETE not taking the whole-segment-drop fast path), so the baseline is intentionally left unblessed until those land.
Bug 1 — json_extract table + loose rows tripped a false layout-mismatch. The heap-tail layout guards compared the partition heap's physical column count against the companion col_names count, which includes json_extract synthetic columns (attnum 0, no pg_attribute row). So any INSERT into a json_extract-configured compressed table followed by a read errored with a cryptic 'layout does not match the compressed layout'. The synthetic column is absent from the physical heap and from a scan that doesn't select it, so the correct test is heap-vs-scan-OUTPUT layout, not heap-vs-total-columns: - decompress.rs heap-tail (begin + emit): compare rel_natts to the scan slot's natts; a query that DOES select a json_extract column widens the slot and is caught with a clear, actionable message. - count_minmax.rs heap-tail: expected_natts = MetadataInfo::physical_col_count() (attnums != 0), not col_names.len(). Queries not referencing the extracted path now work on json_extract tables with loose rows; ones that do get a clear 'run compact/decompress' error. Bug 2 — time-range retention DELETE decomposed/tombstoned every row instead of whole-segment-dropping (~80x slower: 156ms vs 1.9ms on the 300k-row bench). classify_segment_quals proved the min/max coverage (AllPass on the Lt) but its NULL-safety check read the time column's colstats _nonnull_count — which compression writes as a ZERO placeholder (the time column's authoritative stats live in _meta) — so it saw 0 non-null rows and returned Ambiguous, defeating the whole-drop. The time column is non-null by construction in a compressed (non-default) partition (NULL partition keys route to the default), so classify_segment_quals now takes the time_column name and skips its NULL check. Fixes already-compressed data (read-side, no recompress). Retention DELETE now whole-drops: bench 156ms -> 1.9ms (0.1x of plain PG). Tests: TestCompressedInsertJsonExtract (insert+read works, synthetic-col read errors clearly, compaction refused clearly); time-range whole-drop regression asserting has_loose_rows stays false (whole-drop, not decompose). 607 unit (PG17), 430 integration green; clippy clean.
- Describe what's implemented (INSERT loose region + heap tail, catalog-flag gates, tombstone/whole-drop/decompose DELETE+UPDATE, compaction) rather than the original phased proposal with status prefixes and deviations. - Remove all references to other products; state the decompose-vs-bitmap rationale from first principles (metadata-exactness). - Document the catalog-flag gate mechanism, the readers-don't-block change, the cost profile (from bench-dml), limitations (ON CONFLICT, uniqueness, json_extract), and a concrete 'things to improve' list.
The existing DML tests use a narrow (ts, text, int, float8) schema, so they exercise only a few codecs. This adds a wide-schema table — one column per codec-relevant type (int2/4/8, float4/8, hi/lo-card text, varchar, char(n), bool, date, timestamptz, jsonb, a constant column, and nullable columns with NULLs) — loaded across 4 segments with a byte-identical plain-PG twin, and asserts full-row equality after INSERT (loose), UPDATE (decompose, single + multi segment), DELETE (tombstone), compaction, and full decompress. This is the durable guard that a codec or on-disk-format change can't silently break a DML round-trip. Also surfaces (and tracks via xfail) a pre-existing, non-DML compression bug the matrix found: numeric/uuid/bytea columns — types with no specialized codec — corrupt on compressed read (stored/returned as their text-output form). Latent because ClickBench/RTABench never use these types. The main matrix excludes them; test_unspecialized_type_roundtrips_through_compression documents the corruption and flips to XPASS when the codec/fallback is fixed. 438 passed + 3 xfailed (PG17).
Closes the logical-replication gap for DML-on-compressed. Decompose-on-write re-inserts a whole segment's rows into the partition heap before the user's UPDATE/DELETE runs; under Scenario-1 logical replication (subscriber holds the user table, (de)compresses independently) those internal inserts would land on the subscriber as duplicate rows. Fix: tag the WAL of internal maintenance writes (decompose restored-row inserts + companion deletes; compaction segment folds + loose-row removal) with a dedicated 'deltax_internal' replication origin, set per-record via the backend-local replorigin_session_origin (InternalOriginGuard in compress.rs — set directly, not replorigin_session_setup, so concurrent internal writers don't contend on the origin's progress slot). PostgreSQL filters by origin per-WAL-record at decode time, so a subscription created WITH (origin = none) drops exactly this internal churn while the user's own origin-less DML (run after decompose returns, origin reset) replicates normally. Scenario 2 (replicate raw companion+heap storage) uses the default origin = any and receives the tagged changes unchanged — tagging is safe for both topologies. The origin is created in the extension SQL (superuser, any wal_level; name has no reserved 'pg_' prefix); the Rust paths look it up by name via replorigin_by_name and no-op if absent. Test: test_logical_replication.py:: test_decompose_dml_on_compressed_replicates_without_duplication — Scenario 1 + origin=none: 200 rows, compress, UPDATE (decomposes a 40-row segment), assert subscriber stays 200 (no duplication) with the 40 rows updated; compaction is likewise invisible. Scenario 2 still passes. Remaining (documented in COMPRESSED_DML.md §7/§8): tombstone / whole-segment -drop fast-path DELETEs touch only companion tables, so they don't emit a replicable event for a Scenario-1 subscriber — needs a force-decompose switch. 607 unit (PG17), 439 integration + 3 xfail; clippy clean.
… under Scenario 1 Closes the remaining logical-replication gap. The tombstone and whole-segment-drop DELETE fast paths touch only companion tables, so under Scenario-1 replication (publish the user table, companions excluded, origin=none) a fast-path delete produces no replicated event and the subscriber keeps the row. New GUC pg_deltax.replicable_deletes (bool, USERSET, default off): when on, DELETEs on compressed partitions skip both fast paths and decompose-on-write, so the delete runs as an ordinary heap DELETE that replicates like any row delete (the decompose re-inserts are origin-tagged and filtered by origin=none). Off by default — the fast paths are far faster; Scenario 1 publishers turn it on (typically ALTER DATABASE ... SET). Scenario 2 leaves it off and lets tombstone rows replicate through the companion tables. Implemented by gating allow_whole_drop on !replicable_deletes in the ExecutorStart interceptor (tombstone_eligible derives from it, so both fast paths turn off together and all candidates fall to decompose). Test: test_logical_replication.py:: test_replicable_deletes_guc_propagates_delete_on_compressed — Scenario 1 + origin=none + GUC on: DELETE 40 rows on a compressed partition, subscriber goes 200 -> 160 with device gone. Also harden test_worker.py::_cleanup to retry its teardown DROP on DeadlockDetected: these tests run against the postgres DB where the real background worker is active, so the cleanup DROP races the worker's concurrent maintenance of the same table (a normal, retryable DDL-vs-maintenance deadlock). Timing-sensitive; surfaced when the new GUC shifted the worker cycle. Product behavior is correct (PG aborts one side); the test now retries rather than fail on teardown. COMPRESSED_DML.md §7/§8 updated. 607 unit (PG17), 440 integration + 3 xfail; clippy clean.
The correctness harness compared the curated query battery only against a freshly-compressed (pristine) table. But DML changes the physical state a read path sees — loose rows in the heap tail, tombstones subtracted on the fly, decomposed segments, post-compaction segments — which is exactly where a scan/agg/Top-N path's DML-awareness could silently diverge from PostgreSQL. New suite mutates BOTH twins identically (so plain PG stays the oracle), then re-runs curated_smoke_cases against each state: pristine, loose_inserts, tombstone_deletes, decompose_updates, mixed (loose + tombstones + decomposed at once), and compacted. Each state asserts the physical shape it claims to exercise actually exists (has_loose_rows / has_tombstones), so the coverage can't silently collapse into a plain heap. Reuses the existing dataset builder and query generators — no new query definitions. Marked . Full correctness suite: 1041 passed, 3 skipped, 6 xfailed.
The 3 xfail cases I'd added (numeric/uuid/bytea compressed round-trip) only did compress->read, with no DML — so they duplicated the pre-existing, strict=True tracking in tests/correctness/test_codecs_extended.py (numeric / time / uuid / bytea fallback corruption, from commit e1aa83f on main), and were weaker (strict=False). That corruption is a base-codec bug independent of DML. Replace the duplicate xfails with a comment: the type matrix excludes those four types, points at the canonical tracking, and says to add them back once the codec fallback is fixed.
# Conflicts: # src/compress.rs # tests/test_worker.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines and finishes @SferaDev's two stacked PRs — #34 (transparent INSERT, DML P1) and #36 (UPDATE/DELETE + tombstone DELETE, DML P2/P2.5) — rebased onto current
main, so compressed partitions accept plainINSERT/UPDATE/DELETE/MERGEwith full transactional semantics.Full design, cost profile, limitations, and remaining work are in
dev/docs/COMPRESSED_DML.md.On top of the two base PRs, this branch adds:
has_loose_rows/has_tombstones), replacing per-partition physical probes — keeps the analytics happy path at parity (fixes a fresh-backend point-query regression; verified on ClickBench + RTABench EC2).make bench-dml— a DML write-latency benchmark (vs a plain-PG twin) plus a wide type-coverage suite that guards DML round-trips against codec/format changes.Tests: 607 unit (PG17 & PG18), 438 integration + 3 xfail (PG17), clippy clean.