jsonb sub-object extraction + batch @> containment - #48
Open
tsg wants to merge 4 commits into
Open
Conversation
Phase 1 — extract whole jsonb sub-values into companion columns: - json_extract specs accept type: "jsonb"; the named path (object, array, or scalar) is stored as a narrow companion column. Worker threads accumulate JSON text (jsonb_in is not thread-safe); the merge phase and the sequential COPY path produce the binary jsonb varlena form. - walk_chain_shape recognizes bare `->` chains (jsonb leaf) alongside the existing `->>` text-leaf chains; `(chain)::T` casts stay unrecognized. - Planner-walker gating fixes: the phase-0 pre-walk stops at the cscan boundary, so chains referenced only by the scan-level qual (the RTABench Q4/Q8 `@>` shape) never triggered the rewrite — the fire/no- fire gate now also collects the cscan's own qual sigs. Guarded to non-empty scan targetlists: with zero output columns the executor does not fill qual-only synthetic slot positions and the rewritten qual silently dropped every row (same root cause as the DeltaXDecompress crash tracked in TestJsonbContainmentCountStar). Phase 2 — evaluate `@>` once per dictionary entry, not per row: - jsonb companion blobs already flow through the text compression pipeline and get dictionary-encoded at low cardinality; add BatchCompareOp::JsonbContains (recognized from `Var @> Const`, jsonb on both sides, var on the left) and fold it into Phase-1 blob decompression: dict blobs pay O(dict_size) jsonb_contains calls (~10/segment for RTABench's status arrays) instead of O(row_count), materializing jsonb datums only for matching rows. Non-dict blobs fall back to per-row evaluation, which also covers `@>` directly on the physical jsonb column. - Segment-level skip when no dict entry contains the template. Kept separate from segment_skippable_by_dict, which agg worker threads call and must stay free of PG calls. - Thread-safety gates: the Top-N paths disable themselves when a containment qual is present (their candidate collectors run on internal threads and would silently skip the filter); DeltaXAgg never admits `@>` (hook operator whitelist); JsonbContains quals on segment_by columns are handed back to ExecQual. - jsonb_contains is invoked through a raw #[link_name] extern — the pg_sys wrapper is a Rust-ABI pg_guard shim that DirectFunctionCall2Coll cannot take. RTABench EC2 (181M events, c6a.4xlarge), stable across three full runs: Q4 640→263 ms, Q8 240→108 ms vs the pre-branch baseline; the rest of the suite is unchanged. Storage cost of the extra x_status jsonb companion: +0.75%. Extraction specs for status added to the EC2 and local RTABench harnesses. Claude-Session: https://claude.ai/code/session_01EtNpDdiKHU4nrXHs3im3si
The zero-output-column `SELECT count(*) ... WHERE <chain> @> ...` shape no longer crashes on the DeltaXDecompress path: the rewritten qual is now recognized as a JsonbContains batch qual, ExecQual is skipped, and the filter runs during Phase-1 decompression — no scan-slot access. The strict xfail was failing as XPASS on the merged tree; it is a regular test now. The underlying executor gap remains for shapes that are not batch-foldable (docstring updated). Claude-Session: https://claude.ai/code/session_01EtNpDdiKHU4nrXHs3im3si
_cleanup reset pg_deltax.mock_now (ALTER SYSTEM + reload) first, which SIGHUPs the worker awake on the real clock — far ahead of the test's mock time — so its next pass immediately starts partition DDL on the test table. That deadlocks against the session's DROP TABLE CASCADE, which runs while the same transaction holds uncommitted catalog-row DELETEs (worker waits on the session's xact, session waits on the worker's relation lock). The window is a few ms wide, so it surfaced as binary-dependent CI flakiness. Drop the table and catalog rows first, reset the clock last, and retry once on a residual deadlock with an in-flight timer pass. Claude-Session: https://claude.ai/code/session_01EtNpDdiKHU4nrXHs3im3si
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends
json_extractwithtype: "jsonb": a whole sub-value (object/array) is extracted into a narrow companion column, and@>containment quals are evaluated during blob decompression — once per dictionary entry (~10/segment for RTABench's status arrays) instead of once per row. Segments whose dictionary contains no matching entry are skipped entirely.Also fixes the planner-walker gate so chains referenced only by the scan qual (the RTABench Q4/Q8 shape) trigger the synthetic-column rewrite, and — as a side effect of the batch fold — the
SELECT count(*) ... WHERE <chain> @> ...crash shape now works (xfail converted to a regular test).RTABench EC2 (181M events): Q4 640→189 ms, Q8 240→131 ms vs the May baseline; suite total 11.03 s warm (best recorded). Storage cost of the
x_statuscompanion: +0.75%.Thread-safety: Top-N paths self-disable on containment quals (their workers can't call
jsonb_contains); DeltaXAgg never admits@>; segment_by jsonb quals fall back to ExecQual.