Dev docs: perf research notes (round-2 optimization candidates, PG18/19 opportunities) - #29
Closed
SferaDev wants to merge 2 commits into
Closed
Dev docs: perf research notes (round-2 optimization candidates, PG18/19 opportunities)#29SferaDev wants to merge 2 commits into
SferaDev wants to merge 2 commits into
Conversation
…19 opportunities Forward-looking research documentation distilled from the ClickHouse-gap perf session: - dev/docs/NEXT_OPTIMIZATIONS.md: local 10M gap analysis vs ClickHouse, per-query ratio table, round-1 candidates (N1-N9) with outcomes, and round-2 candidates: FSST string encoding, runtime join filters, native rollup segments, Z-order zone maps, per-value count sidecars, numeric ordered TopN, segment-size sweep verdict (30K confirmed; superlinear cost lead at 120K), and the prefetch-v2 design note (whole-relation TOAST streaming reverted after cold-cache regression). - dev/docs/PG18_19_OPPORTUNITIES.md: read_stream/AIO prefetch analysis, PG19 beta 1 survey, TOAST mechanics status, chunked STORAGE MAIN validation, PG18 btree skip-scan audit notes. References to reverted prototypes (decompressed-column cache, whole- relation toast prefetch) keep their verdicts inline rather than linking to docs that never landed on main.
…fy pending-PR status - Replace /tmp/ch_times.txt reference with a description of the CH run; mark the bench-history JSON as an untracked local artifact and note the run included session prototypes now in pending PRs - Fix exec_topn_text line ref (decompress.rs:1349 -> :2435 on main) - Rephrase '#36 rework just landed' as pending PR (main's PERF #36 phase 1 is reverted); same for the ranking section - Add round-1 outcome note to section 2 so N1-N3 sizing vs measured results is consistent within the doc - Point shmem_query_cache post-mortem at QUERY_ANALYSIS.md - PG18 doc: fix _col_idx type (int2, matches schema), clarify 'item 1/item 4' cross-reference, unify arrows/wrapping
SferaDev
marked this pull request as ready for review
June 12, 2026 12:15
SferaDev
marked this pull request as draft
June 12, 2026 22:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two new research documents under
dev/docs/, extracted from a longer performance session (ClickHouse-gap analysis). Documentation only — no code changes.NEXT_OPTIMIZATIONS.md— local 10M-row ClickBench gap analysis vs ClickHouse 24.8 (same machine, both in Docker): pg_deltax 5.74 s vs CH 2.59 s, per-query ratio table, where the gap concentrates. Round-1 candidates (N1–N9) with honest outcomes (N1 measured −8% vs −70% projected; N2/N3 measured ~nothing), and round-2 candidates chosen to change what work exists rather than micro-optimize: FSST string encoding (R1), runtime join filters on the bloom/sentinel infra (R2), native rollup segments (R3), Z-order zone maps (R4), per-value count sidecars (R5), min/max-ordered numeric TopN (R6). Includes the segment-size sweep verdict (R7: 30K rows/segment confirmed — 120K was 10× slower locally, pointing at a possible superlinear per-segment hot path worth profiling) and the prefetch-v2 design note (whole-relation TOAST streaming prototype reverted after catastrophic cold-cache regression, e.g. Q23 cold 1.9 s → 50.5 s at 100M; v2 must stream only surviving blobs' chunk ranges).PG18_19_OPPORTUNITIES.md— survey of PG18/PG19-beta-1 features pg_deltax can exploit:read_stream/AIO prefetch for TOAST (with the reverted-spike verdict cross-referenced), chunked ~7.8 KBSTORAGE MAINblob rows as a promise-preserving TOAST bypass (validated mechanics, prior art, gotchas), PG18 btree skip scan audit notes, and what PG19 does not deliver (no index prefetching, no executor batching, no pluggable TOAST).Why
These notes capture a full round of measurement-grounded research (what was tried, what was falsified, what's next and why) so future optimization work doesn't re-derive or re-attempt rejected ideas. Several verdicts (decompressed-column cache revert, toast-prefetch revert, segment-size sweep) exist nowhere else once the session branch is retired.
Notes for review
main.DECOMPRESSED_CACHE.md, held patch files (n2_held.patch,n3_held*.patch), the toast_prefetch prototype itself,EC2_VALIDATION_PLAN.md(ships with the bench-infra PR), and thePERF_IMPROVEMENTS.md/QUERY_ANALYSIS.mdsession hunks (all feature-specific: UPDATE/DELETE on compressed partitions (DML P2 + tombstone DELETE) #36 merge-side rework and Performance improvements #47 partition-bloom sentinels — those travel with their feature PRs).Testing
Docs only; no build/test run. Verified no EC2 IPs, SSH key paths, or credentials in the shipped files, and that every in-repo doc reference resolves on
main(or ships in this PR).Extracted from a longer perf session branch as part of a PR split.