release: keep compacted query attribution honest

[samo-agent]

Stack / scope

Addresses #136. This reworks and supersedes the approach in #150; it does not modify #150's branch.

This PR is intentionally stacked on #182 (agent/w2-b1 at 2af229a). #182 is still open/draft, so this branch has not been rebased onto main; rebase it after #182 lands. Do not merge this PR before its base. This does not close #122, #156, or #163.

What changed

• Rollup-backed, unfiltered top('query_id') now emits a positive NULL residual equal to max(total wait counts - preserved named query counts, 0) per source row. That residual enters the same ranking stream before n, so compacted-away load can win a top-N slot.
• The residual implementation scans each row's query_counts once with GROUPING SETS and its wait_counts once. It removes fix: keep compacted query attribution honest #150's second relation/query-array scan and window-wide materialized expansion.
• Every explicit query_id filter uses raw samples. The widened retention guard is deliberately narrow: it raises only when ordinary planning selects a rollup that has matching retained coverage in the requested slice before the physical oldest raw sample.
  • _pick_source() naming a fallback rollup is not sufficient.
  • The proof is scoped to the requested database.
  • Valid rollup_1h.minute_counts must contain a non-NULL pre-raw minute; legacy/incomplete hourly detail is handled conservatively.
  • Physical _raw_oldest_sample() is passed to the guard when real coarser history exists; tie-only calls without such history retain _raw_retention_start(). That composes with Release blocker: fresh-install tie drills reject retained raw samples #163's pending logical/physical boundary split.
• compare() NULL-safely pairs rollup residuals, and summary() renders them as (other / unattributed) rather than claiming all compacted load was never captured.

Five-minute install proof

The regression fixture holds one raw row with seven appearances of query 111, five minutes old. It also holds first-minute rollup_1m and valid-detail rollup_1h rows covering only that same retained raw minute. Even though _pick_source(default-left-edge) names a fallback rollup, the exact-query readers remain usable:

• ash.aas(query_id => 111): source=raw, buckets_with_data=1, backend_seconds=7.00
• ash.timeline(query_id => 111): 60 rows, one data row, source=raw, data-row avg_aas=0.12
• ash.top('database', query_id => 111): source=raw, backend_seconds=7.00, pct=100.00
• ash.compare(..., dimension => 'database', query_id => 111): source_1=raw, source_2=raw, avg_delta=0.00, both percentages 100.00

The same fixture proves that unrelated database history does not raise, while a matching old rollup outside raw coverage does raise.

Combined source-selection rule re-derived on #182

Reader	Source selection	Query residual	peak / p99 rule
aas	Any explicit query_id forces raw; matching pre-raw rollup history raises. Wait+query retains the tie guard. Otherwise _pick_source_agg() applies: spans over one hour prefer rollup_1m when the requested start is within its retention; other calls fall through to _pick_source(). Unfiltered/database-only rollup_1h reads use valid minute detail, or disclose rollup_1h_flat; wait-filtered hourly reads use hour arrays.	None.	NULL whenever retained grain exceeds the requested bucket; otherwise computed at the retained/effective grain.
timeline	Same exact-query and aggregate rules as aas, independently for its window.	None.	Same retained-grain rule per effective timeline bucket.
top	Explicit query_id, or dimension='query_id' plus a wait filter, forces raw. Other breakdowns use _pick_source_agg(). Plain hourly database breakdowns may use valid minute detail; hourly wait/query dimensions remain hour-grain.	Emitted only for unfiltered rollup query breakdowns, before ranking and n; raw NULL remains genuinely uncaptured attribution.	Rollup-1m query rows retain minute extrema. Rollup-1h query/wait rows are hour-grain, so sub-hour peak/p99 are NULL.
compare	Coverage/retention comes from per-window aas; dimensional rows come from per-window top. Explicit query filters force both sides raw. A query dimension with a wait filter is raw. Otherwise each side independently selects its aggregate source.	Each rollup-backed query side emits its own residual; the full join pairs NULL keys exactly once.	Inherits top's support rule, then NULLs peak and p99 on both sides when retained grains or effective buckets differ.

There is no disagreement with #182: its effective bounds, disclosed bucket, rollup_1h_flat, and honest extrema NULLing remain authoritative. Exact query filters simply bypass compacted sources. The known _pick_source_agg() end-watermark/completeness gap remains #122 and is not claimed fixed here.

RED / GREEN (verbatim)

1. Residual competes for top-N

RED:

ERROR:  residual top-N mismatch: key=111 source=rollup_1m seconds=3.00 pct=100.00
CONTEXT:  PL/pgSQL function inline_code_block line 98 at ASSERT

GREEN:

NOTICE:  issue #136 rollup_1m residual GREEN: named=3.00/11.54 residual=23.00/88.46 top1=NULL compare=NULL/0.00
DO

The same block exact-asserts two rows, 26.00 backend-seconds, 100.00%, one NULL-safe compare residual with delta 0.00, and the exact summary label/value.

2. rollup_1h top-100 compaction

RED:

ERROR:  rollup_1h top-100 mismatch: rows=100 total=300.00 pct=100.00 residual=/ named=3.00/1.00 source= bucket= peak= p99=
CONTEXT:  PL/pgSQL function inline_code_block line 126 at ASSERT

GREEN:

NOTICE:  issue #136 rollup_1h top-100 GREEN: rows=101 total=400.00 pct=100.00 residual=100.00/25.00
DO

The fixture stores 100 named rows / 300 seconds after hourly top-set truncation while wait totals remain 400 seconds. The final result is 101 rows / 400 seconds / 100%, with an exact 100-second residual and NULL peak/p99 at hour grain.

3. Narrowed exact-query widening

RED from the broad query_id => guard implementation (same-minute first rollup present):

ERROR:  young-install query_id errors: aas=pg_ash: this drill needs exact raw query attribution; raw retention starts at 2026-07-27 18:20:00+00 but the requested window starts at 2026-07-27 17:25:53.654044+00. Narrow the window to start at or after 2026-07-27 18:20:00+00 (the window end is still inside raw retention), or remove query_id. timeline=pg_ash: this drill needs exact raw query attribution; raw retention starts at 2026-07-27 18:20:00+00 but the requested window starts at 2026-07-27 17:25:53.654044+00. Narrow the window to start at or after 2026-07-27 18:20:00+00 (the window end is still inside raw retention), or remove query_id. top=pg_ash: this drill needs exact raw query attribution; raw retention starts at 2026-07-27 18:20:00+00 but the requested window starts at 2026-07-27 17:25:53.654044+00. Narrow the window to start at or after 2026-07-27 18:20:00+00 (the window end is still inside raw retention), or remove query_id. compare=pg_ash: this drill needs exact raw query attribution; raw retention starts at 2026-07-27 18:20:00+00 but the requested window starts at 2026-07-27 18:15:00+00. Narrow the window to start at or after 2026-07-27 18:20:00+00 (the window end is still inside raw retention), or remove query_id.
CONTEXT:  PL/pgSQL function inline_code_block line 128 at ASSERT

GREEN:

NOTICE:  issue #136 young-install GREEN: same-minute 1m/1h rollups ignored; aas=raw/7.00 timeline=raw/0.12 top=raw/7.00 compare=raw/raw; other-db allowed; old rollup rejected
DO

Residual cost measurement

PostgreSQL 17.9; shared_buffers=128MB, work_mem=4MB, JIT off. Fixture: 43,200 rollup_1m rows (30 days × one database, 86 MiB table), each with 100 query pairs, 32 wait pairs, and a 75-count residual. Each variant was fully consumed with count/sum and run three times under EXPLAIN (ANALYZE, BUFFERS, TIMING OFF).

Variant	Three runs	Median	Relative
#182 base, no residual	22.700604s, 27.425866s, 41.263122s	27.425866s	baseline
Straight #150 residual CTE / second query scan	29.271280s, 44.815038s, 34.828003s	34.828003s	27.0% slower than base
Final per-row grouped stream	4.261402s, 4.159544s, 3.967029s	4.159544s	88.1% faster than naive; 84.8% faster than base

Naive #150 spilled 2,628,179 temp blocks read / 2,629,970 written (about 20.05 / 20.07 GiB; shared hit/read 800/20,928). Final spilled 10,172 / 10,192 temp blocks (about 79.5 / 79.6 MiB; shared hit/read 531/10,384). Naive and final outputs were identical: 4,363,200 rows and total count 29,160,000.

This is relevant to #156's rollup-prefix cost class, but it changes only the dimensional query-array work; it does not resolve or close #156.

Validation

• PostgreSQL 17.9 fresh-path reapply and focused regression: GREEN.
• Existing rollup-source readers, 1h seam, release: fix rollup_1h grain honesty #182 grain honesty, calendar/spike ordering, report/compare/retention/chart regressions on PostgreSQL 17.9: GREEN.
• Focused Bug: compacted query rollups are presented as exact AAS #136 fixture on PostgreSQL 14.23 and 18.4: GREEN.
• Simulated Release blocker: fresh-install tie drills reject retained raw samples #163 logical/physical split: same-minute tie allowed; matching physical rollup gap rejected.
• Installer idempotent reapply: GREEN.
• Workflow YAML parse, docs guard, and git diff --check: GREEN.
• Two independent SQL reviews and one docs/test review: no blocker after fixes.

Exact touched line ranges

Relative to #182 head, final-file line numbers:

• .github/workflows/test.yml: 4206-4892
• README.md: 231-246
• blueprints/AAS_API.md: 38-42, 124-125, 132-139, 172, 180-181, 219-240, 244-249, 251, 255-256, 289-293, 462, 464-478
• blueprints/AAS_EXAMPLES.md: 240-253, 320-322, 325, 328-330, 337-338, 341
• blueprints/AAS_USER_STORIES.md: 96-99, 169-170, 227-231
• sql/ash-install.sql: 191-192, 2574, 3496, 3500-3516, 3528, 3585-3589, 3611-3620, 3640-3732, 3756, 3760-3764, 3780-3783, 3793-3797, 3893-3896, 4021-4022, 4062-4063, 4086, 4088, 4090-4119, 4132, 4235-4237, 4274-4275, 4305-4306, 4308-4324, 4503-4506, 4716-4733, 4735-4774, 4776-4784, 4833-4836, 4886-4887, 4924-4933, 4935-4951, 5231, 5348-5355, 5357, 5463-5466, 6512, 6514, 6533, 6536, 6539, 6542

No sibling-owned reader-window validation, privilege, rotation, or production-container surface was changed.