Adaptive decimation by default; fork the pre-3.2 uniform decimator into decimate-uniform/

[slimbuck]

Ships a new decimator and keeps the existing one intact beside it. The two are named for how they allocate removal, not as a ranking: --decimate (adaptive) allocates by local error, --decimate-uniform removes at a uniform rate everywhere. Both are supported and they win on different content, so the choice is the user's.

Adaptive decimation (--decimate, the default)

The cost function is the principled L2 field error — the squared L2 norm between the emitted field of two original Gaussians and that of their moment-matched merge, in closed form over Gaussian–Gaussian products, plus a scale-free DC colour term. Unlike the scale-invariant KL cost it replaces, every term scales with Gaussian volume, so merging large distant Gaussians costs far more than a geometrically similar tiny merge and coarse structure survives deeper into the cascade. On fr-sky this is worth roughly +9.6 to +11.6 dB over the uniform path at every level.

Selection is re-costed rather than one-shot. selectMerges consumed the priority pass's pairwise costs in a single ordered walk, so costs went stale as groups formed; selectMergesRecosted re-evaluates a cluster's best edge after it changes, in commit/refresh waves, so every merge executes at a cost validated against current state. The evaluation kernel (recost-core.ts) is stateless over members — a cluster's moments are recomputed from its ≤ MAX_GROUP immutable cache rows on every evaluation, so no per-root float state is stored and a commit is pure integer structure updates. GpuRecost runs the refresh rounds on WebGPU (the CPU keeps the heap and commit decisions; the GPU replays the commit log against its own structure copy and bulk-evaluates queued roots), with an inline f64 path when there is no device or the buffers exceed the adapter's binding limits.

Scenes too large for one resident block go through block-local planning instead of whole-scene selection:

• block-prepare.ts gathers one core-plus-halo view and runs exact KNN over it on the GPU; the candidate rows are then fixed for the plan's lifetime.
• block-plan.ts runs the re-costed wave loop against that local view and emits an ordered plan of merge pairs plus marginal costs in commit order.
• block-allocation.ts spills each plan to scratch and allocates prefixes across blocks by a global merge-heap over plan costs, so the merges actually executed are globally cost-ordered even though planning was block-local.
• block-merge-stream.ts replays the selected prefixes and emits one core block at a time, keeping selection arrays, gathered fields and moment-matching inputs block-local.

Residency is policy-driven. A private ceiling of min(48 GiB, half of system RAM) — not an allocation, and not exposed on the CLI — steers the core block size, the candidate K, and a per-generation gate that falls back to one-shot selection when the re-costed state would not fit alongside the base state, so a large scene regains re-costing as soon as the cascade shrinks under it. Multi-block adaptive decimation requires WebGPU and scratch storage and fails fast with an actionable message when either is missing. Spatially incoherent multi-block input is staged once to a KD-ordered PLY rather than paying scattered gathers every generation.

Two operational notes for this path specifically: it writes per-block merge plans to --scratch-dir (12 B per planned merge, per generation, removed as each generation is consumed) and, for incoherent input, one full staged copy of the scene — neither of which the previous decimator wrote. Its intermediate-generation RAM-vs-spill threshold also moved with the residency ceiling.

The uniform decimator, forked whole (--decimate-uniform)

src/lib/decimate-uniform/ is the 3.1.x decimator, copied file-for-file. It is kept because it wins: lower memory, and measurably better at depth on scenes of uniformly-sized Gaussians — it leads by up to ~1 dB at L3–L6 on both snow scenes in scenes/DECIMATION-RESULTS.md. It is also the reference baseline every comparison in that document is measured against, which is why bit-for-bit reproducibility matters.

The fork imports nothing from src/lib/decimate/ except moment-match.ts (which has no diff against 3.1.x and whose worker handler is shared) and spatial/kd-tree.ts (build and query paths unchanged, shared with k-means). Work on the adaptive path therefore cannot change uniform output. Each file is provable in one command — git diff main:src/lib/decimate/select.ts src/lib/decimate-uniform/select.ts is empty — and eight of eleven have zero non-import changes. The two deviations are documented in the directory README: repointed import paths, and gpu-knn.ts consuming the current interleaved FlatKdTree instead of packing that same layout itself (buildFlatKdTree is verified structurally identical to the 3.1.x KdTree.flatten() at every size, so the uploaded bytes are unchanged).

Output equivalence is verified end to end against the installed 3.1.6 binary, not argued from inspection: fr-sky (5.81M, 3 SH bands, multi-block) and fr-snow (26.1M, DC only, 13 blocks), six chained halvings each, every level byte-identical, with PSNR reproducing the published old columns exactly. Wall clock matches too (41.6s vs 42.0s on sky, 123.4s vs 124.7s on snow).

This is why the fork exists rather than sharing infrastructure. Sharing the new KNN and partition code with the uniform path cost 30% on fr-sky and 65% on fr-snow, and produced different bytes — the global forest KNN is exact but orders equidistant neighbours differently, the partition's outlier-fence gate had changed for single-block scenes, and the block-size clamp was using the adaptive path's cache stride, which can select a different block size on devices where the binding limit bites.

Shared infrastructure

• KNN moved from per-block trees to a global forest for the adaptive path. knnForestQuery carries top-K state across parts and culls a part outright when its AABB cannot improve the carried worst, making the result exact over the union with no halos and no verification pass. Parts build in parallel on the worker pool into SharedArrayBuffers so query tasks read them without copies, and GpuKnn compiles one kernel per part with the root index and AABB baked in as constants.
• kd-tree.ts gains buildFlatKdTree, which builds straight into the flat GPU layout with no intermediate pointer-node graph, and the layout is compacted to nodePositions / nodeChildren with denormalised positions so a tree walk does one read per visit instead of two. KdTree's own build and query paths are untouched.
• The adaptive merge stream is zero-copy: rows write straight into the consumer's chunk views instead of rolling buffers the consumer copied from,

[Copilot]

🟡 Human review recommended

It introduces a large, high-impact decimation pipeline (including new GPU and multi-block logic) and also contains at least one confirmed CLI behavior bug that should be fixed before merging.

Pull request overview

This PR introduces a new adaptive decimation path as the default (--decimate) while preserving the pre-3.2 decimator as a frozen, reproducible baseline under --decimate-uniform (src/lib/decimate-uniform/). It also refactors supporting infrastructure (KD-tree flattening, KNN querying, merge streaming) and adds extensive test coverage for parity, GPU re-costing, and multi-block behavior.

Changes:

• Add adaptive decimation building blocks: field‑L2 cost, re-costed greedy selection, forest KNN, block-local planning + global prefix allocation, and zero-copy merge streaming into chunk buffers.
• Fork the legacy decimator into src/lib/decimate-uniform/ and add CLI + library entry points to select it (--decimate-uniform, decimateSourceUniform), with parity tripwires.
• Extend/adjust shared infrastructure: introduce buildFlatKdTree, add shared WGSL chunk(s), and factor GPU compute boilerplate into a reusable helper.

File summaries

File	Description
test/kd-tree.test.mjs	Updates KD-tree flattening test coverage to the new buildFlatKdTree layout + adds split-plane correctness test.
test/gpu-recost.test.mjs	New acceptance tests for GPU re-cost refresh parity and selection equality.
test/decimate.test.mjs	Updates degenerate/coincident-scene behavior expectations for the new re-costed selection path.
test/decimate-uniform-parity.test.mjs	New parity tripwires for --decimate-uniform (exact kNN set + pinned digest + GPU/CPU agreement).
test/decimate-source.test.mjs	Replaces legacy-statistics parity check with in-domain/finite aggregate invariants.
test/decimate-select.test.mjs	Updates selection tests to match cost-ordered agglomeration semantics and MAX_GROUP.
test/decimate-recost.test.mjs	New tests for stateless re-costed eval properties + selectMergesRecosted invariants.
test/decimate-priority.test.mjs	Updates priority-pass reference to new CPU edge-cost cache and adds persist-only mode test.
test/decimate-partition.test.mjs	Adds tests for deterministic split jitter and halo building behavior; validates residual marking.
test/decimate-multiblock.test.mjs	New multi-block adaptive-path tests (GPU requirement + scratch-plan cleanup).
test/decimate-merge-stream.test.mjs	Updates tests to the new “fill destination buffers” streaming protocol and new selection semantics.
test/decimate-knn.test.mjs	Reworks KNN tests to validate exact forest KNN and canonical neighbor ordering.
test/decimate-edge-cost.test.mjs	New unit tests for field‑L2 edge-cost properties (coincident ~0; size scaling).
test/decimate-block-plan.test.mjs	New tests for block planning, prefix allocation, replay, and scratch cleanup behavior.
test/cli.test.mjs	Adds CLI coverage for --decimate-uniform.
src/lib/workers/tasks.ts	Adds worker tasks for forest part building + forest KNN queries; updates KD flatten handler to new flat layout.
src/lib/spatial/kd-tree.ts	Introduces buildFlatKdTree and updates FlatKdTree layout (positions/children interleaving).
src/lib/index.ts	Exposes decimateSourceUniform and associated types in the public library entry.
src/lib/gpu/shaders/chunks/gaussian-l2.ts	New shared WGSL chunk for field‑L2 cost components used by GPU kernels.
src/lib/gpu/index.ts	Removes exported EdgeCostCache type from the GPU barrel export.
src/lib/gpu/gpu-kmeans.ts	Refactors compute boilerplate to use new shared makeKernel helper.
src/lib/gpu/compute-kernel.ts	New helper module for consistent WebGPU compute kernel setup (shader + bind groups).
src/lib/decimate/select.ts	Replaces legacy matching/closure selection with cost-ordered agglomeration and exports MAX_GROUP.
src/lib/decimate/select-recost.ts	New re-costed greedy selection implementation with optional GPU refresh waves (GpuRecost).
src/lib/decimate/partition.ts	Extends partitioning with deterministic jitter, residual marking, shared fence, and halo builder.
src/lib/decimate/merge-stream.ts	Changes merge streaming to fill consumer-provided destination buffers (enabling zero-copy output).
src/lib/decimate/knn-core.ts	Replaces block KNN with exact forest KNN query over flat KD parts.
src/lib/decimate/block-producer.ts	Updates block-producer source to drive the new destination-buffer streaming protocol.
src/lib/decimate/block-prepare.ts	New GPU block preparation step (core+halo gather + GPU KNN + canonical ordering).
src/lib/decimate/block-merge-stream.ts	New stream that replays selected block plan prefixes and emits block-local output into dest buffers.
src/lib/decimate/block-allocation.ts	New scratch-plan persistence + global heap allocation of plan prefixes across blocks.
src/lib/decimate-uniform/select.ts	New frozen copy of pre-3.2 selection implementation for --decimate-uniform.
src/lib/decimate-uniform/README.md	Documents the frozen baseline contract and allowed deviations for the uniform fork.
src/lib/decimate-uniform/partition.ts	New frozen copy of pre-3.2 partitioning implementation.
src/lib/decimate-uniform/merge-stream.ts	New frozen copy of pre-3.2 merge stream (payload-based, rolling scratch buffers).
src/lib/decimate-uniform/knn-core.ts	New frozen copy of pre-3.2 CPU block KNN querying.
src/lib/decimate-uniform/index.ts	Exposes decimateSourceUniform and its renamed option/spill types.
src/lib/decimate-uniform/edge-cost-cpu.ts	New frozen CPU edge-cost implementation for uniform path parity and CPU fallback.
src/lib/decimate-uniform/decimate-source.ts	New frozen orchestrator for uniform decimation path, including spill handling and device usage.
src/lib/decimate-uniform/block-producer.ts	New frozen payload-based block producer source for the uniform path.
src/cli/index.ts	Adds --decimate-uniform flag, decimator selection, and derives an internal memory budget ceiling from system RAM.
README.md	Updates user-facing CLI docs to explain adaptive vs uniform decimation.

Review details

• Files reviewed: 53/54 changed files
• Comments generated: 1
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.