Merge the frame assembly's device-to-host copies [#133]

[iderex]

Closes #133.

What changed

The frame assembly loop issued one blocking cudaMemcpy per compressed block.
It now issues the copies on one non-default stream with a single terminal
cudaStreamSynchronize, and before issuing anything it merges consecutive
blocks that are contiguous on both sides into one copy.

The merge is what pays, and what lets it is a property of the frame format
rather than of this corpus. Device slots are strided by the frame's block max,
and every data block but the last decodes to exactly that, so a run of full
blocks is one contiguous device range. Compressed blocks land at consecutive
output offsets wherever no uncompressed block separates them. Both sides are
tested per block, so a frame that breaks either just gets more copies and never
wrong bytes.

Which shape, and why not the other two

The issue proposed two shapes and asked for whichever the bench rewards. Both
were built and measured, and both were rejected.

Shape	64 KB	256 KB	1 MB
Async copies straight into the caller's out, one terminal sync	-1.9 %	+3.0 %	+1.8 %
Async copies through a 32 MiB pinned bounce buffer, in waves	-2.5 %	+25.5 %	+10.8 %
Merge contiguous runs, async on one stream, one terminal sync	-29.9 %	-11.1 %	-3.1 %

The first is flat for the reason the issue's own design note predicts: out is
the caller's buffer and is pageable, and a device-to-pageable
cudaMemcpyAsync is synchronous with respect to the host by specification, so
the stall it was meant to remove is still paid. The second removes the stall for
real and loses anyway, because it adds one host memcpy of the whole output and
past the 64 KB rung there are too few submissions left for the saved latency to
cover it.

So the terminal synchronization is not what paid. That is the first row, and it
is flat. The win is the merge.

Numbers

Three interleaved passes, after / before / after / before / after / before in one
session, both binaries built from the same tree, recorded 2026-08-10 inside the
digest-pinned nvidia/cuda:12.6.2-devel-ubuntu24.04 container
(sha256:738fba0fbdb225b7a2931c58a5c8f03a84d3cd2f6a84975826a157339ef750b8) on
the RTX 3080 (sm_86, driver 560.94, CUDA 12.6, nvcc V12.6.77), 3 warmup + 30
measured runs per rung, output byte-verified against the original once per rung
before timing.

bench_lz4 --frame bench/corpora/silesia/* --warmup 3 --runs 30

Block max	Before p50 (3 samples)	After p50 (3 samples)	Change
64 KB	549.923 / 519.201 / 545.261 ms	374.037 / 375.946 / 381.386 ms	-29.9 %
256 KB	440.591 / 438.248 / 441.267 ms	393.992 / 389.628 / 390.543 ms	-11.1 %
1 MB	448.822 / 440.027 / 442.964 ms	429.779 / 438.233 / 423.059 ms	-3.1 %

The two sides do not overlap at any rung. The gap widens with the block count,
which is what removing a per-block cost looks like; the 1 MB rung has 203 blocks
and correspondingly little to win.

A first attempt at this A/B was discarded rather than reported. Two of its twelve
rung measurements came out near three times their neighbours, which is host
contention rather than a property of either binary. The table above is a second
session with nothing else running against the device.

Proof that the guards bite

The merge test has two halves, and each has a fixture that goes red when that
half is deleted and stays green when it is present. Neither shape was in the
suite before, and the first measurement of that is the reason both exist: with
the output half deleted, the whole suite stayed green.

tests/frame_twin.cu 2d, a stored block between two compressed ones. Only
compressed blocks occupy device slots, so blocks 0 and 2 are adjacent in device
memory and 64 KiB apart in the output. With the output-side test deleted:

first byte mismatch at offset 65540:
  [65540] have 41 want a2
FAIL /w/tests/frame_twin.cu:119: equal_bytes(...) | stored-between-compressed bytes
0% tests passed, 1 tests failed out of 1

The 41 is block 2's text arriving where block 1's stored bytes belong.

tests/frame_twin.cu 2e, a short compressed block followed by a full one. No
compressor emits a short block anywhere but last, so this one is spliced from
two frames carrying the same descriptor; the frame format permits it and liblz4
accepts it, which is asserted in the fixture before the placement is checked.
With the device-side test deleted:

FAIL /w/tests/frame_twin.cu:119: equal_bytes(...) | short-then-full bytes
0% tests passed, 1 tests failed out of 1

Each deletion reddens only its own fixture. Both fixtures assert the frame shape
they claim to build, so they cannot quietly stop covering it if liblz4's
behaviour moves.

Fail-closed and lifetime

The capacity check is untouched and still runs before the first byte is written,
so a too-small dst still yields CUDEC_ERR_OUTPUT_TOO_SMALL with no partial
output. Every CUDA failure still maps to CUDEC_ERR_CUDA through FRAME_CUDA,
and the terminal synchronization's status is checked rather than dropped.

The drain is an RAII guard, so it runs on every exit path from the loop
including the error returns, and it is declared after the device buffer it
protects so it is destroyed first. An in-flight copy can never outlive the
device memory it reads. Neither offset sum in the merge can overflow: the
device side is bounded by n * block_max, whose overflow is already refused
above, and the output side by total, which the capacity walk computed without
one.

Gate

Full suite inside the digest-pinned container on the RTX 3080:

100% tests passed, 0 tests failed out of 36

frame_twin and frame_host_negative are both in that set. Prettier reports
docs/BENCHMARKS.md clean.

The device-side sanitizer net did NOT run. Compute Sanitizer cannot attach to
this device through the WSL2 paravirtualized path, which is #258, and the remedy
the tool names is a machine-wide elevation that is not taken here. Nothing in
this PR should be read as having passed that gate.

This change had no second reader. The evidence above stands in place of one.