cuda: add experimental lossless compressed expert tier

[ZacharyZcR]

Summary

• add an opt-in DietGPU ANS tier for losslessly compressed per-row int4 experts in VRAM
• decode selected compressed tensors into reusable per-device scratch before grouped expert matmul
• add an offline sidecar pack/load flow with shape, size, codec-bound, and truncation validation
• pipeline pinned sidecar uploads across devices and support aligned Linux O_DIRECT reads
• keep the hottest expert prefix raw and account compressed archive bytes in placement
• document the fixed-order sidecar contract, validation, and operational limits

This is a scoped experimental Linux/CUDA backend. It is not enabled by
default.

Current measured result

GLM-5.2 int4 on 6x RTX 5090, rebased onto upstream/dev at eb7a881.
All fixed-replay runs used 19,456 resident experts, 100% expert hits, no disk
fetches during decode, and identical 33,600 routed selections.

configuration	GPU experts	runs	median
full raw tier, 176.6 GB	9,335	8.17, 8.03, 8.23 tok/s	8.17 tok/s
2,500 raw + 8,128 ANS	10,628	8.01, 8.87, 8.54, 8.82, 8.25 tok/s	8.54 tok/s

• GPU expert capacity: +13.9%
• current fixed-replay decode gain: +4.5% median
• CPU-routed rows: 11,210 -> 9,492
• direct archive read: 19.23 seconds for 110 GB
• total expert placement: about 93 seconds

Earlier 32-token and 128-token experiments measured larger gains
(+8.3% to +15.0%). Those results remain historical datapoints, but the current
rebased fixed-replay result above is the merge claim.

Correctness and output semantics

• ANS decompression reconstructs expert weight bytes exactly.
• CUDA backend correctness and complete sidecar validation pass.
• A controlled earlier prompt produced byte-identical generated content.
• Byte-exact weights do not imply token-identical generation against a
  different raw placement: moving experts from CPU execution to CUDA changes
  floating-point accumulation order, and a near tie can change later routing
  and tokens.

The tier is lossless with respect to stored expert weights, not a general
cross-placement token-identity guarantee.

Validation

• make -C c test-c
• make -C c colibri CUDA=1 COLI_ANS=1 DIETGPU_ROOT=/data/test/dietgpu CUDA_ARCH=native
• make -C c cuda-test CUDA=1 COLI_ANS=1 DIETGPU_ROOT=/data/test/dietgpu CUDA_ARCH=native
• RTX 5090 ANS-enabled CUDA correctness test
• complete 24,384-record / 110 GB sidecar validation and load
• fixed-order GLM-5.2 replay on 6x RTX 5090
• all GitHub CI checks pass

Limits

• Linux/CUDA and per-row int4 experts only
• DietGPU is archived and remains an explicit external build dependency
• the sidecar is tied to the exact model, placement profile, and device order
• REPIN is rejected because it invalidates that ordering
• incompressible archives are rejected
• sidecar errors fail fast instead of silently shifting the missing VRAM tier
  into RAM
• startup placement is still about 93 seconds
• no sidecar data or experiment logs are included

Refs #594.

[ZacharyZcR]

Follow-up decode experiments on the 6x RTX 5090 host:

• Nsight kernel share: generic quant_matmul 38.6%, attention 20.7%, ANS
  decode + table 17.5%, grouped expert matmul 19.0%.
• A 128-tensor/device decompressed LRU consumed 4.83 GB but hit only 7.7%;
  decode regressed from the 6.95-7.30 range to 6.47 tok/s. The prototype was
  discarded.
• The hybrid attention path (COLI_CUDA_ATTN_PREFIX=1,
  COLI_CUDA_ATTN=0, COLI_CUDA_PIPE=0) reduced 128-token attention from
  5.97s to 5.47s and measured 7.48 tok/s versus the 7.31 tok/s baseline.
• A new 0.5 GB-reserve sidecar held 11,223 experts, but runtime dense uploads
  OOMed and fell back to CPU (6.39 tok/s). Not safe.
• Reusing that sidecar at a 1.0 GB reserve held 11,025 experts without OOM,
  but still measured only 7.00 tok/s over 32 tokens. Moving more work to the
  scattered GPU critical path is not automatically faster than CPU/GPU
  overlap.

Current best stable balance remains the 2 GB reserve, 10,628-expert compressed
tier. More capacity and a decoded-weight cache both lost performance; the next
useful work is reducing dense/attention projection cost or fusing ANS decode
with expert compute, not simply consuming more VRAM.

[ZacharyZcR]

Additional boundary tests:

• The best stable runtime combination so far is the 2 GB reserve compressed
  tier plus hybrid attention:
  COLI_CUDA_ATTN_PREFIX=1 COLI_CUDA_ATTN=0 COLI_CUDA_PIPE=0.
  It measured 7.48 tok/s over 128 tokens, versus 7.31 tok/s for the full CUDA
  attention/pipe2 baseline. Attention fell from 5.97s to 5.47s.
• A paired q_a + kv_a projection kernel preserved the original accumulation
  order but measured 7.41 tok/s and did not reduce attention time (5.49s).
  The prototype was discarded.
• The expanded sidecar is useful for identifying the capacity boundary:
  0.5 GB reserve held 11,223 experts but caused dense-upload OOM fallback;
  1.0 GB held 11,025 without OOM but measured only 7.00 tok/s.

This confirms the 2 GB reserve is a performance optimum rather than merely a
conservative capacity setting. On this star topology, extra GPU residency can
lengthen the scattered GPU critical path and lose against CPU/GPU overlap.

[ZacharyZcR]

Fused decode/matvec feasibility result (kept outside this PR):

• DietGPU archives are independently decodable in 4,096-byte blocks.
• A prototype decodes one block to shared memory, immediately computes row
  partials, and writes only small partial sums instead of a full 6 MB raw tensor.
• On a real sidecar expert at a 512-block grid:
  • gate 2048x6144: 0.077 -> 0.061 ms (1.26x)
  • up 2048x6144: 0.074 -> 0.059 ms (1.25x)
  • down 6144x2048: 0.076 -> 0.055 ms (1.38x)
• Maximum output difference from changed FP summation grouping was 1.3e-5
  (gate/up) and 5.7e-6 (down). The weights themselves remain byte-lossless.

This is fast enough to justify a separate end-to-end prototype, but the FP
grouping difference means it must pass token/output-quality A/B before it can
be considered. Development continues on a separate exp/ans-fused-mv branch
so this already-green PR remains scoped.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic
(We/our = the joint work; me/my = @monotophic's own hardware and containers).

Wearing the #594 hat: this is great to see — a working compressed expert tier
with byte-identical output, +13.9% VRAM capacity, and +8.3–15.0% decode on
real GLM-5.2 validates the entropy premise harder than our projections did,
and on a host class we can't test ourselves. Thank you for building it and
for the thorough measurements.

On our side, we're moving ahead with building the transport layer for the
broader case: an in-tree portable codec (CPU rANS core first — the GPU-less
decode path the maintainer flagged as decisive, where our 8.73 GB/s Zen 5
number came from — with Metal decode following), and a random-access record
format meant to serve every boundary: disk-bound hosts streaming experts on
demand, and capacity tiers like yours consuming the same records and codec
instead of carrying a separate dependency. When it's done it should slot
into your use case as well as ours.

If you're open to it, we'd like to sync on the record format before we freeze
it so your tier and the transport layer converge on one artifact. And the
standing offer holds: as our GB10 boxes (aarch64/Blackwell) come online we
can run your A/B there for a second-architecture datapoint.

(Figures are calibrations at their stated hosts/dates — yours 6×RTX 5090
2026-07-27, ours M5 Max / Zen 5 2026-07-24/25 on #594.)

[ZacharyZcR]

2026-07-28 current-dev revalidation

Rebased the PR's five implementation commits onto current upstream/dev
(eb7a881) and repeated the CUDA checks and end-to-end measurements on the
6×RTX 5090 / dual Xeon Silver 4510 host.

Validation

• full local C test suite: pass
• make colibri CUDA=1 COLI_ANS=1 ...: pass
• ANS-enabled CUDA backend correctness test on RTX 5090: pass
• 24,384-record / 110 GB sidecar: complete validation and load
• direct-read profile: 19.23 s archive read; 93 s total expert placement
• every performance run below: 19,456 experts resident, 100% hit, 0.000 GB
  fetched during decode, 33,600 fixed routed selections

Fixed 56-token replay

The sidecar remains tied to frozen-general.stats, 2,500 raw experts and 8,128
compressed experts. Five fresh compressed processes measured:

8.01, 8.87, 8.54, 8.82, 8.25 tok/s
median: 8.54 tok/s

Two raw controls were used:

raw control	GPU experts	runs	median	compressed change
same 170.2 GB placement budget	8,997	8.10, 8.37, 8.23	8.23	+3.8%
full current 176.6 GB capacity	9,335	8.17, 8.03, 8.23	8.17	+4.5%

The compressed tier retained 10,628 GPU experts and reduced CPU-routed rows
from 11,210 (full raw capacity) to 9,492. The current gain is therefore real
but materially smaller than the earlier +8.3% to +15.0% runs.

An independent seven-token fixed replay was directionally positive
(7.56 -> 9.26 tok/s; CPU rows 1,153 -> 864), but is too short to use as a
precise throughput claim.

Output semantics correction

The ANS archives still decode weight bytes exactly. However, a current greedy
A/B with the same Chinese prompt, TEMP=0, and 15-token cap produced different
text. The compressed tier moves more experts from CPU execution to CUDA; that
changes floating-point accumulation order, and a near tie can change subsequent
routing and generation even though the weights are byte-exact.

The older byte-identical SHA-256 result remains valid for that measured prompt,
but it is not a general output-identity guarantee. docs/cuda.md now states the
distinction explicitly.

Current verdict

The mechanism survives the latest baseline at roughly +4.5% median decode
and passes correctness/build checks. It remains experimental rather than a
default-path candidate because:

• DietGPU is archived and external;
• the sidecar is model/profile/device-order specific;
• startup placement is still about 93 seconds;
• only Linux/CUDA per-row int4 is supported;
• REPIN is incompatible;
• generated-token identity versus a different raw placement is not guaranteed.

The next merge decision is therefore about accepting a scoped experimental
backend with these limits, not about claiming the original 8–15% as the current
general speedup.

[ZacharyZcR]

@monotophic Thank you for the offer to converge the transport artifact. The
rebased #651 is now Ready, but I agree its DietGPU sidecar should be treated as
an experimental capacity-tier implementation, not as the record format the
project freezes around.

The constraints the 6x5090 implementation has validated and would need from a
portable format are:

• independently decodable tensor/expert records; large aggregate archives hit
  a CUDA boundary that smaller records did not;
• exact model and traversal identity, including device/placement order;
• original size, encoded size, codec bound, shape, and truncation validation
  per record;
• alignment that permits direct reads without a second staging copy;
• fail-fast behavior rather than silently replacing a missing GPU tier with
  RAM;
• a CPU decoder for the streamed transport case and an optional pointer-batch
  GPU decoder for the compressed-VRAM case.

The latest fixed replay puts the honest CUDA capacity-tier gain at +4.5%
median, with +13.9% expert capacity. That is enough to validate the mechanism,
but not enough to justify making an archived external codec a permanent
dependency.

Please tag me when the in-tree record schema is concrete. I can validate
packing, complete-record verification, direct loading, and a 6x5090 A/B
against the existing 110 GB sidecar before the format is frozen.

[monotophic]

phase-1 of the transport layer is now a PR: #671

Still open to a record-format synch as a v2 pass if you see a benefit there. Moving forward with phase 2 development of the entropy transport layer set.

[monotophic]

@ZacharyZcR requirements answered point-by-point on the PR: #671. Short version: mostly met as shipped, container identity routes to #529, and the alignment question is yours to decide with the 6×5090 A/B — details there.