entropy: in-tree rANS codec, int4-rans256-g0 container format, offline tools - #671
Merged
Merged
Conversation
Dependency-free C99 single header (same shape as st.h/quant.h: all-static, tests include it directly), three layers: - scalar stream codec: ryg_rans-style 32-bit-state byte-renormalized encode/decode over the 16-nibble alphabet, ported unchanged from the measured prototype round, plus a checked decode that enforces the invariants every genuine encoder output satisfies (initial state in [L,256L), exact byte consumption, final state == L) so validators can refuse corruption without ground-truth bytes; - chunk record: n_symbols/packed_bytes/stream_offsets[257] header, derived round16 padding, 256 round-robin-interleaved streams per tensor. TRUST-VERIFY-REFUSE reader: every malformation class gets its own named error (truncation, count mismatch, offset malformations, short streams, nonzero padding, trailing bytes), no partial acceptance; - batched decode arms: scalar (branchy, mirrors the reference), scalar_bf (closed-form renorm -- the SIMD algebra kept testable on any host), NEON (16 streams/group, early per-lane loads, vqtbl freq/start lookups, fused vuzp repack; ~1.15x scalar_bf single-thread on an M-series host, and the G=8 variant that measured 0.88x is why the width is 16), AVX-512 F+BW (16 lanes, one vpermd per table lookup, vpmovqb packed store; ported from the prototype kernel measured at 8.7 GB/s batched on Zen 5). Vector arms are byte-identical to scalar by contract -- this is a codec, there is no tolerance -- and each carries the quant.h-style envelope: compile-time ISA gate, first-use round-trip selftest that disables a failing arm loudly, env kill-switch (RANS_NEON=0 / RANS_AVX512=0), and RANS_PATH=<arm> forcing that refuses rather than silently downgrades. The selftest table mixes freq=1 and freq=3 rare symbols deliberately: power-of-two frequencies keep floor(log2 x) on a rigid lattice that never visits some multi-byte renorm bands (measured directly -- a sabotaged renorm threshold passed every comfortable-table test), so the off-lattice f=3 group is what makes the selftest actually bite.
repack_rans.py mints the entropy tier from per-row int4 expert shards: corpus-wide header index (a tensor and its .qs sidecar can live in different shards), ONE global table from the pooled nibble histogram stamped byte-identical into every output shard (decode stays shard-local), every record round-trip-verified byte-exact BEFORE its shard is written, and deterministic output (sorted orders, sort_keys stamps, no timestamps -- two runs diff clean). Writer-side geometry guard mirrors the fp8 tool's discipline: a tensor at the per-group-64 byte signature (weight/scale ratio exactly 32) or without a confirmable per-row .qs is refused by name, never silently coded under this format's identity. rans_verify.py is the TRUST-VERIFY-REFUSE validator: stamp layer (the colibri.fmt stamp is MANDATORY here -- entropy-coded sizes are data-dependent, no byte-arithmetic inference exists, so the stamp is the only dispatch signal, not a cross-check), table layer (JSON shape, freq sum, prefix start, slot banding, crc32), record framing, per-stream decode invariants, and a deterministic re-encode pin. One named refusal class per corruption; exit 0 only on full trust. rans_format.py is the shared reference module: table construction (largest-remainder quantization, unchanged from the measured prototype), record build/parse, minimal deterministic safetensors i/o, and a pure-Python codec fallback. builds tools/librans_c.* (a thin exported-symbol bridge over rans.h, same optional-accelerator shape as tools/libiq3): with it the tools use the C codec, without it they emit identical bytes far more slowly -- the e2e test pins the two implementations byte-equal.
test_rans.c (wired into TEST_BINS): stream round-trip property suite over random and adversarial inputs (all-zero, all-max, single-symbol-freq=M, 15-symbol boundary, empty/odd/edge lengths, low-resolution and rare-symbol tables), checked-decode refusal semantics including an enumerated single-bit-corruption sweep (a corruption that silently PASSes the checked decode is a verifier hole, so it is asserted impossible, not sampled), record framing round trip at the format's real N=256 plus small-N shapes, the full named-refusal battery, and the batched-arm identity sweep: every compiled arm (scalar, scalar_bf, NEON here; AVX-512 on x86 builds) must be BYTE-identical to the packed reference on the whole suite. The rare-symbol case exists because a deliberately sabotaged renorm threshold passed every other case: power-of-two-frequency tables keep the decoder's state off the multi-byte-refill bands entirely (measured), so the f=3 off-lattice group is what makes this suite able to bite. Envelope semantics are asserted too: kill-switches drop the vector arms from selection, and a forced unavailable arm is an error, never a silent downgrade. test_rans_repack.py (unittest, skips without numpy): synthetic per-row int4 fixtures with a cross-shard .qs sidecar; end-to-end byte-exact round trip through real shard files; .qs raw pass-through; MANDATORY-stamp and per-shard-table metadata shape; writer determinism (two runs diff clean, AND a forced pure-Python run emits bytes identical to the C-codec run); the verifier's corruption battery (payload flip, count mismatch, offset break, truncation, dirty padding, dropped stamp, dropped table, wrong table crc, freq-sum break, dangling stamp -- each refused by name); and the writer-side geometry refusals (g64 signature, missing scales, empty selection).
…t claimed) The complete wire contract a third-party consumer needs: safetensors container shape (original tensor names, U8 record blobs, raw .qs sidecars), the chunk record byte layout with derived round16 padding, the 256-stream round-robin interleave and why it makes wide decode output-coalesced, the per-stream codec including the verifiable encoder-output invariants, and the per-shard table-in-metadata blob with its crc field and generation id. The load-bearing section is the MANDATORY stamp: unlike every existing format, entropy-coded byte counts are data-dependent, so no byte-arithmetic inference can identify this format -- the colibri.fmt stamp is the only dispatch signal, and a future engine read path must gate on it BEFORE the existing size-formula inference, which stays untouched. Scope is per-row int4 v1 (the per-group-64 class is excluded with its rationale), and the registry row proposes the NAME only: the numeric fmt ordinal is deliberately left for the maintainer to assign.
…er bound, never-crash validator Review findings (one blocker, four MAJOR): the codec core survived a 4,907-mutant ASan/UBSan fuzz with zero memory errors and zero arm divergence, but the untrusted-input edges around it did not hold. All fixes, each with its regression pinned: - parse hardening: the packed_bytes check is now non-wrapping (n/2 + (n&1) — the additive ceil form accepted n_symbols=UINT64_MAX with packed_bytes=0), and an amplification bound refuses any header claiming more symbols than payload_len*8*M_max could encode under ANY admissible table (E_OVERSIZE) — the decompression-bomb surface E2's loader would have inherited, refused before anything sizes buffers from n_symbols. C and Python refuse identical bytes with identical classes, and that parity is now a TESTED property (same-bytes battery through both implementations), not a documented intention. - encoder bound: the old per-stream scratch rule (n + n/2 + 64) was arithmetically false for freq=1 symbols at scale_bits >= 13 (a symbol costs up to scale_bits bits), so a legitimate rare-symbol-heavy tensor refused as out-of-memory. Correct bound (ceil(n*scale_bits/8)+4+slack), and bound exhaustion is now E_SCRATCH — a logic failure, never conflated with the allocator. Bulk-freq=1 input round-trips at scale_bits 13/14/15. - named encode refusals: input containing a value > 15 or a symbol with table frequency zero refuses as E_SYMBOL_UNCODABLE via a pre-scan, instead of burning the scratch buffer into a misdiagnosed E_NOMEM; the stream primitive masks to the nibble alphabet so no input bytes can index past freq[16] (ASan-verified). - rans_verify.py's "never crashes" is now enforced, not promised: the shard header shape is validated by name (object-ness, entry shape, data_offsets bounds against the file, str->str metadata), record-size allocations refuse as E_NOMEM instead of raising MemoryError, and per-tensor/per-shard catch-alls convert anything unexpected into a named E_INTERNAL refusal. The review round's six traceback classes all print REFUSE lines now. - test vacuum: make test-python builds the ctypes bridge first, the e2e C-vs-Python byte-identity pin skips LOUDLY if the bridge is somehow absent, and a bridge-presence assertion pins that the default run actually used the C codec. - geometry guard exactness: 2-D weights get positive per-row verification from the shape fields; flattened tensors at ratio 64 — exactly the per-group-128 signature, byte-indistinguishable from per-row I=128 — refuse as E_GEOMETRY_AMBIGUOUS instead of being silently stamped with geometry the ratio cannot prove. - n_streams is validated in every public entry (E_NSTREAMS; the ctypes bridge pins the format's 256) — no division by zero, and the generic decode path can no longer return RANS_OK with untouched output. Record buffers must be 4-byte aligned (E_UNALIGNED) and the offsets table is accessed via memcpy on the parse/write side; the little-endian precondition is a compile-time #error and documented. - quantize_freq: stable argsorts (exact fractional ties now break by symbol index on every host — contract 4's cross-host clause) and a provable-termination precondition (m >= present symbols, else ValueError). - multi-shard repack is crash-evident: a non-empty output directory refuses without --force, and repack-manifest.json is written LAST as the completion marker — an interrupted set can no longer masquerade as done. - the selftest's renorm-band coverage is machine-checked (the fixture fails the arm if multi-byte refills stop occurring) and the suite gained the same assertion — the one load-bearing claim that was previously an author assertion. The review fuzzer is adopted as tests/fuzz_rans.c (make fuzz-rans, ASan+UBSan, seeded/bounded; not a test gate). - docs: the encoder renorm emission bound (x_max formula) makes a third-party WRITER doc-buildable; the >=4-bytes-per-stream validity rule, the amplification bound, and header-integer endianness are stated. Sabotage meta-check re-run on the final tree: a t15 off-by-one in the NEON arm still fails both the identity suite and the first-use selftest. Engine object still byte-identical to base.
This was referenced Jul 28, 2026
Contributor
Author
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic @ZacharyZcR — the in-tree record schema is concrete: this PR. Your
Your validation offer is exactly what we'd like this PR to carry into (Throughput figures are calibrations at their stated hosts/dates — |
Contributor
|
@monotophic Thanks for mapping the requirements to concrete code. I reviewed the actual v1 scope and want to separate what the 6x5090 host can validate now from what needs the consumer PR. What is testable in this PR:
What is not yet testable as a capacity-tier A/B:
So my review plan is:
That does not block a deliberately inert codec/format PR from standing on its own. It just keeps “format-ready for GPU dispatch” distinct from “GPU capacity tier validated.” I will post the packing/verification/alignment results here when that host run is complete. |
Contributor
Author
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic @ZacharyZcR — this is the review split we wanted; the g64 coverage — deliberate scope, chosen on census numbers. The The wrong-checkpoint identity gap — agreed, and here's where each layer Alignment — your experiment remains the designated decider, as Looking forward to the per-row host run — the phase-1 validation you've |
…heck Per-record validation proves a record is A well-formed artifact of this format; it cannot prove it is THE record a given mint run produced — a well-formed record swapped in from another checkpoint passes every per-record check (the JustVugg#671 review point). Close that at the build layer: - repack_rans.py records, per output shard, a `records` map — original tensor name -> sha256 of the emitted record bytes exactly as written — hashed inline while the bytes are already in hand (no second read). The record wire format is untouched: the manifest is a sidecar file, still deterministic (sorted keys, no timestamps), still the completion marker. - `--manifest-only <dir>` retro-generates the manifest for already-minted directories by hashing shards in place, no re-encoding. It preserves every field of an existing manifest and only recomputes the digest maps, so retro output is byte-identical to mint output for the same shards (tested byte-for-byte). It lives in the WRITER deliberately: the manifest is the writer's completion/provenance artifact, and rans_verify.py stays side-effect-free — a tool pointed at a stranger's download must never write into it. - rans_verify.py checks each record's bytes against its digest whenever a manifest with digests sits next to the shards, BEFORE record decode (the swap case is exactly what the later checks are blind to). Named refusals: E_DIGEST_MISMATCH (bytes differ from the mint run), E_DIGEST_MISSING (digests exist but not for this shard/tensor — a record the mint never produced), E_MANIFEST_MALFORMED (a manifest that exists but cannot be parsed is evidence of tampering, refused rather than treated as absent). No manifest, or a digest-less manifest from an older mint, is note-and-proceed — never an error. Never-crash holds. - tests: digest round-trip green with "manifest digests checked" pinned; tamper bite (one flipped payload byte in the shard file, manifest untouched -> E_DIGEST_MISMATCH); retro equivalence (byte-identical manifest over a minted dir; from-scratch retro reproduces the digest map); backward compat (stripped digests -> note + green); corrupt manifest -> E_MANIFEST_MALFORMED without traceback; foreign shard -> E_DIGEST_MISSING. The per-record corruption battery now drops the manifest first — that layer must hold even when an attacker removes the whole-artifact evidence. Mint determinism (manifest included) already pinned by the existing two-run byte compare. Build integrity (these bytes came from that mint run) vs container identity (the stamp/registry lineage) is documented in docs/int4-rans256-g0.md; the engine-side load check ships with the consumer PR. Whole-file layer (micro-addendum, same round): each manifest shard entry also carries `sha256` — the complete .safetensors file bytes, hashed while streaming the write before the atomic rename (retro mode hashes files in place; retro-over-minted stays byte-identical to mint). The file hash subsumes what the record map cannot see — .qs sidecars, headers, padding — closing the scale-swap gap; the record map stays for granular diagnosis. Verify checks the file hash FIRST (cheapest whole-artifact gate, E_SHARD_DIGEST_MISMATCH), then the per-record layer, continuing past a file-level mismatch so record digests localize in-record damage. Tests: .qs-region tamper bites as E_SHARD_DIGEST_MISMATCH with NO record-level refusal (the exact gap, demonstrated), record tamper trips both layers, retro identity and mint determinism re-proven, degraded manifests (no digests at all / file-hash-only) note-and-proceed. Validator delta-3 fix (same commit): manifest entry shapes are now type-validated unconditionally BEFORE the digest-evidence scan — an entry whose only digest fields are malformed-typed (sha256 as a number, records as a list, an entry that is not an object) previously zeroed the evidence scan and was silently classed as a digest-less older mint instead of refusing. Wrong-typed digest fields are malformation regardless of whether valid evidence remains elsewhere; both validator corpses are in the shipped battery (E_MANIFEST_MALFORMED, no TRUST line, zero tracebacks). Retro --manifest-only also warns on stderr when `source` provenance is unknowable, so the loss is visible in output, not just by field absence.
Contributor
Author
|
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic Pushed What it adds:
Verification: tamper bites demonstrated for both layers (including a (Digest fields are sidecar-file additions — nothing about record layout, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic
(We/our = the joint work; me/my = @monotophic's own hardware and containers).
What this is
The first deliverable of the transport-layer build declared on #651
(follow-through on #594): a dependency-free C99 rANS codec (
c/rans.h,header-only), the
int4-rans256-g0container record format, and theoffline tools to mint, verify, and document it. No engine inference path
is touched —
colibri.ocompiles byte-identical to base; everything hereis codec + tools + tests + docs, useful standalone and consumable by any
tier (the CUDA capacity tier included — one codec, every consumer, no
external dependency).
each carrying the i4-precedent runtime envelope — first-use round-trip
selftest (with machine-checked renorm-band coverage) and env
kill-switches. Vector arms are byte-identical to scalar by contract and
by test — this is a codec: no tolerance, only identity.
round-robin-interleaved rANS streams, GPU/threadgroup-friendly), the
shared table embedded per-shard in
__metadata__, and a mandatoryformat stamp — entropy-coded sizes are data-dependent, so no
byte-arithmetic inference exists; unstamped tensors can never be
misread (refuse-rather-than-guess, same philosophy as formats: self-describing container stamp (TRUST-VERIFY-REFUSE) + FORMATS.md registry (#524) #529).
repack_rans.py(deterministic: two runs byte-identical;positive per-row geometry verification; crash-evident multi-shard
manifest) and
rans_verify.py(TRUST-VERIFY-REFUSE: a named refusalclass for every corruption class, never a crash — fuzz-hardened).
docs/int4-rans256-g0.mdis complete enough that a third-partyreader AND writer can be built from the doc alone (independently
demonstrated during review).
Measured
Capstone matrix
colibri.obyte-identical base-vs-branch; diff = new files + Makefile/test wiring onlyReview + hardening detail
Built under a two-reviewer gate (blind validator + deep audit, independent
harnesses). The hardening round closed everything found: non-wrapping
header validation + amplification bounds (
E_OVERSIZE), corrected encoderscratch bound (+
E_SCRATCH),E_SYMBOL_UNCODABLEpre-scan, never-crashverifier (hostile-header suite: zero tracebacks), positive 2-D geometry
verification (g64/g128 refused by name — this PR scopes v1 to per-row int4
deliberately),
E_NSTREAMS/E_UNALIGNEDguards, big-endian#error,crash-evident repack manifest. The fuzzer ships as
make fuzz-rans(seeded, sanitizer-built, non-gating). Suite:
make check+227-test Pythonbattery; gcc-13
-Wall -Wextra -Wpedantic -Wshadowclean; the AVX-512arm's first silicon run passed the full suite with the envelope exercised
both ways.
Scope notes: v1 targets per-row int4 expert tensors (the census's 0.763
container class); g64-class and scale-array coding are documented future
rounds with their own (weaker) payoff cases. Engine integration (loader +
CPU streaming decode, then Metal batched decode) follows as separate PRs —
this one is deliberately inert: format + codec + tools, reviewable and
usable on their own.
Durable vs current-state: format, codec, envelope, and refusal semantics
are durable; throughput figures are calibrations at (Zen 5 Strix Halo /
M5 Max, 2026-07-28, dev fa599e0 base) and the ratio at (GLM-5.2 int4
per-row container, 2-shard sample vs 372 GB census).