tools: rate-scale the E8 lattice ball by bit-width (#81) — int3-e8 is now real int3#347
Merged
Merged
Conversation
… real int3 codebook, not a fixed 2-bit ball
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #255, closing the loop @locallypwned asked for. The E8 quantizer's ball radius was hard-coded at
r2=10— the ~2^16 E8P codebook, i.e. 2 bits over 8 dims regardless of theintNprefix. Soint3-e8silently produced the int2 codebook (my earlier -19.4pp cell was really int2).Fix: the E8 lattice's in-ball point count grows ~
r2^4, so +1 bit (×256 codebook) needsr2 ×4. Anchor atr2=10for 2 bits and scale:r2 = 10 · 4^(bits-2)(int2→10, int3→40). One helper, threaded through_quant_e8; prints the radius per bit-width for the record.Measured (OLMoE, same protocol as the #81 matrix, fp16 anchor 57.7)
The headline: with the radius fixed, int3-e8-rot lands at -5.9pp — better than shipped per-row int4 (-9.3) and the uniform int3-g64 reference (-7.5), at ~3.25 b/w. The lattice earns its keep at 3 bits the way it couldn't at 2 (the 2-bit cell stays a cliff, confirming int2 needs more than a bigger ball). And unlike the int2 case, grouping beats per-row here (-5.9 vs -7.9) — the group scales still have adaptation to do at int3.
Practical read: a ~3-bit E8 expert container is quality-viable. That's the lever for the streaming-bound hosts (smaller bytes per expert miss) and, on the residency side, a step toward a container that fits more of the model in fast memory.