Group-scaled int4 (fmt=4): one scale per 128 elements — fixes incoherent output

[woolcoxm]

Summary

Adds a new int4 quantization format (fmt=4) with one scale per 128 elements along the input dimension, instead of one scale per output row. This matches the FP8 source checkpoint's 128×128 block-scale granularity and fixes the root cause of incoherent int4 output.

Closes #225.

Problem

The int4 model produces fluent-but-incoherent output — grammatical English that is completely off-topic (SEO spam, homework-help boilerplate). The root cause is per-row quantization scales: one F32 scale per output row (e.g. 2048 scales for a 2048×6144 matrix), which is 48× coarser than the FP8 source's 128×128 block scales. This destroys fine-grained weight information that handles reasoning and instruction-following while preserving the large-magnitude weights that handle grammar and vocabulary fluency.

GLM-5.2 was QAT-trained for int4 (arXiv 2602.15763, §2.4.3), but only with fine-grained quantization matching what the model saw during training. Per-row scaling is 48× too coarse.

Changes

c/glm.c (+122 lines):

• QT struct: added int gs field (group size, 0=per-row for backward compat, 128=grouped)
• matmul_i4_grouped kernel: AVX2 nibble unpacking, accumulator resets at each 128-element group boundary — verified to 2.98e-08 vs f32 reference
• Format detection (qt_from_disk): auto-detects fmt=4 from .qs scale array size. Old per-row models (fmt=2) work unchanged.
• Expert loading: both mmap and slab+pread paths handle fmt=4
• qt_bytes: fmt=4 case added
• Dispatch (matmul_qt_ex): routes to matmul_i4_grouped when fmt==4, always exact (no IDOT)

c/tools/convert_fp8_to_int4.py (+52 lines):

• New quant_int4_grouped() function + --group-size arg (0=per-row default, 128=grouped)
• Same packed-nibble format as existing int4 — only the scale array is larger

Validation

Metric	Per-row (current)	Grouped (new)
Mean relative error	0.155	0.128
Max abs error	0.0127	0.0127
Improvement	—	1.14-1.22× lower
Kernel vs f32	—	2.98e-08 (PASS)
Cost (model size)	~370 GB	~390 GB (+5%)

Backward compatibility

• Old per-row models (fmt=2) work unchanged — format auto-detected from scale array size
• All existing features (PILOT, EXPERT_BUDGET, CUDA, MTP, PIPE) work identically
• --group-size 0 produces per-row output (converter backward compat)

Test plan

• Converter round-trip: grouped vs per-row dequant error measured
• Engine kernel: AVX2 matmul_i4_grouped matches f32 reference to 2.98e-08
• Engine load: both fmt=2 and fmt=4 models load clean, format auto-detected
• Production quality A/B: grouped vs per-row on the real 372 GB model (requires FP8 download to complete, FP8 download tool: dual-source (ModelScope + HF), parallel, resume — for the 379 GB GLM-5.2-FP8 checkpoint #239)

[JustVugg]

Merged to dev. Reviewed and verified before landing:

Correctness — matmul_i4_grouped is sound: accumulator zeroed per group, nibble unpacking identical to matmul_i4, groups are byte-aligned (gs=128 is even so no cross-byte split at boundaries), and the AVX2 loads stay in-bounds. Your isolated 2.98e-08-vs-f32 number holds up.

Backward compatibility — proven, not assumed. I built the tiny oracle and ran the same snapshot through dev HEAD (without this PR) and this branch: byte-identical results (14/20 oracle match, same 84/0 expert-hit trace on both). The fmt=2 path is untouched — the detection only promotes to fmt=4 when ns > O*4 (a per-row model always emits exactly O scales, so it stays fmt=2), and make check's int4 kernel tests (test_i4_acc512, test_idot) pass. So: no existing model gets worse from this — the grouped format is only active if you rebuild the container with --group-size 128.

Two follow-ups (not blockers):

1. fmt number collision with int3-g64 (fmt=5): per-group-scale 3-bit weights, the size/quality point your own ablation (#132) says the next container should sit at #168. That PR also claims fmt=4, for int3-g64. The detection doesn't actually clash (int3 has a different weight-byte count, so it falls through to fmt=3, not fmt=4), but the integer value 4 in t->fmt can't mean two things. Since this landed first, Group-scaled int4 (fmt=4): one scale per 128 elements — fixes incoherent output #242 keeps fmt=4 and I've asked int3-g64 (fmt=5): per-group-scale 3-bit weights, the size/quality point your own ablation (#132) says the next container should sit at #168 to take fmt=5.
2. Real-model quality A/B is still open (your task 4, blocked on the FP8 download FP8 download tool: dual-source (ModelScope + HF), parallel, resume — for the 379 GB GLM-5.2-FP8 checkpoint #239). The dequant-error drop (0.155→0.128) is encouraging but it's an isolated metric — the end-to-end 'fixes incoherent output' claim in the title isn't measured yet on the 372 GB model. Worth keeping Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 open until that A/B lands. Great work regardless.

[woolcoxm]

Thanks for the thorough review — and for independently verifying the backward compat (the byte-identical oracle match is exactly the right test, glad it held).

On the two follow-ups:

1. fmt=4 collision — agreed, that's the right call. We're fmt=4, int3-g64 (fmt=5): per-group-scale 3-bit weights, the size/quality point your own ablation (#132) says the next container should sit at #168 takes fmt=5. No action needed on our side.

2. Real-model quality A/B — agreed, Root cause of incoherent output: per-row int4 scales — fix with group-scaled quantization (fmt=4) #225 should stay open until the end-to-end A/B lands. The dequant-error metric (0.155→0.128) is necessary but not sufficient. The FP8 download is at 82/141 shards and in progress; once it completes I'll produce both containers (per-row int4 + grouped int4) from the same FP8 source and run the recursion prompt + a small reasoning set side-by-side. That's the test that actually validates the "fixes incoherent output" claim.