Skip to content

convert_olmoe.py: wire --ebits through to quantization (fixes #323)#328

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
mohamedmastouri2000-boop:fix/olmoe-ebits
Jul 17, 2026
Merged

convert_olmoe.py: wire --ebits through to quantization (fixes #323)#328
JustVugg merged 1 commit into
JustVugg:devfrom
mohamedmastouri2000-boop:fix/olmoe-ebits

Conversation

@mohamedmastouri2000-boop

Copy link
Copy Markdown

convert_olmoe.py: wire --ebits through to quantization (fixes #323)

--ebits was parsed (default 4) but never used: quantize_row() hardcoded qmax=127, so every conversion silently emitted int8 while the script's docstring, name, and default all promised int4 β€” blocking the int4-on-OLMoE measurement #323 wanted.

Fix

quantize_row(w, bits) now mirrors the engine's quantize_rows() (olmoe.c) exactly:

  • qmax = 2**(bits-1) - 1 (8 β†’ 127, 4 β†’ 7, 2 β†’ 1)
  • scale = amax(|w|, row) / qmax
  • q = clamp(round(w/scale), -qmax-1, qmax), stored one value per int8 byte (the engine dequantizes as q*scale and never assumes the full int8 range, so no container format change and the I8+.qs fast path in load_expert_w() works unchanged)

Plus: --ebits validated to 2..8 (same range the engine enforces), and the completion message now states the actual bit width.

Behavior note for review

The default invocation now genuinely produces int4, as the docs always claimed β€” anyone who relied on the de-facto int8 output gets a real change. If you'd rather preserve de-facto behavior, flipping the default to --ebits 8 is a one-line tweak; I kept 4 because it's what the script has documented since day one and it's the measurement #323 is after.

Verification

  • Property test vs a pure-Python replica of the C reference: bit-exact q and scales at bits ∈ {8,4,2} on random bf16 tensors; q ranges [-127,127] / [-7,7] / [-1,1]; reconstruction error grows monotonically as bits shrink (0.0003 β†’ 0.005 β†’ 0.033 mean abs).
  • Explicit --ebits 8 reproduces the previous (hardcoded int8) output exactly β€” byte-identical q and scales.
  • Happy to run the end-to-end OLMoE conversion + engine self-test on the Benchmark + Windows GPU-build fixes: GLM-5.2 on Core Ultra 9 285K + RTX 5080 (sm_120)Β #306 box and post the numbers here if useful.

Ref #323.

@JustVugg
JustVugg merged commit 1af9435 into JustVugg:dev Jul 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants