perf(cuda): extend Q8_1 activation memo to the MMQ MoE path#34
Draft
dusterbloom wants to merge 1 commit into
Draft
perf(cuda): extend Q8_1 activation memo to the MMQ MoE path#34dusterbloom wants to merge 1 commit into
dusterbloom wants to merge 1 commit into
Conversation
Companion to the mmvq memo (Luce-Org#33): at spec-decode verify width (and prefill), Laguna's MoE experts route through MMQ (mmq.cu), which re-quantizes src1 per mul_mat_id call. gate_e and up_e share the same gathered activation + ids, so the same src1->q8_1 is quantized twice per MoE layer x39 layers every step. Reuse the existing per-eval ctx.luce_q8_memo (freed LIFO at graph_compute start) for the MMQ ids path: on a hit (up_e after gate_e) reuse gate's q8 buffer and skip the requant. Extends luce_q8_memo_entry with ids_node/ids_data and an mmq flag (an MMQ block_q8_1_mmq buffer must never cross-reuse an mmvq q8_1 one). Keyed on the tensor NODES (data ptrs alias across layers via the graph allocator). The q8 buffer is allocated before the transient id buffers so the strict-LIFO CUDA pool stays valid when a memo buffer is held across the eval. Env-gated LUCE_MMQ_Q8_MEMO=1 (off by default). Builds clean; llama-bench pp512 no crash / ~parity. Output was verified byte-identical (greedy, off vs on) on the dflash server for the same logic; re-confirm on the server after landing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Companion to the mmvq memo (#33): extend the per-eval Q8_1 activation memo to the MMQ
MUL_MAT_ID(MoE) path.At spec-decode verify width (and prefill), Laguna's MoE experts route through MMQ (
mmq.cu), which re-quantizessrc1permul_mat_idcall.gate_eandup_eshare the same gathered activation + ids, so the samesrc1→q8_1is quantized twice per MoE layer × 39 layers every step (reproduced: consecutive quantize calls with identicalsrc1/ids/type/dims).How
Reuse the existing per-eval
ctx.luce_q8_memo(freed LIFO atgraph_computestart) for the MMQ ids path — on a hit (up_eaftergate_e) reuse gate's q8 buffer and skip the requant. Minimal additions:luce_q8_memo_entrygainsids_node/ids_dataand anmmqflag — an MMQblock_q8_1_mmqbuffer must never cross-reuse an mmvqq8_1one.src1/ids), since data pointers alias across layers via the graph allocator.Status
LUCE_MMQ_Q8_MEMO=1(off by default). Zero effect when off.llama-bench pp512no crash, ~parity (prefill; the real payoff is the verify-width>1 decode path).Based on
02cc5286c(the commit the server currently pins viaperf/spec-verify-sm86).Summary by cubic
Extend per-eval Q8_1 activation memoization to the MMQ MoE
MUL_MAT_IDpath to avoid re-quantizing the same gathered activations forgate_eandup_e. Cuts redundant work during spec-decode and prefill with no output changes.ctx.luce_q8_memoin MMQ: keyed by(src1 node, ids node, src0 type, dims); on hit, reuse the q8 buffer and skip requant.ids_node,ids_data, and anmmqflag toluce_q8_memo_entryto ensure MMQ (block_q8_1_mmq) never cross-reuses mmvqq8_1.LUCE_MMQ_Q8_MEMO=1(off by default); no effect when off; outputs validated byte-identical.Written for commit 6c2e686. Summary will update on new commits.