Skip to content

perf(cuda): extend Q8_1 activation memo to the MMQ MoE path#34

Draft
dusterbloom wants to merge 1 commit into
Luce-Org:perf/spec-verify-sm86from
dusterbloom:perf/moe-q8-memo-mmq
Draft

perf(cuda): extend Q8_1 activation memo to the MMQ MoE path#34
dusterbloom wants to merge 1 commit into
Luce-Org:perf/spec-verify-sm86from
dusterbloom:perf/moe-q8-memo-mmq

Conversation

@dusterbloom

@dusterbloom dusterbloom commented Jul 5, 2026

Copy link
Copy Markdown

What

Companion to the mmvq memo (#33): extend the per-eval Q8_1 activation memo to the MMQ MUL_MAT_ID (MoE) path.

At spec-decode verify width (and prefill), Laguna's MoE experts route through MMQ (mmq.cu), which re-quantizes src1 per mul_mat_id call. gate_e and up_e share the same gathered activation + ids, so the same src1→q8_1 is quantized twice per MoE layer × 39 layers every step (reproduced: consecutive quantize calls with identical src1/ids/type/dims).

How

Reuse the existing per-eval ctx.luce_q8_memo (freed LIFO at graph_compute start) for the MMQ ids path — on a hit (up_e after gate_e) reuse gate's q8 buffer and skip the requant. Minimal additions:

  • luce_q8_memo_entry gains ids_node/ids_data and an mmq flag — an MMQ block_q8_1_mmq buffer must never cross-reuse an mmvq q8_1 one.
  • Keyed on the tensor NODES (src1/ids), since data pointers alias across layers via the graph allocator.
  • The q8 buffer is allocated before the transient id buffers, so those sit above it on the strict-LIFO CUDA pool and free first (a held memo buffer must not sit above a buffer that frees before it).

Status

  • Env-gated LUCE_MMQ_Q8_MEMO=1 (off by default). Zero effect when off.
  • Builds clean; llama-bench pp512 no crash, ~parity (prefill; the real payoff is the verify-width>1 decode path).
  • Output verified byte-identical (greedy, off vs on) on the dflash server for this exact logic. Re-confirm on the server after landing, and measure the spec-verify decode delta with the DSpark drafter enabled.

Based on 02cc5286c (the commit the server currently pins via perf/spec-verify-sm86).


Summary by cubic

Extend per-eval Q8_1 activation memoization to the MMQ MoE MUL_MAT_ID path to avoid re-quantizing the same gathered activations for gate_e and up_e. Cuts redundant work during spec-decode and prefill with no output changes.

  • New Features
    • Reuse ctx.luce_q8_memo in MMQ: keyed by (src1 node, ids node, src0 type, dims); on hit, reuse the q8 buffer and skip requant.
    • Add ids_node, ids_data, and an mmq flag to luce_q8_memo_entry to ensure MMQ (block_q8_1_mmq) never cross-reuses mmvq q8_1.
    • Allocate the q8 buffer before transient id buffers to keep the strict LIFO CUDA pool valid; memo buffers free at graph start.
    • Env-gated via LUCE_MMQ_Q8_MEMO=1 (off by default); no effect when off; outputs validated byte-identical.

Written for commit 6c2e686. Summary will update on new commits.

Review in cubic

Companion to the mmvq memo (Luce-Org#33): at spec-decode verify width (and prefill),
Laguna's MoE experts route through MMQ (mmq.cu), which re-quantizes src1 per
mul_mat_id call. gate_e and up_e share the same gathered activation + ids, so
the same src1->q8_1 is quantized twice per MoE layer x39 layers every step.

Reuse the existing per-eval ctx.luce_q8_memo (freed LIFO at graph_compute
start) for the MMQ ids path: on a hit (up_e after gate_e) reuse gate's q8
buffer and skip the requant. Extends luce_q8_memo_entry with ids_node/ids_data
and an mmq flag (an MMQ block_q8_1_mmq buffer must never cross-reuse an mmvq
q8_1 one). Keyed on the tensor NODES (data ptrs alias across layers via the
graph allocator). The q8 buffer is allocated before the transient id buffers
so the strict-LIFO CUDA pool stays valid when a memo buffer is held across the
eval.

Env-gated LUCE_MMQ_Q8_MEMO=1 (off by default). Builds clean; llama-bench pp512
no crash / ~parity. Output was verified byte-identical (greedy, off vs on) on
the dflash server for the same logic; re-confirm on the server after landing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants