CUDA: Fuse MMVQ post-scale for NVFP4

[ORippler]

Overview

This PR fuses post-GEMV-mul of w_s for NVFP4 in the CUDA backend. This gives 1-10% E2E speed-up during decode depending on the platform and model checkpoint. Implements the logic of #24331 w.r.t bias-scale ordering.

Additional information

perf evaluation

build: 5b7d9f272 (9578)
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| CPU                         | Model                    | Test         |   t/s master |   t/s osimons/nvfp4_fuse_mmvq |   Speedup |
|:----------------------------|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B NVFP4     | tg128@d32768 |       151.79 |                        154.10 |      1.02 |
| INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B Q4_K_M    | tg128@d32768 |       187.90 |                        187.27 |      1.00 |
| INTEL(R) XEON(R) GOLD 6542Y | gpt-oss 20B MXFP4 MoE    | tg128@d32768 |       303.77 |                        306.56 |      1.01 |
| INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B NVFP4  | tg128@d32768 |       193.41 |                        207.99 |      1.08 |
| INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B Q4_K_M | tg128@d32768 |       217.60 |                        218.58 |      1.00 |


DGX Spark

build: 5b7d9f272 (9578)
+ ./scripts/compare-llama-bench.py -b master -c osimons/nvfp4_fuse_mmvq --tool llama-bench -i llama-bench.sqlite
| CPU   | Model                    | Test         |   t/s master |   t/s osimons/nvfp4_fuse_mmvq |   Speedup |
|:------|:-------------------------|:-------------|-------------:|------------------------------:|----------:|
| CPU   | gemma4 26B.A4B NVFP4     | tg128@d32768 |        34.61 |                         34.84 |      1.01 |
| CPU   | gemma4 26B.A4B Q4_K_M    | tg128@d32768 |        46.95 |                         46.90 |      1.00 |
| CPU   | gpt-oss 20B MXFP4 MoE    | tg128@d32768 |        64.84 |                         64.62 |      1.00 |
| CPU   | qwen35moe 35B.A3B NVFP4  | tg128@d32768 |        59.63 |                         60.72 |      1.02 |
| CPU   | qwen35moe 35B.A3B Q4_K_M | tg128@d32768 |        56.53 |                         56.55 |      1.00 |

quality evaluation

computed over 5 chunks on wikitext

PR

/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf                                 fusion_enabled   5.2892      0.35389      ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf                                 fusion_disabled  5.2742      0.35215      ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf  fusion_enabled   5.4487      0.36866      ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf  fusion_disabled  5.4403      0.36782      ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf                              fusion_enabled   17342.4348  3703.13932   ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf                              fusion_disabled  18627.0624  3998.42475   ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf                                            fusion_enabled   363.8913    33.14007     ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf                                            fusion_disabled  363.8913    33.14007     ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf                          fusion_enabled   17330.3926  3716.70472   ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf                          fusion_disabled  17933.9524  3883.17066   ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log

Master

model                                                                                                       mode             ppl         uncertainty  log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf                                 fusion_enabled   5.2892      0.35389      ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_enabled.log
/mnt/share/gguf/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf                                 fusion_disabled  5.2742      0.35215      ppl-value-checks/Qwen3.6-35B-A3B-UD-Q4_K_M.fusion_disabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf  fusion_enabled   5.4487      0.36866      ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Qwen3.6-35B-A3B-2.06GB-per-token-CT/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.gguf  fusion_disabled  5.4403      0.36782      ppl-value-checks/Qwen3.6-35B-A3B-2.06GB-per-token-CT_fp8_q8.fusion_disabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf                              fusion_enabled   17342.4348  3703.13932   ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_enabled.log
/mnt/share/gguf/nvidia/Gemma-4-26B-A4B-NVFP4/Gemma-4-26B-A4B-NVFP4_fp8_q8.gguf                              fusion_disabled  18627.0624  3998.42475   ppl-value-checks/Gemma-4-26B-A4B-NVFP4_fp8_q8.fusion_disabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf                                            fusion_enabled   363.8913    33.14007     ppl-value-checks/gpt-oss-20b-mxfp4.fusion_enabled.log
/mnt/share/gguf/ggml-org/gpt-oss-20b-GGUF/gpt-oss-20b-mxfp4.gguf                                            fusion_disabled  363.8913    33.14007     ppl-value-checks/gpt-oss-20b-mxfp4.fusion_disabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf                          fusion_enabled   17330.3926  3716.70472   ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_enabled.log
/mnt/share/gguf/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf                          fusion_disabled  17933.9524  3883.17066   ppl-value-checks/gemma-4-26B-A4B-it-UD-Q4_K_XL.fusion_disabled.log

Additional comments/explanations:

• I restricted the fusion to NVFP4 as I observed some perf regressions on other quants in 28bde65. I'll be looking to refactor mul_mat_vec_q as we seemingly spend a lot of time in pro/epilogue side of things, especially for small feature dims.
• I refactored the fusion-parse logic for MUL_MAT/MUL_MAT_ID in ggml-cuda.cu to be lane-based (one or two lanes that can optionally be fused with GLU later on). This turned out to be more flexible in supporting optionality of components
• I added ggml_cuda_can_fuse_parsed_subgraph since ggml does not offer lane-based, flexible node-fusion validation. It mimicks the logic of ggml_can_fuse_subgraph while allowing view nodes to external inputs be allowed. This was required to support fusion of MUL_MAT_ID [RESHAPE -> REPEAT -> GET_ROWS -> MUL expert_scale] [ADD_ID], where the view-src of RESHAPE sits outside the fused pattern (it is a constant weight so I consider this safe to do). Once other backends look to fuse this, we may wish to generally lessen the restriction for view-srcs.
• We intend to follow-up these style of fusions to BS > 1 in follow-up PRs (both decode and prefill).

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, paired with codex on this.

[am17an]

I think we should add this extension in ggml_can_fuse_subgraph to simplify the fusion logic

[ORippler]

I think we should add this extension in ggml_can_fuse_subgraph to simplify the fusion logic

Care to elaborate on what you mean by "simplify the fusion logic"? With this PR the CUDA backend:

1. Refactors previous mm-fusion-parsing logic to be lane/layer-based instead of checking their representation in linearized ggml_ops.
2. Lessens the constraint on fusion validation to allow for view nodes to external inputs to be allowed.

I agree we should be generalizing the second point to ggml side. To me, lane-based graph-parsing fit my mental model better as it represents the ML "layers" we are mapping to ggml-ops. I personally preferred this over expanding hard-coded patterns ala

    std::initializer_list<enum ggml_op> mul_mat_bias_glu_ops    = { GGML_OP_MUL_MAT,    GGML_OP_ADD,    GGML_OP_MUL_MAT,    GGML_OP_ADD,    GGML_OP_GLU };
    std::initializer_list<enum ggml_op> mul_mat_id_bias_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_GLU };

    std::initializer_list<enum ggml_op> mul_mat_id_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_MUL_MAT_ID, GGML_OP_GLU };
    std::initializer_list<enum ggml_op> mul_mat_glu_ops    = { GGML_OP_MUL_MAT,    GGML_OP_MUL_MAT,    GGML_OP_GLU };

to the possible permutations of add & scale we can now support in the CUDA backend. Happy to hear your thoughts on this.

[am17an]

Adding 2. to the ggml side is what I meant by simplifying the fusion. One more hard-coded pattern should be fine for this fusion rather than adding a new type of fusion detection. I will look at refactoring it in case it becomes too cumbersome.

[ORippler]

Adding 2. to the ggml side is what I meant by simplifying the fusion.

I lessened the restriction to allow vies to tensors marked as GGML_BACKEND_BUFFER_USAGE_WEIGHTS in 8dd49e0. CC @ggerganov in case I am missing some additional check that is needed.

One more hard-coded pattern should be fine for this fusion rather than adding a new type of fusion detection. I will look at refactoring it in case it becomes too cumbersome.

It's actually three:

1. two-lane + optional scale + optional bias + GLU for MM
2. two-lane + optional scale + optional bias + GLU for MMID (scale is per-tensor, so patterns to match are different between MM & MMID)
3. single-lane + scale + optional bias

in addition to the existing

• single-lane + bias

I generally would recommend the version of master...89fcfc2 that still contains the refactoring as it unifies mmvq fusion paths. We currently have dispersed pattern-validation where

• two-lane + GLU paths have some validation happening in ggml_cuda_try_fuse, and some in ggml_cuda_should_fuse_mul_mat.
• for single-lane + scale + optional bias, the new code uses ggml_can_fuse_subgraph, whereas the existing single-lane + bias code dispatches into ggml_can_fuse.

This seemed convoluted to me, hence the refactoring (which comes out at 472 additions and 364 deletions as opposed to 500 additions and 78 deletions when adding 3 more patterns into ggml_cuda_try_fuse). @am17an Given you said you wanted to own an eventual refactoring, we can move forward with any version, your choice.

---

Unrelated question from my side. Why are other backends not affected by the memory-overlap issue we guard for in ggml_cuda_check_fusion_memory_ranges? I thought memory-management was handled by ggml's allocators, not a backend. Consequentially, I'd have thought that such a function should live in ggml, and not the cuda backend (it also uses none of the cuda-specific functions).

[ORippler]

For clarity, the matcher with fixed patterns does it like this

we match in the following node-pattern order:

• scale + optional bias
• bias
• neither scale nor bias

I already gave my take on this verbosity here, recommending to go with the refactored version. If you want I can try to squeeze the bias-only path in the scale + optional bias path, I initially did not to preserve legibility

[am17an]

I think it's fine for now. I don't like how intricate the fusion code w.r.t the gated activations has become. It maybe be easier to execute the whole operation as a ggml-op. Trying to make it easier to do that by introducing #24646

[JohannesGaessler]

I don't like the way ggml_backend_buffer_usage is being integrated into test-backend-ops. My opinion is that we should stick to "normal" usage of the ggml API when possible to minimize discrepancies between the tests and production use. I would suggest the following pattern:

• Instead of just one ggml_context, instead allocate two of them, one for weights and one for compute.
• Allocate both ggml_contexts with ggml_backend_alloc_ctx_tensors but set GGML_BACKEND_BUFFER_USAGE_WEIGHTS for only one of them.
• Add a method build_graph that accepts two ggml_contexts and implement it for the test case for the new NVFP4 fusion code. To avoid having to change every single test when they don't care about GGML_BACKEND_BUFFER_USAGE_WEIGHTS, add a default implementation that simply calls the pre-existing build_graph with only one context.
• Also add sentinels to the newly added weight context to enable the detection of out-of-bounds writes.

[JohannesGaessler]

The MMVQ changes are OK. The changes to the fusion logic in ggml-cuda.cu seem fragile to me; if we are fusing 13 consecutive ggml ops the likelihood of something breaking the fusion is I think non-negligible. This should be fine in terms of at least not causing incorrect outputs but the fallback would be slower. Long-term it may make sense to think about how this could be implemented in a more robust way.

[ORippler]

@JohannesGaessler I implemented the 2nd context as you suggested. Regarding backprop changes:

1. We should be allowed to fuse weight-views even to trainable parameters, as parameters are guaranteed to be constant during a forward/backward pass in SGD (they all change at once when calling optimizer.step()). Therefore, I think the check in CUDA: Fuse MMVQ post-scale for NVFP4 #24481 (comment) shouldn't be needed.
2. Since we don't seem to implement backward passes for fused ops in the cuda backend, I excluded whole_graph tests from the GRAD mode 3b991ef. This avoids triggering the GGML_ASSERT in the fusion case that asserts weight_ctx presence when lane_scale_id=1. Unrelated: various grad tests fail for me, and I even get aborts.

Happy to hear your thoughts on this. I may be missing something, as my assumptions are rooted in how PyTorch implements SGD.

The changes to the fusion logic in ggml-cuda.cu seem fragile to me; if we are fusing 13 consecutive ggml ops the likelihood of something breaking the fusion is I think non-negligible. [...] Long-term it may make sense to think about how this could be implemented in a more robust way.

Agreed, pattern-matching of raw ggml_op nodes is a fragile solution to fusion. Unfortunately, ggml doesn't offer higher-level composite-operators/building blocks like nn.Linear, which would facilitate a more robust approach to this. On first glance #24646 won't resolve this as it doesn't aim to add composite-operators/building blocks.

[JohannesGaessler]

@ORippler I'm not surprised that the backpropagation tests are broken because they are (to my knowledge) not part of our CI and the implementation is nowhere near an actually usable state. The way backpropagation works in ggml is that the forward and backward pass are part of a single compute graph with gradient updates being a ggml op. So you can in fact not assume that weights will remain constant during the graph execution if they have the corresponding flag. But in any case, as of right now training is broken anyways so it doesn't really matter.

[ORippler]

The way backpropagation works in ggml is that the forward and backward pass are part of a single compute graph with gradient updates being a ggml op. So you can in fact not assume that weights will remain constant during the graph execution if they have the corresponding flag. But in any case, as of right now training is broken anyways so it doesn't really matter.

@JohannesGaessler I see, thanks for the explanation. That would still mean we have to add backward passes for the fusion patterns though right? In this light, I'd just keep them excluded via 3b991ef for now.

[ORippler]

HIP CI failure seem unrelated, filed #25361 to track