Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
f3d3398
CUDA: Fuse MMVQ for NVFP4 and BS 1
ORippler Jun 2, 2026
45d4704
Add dense MMVQ fusion as well
ORippler May 29, 2026
2173d97
Add tests for the added fusion ops
ORippler May 29, 2026
4ed8ef2
Cleanup test-backend-ops
ORippler Jun 2, 2026
d6190b9
Cleanup ggml-cuda/mmvq
ORippler Jun 2, 2026
b07d0f0
Merge old mul_mat patterns into the lane-based approach
ORippler Jun 2, 2026
cf97e37
Enable fusion for MoE in shared MMVQ
ORippler Jun 3, 2026
c1b9381
Restrict scale_view_nodes, enroll MM + ADD into lane-matcher
ORippler Jun 3, 2026
2971360
Refactor mmvq loads, still does not help non-nvfp4 kernels
ORippler Jun 3, 2026
28bde65
Restrict scale-fusion to NVFP4
ORippler Jun 10, 2026
9d9f1a0
Reorder scale & bias-add to adhere to #24331
ORippler Jun 10, 2026
0965978
Restrict lane scale to NVFP4
ORippler Jun 10, 2026
9088a3c
Cleanup
ORippler Jun 10, 2026
c91327f
Merge single-lane mm-fusion helpers
ORippler Jun 11, 2026
6d815ca
Refactor and clean-up host-side fusion logic
ORippler Jun 11, 2026
6b9024e
Move gate_bias and scale into the same active-thread guard
ORippler Jun 11, 2026
8dd49e0
Allow views to weights in ggml_can_fuse_subgraph
ORippler Jun 16, 2026
89fcfc2
Remove gate_first from test_mul_mat_vec_fusion
ORippler Jun 17, 2026
d8955f6
Ditch lane-parsing approach in favor of hard-coded patterns
ORippler Jun 16, 2026
0905129
Apply suggestions from code review
ORippler Jun 26, 2026
637bafd
Rename ggml_is_constant_view_src to ggml_is_constant
ORippler Jun 26, 2026
b46e632
Finish renaming of 0905129e9d12e2bc6f16d6d3cc4e6b40606fc893
ORippler Jun 26, 2026
30422a8
Readd descriptive prints for fusion debugging
ORippler Jun 26, 2026
764f9da
Add weight-buffer pre-allocation to `test_case`
ORippler Jun 26, 2026
d5a8272
Update ggml/src/ggml.c
ORippler Jul 2, 2026
e9f8b33
Add 2nd context for weights as suggested by @JohannesGaessler
ORippler Jul 3, 2026
3b991ef
Exclude fused tests from gradient mode
ORippler Jul 3, 2026
216097f
Apply suggestions from code review
ORippler Jul 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions ggml/src/ggml-cuda/common.cuh
Original file line number Diff line number Diff line change
Expand Up @@ -1505,12 +1505,16 @@ struct ggml_cuda_mm_fusion_args_host {
const ggml_tensor * x_bias = nullptr;
const ggml_tensor * gate = nullptr;
const ggml_tensor * gate_bias = nullptr;
const ggml_tensor * x_scale = nullptr;
const ggml_tensor * gate_scale = nullptr;
ggml_glu_op glu_op;
};
struct ggml_cuda_mm_fusion_args_device {
const void * x_bias = nullptr;
const void * gate = nullptr;
const void * gate_bias = nullptr;
const void * x_scale = nullptr;
const void * gate_scale = nullptr;
ggml_glu_op glu_op;
};

Expand Down
Loading
Loading