cuda: fuse independent quantized SET_ROWS pairs into one launch

[davide221]

Per-layer K/V cache appends are two independent quantized SET_ROWS nodes (separated only by the elidable view feeding the second). This fuses the pair into one kernel launch; each grid half runs the exact k_set_rows_quant element math, so outputs are bit-identical to separate launches. Supported for matching q4_0/q4_1/q5_0/q5_1/q8_0/iq4_nl dst types with f32 src and i32/i64 ids; GGML_CUDA_DISABLE_FUSION covers it.

Validated on RTX 3090 (laguna-xs2 + v24 q4mix, w6): output hashes byte-identical across code/math/long-context prompts, lucebox ctest 12/12, harness suites unchanged. Perf is a small launch/tail saving (~0-2 tok/s at w6 since CUDA-graph replay already amortizes most launch cost); the main value is 80→40 kernel launches per step and bit-exactness.

Consumed by Luce-Org/lucebox#482 (pin e0e6b2c).

🤖 Generated with Claude Code

---

Summary by cubic

Fuses pairs of independent quantized SET_ROWS ops (per-layer K/V cache appends) into a single CUDA launch. Outputs are bit-identical; per-step launches drop ~80→40 with a small tail-latency win.

• New Features - New features added
  • Adds a fused dual SET_ROWS CUDA kernel for matching quantized dst types (q4_0, q4_1, q5_0, q5_1, q8_0, iq4_nl) with f32 src and i32/i64 ids; otherwise falls back.
  • Graph pass auto-detects and fuses adjacent independent pairs, skipping GGML_OP_VIEW/GGML_OP_RESHAPE/GGML_OP_PERMUTE.
  • Guarded by GGML_CUDA_DISABLE_FUSION. Outputs are bit-identical to separate launches.

^{Written for commit e0e6b2c. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="ggml/src/ggml-cuda/set-rows.cu">

<violation number="1" location="ggml/src/ggml-cuda/set-rows.cu:538">
P2: The dual SET_ROWS fusion lacks a check for overlapping destination memory ranges. The `ggml_cuda_set_rows_dual_supported` guard only rejects the trivial case `a->data == b->data`, but GGML tensor views into the same backing buffer can have different `data` pointers while still mapping to overlapping regions. Because the fused kernel runs both halves concurrently without synchronization, overlapping destinations would turn the deterministic sequential overwrite semantics of separate launches into nondeterministic CUDA write races. Consider adding an explicit non-overlap assertion or range check before fusion, or document that callers must guarantee disjoint destinations.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The dual SET_ROWS fusion lacks a check for overlapping destination memory ranges. The ggml_cuda_set_rows_dual_supported guard only rejects the trivial case a->data == b->data, but GGML tensor views into the same backing buffer can have different data pointers while still mapping to overlapping regions. Because the fused kernel runs both halves concurrently without synchronization, overlapping destinations would turn the deterministic sequential overwrite semantics of separate launches into nondeterministic CUDA write races. Consider adding an explicit non-overlap assertion or range check before fusion, or document that callers must guarantee disjoint destinations.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At ggml/src/ggml-cuda/set-rows.cu, line 538:

<comment>The dual SET_ROWS fusion lacks a check for overlapping destination memory ranges. The `ggml_cuda_set_rows_dual_supported` guard only rejects the trivial case `a->data == b->data`, but GGML tensor views into the same backing buffer can have different `data` pointers while still mapping to overlapping regions. Because the fused kernel runs both halves concurrently without synchronization, overlapping destinations would turn the deterministic sequential overwrite semantics of separate launches into nondeterministic CUDA write races. Consider adding an explicit non-overlap assertion or range check before fusion, or document that callers must guarantee disjoint destinations.</comment>

<file context>
@@ -405,6 +405,171 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
+    if (a->op != GGML_OP_SET_ROWS || b->op != GGML_OP_SET_ROWS) {
+        return false;
+    }
+    if (a->type != b->type || a->data == b->data) {
+        return false;
+    }
</file context>