Summary
PR #20463 ("disable graph reuse with pipeline parallelism") introduces a 16% decode speed regression on V100 (SM70) in pipeline-parallel (layer split) mode. Graph reuse was disabled as a precaution, but testing shows it works correctly for the Qwen3.5-35B-A3B MoE architecture on V100.
Benchmark Data
Configuration: Qwen3.5-35B-A3B Q4_K_M, 2× Tesla V100-SXM2-32GB (NVLink, 155 GB/s), --split-mode layer, --flash-attn on, llama-bench -n 128 -r 5
| Build |
Graph Reuse |
tg128 tok/s |
Stddev |
| 8289 (182acfe) |
ON |
91.1 |
±0.4 |
| 8367 (88915cb) |
OFF (#20463) |
83.2 |
±5.1 |
| 8367 + revert #20463 |
ON |
92.8 |
±0.3 |
Root Cause
Without graph reuse, llama.cpp rebuilds the compute graph every decode step. This adds CPU overhead that:
- Reduces steady-state throughput by 16% (91→76 tok/s)
- Introduces high variance (±5 tok/s) as the CPU overhead generates additional heat
- Creates a thermal ramp-down pattern: 93→80 tok/s over 5 runs
Correctness Verification
With graph reuse re-enabled for PP:
- 3 identical generations at temperature=0 (deterministic)
- No NaN or garbled output
- Context scaling works correctly (tested 0-32K context)
- Architecture: 40 layers (28 GatedDeltaNet + 12 Gated Attention), MoE with 256 experts top-8
Suggestion
Consider a more targeted fix than blanket disabling:
- Only disable graph reuse for specific architectures that are known to be affected
- Add a runtime flag to override (e.g.,
--graph-reuse-enable-pp)
- Further investigate what PP scenarios break with graph reuse
The original #20463 TODO says "figure out a way to make graph reuse work with pipeline parallelism" — the data suggests it already works for at least this architecture on V100.
Environment
- 2× Tesla V100-SXM2-32GB
- NVLink 2.0 (6 links, 155 GB/s)
- CUDA 12.8
- Build: SM70,
USE_GRAPHS=1, ARCHS=700
Summary
PR #20463 ("disable graph reuse with pipeline parallelism") introduces a 16% decode speed regression on V100 (SM70) in pipeline-parallel (layer split) mode. Graph reuse was disabled as a precaution, but testing shows it works correctly for the Qwen3.5-35B-A3B MoE architecture on V100.
Benchmark Data
Configuration: Qwen3.5-35B-A3B Q4_K_M, 2× Tesla V100-SXM2-32GB (NVLink, 155 GB/s),
--split-mode layer,--flash-attn on,llama-bench -n 128 -r 5Root Cause
Without graph reuse, llama.cpp rebuilds the compute graph every decode step. This adds CPU overhead that:
Correctness Verification
With graph reuse re-enabled for PP:
Suggestion
Consider a more targeted fix than blanket disabling:
--graph-reuse-enable-pp)The original #20463 TODO says "figure out a way to make graph reuse work with pipeline parallelism" — the data suggests it already works for at least this architecture on V100.
Environment
USE_GRAPHS=1,ARCHS=700