Sched: Reintroduce less synchronizations between token, with fixed pipeline parallelism. by aendk · Pull Request #20793 · ggml-org/llama.cpp

aendk · 2026-03-20T10:19:40Z

Follow up to #20463 (comment).

#17795 improved performance in the single GPU setting on CUDA, but it was rolled back due to a bug surfacing in multi-GPU pipeline parallel settings.

For the single GPU setting, it moved the scheduling from sassassasg to the more efficient saaasg pattern, where s= sync, a= async copy, g= graph execution.
Each asynchronous copy was enclosed in two synchronizations. Removing some superfluous synchronizations improved performance, especially on windows. The change was to only do a single synchronization between memory copies and graph execution.

However in multi-GPU settings, we saw llama-perplexity regressions indicating incorrect scheduling (#20463).

I found that the event-based pipeline parallelism scheduling mechanism very likely implicitly relies on synchronous copies, as (i) in my testing copy_from_host worked as intended, and (ii) disabling it and therefore introducing synchronous copies fixed the bug, llama-perplexity perplexity was then identical to master.

The proposed fix here is therefore to enroll pipeline parallelism into the same synchronization between async copies and graph execution as the single GPU case already has.
I think this can be a good solution as it keeps scheduling similar between single GPU and multi GPU, and because it is simpler and safer than reworking the event-driven pipeline parallelism logic.

In my testing, this proposal has same performance benefits as the initial PR, and it yields correct perplexity scores both in single and multi-GPU.
As this bug surfaced in the community with their more diverse hardware setups and usage scenarios, it would be awesome if you could test-drive this change both with llama-bench and llama-perplexity with your usual model and launch-options usage!
@mxxm-t @slavap @Superbobo75 @thejacer

If you can, check out this branch and compare this against its master (git checkout HEAD~2). Let me know if you run into performance or accuracy issues!

mxxm-t · 2026-03-22T14:30:40Z

Will test soon.

aendk · 2026-03-24T09:30:29Z

@mxxm-t don't bother right now.
I will keep this open for now, but the solution is incomplete. This just adds a "speed bump" of some sorts so the race condition does not appear, but it is not a real scheduling fix.

Once I think I have the correct solution, I'll force-push and ping you again.

aendk · 2026-03-24T14:39:03Z

With #20927 being merged, I now see identical PPL in master and the reapplied original PR on my RTX PRO 6000 Max-Q / RTX PRO 4500 setup of Final estimate: PPL = 28.8008 +/- 1.50705

@mxxm-t feel free to try it if time allows.

Linux Performance

scripts$ python compare-llama-bench.py -c akieslinger/rework-reduce-per-token-syncs -b master -i ../llama-bench.sqlite
| Model                    | Test   |   t/s master |   t/s akieslinger/rework-reduce-per-token-syncs |   Speedup |
|:-------------------------|:-------|-------------:|------------------------------------------------:|----------:|
| gpt-oss 20B MXFP4 MoE    | tg128  |       285.17 |                                          286.79 |      1.01 |
| gpt-oss 20B MXFP4 MoE    | tg256  |       285.33 |                                          287.04 |      1.01 |
| gpt-oss 20B MXFP4 MoE    | tg512  |       278.70 |                                          281.61 |      1.01 |
| qwen3next 80B.A3B Q4_K_M | tg128  |       163.12 |                                          163.91 |      1.00 |
| qwen3next 80B.A3B Q4_K_M | tg256  |       163.15 |                                          164.56 |      1.01 |
| qwen3next 80B.A3B Q4_K_M | tg512  |       163.67 |                                          164.46 |      1.00 |

sjoerdmaessen · 2026-03-26T13:13:27Z

Benchmark: 2x NVIDIA L40S (sm_89 / Lovelace)

Tested with model: Qwen3.5-122B-A10B Q5_K_S

Hardware: 2x NVIDIA L40S 48GB, AMD EPYC 9354P, Linux
Model: Qwen3.5-122B-A10B Q5_K_S (80.44 GiB, split across both GPUs)
Flags: -ngl 99 -fa 1 -t 4, 3 repetitions per test

Master (`dc8d14c`, b8537)

test	t/s
pp512	2103.93 ± 42.88
pp1024	2472.55 ± 11.66
pp2048	2694.09 ± 6.61
tg128	62.05, 62.13, 62.14

PR branch (`06e8b36`, b8508)

test	t/s
pp512	2122.19 ± 6.42
pp1024	2454.35 ± 5.46
pp2048	2677.33 ± 8.63
tg128	62.20, 62.20, 62.23

Summary

All results within noise on this setup. No regression, no measurable improvement. tg128 ~+0.1 t/s (within margin of error). This is consistent with your observation that the benefit is primarily on Windows I think, Linux multi-GPU pipeline parallelism scheduling appears unaffected by this change for my setup.

aendk · 2026-03-31T09:46:52Z

@mxxm-t @slavap @Superbobo75 @thejacer if you have the time, please test PPL (as outlined in #20463 (comment)) and performance on your setups.

thejacer · 2026-04-01T23:51:02Z

PR #20793

perplexity: calculating perplexity over 8 chunks, n_ctx=512, batch_size=2048, n_seq=4
perplexity: 3.82 seconds per pass - ETA 0.12 minutes
[1]2.7627,[2]2.0251,[3]2.4352,[4]2.2139,[5]2.3001,[6]2.3767,[7]2.3087,[8]2.3420,
Final estimate: PPL = 2.3420 +/- 0.10001

| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium |  20.27 GiB |    34.66 B | ROCm       |  99 |  1 |    0 |           pp512 |        870.66 ± 4.24 |
| qwen35moe 35B.A3B Q4_K - Medium |  20.27 GiB |    34.66 B | ROCm       |  99 |  1 |    0 |           tg128 |         55.22 ± 0.11 |

build: 06e8b36a6 (8508)

Master

perplexity: calculating perplexity over 8 chunks, n_ctx=512, batch_size=2048, n_seq=4
perplexity: 3.84 seconds per pass - ETA 0.12 minutes
[1]2.7627,[2]2.0251,[3]2.4352,[4]2.2139,[5]2.3001,[6]2.3767,[7]2.3087,[8]2.3420,
Final estimate: PPL = 2.3420 +/- 0.10001

| model                          |       size |     params | backend    | ngl | fa | mmap |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ---: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium |  20.27 GiB |    34.66 B | ROCm       |  99 |  1 |    0 |           pp512 |        872.38 ± 2.69 |
| qwen35moe 35B.A3B Q4_K - Medium |  20.27 GiB |    34.66 B | ROCm       |  99 |  1 |    0 |           tg128 |         55.27 ± 0.09 |

build: 6de97b9d3 (8623)

aendk · 2026-04-07T12:49:48Z

@ggerganov
Looks like my initial PR works with @am17an's multi-GPU fix, as it yields correct PPL in the multi-GPU, too.
Do you think additional testing is required to merge this again?

JohannesGaessler

Sorry, I'm confused. The way I read the linked PR in which the original one got reverted @ggerganov is saying that the original PR had to be reverted to restore correct results. But isn't this PR making the exact same changes as before?

aendk · 2026-04-08T08:08:58Z

Sorry, I'm confused. The way I read the linked PR in which the original one got reverted @ggerganov is saying that the original PR had to be reverted to restore correct results. But isn't this PR making the exact same changes as before?

correct in both aspects. Since then @am17an merged his fix. Reapplying my changes now still yields correct results, since the scheduling bug was not part of my PR. My PR just removed a lot of "speed bumps" in the form of synchronizations, which exposed the bug.

JohannesGaessler · 2026-04-08T08:13:05Z

Can you also link the fix in question?

am17an · 2026-04-08T08:13:18Z

I think we should wait for the TP PR (#13776) to be merged and stable before merging this again. Debugging synchronization issues is a pain.

am17an · 2026-04-08T08:15:12Z

@JohannesGaessler it's #20927

aendk · 2026-04-14T09:48:20Z

@am17an @JohannesGaessler do we have a rough timeline when we consider #19378 to be stable?

Since this closes a major perf gap between windows and linux, most of the user base is on windows, and there is no real benefit leaving it sitting, revisiting/merging this sooner than later makes sense to me.

IMbackK · 2026-04-14T10:21:13Z

I need to find the time to test this on HIP first for sure, since the previous attempt at this broke HIP entirely.

JohannesGaessler · 2026-04-14T10:21:14Z

Sorry, I forgot about this PR. Please rebase on top of master and I'll check whether there are issues (though the synchronization logic should be backend-agnostic).

aendk · 2026-04-14T11:55:29Z

I rebased and spot-checked single and multi-GPU again. In both cases, PPL is identical to master.
Also, note that only cosmetics were changed in the rebase (dst->buffer -> buf_dst), so the findings from the community above should still apply.
@IMbackK good call. it should not break hip, otherwise let me know.
@JohannesGaessler thanks, I consider the PR to be ready now.

JohannesGaessler · 2026-04-14T12:21:48Z

Using 4x RTX 4090 I am unable to provoke issues, the PPL values I get are bit-for-bit identical both for --split-mode layer and --split-mode tensor.

thejacer · 2026-04-14T13:25:38Z

I need to find the time to test this on HIP first for sure, since the previous attempt at this broke HIP entirely.

My testing above was done on 2xMi50. Does that satisfy HIP testing?

IMbackK · 2026-04-14T13:53:40Z

@thejacer that helps, yeah

IMbackK · 2026-04-14T17:05:53Z

Unfortionatly this reintroduces #20433 on hip, repdoucer from that issue:

This pr:
[1]281.8696,[2]238.4758,[3]252.0735,[4]237.7406,[5]251.8294,[6]256.8782,[7]245.3641,[8]239.9153,[9]239.4646,[10]237.5819,[11]235.9763,[12]238.7037,[13]236.2493,

Master @b8785

[1]271.8923,[2]241.0099,[3]242.2813,[4]233.9529,[5]234.6784,[6]236.7585,[7]240.0973,[8]239.3741,[9]238.8985,[10]237.3241,[11]239.6173,[12]239.7815,[13]237.2340,

aendk · 2026-04-15T11:48:13Z

@IMbackK I guess there is a subtle difference in the inherent ordering of memcpy and compute between CUDA and hip. That means it makes sense to make the saaasg-pattern (s=sync, a=async memcpy, g=graph compute) in single-GPU explicit in multi-GPU, too.

Just out of curiosity, can you check if unguarding the ggml_backend_synchronize(split_backend); either in L1547-L1556 or in L1670-L1673 of this diff is required for hip consistency, or both? So just replace these two if-conditions to if(true) and test these 3 cases (both modified to true + top true, bottom default + top default, bottom true)?
Can you check both performance and PPL?

IMbackK · 2026-04-19T11:25:31Z

llama-bench -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -fa 1 -p 4096 -n 128 -ub 256 -sm layer,tensor

patch1.patch
patch2.patch

Master:

model	size	params	backend	ngl	sm	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	pp4096	4359.42 ± 6.17
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	tg128	109.77 ± 0.56
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	pp4096	3490.72 ± 24.08
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	tg128	98.57 ± 0.45

PR (rebased):
PPL: fail

model	size	params	backend	ngl	sm	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	pp4096	4388.10 ± 9.35
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	tg128	110.74 ± 0.66
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	pp4096	3544.33 ± 9.81
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	tg128	98.67 ± 0.37

Patch1:
PPL: pass

model	size	params	backend	ngl	sm	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	pp4096	4294.21 ± 9.30
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	tg128	109.74 ± 0.66
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	pp4096	3505.95 ± 14.07
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	tg128	98.47 ± 0.36

Patch2:
PPL: pass

model	size	params	backend	ngl	sm	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	pp4096	4352.22 ± 8.19
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	tg128	109.03 ± 0.45
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	pp4096	3476.17 ± 14.28
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	tg128	97.45 ± 0.37

Both:

PPL: pass

model	size	params	backend	ngl	sm	fa	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	pp4096	4312.89 ± 11.18
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	layer	1	tg128	109.52 ± 0.73
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	pp4096	3559.21 ± 4.11
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	ROCm	99	tensor	1	tg128	98.42 ± 0.29

All of the performance values are within run to run variance. From the perspective of purely the hip backend, this pr is not worthwhile in the first place.

…kend pipeline parallel bugs.

GPU split optimization

non-async backends

…hat it is out of precaution, but that no perf-impact is visible, and that it can be revisited separately anytime.

DEV-DUFORD · 2026-06-26T17:58:34Z

@ggerganov @aendk I can confirm on my 4x MI50 Rig, running ROCm 6.3.3 Official, this is causing a major regression in speed, around a 50% loss in prefill speed:

Before:

nerd-ai@nerd-ai:/llama.cpp$ ./build/bin/llama-bench -m /mnt/gguf_store/Qwen3.6-35B-A3B/Qwen_Qwen3.6-35B-A3B-Q8_0.gguf -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 512,2048,4096 -n 128,1024<br>ggml_cuda_init: found 4 ROCm devices (Total VRAM: 131008 MiB):<br>Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 2: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 3: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB</p><div class="table-wrapper" style="overflow-x: auto;">
model | size | params | backend | ngl | fa | mmap | dio | test | t/s
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp512 | 725.34 ± 4.62
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp2048 | 1335.37 ± 5.31
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp4096 | 1497.48 ± 9.47
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg128 | 64.35 ± 0.55
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg1024 | 64.43 ± 0.46

build: 60bc8866b (9796)

nerd-ai@nerd-ai:/llama.cpp$ ./build/bin/llama-bench -m /mnt/gguf_store/Qwen3.6-27B/Qwen_Qwen3.6-27B-Q8_0.gguf -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 512,2048,4096 -n 128,1024<br>ggml_cuda_init: found 4 ROCm devices (Total VRAM: 131008 MiB):<br>Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 2: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 3: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB</p><div class="table-wrapper" style="overflow-x: auto;">
model | size | params | backend | ngl | fa | mmap | dio | test | t/s
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp512 | 155.27 ± 0.31
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp2048 | 306.17 ± 0.14
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp4096 | 347.33 ± 0.13
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | tg128 | 18.80 ± 0.02
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | tg1024 | 18.70 ± 0.03

build: 60bc8866b (9796)

After:

nerd-ai@nerd-ai:/llama.cpp$ ./build/bin/llama-bench -m /mnt/gguf_store/Qwen3.6-35B-A3B/Qwen_Qwen3.6-35B-A3B-Q8_0.gguf -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -
p 512,2048,4096 -n 128,1024<br>ggml_cuda_init: found 4 ROCm devices (Total VRAM: 131008 MiB):<br>Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 2: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 3: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB</p><div class="table-wrapper" style="overflow-x: auto;">
model | size | params | backend | ngl | fa | mmap | dio | test | t/s
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp512 | 724.57 ± 4.94
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp2048 | 831.74 ± 2.71
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp4096 | 842.52 ± 4.55
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg128 | 60.04 ± 0.59
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg1024 | 59.47 ± 0.32

build: 3fc4e1052 (9820)

nerd-ai@nerd-ai:/llama.cpp$ ./build/bin/llama-bench -m /mnt/gguf_store/Qwen3.6-27B/Qwen_Qwen3.6-27B-Q8_0.gguf -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 512,20
48,4096 -n 128,1024<br>ggml_cuda_init: found 4 ROCm devices (Total VRAM: 131008 MiB):<br>Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 2: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 3: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB</p><div class="table-wrapper" style="overflow-x: auto;">
model | size | params | backend | ngl | fa | mmap | dio | test | t/s
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp512 | 154.90 ± 1.10
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp2048 | 189.75 ± 0.05
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | pp4096 | 195.71 ± 0.03
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | tg128 | 18.53 ± 0.02
qwen35 27B Q8_0 | 26.69 GiB | 26.90 B | ROCm | 999 | 1 | 0 | 1 | tg1024 | 18.64 ± 0.07

build: 3fc4e1052 (9820)

While these cards are EOL the version of ROCm that I am using does officially support them.

Edit: To be thorough, pulled down the commit prior to this merge, then built:

nerd-ai@nerd-ai:/llama.cpp$ ./build/bin/llama-bench -m /mnt/gguf_store/Qwen3.6-35B-A3B/Qwen_Qwen3.6-35B-A3B-Q8_0.gguf -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -
p 512,2048,4096 -n 128,1024<br>ggml_cuda_init: found 4 ROCm devices (Total VRAM: 131008 MiB):<br>Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 2: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB<br>Device 3: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64, VRAM: 32752 MiB</p><div class="table-wrapper" style="overflow-x: auto;">
model | size | params | backend | ngl | fa | mmap | dio | test | t/s
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp512 | 726.03 ± 4.90
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp2048 | 1336.90 ± 6.75
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | pp4096 | 1497.06 ± 9.76
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg128 | 64.77 ± 0.51
qwen35moe 35B.A3B Q8_0 | 34.37 GiB | 34.66 B | ROCm | 999 | 1 | 0 | 1 | tg1024 | 64.83 ± 0.50

build: 5d8ccdf9d (9819)

…aster

@ggerganov

…org#20793) * CUDA: Improve performance via less synchronizations between token (ggml-org#17795) * Adds CPU-to-CUDA copy capability to ggml_backend_cuda_cpy_tensor_async() * Adds function to relax sync requirements between input copies on supported backends (CUDA for now) * Exchanges synchronous copy with async copy function. * Adds macro guards to allow compilation in non-CUDA builds * Reworked backend detection in ggml-backend.cpp to avoid linking conflicts * Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues * Minor cleanup * Makes opt-in to relax use of explicit syncs more general. Backends like vulkan which require a synchronization between HtoD copies and graph execution could also adopt this change now. * Reintroduces stricter check for CPU->CUDA backend async copy via GGML_DEVICE_TYPE_CPU. * Corrects initialization of ggml_backend_sync_mode in ggml_backend_sched_split initialization * Simplifies synchronizations to adhere to `saaasg` pattern. * Apply suggestion from @ggerganov (src->buffer to buf_src) Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestion from @ggerganov (src->buffer to buf_src) v2 Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestions from @JohannesGaessler code review Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs. * Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split -> GPU split optimization * Scheduler Hardening: Re-adding original additional synchronizations for non-async backends * Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of precaution, but that no perf-impact is visible, and that it can be revisited separately anytime. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

slavap · 2026-06-27T04:55:30Z

@aendk @ggerganov
-> Exact wording relevant here: " Unsupported - The current ROCm release does not support this hardware. The HIP runtime might continue to run applications for an unsupported GPU, but prebuilt ROCm libraries are not officially supported and will cause runtime errors" [emphasis mine].
-> ROCM-7.0.1 removed support for MI50 (https://newreleases.io/project/github/ROCm/ROCm/release/rocm-7.0.1)
Because there is no support and testing for his HW/SW stack, run time issues cannot be ruled out.

This type of argumentation is really disappointing. The thing is that mi50 is old, but still very popular, and support has been restored for it in modernized Rocm (TheRock) https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md
Also despite of pointless "blah-blah-blah unsupported" - it works fine in Rocm 7.2.3 (and actually faster than in 6.3.3)

nathanmp · 2026-06-27T06:13:39Z

It definitely isn't just EOL hardware, this is a pretty big regression on my 3 R9700s. On the Qwen models and Gemma 31B, I get around a 35% decrease in prefill. Decode seems unaffected for dense models but it takes a hit for MOE models. Software: Ubuntu 24.04, ROCm version 7.2.1.

Tested on commit 5d8ccdf (immediately before this PR) and commit 3fc4e10 (this PR).

Before/5d8ccdf:

nathan@Theseus:~/bin/llama.cpp-prechange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/Qwen3.6-35B-A3B-Q8_0.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q8_0         |  35.19 GiB |    35.51 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |      5471.62 ± 14.17 |
| qwen35moe 35B.A3B Q8_0         |  35.19 GiB |    35.51 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         72.46 ± 0.10 |

nathan@Theseus:~/bin/llama.cpp-prechange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/Qwen3.6-27B-Q6_K.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |       1312.72 ± 2.98 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         22.32 ± 0.04 |

nathan@Theseus:~/bin/llama.cpp-prechange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q6_K            |  22.17 GiB |    25.23 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |      2308.53 ± 12.22 |
| gemma4 26B.A4B Q6_K            |  22.17 GiB |    25.23 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         71.94 ± 0.20 |

nathan@Theseus:~/bin/llama.cpp-prechange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/gemma-4-31B-it-Q8_0.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| gemma4 31B Q8_0                |  30.38 GiB |    30.70 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |       1781.16 ± 4.75 |
| gemma4 31B Q8_0                |  30.38 GiB |    30.70 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         16.18 ± 0.02 |

After/3fc4e10:

nathan@Theseus:~/bin/llama.cpp-postchange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/Qwen3.6-35B-A3B-Q8_0.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q8_0         |  35.19 GiB |    35.51 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |      3547.74 ± 10.50 |
| qwen35moe 35B.A3B Q8_0         |  35.19 GiB |    35.51 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         63.47 ± 0.08 |

nathan@Theseus:~/bin/llama.cpp-postchange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/Qwen3.6-27B-Q6_K.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |        851.84 ± 1.45 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         21.77 ± 0.03 |

nathan@Theseus:~/bin/llama.cpp-postchange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| gemma4 26B.A4B Q6_K            |  22.17 GiB |    25.23 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |      2290.15 ± 13.22 |
| gemma4 26B.A4B Q6_K            |  22.17 GiB |    25.23 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         59.96 ± 0.34 |

nathan@Theseus:~/bin/llama.cpp-postchange$ ./build/bin/llama-bench -ngl 999 -fa 1 -mmp 0 -dio 1 -r 5 -sm layer -p 4096 -n 1024 -m ~/models/gemma-4-31B-it-Q8_0.gguf --device ROCm0/ROCm1/ROCm2
| model                          |       size |     params | backend    | ngl |  fa | dev          | mmap | dio |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---: | --: | --------------: | -------------------: |
| gemma4 31B Q8_0                |  30.38 GiB |    30.70 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          pp4096 |       1165.06 ± 1.54 |
| gemma4 31B Q8_0                |  30.38 GiB |    30.70 B | ROCm,Vulkan | 999 |   1 | ROCm0/ROCm1/ROCm2 |    0 |   1 |          tg1024 |         16.01 ± 0.01 |

ggml-org#20793)" This reverts commit 3fc4e10.

JohannesGaessler · 2026-06-28T17:00:09Z

We definitely should not be breaking HIP performance like this for a few % on other hardware. Revert this PR.

ORippler · 2026-06-29T10:20:42Z

We definitely should not be breaking HIP performance like this for a few % on other hardware.

Agreed, though I must say I'm surprised as we did validate HIP performance to be fine at some point of the journey #20793 (comment) (though only for a single dense model it seem).

As I understand it, #20927 makes sure set_inputs() does not rewrite reused graph inputs while the previous graph execution is still in flight.

The issue I am pointing at is one level lower where the scheduler starts an async H2D copy from a GGML_TENSOR_FLAG_INPUT source then the same source tensor can be reused later. If the H2D copy has not finished reading the host source, the later write can corrupt the copy.

Tensors flagged as GGML_TENSOR_FLAG_INPUT are guaranteed to be allocated in non-overlapping regions without reuse though, meaning that a fence between ubatch invocations should be sufficient (though there may be some issues/nuances in mapping global-inputs to per-split-graph-inputs.)

am17an · 2026-06-29T11:05:10Z

without reuse though

I don't think this is generally true for this flag although it is true for GGML_TENSOR_FLAG_OUTPUT.

ORippler · 2026-06-29T13:10:08Z

I don't think this is generally true for this flag although it is true for GGML_TENSOR_FLAG_OUTPUT.

I stand corrected :) Though for our intents, the non-overlapping aspect in itself should be sufficient as we effectively "batch-copy" all inputs at the beginning 🤔

I think a bisect of this PR's commits on PPL/Perf on AMD GPUs would help greatly root-cause this as it should be performance positive across backends - unfortunately I don't have access to AMD GPUs.

#20793)" (#25138)

ggml-org#20793)" (ggml-org#25138)

@ggerganov

…org#20793) * CUDA: Improve performance via less synchronizations between token (ggml-org#17795) * Adds CPU-to-CUDA copy capability to ggml_backend_cuda_cpy_tensor_async() * Adds function to relax sync requirements between input copies on supported backends (CUDA for now) * Exchanges synchronous copy with async copy function. * Adds macro guards to allow compilation in non-CUDA builds * Reworked backend detection in ggml-backend.cpp to avoid linking conflicts * Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues * Minor cleanup * Makes opt-in to relax use of explicit syncs more general. Backends like vulkan which require a synchronization between HtoD copies and graph execution could also adopt this change now. * Reintroduces stricter check for CPU->CUDA backend async copy via GGML_DEVICE_TYPE_CPU. * Corrects initialization of ggml_backend_sync_mode in ggml_backend_sched_split initialization * Simplifies synchronizations to adhere to `saaasg` pattern. * Apply suggestion from @ggerganov (src->buffer to buf_src) Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestion from @ggerganov (src->buffer to buf_src) v2 Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Apply suggestions from @JohannesGaessler code review Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs. * Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split -> GPU split optimization * Scheduler Hardening: Re-adding original additional synchronizations for non-async backends * Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of precaution, but that no perf-impact is visible, and that it can be revisited separately anytime. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

ggml-org#20793)" (ggml-org#25138)

sched : reintroduce less synchronizations during split compute (ggml-org#20793)

This comment was marked as outdated.

Sign in to view

github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Mar 20, 2026

aendk force-pushed the akieslinger/rework-reduce-per-token-syncs branch from a48fd3b to 06e8b36 Compare March 24, 2026 14:38

hai-pilgrim mentioned this pull request Mar 29, 2026

perf(cuda): reduce redundant per-token synchronizations heiervang-technologies/ht-llama.cpp#22

Closed

4 tasks

aendk marked this pull request as ready for review March 31, 2026 09:44

aendk requested a review from a team as a code owner March 31, 2026 09:44

JohannesGaessler reviewed Apr 7, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated

Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated

aendk force-pushed the akieslinger/rework-reduce-per-token-syncs branch from 5002405 to 38a6f1e Compare April 14, 2026 11:41

aendk added 4 commits June 26, 2026 14:10

Adds single-GPU synchronizations to multi-GPU settings to fix hip bac…

c497965

…kend pipeline parallel bugs.

Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split ->

ede1d0e

GPU split optimization

Scheduler Hardening: Re-adding original additional synchronizations for

2150d7d

non-async backends

Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights t…

0234bb9

…hat it is out of precaution, but that no perf-impact is visible, and that it can be revisited separately anytime.

aendk force-pushed the akieslinger/rework-reduce-per-token-syncs branch from 29baa60 to 0234bb9 Compare June 26, 2026 12:11

github-actions Bot added the CUDA Related to the CUDA backend label Jun 26, 2026

ggerganov merged commit 3fc4e10 into ggml-org:master Jun 26, 2026
30 of 31 checks passed

DEV-DUFORD mentioned this pull request Jun 26, 2026

50% loss in Prefill Speed on Multiple AMD ISAs, Build 9820 #25062

Closed

DEV-DUFORD added a commit to DEV-DUFORD/llama.cpp that referenced this pull request Jun 26, 2026

Merge revert of ggml-org#20793 (prefill regression) into DEV-DUFORD/m…

9c81825

…aster

LostRuins added a commit to LostRuins/koboldcpp that referenced this pull request Jun 27, 2026

Revert "sched : reintroduce less synchronizations during split compute (

16f197a

ggml-org#20793)" This reverts commit 3fc4e10.

nathanmp mentioned this pull request Jun 29, 2026

Revert PR #20793 #25120

Closed

ORippler mentioned this pull request Jun 29, 2026

Revert "sched : reintroduce less synchronizations during split comput… #25138

Merged

am17an pushed a commit that referenced this pull request Jun 30, 2026

Revert "sched : reintroduce less synchronizations during split compute (

86b9470

#20793)" (#25138)

turbo-tan pushed a commit to turbo-tan/llama.cpp-tq3 that referenced this pull request Jul 1, 2026

Revert "sched : reintroduce less synchronizations during split compute (

30033a4

ggml-org#20793)" (ggml-org#25138)

JohannesGaessler mentioned this pull request Jul 5, 2026

ggml: asynchronous scheduler memory copies #25319

Draft

adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026

Revert "sched : reintroduce less synchronizations during split compute (

becdfac

ggml-org#20793)" (ggml-org#25138)

XZiar pushed a commit to XZiar/llama.cpp that referenced this pull request Jul 6, 2026

fix for nex-n2

43433c9

sched : reintroduce less synchronizations during split compute (ggml-org#20793)

XZiar pushed a commit to XZiar/llama.cpp that referenced this pull request Jul 8, 2026

fix for nex-n2

91a2819

sched : reintroduce less synchronizations during split compute (ggml-org#20793)

am17an mentioned this pull request Jul 9, 2026

MTP feature performance drops since b9935 #25489

Open

XZiar pushed a commit to XZiar/llama.cpp that referenced this pull request Jul 9, 2026

fix for nex-n2

c7d05a8

sched : reintroduce less synchronizations during split compute (ggml-org#20793)

XZiar pushed a commit to XZiar/llama.cpp that referenced this pull request Jul 9, 2026

fix for nex-n2

f328696

sched : reintroduce less synchronizations during split compute (ggml-org#20793)

Uh oh!

Conversation

aendk commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

mxxm-t commented Mar 22, 2026

Uh oh!

aendk commented Mar 24, 2026

Uh oh!

aendk commented Mar 24, 2026

Uh oh!

sjoerdmaessen commented Mar 26, 2026

Benchmark: 2x NVIDIA L40S (sm_89 / Lovelace)

Master (dc8d14c, b8537)

PR branch (06e8b36, b8508)

Summary

Uh oh!

aendk commented Mar 31, 2026

Uh oh!

thejacer commented Apr 1, 2026

Uh oh!

aendk commented Apr 7, 2026

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aendk commented Apr 8, 2026

Uh oh!

JohannesGaessler commented Apr 8, 2026

Uh oh!

am17an commented Apr 8, 2026

Uh oh!

am17an commented Apr 8, 2026

Uh oh!

aendk commented Apr 14, 2026

Uh oh!

IMbackK commented Apr 14, 2026

Uh oh!

JohannesGaessler commented Apr 14, 2026

Uh oh!

aendk commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented Apr 14, 2026

Uh oh!

thejacer commented Apr 14, 2026

Uh oh!

IMbackK commented Apr 14, 2026

Uh oh!

IMbackK commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aendk commented Apr 15, 2026

Uh oh!

IMbackK commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

DEV-DUFORD commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slavap commented Jun 27, 2026

Uh oh!

nathanmp commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented Jun 28, 2026

Uh oh!

ORippler commented Jun 29, 2026

Uh oh!

am17an commented Jun 29, 2026

Uh oh!

ORippler commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

aendk commented Mar 20, 2026 •

edited

Loading

Master (`dc8d14c`, b8537)

PR branch (`06e8b36`, b8508)

aendk commented Apr 14, 2026 •

edited

Loading

IMbackK commented Apr 14, 2026 •

edited

Loading

IMbackK commented Apr 19, 2026 •

edited

Loading

DEV-DUFORD commented Jun 26, 2026 •

edited

Loading

nathanmp commented Jun 27, 2026 •

edited

Loading

ORippler commented Jun 29, 2026 •

edited

Loading