vulkan: add allreduce function with cross-device CPU proxy and fix Tensor Parallel crash [EXPERIMENTAL]#25051
vulkan: add allreduce function with cross-device CPU proxy and fix Tensor Parallel crash [EXPERIMENTAL]#25051pwilkin wants to merge 7 commits into
Conversation
github-actions
Bot
added
Vulkan
…cache views When a per-device allocation exceeds the backend's max buffer size (e.g. a large KV cache), ggml-alloc returns a multi_buffer wrapping several real buffers. Compute-graph views inherited that multi_buffer as their backend buffer, so a backend that casts tensor->buffer->context to its own buffer-context type (the Vulkan backend does, e.g. in ggml_vk_tensors_overlap) dereferenced garbage and crashed with -sm tensor (issue ggml-org#22197). A view aliases its source's storage, so it must reference the source's real sub-buffer: set t_ij->buffer = t_ij->view_src->buffer. This is the correct ggml invariant and a no-op in the single-buffer case. Assisted-by: Claude Opus 4.8
Implements the backend-agnostic comm hook (ggml_backend_comm_init / _allreduce_tensor / _free, discovered by the meta backend via get_proc_address) for the Vulkan backend, so tensor-parallel inference no longer falls back to the meta backend's CPU-barriered butterfly AllReduce. Consumer GPUs have no P2P here, so the reduce stages through host memory, but everything is ordered on the GPU via exported timeline semaphores (no CPU barriers between layers). Each slice is split into chunks: the dedicated transfer queue streams this device's slice out to shared host memory while the compute queue pulls each peer chunk back as soon as it lands, so the two PCIe directions overlap (full-duplex). Partials are cast to F16 before the host transfer to halve the bytes on the bandwidth-bound link and added straight into the fp32 result via the mixed-type add pipeline. Large prefill activations use this pipeline; small (decode) tensors take a single-shot path where the fixed per-call overhead dominates. Roughly 2.5-3x the butterfly fallback; at long context it overtakes -sm layer and is competitive with CUDA/NCCL on prefill. GGML_VK_COMM_OFF disables the custom comm (falls back to butterfly); GGML_VK_COMM_FP32 forces fp32 staging. Assisted-by: Claude Opus 4.8
…duce The GPU-side cross-device ordering imports each peer's OPAQUE_FD timeline semaphore, but OPAQUE_FD payloads are driver-private, so the import only works when all devices share a driver (e.g. two NVIDIA GPUs). On mixed drivers or vendors it is out of spec. Add a portable fallback: a helper thread polls each peer's progress/upload timeline and host-signals a local timeline that the consumer's download is parked on (core timeline semaphores plus host signal/wait, no imported handle). Both the chunked pipeline (prefill) and the single-shot (decode) paths are bridged, so proxy mode no longer drops decode to the meta-backend butterfly. A capability gate (vkGetPhysicalDeviceExternalSemaphoreProperties plus a driverUUID match) selects the proxy deterministically on unsupported configs; GGML_VK_COMM_PROXY forces it, and the import try/catch stays as a safety net. Measured within ~4% of the native-import path on decode and on par for prefill, with byte-identical output. Assisted-by: Claude Opus 4.8
|
Did a few more tests with an A16 and a 4090: 4090 + A16-sm layer -sm tensor 2xA16-sm layer -sm tensor So as you can see, the 4090 is held down by the A16 and the boost from tensor parallel there is really small, but on 2xA16, the TG boost is almost double while the PP loss is negligible (almost 90% of the original). |
|
AMD Radeon RX 9070 XT & AMD Radeon AI PRO R9700
|
AMD RX7900XTX (2, 4, 8 GPUs)ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: Found 4 Vulkan devices:
ggml_vulkan: Found 8 Vulkan devices:
And ROCm for comparison:ggml_cuda_init: found 2 ROCm devices (Total VRAM: 49120 MiB):
ggml_cuda_init: found 4 ROCm devices (Total VRAM: 98240 MiB):
ggml_cuda_init: found 8 ROCm devices (Total VRAM: 196480 MiB):
|
There was a problem hiding this comment.
Your way of dealing with the crash seems to be similar to the one proposed here (also AI generated, lol) that basically uses the view src to get around the multi buffer issue. I explain this a bit more in #22197 but I think this only works if the tensor is a view tensor. If it's not a view tensor then we don't have any way of knowing which buffer in the multi buffer the tensor is stored in.
There was a problem hiding this comment.
Yup, valid point. Didn't surface in the small models I've tested, but probably will in any bigger one. Paging @0cc4m here: do you have any qualms about just adding a tensor to sub-buffer map in multi_buffer?
|
OK, bit of oddness on my R9700s compared with the 7900XTX above - prefill takes a significant hit compared with Also, even more weirdness - when run as a server, the first request goes through at 36t/s, but the second falls off to 2t/s: |
Synthetic benchmark (pp4096 / tg128, ~4K context)Without this PR:
With this PR applied:
At short context the PR delivers Real-world big prompt benchmark (88K prompt, 164K context)I've asked the LLM to provide me a resume of a book which content has been sent in the prompt Without this PR:
With this PR applied:
The tensor-mode segfault (fixed by this PR)Without this PR, followed by The tensor decode regression at long contextWith the PR applied, tensor-mode tg drops from 43.6 (short ctx) to 9 (long ctx) — 4.8× worse. Row/layer stay at 37. This appears to be the For reference, |
This comment was marked as off-topic.
This comment was marked as off-topic.
2x Intel Arc Pro B60 (Battlemage, BMG G21)
Observations
|
|
@marksverdhei it's a prototype, don't worry :) when it's been decently tested and cleaned up I'll deslopify it :) |
0cc4m
changed the title
pwilkin
changed the title
pwilkin
changed the title
Assisted-by: Claude Opus 4.8
|
So the damn clanker decided to only implement the proper allreduce for 2 GPUs, because why bother ;) sorry for all the people with 4+ who posted their results, could you please retest with the new commit? (I've tested up to 8 GPUs now for correctness) |
…ackend Assisted-by: Claude Opus 4.8
|
Sharing some benchmark results with 2/4/5x Radeon PRO W7900's:
##Vulkan PR TP (2 GPU):
##Vulkan PR TP (4 GPU):
##Vulkan PR TP (5 GPU):
#For comparison with ROCm:
build: 050ee92 (9821)
##ROCm TP RCCL (4 GPU):
##ROCm TP RCCL (5 GPU):
|
|
Could someone else check multi-turn conversations? As noted in my comment above, I'm finding that the first prompt works as expected, but any follow up tanks down to 2t/s. |
|
@AbdullahMPrograms is this after the multiGPU fix already? @netrunnereve I risked a proper fix to the real buffer mapping issue. |
@pwilkin This is with commit: 362fdd2 |
I can reproduce this behavior as well, first prompt: follow up prompt: |
Thank you! At least I know I'm not holding it wrong ;) (EDIT: ...or we both are) |
|
I've also encountered some gibberish in llama-server with TP on 5 GPU's: above on commit e578ca2 |
|
It's faster now on TP=4 and TP=8 (RX7900XTX). Shorter table this time:
And for a multiturn conversation, I don't see any significant degradation. 2nd message: 3rd message: |
|
Looking into the multiGPU degradation, as for the slowdown, can one of you possibly capture a perf profile? |
Assisted-by: Claude Opus 4.8
Sure - not done that before, though. Totally revealing my lack of knowledge here, but can you point me to documentation for doing that? |
Sure: https://perfwiki.github.io/main/tutorial/ It's actually pretty easy: you run |
9 participants
Overview
I've heard from @0cc4m that Vulkan maintainers really like large, LLM assisted PRs, so here's one that should make them happy 😁
This fixes crashes in the pipeline and introduces a Vulkan F16 AllReduce.
Additional information
Benchmark results on my box - would love someone with non-NVidia hardware to test this:
Vulkan tensor-parallel benchmark v2 (rebased+cleaned build) — RTX 3080 + RTX 5060 Ti
pp512 / tg128 (tok/s).
llama-bench -fa 1, default ubatch. 19 LLMs, 2-16GB. Each split-mode in its own run.Cases: Vk-layer / CUDA-layer / Vk-tensor crashfix (butterfly,
GGML_VK_COMM_OFF) / Vk-tensor F16 (this work) / CUDA-tensor (NCCL).Depth 0
Depth 4096
Depth 40000
Requirements