Eval bug: AMD Vulkan vk::DeviceLostError crash sensitive to ubatch-size and context length

[fkroener]

Name and Version

version: 8327 (8f974d2)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

ggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
~120 GB unified memory

Models

unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL

Problem description & steps to reproduce

Similar to #20462, but using a single GPU only llama-server crashes with medium (~65k-80k) token usage. Usually the crash happens when being fed the context and re-processing it entirely, sometimes I can exchange one or two messages before it crashes.

I thought it might be connected to recent GATED_DELTA_NET op vulkan additions, but reverting those didn't make a difference and git bisect has me kinda stuck, finding only bad revisions:

First Bad Commit

No response

Relevant log output

git bisect log

git bisect start
# status: waiting for both good and bad commits
# bad: [983df142a99b764f5cbe9acb6b9ec862f861d353] convert : fix/suppress pyright errors (#20442)
git bisect bad 983df142a99b764f5cbe9acb6b9ec862f861d353
# status: waiting for good commit(s), bad commit known
# bad: [de190154c85d20e24dbeae8c8af1849402ae5098] New conversations now auto-select the first loaded model (#20403)
git bisect bad de190154c85d20e24dbeae8c8af1849402ae5098
# status: waiting for good commit(s), bad commit known
# bad: [0516e04bf9c002bf190519af6795415ab3ebd059] opencl: use larger workgroup size for get_rows (#20316)
git bisect bad 0516e04bf9c002bf190519af6795415ab3ebd059
# status: waiting for good commit(s), bad commit known
# bad: [9ef7523ee99e13fe57a3da5cd600c206ed997630] cuda/hip: fix loop unrolling in ssm-conv (#20369)
git bisect bad 9ef7523ee99e13fe57a3da5cd600c206ed997630
# status: waiting for good commit(s), bad commit known
# bad: [6fce5c6a7dba6a3e1df0aad1574b78d1a1970621] opencl: add l2_norm (#20160)
git bisect bad 6fce5c6a7dba6a3e1df0aad1574b78d1a1970621
# status: waiting for good commit(s), bad commit known
# bad: [f6235a41efc8fe10b3f85619d46ee86dd50142f7] webui: Agentic Loop + MCP Client with support for Tools, Resources and Prompts (#18655)
git bisect bad f6235a41efc8fe10b3f85619d46ee86dd50142f7
# status: waiting for good commit(s), bad commit known
# bad: [872646b30c342c5fa98efec002f94fdb4f1f1ea5] model : update Qwen3.5 model type detection (#20126)
git bisect bad 872646b30c342c5fa98efec002f94fdb4f1f1ea5
# status: waiting for good commit(s), bad commit known
# bad: [66d65ec29ba7c440cbc31b6f63b74a17b536ba65] cuda: cap grid.y at 65535 in non-contiguous dequantize/convert kernels (#19999)
git bisect bad 66d65ec29ba7c440cbc31b6f63b74a17b536ba65
# status: waiting for good commit(s), bad commit known
# bad: [8f974d2392da4e6fa422a67050e90f1471d72966] mtmd : rename mtmd_get_audio_bitrate to mtmd_get_audio_sample_rate (#20105)
git bisect bad 8f974d2392da4e6fa422a67050e90f1471d72966
# status: waiting for good commit(s), bad commit known

Using

llama-server --host 127.0.0.1 --min-p 0.0 --port 44827 --presence-penalty 0.0 --repeat-penalty 1.0 --sleep-idle-seconds 900 --spec-ngram-size-n 48 --spec-type ngram-mod --temperature 0.6 --top-k 20 --top-p 0.95 --alias unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL --batch-size 2048 --device Vulkan0 --direct-io --hf-repo unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL --parallel 2 --ubatch-size 2048'

tail of my logfile with -DGGML_VULKAN_DEBUG:BOOL=ON: amdgpu-crash.txt

[0cc4m]

Which driver version are you using? If you're not sure, vulkaninfo --summary shows it.

[fkroener]

GPU0:
        apiVersion         = 1.4.344
        driverVersion      = 26.0.99
        vendorID           = 0x1002
        deviceID           = 0x1586
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = Radeon 8060S Graphics (RADV STRIX_HALO)
        driverID           = DRIVER_ID_MESA_RADV
        driverName         = radv
        driverInfo         = Mesa 26.1.0-devel (git-17b18496f6)

btw. I found my first 'good' commit with [fc0fe40] models : support qwen3.5 series (#19468) - at least I could process the entire context twice and called it good enough since checkpoints didn't work properly back then with qwen3.5

[0cc4m]

Try a driver version that's not quite as bleeding edge, maybe it's a regression.

[fkroener]

Downgrading to Mesa 26.0.2 shows the same crash. Tested #20518 just in case, but it also crashed. I'm back to bisecting for now, but I don't really trust the two 'good' revisions I identified so far, since possibly prompt processing the context twice might have just been a fluke.

[fkroener]

So according to bisect first bad commit is 65cede7, which I don't really trust, because my criteria for 'good' so far was 'let's me reprocess the context at least twice without crashing', since it takes so long.

I'll try if it crashes for smaller models too, like Qwen3.5 35B-A3B, this should speed up more thorough testing.

git bisect log

git bisect start
# status: waiting for both good and bad commits
# bad: [983df142a99b764f5cbe9acb6b9ec862f861d353] convert : fix/suppress pyright errors (#20442)
git bisect bad 983df142a99b764f5cbe9acb6b9ec862f861d353
# status: waiting for good commit(s), bad commit known
# bad: [de190154c85d20e24dbeae8c8af1849402ae5098] New conversations now auto-select the first loaded model (#20403)
git bisect bad de190154c85d20e24dbeae8c8af1849402ae5098
# status: waiting for good commit(s), bad commit known
# bad: [0516e04bf9c002bf190519af6795415ab3ebd059] opencl: use larger workgroup size for get_rows (#20316)
git bisect bad 0516e04bf9c002bf190519af6795415ab3ebd059
# status: waiting for good commit(s), bad commit known
# bad: [9ef7523ee99e13fe57a3da5cd600c206ed997630] cuda/hip: fix loop unrolling in ssm-conv (#20369)
git bisect bad 9ef7523ee99e13fe57a3da5cd600c206ed997630
# status: waiting for good commit(s), bad commit known
# bad: [6fce5c6a7dba6a3e1df0aad1574b78d1a1970621] opencl: add l2_norm (#20160)
git bisect bad 6fce5c6a7dba6a3e1df0aad1574b78d1a1970621
# status: waiting for good commit(s), bad commit known
# bad: [f6235a41efc8fe10b3f85619d46ee86dd50142f7] webui: Agentic Loop + MCP Client with support for Tools, Resources and Prompts (#18655)
git bisect bad f6235a41efc8fe10b3f85619d46ee86dd50142f7
# status: waiting for good commit(s), bad commit known
# bad: [872646b30c342c5fa98efec002f94fdb4f1f1ea5] model : update Qwen3.5 model type detection (#20126)
git bisect bad 872646b30c342c5fa98efec002f94fdb4f1f1ea5
# status: waiting for good commit(s), bad commit known
# bad: [66d65ec29ba7c440cbc31b6f63b74a17b536ba65] cuda: cap grid.y at 65535 in non-contiguous dequantize/convert kernels (#19999)
git bisect bad 66d65ec29ba7c440cbc31b6f63b74a17b536ba65
# status: waiting for good commit(s), bad commit known
# bad: [8f974d2392da4e6fa422a67050e90f1471d72966] mtmd : rename mtmd_get_audio_bitrate to mtmd_get_audio_sample_rate (#20105)
git bisect bad 8f974d2392da4e6fa422a67050e90f1471d72966
# status: waiting for good commit(s), bad commit known
# bad: [c747294b2d70a00a91713abe62fb7890c5893c5c] scripts: update corpus of compare-logprobs (#19326)
git bisect bad c747294b2d70a00a91713abe62fb7890c5893c5c
# status: waiting for good commit(s), bad commit known
# good: [fc0fe4004985d6749a7a05e250d161f9dbe41d65] models : support qwen3.5 series (#19468)
git bisect good fc0fe4004985d6749a7a05e250d161f9dbe41d65
# good: [cc45f2ada695644c6697c0fb0e70a5e95563ad0f] models : deduplicate delta-net graphs for Qwen family (#19597)
git bisect good cc45f2ada695644c6697c0fb0e70a5e95563ad0f
# bad: [237958db339300bdd8028608cc08b2ba2685ec33] model: Add PaddleOCR-VL model support (#18825)
git bisect bad 237958db339300bdd8028608cc08b2ba2685ec33
# bad: [ea003229d30148d0cd268a25e93e57ba51bb354a] Pre-MCP UI and architecture cleanup (#19689)
git bisect bad ea003229d30148d0cd268a25e93e57ba51bb354a
# bad: [e48349a49d55a02785ebf55c5531131c6d90d453] ci : bump komac version (#19682)
git bisect bad e48349a49d55a02785ebf55c5531131c6d90d453
# good: [cceb1b4e33cfd9595b4ac1949f2c0857e43af427] common : inline functions (#18639)
git bisect good cceb1b4e33cfd9595b4ac1949f2c0857e43af427
# good: [05fa625eac5bbdbe88b43f857156c35501421d6e] convert : add JoyAI-LLM-Flash (#19651)
git bisect good 05fa625eac5bbdbe88b43f857156c35501421d6e
# bad: [ae46a61e41be4e33ff7eabb36ddc4a7f43e1d0fb] build : link ws2_32 as PUBLIC on Windows (#19666)
git bisect bad ae46a61e41be4e33ff7eabb36ddc4a7f43e1d0fb
# bad: [65cede7c700a71180216777bbb107fa254ba9f7b] build : cleanup library linking logic (#19665)
git bisect bad 65cede7c700a71180216777bbb107fa254ba9f7b
# first bad commit: [65cede7c700a71180216777bbb107fa254ba9f7b] build : cleanup library linking logic (#19665)

[fkroener]

So it seems to be something model size/memory/vulkan specific.

Using unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q6_K_XL (~30 GB) worked with this context with vulkan.

Models like Sabomako/Qwen3.5-122B-A10B-heretic-GGUF:MXFP4_MOE (~65 GB), AesSedai/Step-3.5-Flash-GGUF:IQ4_XS(~95 GB, crashes at 25k tokens processed already) and the previously tested unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL (~75 GB) all crash reliably on master, using the vulkan backend.

I tested unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL with ROCm without apparent issues, which imho rules out faulty memory.

[fkroener]

Another data point: the crashes appear to be sensitive to ubatch-size / batch-size settings with the vulkan backend. Where Step-3.5-Flash would crash at around 25k tokens pre-fill with batch-size = 2048, ubatch-size = 2048. Using the default (2048 / 512) it made it to 83k tokens before it crashed at 96 % progress. This was with 15-16 GiB memory available according to the system. Reducing ubatch-size further to 256 seems to help even with Step-3.5-Flash with a context around ~83k tokens - but somehow at this point the model turns incoherent, returning reasoning traces like "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<Cases<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<Cat<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<临<<<<<<<<<<<<<<".

Returning to Qwen3.5-122B-A10B I am able to keep and increase the context with the 512 u-batch-size setting without crashing in the last couple of minutes. Overall memory pressure is likely less due to model size (currently sitting at around 32 GiB available with the context read in and 32 checkpoints). Context size is now at 105k+ and no crash so far.

[fala13]

Getting same error on R9700 + 7900XT loading Qwen3.5-122B after few prompts on latest versions. Getting 44 tokens/s with latest, 33 tokens/sec with some previous stable version. Tried all different options and disabling GATED_DELTA_NET, but no luck.

[fkroener]

@fala13 since you appear to use dual gpus, #20462 might be more specific to your setup. Maybe try the patches in #20518 and report your findings in the other bug.

[fala13]

@fala13 since you appear to use dual gpus, #20462 might be more specific to your setup. Maybe try the patches in #20518 and report your findings in the other bug.

yes, #20518 helped for my 2 GPU setup to prevent vk::DeviceLostError

[bonswouar]

Came here from #20462 (comment) as apparently this issue is more suited.

Single (i)GPU, 8845HS / 780M, 32GB of shared memory
Devstral-Small-2-24B-Instruct-2512-UD-Q6_K_XL crashes at ~39k tokens, plus my laptop seems way less responsive than before when running a model (I think I saw another recent maybe-related issue but can't find it anymore)

Here are the params I currently use:

LLAMA_SET_ROWS=1 llama-server --port 5800 --flash-attn 1 --n-gpu-layers 65 --threads -1 --prio 2 --timeout 86400 --threads-http 1 --cache-reuse 1 --slot-save-path /kv-cache/ --cache-ram 2048 --jinja --model /models/Devstral-Small-2-24B-Instruct-2512-UD-Q6_K_XL.gguf --temp 0.15 --min-p 0.01 --cache-type-k q4_0 --cache-type-v q4_0 --ctx-size 85000

I've just tried with smaller ubatch (--ubatch-size 256) and it seems to improve things: the unresponsiveness seems much better, and I've been able to parse up to ~70k tokens without any crash (although I'll need more testing - this isn't a veryyy fast gpu unfortunately)

And I don't have the "bleeding edge" drivers afaik:

• Driver: Mesa 25.3.6
• Api version: 1.4.328
• Linux Kernel: 6.19.8-200