CUDA illegal memory access with Qwen3-Next on multi-GPU using -ot (regression)

[nerdpudding]

Name and Version

Last known working: b48e80f67 (b8022, 2026-02-13)
First observed broken: ed4837891 (b8022+93 commits, 2026-02-22, head of master at time of fetch)

Operating systems

Linux (Ubuntu 24.04)

GGML backends

CUDA

Hardware

• GPU 0: NVIDIA RTX 4090 (24 GB VRAM) — sm_89
• GPU 1: NVIDIA RTX 5070 Ti (16 GB VRAM) — sm_120
• CPU: AMD Ryzen 7 5800X3D
• RAM: 64 GB DDR4
• Driver: 580.x (open kernel)
• CUDA: 13.0

Models

• Qwen3-Next-80B-A3B-Instruct UD-Q5_K_XL (from unsloth)
• Qwen3-Coder-Next UD-Q5_K_XL (from unsloth)

Both models exhibit the same behavior.

Problem description & steps to reproduce

When using -ot (per-layer tensor overrides) to split layers across two GPUs with expert offload to CPU, the server crashes with CUDA error: an illegal memory access was encountered on the first generated token after prompt processing.

Key observations:

• Prompt processing completes successfully (confirmed in logs)
• Sampler initialization completes successfully
• Crash occurs at the start of token generation (first decode step)
• Short prompts (~20 tokens) work fine
• Longer prompts (~100+ tokens) consistently crash
• The same configuration and same models work without issues on commit b48e80f67

Server command (via docker-compose, effective flags):

llama-server \
  --model Qwen3-Next-80B-A3B-Instruct-UD-Q5_K_XL-00001-of-00002.gguf \
  --ctx-size 10240 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --jinja -np 1 -b 512 -ub 512 \
  --no-context-shift \
  -ot blk\.([0-9]|1[0-8])\.=CUDA0,blk\.(1[9]|2[0-7])\.=CUDA1,exps=CPU

Steps to reproduce:

1. Load either Qwen3-Next or Qwen3-Coder-Next with the -ot configuration above (layers split across two GPUs, experts on CPU)
2. Send a chat completion request with a prompt of ~100+ tokens
3. Server crashes during first token generation

What I tested:

• Reverted to b48e80f67 → works perfectly, no CUDA errors
• Tried different -ot layer splits (both production and bench profiles) → same crash on new code
• Tested both Qwen3-Next and Qwen3-Coder-Next → both crash identically
• Short prompts (~20 tokens) succeed even on the broken build

Possibly related

This resembles #18580, which reported the same error (CUDA illegal memory access with Qwen3-Next on multi-GPU) and was fixed by #18593 (merged in b7625). The fix is included in our last working commit, so this appears to be a new regression of a similar issue rather than the original bug.

First Bad Commit

Not bisected. There are 93 commits between the last known good (b48e80f67) and the broken head. I have not narrowed it down further.

Relevant log output

Logs

Server startup (loads fine):

load_tensors: offloading 49/49 layers to GPU
load_tensors:   CPU_Mapped model buffer size = 47115.28 MiB
load_tensors:   CPU_Mapped model buffer size =  6669.17 MiB
load_tensors:        CUDA0 model buffer size = 21203.75 MiB
load_tensors:        CUDA1 model buffer size = 10666.26 MiB

Crash (repeats on every request with 100+ token prompts):

slot update_slots: id  0 | task 0 | prompt processing progress, n_tokens = 162, batch.n_tokens = 162, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_tokens = 162, batch.n_tokens = 162
slot init_sampler: id  0 | task 0 | init sampler, took 0.02 ms, tokens: text = 162, total = 162
/build/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
CUDA error: an illegal memory access was encountered

The server restarts (docker restart: unless-stopped), reloads the model, and crashes again on the next request. Observed 22 CUDA errors and 11 model reloads before stopping the container.

[ggerganov]

It would be best to bisect and find the exact commit that stops working.

[nerdpudding]

Bisected. The first bad commit is 1725e31 — PR #19375 (models : optimize qwen3next graph).

1725e316c1a780759ec134ca5a2999f4d53ce273 is the first bad commit
commit 1725e316c1a780759ec134ca5a2999f4d53ce273
Author: Georgi Gerganov <ggerganov@gmail.com>
Date:   Sat Feb 14 12:57:36 2026 +0200

    models : optimize qwen3next graph (#19375)

Tested on Qwen3-Next-80B-A3B UD-Q5_K_XL with multi-GPU -ot splits (layers across CUDA0/CUDA1, experts on CPU). The commit right before it (b7742cf) works fine, this one crashes with the CUDA illegal memory access on first decode.

I noticed the last line of the commit message mentions "use prefix to disable CUDA graphs" — it seems that mitigation doesn't fully cover the multi-GPU -ot case.

[nerdpudding]

Tested the latest (ed4837891) with GGML_CUDA_DISABLE_GRAPHS=1 — same crash. So the issue isn't CUDA graph capture; the graph rewrite itself appears to not work correctly with multi-GPU -ot splits.

For now I'm staying on the last working commit (b48e80f67). Happy to test patches or specific commits if that helps.

[ggerganov]

Does this patch fix it?

diff --git a/src/models/delta-net-base.cpp b/src/models/delta-net-base.cpp
index 99f1fdd95..ccd03be50 100644
--- a/src/models/delta-net-base.cpp
+++ b/src/models/delta-net-base.cpp
@@ -259,7 +259,7 @@ std::pair<ggml_tensor *, ggml_tensor *> llm_build_delta_net_base::build_delta_ne
         ggml_tensor * o_ch = ggml_add(ctx0, attn_inter, v_attn);
         cb(o_ch, "dnet_add_ch_attn_out", il);
 
-        v = ggml_set_inplace(ctx0, v, o_ch, v->nb[1], v->nb[2], v->nb[3], chunk * v->nb[2]);
+        v = ggml_set(ctx0, v, o_ch, v->nb[1], v->nb[2], v->nb[3], chunk * v->nb[2]);
 
         // kgdmulvnew = (key_gdiff).transpose(-1, -2) @ v_new
         // TODO: head broadcast might not work here - probably will need a transpose

[nerdpudding]

Confirmed — the ggml_set_inplace → ggml_set fix resolves the issue. Tested on latest (ed4837891) with multi-GPU -ot splits, long prompts work fine now. Thanks for the quick turnaround!

[ggerganov]

Can you confirm that replacing -ot blk\.([0-9]|1[0-8])\.=CUDA0,blk\.(1[9]|2[0-7])\.=CUDA1,exps=CPU with just -ot exps=CPU also works, even without the patch above?

[nerdpudding]

Tested with just -ot exps=CPU (no explicit per-layer GPU splits) on the unpatched code — works fine, no crash.

Here's what I've tested so far:

• -ot blk.0-18=CUDA0,blk.19-27=CUDA1,exps=CPU without patch: crash (CUDA illegal memory access)
• -ot blk.0-18=CUDA0,blk.19-27=CUDA1,exps=CPU with ggml_set patch: works, ~29.5 t/s
• -ot exps=CPU with --fit on, without patch: works, ~21.7 t/s

So the crash is specifically triggered by explicit per-layer multi-GPU assignment (-ot blk.X=CUDA0,blk.Y=CUDA1), not by -ot exps=CPU alone.

The reason I use explicit per-layer splits is to maximize GPU utilization on my asymmetric dual-GPU setup (RTX 4090 24GB + RTX 5070 Ti 16GB). With explicit splits: ~21 GB on CUDA0 + ~10 GB on CUDA1, giving ~29.5 t/s. Without them, --fit on auto-places very conservatively — only ~1.6 GB on the GPUs and ~53 GB on CPU — resulting in ~21.7 t/s (~27% slower).

[ggerganov]

I think the proper way to split the GPU load is using the -ts arguments and avoid -ot for CUDA devices.

The original -ot expression is problematic because it imposes conflicting requirements with the default -ngl N -ts ... arguments. It's not ideal to crash, but also I am not sure it warrants changes - it's more an invalid user configuration.

Can you try to achieve the same performance just with -ngl 99 -ts 21,10 -ot exps=CPU?

[pwilkin]

@nerdpudding try just --fit without -ot exps=CPU, should work fine if you specify the context size.

[nerdpudding]

The earlier tests (with and without patch) were at 10K context with -ot blk.0-18=CUDA0,blk.19-27=CUDA1,exps=CPU (~29.5 t/s).

In practice I run at 262K context with a different split that I tuned to fit: -ot blk.0-16=CUDA0,blk.17-24=CUDA1,exps=CPU (~28 t/s, ~21 GB on CUDA0, ~10 GB on CUDA1).

I tried your suggestion with -ngl 99 -ts 21,10 -ot exps=CPU at 262K context. It doesn't crash, but the GPUs are barely used:

• CUDA0 model buffer: 959 MiB (of 24 GB)
• CUDA1 model buffer: 647 MiB (of 16 GB)
• CPU: 53,784 MiB
• Speed: ~19.8 t/s

It looks like -ot exps=CPU moves the expert weights to CPU first, and -ts then only distributes the remaining ~1.6 GB of non-expert data across GPUs.

I'd like to use -ts properly, but I'm not sure how to calculate the right split to get the same GPU utilization as with the explicit -ot layer assignments. Any guidance?

[nerdpudding]

@pwilkin Tried --fit on without -ot exps=CPU at 262K context (--ctx-size 262144 --n-gpu-layers 99 --fit on --jinja -np 1 --no-context-shift) — OOM. The model is ~53 GB, my GPUs are 24 GB + 16 GB = 40 GB total. FIT tries to allocate 36 GB on CUDA0 and fails. It also won't reduce n_gpu_layers because it considers it user-set.

Without offloading the expert weights (~47 GB) to CPU, this model simply doesn't fit. That's why I use -ot exps=CPU combined with explicit per-layer GPU assignments — it's the only way I've found to keep the non-expert parts on GPU for decent speed (~28 t/s at 262K context) while offloading experts to CPU/RAM.

[pwilkin]

@nerdpudding Yeah, don't specify --n-gpu-layers. Only specify --ctx-size.

[nerdpudding]

Solved — this was a configuration issue on my end. I run llama-server via Docker, and I had --n-gpu-layers 99 hardcoded as a default in both my Dockerfile and docker-compose.yml. My config file generates the .env for docker-compose, but when a profile didn't specify N_GPU_LAYERS, the hardcoded 99 was used as fallback. I assumed omitting it would let --fit decide, but the Docker defaults kicked in first, so --fit always saw it as "user set" and refused to adjust.

Once I changed the default to auto, --fit handles everything correctly — no -ot needed at all, and the ggml_set_inplace patch is no longer needed either since I'm not using -ot for GPU device assignments anymore.

I also found that --fit-target per device matters for tight-fit models. My setup has a dedicated RTX 4090 (nothing else running) and an RTX 5070 Ti (shares with OS/display). Using --fit-target 128,1024 instead of the default 1024,1024 lets FIT use more of the 4090's VRAM.

Final results — all models faster than my previous hand-tuned -ot configurations:

• GLM Q4 (single GPU): 147 t/s (was ~140)
• GLM Q8 (dual GPU): 112 t/s (was ~105), 5 graph splits (was 33)
• GPT-OSS 120B: 22 t/s (was ~21)
• Qwen3-Coder-Next: 33 t/s (was ~28)
• Qwen3-Next: 33 t/s (was ~27)

Thanks @ggerganov and @pwilkin — I learned a lot from this.

[JohannesGaessler]

If it is possible at all to provoke an "illegal memory access" CUDA error that is still indicative of a bug that needs to be fixed.