# Gemma 4 MTP draft-mtp crashes: `ubatch->embd` is never populated → NaN in `nextn.pre_projection` → RMS-norm assertion (`invalid vector subscript` in release)

[Dmitriy-Romanov]

Summary

Using an unsloth gemma-4-12b-it GGUF together with its mtp-gemma-4-12b-it-Q8_0.gguf drafter via --spec-type draft-mtp, llama-server (and llama-cli) crash during warmup, before any prompt is processed. The crash reproduces identically on:

• CPU backend (MSVC debug build → assert(scale > 0.0f) in ops.cpp:3838)
• Vulkan backend (AMD RDNA4, prebuilt b10068 and b9553 → error loading model: invalid vector subscript)
• Any build version tested (b9553, b10068, current master 571d0d5)

So this is not a backend-specific regression, and not tied to a recent Unsloth re-upload (the MTP weights' SHA256 has been stable since 2026-06-05; only the chat template changed on 2026-07-17).

Name and Version

version: 1 (571d0d5)
built with MSVC 19.44.35228.0 for x64

Reproduced on prebuilt Windows releases b9553 and b10068 (both Vulkan), and on a self-built CPU debug build from 571d0d5.

Operating systems

Windows 11 26220 x64. Also reported on Linux/CUDA in related issues (#24795, #24492, #22337).

Command line

llama-server \
  -m gemma-4-12b-it-UD-Q4_K_XL.gguf \
  --model-draft mtp-gemma-4-12b-it-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  -ngl 99 -ngld 99 -c 16384 -fa on \
  --host 127.0.0.1 --port 8088 --jinja

The same crash happens with -ngl 0 -ngld 0 -c 1024, with/without -fa, with -md instead of --model-draft, with --spec-draft-n-max 2, and with --spec-draft-p-min. Crashing does not depend on flags.

Models

• Main: unsloth/gemma-4-12b-it-GGUF → gemma-4-12b-it-UD-Q4_K_XL.gguf (arch gemma4, 48 layers)
• Draft: unsloth/gemma-4-12b-it-GGUF/MTP/mtp-gemma-4-12b-it-Q8_0.gguf (arch gemma4-assistant, nextn_predict_layers = 4)
  • size 465109248, sha256 145db9094bc0f85f1701e255a2ed216dcc9800fc8bc8631ad00905b456bd451b
  • identical to the current HF copy (verified via LFS pointer). Last weight change: 2026-06-05.

Both files load fine individually; the crash is specifically in the draft-mtp path.

What happens

After both models load successfully and common_specu reports:

common_specu: adding speculative implementation 'draft-mtp'
common_specu: - n_max=4, n_min=0, p_min=0.00, n_embd=3840, backend_sampling=1
common_specu: - gpu_layers=0, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes

the warmup llama_decode(ctx_dft, llama_batch_get_one(tokens)) (common.cpp:1450) runs the draft graph and crashes inside the first RMS-norm after nextn.pre_projection.

Debug build:

0.05.936 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)
Assertion failed: scale > 0.0f, file .../ggml/src/ggml-cpu/ops.cpp, line 3838

Release build (prebuilt Vulkan b10068):

error loading model: invalid vector subscript
error loading model: failed to load draft model

The invalid vector subscript message is MSVC STL's std::vector out-of-bounds assertion triggered downstream after NaN propagation — it is the same crash as scale > 0.0f, just caught later in a release build without assert(). Adding instrumentation to ggml_compute_forward_rms_norm_f32 showed the RMS-norm input (pre_proj, the output of mul_mat(nextn_proj_pre, xh)) is entirely NaN (nnan=1024 for ne00=1024).

Root cause (investigated)

The drafter graph in src/models/gemma4-assistant.cpp builds:

auto inp = std::make_unique<llm_graph_input_embd>(n_embd_backbone);   // n_embd_backbone = 3840
...
inp->embd = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_embd_backbone, ubatch.n_tokens);
res->t_inp_embd = inp->embd;                                         // this is the "inp_h" input
...
ggml_tensor * xh = ggml_concat(ctx0, x, inp_h, 0);                   // [2*3840, n_tokens]
ggml_tensor * cur = ggml_mul_mat(ctx0, model.nextn_proj_pre, xh);    // "pre_proj"

inp_h is meant to carry the target's h_nextn (post-final-norm hidden state). It is populated by llm_graph_input_embd::set_input (src/llama-graph.cpp:66):

void llm_graph_input_embd::set_input(const llama_ubatch * ubatch) {
    if (ubatch->token) {
        ggml_backend_tensor_set(tokens, ubatch->token, ...);
    }
    if (ubatch->embd) {                                              // <-- only filled if ubatch->embd != NULL
        ggml_backend_tensor_set(embd, ubatch->embd, ...);
    }
    // else: embd tensor is left uninitialized
}

Instrumenting this function across a full session (warmup + a real /v1/chat/completions request) shows:

[EMBD COUNTER] null=1 nonnull=0

i.e. ubatch->embd is NULL on every call, including during real generation — not only during warmup. The expected target→draft h_nextn transfer does not happen, so inp_h stays uninitialized memory → NaNs → pre_proj is NaN → RMS-norm asserts.

Contrast with EAGLE3 (common/speculative.cpp:483-486) and DFlash (:968-969), which feed target features into the draft batch via common_batch_add(... embd ...) and a dedicated batch_inject. The draft-mtp path (:1264-1267) allocates batch.embd and writes into it in process() (:1406, :1411) and draft() (:1498, :1587), but those writes are against the common_speculative_impl_draft_mtp::batch member — and the instrumentation above shows ubatch->embd never becomes non-NULL by the time the drafter's set_input runs. So either the batch isn't reaching the draft context's llama_decode, or the batch→ubatch split (llama_batch_allocr / split_simple) is dropping embd for this input class.

Why "it used to work"

Most public "works for me" reports (#24795 "works on b9553", Reddit "2.5x speedup") are CUDA-only, short generations, and none publish a draft acceptance rate. The PR that added this code (#23398, merged 2026-06-07) is recent, and the hidden-state transfer path for gemma4-assistant does not appear to have been validated against a real acceptance benchmark on the Vulkan/CPU path. On the systems where warmup was disabled or short enough, the crash may not have surfaced.

What is NOT the cause

• ❌ Not the Unsloth GGUF — file SHA256 matches HF, metadata is well-formed, both files load standalone.
• ❌ Not a version regression — crashes identically on b9553, b10068, master 571d0d5.
• ❌ Not a backend bug — reproduces on CPU, Vulkan (AMD RDNA4); same symptom reported on CUDA (Eval bug: gemma4-assistant MTP draft model fails to load — "invalid vector subscript" (regression: works on b9553, broken on b9702/b9717) #24795) and Vulkan 31B (Eval bug: Gemma 4 31B MTP (draft-mtp) crashes on Vulkan backend, pre-allocated tensor cannot run operation NONE #24492).
• ❌ Not a llama_batch_init token/embd allocation issue — the batch.token workaround is already present at speculative.cpp:1267.

Minimal reproducer (debug build, ~10 lines)

git clone https://github.com/ggml-org/llama.cpp
cmake -S llama.cpp -B build -G "Visual Studio 17 2022" -A x64 -DLLAMA_BUILD_SERVER=ON
cmake --build build --config Debug --target llama-server
build/bin/Debug/llama-server.exe \
  -m gemma-4-12b-it-UD-Q4_K_XL.gguf \
  --model-draft mtp-gemma-4-12b-it-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -ngl 0 -ngld 0 -c 1024 --host 127.0.0.1 --port 8099
# -> Assertion failed: scale > 0.0f, ops.cpp line 3838

Related

• llama : add Gemma4 MTP #23398 — original "llama: add Gemma4 MTP" PR
• mtp: support for gemma-4 E2B and E4B assistants #24282 — E2B/E4B assistant support
• Eval bug: gemma4-assistant MTP draft model fails to load — "invalid vector subscript" (regression: works on b9553, broken on b9702/b9717) #24795 — same symptom ("invalid vector subscript"), tagged as regression on b9702/b9717 vs b9553 CUDA
• Eval bug: Gemma 4 31B MTP (draft-mtp) crashes on Vulkan backend, pre-allocated tensor cannot run operation NONE #24492 — "Gemma 4 31B MTP crashes on Vulkan"
• Misc. bug: E llama_init_from_model: failed to initialize the context: Gemma4Assistant #24343 — Gemma4Assistant requires ctx_other to be set (benign, but appears in every repro log)
• draft issue Gemma E2B #22337 — original "Gemma E2B/E4B fail to load as draft model: invalid vector subscript"

[jordan1776]

I see this exact same issue too. Hopefully someone finds the reason behind the "invalid vector subscript". For now Qwen seems to work find for me in MTP mode, so there's that.

[poog26]

@jordan1776 It seems to have something to do with partially offloading the model. For me it works fine until I have to partially offload to cpu, or get too close to the vram limit.