GGUF tensor loader bug: blk.56.ssm_out.weight falsely reports 'overflows file' on valid Qwen3.6-27B Q4_K_M GGUF

[batumilove]

Bug Summary

Lucebox dflash_server fails to load unsloth/Qwen3.6-27B-GGUF (Q4_K_M quantization) with error:

target load: tensor 'blk.56.ssm_out.weight' overflows file
[backend_factory] Qwen35Backend init failed
[server] backend creation failed

The GGUF file is valid — verified with the standard Python gguf library (v0.19.0): all 851 tensors fit perfectly with 0 bytes gap. The tensor blk.56.ssm_out.weight is Q5_K type, shape [6144, 5120], 21,626,880 bytes, at file offset 15,038,670,688. File size is 16,817,244,384 bytes. The tensor ends at 15,060,297,568 — well within the file.

Environment

• GPU: NVIDIA GB10 (DGX Spark), compute capability 12.1, 122,542 MiB VRAM
• CUDA: 13.0, driver 580.126.09
• OS: Ubuntu on ARM64 (aarch64)
• Model: unsloth/Qwen3.6-27B-GGUF → Qwen3.6-27B-Q4_K_M.gguf (16.82 GB, 851 tensors)
• Draft model: dflash-draft-3.6-q4_k_m.gguf (1008 MB)
• Commit: c95dfca (latest main as of May 31, 2026)
• Build: Clean build from source, dflash_server binary (13 MB)

Reproduction

DFLASH27B_KV_TQ3=1 ./server/build/dflash_server server/models/Qwen3.6-27B-Q4_K_M.gguf \
  --draft server/models/draft/dflash-draft-3.6-q4_k_m.gguf \
  --ddtree --ddtree-budget 22 \
  --fa-window 2048 \
  --host 0.0.0.0 --port 8000

Full server log

[server] loading tokenizer from /root/lucebox-hub/server/models/Qwen3.6-27B-Q4_K_M.gguf
[tokenizer] added_tokens: 33 special tokens
[tokenizer] loaded vocab=248320 merges=247587 bos=248044 eos=248046 eot=248046 pre=qwen35 sp=no
[server] creating backend...
[backend_factory] detected arch=qwen35
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 122542 MiB):
  Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes, VRAM: 122542 MiB
target load: tensor 'blk.56.ssm_out.weight' overflows file
[backend_factory] Qwen35Backend init failed
[server] backend creation failed
[loader] eos_id=248046 eos_chat_id=-1

Analysis

The error originates in server/src/qwen35/gguf_target_loader.cpp line ~649-652:

const size_t off = data_start + gguf_get_tensor_offset(gctx, tid);
const size_t sz  = gguf_get_tensor_size(gctx, tid);
if (off + sz > mm.len) {
    set_last_error(std::string("tensor '") + tname + "' overflows file");

gguf_get_tensor_size() calls ggml_nbytes() on the tensor in the meta context. The standard gguf Python library computes n_bytes = 21,626,880 correctly for this Q5_K [6144, 5120] tensor. The Lucebox C++ path appears to compute a larger value via ggml_nbytes, which uses stride-based computation (ne[0]*nb[0]/blck_size + (ne[1]-1)*nb[1]). This suggests the strides (nb[]) are being set incorrectly for SSM tensors in the meta context created by gguf_init_from_file.

Verification

Python verification confirming the GGUF is valid:

from gguf import GGUFReader
reader = GGUFReader("Qwen3.6-27B-Q4_K_M.gguf")
# File: 16,817,244,384 bytes (16.82 GB), 851 tensors
# blk.56.ssm_out.weight: data_offset=15,038,670,688, n_bytes=21,626,880, end=15,060,297,568 — NO overflow
# Last tensor: blk.63.post_attention_norm.weight: end=16,817,244,384 = exact file size (0 bytes gap)

Note

Issue #259 shows the same Qwen3.6-27B Q4_K_M GGUF loading successfully on a Tesla V100 with 850 tensors. Our GGUF from unsloth has 851 tensors — possibly a slightly different conversion that exposes a latent bug in the stride computation for SSM (hybrid Mamba) layers.