SYCL: MUL_MAT_ID prefill path produces wrong results on Intel Arc Pro B70 (Battlemage G31) — causes garbage output on MoE models

[malakhovskiy]

Description

test-backend-ops -b SYCL0 -o MUL_MAT_ID fails 28/792 cases on Intel Arc Pro B70 (Battlemage G31), all in the non-decode (prefill/batched, ne12 > 1) path of ggml_sycl_mul_mat_id(). Errors are large numerical mismatches (0.3–1.9, tolerance is 0.0005), not close-call rounding.

I originally hit this while debugging garbage/gibberish output from a real MoE model (Qwen3.6-35B-A3B) running on the SYCL backend — the same model on the Vulkan backend of the same GPU produces fully correct, coherent output at similar speed, so the corruption is SYCL-specific.

Investigation so far

• Confirmed not related to any of GGML_SYCL_ENABLE_OPT, GGML_SYCL_ENABLE_VMM, GGML_SYCL_ENABLE_FUSION, GGML_SYCL_USE_ASYNC_MEM_OP — disabling each individually (and all together) does not change the failure set.
• Suspected the counting-sort refactor from [SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 (mmid_counting_sort_rows / current k_copy_src1_to_contiguous) — tested by temporarily reverting ggml_sycl_mul_mat_id()'s ne12 > 1 branch to the pre-[SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 atomic-scan implementation (old k_copy_src1_to_contiguous/k_copy_dst_from_contiguous with dev_cur_src1_row atomic counter). Same failure set reproduces, and in fact gets worse — several cases that were "just" large numerical errors under the new code produce outright NaN/inf under the old code. This points to a bug in a lower-level shared path (e.g. ggml_sycl_op_mul_mat/ggml_sycl_op_mul_mat_sycl when fed the row-gathered/contiguous tensor shapes used by mul_mat_id), not in the [SYCL] improve MoE prefill throughput (+70% with Qwen3.6-35B) #23142 refactor itself.
• Already has SYCL: fix use-after-free bug with async memcpy in MoE prefill #24676 (use-after-free fix) and sycl: support reordered Q4_K/Q5_K/Q6_K MoE MUL_MAT_ID #24452 (reordered Q4_K/Q5_K/Q6_K MoE support) merged — neither affects this.
• Decode path (ne12 == 1, ggml_sycl_mul_mat_id_mmvq_fused) is unaffected — only the prefill/batched path with more than one token per sequence.

Failing cases (764/792 pass; 28 fail)

MUL_MAT_ID(type_a=bf16,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=50,n=200,k=64)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=1,m=50,n=200,k=64)
MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=f32,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=3)
MUL_MAT_ID(type_a=iq1_m,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq1_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_xs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq2_xxs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq3_s,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq3_xxs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=iq4_nl,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=iq4_xs,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=mxfp4,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q1_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=384)
MUL_MAT_ID(type_a=q2_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q3_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q4_1,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q4_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q5_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q5_1,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)
MUL_MAT_ID(type_a=q5_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q6_K,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=768)
MUL_MAT_ID(type_a=q8_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96)

Sample error magnitudes:

[MUL_MAT_ID] ERR = 1.564748538 > 0.000500000   MUL_MAT_ID(type_a=q4_0,type_b=f32,n_mats=4,n_used=2,b=0,m=64,n=16,k=96): FAIL
[MUL_MAT_ID] ERR = 0.327848188 > 0.000500000   MUL_MAT_ID(type_a=f16,type_b=f32,n_mats=16,n_used=16,b=0,m=50,n=200,k=64): FAIL

Steps to reproduce

llama-bench --list-devices   # confirm SYCL0 loaded
test-backend-ops test -b SYCL0 -o MUL_MAT_ID

Environment

• GPU: Intel Arc Pro B70 (Battlemage G31)
• Level Zero driver: 1.14.36605
• oneAPI: 2026.1.0 (icx 2026.1.0 build 20260617)
• Build flags: -DGGML_SYCL=ON -DGGML_SYCL_F16=ON -DGGML_SYCL_DEVICE_ARCH=bmg-g31 (AOT via spir64_gen)
• OS: Windows 11
• All SYCL runtime env vars at default (GGML_SYCL_ENABLE_OPT=1, GGML_SYCL_ENABLE_VMM=1, GGML_SYCL_ENABLE_FUSION=1, GGML_SYCL_USE_ASYNC_MEM_OP=1) — disabling any/all of these does not change the result.

Other backends (CUDA, Vulkan, CPU) on the same build pass all MUL_MAT_ID tests cleanly.