Delta weight sync (NCCL) produces NaN weights on Qwen3.5-122B MoE → "probability tensor contains inf/nan" crash

[leofan-i]

Summary

With --update-weight-mode delta --update-weight-transport nccl, the first delta apply on Qwen3.5-122B-A10B (grouped MoE) leaves NaN/inf in the rollout engine's weights. The next rollout crashes in sampling. Full-sync mode (--update-weight-mode full) on the same model/config is stable. Reproduced 4/4 runs.

Symptom

TensorCompare.cu:109 _assert_async_cuda_kernel: Assertion `probability tensor contains either inf, nan or element < 0` failed
Subprocess scheduler_0 crashed with exit code -3. Triggering SIGQUIT for cleanup...

Timeline: initial full seed sync ✓ → step-1 train ✓ → step-2 delta sync applied (checksum passes) ✓ → next rollout crashes.

Config

• slime with delta sync ([1/n] Disaggregated rollout: delta weight sync (disk + NCCL transports) #1806); SGLang v0.5.13 + docker/patch/latest/sglang.patch
• Qwen3.5-122B-A10B: --num-experts 256 --moe-grouped-gemm --moe-token-dispatcher-type alltoall --attention-backend flash (not flashinfer_trtllm)
• --update-weight-encoding indices, per-sync density ~1%
• Reproduced on both TP2/PP2/EP16 and TP2/PP4/EP8 (parallelism-independent)

Notes

• Checksum passes, so the (positions, values) payload arrives intact — corruption is in the apply into grouped-MoE params, not on the wire.
• Possibly related: fix(update_weight): preserve grouped MoE expert axis during GLU rechunk #2193 (grouped-MoE GLU rechunk expert-axis), Delta weight sync: failed engine apply is silently swallowed, leaving sender snapshot ahead of receiver state #2104 (failed apply not surfaced) — but neither reports this crash signature.