Distributed CUDA worker host-registers the whole GGUF, not just its layer slice → Q4 OOM on 128GB DGX Spark

[ngt1999]

Summary

A distributed CUDA worker host-registers the ENTIRE gguf mapping for device
access instead of only its assigned --layers slice. A Q4 worker therefore needs
~150GB and fails on a 128GB machine, even though its slice is only ~78GB.
Q2 (~81GB whole file) fits and runs fine.

Setup

• Coordinator: MacBook Pro M5 Max 128GB (Metal), --layers 0:20
• Worker: NVIDIA DGX Spark (GB10, sm_121) 128GB (CUDA), --layers 21:output
• 5GbE link; route forms fine ("complete route ready")
• Same commit on both; q4-imatrix / q2-imatrix from download_model.sh

Q4 (fails)

ds4: restricting cuda model map to layers 21:output (48 spans, 78.21 GiB tensor span)
ds4: CUDA host registration skipped: out of memory
ds4: cuda backend initialized for graph diagnostics
(coordinator) prompt processing failed: cuda layer-slice failed

Q2 (works)

ds4: restricting cuda model map to layers 21:output (48 spans, 41.08 GiB tensor span)
ds4: CUDA registered 80.76 GiB model mapping for device access
prefill: 9.36 t/s, generation: 16.59 t/s

Likely cause

"registered 80.76 GiB" ≈ the whole Q2 file while the slice is only 41 GiB, so
registration covers the full mmap, not the slice. For Q4 the whole file (~150GB)
exceeds the 128GB worker, so cudaHostRegister OOMs and the slice can't run.
Registering only the slice's pages (~78GB) would fit and let Q4 run distributed,
which is the whole point of the layer split. GB10 has coherent unified memory,
so full host pinning may be avoidable on this platform entirely.

[hexxyan]

Hi @ngt1999 — I opened PR #317 that addresses the full-GGUF registration issue you reported.

Root cause: ds4_gpu_set_model_map_spans() was passing the full file size to every fallback path. Even though the span calculation was correct, all downstream code (cuda_model_range_ptr, cuda_model_range_is_cached, cuda_model_direct_fallback_ptr, Q2 tail expansion) assumed the full mmap was accessible — which is why Q2 "worked" (80.76 GiB registered ≈ whole file) but Q4 OOMed (150 GiB > 128 GB).

Fix: Added g_model_span_cover_offset/end to track the actual page-aligned span cover, and a cuda_model_range_in_span_cover() helper that gates all direct/full-file fallback paths. The new log line makes it immediately visible:

ds4: CUDA span cover 78.21 GiB of 150.00 GiB model (48 spans, offset 72.00 GiB)

CUDA compile verified with nvcc -arch=sm_90 (zero warnings/errors).

If you can test on your DGX Spark:

# Option 1: with gh CLI
gh pr checkout 317 --repo antirez/ds4

# Option 2: without gh
git fetch https://github.com/hexxyan/ds4.git fix/cuda-distributed-span-cover:pr-317
git checkout pr-317

# Build for GB10
make cuda CUDA_ARCH=sm_121
# or: make cuda-spark

# Test with ds4-server (distributed worker)
./ds4-server --layers 21:output --distributed worker ... -m q4-imatrix.gguf

# Or test with ds4 directly
./ds4 --layers 21:output --distributed worker ... -m q4-imatrix.gguf

The key things to check:

1. Does the new "CUDA span cover" log line appear and show a reasonable size (should be ~78 GiB, not ~150 GiB)?
2. Does Q4 distributed mode start without OOM?
3. Does inference actually work (no GPU faults from out-of-span access)?

Thanks for the detailed bug report — the exact log output + root cause analysis made this much easier to track down.

[ngt1999]

Tested PR #317 on real hardware — works! 🎉

Setup: M5 Max 128GB (Metal coordinator, --layers 0:20) +
DGX Spark GB10/sm_121 128GB (CUDA worker, --layers 21:output), 5GbE.
Built both from pr-317 (DGX: make cuda CUDA_ARCH=sm_121).

Worker now logs the new span-cover line and starts without OOM:

ds4: CUDA span cover 82.44 GiB of 153.33 GiB model (48 spans, offset 70.89 GiB)
ds4: CUDA preparing model tensor mappings: 78.21 GiB
ds4: CUDA startup model preparation covered 78.21 GiB of tensor spans in 20.201s

(No more "host registration skipped: out of memory".)

Q4 distributed works end-to-end — coherent output, no GPU faults:
prefill: 11.59 t/s, generation: 15.74 t/s (short prompt).

All three checks pass. Thanks for the quick fix!

[hexxyan]

Thanks @ngt1999 for the hardware validation! 🎉

The fix is confirmed working on real DGX Spark GB10 hardware — Q4 distributed mode no longer OOMs, span cover correctly logs 82.44 GiB of 153.33 GiB, and inference produces coherent output at 15.74 t/s.

@antirez — since the fix is now verified on real hardware, could you review PR #317 and consider merging? The change is scoped and well-contained:

• Root cause: ds4_gpu_set_model_map_spans() registered the full GGUF mmap instead of the layer slice
• Fix: Added cuda_model_compute_span_cover() + cuda_model_range_in_span_cover() to compute a tight page-aligned cover and gate all 5 fallback paths (Q2 tail expansion, direct ptr, HMM, fd cache, chunked copy)
• New global state: g_model_span_only, g_model_span_cover_offset, g_model_span_cover_end — all reset in ds4_gpu_cleanup() and cuda_model_set_host_map()
• No behavior change for non-distributed mode (span-only flag is never set)

Thanks!

[ngt1999]

Glad it helped, and thanks @hexxyan for the quick fix!

A few more data points from the DGX Spark (GB10) + M5 Max setup — plus
one follow-up that is separate from #317 and not a blocker for merging.

Distributed bench @ 8192 ctx, split 0:20 (Metal coordinator) /
21:output (CUDA worker), 5GbE:

quant	prefill	generation
Q2	513 t/s	12.7 t/s
Q4	136 t/s	12.5 t/s

Decode is ~equal (bandwidth-bound), but Q4 prefill is ~3.8x slower.
The cause looks like the fp16 weight-cache cap: the Q4 worker logs

CUDA q8 fp16 cache limit reached; using q8 kernels (cached=3.94 GiB limit=4.00 GiB)

and falls back to q8 kernels for prefill, while Q2 (41 GiB slice) never
hits the cap. The span-cover fix is NOT the cause — Q2 uses span cover
too and stays fast. On a 128 GB DGX the 78 GB Q4 slice + KV leaves little
headroom, so the fallback may be hard to avoid, but there is currently no
flag to raise the cap (--help only exposes --metal/--cuda/--cpu/--backend).

Happy to open a separate issue for this prefill / fp16-cache behavior so
it doesn't clutter #317 — just let me know which you'd prefer.

[ngt1999]

Follow-up isolation — it is NOT inherent quant cost:

Same 4096-token chunk, same worker layers (21:42), 8192 ctx:

quant	worker eval / 4096 tok	coordinator (Metal) local
Q2	~5.5s	8.60s
Q4	~27.5s	8.63s

The Metal coordinator runs Q2 and Q4 prefill at the SAME speed (~8.6s) —
prefill is quant-independent there. Only the CUDA worker shows Q4 ~5x
slower than Q2, exactly where the "q8 fp16 cache limit reached; using q8
kernels" fallback triggers (Q4 only). So the slowdown is specific to the
CUDA Q4 path, not inherent quantization.

[ngt1999]

Full bench after main update (prefill + 32 gen, 8192 ctx):
prefill 207 t/s, generation 12.55 t/s.
Prefill improved 136 -> 207 (+52%) vs the pre-PRO-Q4-rework build; the
Metal coordinator's local time is unchanged (~8.5s), so the speedup is
entirely CUDA-side. Decode unchanged (~12.5 t/s, bandwidth-bound). The
q8 fp16 cache fallback still logs, so the ~2.5x prefill gap vs Q2
(513 t/s) remains, but the fallback kernels are clearly faster now.
OOM fully resolved on stock main.