Add JACCL expert-parallel distributed inference

[machiabeli]

Summary

Expert-parallel distributed inference for DeepSeek V4 Flash/Pro via JACCL
(Apple's RDMA collective library for Thunderbolt 5). Each rank owns a subset
of the 256 MoE experts; after each layer's routed MoE forward pass,
jaccl_group_all_sum() merges partial results across ranks.

Opt-in: make JACCL=1. Requires JACCL headers from ml-explore/mlx.

What changed (1,559 lines across 17 files)

Core integration (ds4.c, ds4.h, ds4_cli.c, ds4_server.c):

• --distributed CLI flag enables JACCL init at engine startup
• Expert range [start, end) partitioned by rank; 256/N experts per rank
• all_sum inserted after every routed MoE output (CPU decode, CPU batch, Metal graph)
• GPU sync (ds4_gpu_end_commands) before all_sum to ensure Metal kernel completion

C shim (jaccl_shim.h, jaccl_shim.cpp):

• 46-line C wrapper around JACCL's C++ Group API (all_sum, barrier, send, recv)
• Statically linked via libjaccl.a (built by CMake in build/jaccl/)

Metal kernels (metal/expert_mask.metal, metal/moe.metal, ds4_metal.m):

• expert_mask: zero router weights for non-owned experts
• expert_compact: rewrite selected[] to owned-only (env-gated DS4_EXPERT_COMPACT)
• Blit-zero mid buffer before swiglu to prevent stale GPU memory reads
• Zero mid[] in swiglu early-exit path (IQ2_XXS + Q4_K kernel variants)

Build (Makefile):

• JACCL=1 opt-in flag; JACCL_SRC defaults to mlx submodule path
• CMake builds libjaccl.a; C++ shim links against it

Tools:

• distributed_launch.sh: multi-node launcher with asmi topology discovery
• tests/test_distributed_correctness.sh: automated 2-node correctness test
• tests/test_jaccl_shim.c: shim linkage smoke test
• docs/distributed-benchmark.md: methodology and baseline results

Key design decisions

1. Expert-parallel, not layer-parallel. All ranks run all 61 layers; only MoE
   experts are split. The all_sum payload is ~28KB per layer (n_embd x float32),
   which completes in <0.3ms at 5.5 GB/s RDMA bandwidth.

2. GPU sync before RDMA read. Metal GPU kernels are asynchronous. Without
   ds4_gpu_end_commands() before reading routed_out, each rank transmits
   partially-computed data. This was the root cause of the garbage output bug.

3. Defense-in-depth for expert masking. Three layers: (A) blit-zero the mid
   buffer, (B) zero mid[] in kernel early-exit, (C) expert_compact rewrites
   selected[] to owned-only. Any one suffices; all three prevent regressions.

4. Env var device matrix. JACCL_IBV_DEVICES points to a JSON file (not
   inline JSON) containing the NxN RDMA device connectivity matrix.

Test plan

• Single-node regression: correct output at 36.7 tok/s
• 2-node hub+m3u4: correct output ("Paris") on both ranks at 14.5 tok/s
• 4-node full mesh: 256/4 = 64 experts per rank
• DeepSeek V4 Pro Q4 (384 experts, 838GB GGUF)
• Sustained generation benchmark (100+ tokens)