Skip to content

feat(ggml): add ROCmFPX quant support for DS4#36

Closed
davide221 wants to merge 7 commits into
perf/luce-verify-kernelsfrom
codex/rocmfpx-ds4-strix
Closed

feat(ggml): add ROCmFPX quant support for DS4#36
davide221 wants to merge 7 commits into
perf/luce-verify-kernelsfrom
codex/rocmfpx-ds4-strix

Conversation

@davide221

@davide221 davide221 commented Jul 6, 2026

Copy link
Copy Markdown

Summary

  • add ROCmFP4/ROCmFPX ggml quant types and CPU reference conversions
  • wire ROCmFPX CUDA/HIP dequant, copy, getrows, MMVQ vecdot, MMVF, unary, and FA paths
  • preserve existing luce verify-kernel optimizations: graph stats, LUCE_Q8_MEMO, LUCE_MMVQ_MAX_NCOLS, and richer CUDA op diagnostics
  • add DS4 SWIGLU parameter plumbing for fused FFN matvec paths

Used By

Verification

  • git diff --check
  • source audit confirmed ROCmFPX type coverage in ggml type registry, CUDA/HIP dequant/copy/getrows/MMVQ paths, and preserved ac06 perf hooks
  • lucebox2 HIP release build succeeded as the .NET LLaMA.cpp library! 🦙 ggml-org/llama.cpp#494 submodule stack: cmake --build build-hip-pr494 -j 8 --target dflash_server
  • ROCmFPX DS4 model loaded from /home/lucebox2/models/DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf under the updated hub build
  • n_gen=64 warm benchmark with DFLASH_DS4_FFN_RAW_MMID=1 and DFLASH_DS4_HC_DIRECT_NO_SYNC=1: decode tok/s aggregate 21.85, range 16.16 - 23.17

Notes

  • Draft while reviewers inspect the imported ROCmFPX format code and decide whether to keep all enum values/ftypes exposed in ggml.

davide221 added 4 commits July 7, 2026 01:07
Adds ds4-hc kernels (pre/post/out modes) used by the opt-in
DFLASH_DS4_FUSED_DECODE single-graph decode path on DeepSeek V4 Flash.
The fn/fn_out matvecs stay on ggml mul_mat; the op fuses sigmoid gates,
sinkhorn normalization and stream mixing. CPU fallback included; the op
is inert unless emitted by the server.
@davide221 davide221 force-pushed the codex/rocmfpx-ds4-strix branch from b8390df to 7c3a6cb Compare July 7, 2026 22:44
davide221 added 2 commits July 8, 2026 11:15
The generic sinkhorn's runtime-bound loops forced its c[] matrix into
scratch (private, VRAM-backed) memory, so the serial thread-0 sinkhorn
cost ~97us per launch (measured on gfx1151; 86 launches per decoded
token ~ 8ms/token). A fully-unrolled template<NHC=4> variant keeps c[]
in registers: 5.3us per launch. Also grid over n_embd instead of a
single block and stage mix/base through shared memory.

DS4-Flash fused decode on Strix gfx1151: 19.1 -> 23.0 tok/s.
@davide221

Copy link
Copy Markdown
Author

Superseded by Luce-Org/lucebox-hub#502, which ports this ROCmFPX ggml work into the hub's vendored server/deps/llama.cpp tree (ggml is no longer a submodule after lucebox-hub#486). The port is layered on current hub main so main-side ggml work is preserved, adds two review fixes (ftype-switch -Wswitch coverage; FP6 vecdot over-read), and was verified on lucebox2 (clean HIP build, decode 22.8-22.9 tok/s).

@davide221 davide221 closed this Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA documentation Improvements or additions to documentation ggml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant