Skip to content

perf(cuda): recover qwen dense decode path#27

Open
dusterbloom wants to merge 1 commit into
Luce-Org:luce-dflashfrom
dusterbloom:perf/qwen-dense-decode-recovery
Open

perf(cuda): recover qwen dense decode path#27
dusterbloom wants to merge 1 commit into
Luce-Org:luce-dflashfrom
dusterbloom:perf/qwen-dense-decode-recovery

Conversation

@dusterbloom

@dusterbloom dusterbloom commented Jul 1, 2026

Copy link
Copy Markdown

What

Split out the second concern that used to ride in #24.

This PR keeps the dense decode recovery work independent of the in-place GDN state-write API:

  • adds the thread-local CUDA graph property-check skip hook (ggml_cuda_set_skip_props_check)
  • makes GDN intermediate-state writes a compile-time WRITE_INTER specialization so pure AR can
    compile out that store path
  • disables the grouped-columns GDN path on Ampere by default, with DFLASH_GDN_FORCE_GROUPED_COLS /
    DFLASH_GDN_NO_GROUPED_COLS overrides

Relationship to #24

#24 is now only the in-place final-state write API/kernel path.

This branch is based directly on luce-dflash, so reviewers can inspect it as a separate concern.
The Lucebox consumer can combine #24 and this PR to recover the local dense AR replay fast path
without mixing unrelated ggml changes in one PR.

Validation

Built both split branches locally with CUDA 12.6 and explicit SM86 because this environment cannot
probe CUDA_ARCHITECTURES=native without a visible GPU:

  • env CUDACXX=/usr/local/cuda/bin/nvcc cmake -S /tmp/ggml-pr24-inplace -B /tmp/ggml-pr24-inplace/ build-cuda126-sm86 -DGGML_CUDA=ON -DGGML_CCACHE=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
  • cmake --build /tmp/ggml-pr24-inplace/build-cuda126-sm86 --target ggml-cuda -j 8
  • env CUDACXX=/usr/local/cuda/bin/nvcc cmake -S /tmp/ggml-pr24-props-independent -B /tmp/ggml-pr24- props-independent/build-cuda126-sm86 -DGGML_CUDA=ON -DGGML_CCACHE=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86
  • cmake --build /tmp/ggml-pr24-props-independent/build-cuda126-sm86 --target ggml-cuda -j 8

@dusterbloom dusterbloom marked this pull request as ready for review July 1, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant