llama: optimize RWKV7 inference by fusing some graph operators

[MollySophia]

Summary

This PR optimizes RWKV7 inference by reducing graph-level operator overhead
across CUDA and Vulkan, with CPU fallback coverage for the new RWKV-specific
operators.

Although RWKV is architecturally more linear/recurrent than Qwen3.5-style
hybrid models, the current ggml graph for RWKV7 expands into many small
elementwise and reduction operators around WKV. In decode, these small ops
create significant launch/dispatch overhead. This PR makes the main RWKV7 hot
paths explicit ggml ops so backends do not need to rediscover RWKV semantics
through fragile graph-pattern fusion.

Changes

• Specialize RWKV7 WKV decode paths.

  • CUDA: optimized WKV7 decode kernel.
  • Vulkan: added specialized wkv7_t1 decode shader.

• Add explicit RWKV time-mix lerp op:

  • RWKV_LERP: cur + (x_prev - cur) * weight
  • CUDA and Vulkan call the existing specialized kernels directly.
  • CPU has a F32 fallback with a contiguous RWKV fast path.

• Fuse CUDA RWKV key-update pattern:

  • k + (a * ka - ka)

• Add explicit RWKV7 r_k correction op:

  • RWKV_RK: cur + reshape(v * sum_rows((k * r) * r_k))
  • CUDA: added standalone fused kernel.
  • Vulkan: added standalone rwkv_rk.comp shader.

• Work around a Vulkan CI correctness failure by disabling the specialized
  RWKV7 T=1 subgroup shader on Intel proprietary Windows. This is a separate
  commit from the RWKV op changes so it can be reverted independently if the
  driver-specific issue is resolved.

Performance

Hardware:

• GPU: NVIDIA GeForce RTX 5090
• CPU: AMD Ryzen 9 9950X
• CPU thread count: 16
• Baseline commit: 0eca4d490

RWKV7 1.5B

GPU model: rwkv7-g1g-1.5b-20260526-ctx8192-FP16.gguf

CPU model: rwkv7-g1g-1.5b-20260526-ctx8192-q4_0.gguf

Prefill

Backend	Model	Test	Baseline	Current	Delta
CUDA	RWKV7 1.5B F16	pp512	23092.43 +/- 1706.81	25148.17 +/- 2650.54	+8.9%
Vulkan	RWKV7 1.5B F16	pp512	16885.02 +/- 37.00	20720.94 +/- 47.62	+22.7%
CPU 16t	RWKV7 1.5B Q4_0	pp512	1112.15 +/- 2.48	1192.53 +/- 7.59	+7.2%

Decode

Backend	Model	Test	Baseline	Current	Delta
CUDA	RWKV7 1.5B F16	tg128	282.29 +/- 1.52	322.79 +/- 1.29	+14.3%
Vulkan	RWKV7 1.5B F16	tg128	244.56 +/- 0.60	287.48 +/- 0.97	+17.5%
CPU 16t	RWKV7 1.5B Q4_0	tg128	64.47 +/- 0.51	65.54 +/- 0.51	+1.7%

RWKV7 7.2B

Model: rwkv7-g1g-7.2b-20260523-ctx8192-F16.gguf

Prefill

Backend	Model	Test	Baseline	Current	Delta
CUDA	RWKV7 7.2B F16	pp512	8190.77 +/- 562.32	9336.82 +/- 648.37	+14.0%
Vulkan	RWKV7 7.2B F16	pp512	7289.55 +/- 11.08	8464.32 +/- 5.15	+16.1%

Decode

Backend	Model	Test	Baseline	Current	Delta
CUDA	RWKV7 7.2B F16	tg128	90.32 +/- 0.19	95.76 +/- 0.13	+6.0%
CUDA	RWKV7 7.2B F16	tg512	90.51 +/- 0.03	95.91 +/- 0.03	+6.0%
Vulkan	RWKV7 7.2B F16	tg128	83.84 +/- 0.01	90.10 +/- 0.02	+7.5%
Vulkan	RWKV7 7.2B F16	tg512	83.95 +/- 0.05	90.26 +/- 0.04	+7.5%

Intel Arc Pro B50 / SYCL

Models:

• rwkv7-g1g-1.5b-20260526-ctx8192-FP16.gguf
• rwkv7-g1g-2.9b-20260526-ctx8192-FP16.gguf
• rwkv7-g1g-7.2b-20260523-ctx8192-F16.gguf

Prefill

Backend	Model	Test	Baseline	Current	Delta
SYCL	RWKV7 1.5B F16	pp512	1650.40 +/- 3.24	1814.28 +/- 0.34	+9.9%
SYCL	RWKV7 2.9B F16	pp512	843.10 +/- 0.36	914.79 +/- 0.58	+8.5%
SYCL	RWKV7 7.2B F16	pp512	379.63 +/- 0.82	399.85 +/- 0.30	+5.3%

Decode

Backend	Model	Test	Baseline	Current	Delta
SYCL	RWKV7 1.5B F16	tg128	49.77 +/- 0.00	51.49 +/- 0.02	+3.5%
SYCL	RWKV7 2.9B F16	tg128	28.77 +/- 0.03	29.52 +/- 0.00	+2.6%
SYCL	RWKV7 7.2B F16	tg128	11.49 +/- 0.01	11.63 +/- 0.01	+1.2%

Validation

• Backend op smoke tests:
  • test-backend-ops test -o RWKV_WKV7 -b CUDA0 -j 1
  • test-backend-ops test -o RWKV_WKV7 -b Vulkan0 -j 1
• WikiText-2 perplexity is unchanged within noise:
  • 1.5B F16: 9.5497 -> 9.5503
  • 7.2B F16: 6.4011 -> 6.4013

Notes

The largest gains come from reducing small graph operators and backend launches
around RWKV decode. On larger models, decode gains are smaller because mat-vec
work dominates more strongly, but both CUDA and Vulkan still show consistent
positive improvements.

TODO

• [] SYCL/Metal fused kernels

[MollySophia]

Hi @MollySophia, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.
• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Regard PR flags:

1 & 2: This PR makes changes on the existing RWKV7 path, rather than support a new model or feature. The WKV7 op has some semantic changes so that all the supported backends need to be modified at once.

[Copilot]

Pull request overview

This PR targets RWKV7 inference performance by making RWKV7 decode/time-mix semantics explicit in ggml (new ops + fused kernels/shaders), reducing graph-level operator count and backend launch/dispatch overhead across CUDA, Vulkan, SYCL, Metal, and CPU.

Changes:

• Introduce explicit RWKV ops (GGML_OP_RWKV_LERP, GGML_OP_RWKV_RK) and route RWKV7 model graph construction through them.
• Specialize/fuse RWKV7 hot paths across backends (CUDA kernels, Vulkan shaders/pipelines, SYCL kernels, Metal kernel updates) and adjust RWKV7 WKV7 parameterization (kk, a) consistently across backends.
• Add/extend backend-side graph fusions (e.g., CUDA key-update pattern; Vulkan/CPU add-mul and norm-mul-add fusions) and expand backend op test coverage.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test-backend-ops.cpp	Adds backend-op tests for RWKV_LERP and RWKV_RK; updates RWKV_WKV7 test inputs to new signature.
src/models/rwkv7-base.cpp	Uses new ggml RWKV ops (lerp/rk) and updates RWKV7 WKV7 call to revised inputs.
ggml/src/ggml.c	Adds new ggml ops (RWKV_LERP/RWKV_RK); updates RWKV_WKV7 op signature and metadata strings.
ggml/include/ggml.h	Exposes new RWKV APIs and updates RWKV_WKV7 function signature in the public header.
ggml/include/ggml-rpc.h	Bumps RPC patch version and updates op-count static_assert to match new ops.
ggml/src/ggml-backend-meta.cpp	Marks new RWKV ops as supported for meta backend split-state handling.
ggml/src/ggml-cpu/ops.h	Declares CPU implementations for RWKV_LERP/RWKV_RK and new fused CPU paths.
ggml/src/ggml-cpu/ops.cpp	Implements CPU RWKV_LERP/RWKV_RK plus fused ADD+MUL and NORM+MUL+ADD compute paths.
ggml/src/ggml-cpu/ggml-cpu.c	Wires new RWKV ops into CPU execution and adds CPU-side fusion detection.
ggml/src/ggml-cuda/wkv.cu	Updates CUDA RWKV_WKV7 kernel signature and adds a specialized T=1 decode kernel path.
ggml/src/ggml-cuda/rwkv.cuh	Declares CUDA fused RWKV helper ops (lerp, rk, and elementwise fusions).
ggml/src/ggml-cuda/rwkv.cu	Implements CUDA fused RWKV ops (lerp/rk) and elementwise fusion kernels.
ggml/src/ggml-cuda/norm.cuh	Declares fused CUDA norm entry points (norm+mul, norm+mul+add).
ggml/src/ggml-cuda/norm.cu	Extends CUDA norm kernel to optionally fuse mul/add, and adds fused wrapper functions.
ggml/src/ggml-cuda/ggml-cuda.cu	Dispatches new RWKV ops on CUDA and adds new fusion patterns (incl. RWKV key-update, add-mul, norm fusions).
ggml/src/ggml-vulkan/vulkan-shaders/wkv7.comp	Updates Vulkan WKV7 shader to revised (kk, a) math and inline w transform.
ggml/src/ggml-vulkan/vulkan-shaders/wkv7_t1.comp	Adds a Vulkan subgroup-based specialized WKV7 T=1 decode shader.
ggml/src/ggml-vulkan/vulkan-shaders/rwkv_rk.comp	Adds Vulkan compute shader for fused RWKV_RK.
ggml/src/ggml-vulkan/vulkan-shaders/lerp.comp	Adds Vulkan shader for RWKV_LERP.
ggml/src/ggml-vulkan/vulkan-shaders/add_mul.comp	Adds Vulkan shader for fused (a+b)*scale pattern.
ggml/src/ggml-vulkan/vulkan-shaders/norm_mul_add.comp	Adds Vulkan shader for fused norm*mul+add.
ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp	Registers new Vulkan shaders for SPIR-V generation.
ggml/src/ggml-vulkan/ggml-vulkan.cpp	Adds Vulkan pipelines/dispatch for new RWKV ops, adds WKV7 T=1 pipeline, and implements new fusion paths.
ggml/src/ggml-sycl/wkv.cpp	Updates SYCL WKV7 implementation to revised (kk, a) math and inline w transform.
ggml/src/ggml-sycl/rwkv.hpp	Declares SYCL RWKV_LERP and RWKV_RK ops.
ggml/src/ggml-sycl/rwkv.cpp	Implements SYCL RWKV_LERP and RWKV_RK kernels.
ggml/src/ggml-sycl/backend.hpp	Includes the new SYCL RWKV header into the backend.
ggml/src/ggml-sycl/ggml-sycl.cpp	Wires RWKV_LERP and RWKV_RK into SYCL compute and device support checks.
ggml/src/ggml-metal/ggml-metal.metal	Updates Metal WKV7 kernel signature and math to match revised (kk, a) formulation and inline w transform.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[am17an]

I don't think adding new ops for this architecture is worth it IMO.

[MollySophia]

I don't think adding new ops for this architecture is worth it IMO.

I tried to add some fusing rules in the first version, but I didn't find an really elegant way by myself either. Any suggestions on doing fusion instead of adding ops?

[am17an]

I'm trying to figure out a way to add ops without adding a lot of maintenance burden #24646, perhaps wait for some kind of resolution on that. I agree that fusion is not the right way for complicated patterns

[MollySophia]

I'm trying to figure out a way to add ops without adding a lot of maintenance burden #24646, perhaps wait for some kind of resolution on that. I agree that fusion is not the right way for complicated patterns

That would be nice! I'll wait for that and avoid these ops. They are indeed not really meaningful enough.