diff --git a/AMPERE-EXTENSION.md b/AMPERE-EXTENSION.md new file mode 100644 index 000000000..184b2fdf0 --- /dev/null +++ b/AMPERE-EXTENSION.md @@ -0,0 +1,36 @@ +# Ampere Extension (sm_80 / sm_86) + +The knowledge base is Blackwell/Hopper-first. This extension makes the same pages actionable on Ampere hardware: A100 (sm_80) and GA10x consumer/workstation cards (sm_86: RTX 3090/3080, RTX A6000, A40). It was written against a 4x RTX 3090 rig, and numbers marked "measured" come from that machine. + +## Design + +All changes are **additive** (new files + vocabulary entries) so upstream merges stay trivial: + +- `data/tags.yaml` — added architectures `sm80`, `sm86`; hardware features `cp-async`, `mma-sync`, `l2-persistence`. +- `data/aliases.yaml` — added alias groups for `sm80`, `sm86`, `cp-async`, `mma-sync`, `l2-persistence`. +- `sources/docs/` — 4 new source pages: `doc-ampere-tuning-guide`, `doc-ga102-whitepaper`, `doc-ptx-isa-ampere`, `doc-cutlass-ampere`. +- `wiki/hardware/` — 3 new pages: `hw-cp-async`, `hw-mma-sync-ampere`, `hw-ampere-memory-model`. +- `wiki/migration/` — 1 new page: `migration-hopper-to-ampere` (the entry point: instruction replacement table, capacity re-planning, scheduling paradigm, ncu checklist). +- `references/primer.md` — "Ampere Extension" section + alias cheat-sheet rows. +- `SKILL.md` — trigger description now includes Ampere/SM86/RTX 3090 and backport questions. + +The upstream validator rules are untouched and still pass: Ampere pages don't trip the Blackwell-first rule (it only constrains Hopper-only pages), and every new page carries the full required frontmatter for its type. + +## Query examples + +```bash +python3 scripts/query.py --architecture sm86 --compact # aliases: "RTX 3090", GA102 … +python3 scripts/query.py --tag cp-async --type hardware +python3 scripts/get_page.py migration-hopper-to-ampere +python3 scripts/grep_wiki.py "wait_group" --only wiki +``` + +## Verification status + +Snippets on the new pages are compiled against `nvcc -arch=sm_86` on a real 4x RTX 3090 machine (see per-page `reproducibility`). Facts sourced from: NVIDIA Ampere tuning guide (CC 8.0/8.6 table), GA102 whitepaper v2.1 (tensor throughput incl. the FP32-accumulate half-rate), PTX ISA (instruction availability), CUTLASS (SM80 mainloop idiom). + +## Candidate follow-ups (not done) + +- `source-pr` pages for canonical Ampere kernels in tracked repos (e.g. vLLM Marlin W4A16, exllama kernels) — would upgrade wiki-page confidence to `verified` (needs an `upstream-code` source). +- `kernel-` case-study pages with benchmarked claims from the 4x3090 rig (FlashAttention-2, Marlin, Triton GEMM autotune points). +- sm_89 (Ada) column where it differs (FP8 tensor cores present, SMEM 100 KB like sm_86). diff --git a/README.md b/README.md index aa99ef8d9..9dbbd29c7 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,8 @@ > **Last repository update: 2026-06-09.** Information after this date is not included in KernelWiki yet. +> **Downstream branch note:** this clone carries an additive **Ampere (sm_80/sm_86) extension** — see [AMPERE-EXTENSION.md](AMPERE-EXTENSION.md). Entry point: `migration-hopper-to-ampere`. + A structured knowledge base of NVIDIA Blackwell (SM100, B200) and Hopper (SM90, H100) GPU kernel optimization, packaged as a Claude Code skill. The repository root **is** the skill directory — clone it directly into `~/.claude/skills/` and it works out of the box. ## Install as a Claude Code Skill diff --git a/SKILL.md b/SKILL.md index 0ec65d34b..2e09a96f6 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,6 +1,6 @@ --- name: KernelWiki -description: Use when the user asks about optimizing NVIDIA Blackwell (SM100, B200) or Hopper (SM90, H100) GPU kernels — tcgen05/TMEM/CLC/NVFP4/2-SM cooperative, warp specialization, FlashAttention-4, DeepGEMM, FlashMLA, MoE, grouped GEMM, CuTe-DSL/PTX/Triton on Blackwell, or wants concrete PR references from CUTLASS/SGLang/vLLM/FlashInfer/PyTorch. Do NOT use for generic CUDA Q&A that is not Blackwell/Hopper-specific, host-side framework integration, or distributed systems (DeepEP/EPLB/DualPipe). +description: Use when the user asks about optimizing NVIDIA Blackwell (SM100, B200), Hopper (SM90, H100), or Ampere (SM80/SM86, A100, RTX 3090) GPU kernels — tcgen05/TMEM/CLC/NVFP4/2-SM cooperative, warp specialization, FlashAttention-4, DeepGEMM, FlashMLA, MoE, grouped GEMM, CuTe-DSL/PTX/Triton on Blackwell, cp.async/mma.sync/ldmatrix on Ampere, backporting Hopper/Blackwell kernel designs to sm_86 rigs, or wants concrete PR references from CUTLASS/SGLang/vLLM/FlashInfer/PyTorch. Do NOT use for generic CUDA Q&A that is not architecture-specific, host-side framework integration, or distributed systems (DeepEP/EPLB/DualPipe). argument-hint: "[natural-language-question] | [--tag foo --type kernel] | [page-id]" allowed-tools: "Bash Read Grep Glob" --- @@ -18,6 +18,7 @@ Trigger this skill when the user asks about: - **Performance patterns** — low SM utilization, memory-bound, register pressure, compute-bound, tail effects, pipeline stalls - **DSLs for Blackwell** — CuTe DSL, CUDA C++ with PTX inline, Triton on Blackwell - **Hopper → Blackwell migration** — wgmma → tcgen05, register → TMEM accumulators +- **Ampere (SM80/SM86) kernel programming and backports** — cp.async pipelines, mma.sync/ldmatrix, GA10x quirks (99 KB SMEM, FP32-accumulate half-rate), porting Hopper/Blackwell designs to A100/RTX 3090 rigs — start at `migration-hopper-to-ampere` - **PR references** — "how did vLLM/SGLang/FlashInfer/CUTLASS/PyTorch implement X for SM100?" - **Competition solutions** — GPU Mode NVFP4 hackathon, FlashInfer MLSys 2026 submissions diff --git a/data/aliases.yaml b/data/aliases.yaml index f2f754e92..c85c6b5cd 100644 --- a/data/aliases.yaml +++ b/data/aliases.yaml @@ -52,6 +52,30 @@ wgmma: - "wgmma.mma_async" - WGMMA +cp-async: + - "cp.async" + - LDGSTS + - "async copy" + - "cp.async.ca" + - "cp.async.cg" + - "cuda::memcpy_async" + - memcpy_async + +mma-sync: + - "mma.sync" + - HMMA + - m16n8k16 + - m16n8k8 + - "mma.sync.aligned" + - "warp MMA" + +l2-persistence: + - "L2 persistence" + - cudaAccessPolicyWindow + - "persisting L2 cache" + - "L2 set-aside" + - "L2 residency control" + block-scale: - "block scaling" - "block-scaled" @@ -77,6 +101,25 @@ sm90: - H800 - "SM90" +sm80: + - Ampere + - ampere + - A100 + - A800 + - GA100 + - "SM80" + +sm86: + - GA102 + - "RTX 3090" + - "RTX 3090 Ti" + - "3090" + - "RTX 3080" + - "RTX A6000" + - A40 + - A10 + - "SM86" + # Kernel types moe: - MoE diff --git a/data/tags.yaml b/data/tags.yaml index 114146f0a..d99982ede 100644 --- a/data/tags.yaml +++ b/data/tags.yaml @@ -8,6 +8,9 @@ architectures: - sm90 - sm90a - sm120 + # Ampere extension (backport target: A100 / GA10x consumer cards) + - sm80 + - sm86 hardware_features: - tcgen05 @@ -27,6 +30,10 @@ hardware_features: - mbarrier - ldmatrix - stmatrix + # Ampere extension + - cp-async + - mma-sync + - l2-persistence techniques: - warp-specialization diff --git a/queries/by-hardware-feature.md b/queries/by-hardware-feature.md index 988772a5a..05a7860e2 100644 --- a/queries/by-hardware-feature.md +++ b/queries/by-hardware-feature.md @@ -8,14 +8,18 @@ | `block-scale` | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [NVFP4 Format Details](../sources/blogs/nvfp4-format-details.md), [NVFP4 GEMV and Improved NVFP4 GEMV](../sources/blogs/simon-nvfp4-gemv.md), [TFLOPS Gap: Why FP4 MoE Kernel Engineering Matters on Blackwell](../sources/blogs/tflops-gap-fp4-moe.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Triton 3.6.0 Release Notes — Blackwell (SM100) Lowering](../sources/docs/triton-3.6-blackwell.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [[TRTLLM-9457][feat] Add cute dsl fp8 gemm for Blackwell](../sources/prs/TensorRT-LLM/PR-10130.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [[CuTeDSL] Fix: SM100 block-scale gemm overlapping accumulator](../sources/prs/cutlass/PR-2995.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [feat: enable deepgemm jit for fp8 block-scale on SM90](../sources/prs/flashinfer/PR-1969.md), [feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe](../sources/prs/flashinfer/PR-2217.md), [fix: trtllm_mxint4_block_scale_moe unit test to index output list](../sources/prs/flashinfer/PR-2627.md), [int16 Block-Scaled State and Stochastic Rounding for SSU (mamba)](../sources/prs/flashinfer/PR-2645.md), [Support in-place update for `trtllm_fp8_block_scale_moe`](../sources/prs/flashinfer/PR-2739.md), [fix: int32 overflow in `trtllm_fp4_block_scale_moe` causing "Unsupported hidden state scale shape" for EP32+ configs](../sources/prs/flashinfer/PR-2853.md), [fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120](../sources/prs/flashinfer/PR-2898.md), [fix: Fix autotuner crash on meta-device tensor in trtllm_fp4_block_scale_routed_moe](../sources/prs/flashinfer/PR-2916.md), [Only swizzle on v block scale; rename kv_block_scales to kv_cache_sf](../sources/prs/flashinfer/PR-2954.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [[Perf] Eliminate the slice op for Flashinfer `trtllm_fp4_block_scale_moe`](../sources/prs/sglang/PR-15731.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature](../sources/prs/vllm/PR-14968.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Bugfix] Handle num_expert_group=None in flashinfer block-scale FP8 MoE](../sources/prs/vllm/PR-34494.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md), [Fine-Grained FP8/FP4 Quantization](../wiki/techniques/fine-grained-quantization.md) | | `clc` | [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [CUTLASS Cluster Launch Control (CLC) Documentation](../sources/docs/cutlass-clc-documentation.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [Support for GEMM-K=0 for Blackwell Grouped GEMMs](../sources/prs/cutlass/PR-2746.md), [[Cute-DSL] Add option for issue_clc_query without multicast](../sources/prs/cutlass/PR-3021.md), [[Ai-assisted] CLC work stealing](../sources/prs/flash-attention/PR-2218.md), [Add CLC scheduler heuristic](../sources/prs/flash-attention/PR-2455.md), [[V1][P/D]P2pNcclConnector supports flashinfer](../sources/prs/vllm/PR-23536.md), [Cluster Launch Control (CLC)](../wiki/hardware/clc.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Low SM Utilization](../wiki/patterns/low-sm-utilization.md), [MoE Expert Load Imbalance](../wiki/patterns/moe-load-imbalance.md), [Tail Effect — Last Wave Underutilization](../wiki/patterns/tail-effect.md), [Persistent Kernels with CLC](../wiki/techniques/persistent-kernels.md), [Tile Scheduling Strategies](../wiki/techniques/tile-scheduling.md) | | `cluster` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Qwen3-Next: Hybrid GDN+MoE Architecture on NVIDIA Blackwell](../sources/blogs/qwen3-next-architecture.md), [CUTLASS Cluster Launch Control (CLC) Documentation](../sources/docs/cutlass-clc-documentation.md), [Two-SM Cooperative MMA](../wiki/hardware/2sm-cooperative.md) | +| `cp-async` | [CUTLASS Ampere (SM80/SM86) Support: MmaMultistage, CuTe SM80 Atoms, 3xTF32](../sources/docs/cutlass-ampere-support.md), [NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6)](../sources/docs/nvidia-ampere-tuning-guide.md), [PTX ISA Ampere (sm_80/sm_86) Instructions Reference](../sources/docs/nvidia-ptx-isa-ampere.md), [Ampere Memory Model: SMEM/L2/Occupancy on sm_80 vs sm_86](../wiki/hardware/ampere-memory-model.md), [cp.async — Asynchronous Global→Shared Copy (Ampere)](../wiki/hardware/cp-async.md), [Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86)](../wiki/migration/hopper-to-ampere-backport.md) | | `fp4` | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Anatomy of a Reward Hack](../sources/blogs/gpu-mode-reward-hack.md), [NVFP4 Format Details](../sources/blogs/nvfp4-format-details.md), [NVFP4 GEMV and Improved NVFP4 GEMV](../sources/blogs/simon-nvfp4-gemv.md), [TFLOPS Gap: Why FP4 MoE Kernel Engineering Matters on Blackwell](../sources/blogs/tflops-gap-fp4-moe.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [[TRTLLM-9661][chore] Further reduce tuning time for cuteDSL nvFP4 dense gemm.](../sources/prs/TensorRT-LLM/PR-10339.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11165.md), [[None][feat] Optimize super-v3 nvfp4 for better perf](../sources/prs/TensorRT-LLM/PR-11273.md), [[None][feat] Optimize by fuse nvfp4_quant to layernorm_gated for mamba2_mixer](../sources/prs/TensorRT-LLM/PR-11473.md), [[None][feat] Add support for expert_number<=2048 and K<=32](../sources/prs/TensorRT-LLM/PR-11510.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11733.md), [[None][fix] Fix SM120 issue for rms_norm with nvfp4_quant_fusion](../sources/prs/TensorRT-LLM/PR-11774.md), [[None][feat] Support update weight for nvfp4](../sources/prs/TensorRT-LLM/PR-12320.md), [[None][feat] Update rms_norm + fp4_qaunt kernel supporting more dim](../sources/prs/TensorRT-LLM/PR-13033.md), [[None][feat] Add FP4 residual quantization kernel without channel reo…](../sources/prs/TensorRT-LLM/PR-13117.md), [[None][feat] Integrate FP4 indexer for DSA on Blackwell](../sources/prs/TensorRT-LLM/PR-13340.md), [[TRTLLM-11127][feat] add W4A8_MXFP4_FP8 MoE unit test support](../sources/prs/TensorRT-LLM/PR-13401.md), [[TRTLLM-12316][feat] Integrate FP4 indexer for DSv4](../sources/prs/TensorRT-LLM/PR-13575.md), [[TRTLLM-12503][feat] Parallel VAE independent scaling and fix arg passing](../sources/prs/TensorRT-LLM/PR-13873.md), [[TRTLLM-35237][feat] Add cute dsl FP4 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13929.md), [[https://nvbugs/6025177][test] rcca tests using kimi k2.5 fp4](../sources/prs/TensorRT-LLM/PR-14172.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[TRTLLM-8535][feat] Support DeepSeek V3.2 with FP8 + BF16 KV cache/NVFP4 + BF16 KV cache](../sources/prs/TensorRT-LLM/PR-8405.md), [[None][feat] Enable nvfp4 cuda core for sm120](../sources/prs/TensorRT-LLM/PR-8620.md), [[None][feat] Update TRTLLM MoE cubins; reduce mxfp4 weight padding requirement; tighten TMA bound](../sources/prs/TensorRT-LLM/PR-9025.md), [[None][feat] add fp4 gemm + allreduce](../sources/prs/TensorRT-LLM/PR-9729.md), [[None][feat] Port fp4 quantization kernel optimization from FlashInfer](../sources/prs/TensorRT-LLM/PR-9854.md), [bugfix: Fix test and output shape of fp4 quantize](../sources/prs/flashinfer/PR-1114.md), [Add fp4 quantization swizzling tests](../sources/prs/flashinfer/PR-1157.md), [Expose fp4 blockscale swizzling kernel](../sources/prs/flashinfer/PR-1176.md), [Feature/sm100 low latency nvfp4 kernels](../sources/prs/flashinfer/PR-1214.md), [feat: Support MXFP8 x MXFP4 CUTLASS grouped GEMM](../sources/prs/flashinfer/PR-1241.md), [Reduce the JIT compilation time of gen_gemm_sm100_module](../sources/prs/flashinfer/PR-1251.md), [add mm_fp4 use cudnn backend](../sources/prs/flashinfer/PR-1288.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [add cutlass backend for mm_fp4](../sources/prs/flashinfer/PR-1296.md), [Refactor Fused Moe Module](../sources/prs/flashinfer/PR-1309.md), [Support loading autotuned results from json for cutlass fp4 moe backends](../sources/prs/flashinfer/PR-1310.md), [feat: support output nvfp4 in trtllm-gen function call.](../sources/prs/flashinfer/PR-1318.md), [feat: masked layout fp4 gemm using cute-dsl](../sources/prs/flashinfer/PR-1331.md), [add torch float4_e2m1fn_x2 check for cudnn fp4 backend](../sources/prs/flashinfer/PR-1333.md), [[Fix] remove torch 2.8 requirement for FP4 GEMM](../sources/prs/flashinfer/PR-1334.md), [feature: add fp4 mm using trtllm backend](../sources/prs/flashinfer/PR-1355.md), [hotfix: update mxfp4 groupwise-scaled gemm unittests](../sources/prs/flashinfer/PR-1359.md), [support trtllm-gen prefill fp4 output](../sources/prs/flashinfer/PR-1360.md), [Update autotune results for the nvfp4 cutlass moe backends for v0.2.9](../sources/prs/flashinfer/PR-1361.md), [Support scale factor start index for fp4 mha prefill/decode](../sources/prs/flashinfer/PR-1363.md), [bugfix: fixed cutlass fused moe usage of FP4QuantizationSFLayout::SWIZZLED](../sources/prs/flashinfer/PR-1371.md), [bugfix: Add guard for fp4/fp8 related include headers](../sources/prs/flashinfer/PR-1376.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [Add Mxfp4 trtllm-gen moe unit tests](../sources/prs/flashinfer/PR-1399.md), [feature: enable cublas for fp4 gemm when cudnn == 9.11.1 or >= 9.13](../sources/prs/flashinfer/PR-1405.md), [Faster weight processing (moe nvfp4)](../sources/prs/flashinfer/PR-1412.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [Fix TRTLLM NVFP4-out attention kernel scale factor dim issue](../sources/prs/flashinfer/PR-1460.md), [tuner: Trtllm-gen Fp4 MoE Autotunner](../sources/prs/flashinfer/PR-1475.md), [fix missing enable_pdl argument in trtllm-gen fp4 moe](../sources/prs/flashinfer/PR-1480.md), [Add python API for masked grouped gemm](../sources/prs/flashinfer/PR-1481.md), [fix: update masked moe gemm fp4 tensor reshape](../sources/prs/flashinfer/PR-1495.md), [feat: scaling at fp4 gemm epilogue](../sources/prs/flashinfer/PR-1498.md), [refactor fp4 masked gemm cute-dsl implementation and add manual cache](../sources/prs/flashinfer/PR-1521.md), [Add GeGLU support to trtllm-gen NVFP4 Fused MoE Kernel](../sources/prs/flashinfer/PR-1525.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [fix: separate out fp4 lib into sm90 and sm100 versions, add oob checking in fused moe](../sources/prs/flashinfer/PR-1565.md), [update trtllm-gen fp4 autotuner and routing](../sources/prs/flashinfer/PR-1573.md), [bugfix: Fix test_fp4_quantize test bug](../sources/prs/flashinfer/PR-1585.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [feat: cutlass fp4 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1609.md), [bugfix: fix fp4 quantization with 8x4 scale factor layout](../sources/prs/flashinfer/PR-1611.md), [feat: add support of fp4_batched_quantize](../sources/prs/flashinfer/PR-1633.md), [Added mx_fp4 support using the cudnn backend](../sources/prs/flashinfer/PR-1644.md), [[Hotfix] `test_fp4_quantize.py` failure on sm103](../sources/prs/flashinfer/PR-1666.md), [feat: Benchmark mm_fp4 mxfp4 support and gemm autotune support. Restore mm_fp4 API behavior](../sources/prs/flashinfer/PR-1706.md), [test: skip the unsupported test cases for sm120/121](../sources/prs/flashinfer/PR-1710.md), [Added xfail for mx_fp4 matmul on SM120](../sources/prs/flashinfer/PR-1766.md), [Masked batch nvfp4 quantization](../sources/prs/flashinfer/PR-1774.md), [Support checks PoC](../sources/prs/flashinfer/PR-1809.md), [fix: fp4 moe on sm120](../sources/prs/flashinfer/PR-1817.md), [[Quantization] Add per-expert global scaling factor for fp4 batched quantize](../sources/prs/flashinfer/PR-1835.md), [raise error for group_gemm_fp8_nt_groupwise then num_groups > 1 on sm120/121](../sources/prs/flashinfer/PR-1862.md), [feat: Add FP4 TRTLLM-Gen throughput MOE batched gemms](../sources/prs/flashinfer/PR-1882.md), [silu_and_mul nvfp4 quanization fusion rework](../sources/prs/flashinfer/PR-1927.md), [fix: Add cutlass as an mm_fp4 backend in compute capability 12.0 in benchmark code](../sources/prs/flashinfer/PR-1959.md), [feat: Add backend='auto' to mm_fp4 and enable autotune for backend='cudnn'](../sources/prs/flashinfer/PR-1979.md), [Feature: Support non-gated activation in cutlass fused MoE nvfp4](../sources/prs/flashinfer/PR-2011.md), [fix: Enable SM121 for mm_fp4](../sources/prs/flashinfer/PR-2012.md), [[DSV3] Optimized Router Gemm](../sources/prs/flashinfer/PR-2019.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [perf: Speed up fp4 quantization for small batch with swizzling for cutlass MoE](../sources/prs/flashinfer/PR-2025.md), [[BUG] Fix trtllm-gen fp4 moe renormalize routing](../sources/prs/flashinfer/PR-2049.md), [Patch sm103 for 3xfp4 moe generation](../sources/prs/flashinfer/PR-2082.md), [perf: enable pdl for cutlass fp4 gemm](../sources/prs/flashinfer/PR-2095.md), [feat: MxInt4 x Bf16 TRT-LLM Gen MoE support](../sources/prs/flashinfer/PR-2159.md), [Add data type check for deepseek fp4 moe](../sources/prs/flashinfer/PR-2165.md), [feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe](../sources/prs/flashinfer/PR-2217.md), [feat: Fused RMSNorm + FP4 Quantization Kernels in CuTe-DSL](../sources/prs/flashinfer/PR-2233.md), [fix: Add global scale support and optional output allocation for RMSNorm+FP4Quant fusion kernels](../sources/prs/flashinfer/PR-2260.md), [[performance]optimize for nvfp4](../sources/prs/flashinfer/PR-2268.md), [[WIP] Refactor: simplify torch -> cute-dsl boilerplate and enable tvm-ffi for cute-dsl kernels](../sources/prs/flashinfer/PR-2279.md), [[Perf][Feature] Add SM103-specific schedulers for NVFP4 CUTLASS kernels](../sources/prs/flashinfer/PR-2303.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [Optimize quantization function in large problem size](../sources/prs/flashinfer/PR-2343.md), [fix: In-place Residual Update for add_rmsnorm_fp4quant](../sources/prs/flashinfer/PR-2385.md), [feat: Add output_both_sf_layouts option to add_rmsnorm_fp4quant API](../sources/prs/flashinfer/PR-2395.md), [feat: cuteDSL fp4 moe for better DSR1 performance.](../sources/prs/flashinfer/PR-2398.md), [perf: mm_fp4 heuristic prioritizes CUTLASS over cuDNN on SM103](../sources/prs/flashinfer/PR-2404.md), [refactor: simplify fp4 rmsnorm](../sources/prs/flashinfer/PR-2421.md), [refactor: refactoring cuda code to cute-dsl (part 1)](../sources/prs/flashinfer/PR-2428.md), [Add cute-dsl backends to mxfp[8,4]_quantization for future refactor](../sources/prs/flashinfer/PR-2443.md), [perf: add fp4 GEMM tile configs and streamK scheduler for SM120](../sources/prs/flashinfer/PR-2460.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [Support NVFP4 KV cache decode on SM120](../sources/prs/flashinfer/PR-2520.md), [feat: cute dsl mmfp4 for blackwell](../sources/prs/flashinfer/PR-2540.md), [[Bugfix][comm] Fix FP4 one-shot launch config instability in trtllm_allreduce_fusion](../sources/prs/flashinfer/PR-2557.md), [[Bug] Fix spark unit test failures for test_add_rmsnorm_fp4_quant_cute_dsl](../sources/prs/flashinfer/PR-2573.md), [tests: add bias testing to nvfp4 moe](../sources/prs/flashinfer/PR-2585.md), [fix: cute dsl nvfp4 moe routing index error](../sources/prs/flashinfer/PR-2629.md), [fix: add SM121 support to SM120 version guards](../sources/prs/flashinfer/PR-2631.md), [benchmark: Add MXFP4/MXFP8 quantization mode support to FP4 MoE benchmark](../sources/prs/flashinfer/PR-2635.md), [Enable sm120f compilation](../sources/prs/flashinfer/PR-2650.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [feat: support mxfp4 & mxfp8 entrypoint for blackwell cutedsl dense gemm](../sources/prs/flashinfer/PR-2660.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [Add NVFP4 KV cache quantization support for SM100](../sources/prs/flashinfer/PR-2702.md), [fix: Add SM120 (RTX Blackwell desktop) support for NVFP4 MoE kernels](../sources/prs/flashinfer/PR-2725.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [feat: Add FP4 KV cache quant/dequant kernels ](../sources/prs/flashinfer/PR-2757.md), [feat: Add CuTe-DSL backend for NVFP4 quantization](../sources/prs/flashinfer/PR-2838.md), [fix: int32 overflow in `trtllm_fp4_block_scale_moe` causing "Unsupported hidden state scale shape" for EP32+ configs](../sources/prs/flashinfer/PR-2853.md), [fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120](../sources/prs/flashinfer/PR-2898.md), [perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels](../sources/prs/flashinfer/PR-2904.md), [fix: Fix autotuner crash on meta-device tensor in trtllm_fp4_block_scale_routed_moe](../sources/prs/flashinfer/PR-2916.md), [CuTe DSL FP4 GEMM Heuristic](../sources/prs/flashinfer/PR-2940.md), [Only swizzle on v block scale; rename kv_block_scales to kv_cache_sf](../sources/prs/flashinfer/PR-2954.md), [[Fmha] support nvfp4 output keepsMmaAb generation kernels](../sources/prs/flashinfer/PR-2988.md), [ Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] ](../sources/prs/flashinfer/PR-2994.md), [feat: add PDL support to rmsnorm_fp4quant and add_rmsnorm_fp4quant CuTe DSL kernels](../sources/prs/flashinfer/PR-3008.md), [perf: Port TRT-LLM SM120/SM121 FP4 CUTLASS GEMM optimizations. Add PDL](../sources/prs/flashinfer/PR-3026.md), [[feat] Trtllm-gen Per-token Nvfp4 MoE](../sources/prs/flashinfer/PR-3027.md), [feat: Add backend="b12x" for mm_fp4 on SM120](../sources/prs/flashinfer/PR-3051.md), [feat: Add b12x CuTe DSL fused MoE for SM120](../sources/prs/flashinfer/PR-3066.md), [Support NVFP4 KV for prefill and batch attention kernels](../sources/prs/flashinfer/PR-3097.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [perf: optimize per-token nvfp4 quantization kernel.](../sources/prs/flashinfer/PR-3237.md), [feat(moe): add SM120 W4A16 b12x kernels](../sources/prs/flashinfer/PR-3271.md), [feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA](../sources/prs/sglang/PR-10078.md), [Optimize nvfp4 block scaled gemm kernel when M is small.](../sources/prs/sglang/PR-10101.md), [Fix chunked prefix cache for nvfp4](../sources/prs/sglang/PR-10180.md), [Fix correction bias undefined behavior for nvfp4 models](../sources/prs/sglang/PR-10426.md), [feat: add dsv3 fp4 cutlass moe etp ut](../sources/prs/sglang/PR-10433.md), [Fix bias handling in TritonMoeQuantInfo within quantization/mxfp4.py](../sources/prs/sglang/PR-10579.md), [Fix MTP MoE weight loading with NVFP4 target model.](../sources/prs/sglang/PR-10758.md), [[NVIDIA] Add new SMs support for Spark & Thor](../sources/prs/sglang/PR-11287.md), [Support running FP4 Deepseek on SM120.](../sources/prs/sglang/PR-11708.md), [support cutlass fp4 kernel in sm120](../sources/prs/sglang/PR-11737.md), [Use cutlass fp4 gemm by default](../sources/prs/sglang/PR-11813.md), [Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4](../sources/prs/sglang/PR-11866.md), [Feature/nano v2 offline modelopt fp8 and nvfp4](../sources/prs/sglang/PR-12018.md), [Replace [silu_and_mul_]scaled_fp4_group_quant by Flashinfer equivalent](../sources/prs/sglang/PR-12376.md), [Use sgl fp4 quant kernel by default](../sources/prs/sglang/PR-12482.md), [[NVIDIA] Fix CUDA arch requirement in nvfp4 cast](../sources/prs/sglang/PR-12581.md), [feat: Add FP4 (E2M1) KV Cache Support for MHA](../sources/prs/sglang/PR-12612.md), [[NVIDIA] Fix wrong symmetric sizes for fp4 cases](../sources/prs/sglang/PR-12640.md), [[Bugfix] Fix illegal memory access](../sources/prs/sglang/PR-12758.md), [ignore the deepgemm check when the model weight with nvfp4 and moe ba…](../sources/prs/sglang/PR-12782.md), [support mtp with deepseek r1 nvfp4 model](../sources/prs/sglang/PR-13115.md), [Fix nan in global scaling factor for large scale nvfp4 EP](../sources/prs/sglang/PR-13162.md), [Fix EPLB + FP4 Quantization Compatibility Issue](../sources/prs/sglang/PR-13715.md), [[Feat][NVFP4] Enable NVFP4 MoE for Qwen series models (eg. Qwen3-Next) #13761](../sources/prs/sglang/PR-13761.md), [Support fp4 fp8 non gated moe](../sources/prs/sglang/PR-13794.md), [Fix flashinfer cutlass MoE output shape for non-FP4-packed inputs](../sources/prs/sglang/PR-14028.md), [[FIX] trtllm-moe-fp4-renorm for Qwen series models](../sources/prs/sglang/PR-14350.md), [[CPU] Implement MXFP4 Gemm kernels for intel AMX to support GPT OSS series.](../sources/prs/sglang/PR-14385.md), [Mistral Large 3 NVFP4 support](../sources/prs/sglang/PR-14485.md), [[AMD] Support fused_rms_mxfp4_quant in the prefill stage for DeepSeek-R1-MXFP4](../sources/prs/sglang/PR-14975.md), [Mistral Large 3 NVFP4 TRTLLM MoE support](../sources/prs/sglang/PR-15049.md), [[NVIDIA] Fixes for NVFP4 all-gather with spec decoding](../sources/prs/sglang/PR-15280.md), [Fix the accuracy issue when running mxfp4 dsv3 model and enable ep](../sources/prs/sglang/PR-15304.md), [Update flashinfer to 0.6.1](../sources/prs/sglang/PR-15551.md), [[Perf] Eliminate the slice op for Flashinfer `trtllm_fp4_block_scale_moe`](../sources/prs/sglang/PR-15731.md), [Tiny fix cannot launch nvfp4 checkpoint with bf16 kv cache](../sources/prs/sglang/PR-15986.md), [[Performance] Force split_k=1 for MXFP4 Triton kernels on Hopper](../sources/prs/sglang/PR-16014.md), [Inclusion of nvfp4 blockscale in EPLB Rebalance](../sources/prs/sglang/PR-17158.md), [[Fix] GLM 4.7 + NVFP4 + MTP](../sources/prs/sglang/PR-17166.md), [[FIX] Always support TP > 4 for FP4 Gemm](../sources/prs/sglang/PR-17300.md), [[feat] Support nvfp4 quantized model of Qwen3-Next](../sources/prs/sglang/PR-17627.md), [fix(quantization): add sgl_kernel fallback for FP4 quantize on Blackwell GPUs](../sources/prs/sglang/PR-17816.md), [[Bugfix] Fix Mistral Large 3 NVFP4 TRTLLM MoE](../sources/prs/sglang/PR-18065.md), [Fix nvfp4 weight update](../sources/prs/sglang/PR-18085.md), [[ModelOpt] Fix broken Qwen3-235B-A22B-Instruct-2507-NVFP4 launch](../sources/prs/sglang/PR-18189.md), [[ModelOPT] Support Qwen 3 Next Coder NVFP4](../sources/prs/sglang/PR-18224.md), [[Kimi-K2.5] Fix NVFP4 Kimi-K2.5 weight mapping and exclude list](../sources/prs/sglang/PR-18370.md), [Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4](../sources/prs/sglang/PR-18389.md), [[Perf] ~9.5x faster Blackwell MXFP4 MoE weight loading](../sources/prs/sglang/PR-18858.md), [[Qwen3.5] Enable nvfp4 checkpoint](../sources/prs/sglang/PR-18937.md), [feat: Support MXFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-19143.md), [Adjust padding size to improve triton_kernels moe performance](../sources/prs/sglang/PR-19174.md), [Fix nightly Mistral-Large-3 NVFP4 accuracy threshold](../sources/prs/sglang/PR-19402.md), [[AMD] Fix weight load shape mismatch for amd dsr1 0528 mxfp4](../sources/prs/sglang/PR-19425.md), [[Kernel Slimming] Migrate NVFP4 kernels to JIT](../sources/prs/sglang/PR-19437.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [Support `triton_kernels` for GPT-OSS on SM120](../sources/prs/sglang/PR-19718.md), [[AMD] Fix FP8 assertion failure in aiter MLA decode by falling back to self.k_scale](../sources/prs/sglang/PR-19935.md), [[JIT Kernel] Reland NVFP4 kernels to JIT](../sources/prs/sglang/PR-20012.md), [Fix SM120 `triton_kernels` MXFP4 `block_k` for GPT-OSS](../sources/prs/sglang/PR-20040.md), [MiniMax-M2.5 - Support dp attention, dp reduce scatter, FP4 all gather, AR fusion in prepare_attn](../sources/prs/sglang/PR-20067.md), [[V32/GLM5] Change default setting of V32 nvfp4 on TP4](../sources/prs/sglang/PR-20086.md), [[diffusion] Support nvfp4 for Flux.2](../sources/prs/sglang/PR-20137.md), [[4/n jit_kernel restruct] speed up CI tests and add benchmark workflow](../sources/prs/sglang/PR-20268.md), [[Benchmark] use flashinfer bench_gpu_time instead of triton do_bench](../sources/prs/sglang/PR-20305.md), [[Model] Support Nemotron 3 Super NVFP4](../sources/prs/sglang/PR-20407.md), [[JIT Kernel] Fix NVFP4 multi-arch compilation failure](../sources/prs/sglang/PR-20874.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Chore] Clean up JIT compilation flags](../sources/prs/sglang/PR-21022.md), [[AMD]: Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5…](../sources/prs/sglang/PR-21213.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [[NVIDIA] Enable FP4 flashinfer trtllm routed moe](../sources/prs/sglang/PR-21240.md), [CUTLASS NVFP4 GEMM improvement of SM120](../sources/prs/sglang/PR-21314.md), [[Kernel] Support FlashInfer TRTLLM-Gen fused MoE for non-gated FP4 & FP8 (Nemotron)](../sources/prs/sglang/PR-21321.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [Add dedicated FlashInferCuteDslMoE layer for standard-path FP4 MoE](../sources/prs/sglang/PR-21339.md), [Migrate all callers from /get_server_info to /server_info](../sources/prs/sglang/PR-21463.md), [Harden FlashInfer FP4 imports in standard dispatcher](../sources/prs/sglang/PR-21776.md), [[Diffusion] Fix weight scale swizzle and add large-M kernel config for FLUX.2-dev-NVFP4](../sources/prs/sglang/PR-22064.md), [[nvidia] Gemma4 nvfp4 fix](../sources/prs/sglang/PR-22079.md), [[diffusion] Default NVFP4 to CUTLASS and add all-model shape benchmarks](../sources/prs/sglang/PR-22091.md), [[Diffusion] Add diffusion NVFP4 scaled-mm correctness test](../sources/prs/sglang/PR-22127.md), [[RL] Refactor NVFP4 shuffling/swizzling to in-place replacement](../sources/prs/sglang/PR-22204.md), [:sparkles: [diffusion][npu][quant] Add MXFP4 quantization support for Wan2.2 Diffusion on Ascend NPU](../sources/prs/sglang/PR-22338.md), [GLM-5/5.1 MXFP4 Checkpoint Inference Compatibility Fix](../sources/prs/sglang/PR-22543.md), [[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22574.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [[Diffusion] Add Wan2.2 ModelOpt NVFP4 support](../sources/prs/sglang/PR-22681.md), [[FlashInfer v0.6.11] [RL] Support FlashInfer per-token NVFP4 MoE](../sources/prs/sglang/PR-22918.md), [Reland Cute-DSL FP4 dense GEMM](../sources/prs/sglang/PR-23590.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [Use Cute-DSL NVFP4 quantization kernels](../sources/prs/sglang/PR-23745.md), [[MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute](../sources/prs/sglang/PR-23760.md), [Port MXFP4 Marlin MoE support to JIT kernel path](../sources/prs/sglang/PR-24490.md), [Add FlashInfer SM90 cutlass MXFP4 MoE backend (W4A16) for GPT-OSS + DeepSeek-V4](../sources/prs/sglang/PR-24816.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [perf(nvfp4): free unused source scales after weight processing](../sources/prs/sglang/PR-25107.md), [[Intel GPU] Enable DeepSeek V4 Inference on XPU](../sources/prs/sglang/PR-25336.md), [[AMD] Enable shared-experts fusion with new KIMI-K2.5-MXFP4 model.](../sources/prs/sglang/PR-25390.md), [[codex] Update Wan2.2 ModelOpt CI checkpoints](../sources/prs/sglang/PR-25483.md), [Add no_combine support to cutlass_moe_fp4](../sources/prs/sglang/PR-25688.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [Support FP4 gemm (1/2)](../sources/prs/sglang/PR-3899.md), [[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu](../sources/prs/sglang/PR-4953.md), [[1/2] Add Kernel support for Cutlass based Fused FP4 MoE](../sources/prs/sglang/PR-6093.md), [[DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model](../sources/prs/sglang/PR-6853.md), [Fix Deepseek R1 0528 FP4 tensor name mismatch issue during weights loading.](../sources/prs/sglang/PR-7164.md), [Support NVFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-7302.md), [FlashInfer NVFP4 MoE with EP & 2-stream shared expert](../sources/prs/sglang/PR-7327.md), [Fix MTP with Deepseek R1 Fp4](../sources/prs/sglang/PR-7376.md), [Add fp4 quantize before all-gather for Flashinfer cutlass MoE DP (max throughput)](../sources/prs/sglang/PR-7667.md), [Qwen FP8/NVFP4 ModelOPT Quantization support](../sources/prs/sglang/PR-7912.md), [[Fix][Ready]Fix register spilling in cutlass nvfp4 gemm kernel on Blackwell](../sources/prs/sglang/PR-8127.md), [[fix] fix modelopt fp4 on b200](../sources/prs/sglang/PR-8195.md), [[NVIDIA] Add Low Latency NVFP4 decode kernels from Flashinfer](../sources/prs/sglang/PR-8552.md), [[Perf] Auto enable best flashinfer mxfp4 kernel in b200](../sources/prs/sglang/PR-8898.md), [Fix hopper launch gpt-oss model illegal memory](../sources/prs/sglang/PR-8908.md), [Faster weight processing (trtllm-gen moe nvfp4)](../sources/prs/sglang/PR-9162.md), [[NVIDIA] [3/N] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked ](../sources/prs/sglang/PR-9199.md), [[NVIDA] [1/N] Nvfp4 Masked Gemm: Add quant op for the flashinfer grouped gemm](../sources/prs/sglang/PR-9200.md), [Fix FP4 inference corruption issue in glm4.5-air model](../sources/prs/sglang/PR-9346.md), [[fix] Fix mxfp4 triton MoE tp bug](../sources/prs/sglang/PR-9473.md), [[NVIDIA] [2/N] Optimize `silu_and_mul_scaled_fp4_grouped_quant` perf](../sources/prs/sglang/PR-9556.md), [move is_sm90_supported/is_sm100_supported to python/sglang/srt/utils.py](../sources/prs/sglang/PR-9679.md), [[ModelOpt] Fix Weight Loading for DSR1-FP4 Quantization](../sources/prs/sglang/PR-9712.md), [Make fp4_quantize kernels work on sm103](../sources/prs/sglang/PR-9807.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[TMA] Support FP4 TensorMap TMA copies](../sources/prs/tilelang/PR-2107.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[NVIDIA] Support nvfp4 quantization](../sources/prs/vllm/PR-12784.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[Hardware/NVIDIA/Kernel] [Functional Enablement] [1/N] Enable nvidia/DeepSeek-R1-FP4 Model](../sources/prs/vllm/PR-16362.md), [[Kernel] fp4 marlin kernel](../sources/prs/vllm/PR-17687.md), [[Misc] Add compressed-tensors NVFP4A16 emulation support](../sources/prs/vllm/PR-17914.md), [Use NVFP4 Marlin for CompressedTensorsW4A16Fp4](../sources/prs/vllm/PR-18000.md), [[Quantization] Add compressed-tensors NVFP4 support](../sources/prs/vllm/PR-18312.md), [[V1] Support `LLM.apply_model`](../sources/prs/vllm/PR-18465.md), [[Hardware][NVIDIA] FP4 MoE kernel optimization](../sources/prs/vllm/PR-19110.md), [[Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization](../sources/prs/vllm/PR-19500.md), [[Quantization] Add compressed-tensors emulations support for NVFP4](../sources/prs/vllm/PR-19879.md), [[Quantization] Add compressed-tensors NVFP4 MoE Support](../sources/prs/vllm/PR-19990.md), [Add ModelOpt Qwen3 nvfp4 support](../sources/prs/vllm/PR-20101.md), [[Bugfix] Fix some narrowing conversion warnings](../sources/prs/vllm/PR-20141.md), [[Kernel][Bugfix] Fixup some warnings in nvfp4_blockwise_moe when CUDA < 12.8](../sources/prs/vllm/PR-20324.md), [Support Llama 4 for cutlass_moe_fp4](../sources/prs/vllm/PR-20453.md), [[Perf] Reuse workspace for FP8+FP4 Marlin MoE](../sources/prs/vllm/PR-20500.md), [Support mnnvl all2allv from Flashinfer](../sources/prs/vllm/PR-21003.md), [[Feature][OCP MX] Support mxfp6 and mixed mxfp6-mxfp4](../sources/prs/vllm/PR-21166.md), [Support CUTLASS NVFP4 (w4a4) for Blackwell Geforce GPUs (SM120)](../sources/prs/vllm/PR-21309.md), [Support Tensorrt-LLM MoE fp4 for low-latency](../sources/prs/vllm/PR-21331.md), [Update flashinfer CUTLASS NVFP4 MoE Kernel to use per expert global scaling factor](../sources/prs/vllm/PR-21408.md), [[Bug] Fix Compressed Tensor NVFP4 `cutlass_fp4_group_mm` illegal memory access](../sources/prs/vllm/PR-21465.md), [[NVIDIA] Fix Llama4 Scout FP4 functionality issues](../sources/prs/vllm/PR-21499.md), [[Feature] Add Flashinfer MoE Support for Compressed Tensor NVFP4](../sources/prs/vllm/PR-21639.md), [[gpt-oss] flashinfer mxfp4](../sources/prs/vllm/PR-22339.md), [[gpt-oss] triton kernel mxfp4](../sources/prs/vllm/PR-22421.md), [Fix Llama4 FlashInfer FP4 MoE issues](../sources/prs/vllm/PR-22511.md), [Quantization: support FP4 quantized models on AMD CDNA2/CDNA3 GPUs](../sources/prs/vllm/PR-22527.md), [Fix torch version check for SM100 mxfp4 ](../sources/prs/vllm/PR-22535.md), [[Quantization] Expand compressed-tensors MoE matching logic to support NFP4 + FP8 MoEs](../sources/prs/vllm/PR-22674.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv NVFP4-out Attention Kernel](../sources/prs/vllm/PR-22703.md), [Use Blackwell FlashInfer MXFP4 MoE by default if available ](../sources/prs/vllm/PR-23008.md), [Add routed_scaling_factor to MoE grouped topk](../sources/prs/vllm/PR-23123.md), [Fix nvfp4 swizzling](../sources/prs/vllm/PR-23140.md), [[Kernels] Overlap shared experts with send/recv](../sources/prs/vllm/PR-23273.md), [fix incompatibililty with non cuda platform for nvfp4](../sources/prs/vllm/PR-23478.md), [Update Flashinfer to 0.2.14.post1](../sources/prs/vllm/PR-23537.md), [DP/EP Support for gpt-oss with deepep-ht comm kernel on SM100](../sources/prs/vllm/PR-23608.md), [[Bugfix] Fix Marlin NVFP4 for modelopt](../sources/prs/vllm/PR-23659.md), [[NVIDIA] Support SiluMul + NVFP4 quant fusion](../sources/prs/vllm/PR-23671.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files](../sources/prs/vllm/PR-23727.md), [[Model][gpt-oss] Support DP+EP for GPT-OSS with FlashInfer trtllm-gen MoE](../sources/prs/vllm/PR-23819.md), [[BUGFIX ] fix undefined silu_and_mul_nvfp4_quant](../sources/prs/vllm/PR-23929.md), [[Model] Add LongCat-Flash ](../sources/prs/vllm/PR-23991.md), [[Transform] [Quantization] Add QuTLASS support to vLLM](../sources/prs/vllm/PR-24440.md), [[Kernel][Quantization] add w4a8 support for marlin kernel](../sources/prs/vllm/PR-24722.md), [[Bugfix] Fix accuracy issue for silu_mul + nvfp4 quant fusion kernel](../sources/prs/vllm/PR-24833.md), [[Compile] Fix Compile Warning for Ignoring `MIN_BLOCK_PER_SM`](../sources/prs/vllm/PR-25193.md), [[ROCm] Small functional changes for gptoss](../sources/prs/vllm/PR-25201.md), [Enable Fbgemm NVFP4 on Dense models](../sources/prs/vllm/PR-25609.md), [[Bugfix] Enable padded FP4 quantization](../sources/prs/vllm/PR-25947.md), [[Quantization/NVFP4] Speed up TRTLLM NVFP4 MOE weight loading and fix K/V scale loading for MLA Attn](../sources/prs/vllm/PR-25968.md), [[Bugfix] Allow skipping MoE in NVFP4 (fix for MTP)](../sources/prs/vllm/PR-25987.md), [[MoE] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked](../sources/prs/vllm/PR-25990.md), [[NVIDIA] Add support for cudnn fp4 gemm via flashinfer](../sources/prs/vllm/PR-26107.md), [[ModelOpt] Load w13/w2_input_scale for all experts, nvfp4](../sources/prs/vllm/PR-26135.md), [[ROCM] MoE fp4 CK kernel](../sources/prs/vllm/PR-26545.md), [support flashinfer_fp4 moe for 5090 gpu](../sources/prs/vllm/PR-26669.md), [[NVIDIA] [Perf] Update to leverage flashinfer trtllm FP4 MOE throughput kernel](../sources/prs/vllm/PR-26714.md), [[Bugfix] Fix gpt-oss w4a8 DP/EP on B200](../sources/prs/vllm/PR-26729.md), [Flashinfer_CUTLASS_MOE fuses quantization for TP](../sources/prs/vllm/PR-27223.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [[Bugfix] Fix GPT-OSS on AMD after #28603](../sources/prs/vllm/PR-28816.md), [Add TRTLLM MoE NVFP4 kernel to CompressedTensorsW4A4MoeMethod](../sources/prs/vllm/PR-28892.md), [[Feat] Support non-gated activations in NVFP4 modelopt path](../sources/prs/vllm/PR-29004.md), [[Kernel] Add NVFP4 MoE CUTLASS support for SM120](../sources/prs/vllm/PR-29242.md), [[Bugfix] Only use triton_kernels for MXFP4 on SM90 and SM100](../sources/prs/vllm/PR-29339.md), [[Bugfix] Fix mismatched nvfp4 gemm output shape](../sources/prs/vllm/PR-29742.md), [[ROCm][MXFP4] Infer w4a4 quant method in rocm aiter fused moe](../sources/prs/vllm/PR-29775.md), [[EPLB] Support EPLB w/ NVFP4](../sources/prs/vllm/PR-29804.md), [[Perf] Do FP4 quant before All gather on flashinfer trtllmgen MOE ](../sources/prs/vllm/PR-30014.md), [[ROCm][Quantization] GPT OSS Upstream MoE wmxfp4_afp8 with static scales](../sources/prs/vllm/PR-30357.md), [[Feature] Add SM103 (Blackwell Ultra) Support to vLLM](../sources/prs/vllm/PR-30484.md), [[Perf] Set split_k to 1 for triton_kernels](../sources/prs/vllm/PR-30528.md), [[Perf] Eliminate padding and slicing op for GPT-OSS with Flashinfer MXFP4 MXFP8 MoE](../sources/prs/vllm/PR-30647.md), [[SM100] Enable fp8 compute for prefill MLA](../sources/prs/vllm/PR-30746.md), [[Compressed-Tensors] Simplify NVFP4 Conditions, enable marlin support for NVFP4A16 MoEs](../sources/prs/vllm/PR-30881.md), [[Kernel][Performance] Enable smaller Scaling Factor tiling for NVFP4 small-batch decoding](../sources/prs/vllm/PR-30885.md), [[NVFP4][Perf] Tune NVFP4 input quant kernel for small batch size](../sources/prs/vllm/PR-30897.md), [ [FIX] Always support TP > 4 for FP4 Gemm](../sources/prs/vllm/PR-31099.md), [[Bugfix] Fix Broken ModelOpt NVFP4 MoE](../sources/prs/vllm/PR-31742.md), [[Perf] Fuse stride preparation for NVFP4 cutlass_moe](../sources/prs/vllm/PR-31837.md), [[5/N][Attention] Finish eliminating `vllm/attention` folder](../sources/prs/vllm/PR-32064.md), [[Perf][Kernel] Optimize FP4 quantization kernels (SM100F)](../sources/prs/vllm/PR-32520.md), [Support compress-tensors with nvfp4 or fp8 weights and modelopt with nvfp4 weights on Turing](../sources/prs/vllm/PR-33076.md), [fix: Add SM120 (RTX Blackwell) support for FlashInfer CUTLASS NVFP4 MoE kernels](../sources/prs/vllm/PR-33417.md), [[Kernel] Support Flashinfer trtllm fused MoE non gated FP8 & NVFP4](../sources/prs/vllm/PR-33506.md), [[Bugfix] Fix DSV3.2 NVFP4](../sources/prs/vllm/PR-33932.md), [[Bugfix]fix output Nan/Inf in marlin if dtype=float16](../sources/prs/vllm/PR-33972.md), [[ModelBash][DSR1 NVFp4] Avoid Bf16 Bias Cast](../sources/prs/vllm/PR-34298.md), [[Custom Ops] Add functional + out variant for scaled_fp4_quant](../sources/prs/vllm/PR-34389.md), [[BUGFIX] Fix accuracy regression for NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with TP>1](../sources/prs/vllm/PR-34476.md), [[Model] Add NVFP4 quantization support for Step3.5-Flash](../sources/prs/vllm/PR-34478.md), [[Bugfix] Rescale NVFP4 weight scales to fix BF16 dequant underflow](../sources/prs/vllm/PR-34577.md), [[torch.compile] Turn on silu+fp4 quant fusion by default for O1+](../sources/prs/vllm/PR-34718.md), [[Bugfix] Fix NVFP4 TRTLLM MoE non-gated support; add gsm8k for Nemotron-3-Nano FP8+NVFP4](../sources/prs/vllm/PR-34725.md), [[BugFix] Fix fp4 quant kernel on CUDA 12.8](../sources/prs/vllm/PR-35210.md), [fix(mxfp4): return is_monolithic=False when LoRA is enabled for Triton backend](../sources/prs/vllm/PR-35382.md), [[NVFP4] Support NVFP4 dense models from `modelopt` and `compressed-tensors` on AMD Instinct MI300, MI355X and Hopper through emulation](../sources/prs/vllm/PR-35733.md), [[Bugfix] Fix passing of activation_type to trtllm fused MoE NVFP4 and FP8](../sources/prs/vllm/PR-36017.md), [[Mamba] Flashinfer selective_state_update](../sources/prs/vllm/PR-36162.md), [[mla] Support fused FP8/NVFP4 output quantization in MLA attention (#35792)](../sources/prs/vllm/PR-36205.md), [[Bug][MoE] Fix TRTLLM NVFP4 Routing Kernel Precision](../sources/prs/vllm/PR-36725.md), [[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts](../sources/prs/vllm/PR-36728.md), [[MoE Refactor] Mxfp4 oracle rebased](../sources/prs/vllm/PR-37128.md), [Fix minimax m2.5 nvfp4 kv scales weight loading](../sources/prs/vllm/PR-37214.md), [[MoE/EPLB] Fix FlashInfer nvfp4 experts + EPLB correctness](../sources/prs/vllm/PR-37217.md), [[Kernel] Add non-gated support for NVFP4 CUTLASS MoE](../sources/prs/vllm/PR-37320.md), [[Bugfix] Fix EP weight filter breaking EPLB and NVFP4 accuracy](../sources/prs/vllm/PR-37322.md), [Add nvfp4 support to reshape_and_cache_flash](../sources/prs/vllm/PR-37332.md), [[Kernel] Add MXFP4 W4A4 CUTLASS MoE kernel for SM100](../sources/prs/vllm/PR-37463.md), [[Bugfix] Remove assertion for NVFP4 scale dynamic range](../sources/prs/vllm/PR-37465.md), [[Bugfix] Fix marlin nvfp4 rescaling](../sources/prs/vllm/PR-37502.md), [[4/n] Migrate FP4/W4A8 CUTLASS kernels to torch stable ABI](../sources/prs/vllm/PR-37503.md), [[Perf] Use torch compile to fuse pack topk in trtllm moe](../sources/prs/vllm/PR-37695.md), [[Bugfix] Preserve CUDA arch suffix (a/f) for SM12x — fixes NVFP4 NaN on desktop Blackwell](../sources/prs/vllm/PR-37725.md), [[MoE] Move FlashInfer CuteDSL experts into fused_moe/experts/](../sources/prs/vllm/PR-37759.md), [[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration](../sources/prs/vllm/PR-38050.md), [[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell](../sources/prs/vllm/PR-38083.md), [Fix NaN from stale FP4 scale padding in create_fp4_scale_tensor](../sources/prs/vllm/PR-38148.md), [[Quantization] Add FlashInfer CuteDSL batched experts backend for NVFP4 MoE](../sources/prs/vllm/PR-38251.md), [[MoE] Add RoutingMethodType.Simulated to TRT-LLM FP8/NVFP4 kernel allowlists](../sources/prs/vllm/PR-38329.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[Compile] Fix nvfp4 compile warning](../sources/prs/vllm/PR-38573.md), [[Bugfix] Fix NVFP4+MTP crash: force unquantized mtp.fc for Qwen3.5](../sources/prs/vllm/PR-38832.md), [[MoE Refactor] Split up compressed_tensors_moe.py](../sources/prs/vllm/PR-38960.md), [[MoE] Move GPT OSS Triton kernel experts into fused_moe/experts/](../sources/prs/vllm/PR-39007.md), [[Refactor] Move NVFP4 GEMM management into NvFp4LinearKernel](../sources/prs/vllm/PR-39129.md), [[Feature] Batch invariant nvfp4 linear support](../sources/prs/vllm/PR-39322.md), [[Kernel] Support TRTLLM GEN NVFP4 MoE for non-512-aligned hidden dims via weight padding](../sources/prs/vllm/PR-39510.md), [[Bugfix] Reject non-nvfp4 dtypes when using the flashinfer_nvlink_one_sided all2all backend](../sources/prs/vllm/PR-39717.md), [[Bug] Fix batch invariance nvfp4 support](../sources/prs/vllm/PR-39820.md), [[Bugfix] Temporarily disable B200 fp4 MoE layer tests](../sources/prs/vllm/PR-40057.md), [Add nvfp4 kv cache support](../sources/prs/vllm/PR-40177.md), [[Bugfix] Guard mxfp4_experts_quant bindings on ENABLE_NVFP4_SM100](../sources/prs/vllm/PR-40191.md), [[MoE] Move cutlass moe to fused_moe/experts/](../sources/prs/vllm/PR-40574.md), [[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided](../sources/prs/vllm/PR-40960.md), [[Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4](../sources/prs/vllm/PR-41050.md), [[ROCm][Quantization][3/N] Refactor quark_moe w4a4 w/ oracle](../sources/prs/vllm/PR-41436.md), [[Quantization] Rework quantization_config to use QuantKey and allow for activation override](../sources/prs/vllm/PR-41566.md), [[MXFP4] Support for linear layers + compressed-tensors integration](../sources/prs/vllm/PR-41664.md), [Add NVFP4 all-gather GEMM fusion for AsyncTP](../sources/prs/vllm/PR-41882.md), [[CPU] Add MXFP4 W4A16 MoE support](../sources/prs/vllm/PR-41922.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Bugfix] Add swiglu limits to deepgemm fp8 methods](../sources/prs/vllm/PR-41986.md), [[Kernel] Pack topk id/weights triton kernel](../sources/prs/vllm/PR-42527.md), [[Bugfix] fix swiglu limit issue for humming backend + deepseek v4](../sources/prs/vllm/PR-42541.md), [[Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement](../sources/prs/vllm/PR-42774.md), [[Perf] Re-enable flashinfer autotune by default and cleanup](../sources/prs/vllm/PR-42857.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md), [TensorRT-LLM Blackwell FP4 DSA Indexer](../wiki/kernels/tensorrt-llm-blackwell-indexer.md), [Fine-Grained FP8/FP4 Quantization](../wiki/techniques/fine-grained-quantization.md) | | `fp6` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md) | -| `fp8` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../sources/blogs/flashmla.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [DeepSeek-V3.2-Exp in vLLM: Fine-Grained Sparse Attention in Action](../sources/blogs/vllm-deepseek-v3-sparse-attention.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Use swizzling instead of padding](../sources/prs/DeepGEMM/PR-86.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[TRTLLM-9457][feat] Add cute dsl fp8 gemm for Blackwell](../sources/prs/TensorRT-LLM/PR-10130.md), [[None][fix] impl fused triton kernel for e8m0 resmooth to reduce memory footprint](../sources/prs/TensorRT-LLM/PR-10327.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[None][feat] Add support for expert_number<=2048 and K<=32](../sources/prs/TensorRT-LLM/PR-11510.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[TRTLLM-10421][perf] Add fused cat+fp8_quantize CUDA kernel for DSA indexer](../sources/prs/TensorRT-LLM/PR-11899.md), [[None][feat] GLM 5 support and DSA MTP fixes](../sources/prs/TensorRT-LLM/PR-11990.md), [[https://nvbugs/5879577][fix] Fix KeyError in DeepSeekV3Lite FP8 MTP weight loading](../sources/prs/TensorRT-LLM/PR-12530.md), [[#12784][feat] AutoDeploy: Optimize DeepSeek-R1 model performance](../sources/prs/TensorRT-LLM/PR-12946.md), [[TRTLLM-34871][feat] Add cute dsl FP8 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13219.md), [[TRTLLM-11127][feat] add W4A8_MXFP4_FP8 MoE unit test support](../sources/prs/TensorRT-LLM/PR-13401.md), [[None][feat] Fuse FP8 1x128 quantize + UE8M0 scale pack on SM100](../sources/prs/TensorRT-LLM/PR-13628.md), [[None][fix] Plumb swiglu_limit through DeepGEMM and TRTLLMGen FP8 fused MoE](../sources/prs/TensorRT-LLM/PR-13767.md), [[TRTLLM-12503][feat] Parallel VAE independent scaling and fix arg passing](../sources/prs/TensorRT-LLM/PR-13873.md), [[TRTLLM-35237][feat] Add cute dsl FP4 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13929.md), [[None][feat] Keep DSv4 o_a_proj as FP8, and port vLLM's fused_inv_rope_fp8_quant](../sources/prs/TensorRT-LLM/PR-13938.md), [[TRTLLM-12462][fix] Fix FP8 block scaling GEMM autotuner cache growth](../sources/prs/TensorRT-LLM/PR-14165.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[TRTLLM-8535][feat] Support DeepSeek V3.2 with FP8 + BF16 KV cache/NVFP4 + BF16 KV cache](../sources/prs/TensorRT-LLM/PR-8405.md), [[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP](../sources/prs/TensorRT-LLM/PR-8501.md), [[None][feat] TRT-LLM Gen MoE optimize DeepSeek Fp8 activation kernel](../sources/prs/TensorRT-LLM/PR-9175.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Hopper Grouped GEMM support for FP8 Accum](../sources/prs/cutlass/PR-2123.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [support fp16 accmulator for sm89 fp8 mma](../sources/prs/cutlass/PR-2378.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [[Cutlass gemm] Fix SM100 FP8 nosmem epilogue-fusion shape_div 'Divisibility Condition' for non-multiple-of-64 N tiles](../sources/prs/cutlass/PR-2946.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG](../sources/prs/flash-attention/PR-1173.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [[Cute,Fwd,Sm100] fp8 e4m3 and e5m2 support](../sources/prs/flash-attention/PR-2109.md), [feat: add functional per-head FP8 quantization for FA3](../sources/prs/flashinfer/PR-1033.md), [perf: accelerate blackwell grouped gemm](../sources/prs/flashinfer/PR-1086.md), [bugfix: fix fp8 attention kernels aot compilation issue](../sources/prs/flashinfer/PR-1087.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [[Feature] Support PDL for batch Prefill and Decode](../sources/prs/flashinfer/PR-1117.md), [Add DeepGEMM kernels](../sources/prs/flashinfer/PR-1209.md), [Fix test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-1211.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [Patch fp8 cubin availability](../sources/prs/flashinfer/PR-1240.md), [feat: Support MXFP8 x MXFP4 CUTLASS grouped GEMM](../sources/prs/flashinfer/PR-1241.md), [Add trtllm-gen attention mha kernel with FP8 Q/K/V and FP8 output](../sources/prs/flashinfer/PR-1242.md), [Reduce the JIT compilation time of gen_gemm_sm100_module](../sources/prs/flashinfer/PR-1251.md), [init add gemm fp8 using cudnn backend](../sources/prs/flashinfer/PR-1264.md), [feat: add masked deepgemm support and benchmarking](../sources/prs/flashinfer/PR-1266.md), [Unify groupwise fp8 GEMM test](../sources/prs/flashinfer/PR-1281.md), [Bug fix: guard fp8 e8m0 and e2m1 compile ](../sources/prs/flashinfer/PR-1287.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [Make Fp8 MoE routing_bias optional](../sources/prs/flashinfer/PR-1319.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [feat: Fused rope fp8 quantize kernel for MLA](../sources/prs/flashinfer/PR-1339.md), [[fix] remove (view) transpose to keep consistent with majorness MN requirement.](../sources/prs/flashinfer/PR-1358.md), [bugfix: Add guard for fp4/fp8 related include headers](../sources/prs/flashinfer/PR-1376.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [Adding FP8 benchmark on attention and matmul testing](../sources/prs/flashinfer/PR-1390.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [feature: add cutlass as bmm_fp8 backend.](../sources/prs/flashinfer/PR-1397.md), [bugfix: fix perf issue by using fp8 graph that can use cublaslt](../sources/prs/flashinfer/PR-1435.md), [Add alignment in MxFP8Quantization](../sources/prs/flashinfer/PR-1445.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [perf: add 1x4x1 cluster shape for fp8 bmm M<16 cases](../sources/prs/flashinfer/PR-1473.md), [refactor: unify autotuner for bmm_fp8](../sources/prs/flashinfer/PR-1479.md), [feat: Support fp8 qkv, fp16/bf16 out MHA for trtllm-gen.](../sources/prs/flashinfer/PR-1490.md), [Perf: support scale_a/scale_b instead of combined scale in cutlass bmm_fp8](../sources/prs/flashinfer/PR-1491.md), [flashinfer_benchmark QoL Improvements and Attention FP8 Support](../sources/prs/flashinfer/PR-1512.md), [bugfix: Fix compile error for undefined swizzle enum.](../sources/prs/flashinfer/PR-1530.md), [feat: Add fp8-qkv, fp16/bf16 output MHA](../sources/prs/flashinfer/PR-1540.md), [bugfix: fix unittest test_fp8_quantize](../sources/prs/flashinfer/PR-1599.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [feat: cutlass fp8 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1610.md), [test: pytest.mark.xfail on deepgemm](../sources/prs/flashinfer/PR-1636.md), [Add benchmark for MLARopeQuantize](../sources/prs/flashinfer/PR-1656.md), [perf&bugfix: skip kv-tile computation out of sliding window in FA2; fix __syncthreads in mergestate](../sources/prs/flashinfer/PR-1661.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [test: better fp8 quantization init for fused_moe test](../sources/prs/flashinfer/PR-1674.md), [Update deepgemm backend for 103a](../sources/prs/flashinfer/PR-1694.md), [test: skip the unsupported test cases for sm120/121](../sources/prs/flashinfer/PR-1710.md), [TVM: support TVM binding for GroupedGemm](../sources/prs/flashinfer/PR-1725.md), [tests: xfail moe quantization classes mxfp8_bf16 UTs on sm103 ](../sources/prs/flashinfer/PR-1754.md), [tests: skip non SM100/103 for grouped deepgemm](../sources/prs/flashinfer/PR-1767.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [raise error for group_gemm_fp8_nt_groupwise then num_groups > 1 on sm120/121](../sources/prs/flashinfer/PR-1862.md), [misc: fix some B200 GEMM bench](../sources/prs/flashinfer/PR-1883.md), [MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA](../sources/prs/flashinfer/PR-1924.md), [Fix: Verify scales are not None for Cutlass FP8 FusedMoE](../sources/prs/flashinfer/PR-1961.md), [feat: enable deepgemm jit for fp8 block-scale on SM90](../sources/prs/flashinfer/PR-1969.md), [fix: correct PDL parameter handling in RopeQuantize kernel](../sources/prs/flashinfer/PR-1982.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [feat: suitable_auto_backends to prune auto backends, bmm_fp8 refactor, heuristic_func intake](../sources/prs/flashinfer/PR-2029.md), [Enable renormalize(naive) routing for fp8 per-tensor](../sources/prs/flashinfer/PR-2030.md), [Added an initial implementation of Q and KV Cache in fp8 and to use t…](../sources/prs/flashinfer/PR-2035.md), [feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA) ](../sources/prs/flashinfer/PR-2037.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [test: Skip test_fp8_quantize.py on Hopper](../sources/prs/flashinfer/PR-2052.md), [Fix moe fp8 failure for sm121](../sources/prs/flashinfer/PR-2061.md), [perf: TRT-LLM MoE Block-FP8 activation optimization](../sources/prs/flashinfer/PR-2063.md), [enable xqa fp8 output](../sources/prs/flashinfer/PR-2081.md), [refactor: update dpsk fused_moe test [1]](../sources/prs/flashinfer/PR-2088.md), [refactor: update dpsk fused_moe test [2]](../sources/prs/flashinfer/PR-2097.md), [refactor: update fa3 codebase and fix hopper unittest [part 1]](../sources/prs/flashinfer/PR-2111.md), [fix: Fix bench_mm_fp8.py](../sources/prs/flashinfer/PR-2129.md), [make DeepGEMM swapAB available for linear gemm SM90](../sources/prs/flashinfer/PR-2131.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [Enable Hopper FA3 FP8 attention in decode.py](../sources/prs/flashinfer/PR-2148.md), [Fp8 attention are now part of cuDNN 9.17.1](../sources/prs/flashinfer/PR-2241.md), [feat: RMSNorm/Fused RMSNorm + FP8 Quantization kernels](../sources/prs/flashinfer/PR-2243.md), [fix: support int64 IdType for RoPE part argument in `rope_quantize_fp8_append_paged_kv_cache`](../sources/prs/flashinfer/PR-2255.md), [feat: Add support for bmm mxfp8](../sources/prs/flashinfer/PR-2256.md), [Fix CUTLASS FP8 gemm correctness issue on SM120/SM121 for shapes where N is not divisible by ScaleGranularityN.](../sources/prs/flashinfer/PR-2261.md), [test: use .float() in in F.cosine_similarity() in bmm_fp8 test](../sources/prs/flashinfer/PR-2266.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [[perf] Improve gemm_fp8_nt_groupwise (cutlass backend) by 10-40% for batch sizes <= 32](../sources/prs/flashinfer/PR-2327.md), [fix: guard batchWarpReduceSum with ENABLE_FP8 to fix compilation without FP8](../sources/prs/flashinfer/PR-2328.md), [fix: Fix NaN output in mxfp8_quantize for very small input values](../sources/prs/flashinfer/PR-2441.md), [Add cute-dsl backends to mxfp[8,4]_quantization for future refactor](../sources/prs/flashinfer/PR-2443.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [feat: Add MXFP8 GEMM mm_mxfp8 (cutlass)](../sources/prs/flashinfer/PR-2464.md), [Feat: Trtllm-gen MxFP8 MoE integration](../sources/prs/flashinfer/PR-2505.md), [fix: include fp8_blockscale_gemm_90 in AOT jit-cache](../sources/prs/flashinfer/PR-2533.md), [fallback to fa2 (instead of fa3) for unsupported configuration (bf16 Q, Fp8 KV)](../sources/prs/flashinfer/PR-2536.md), [tests: bmm_fp8 for SM110](../sources/prs/flashinfer/PR-2538.md), [Add gen_gemm_sm100_module_cutlass_mxfp8 to jit-cache](../sources/prs/flashinfer/PR-2549.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [fix: add SM121 support to SM120 version guards](../sources/prs/flashinfer/PR-2631.md), [benchmark: Add MXFP4/MXFP8 quantization mode support to FP4 MoE benchmark](../sources/prs/flashinfer/PR-2635.md), [[fp8_blockwise]Fix int32 overflow in TRTLLM fused MoE activation kernel](../sources/prs/flashinfer/PR-2642.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [feat: support mxfp4 & mxfp8 entrypoint for blackwell cutedsl dense gemm](../sources/prs/flashinfer/PR-2660.md), [benchmarks: Add FP8 input / BF16 output in ragged prefill benchmark](../sources/prs/flashinfer/PR-2666.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [feat: Add support for TRTLLM MXFP8 non-gated MoE with ReLU2](../sources/prs/flashinfer/PR-2707.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [Support in-place update for `trtllm_fp8_block_scale_moe`](../sources/prs/flashinfer/PR-2739.md), [Add cute dsl mla decode op](../sources/prs/flashinfer/PR-2743.md), [[Spark unit test debugging] Fix for tests/gemm/test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-2751.md), [feat: FP8 output support for CUTLASS MLA paged attention](../sources/prs/flashinfer/PR-2779.md), [tests: skip sliding window + fp8 to prevent hang in fmha_v2 unit tests](../sources/prs/flashinfer/PR-2781.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [Fix silent bug with FP8 per tensor non-gated MoE](../sources/prs/flashinfer/PR-2882.md), [feat: add pdl support for cute dsl mla decode kernel support](../sources/prs/flashinfer/PR-2901.md), [feat: add MXFP8 GEMM support for SM120](../sources/prs/flashinfer/PR-2902.md), [perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels](../sources/prs/flashinfer/PR-2904.md), [[NVIDIA] fix(jit): enable GDC for CUTLASS fused MoE PDL — prevent random crashes on SM12x](../sources/prs/flashinfer/PR-2913.md), [feat: Add cuBLASLt backend for `mm_bf16` and enable multi-tactic autotuning for FP8/MXFP8 runners](../sources/prs/flashinfer/PR-2914.md), [test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9](../sources/prs/flashinfer/PR-2963.md), [test: skip unsupported mm_mxfp8 configurations on SM12x](../sources/prs/flashinfer/PR-2974.md), [ Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] ](../sources/prs/flashinfer/PR-2994.md), [Support Allreduce + Norm + Per-token Group Fp8 Quant Fusion](../sources/prs/flashinfer/PR-3059.md), [feat: Enable FP8 (E4M3/E5M2) in concat_mla_k for optimize long-context prefill performance and refactor type dispatch for BF16/FP16](../sources/prs/flashinfer/PR-3129.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [test: enable bmm_mxfp8 cutlass backend coverage on SM12x](../sources/prs/flashinfer/PR-3183.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [Support Kimi K2.5 H64 CuTe DSL MLA decode](../sources/prs/flashinfer/PR-3235.md), [fix(fmha_v2): fix FP8 V-scratch pipeline and varlen scheduler on SM90](../sources/prs/flashinfer/PR-3276.md), [Naive Support for Hopper FP8 Prefill Kernel with Per-Head Quantization](../sources/prs/flashinfer/PR-869.md), [perf: Fix python API overhead when CUDAGraph is not enabled](../sources/prs/flashinfer/PR-969.md), [Update CUTLASS. Refine KernelSchedule for fp8 (grouped) gemm.](../sources/prs/sglang/PR-10491.md), [support qwen3-next-fp8 deepep](../sources/prs/sglang/PR-10622.md), [Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization](../sources/prs/sglang/PR-11081.md), [[sgl-kernel][1/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-11432.md), [[DeepseekV32] Enable flashmla_prefill kernel with fp8 kvcache](../sources/prs/sglang/PR-11655.md), [Support running FP4 Deepseek on SM120.](../sources/prs/sglang/PR-11708.md), [Change bf16 to fp8 for some gemms in attention for DeepSeek ckpt v2](../sources/prs/sglang/PR-11805.md), [Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4](../sources/prs/sglang/PR-11866.md), [Feature/nano v2 offline modelopt fp8 and nvfp4](../sources/prs/sglang/PR-12018.md), [[sgl-kernel][4/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-12080.md), [[hotfix] missing `w13_weight_fp8` and `w2_weight_fp8` in UE8M0 requantization](../sources/prs/sglang/PR-12259.md), [fix: llama 4 + trtllm gen + fp8 kv cache incompatibility](../sources/prs/sglang/PR-12347.md), [Enable Flashinfer TRTLLM-GEN-MoE FP8 blockwise kernel for Qwen3-Next on Blackwell](../sources/prs/sglang/PR-12543.md), [[sgl-kernel] support custom fp8 flashmla kernel](../sources/prs/sglang/PR-13087.md), [Aiter fp8 kv cache](../sources/prs/sglang/PR-13147.md), [[NVIDIA] Fix broken fp8 MoE of deepseek v3](../sources/prs/sglang/PR-13264.md), [[NVIDIA] Fix use case of SGLANG_ENABLE_FLASHINFER_GEMM](../sources/prs/sglang/PR-13274.md), [[ROCM] Optimized deepseek-r1 fp8 model with + triton_gemm_a8w8 + batch_gemm_a8w8 + fused set_mla_kv_buffer kernel](../sources/prs/sglang/PR-13617.md), [[sgl-kernel][Feat][B200][1/N]Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-13731.md), [Support fp4 fp8 non gated moe](../sources/prs/sglang/PR-13794.md), [Feat: GLM-4.6 supports shared experts fusion](../sources/prs/sglang/PR-13873.md), [Fix update weight error for blackwell DeepGEMM](../sources/prs/sglang/PR-13910.md), [[DeepSeek v3.2] opt Context Parallelism: support fused moe, multi batch and fp8 kvcache](../sources/prs/sglang/PR-13959.md), [Support checking fp8 params in weight_checker](../sources/prs/sglang/PR-14147.md), [Add Mistral Large 3 support.](../sources/prs/sglang/PR-14213.md), [[CPU] Implement MXFP4 Gemm kernels for intel AMX to support GPT OSS series.](../sources/prs/sglang/PR-14385.md), [Support FP8 MLA prefill and 128k context.](../sources/prs/sglang/PR-14395.md), [Add Mistral Large 3 Eagle Support](../sources/prs/sglang/PR-14466.md), [[sgl-kernel][Feat][B200][2/N] Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-14640.md), [[sgl-kernel] Update flashmla to include fp8 sparse_mla optimizations](../sources/prs/sglang/PR-15242.md), [Flashinfer MOE FP8 support for Mistral Large 3.](../sources/prs/sglang/PR-15422.md), [[sgl-kernel][6/7]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-15471.md), [[Perf] Add Flashinfer DeepGEMM SM90 for SwapAB Optimization](../sources/prs/sglang/PR-15514.md), [Optimize FP8 MLA KV cache writes with Triton kernel](../sources/prs/sglang/PR-15522.md), [[JIT kernel] Apply jit per_tensor_quant_fp8 kernel](../sources/prs/sglang/PR-15836.md), [Fix FP8 MoE NaN with DeepGEMM on Blackwell](../sources/prs/sglang/PR-16622.md), [[GLM 4.7] Add RTX 6000 Pro aka sm120](../sources/prs/sglang/PR-17235.md), [Disable mla persistent kernel when not using fp8 kv_cache](../sources/prs/sglang/PR-17327.md), [Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE](../sources/prs/sglang/PR-17449.md), [[ROCm] Optimize Deepseek R1 on MI300X](../sources/prs/sglang/PR-18242.md), [Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4](../sources/prs/sglang/PR-18389.md), [[AMD] Update aiter to v0.1.10.post2](../sources/prs/sglang/PR-18423.md), [Fp8 prefill attn kernel integration](../sources/prs/sglang/PR-18528.md), [[AMD] DSR1/V3 use fp8 bmm in MLA for MI300X](../sources/prs/sglang/PR-18624.md), [[RL] Support per-layer mixed FP8/BF16 serving for FP8 checkpoints](../sources/prs/sglang/PR-18742.md), [Fix NSA FP8 KV cache path for both-trtllm MHA one-shot](../sources/prs/sglang/PR-18931.md), [[FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE](../sources/prs/sglang/PR-19537.md), [Various SM120 improvements](../sources/prs/sglang/PR-19721.md), [[AMD] Fix FP8 assertion failure in aiter MLA decode by falling back to self.k_scale](../sources/prs/sglang/PR-19935.md), [Enable modelopt quantized FLUX deployment](../sources/prs/sglang/PR-20082.md), [[AMD] Fp8 prefill integration with radix cache path for dpsk models](../sources/prs/sglang/PR-20187.md), [[Benchmark] use flashinfer bench_gpu_time instead of triton do_bench](../sources/prs/sglang/PR-20305.md), [[NVIDIA] Enable fp8 flashinfer_trtllm_routed MoE for MiniMax-M2.5](../sources/prs/sglang/PR-20394.md), [Support Triton MLA FP8 KV cache](../sources/prs/sglang/PR-20479.md), [FIX: (NSA) Compute topk_indices_offset when NSA prefill flashmla_sparse is used with FP8 KV cache](../sources/prs/sglang/PR-20606.md), [CUTLASS FP8 Blockwise GEMM improvement of SM120](../sources/prs/sglang/PR-20887.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [ci: run Stage A CUDA tests as stage-a-test-small-1-gpu on 5090](../sources/prs/sglang/PR-20988.md), [[AMD]: Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5…](../sources/prs/sglang/PR-21213.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [[RL] Support mxfp8 DeepSeek V3](../sources/prs/sglang/PR-21280.md), [[Kernel] Support FlashInfer TRTLLM-Gen fused MoE for non-gated FP4 & FP8 (Nemotron)](../sources/prs/sglang/PR-21321.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [Migrate all callers from /get_server_info to /server_info](../sources/prs/sglang/PR-21463.md), [[AMD] Enable FP8 KV cache and FP8 attention kernel for NSA on MI300/MI355 with TileLang backend](../sources/prs/sglang/PR-21511.md), [[FlashInver v0.6.7] Integrate flashinfer_trtllm mxfp8 gemm](../sources/prs/sglang/PR-21576.md), [[AMD] Add GLM-5-FP8 nightly performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-21710.md), [[Misc] [MXFP8] Drop sm100 mxfp8 warning](../sources/prs/sglang/PR-21881.md), [fix pcg torch dynamo recompile in mxfp8 Triton path](../sources/prs/sglang/PR-21888.md), [Tiny fix trtllm_fp8_per_tensor_scale_moe_wrapper router_logits dtype](../sources/prs/sglang/PR-22006.md), [[AMD][HIP] NSA: bf16 passthrough from RMSNorm to eliminate FP8 dequantization](../sources/prs/sglang/PR-22258.md), [[AMD] Fix GLM-5 fp8 KV quant path dispatch on MI300](../sources/prs/sglang/PR-22314.md), [[Reland] DeepSeek-R1-0528-w4a8: DeepEP Low Latency Dispatch Adopts FP8 Communication](../sources/prs/sglang/PR-22316.md), [[Lora] Lora quat info re-factor and support deepseekv3 mla lora](../sources/prs/sglang/PR-22323.md), [[AMD] Add GLM-5.1-FP8 nightly accuracy and performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-22336.md), [[Diffusion] modelopt diffusion fp8 support for flux1/flux2 and wan2.2](../sources/prs/sglang/PR-22365.md), [[DSA] Hopper FP8 FlashMLA KV padding](../sources/prs/sglang/PR-22372.md), [[RL] Fix weight update for mxfp8 flashinfer_cutlass gemm backend](../sources/prs/sglang/PR-22484.md), [[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22574.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [[MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute](../sources/prs/sglang/PR-23760.md), [Refactor device timer, clean up metrics collector, and add fwd occupancy metric](../sources/prs/sglang/PR-24197.md), [Port MXFP4 Marlin MoE support to JIT kernel path](../sources/prs/sglang/PR-24490.md), [[Codex] Opt Mistral Large performace ](../sources/prs/sglang/PR-24611.md), [[Gemma4] Optimize Gemm4 with fused Q/K/V RMSNorm + per-expert FP8 ckpt loader](../sources/prs/sglang/PR-24696.md), [Add FlashInfer SM90 cutlass MXFP4 MoE backend (W4A16) for GPT-OSS + DeepSeek-V4](../sources/prs/sglang/PR-24816.md), [[attn backend] Integrate tokenspeed_mla prefill/decode kernels (fp8 kv cache, blackwell)](../sources/prs/sglang/PR-24925.md), [Amd/deepseek v4 rebase main 0509](../sources/prs/sglang/PR-24933.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [[Gemma4]: Fix FP8 Triton scale layout](../sources/prs/sglang/PR-25286.md), [[Intel GPU] Enable DeepSeek V4 Inference on XPU](../sources/prs/sglang/PR-25336.md), [[fp8] SM90 swap-AB scaled_mm dispatch (~1.16x kernel geomean, +5.8-18.5% end-to-end)](../sources/prs/sglang/PR-25532.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [support w8a8 fp8 kernel with CUTLASS](../sources/prs/sglang/PR-3047.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [Apply sgl w8a8 fp8 kernel](../sources/prs/sglang/PR-3148.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [add control for cutlass fp8 blockwise gemm](../sources/prs/sglang/PR-3727.md), [[tools] add fp8 max/min constant in utils](../sources/prs/sglang/PR-3959.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [linear support deepgemm](../sources/prs/sglang/PR-4199.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [Clean up fp8 support](../sources/prs/sglang/PR-4230.md), [fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0](../sources/prs/sglang/PR-4231.md), [Support Blackwell Block Scale FP8 Gemm](../sources/prs/sglang/PR-4278.md), [[FIX] fix incorrect output when enable both deepgemm and torch compile](../sources/prs/sglang/PR-4359.md), [[ROCm] fix dtype](../sources/prs/sglang/PR-4510.md), [Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt](../sources/prs/sglang/PR-4515.md), [Support fp8 gemm for blackwell](../sources/prs/sglang/PR-4558.md), [[quantization] fix channelwise conversion with scalar weight scale](../sources/prs/sglang/PR-4596.md), [Set deepgemm to the default value in the hopper architecture.](../sources/prs/sglang/PR-4613.md), [Add DeepSeek V3/R1 shared experts fusion](../sources/prs/sglang/PR-4918.md), [feat: add DeepGEMM build warning](../sources/prs/sglang/PR-5176.md), [[Fix] Turn off DeepGEMM by default](../sources/prs/sglang/PR-5263.md), [[1/2] Add FP8 Blockscale MoE CUTLASS kernel for Blackwell](../sources/prs/sglang/PR-5281.md), [fix: use deepgemm only on hopper](../sources/prs/sglang/PR-5310.md), [[perf] experimental enhance fp8 per-tensor quant](../sources/prs/sglang/PR-5370.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [[feature] enable pre compile jit deep_gemm](../sources/prs/sglang/PR-5580.md), [ DeepEP normal support deepgemm-contiguous](../sources/prs/sglang/PR-5626.md), [[perf] dsv3 bmm fallback to bf16](../sources/prs/sglang/PR-5662.md), [[2/2] Add python wrapper for CUTLASS FP8 Blockscale MoE Kernel. ](../sources/prs/sglang/PR-5694.md), [cutlass 3.9 supported to improve fp8_blockwise_gemm](../sources/prs/sglang/PR-5820.md), [chore: upgrade cutlass 3.9.2](../sources/prs/sglang/PR-6004.md), [Upgrade CUTLASS 4.0](../sources/prs/sglang/PR-6336.md), [Add fp8 fused_experts kernel for CPU in sgl-kernel and add UT](../sources/prs/sglang/PR-6404.md), [Fix bug of deepseek-v3 under DP+EP mode with large batchsize/seqlen](../sources/prs/sglang/PR-6449.md), [[Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell](../sources/prs/sglang/PR-6479.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [[CPU] add optimizations for INT8 and FP8 DeepSeek](../sources/prs/sglang/PR-6769.md), [feat: integrate deepgemm into EPMoE](../sources/prs/sglang/PR-6821.md), [CPU: map changes from developing branch in sgl-kernel](../sources/prs/sglang/PR-6833.md), [Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa](../sources/prs/sglang/PR-6890.md), [Add a CUDA kernel for fusing mapping and weighted sum for MoE.](../sources/prs/sglang/PR-6916.md), [[Feature] Support Flashinfer fmha on Blackwell](../sources/prs/sglang/PR-6930.md), [Fuse routed scaling factor in deepseek](../sources/prs/sglang/PR-6970.md), [Update default settings for blackwell](../sources/prs/sglang/PR-7023.md), [Fix positional argument](../sources/prs/sglang/PR-7093.md), [fix amd EP MoE FP8 issue](../sources/prs/sglang/PR-7125.md), [Enable ModelOpt Llama4 fp8 checkpoint deployment in SGLang](../sources/prs/sglang/PR-7129.md), [[amd] Opt dsv3 moe](../sources/prs/sglang/PR-7160.md), [Support new DeepGEMM](../sources/prs/sglang/PR-7172.md), [Tiny let DeepGEMM scale checks cover more cases](../sources/prs/sglang/PR-7182.md), [[AMD] Fail gracefully when AITER is unavailable gfx90a GPUs](../sources/prs/sglang/PR-7187.md), [[fix] fix DeepGEMM blackwell input quant & ut & fix style and log](../sources/prs/sglang/PR-7247.md), [Add CUTLASS FP8 Blockscale MoE kernel for Hopper architecture](../sources/prs/sglang/PR-7278.md), [Fix torch compile run](../sources/prs/sglang/PR-7391.md), [[AMD][Quantization] Add `int4fp8_moe` online quantization on ROCm](../sources/prs/sglang/PR-7392.md), [feat: support DeepSeek-R1-W4AFP8 model with ep-moe mode](../sources/prs/sglang/PR-7762.md), [Qwen FP8/NVFP4 ModelOPT Quantization support](../sources/prs/sglang/PR-7912.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [[sgl-kernel] Opt per_token_quant_fp8 with warp reduce](../sources/prs/sglang/PR-8130.md), [[1/N]Support DeepSeek-R1 w4a8 normal deepep](../sources/prs/sglang/PR-8247.md), [[2/N]Support DeepSeek-R1 w4a8 low latency deepep](../sources/prs/sglang/PR-8464.md), [TRTLLM-MLA FP8 path](../sources/prs/sglang/PR-8638.md), [feat: support cutlass_moe_fp8 kernel for fusedmoe in sm90](../sources/prs/sglang/PR-8678.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/sglang/PR-8818.md), [optimize: reduce shulffle and quantization overhead in cutlass_moe sm90](../sources/prs/sglang/PR-8962.md), [[fix]: fix cutlass moe ut and and Opt H20 cutlass groupGemm performance](../sources/prs/sglang/PR-9272.md), [[sgl-kernel] feat: Support sm120 cutlass fp8 gemm kernel](../sources/prs/sglang/PR-9403.md), [fix: blackwell dsv3 fp8 issue temporary solution](../sources/prs/sglang/PR-9530.md), [Update CUTLASS 4.2 & Enable K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM](../sources/prs/sglang/PR-9559.md), [move is_sm90_supported/is_sm100_supported to python/sglang/srt/utils.py](../sources/prs/sglang/PR-9679.md), [[CPU] Add FP8 Bmm support](../sources/prs/sglang/PR-9744.md), [Make sm100 fp8 kernels available on sm103](../sources/prs/sglang/PR-9789.md), [CUTLASS fp8 blockwise gemm support of sm120](../sources/prs/sglang/PR-9969.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support](../sources/prs/vllm/PR-10995.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [Add: Support for Sparse24Bitmask Compressed Models](../sources/prs/vllm/PR-12097.md), [Expert Parallelism (EP) Support for DeepSeek Models](../sources/prs/vllm/PR-12583.md), [[Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3](../sources/prs/vllm/PR-12587.md), [[Attention] Deepseek v3 MLA support with FP8 compute](../sources/prs/vllm/PR-12601.md), [[Attention] MLA with chunked prefill](../sources/prs/vllm/PR-12639.md), [[AMD][ROCm] Enable DeepSeek model on ROCm](../sources/prs/vllm/PR-12662.md), [[Bugfix] Better FP8 supported defaults](../sources/prs/vllm/PR-12796.md), [[Kernel]Add streamK for block-quantized CUTLASS kernels](../sources/prs/vllm/PR-12978.md), [[core] Perf improvement for DSv3 on AMD GPUs](../sources/prs/vllm/PR-13718.md), [[V1] V1 Enablement Oracle ](../sources/prs/vllm/PR-13726.md), [[Bugfix][Quantization] Fix FP8 + EP](../sources/prs/vllm/PR-13784.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[Kernel] CUTLASS grouped gemm fp8 MoE kernel](../sources/prs/vllm/PR-13972.md), [[core] moe fp8 block quant tuning support](../sources/prs/vllm/PR-14068.md), [dynamic distpatch of fp8 kernels](../sources/prs/vllm/PR-14245.md), [Add cutlass support for blackwell fp8 blockwise gemm](../sources/prs/vllm/PR-14383.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [[ROCm][Kernel] MoE weights padding](../sources/prs/vllm/PR-14454.md), [permute/unpermute kernel for moe optimization](../sources/prs/vllm/PR-14568.md), [[Attention] Flash Attention 3 - fp8](../sources/prs/vllm/PR-14570.md), [[Quantization][FP8] Adding support for fp8 gemm layer input in fp8](../sources/prs/vllm/PR-14578.md), [[Attention] MLA get rid of materialization](../sources/prs/vllm/PR-14770.md), [[FEAT][ROCm] Integrate Fused MoE Kernels from AITER](../sources/prs/vllm/PR-14967.md), [[FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature](../sources/prs/vllm/PR-14968.md), [[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM](../sources/prs/vllm/PR-15587.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [Upstream Llama4 Support to Main](../sources/prs/vllm/PR-16113.md), [[Kernel] Support W8A8 channel-wise weights and per-token activations in triton fused_moe_kernel](../sources/prs/vllm/PR-16366.md), [Enable PTPC FP8 for CompressedTensorsW8A8Fp8MoEMethod (triton fused_moe)](../sources/prs/vllm/PR-16537.md), [[ROCm] Add aiter tkw1 kernel for Llama4 fp8](../sources/prs/vllm/PR-16727.md), [[Kernel] some optimizations for dense marlin and moe marlin](../sources/prs/vllm/PR-16850.md), [[Kernel] Add expert_map support to Cutlass FP8 MOE](../sources/prs/vllm/PR-16861.md), [[FEAT] [ROCm]: Add AITER CK 2 Stages MoE support](../sources/prs/vllm/PR-17110.md), [[ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention](../sources/prs/vllm/PR-17139.md), [[NVIDIA] Support Cutlass w8a8 FP8 for Blackwell Geforce GPUs (sm120)](../sources/prs/vllm/PR-17280.md), [[Kernel] fp4 marlin kernel](../sources/prs/vllm/PR-17687.md), [use ceil_div in cutlass block scaling shape check](../sources/prs/vllm/PR-17918.md), [[Feature] Expert Parallelism Load Balancer (EPLB)](../sources/prs/vllm/PR-18343.md), [[V1] Support `LLM.apply_model`](../sources/prs/vllm/PR-18465.md), [Sm100 blockwise fp8 swap ab](../sources/prs/vllm/PR-18564.md), [[Kernel] Integrate CUTLASS MoE kernel with PPLX](../sources/prs/vllm/PR-18762.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-18778.md), [[Kernel] Enable fp8 support for pplx and BatchedTritonExperts.](../sources/prs/vllm/PR-18864.md), [[Kernel] Support deep_gemm for linear methods](../sources/prs/vllm/PR-19085.md), [[Kernels] Add activation chunking logic to FusedMoEModularKernel](../sources/prs/vllm/PR-19168.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [[Feature] Integrate new deepgemm](../sources/prs/vllm/PR-19820.md), [ [Feature] Integrate SM100 DeepGEMM support](../sources/prs/vllm/PR-20087.md), [[Bugfix] Fix topk_ids indices_type for CUTLASS w8a8 FP8 MoE](../sources/prs/vllm/PR-20166.md), [[V1] [ROCm] Enable EP with AITER Fused MoE](../sources/prs/vllm/PR-20270.md), [[Kernel] SM90 CUTLASS FP8 GEMM: add support for swap AB + kernel tuning](../sources/prs/vllm/PR-20396.md), [[feat]: add SM100 support for cutlass FP8 groupGEMM](../sources/prs/vllm/PR-20447.md), [Support Llama 4 for fused_marlin_moe](../sources/prs/vllm/PR-20457.md), [[Perf] Reuse workspace for FP8+FP4 Marlin MoE](../sources/prs/vllm/PR-20500.md), [[Performance] Performance improvements in non-blockwise fp8 CUTLASS MoE](../sources/prs/vllm/PR-20762.md), [[Perf] Use Triton instead of Torch for DeepGEMM Per Token Group Quant](../sources/prs/vllm/PR-20841.md), [[Perf] Add swap_ab to SM90 FP8 non-block CUTLASS moe grouped gemm](../sources/prs/vllm/PR-20911.md), [[Perf] Cuda Kernel for Per Token Group Quant](../sources/prs/vllm/PR-21083.md), [[perf] Add fused MLA QKV + strided layernorm](../sources/prs/vllm/PR-21116.md), [[Feature][OCP MX] Support mxfp6 and mixed mxfp6-mxfp4](../sources/prs/vllm/PR-21166.md), [[Bug] DeepGemm: Fix TypeError: per_block_cast_to_fp8() missing 1 required positional argument: 'use_ue8m0' for SM100](../sources/prs/vllm/PR-21187.md), [[NVIDIA] Explicitly disable shuffled weights for flashinfer blockscale moe fp8 kernels](../sources/prs/vllm/PR-21411.md), [[Bugfix][CUDA] fixes CUDA FP8 kv cache dtype supported](../sources/prs/vllm/PR-21420.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv/out Attention Kernel](../sources/prs/vllm/PR-21716.md), [[Kernel] Add support for block FP8 on SM120 (NVIDIA 5090 and RTX PRO 6000)](../sources/prs/vllm/PR-22131.md), [Fp8 paged attention update](../sources/prs/vllm/PR-22222.md), [[Bug] Fix B200 DeepGEMM E8M0 Accuracy Issue](../sources/prs/vllm/PR-22399.md), [[Quantization] Expand compressed-tensors MoE matching logic to support NFP4 + FP8 MoEs](../sources/prs/vllm/PR-22674.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv NVFP4-out Attention Kernel](../sources/prs/vllm/PR-22703.md), [fp8 kv cache support fix for torch.compile](../sources/prs/vllm/PR-22758.md), [[Kernel] Added flashinfer fp8 per-tensor gemms](../sources/prs/vllm/PR-22895.md), [[Bugfix] fix qwen3 moe fp8 accuracy issue](../sources/prs/vllm/PR-23031.md), [[Kernel] CUTLASS MoE FP8: Integrate cuda moe permute/unpermute](../sources/prs/vllm/PR-23045.md), [Add routed_scaling_factor to MoE grouped topk](../sources/prs/vllm/PR-23123.md), [[XPU][Feature] fp8 online quantization support for XPU](../sources/prs/vllm/PR-23148.md), [[kernel] Support W4A8 on Hopper](../sources/prs/vllm/PR-23198.md), [[ROCm][Aiter] Add triton fp8 bmm kernel for mla](../sources/prs/vllm/PR-23264.md), [[Perf] Small optimizations for silu_mul_fp8_quant_deep_gemm](../sources/prs/vllm/PR-23265.md), [[Kernels] Overlap shared experts with send/recv](../sources/prs/vllm/PR-23273.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Bug] Fix R1 Accuracy 0 Bug](../sources/prs/vllm/PR-23294.md), [DP/EP Support for gpt-oss with deepep-ht comm kernel on SM100](../sources/prs/vllm/PR-23608.md), [[Flashinfer] Support Flashinfer TRTLLM FP8-qkv BF16/FP16-out Attention Kernel](../sources/prs/vllm/PR-23647.md), [[Feature] Add Hopper DeepGEMM E8M0 for DeepSeekV3.1 scale_fmt](../sources/prs/vllm/PR-23666.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Model] Add LongCat-Flash ](../sources/prs/vllm/PR-23991.md), [[Performance] Move apply_w8a8_block_fp8_linear to an op class](../sources/prs/vllm/PR-24666.md), [[NVIDIA] Blackwell Family](../sources/prs/vllm/PR-24673.md), [[Kernel][Quantization] add w4a8 support for marlin kernel](../sources/prs/vllm/PR-24722.md), [[Flashinfer][gpt-oss] Support FP8-qkv Flashinfer TRTLLM Sinks Attention](../sources/prs/vllm/PR-25674.md), [[Bugfix] Convert untraceable GroupShape to list for AMD impl](../sources/prs/vllm/PR-26535.md), [[Bugfix] Fix gpt-oss w4a8 DP/EP on B200](../sources/prs/vllm/PR-26729.md), [[Feature] Batch Invariant: Support DeepGEMM and Blackwell](../sources/prs/vllm/PR-27127.md), [[Kernels] Enable FlashInfer FP8 Blockscale on SM90 (for TEP DSR1)](../sources/prs/vllm/PR-27134.md), [[torch.compile] Enable silu_mul_fp8_quant fusion without custom ops enabled](../sources/prs/vllm/PR-27146.md), [Bugfix: Cutlass FP8 FusedMoE bad scaling factors](../sources/prs/vllm/PR-27255.md), [Feature: Support Relu2 in FusedMoE fp8 cutlass path](../sources/prs/vllm/PR-27261.md), [[Perf] SM100 - add swap AB optimization to CUTLASS FP8 GEMM](../sources/prs/vllm/PR-27284.md), [[Performance] Support FP8 flashinfer TRTLLM MOE on Qwen3 and Qwen-3next](../sources/prs/vllm/PR-27492.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [[Feature] Batch invariant torch.compile](../sources/prs/vllm/PR-27660.md), [[Performance] Fused blockwise quant RMS norm](../sources/prs/vllm/PR-27883.md), [[Performance][B200] Fix deepgemm prologue](../sources/prs/vllm/PR-27897.md), [[ROCm][MLA] enable fp8 MLA decode on ROCm](../sources/prs/vllm/PR-28032.md), [[Performance][B200] silu_mul_quant: pack scales in int32](../sources/prs/vllm/PR-28358.md), [[Performance] Reduce DeepGEMM N dim restriction from 128 to 64 multiplier ](../sources/prs/vllm/PR-28687.md), [[Perf][Kernels] Enable FlashInfer DeepGEMM swapAB on SM90 (for W8A8 Linear Op)](../sources/prs/vllm/PR-29213.md), [chore: add RTX_PRO_6000 GLM4.6-FP8 kernel tuning](../sources/prs/vllm/PR-29240.md), [[Perf] Disable DeepGEMM MoE by default when TP=8 is used](../sources/prs/vllm/PR-29346.md), [[Kernel]Support W4A8 Grouped GEMM on Hopper](../sources/prs/vllm/PR-29691.md), [[MoE-FP8-modelopt] Add FlashInfer alignment padding for intermediate dimensions](../sources/prs/vllm/PR-29748.md), [Add Mistral Large 3 and Ministral 3](../sources/prs/vllm/PR-29757.md), [[Perf] Improve fp8 quant in mla; replace ReduceSum with ReduceScatterSum](../sources/prs/vllm/PR-29795.md), [[Bugfix] Fix FP8 MoE LoRA](../sources/prs/vllm/PR-29890.md), [[Kernel][Quantization][MoE] add marlin kernel support for turing (sm75)](../sources/prs/vllm/PR-29901.md), [[ROCm] add fallback for aiter fp8 decode mla](../sources/prs/vllm/PR-30005.md), [[Quantization] Support Quark int4-fp8 w4a8 for MoE](../sources/prs/vllm/PR-30071.md), [Add llmcompressor fp8 kv-cache quant (per-tensor and per-attn_head)](../sources/prs/vllm/PR-30141.md), [[Bugfix] Fix DeepGEMM after #29546 ](../sources/prs/vllm/PR-30267.md), [[LoRA] Support Quantized Adapters](../sources/prs/vllm/PR-30286.md), [[Bugfix] Fix fp8 DeepGemm compilation issues](../sources/prs/vllm/PR-30336.md), [[ROCm][Quantization] GPT OSS Upstream MoE wmxfp4_afp8 with static scales](../sources/prs/vllm/PR-30357.md), [[Feature] Add SM103 (Blackwell Ultra) Support to vLLM](../sources/prs/vllm/PR-30484.md), [[Perf] Eliminate padding and slicing op for GPT-OSS with Flashinfer MXFP4 MXFP8 MoE](../sources/prs/vllm/PR-30647.md), [[SM100] Enable fp8 compute for prefill MLA](../sources/prs/vllm/PR-30746.md), [[Feature]: Support NVIDIA ModelOpt HF FP8 variants FP8_PER_CHANNEL_PER_TOKEN and FP8_PB_WO in vLLM](../sources/prs/vllm/PR-30957.md), [[Bugfix][Hardware][AMD] Consolidate FP8 min/max values helper function](../sources/prs/vllm/PR-31106.md), [[Bugfix][Hardware][AMD] Fix exception types in AITER MLA FP8 check](../sources/prs/vllm/PR-31177.md), [[SM100] Resubmit FMHA FP8 prefill for MLA](../sources/prs/vllm/PR-31195.md), [[Bugfix][ROCm] Fix Static Quant Issue](../sources/prs/vllm/PR-31502.md), [[Bugfix] Fix Broken ModelOpt NVFP4 MoE](../sources/prs/vllm/PR-31742.md), [[1/N][Attention] Restructure attention: move files](../sources/prs/vllm/PR-31916.md), [[5/N][Attention] Finish eliminating `vllm/attention` folder](../sources/prs/vllm/PR-32064.md), [[Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper](../sources/prs/vllm/PR-32619.md), [[Bugfix] Fix FP8 MoE EP Weight Loading for ModelOpt Llama4](../sources/prs/vllm/PR-32886.md), [Support compress-tensors with nvfp4 or fp8 weights and modelopt with nvfp4 weights on Turing](../sources/prs/vllm/PR-33076.md), [Add support for Mistral Large 3 inference with Flashinfer MoE](../sources/prs/vllm/PR-33174.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Bugfix] Register fp8 cutlass_group_gemm as supported for only SM90+SM100](../sources/prs/vllm/PR-33285.md), [[Kernel] Support Flashinfer trtllm fused MoE non gated FP8 & NVFP4](../sources/prs/vllm/PR-33506.md), [[Kernel] Add enable_sm120_or_later for SM121 (DGX Spark) CUTLASS support](../sources/prs/vllm/PR-33517.md), [[Perf] Disable clean_logits in deepgemm fp8_mqa_logits kernel](../sources/prs/vllm/PR-33568.md), [enable skipping of SW attention layers when using FP8 KV cache](../sources/prs/vllm/PR-33695.md), [[Kernel] Integrate SM100 MXFP8 blockscaled grouped MM and quant kernels](../sources/prs/vllm/PR-34448.md), [[Bugfix] Handle num_expert_group=None in flashinfer block-scale FP8 MoE](../sources/prs/vllm/PR-34494.md), [[Kernel] Add FP8 KV cache support to Triton MLA decode attention](../sources/prs/vllm/PR-34597.md), [[Bugfix] Fix NVFP4 TRTLLM MoE non-gated support; add gsm8k for Nemotron-3-Nano FP8+NVFP4](../sources/prs/vllm/PR-34725.md), [Integrate flashinfer mm_mxfp8 in ModelOpt MXFP8](../sources/prs/vllm/PR-35053.md), [[Feat] Add CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function](../sources/prs/vllm/PR-35271.md), [[Attention][Perf] Optimize cp_gather_and_upconvert_fp8_kv_cache - DeepSeek-v3.2](../sources/prs/vllm/PR-35290.md), [[Quant][Feature] Support online MXFP8 quantization for MoE and dense models](../sources/prs/vllm/PR-35448.md), [[ROCm] Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5/Linear)](../sources/prs/vllm/PR-35850.md), [[Perf] Support FP8 KV cache for Flashinfer MLA Sparse](../sources/prs/vllm/PR-35891.md), [Add support for ModelOpt MXFP8 MoE models](../sources/prs/vllm/PR-35986.md), [[Bugfix] Fix passing of activation_type to trtllm fused MoE NVFP4 and FP8](../sources/prs/vllm/PR-36017.md), [[mla] Support fused FP8/NVFP4 output quantization in MLA attention (#35792)](../sources/prs/vllm/PR-36205.md), [[Perf] Add TRTLLM FP8 MoE Modular Kernel](../sources/prs/vllm/PR-36307.md), [[XPU] Support block fp8 moe by fallback to TritonExpert on XPU](../sources/prs/vllm/PR-36458.md), [[Kernel] Fuse FP8 output quantization into merge_attn_states](../sources/prs/vllm/PR-36518.md), [[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts](../sources/prs/vllm/PR-36728.md), [[Bugfix] Fix KV scales inconsistency in fp8 MLA & FlashInfer kv_cache_dtype "auto" leading to gibberish](../sources/prs/vllm/PR-37054.md), [[XPU] support MLA model on Intel GPU](../sources/prs/vllm/PR-37143.md), [[Perf] Set Flashinfer sparse MLA as default backend for FP8 kv cache](../sources/prs/vllm/PR-37252.md), [[Bugfix] Disable monolithic TRTLLM MoE for Renormalize routing (#37591)](../sources/prs/vllm/PR-37605.md), [[Perf] Use torch compile to fuse pack topk in trtllm moe](../sources/prs/vllm/PR-37695.md), [[Bug] Fix fp8 deepgemm batch invariant](../sources/prs/vllm/PR-37718.md), [[Kernel] Optimize SM120 CUTLASS blockwise FP8 GEMM](../sources/prs/vllm/PR-37970.md), [[Perf] FP8 FlashInfer Attn for ViT](../sources/prs/vllm/PR-38065.md), [[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell](../sources/prs/vllm/PR-38083.md), [[Kernel] Add swapAB support for SM120 CUTLASS blockwise FP8 GEMM ](../sources/prs/vllm/PR-38325.md), [[MoE] Add RoutingMethodType.Simulated to TRT-LLM FP8/NVFP4 kernel allowlists](../sources/prs/vllm/PR-38329.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[QeRL] Fix online quantized reloading](../sources/prs/vllm/PR-38442.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[XPU] add xpu backend implementation of mxfp8 quant](../sources/prs/vllm/PR-38682.md), [[Quant] add CompressedTensorsW8A8Mxfp8 for linear and MoE layers](../sources/prs/vllm/PR-38815.md), [[Bugfix] Re-enable Renormalize routing for TRT-LLM MoE experts](../sources/prs/vllm/PR-38859.md), [[Bugfix] Fix broken explicit unquantized kv cache dtype support](../sources/prs/vllm/PR-38922.md), [[MoE Refactor] Split up compressed_tensors_moe.py](../sources/prs/vllm/PR-38960.md), [[Bug] Fix routing bias dtype for trtllm per-block fp8 moe](../sources/prs/vllm/PR-38989.md), [[Perf] Change Trtllm fp8 MoE to use Shuffled Weights and BlockMajorK Layout](../sources/prs/vllm/PR-38993.md), [[Bug] Fix Trtllm Fp8 MoE Weight Shuffle Memory Fragamentation](../sources/prs/vllm/PR-39054.md), [perf(moe): add tuned fused_moe config for RTX PRO 6000 Blackwell Server Edition](../sources/prs/vllm/PR-39183.md), [[Refactor] Move MXFP8 GEMM management into MxFp8LinearKernel](../sources/prs/vllm/PR-39205.md), [[Perf] Fuse Zero Initializer for FP8 DeepGemm Block Quant Kernel](../sources/prs/vllm/PR-39547.md), [add warning when FP8 KV cache misses prefill query quantization](../sources/prs/vllm/PR-39752.md), [[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement](../sources/prs/vllm/PR-40408.md), [[MoE] Move cutlass moe to fused_moe/experts/](../sources/prs/vllm/PR-40574.md), [[Kernel][Helion] Optimize Helion config parsing latency](../sources/prs/vllm/PR-40850.md), [[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided](../sources/prs/vllm/PR-40960.md), [Faster per-token fp8 group quant packed kernel for blackwell](../sources/prs/vllm/PR-41326.md), [[Quantization] Rework quantization_config to use QuantKey and allow for activation override](../sources/prs/vllm/PR-41566.md), [[CPU] Add MXFP4 W4A16 MoE support](../sources/prs/vllm/PR-41922.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Bugfix] Add swiglu limits to deepgemm fp8 methods](../sources/prs/vllm/PR-41986.md), [[feat] Add FP8 per-tensor Q scale support to Triton attention backend](../sources/prs/vllm/PR-42080.md), [[Perf] Use 2D-grid to eliminate divmod in W8W8 group quant](../sources/prs/vllm/PR-42153.md), [[Perf] Wire silu_and_mul_per_block_quant into TritonFP8MoE (MiniMax-M2) ](../sources/prs/vllm/PR-42497.md), [[6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued)](../sources/prs/vllm/PR-42663.md), [[Perf] Add do_not_specialize in fused FP8 RoPE kernel](../sources/prs/vllm/PR-42849.md), [add cutedsl dsv4 indexer fp8 kernel](../sources/prs/vllm/PR-42899.md), [[Model Refactoring] Move deepseek_v4_ops to models/deepseek_v4 [3/N]](../sources/prs/vllm/PR-43073.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../wiki/kernels/flashmla.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [Sparse MLA (DeepSeek V3.2)](../wiki/kernels/sparse-mla.md), [Fine-Grained FP8/FP4 Quantization](../wiki/techniques/fine-grained-quantization.md) | +| `fp8` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../sources/blogs/flashmla.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [DeepSeek-V3.2-Exp in vLLM: Fine-Grained Sparse Attention in Action](../sources/blogs/vllm-deepseek-v3-sparse-attention.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [NVIDIA Ampere GA102 GPU Architecture Whitepaper (v2.1)](../sources/docs/nvidia-ga102-whitepaper.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Use swizzling instead of padding](../sources/prs/DeepGEMM/PR-86.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[TRTLLM-9457][feat] Add cute dsl fp8 gemm for Blackwell](../sources/prs/TensorRT-LLM/PR-10130.md), [[None][fix] impl fused triton kernel for e8m0 resmooth to reduce memory footprint](../sources/prs/TensorRT-LLM/PR-10327.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[None][feat] Add support for expert_number<=2048 and K<=32](../sources/prs/TensorRT-LLM/PR-11510.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[TRTLLM-10421][perf] Add fused cat+fp8_quantize CUDA kernel for DSA indexer](../sources/prs/TensorRT-LLM/PR-11899.md), [[None][feat] GLM 5 support and DSA MTP fixes](../sources/prs/TensorRT-LLM/PR-11990.md), [[https://nvbugs/5879577][fix] Fix KeyError in DeepSeekV3Lite FP8 MTP weight loading](../sources/prs/TensorRT-LLM/PR-12530.md), [[#12784][feat] AutoDeploy: Optimize DeepSeek-R1 model performance](../sources/prs/TensorRT-LLM/PR-12946.md), [[TRTLLM-34871][feat] Add cute dsl FP8 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13219.md), [[TRTLLM-11127][feat] add W4A8_MXFP4_FP8 MoE unit test support](../sources/prs/TensorRT-LLM/PR-13401.md), [[None][feat] Fuse FP8 1x128 quantize + UE8M0 scale pack on SM100](../sources/prs/TensorRT-LLM/PR-13628.md), [[None][fix] Plumb swiglu_limit through DeepGEMM and TRTLLMGen FP8 fused MoE](../sources/prs/TensorRT-LLM/PR-13767.md), [[TRTLLM-12503][feat] Parallel VAE independent scaling and fix arg passing](../sources/prs/TensorRT-LLM/PR-13873.md), [[TRTLLM-35237][feat] Add cute dsl FP4 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13929.md), [[None][feat] Keep DSv4 o_a_proj as FP8, and port vLLM's fused_inv_rope_fp8_quant](../sources/prs/TensorRT-LLM/PR-13938.md), [[TRTLLM-12462][fix] Fix FP8 block scaling GEMM autotuner cache growth](../sources/prs/TensorRT-LLM/PR-14165.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[TRTLLM-8535][feat] Support DeepSeek V3.2 with FP8 + BF16 KV cache/NVFP4 + BF16 KV cache](../sources/prs/TensorRT-LLM/PR-8405.md), [[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP](../sources/prs/TensorRT-LLM/PR-8501.md), [[None][feat] TRT-LLM Gen MoE optimize DeepSeek Fp8 activation kernel](../sources/prs/TensorRT-LLM/PR-9175.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Hopper Grouped GEMM support for FP8 Accum](../sources/prs/cutlass/PR-2123.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [support fp16 accmulator for sm89 fp8 mma](../sources/prs/cutlass/PR-2378.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [[Cutlass gemm] Fix SM100 FP8 nosmem epilogue-fusion shape_div 'Divisibility Condition' for non-multiple-of-64 N tiles](../sources/prs/cutlass/PR-2946.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG](../sources/prs/flash-attention/PR-1173.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [[Cute,Fwd,Sm100] fp8 e4m3 and e5m2 support](../sources/prs/flash-attention/PR-2109.md), [feat: add functional per-head FP8 quantization for FA3](../sources/prs/flashinfer/PR-1033.md), [perf: accelerate blackwell grouped gemm](../sources/prs/flashinfer/PR-1086.md), [bugfix: fix fp8 attention kernels aot compilation issue](../sources/prs/flashinfer/PR-1087.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [[Feature] Support PDL for batch Prefill and Decode](../sources/prs/flashinfer/PR-1117.md), [Add DeepGEMM kernels](../sources/prs/flashinfer/PR-1209.md), [Fix test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-1211.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [Patch fp8 cubin availability](../sources/prs/flashinfer/PR-1240.md), [feat: Support MXFP8 x MXFP4 CUTLASS grouped GEMM](../sources/prs/flashinfer/PR-1241.md), [Add trtllm-gen attention mha kernel with FP8 Q/K/V and FP8 output](../sources/prs/flashinfer/PR-1242.md), [Reduce the JIT compilation time of gen_gemm_sm100_module](../sources/prs/flashinfer/PR-1251.md), [init add gemm fp8 using cudnn backend](../sources/prs/flashinfer/PR-1264.md), [feat: add masked deepgemm support and benchmarking](../sources/prs/flashinfer/PR-1266.md), [Unify groupwise fp8 GEMM test](../sources/prs/flashinfer/PR-1281.md), [Bug fix: guard fp8 e8m0 and e2m1 compile ](../sources/prs/flashinfer/PR-1287.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [Make Fp8 MoE routing_bias optional](../sources/prs/flashinfer/PR-1319.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [feat: Fused rope fp8 quantize kernel for MLA](../sources/prs/flashinfer/PR-1339.md), [[fix] remove (view) transpose to keep consistent with majorness MN requirement.](../sources/prs/flashinfer/PR-1358.md), [bugfix: Add guard for fp4/fp8 related include headers](../sources/prs/flashinfer/PR-1376.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [Adding FP8 benchmark on attention and matmul testing](../sources/prs/flashinfer/PR-1390.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [feature: add cutlass as bmm_fp8 backend.](../sources/prs/flashinfer/PR-1397.md), [bugfix: fix perf issue by using fp8 graph that can use cublaslt](../sources/prs/flashinfer/PR-1435.md), [Add alignment in MxFP8Quantization](../sources/prs/flashinfer/PR-1445.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [perf: add 1x4x1 cluster shape for fp8 bmm M<16 cases](../sources/prs/flashinfer/PR-1473.md), [refactor: unify autotuner for bmm_fp8](../sources/prs/flashinfer/PR-1479.md), [feat: Support fp8 qkv, fp16/bf16 out MHA for trtllm-gen.](../sources/prs/flashinfer/PR-1490.md), [Perf: support scale_a/scale_b instead of combined scale in cutlass bmm_fp8](../sources/prs/flashinfer/PR-1491.md), [flashinfer_benchmark QoL Improvements and Attention FP8 Support](../sources/prs/flashinfer/PR-1512.md), [bugfix: Fix compile error for undefined swizzle enum.](../sources/prs/flashinfer/PR-1530.md), [feat: Add fp8-qkv, fp16/bf16 output MHA](../sources/prs/flashinfer/PR-1540.md), [bugfix: fix unittest test_fp8_quantize](../sources/prs/flashinfer/PR-1599.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [feat: cutlass fp8 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1610.md), [test: pytest.mark.xfail on deepgemm](../sources/prs/flashinfer/PR-1636.md), [Add benchmark for MLARopeQuantize](../sources/prs/flashinfer/PR-1656.md), [perf&bugfix: skip kv-tile computation out of sliding window in FA2; fix __syncthreads in mergestate](../sources/prs/flashinfer/PR-1661.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [test: better fp8 quantization init for fused_moe test](../sources/prs/flashinfer/PR-1674.md), [Update deepgemm backend for 103a](../sources/prs/flashinfer/PR-1694.md), [test: skip the unsupported test cases for sm120/121](../sources/prs/flashinfer/PR-1710.md), [TVM: support TVM binding for GroupedGemm](../sources/prs/flashinfer/PR-1725.md), [tests: xfail moe quantization classes mxfp8_bf16 UTs on sm103 ](../sources/prs/flashinfer/PR-1754.md), [tests: skip non SM100/103 for grouped deepgemm](../sources/prs/flashinfer/PR-1767.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [raise error for group_gemm_fp8_nt_groupwise then num_groups > 1 on sm120/121](../sources/prs/flashinfer/PR-1862.md), [misc: fix some B200 GEMM bench](../sources/prs/flashinfer/PR-1883.md), [MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA](../sources/prs/flashinfer/PR-1924.md), [Fix: Verify scales are not None for Cutlass FP8 FusedMoE](../sources/prs/flashinfer/PR-1961.md), [feat: enable deepgemm jit for fp8 block-scale on SM90](../sources/prs/flashinfer/PR-1969.md), [fix: correct PDL parameter handling in RopeQuantize kernel](../sources/prs/flashinfer/PR-1982.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [feat: suitable_auto_backends to prune auto backends, bmm_fp8 refactor, heuristic_func intake](../sources/prs/flashinfer/PR-2029.md), [Enable renormalize(naive) routing for fp8 per-tensor](../sources/prs/flashinfer/PR-2030.md), [Added an initial implementation of Q and KV Cache in fp8 and to use t…](../sources/prs/flashinfer/PR-2035.md), [feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA) ](../sources/prs/flashinfer/PR-2037.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [test: Skip test_fp8_quantize.py on Hopper](../sources/prs/flashinfer/PR-2052.md), [Fix moe fp8 failure for sm121](../sources/prs/flashinfer/PR-2061.md), [perf: TRT-LLM MoE Block-FP8 activation optimization](../sources/prs/flashinfer/PR-2063.md), [enable xqa fp8 output](../sources/prs/flashinfer/PR-2081.md), [refactor: update dpsk fused_moe test [1]](../sources/prs/flashinfer/PR-2088.md), [refactor: update dpsk fused_moe test [2]](../sources/prs/flashinfer/PR-2097.md), [refactor: update fa3 codebase and fix hopper unittest [part 1]](../sources/prs/flashinfer/PR-2111.md), [fix: Fix bench_mm_fp8.py](../sources/prs/flashinfer/PR-2129.md), [make DeepGEMM swapAB available for linear gemm SM90](../sources/prs/flashinfer/PR-2131.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [Enable Hopper FA3 FP8 attention in decode.py](../sources/prs/flashinfer/PR-2148.md), [Fp8 attention are now part of cuDNN 9.17.1](../sources/prs/flashinfer/PR-2241.md), [feat: RMSNorm/Fused RMSNorm + FP8 Quantization kernels](../sources/prs/flashinfer/PR-2243.md), [fix: support int64 IdType for RoPE part argument in `rope_quantize_fp8_append_paged_kv_cache`](../sources/prs/flashinfer/PR-2255.md), [feat: Add support for bmm mxfp8](../sources/prs/flashinfer/PR-2256.md), [Fix CUTLASS FP8 gemm correctness issue on SM120/SM121 for shapes where N is not divisible by ScaleGranularityN.](../sources/prs/flashinfer/PR-2261.md), [test: use .float() in in F.cosine_similarity() in bmm_fp8 test](../sources/prs/flashinfer/PR-2266.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [[perf] Improve gemm_fp8_nt_groupwise (cutlass backend) by 10-40% for batch sizes <= 32](../sources/prs/flashinfer/PR-2327.md), [fix: guard batchWarpReduceSum with ENABLE_FP8 to fix compilation without FP8](../sources/prs/flashinfer/PR-2328.md), [fix: Fix NaN output in mxfp8_quantize for very small input values](../sources/prs/flashinfer/PR-2441.md), [Add cute-dsl backends to mxfp[8,4]_quantization for future refactor](../sources/prs/flashinfer/PR-2443.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [feat: Add MXFP8 GEMM mm_mxfp8 (cutlass)](../sources/prs/flashinfer/PR-2464.md), [Feat: Trtllm-gen MxFP8 MoE integration](../sources/prs/flashinfer/PR-2505.md), [fix: include fp8_blockscale_gemm_90 in AOT jit-cache](../sources/prs/flashinfer/PR-2533.md), [fallback to fa2 (instead of fa3) for unsupported configuration (bf16 Q, Fp8 KV)](../sources/prs/flashinfer/PR-2536.md), [tests: bmm_fp8 for SM110](../sources/prs/flashinfer/PR-2538.md), [Add gen_gemm_sm100_module_cutlass_mxfp8 to jit-cache](../sources/prs/flashinfer/PR-2549.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [fix: add SM121 support to SM120 version guards](../sources/prs/flashinfer/PR-2631.md), [benchmark: Add MXFP4/MXFP8 quantization mode support to FP4 MoE benchmark](../sources/prs/flashinfer/PR-2635.md), [[fp8_blockwise]Fix int32 overflow in TRTLLM fused MoE activation kernel](../sources/prs/flashinfer/PR-2642.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [feat: support mxfp4 & mxfp8 entrypoint for blackwell cutedsl dense gemm](../sources/prs/flashinfer/PR-2660.md), [benchmarks: Add FP8 input / BF16 output in ragged prefill benchmark](../sources/prs/flashinfer/PR-2666.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [feat: Add support for TRTLLM MXFP8 non-gated MoE with ReLU2](../sources/prs/flashinfer/PR-2707.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [Support in-place update for `trtllm_fp8_block_scale_moe`](../sources/prs/flashinfer/PR-2739.md), [Add cute dsl mla decode op](../sources/prs/flashinfer/PR-2743.md), [[Spark unit test debugging] Fix for tests/gemm/test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-2751.md), [feat: FP8 output support for CUTLASS MLA paged attention](../sources/prs/flashinfer/PR-2779.md), [tests: skip sliding window + fp8 to prevent hang in fmha_v2 unit tests](../sources/prs/flashinfer/PR-2781.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [Fix silent bug with FP8 per tensor non-gated MoE](../sources/prs/flashinfer/PR-2882.md), [feat: add pdl support for cute dsl mla decode kernel support](../sources/prs/flashinfer/PR-2901.md), [feat: add MXFP8 GEMM support for SM120](../sources/prs/flashinfer/PR-2902.md), [perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels](../sources/prs/flashinfer/PR-2904.md), [[NVIDIA] fix(jit): enable GDC for CUTLASS fused MoE PDL — prevent random crashes on SM12x](../sources/prs/flashinfer/PR-2913.md), [feat: Add cuBLASLt backend for `mm_bf16` and enable multi-tactic autotuning for FP8/MXFP8 runners](../sources/prs/flashinfer/PR-2914.md), [test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9](../sources/prs/flashinfer/PR-2963.md), [test: skip unsupported mm_mxfp8 configurations on SM12x](../sources/prs/flashinfer/PR-2974.md), [ Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] ](../sources/prs/flashinfer/PR-2994.md), [Support Allreduce + Norm + Per-token Group Fp8 Quant Fusion](../sources/prs/flashinfer/PR-3059.md), [feat: Enable FP8 (E4M3/E5M2) in concat_mla_k for optimize long-context prefill performance and refactor type dispatch for BF16/FP16](../sources/prs/flashinfer/PR-3129.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [test: enable bmm_mxfp8 cutlass backend coverage on SM12x](../sources/prs/flashinfer/PR-3183.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [Support Kimi K2.5 H64 CuTe DSL MLA decode](../sources/prs/flashinfer/PR-3235.md), [fix(fmha_v2): fix FP8 V-scratch pipeline and varlen scheduler on SM90](../sources/prs/flashinfer/PR-3276.md), [Naive Support for Hopper FP8 Prefill Kernel with Per-Head Quantization](../sources/prs/flashinfer/PR-869.md), [perf: Fix python API overhead when CUDAGraph is not enabled](../sources/prs/flashinfer/PR-969.md), [Update CUTLASS. Refine KernelSchedule for fp8 (grouped) gemm.](../sources/prs/sglang/PR-10491.md), [support qwen3-next-fp8 deepep](../sources/prs/sglang/PR-10622.md), [Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization](../sources/prs/sglang/PR-11081.md), [[sgl-kernel][1/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-11432.md), [[DeepseekV32] Enable flashmla_prefill kernel with fp8 kvcache](../sources/prs/sglang/PR-11655.md), [Support running FP4 Deepseek on SM120.](../sources/prs/sglang/PR-11708.md), [Change bf16 to fp8 for some gemms in attention for DeepSeek ckpt v2](../sources/prs/sglang/PR-11805.md), [Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4](../sources/prs/sglang/PR-11866.md), [Feature/nano v2 offline modelopt fp8 and nvfp4](../sources/prs/sglang/PR-12018.md), [[sgl-kernel][4/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-12080.md), [[hotfix] missing `w13_weight_fp8` and `w2_weight_fp8` in UE8M0 requantization](../sources/prs/sglang/PR-12259.md), [fix: llama 4 + trtllm gen + fp8 kv cache incompatibility](../sources/prs/sglang/PR-12347.md), [Enable Flashinfer TRTLLM-GEN-MoE FP8 blockwise kernel for Qwen3-Next on Blackwell](../sources/prs/sglang/PR-12543.md), [[sgl-kernel] support custom fp8 flashmla kernel](../sources/prs/sglang/PR-13087.md), [Aiter fp8 kv cache](../sources/prs/sglang/PR-13147.md), [[NVIDIA] Fix broken fp8 MoE of deepseek v3](../sources/prs/sglang/PR-13264.md), [[NVIDIA] Fix use case of SGLANG_ENABLE_FLASHINFER_GEMM](../sources/prs/sglang/PR-13274.md), [[ROCM] Optimized deepseek-r1 fp8 model with + triton_gemm_a8w8 + batch_gemm_a8w8 + fused set_mla_kv_buffer kernel](../sources/prs/sglang/PR-13617.md), [[sgl-kernel][Feat][B200][1/N]Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-13731.md), [Support fp4 fp8 non gated moe](../sources/prs/sglang/PR-13794.md), [Feat: GLM-4.6 supports shared experts fusion](../sources/prs/sglang/PR-13873.md), [Fix update weight error for blackwell DeepGEMM](../sources/prs/sglang/PR-13910.md), [[DeepSeek v3.2] opt Context Parallelism: support fused moe, multi batch and fp8 kvcache](../sources/prs/sglang/PR-13959.md), [Support checking fp8 params in weight_checker](../sources/prs/sglang/PR-14147.md), [Add Mistral Large 3 support.](../sources/prs/sglang/PR-14213.md), [[CPU] Implement MXFP4 Gemm kernels for intel AMX to support GPT OSS series.](../sources/prs/sglang/PR-14385.md), [Support FP8 MLA prefill and 128k context.](../sources/prs/sglang/PR-14395.md), [Add Mistral Large 3 Eagle Support](../sources/prs/sglang/PR-14466.md), [[sgl-kernel][Feat][B200][2/N] Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-14640.md), [[sgl-kernel] Update flashmla to include fp8 sparse_mla optimizations](../sources/prs/sglang/PR-15242.md), [Flashinfer MOE FP8 support for Mistral Large 3.](../sources/prs/sglang/PR-15422.md), [[sgl-kernel][6/7]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-15471.md), [[Perf] Add Flashinfer DeepGEMM SM90 for SwapAB Optimization](../sources/prs/sglang/PR-15514.md), [Optimize FP8 MLA KV cache writes with Triton kernel](../sources/prs/sglang/PR-15522.md), [[JIT kernel] Apply jit per_tensor_quant_fp8 kernel](../sources/prs/sglang/PR-15836.md), [Fix FP8 MoE NaN with DeepGEMM on Blackwell](../sources/prs/sglang/PR-16622.md), [[GLM 4.7] Add RTX 6000 Pro aka sm120](../sources/prs/sglang/PR-17235.md), [Disable mla persistent kernel when not using fp8 kv_cache](../sources/prs/sglang/PR-17327.md), [Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE](../sources/prs/sglang/PR-17449.md), [[ROCm] Optimize Deepseek R1 on MI300X](../sources/prs/sglang/PR-18242.md), [Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4](../sources/prs/sglang/PR-18389.md), [[AMD] Update aiter to v0.1.10.post2](../sources/prs/sglang/PR-18423.md), [Fp8 prefill attn kernel integration](../sources/prs/sglang/PR-18528.md), [[AMD] DSR1/V3 use fp8 bmm in MLA for MI300X](../sources/prs/sglang/PR-18624.md), [[RL] Support per-layer mixed FP8/BF16 serving for FP8 checkpoints](../sources/prs/sglang/PR-18742.md), [Fix NSA FP8 KV cache path for both-trtllm MHA one-shot](../sources/prs/sglang/PR-18931.md), [[FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE](../sources/prs/sglang/PR-19537.md), [Various SM120 improvements](../sources/prs/sglang/PR-19721.md), [[AMD] Fix FP8 assertion failure in aiter MLA decode by falling back to self.k_scale](../sources/prs/sglang/PR-19935.md), [Enable modelopt quantized FLUX deployment](../sources/prs/sglang/PR-20082.md), [[AMD] Fp8 prefill integration with radix cache path for dpsk models](../sources/prs/sglang/PR-20187.md), [[Benchmark] use flashinfer bench_gpu_time instead of triton do_bench](../sources/prs/sglang/PR-20305.md), [[NVIDIA] Enable fp8 flashinfer_trtllm_routed MoE for MiniMax-M2.5](../sources/prs/sglang/PR-20394.md), [Support Triton MLA FP8 KV cache](../sources/prs/sglang/PR-20479.md), [FIX: (NSA) Compute topk_indices_offset when NSA prefill flashmla_sparse is used with FP8 KV cache](../sources/prs/sglang/PR-20606.md), [CUTLASS FP8 Blockwise GEMM improvement of SM120](../sources/prs/sglang/PR-20887.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [ci: run Stage A CUDA tests as stage-a-test-small-1-gpu on 5090](../sources/prs/sglang/PR-20988.md), [[AMD]: Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5…](../sources/prs/sglang/PR-21213.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [[RL] Support mxfp8 DeepSeek V3](../sources/prs/sglang/PR-21280.md), [[Kernel] Support FlashInfer TRTLLM-Gen fused MoE for non-gated FP4 & FP8 (Nemotron)](../sources/prs/sglang/PR-21321.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [Migrate all callers from /get_server_info to /server_info](../sources/prs/sglang/PR-21463.md), [[AMD] Enable FP8 KV cache and FP8 attention kernel for NSA on MI300/MI355 with TileLang backend](../sources/prs/sglang/PR-21511.md), [[FlashInver v0.6.7] Integrate flashinfer_trtllm mxfp8 gemm](../sources/prs/sglang/PR-21576.md), [[AMD] Add GLM-5-FP8 nightly performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-21710.md), [[Misc] [MXFP8] Drop sm100 mxfp8 warning](../sources/prs/sglang/PR-21881.md), [fix pcg torch dynamo recompile in mxfp8 Triton path](../sources/prs/sglang/PR-21888.md), [Tiny fix trtllm_fp8_per_tensor_scale_moe_wrapper router_logits dtype](../sources/prs/sglang/PR-22006.md), [[AMD][HIP] NSA: bf16 passthrough from RMSNorm to eliminate FP8 dequantization](../sources/prs/sglang/PR-22258.md), [[AMD] Fix GLM-5 fp8 KV quant path dispatch on MI300](../sources/prs/sglang/PR-22314.md), [[Reland] DeepSeek-R1-0528-w4a8: DeepEP Low Latency Dispatch Adopts FP8 Communication](../sources/prs/sglang/PR-22316.md), [[Lora] Lora quat info re-factor and support deepseekv3 mla lora](../sources/prs/sglang/PR-22323.md), [[AMD] Add GLM-5.1-FP8 nightly accuracy and performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-22336.md), [[Diffusion] modelopt diffusion fp8 support for flux1/flux2 and wan2.2](../sources/prs/sglang/PR-22365.md), [[DSA] Hopper FP8 FlashMLA KV padding](../sources/prs/sglang/PR-22372.md), [[RL] Fix weight update for mxfp8 flashinfer_cutlass gemm backend](../sources/prs/sglang/PR-22484.md), [[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22574.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [[MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute](../sources/prs/sglang/PR-23760.md), [Refactor device timer, clean up metrics collector, and add fwd occupancy metric](../sources/prs/sglang/PR-24197.md), [Port MXFP4 Marlin MoE support to JIT kernel path](../sources/prs/sglang/PR-24490.md), [[Codex] Opt Mistral Large performace ](../sources/prs/sglang/PR-24611.md), [[Gemma4] Optimize Gemm4 with fused Q/K/V RMSNorm + per-expert FP8 ckpt loader](../sources/prs/sglang/PR-24696.md), [Add FlashInfer SM90 cutlass MXFP4 MoE backend (W4A16) for GPT-OSS + DeepSeek-V4](../sources/prs/sglang/PR-24816.md), [[attn backend] Integrate tokenspeed_mla prefill/decode kernels (fp8 kv cache, blackwell)](../sources/prs/sglang/PR-24925.md), [Amd/deepseek v4 rebase main 0509](../sources/prs/sglang/PR-24933.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [[Gemma4]: Fix FP8 Triton scale layout](../sources/prs/sglang/PR-25286.md), [[Intel GPU] Enable DeepSeek V4 Inference on XPU](../sources/prs/sglang/PR-25336.md), [[fp8] SM90 swap-AB scaled_mm dispatch (~1.16x kernel geomean, +5.8-18.5% end-to-end)](../sources/prs/sglang/PR-25532.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [support w8a8 fp8 kernel with CUTLASS](../sources/prs/sglang/PR-3047.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [Apply sgl w8a8 fp8 kernel](../sources/prs/sglang/PR-3148.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [add control for cutlass fp8 blockwise gemm](../sources/prs/sglang/PR-3727.md), [[tools] add fp8 max/min constant in utils](../sources/prs/sglang/PR-3959.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [linear support deepgemm](../sources/prs/sglang/PR-4199.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [Clean up fp8 support](../sources/prs/sglang/PR-4230.md), [fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0](../sources/prs/sglang/PR-4231.md), [Support Blackwell Block Scale FP8 Gemm](../sources/prs/sglang/PR-4278.md), [[FIX] fix incorrect output when enable both deepgemm and torch compile](../sources/prs/sglang/PR-4359.md), [[ROCm] fix dtype](../sources/prs/sglang/PR-4510.md), [Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt](../sources/prs/sglang/PR-4515.md), [Support fp8 gemm for blackwell](../sources/prs/sglang/PR-4558.md), [[quantization] fix channelwise conversion with scalar weight scale](../sources/prs/sglang/PR-4596.md), [Set deepgemm to the default value in the hopper architecture.](../sources/prs/sglang/PR-4613.md), [Add DeepSeek V3/R1 shared experts fusion](../sources/prs/sglang/PR-4918.md), [feat: add DeepGEMM build warning](../sources/prs/sglang/PR-5176.md), [[Fix] Turn off DeepGEMM by default](../sources/prs/sglang/PR-5263.md), [[1/2] Add FP8 Blockscale MoE CUTLASS kernel for Blackwell](../sources/prs/sglang/PR-5281.md), [fix: use deepgemm only on hopper](../sources/prs/sglang/PR-5310.md), [[perf] experimental enhance fp8 per-tensor quant](../sources/prs/sglang/PR-5370.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [[feature] enable pre compile jit deep_gemm](../sources/prs/sglang/PR-5580.md), [ DeepEP normal support deepgemm-contiguous](../sources/prs/sglang/PR-5626.md), [[perf] dsv3 bmm fallback to bf16](../sources/prs/sglang/PR-5662.md), [[2/2] Add python wrapper for CUTLASS FP8 Blockscale MoE Kernel. ](../sources/prs/sglang/PR-5694.md), [cutlass 3.9 supported to improve fp8_blockwise_gemm](../sources/prs/sglang/PR-5820.md), [chore: upgrade cutlass 3.9.2](../sources/prs/sglang/PR-6004.md), [Upgrade CUTLASS 4.0](../sources/prs/sglang/PR-6336.md), [Add fp8 fused_experts kernel for CPU in sgl-kernel and add UT](../sources/prs/sglang/PR-6404.md), [Fix bug of deepseek-v3 under DP+EP mode with large batchsize/seqlen](../sources/prs/sglang/PR-6449.md), [[Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell](../sources/prs/sglang/PR-6479.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [[CPU] add optimizations for INT8 and FP8 DeepSeek](../sources/prs/sglang/PR-6769.md), [feat: integrate deepgemm into EPMoE](../sources/prs/sglang/PR-6821.md), [CPU: map changes from developing branch in sgl-kernel](../sources/prs/sglang/PR-6833.md), [Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa](../sources/prs/sglang/PR-6890.md), [Add a CUDA kernel for fusing mapping and weighted sum for MoE.](../sources/prs/sglang/PR-6916.md), [[Feature] Support Flashinfer fmha on Blackwell](../sources/prs/sglang/PR-6930.md), [Fuse routed scaling factor in deepseek](../sources/prs/sglang/PR-6970.md), [Update default settings for blackwell](../sources/prs/sglang/PR-7023.md), [Fix positional argument](../sources/prs/sglang/PR-7093.md), [fix amd EP MoE FP8 issue](../sources/prs/sglang/PR-7125.md), [Enable ModelOpt Llama4 fp8 checkpoint deployment in SGLang](../sources/prs/sglang/PR-7129.md), [[amd] Opt dsv3 moe](../sources/prs/sglang/PR-7160.md), [Support new DeepGEMM](../sources/prs/sglang/PR-7172.md), [Tiny let DeepGEMM scale checks cover more cases](../sources/prs/sglang/PR-7182.md), [[AMD] Fail gracefully when AITER is unavailable gfx90a GPUs](../sources/prs/sglang/PR-7187.md), [[fix] fix DeepGEMM blackwell input quant & ut & fix style and log](../sources/prs/sglang/PR-7247.md), [Add CUTLASS FP8 Blockscale MoE kernel for Hopper architecture](../sources/prs/sglang/PR-7278.md), [Fix torch compile run](../sources/prs/sglang/PR-7391.md), [[AMD][Quantization] Add `int4fp8_moe` online quantization on ROCm](../sources/prs/sglang/PR-7392.md), [feat: support DeepSeek-R1-W4AFP8 model with ep-moe mode](../sources/prs/sglang/PR-7762.md), [Qwen FP8/NVFP4 ModelOPT Quantization support](../sources/prs/sglang/PR-7912.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [[sgl-kernel] Opt per_token_quant_fp8 with warp reduce](../sources/prs/sglang/PR-8130.md), [[1/N]Support DeepSeek-R1 w4a8 normal deepep](../sources/prs/sglang/PR-8247.md), [[2/N]Support DeepSeek-R1 w4a8 low latency deepep](../sources/prs/sglang/PR-8464.md), [TRTLLM-MLA FP8 path](../sources/prs/sglang/PR-8638.md), [feat: support cutlass_moe_fp8 kernel for fusedmoe in sm90](../sources/prs/sglang/PR-8678.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/sglang/PR-8818.md), [optimize: reduce shulffle and quantization overhead in cutlass_moe sm90](../sources/prs/sglang/PR-8962.md), [[fix]: fix cutlass moe ut and and Opt H20 cutlass groupGemm performance](../sources/prs/sglang/PR-9272.md), [[sgl-kernel] feat: Support sm120 cutlass fp8 gemm kernel](../sources/prs/sglang/PR-9403.md), [fix: blackwell dsv3 fp8 issue temporary solution](../sources/prs/sglang/PR-9530.md), [Update CUTLASS 4.2 & Enable K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM](../sources/prs/sglang/PR-9559.md), [move is_sm90_supported/is_sm100_supported to python/sglang/srt/utils.py](../sources/prs/sglang/PR-9679.md), [[CPU] Add FP8 Bmm support](../sources/prs/sglang/PR-9744.md), [Make sm100 fp8 kernels available on sm103](../sources/prs/sglang/PR-9789.md), [CUTLASS fp8 blockwise gemm support of sm120](../sources/prs/sglang/PR-9969.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support](../sources/prs/vllm/PR-10995.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [Add: Support for Sparse24Bitmask Compressed Models](../sources/prs/vllm/PR-12097.md), [Expert Parallelism (EP) Support for DeepSeek Models](../sources/prs/vllm/PR-12583.md), [[Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3](../sources/prs/vllm/PR-12587.md), [[Attention] Deepseek v3 MLA support with FP8 compute](../sources/prs/vllm/PR-12601.md), [[Attention] MLA with chunked prefill](../sources/prs/vllm/PR-12639.md), [[AMD][ROCm] Enable DeepSeek model on ROCm](../sources/prs/vllm/PR-12662.md), [[Bugfix] Better FP8 supported defaults](../sources/prs/vllm/PR-12796.md), [[Kernel]Add streamK for block-quantized CUTLASS kernels](../sources/prs/vllm/PR-12978.md), [[core] Perf improvement for DSv3 on AMD GPUs](../sources/prs/vllm/PR-13718.md), [[V1] V1 Enablement Oracle ](../sources/prs/vllm/PR-13726.md), [[Bugfix][Quantization] Fix FP8 + EP](../sources/prs/vllm/PR-13784.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[Kernel] CUTLASS grouped gemm fp8 MoE kernel](../sources/prs/vllm/PR-13972.md), [[core] moe fp8 block quant tuning support](../sources/prs/vllm/PR-14068.md), [dynamic distpatch of fp8 kernels](../sources/prs/vllm/PR-14245.md), [Add cutlass support for blackwell fp8 blockwise gemm](../sources/prs/vllm/PR-14383.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [[ROCm][Kernel] MoE weights padding](../sources/prs/vllm/PR-14454.md), [permute/unpermute kernel for moe optimization](../sources/prs/vllm/PR-14568.md), [[Attention] Flash Attention 3 - fp8](../sources/prs/vllm/PR-14570.md), [[Quantization][FP8] Adding support for fp8 gemm layer input in fp8](../sources/prs/vllm/PR-14578.md), [[Attention] MLA get rid of materialization](../sources/prs/vllm/PR-14770.md), [[FEAT][ROCm] Integrate Fused MoE Kernels from AITER](../sources/prs/vllm/PR-14967.md), [[FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature](../sources/prs/vllm/PR-14968.md), [[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM](../sources/prs/vllm/PR-15587.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [Upstream Llama4 Support to Main](../sources/prs/vllm/PR-16113.md), [[Kernel] Support W8A8 channel-wise weights and per-token activations in triton fused_moe_kernel](../sources/prs/vllm/PR-16366.md), [Enable PTPC FP8 for CompressedTensorsW8A8Fp8MoEMethod (triton fused_moe)](../sources/prs/vllm/PR-16537.md), [[ROCm] Add aiter tkw1 kernel for Llama4 fp8](../sources/prs/vllm/PR-16727.md), [[Kernel] some optimizations for dense marlin and moe marlin](../sources/prs/vllm/PR-16850.md), [[Kernel] Add expert_map support to Cutlass FP8 MOE](../sources/prs/vllm/PR-16861.md), [[FEAT] [ROCm]: Add AITER CK 2 Stages MoE support](../sources/prs/vllm/PR-17110.md), [[ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention](../sources/prs/vllm/PR-17139.md), [[NVIDIA] Support Cutlass w8a8 FP8 for Blackwell Geforce GPUs (sm120)](../sources/prs/vllm/PR-17280.md), [[Kernel] fp4 marlin kernel](../sources/prs/vllm/PR-17687.md), [use ceil_div in cutlass block scaling shape check](../sources/prs/vllm/PR-17918.md), [[Feature] Expert Parallelism Load Balancer (EPLB)](../sources/prs/vllm/PR-18343.md), [[V1] Support `LLM.apply_model`](../sources/prs/vllm/PR-18465.md), [Sm100 blockwise fp8 swap ab](../sources/prs/vllm/PR-18564.md), [[Kernel] Integrate CUTLASS MoE kernel with PPLX](../sources/prs/vllm/PR-18762.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-18778.md), [[Kernel] Enable fp8 support for pplx and BatchedTritonExperts.](../sources/prs/vllm/PR-18864.md), [[Kernel] Support deep_gemm for linear methods](../sources/prs/vllm/PR-19085.md), [[Kernels] Add activation chunking logic to FusedMoEModularKernel](../sources/prs/vllm/PR-19168.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [[Feature] Integrate new deepgemm](../sources/prs/vllm/PR-19820.md), [ [Feature] Integrate SM100 DeepGEMM support](../sources/prs/vllm/PR-20087.md), [[Bugfix] Fix topk_ids indices_type for CUTLASS w8a8 FP8 MoE](../sources/prs/vllm/PR-20166.md), [[V1] [ROCm] Enable EP with AITER Fused MoE](../sources/prs/vllm/PR-20270.md), [[Kernel] SM90 CUTLASS FP8 GEMM: add support for swap AB + kernel tuning](../sources/prs/vllm/PR-20396.md), [[feat]: add SM100 support for cutlass FP8 groupGEMM](../sources/prs/vllm/PR-20447.md), [Support Llama 4 for fused_marlin_moe](../sources/prs/vllm/PR-20457.md), [[Perf] Reuse workspace for FP8+FP4 Marlin MoE](../sources/prs/vllm/PR-20500.md), [[Performance] Performance improvements in non-blockwise fp8 CUTLASS MoE](../sources/prs/vllm/PR-20762.md), [[Perf] Use Triton instead of Torch for DeepGEMM Per Token Group Quant](../sources/prs/vllm/PR-20841.md), [[Perf] Add swap_ab to SM90 FP8 non-block CUTLASS moe grouped gemm](../sources/prs/vllm/PR-20911.md), [[Perf] Cuda Kernel for Per Token Group Quant](../sources/prs/vllm/PR-21083.md), [[perf] Add fused MLA QKV + strided layernorm](../sources/prs/vllm/PR-21116.md), [[Feature][OCP MX] Support mxfp6 and mixed mxfp6-mxfp4](../sources/prs/vllm/PR-21166.md), [[Bug] DeepGemm: Fix TypeError: per_block_cast_to_fp8() missing 1 required positional argument: 'use_ue8m0' for SM100](../sources/prs/vllm/PR-21187.md), [[NVIDIA] Explicitly disable shuffled weights for flashinfer blockscale moe fp8 kernels](../sources/prs/vllm/PR-21411.md), [[Bugfix][CUDA] fixes CUDA FP8 kv cache dtype supported](../sources/prs/vllm/PR-21420.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv/out Attention Kernel](../sources/prs/vllm/PR-21716.md), [[Kernel] Add support for block FP8 on SM120 (NVIDIA 5090 and RTX PRO 6000)](../sources/prs/vllm/PR-22131.md), [Fp8 paged attention update](../sources/prs/vllm/PR-22222.md), [[Bug] Fix B200 DeepGEMM E8M0 Accuracy Issue](../sources/prs/vllm/PR-22399.md), [[Quantization] Expand compressed-tensors MoE matching logic to support NFP4 + FP8 MoEs](../sources/prs/vllm/PR-22674.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv NVFP4-out Attention Kernel](../sources/prs/vllm/PR-22703.md), [fp8 kv cache support fix for torch.compile](../sources/prs/vllm/PR-22758.md), [[Kernel] Added flashinfer fp8 per-tensor gemms](../sources/prs/vllm/PR-22895.md), [[Bugfix] fix qwen3 moe fp8 accuracy issue](../sources/prs/vllm/PR-23031.md), [[Kernel] CUTLASS MoE FP8: Integrate cuda moe permute/unpermute](../sources/prs/vllm/PR-23045.md), [Add routed_scaling_factor to MoE grouped topk](../sources/prs/vllm/PR-23123.md), [[XPU][Feature] fp8 online quantization support for XPU](../sources/prs/vllm/PR-23148.md), [[kernel] Support W4A8 on Hopper](../sources/prs/vllm/PR-23198.md), [[ROCm][Aiter] Add triton fp8 bmm kernel for mla](../sources/prs/vllm/PR-23264.md), [[Perf] Small optimizations for silu_mul_fp8_quant_deep_gemm](../sources/prs/vllm/PR-23265.md), [[Kernels] Overlap shared experts with send/recv](../sources/prs/vllm/PR-23273.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Bug] Fix R1 Accuracy 0 Bug](../sources/prs/vllm/PR-23294.md), [DP/EP Support for gpt-oss with deepep-ht comm kernel on SM100](../sources/prs/vllm/PR-23608.md), [[Flashinfer] Support Flashinfer TRTLLM FP8-qkv BF16/FP16-out Attention Kernel](../sources/prs/vllm/PR-23647.md), [[Feature] Add Hopper DeepGEMM E8M0 for DeepSeekV3.1 scale_fmt](../sources/prs/vllm/PR-23666.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Model] Add LongCat-Flash ](../sources/prs/vllm/PR-23991.md), [[Performance] Move apply_w8a8_block_fp8_linear to an op class](../sources/prs/vllm/PR-24666.md), [[NVIDIA] Blackwell Family](../sources/prs/vllm/PR-24673.md), [[Kernel][Quantization] add w4a8 support for marlin kernel](../sources/prs/vllm/PR-24722.md), [[Flashinfer][gpt-oss] Support FP8-qkv Flashinfer TRTLLM Sinks Attention](../sources/prs/vllm/PR-25674.md), [[Bugfix] Convert untraceable GroupShape to list for AMD impl](../sources/prs/vllm/PR-26535.md), [[Bugfix] Fix gpt-oss w4a8 DP/EP on B200](../sources/prs/vllm/PR-26729.md), [[Feature] Batch Invariant: Support DeepGEMM and Blackwell](../sources/prs/vllm/PR-27127.md), [[Kernels] Enable FlashInfer FP8 Blockscale on SM90 (for TEP DSR1)](../sources/prs/vllm/PR-27134.md), [[torch.compile] Enable silu_mul_fp8_quant fusion without custom ops enabled](../sources/prs/vllm/PR-27146.md), [Bugfix: Cutlass FP8 FusedMoE bad scaling factors](../sources/prs/vllm/PR-27255.md), [Feature: Support Relu2 in FusedMoE fp8 cutlass path](../sources/prs/vllm/PR-27261.md), [[Perf] SM100 - add swap AB optimization to CUTLASS FP8 GEMM](../sources/prs/vllm/PR-27284.md), [[Performance] Support FP8 flashinfer TRTLLM MOE on Qwen3 and Qwen-3next](../sources/prs/vllm/PR-27492.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [[Feature] Batch invariant torch.compile](../sources/prs/vllm/PR-27660.md), [[Performance] Fused blockwise quant RMS norm](../sources/prs/vllm/PR-27883.md), [[Performance][B200] Fix deepgemm prologue](../sources/prs/vllm/PR-27897.md), [[ROCm][MLA] enable fp8 MLA decode on ROCm](../sources/prs/vllm/PR-28032.md), [[Performance][B200] silu_mul_quant: pack scales in int32](../sources/prs/vllm/PR-28358.md), [[Performance] Reduce DeepGEMM N dim restriction from 128 to 64 multiplier ](../sources/prs/vllm/PR-28687.md), [[Perf][Kernels] Enable FlashInfer DeepGEMM swapAB on SM90 (for W8A8 Linear Op)](../sources/prs/vllm/PR-29213.md), [chore: add RTX_PRO_6000 GLM4.6-FP8 kernel tuning](../sources/prs/vllm/PR-29240.md), [[Perf] Disable DeepGEMM MoE by default when TP=8 is used](../sources/prs/vllm/PR-29346.md), [[Kernel]Support W4A8 Grouped GEMM on Hopper](../sources/prs/vllm/PR-29691.md), [[MoE-FP8-modelopt] Add FlashInfer alignment padding for intermediate dimensions](../sources/prs/vllm/PR-29748.md), [Add Mistral Large 3 and Ministral 3](../sources/prs/vllm/PR-29757.md), [[Perf] Improve fp8 quant in mla; replace ReduceSum with ReduceScatterSum](../sources/prs/vllm/PR-29795.md), [[Bugfix] Fix FP8 MoE LoRA](../sources/prs/vllm/PR-29890.md), [[Kernel][Quantization][MoE] add marlin kernel support for turing (sm75)](../sources/prs/vllm/PR-29901.md), [[ROCm] add fallback for aiter fp8 decode mla](../sources/prs/vllm/PR-30005.md), [[Quantization] Support Quark int4-fp8 w4a8 for MoE](../sources/prs/vllm/PR-30071.md), [Add llmcompressor fp8 kv-cache quant (per-tensor and per-attn_head)](../sources/prs/vllm/PR-30141.md), [[Bugfix] Fix DeepGEMM after #29546 ](../sources/prs/vllm/PR-30267.md), [[LoRA] Support Quantized Adapters](../sources/prs/vllm/PR-30286.md), [[Bugfix] Fix fp8 DeepGemm compilation issues](../sources/prs/vllm/PR-30336.md), [[ROCm][Quantization] GPT OSS Upstream MoE wmxfp4_afp8 with static scales](../sources/prs/vllm/PR-30357.md), [[Feature] Add SM103 (Blackwell Ultra) Support to vLLM](../sources/prs/vllm/PR-30484.md), [[Perf] Eliminate padding and slicing op for GPT-OSS with Flashinfer MXFP4 MXFP8 MoE](../sources/prs/vllm/PR-30647.md), [[SM100] Enable fp8 compute for prefill MLA](../sources/prs/vllm/PR-30746.md), [[Feature]: Support NVIDIA ModelOpt HF FP8 variants FP8_PER_CHANNEL_PER_TOKEN and FP8_PB_WO in vLLM](../sources/prs/vllm/PR-30957.md), [[Bugfix][Hardware][AMD] Consolidate FP8 min/max values helper function](../sources/prs/vllm/PR-31106.md), [[Bugfix][Hardware][AMD] Fix exception types in AITER MLA FP8 check](../sources/prs/vllm/PR-31177.md), [[SM100] Resubmit FMHA FP8 prefill for MLA](../sources/prs/vllm/PR-31195.md), [[Bugfix][ROCm] Fix Static Quant Issue](../sources/prs/vllm/PR-31502.md), [[Bugfix] Fix Broken ModelOpt NVFP4 MoE](../sources/prs/vllm/PR-31742.md), [[1/N][Attention] Restructure attention: move files](../sources/prs/vllm/PR-31916.md), [[5/N][Attention] Finish eliminating `vllm/attention` folder](../sources/prs/vllm/PR-32064.md), [[Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper](../sources/prs/vllm/PR-32619.md), [[Bugfix] Fix FP8 MoE EP Weight Loading for ModelOpt Llama4](../sources/prs/vllm/PR-32886.md), [Support compress-tensors with nvfp4 or fp8 weights and modelopt with nvfp4 weights on Turing](../sources/prs/vllm/PR-33076.md), [Add support for Mistral Large 3 inference with Flashinfer MoE](../sources/prs/vllm/PR-33174.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Bugfix] Register fp8 cutlass_group_gemm as supported for only SM90+SM100](../sources/prs/vllm/PR-33285.md), [[Kernel] Support Flashinfer trtllm fused MoE non gated FP8 & NVFP4](../sources/prs/vllm/PR-33506.md), [[Kernel] Add enable_sm120_or_later for SM121 (DGX Spark) CUTLASS support](../sources/prs/vllm/PR-33517.md), [[Perf] Disable clean_logits in deepgemm fp8_mqa_logits kernel](../sources/prs/vllm/PR-33568.md), [enable skipping of SW attention layers when using FP8 KV cache](../sources/prs/vllm/PR-33695.md), [[Kernel] Integrate SM100 MXFP8 blockscaled grouped MM and quant kernels](../sources/prs/vllm/PR-34448.md), [[Bugfix] Handle num_expert_group=None in flashinfer block-scale FP8 MoE](../sources/prs/vllm/PR-34494.md), [[Kernel] Add FP8 KV cache support to Triton MLA decode attention](../sources/prs/vllm/PR-34597.md), [[Bugfix] Fix NVFP4 TRTLLM MoE non-gated support; add gsm8k for Nemotron-3-Nano FP8+NVFP4](../sources/prs/vllm/PR-34725.md), [Integrate flashinfer mm_mxfp8 in ModelOpt MXFP8](../sources/prs/vllm/PR-35053.md), [[Feat] Add CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function](../sources/prs/vllm/PR-35271.md), [[Attention][Perf] Optimize cp_gather_and_upconvert_fp8_kv_cache - DeepSeek-v3.2](../sources/prs/vllm/PR-35290.md), [[Quant][Feature] Support online MXFP8 quantization for MoE and dense models](../sources/prs/vllm/PR-35448.md), [[ROCm] Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5/Linear)](../sources/prs/vllm/PR-35850.md), [[Perf] Support FP8 KV cache for Flashinfer MLA Sparse](../sources/prs/vllm/PR-35891.md), [Add support for ModelOpt MXFP8 MoE models](../sources/prs/vllm/PR-35986.md), [[Bugfix] Fix passing of activation_type to trtllm fused MoE NVFP4 and FP8](../sources/prs/vllm/PR-36017.md), [[mla] Support fused FP8/NVFP4 output quantization in MLA attention (#35792)](../sources/prs/vllm/PR-36205.md), [[Perf] Add TRTLLM FP8 MoE Modular Kernel](../sources/prs/vllm/PR-36307.md), [[XPU] Support block fp8 moe by fallback to TritonExpert on XPU](../sources/prs/vllm/PR-36458.md), [[Kernel] Fuse FP8 output quantization into merge_attn_states](../sources/prs/vllm/PR-36518.md), [[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts](../sources/prs/vllm/PR-36728.md), [[Bugfix] Fix KV scales inconsistency in fp8 MLA & FlashInfer kv_cache_dtype "auto" leading to gibberish](../sources/prs/vllm/PR-37054.md), [[XPU] support MLA model on Intel GPU](../sources/prs/vllm/PR-37143.md), [[Perf] Set Flashinfer sparse MLA as default backend for FP8 kv cache](../sources/prs/vllm/PR-37252.md), [[Bugfix] Disable monolithic TRTLLM MoE for Renormalize routing (#37591)](../sources/prs/vllm/PR-37605.md), [[Perf] Use torch compile to fuse pack topk in trtllm moe](../sources/prs/vllm/PR-37695.md), [[Bug] Fix fp8 deepgemm batch invariant](../sources/prs/vllm/PR-37718.md), [[Kernel] Optimize SM120 CUTLASS blockwise FP8 GEMM](../sources/prs/vllm/PR-37970.md), [[Perf] FP8 FlashInfer Attn for ViT](../sources/prs/vllm/PR-38065.md), [[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell](../sources/prs/vllm/PR-38083.md), [[Kernel] Add swapAB support for SM120 CUTLASS blockwise FP8 GEMM ](../sources/prs/vllm/PR-38325.md), [[MoE] Add RoutingMethodType.Simulated to TRT-LLM FP8/NVFP4 kernel allowlists](../sources/prs/vllm/PR-38329.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[QeRL] Fix online quantized reloading](../sources/prs/vllm/PR-38442.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[XPU] add xpu backend implementation of mxfp8 quant](../sources/prs/vllm/PR-38682.md), [[Quant] add CompressedTensorsW8A8Mxfp8 for linear and MoE layers](../sources/prs/vllm/PR-38815.md), [[Bugfix] Re-enable Renormalize routing for TRT-LLM MoE experts](../sources/prs/vllm/PR-38859.md), [[Bugfix] Fix broken explicit unquantized kv cache dtype support](../sources/prs/vllm/PR-38922.md), [[MoE Refactor] Split up compressed_tensors_moe.py](../sources/prs/vllm/PR-38960.md), [[Bug] Fix routing bias dtype for trtllm per-block fp8 moe](../sources/prs/vllm/PR-38989.md), [[Perf] Change Trtllm fp8 MoE to use Shuffled Weights and BlockMajorK Layout](../sources/prs/vllm/PR-38993.md), [[Bug] Fix Trtllm Fp8 MoE Weight Shuffle Memory Fragamentation](../sources/prs/vllm/PR-39054.md), [perf(moe): add tuned fused_moe config for RTX PRO 6000 Blackwell Server Edition](../sources/prs/vllm/PR-39183.md), [[Refactor] Move MXFP8 GEMM management into MxFp8LinearKernel](../sources/prs/vllm/PR-39205.md), [[Perf] Fuse Zero Initializer for FP8 DeepGemm Block Quant Kernel](../sources/prs/vllm/PR-39547.md), [add warning when FP8 KV cache misses prefill query quantization](../sources/prs/vllm/PR-39752.md), [[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement](../sources/prs/vllm/PR-40408.md), [[MoE] Move cutlass moe to fused_moe/experts/](../sources/prs/vllm/PR-40574.md), [[Kernel][Helion] Optimize Helion config parsing latency](../sources/prs/vllm/PR-40850.md), [[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided](../sources/prs/vllm/PR-40960.md), [Faster per-token fp8 group quant packed kernel for blackwell](../sources/prs/vllm/PR-41326.md), [[Quantization] Rework quantization_config to use QuantKey and allow for activation override](../sources/prs/vllm/PR-41566.md), [[CPU] Add MXFP4 W4A16 MoE support](../sources/prs/vllm/PR-41922.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Bugfix] Add swiglu limits to deepgemm fp8 methods](../sources/prs/vllm/PR-41986.md), [[feat] Add FP8 per-tensor Q scale support to Triton attention backend](../sources/prs/vllm/PR-42080.md), [[Perf] Use 2D-grid to eliminate divmod in W8W8 group quant](../sources/prs/vllm/PR-42153.md), [[Perf] Wire silu_and_mul_per_block_quant into TritonFP8MoE (MiniMax-M2) ](../sources/prs/vllm/PR-42497.md), [[6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued)](../sources/prs/vllm/PR-42663.md), [[Perf] Add do_not_specialize in fused FP8 RoPE kernel](../sources/prs/vllm/PR-42849.md), [add cutedsl dsv4 indexer fp8 kernel](../sources/prs/vllm/PR-42899.md), [[Model Refactoring] Move deepseek_v4_ops to models/deepseek_v4 [3/N]](../sources/prs/vllm/PR-43073.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../wiki/kernels/flashmla.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [Sparse MLA (DeepSeek V3.2)](../wiki/kernels/sparse-mla.md), [Fine-Grained FP8/FP4 Quantization](../wiki/techniques/fine-grained-quantization.md) | | `gdc` | [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [Programmatic Dependent Launch / Grid Dependency Control](../wiki/hardware/pdl-gdc.md) | -| `mbarrier` | [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [CUTLASS Cluster Launch Control (CLC) Documentation](../sources/docs/cutlass-clc-documentation.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Vectorize mbarrier initialization in warpspeed scan](../sources/prs/cccl/PR-8423.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [mbarrier (Memory Barrier Primitives)](../wiki/hardware/mbarrier.md), [tcgen05.mma — Blackwell MMA Instruction](../wiki/hardware/tcgen05-mma.md), [Tensor Memory Accelerator (TMA)](../wiki/hardware/tma.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Pipeline Stalls](../wiki/patterns/pipeline-stalls.md), [Software Pipelining and Multi-Stage Buffering](../wiki/techniques/pipeline-stages.md) | +| `l2-persistence` | [NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6)](../sources/docs/nvidia-ampere-tuning-guide.md), [Ampere Memory Model: SMEM/L2/Occupancy on sm_80 vs sm_86](../wiki/hardware/ampere-memory-model.md) | +| `ldmatrix` | [CUTLASS Ampere (SM80/SM86) Support: MmaMultistage, CuTe SM80 Atoms, 3xTF32](../sources/docs/cutlass-ampere-support.md), [PTX ISA Ampere (sm_80/sm_86) Instructions Reference](../sources/docs/nvidia-ptx-isa-ampere.md), [mma.sync + ldmatrix — Ampere Tensor Core Programming](../wiki/hardware/mma-sync-ampere.md) | +| `mbarrier` | [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [CUTLASS Cluster Launch Control (CLC) Documentation](../sources/docs/cutlass-clc-documentation.md), [NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6)](../sources/docs/nvidia-ampere-tuning-guide.md), [PTX ISA Ampere (sm_80/sm_86) Instructions Reference](../sources/docs/nvidia-ptx-isa-ampere.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Vectorize mbarrier initialization in warpspeed scan](../sources/prs/cccl/PR-8423.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [Ampere Memory Model: SMEM/L2/Occupancy on sm_80 vs sm_86](../wiki/hardware/ampere-memory-model.md), [cp.async — Asynchronous Global→Shared Copy (Ampere)](../wiki/hardware/cp-async.md), [mbarrier (Memory Barrier Primitives)](../wiki/hardware/mbarrier.md), [tcgen05.mma — Blackwell MMA Instruction](../wiki/hardware/tcgen05-mma.md), [Tensor Memory Accelerator (TMA)](../wiki/hardware/tma.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Pipeline Stalls](../wiki/patterns/pipeline-stalls.md), [Software Pipelining and Multi-Stage Buffering](../wiki/techniques/pipeline-stages.md) | +| `mma-sync` | [CUTLASS Ampere (SM80/SM86) Support: MmaMultistage, CuTe SM80 Atoms, 3xTF32](../sources/docs/cutlass-ampere-support.md), [NVIDIA Ampere GA102 GPU Architecture Whitepaper (v2.1)](../sources/docs/nvidia-ga102-whitepaper.md), [PTX ISA Ampere (sm_80/sm_86) Instructions Reference](../sources/docs/nvidia-ptx-isa-ampere.md), [mma.sync + ldmatrix — Ampere Tensor Core Programming](../wiki/hardware/mma-sync-ampere.md), [Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86)](../wiki/migration/hopper-to-ampere-backport.md) | | `nvfp4` | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [Anatomy of a Reward Hack](../sources/blogs/gpu-mode-reward-hack.md), [NVFP4 Format Details](../sources/blogs/nvfp4-format-details.md), [NVFP4 GEMV and Improved NVFP4 GEMV](../sources/blogs/simon-nvfp4-gemv.md), [TFLOPS Gap: Why FP4 MoE Kernel Engineering Matters on Blackwell](../sources/blogs/tflops-gap-fp4-moe.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Triton 3.6.0 Release Notes — Blackwell (SM100) Lowering](../sources/docs/triton-3.6-blackwell.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [[TRTLLM-9661][chore] Further reduce tuning time for cuteDSL nvFP4 dense gemm.](../sources/prs/TensorRT-LLM/PR-10339.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11165.md), [[None][feat] Optimize super-v3 nvfp4 for better perf](../sources/prs/TensorRT-LLM/PR-11273.md), [[None][feat] Optimize by fuse nvfp4_quant to layernorm_gated for mamba2_mixer](../sources/prs/TensorRT-LLM/PR-11473.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11733.md), [[None][fix] Fix SM120 issue for rms_norm with nvfp4_quant_fusion](../sources/prs/TensorRT-LLM/PR-11774.md), [[None][feat] Support update weight for nvfp4](../sources/prs/TensorRT-LLM/PR-12320.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[TRTLLM-8535][feat] Support DeepSeek V3.2 with FP8 + BF16 KV cache/NVFP4 + BF16 KV cache](../sources/prs/TensorRT-LLM/PR-8405.md), [[None][feat] Enable nvfp4 cuda core for sm120](../sources/prs/TensorRT-LLM/PR-8620.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Feature/sm100 low latency nvfp4 kernels](../sources/prs/flashinfer/PR-1214.md), [feat: support output nvfp4 in trtllm-gen function call.](../sources/prs/flashinfer/PR-1318.md), [Update autotune results for the nvfp4 cutlass moe backends for v0.2.9](../sources/prs/flashinfer/PR-1361.md), [Faster weight processing (moe nvfp4)](../sources/prs/flashinfer/PR-1412.md), [Fix TRTLLM NVFP4-out attention kernel scale factor dim issue](../sources/prs/flashinfer/PR-1460.md), [Add GeGLU support to trtllm-gen NVFP4 Fused MoE Kernel](../sources/prs/flashinfer/PR-1525.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [Masked batch nvfp4 quantization](../sources/prs/flashinfer/PR-1774.md), [silu_and_mul nvfp4 quanization fusion rework](../sources/prs/flashinfer/PR-1927.md), [Feature: Support non-gated activation in cutlass fused MoE nvfp4](../sources/prs/flashinfer/PR-2011.md), [[performance]optimize for nvfp4](../sources/prs/flashinfer/PR-2268.md), [[Perf][Feature] Add SM103-specific schedulers for NVFP4 CUTLASS kernels](../sources/prs/flashinfer/PR-2303.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [Support NVFP4 KV cache decode on SM120](../sources/prs/flashinfer/PR-2520.md), [tests: add bias testing to nvfp4 moe](../sources/prs/flashinfer/PR-2585.md), [fix: cute dsl nvfp4 moe routing index error](../sources/prs/flashinfer/PR-2629.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [Add NVFP4 KV cache quantization support for SM100](../sources/prs/flashinfer/PR-2702.md), [fix: Add SM120 (RTX Blackwell desktop) support for NVFP4 MoE kernels](../sources/prs/flashinfer/PR-2725.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [feat: Add CuTe-DSL backend for NVFP4 quantization](../sources/prs/flashinfer/PR-2838.md), [perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels](../sources/prs/flashinfer/PR-2904.md), [[Fmha] support nvfp4 output keepsMmaAb generation kernels](../sources/prs/flashinfer/PR-2988.md), [[feat] Trtllm-gen Per-token Nvfp4 MoE](../sources/prs/flashinfer/PR-3027.md), [Support NVFP4 KV for prefill and batch attention kernels](../sources/prs/flashinfer/PR-3097.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [perf: optimize per-token nvfp4 quantization kernel.](../sources/prs/flashinfer/PR-3237.md), [Optimize nvfp4 block scaled gemm kernel when M is small.](../sources/prs/sglang/PR-10101.md), [Fix chunked prefix cache for nvfp4](../sources/prs/sglang/PR-10180.md), [Fix correction bias undefined behavior for nvfp4 models](../sources/prs/sglang/PR-10426.md), [Fix MTP MoE weight loading with NVFP4 target model.](../sources/prs/sglang/PR-10758.md), [[NVIDIA] Add new SMs support for Spark & Thor](../sources/prs/sglang/PR-11287.md), [support cutlass fp4 kernel in sm120](../sources/prs/sglang/PR-11737.md), [Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4](../sources/prs/sglang/PR-11866.md), [Feature/nano v2 offline modelopt fp8 and nvfp4](../sources/prs/sglang/PR-12018.md), [[NVIDIA] Fix CUDA arch requirement in nvfp4 cast](../sources/prs/sglang/PR-12581.md), [ignore the deepgemm check when the model weight with nvfp4 and moe ba…](../sources/prs/sglang/PR-12782.md), [support mtp with deepseek r1 nvfp4 model](../sources/prs/sglang/PR-13115.md), [Fix nan in global scaling factor for large scale nvfp4 EP](../sources/prs/sglang/PR-13162.md), [[Feat][NVFP4] Enable NVFP4 MoE for Qwen series models (eg. Qwen3-Next) #13761](../sources/prs/sglang/PR-13761.md), [Mistral Large 3 NVFP4 support](../sources/prs/sglang/PR-14485.md), [Mistral Large 3 NVFP4 TRTLLM MoE support](../sources/prs/sglang/PR-15049.md), [[NVIDIA] Fixes for NVFP4 all-gather with spec decoding](../sources/prs/sglang/PR-15280.md), [Tiny fix cannot launch nvfp4 checkpoint with bf16 kv cache](../sources/prs/sglang/PR-15986.md), [Inclusion of nvfp4 blockscale in EPLB Rebalance](../sources/prs/sglang/PR-17158.md), [[Fix] GLM 4.7 + NVFP4 + MTP](../sources/prs/sglang/PR-17166.md), [[feat] Support nvfp4 quantized model of Qwen3-Next](../sources/prs/sglang/PR-17627.md), [[Bugfix] Fix Mistral Large 3 NVFP4 TRTLLM MoE](../sources/prs/sglang/PR-18065.md), [Fix nvfp4 weight update](../sources/prs/sglang/PR-18085.md), [[ModelOpt] Fix broken Qwen3-235B-A22B-Instruct-2507-NVFP4 launch](../sources/prs/sglang/PR-18189.md), [[ModelOPT] Support Qwen 3 Next Coder NVFP4](../sources/prs/sglang/PR-18224.md), [[Kimi-K2.5] Fix NVFP4 Kimi-K2.5 weight mapping and exclude list](../sources/prs/sglang/PR-18370.md), [Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4](../sources/prs/sglang/PR-18389.md), [[Qwen3.5] Enable nvfp4 checkpoint](../sources/prs/sglang/PR-18937.md), [feat: Support MXFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-19143.md), [Fix nightly Mistral-Large-3 NVFP4 accuracy threshold](../sources/prs/sglang/PR-19402.md), [[Kernel Slimming] Migrate NVFP4 kernels to JIT](../sources/prs/sglang/PR-19437.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [[JIT Kernel] Reland NVFP4 kernels to JIT](../sources/prs/sglang/PR-20012.md), [[V32/GLM5] Change default setting of V32 nvfp4 on TP4](../sources/prs/sglang/PR-20086.md), [[diffusion] Support nvfp4 for Flux.2](../sources/prs/sglang/PR-20137.md), [[4/n jit_kernel restruct] speed up CI tests and add benchmark workflow](../sources/prs/sglang/PR-20268.md), [[Model] Support Nemotron 3 Super NVFP4](../sources/prs/sglang/PR-20407.md), [[JIT Kernel] Fix NVFP4 multi-arch compilation failure](../sources/prs/sglang/PR-20874.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Chore] Clean up JIT compilation flags](../sources/prs/sglang/PR-21022.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [CUTLASS NVFP4 GEMM improvement of SM120](../sources/prs/sglang/PR-21314.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [[Diffusion] Fix weight scale swizzle and add large-M kernel config for FLUX.2-dev-NVFP4](../sources/prs/sglang/PR-22064.md), [[nvidia] Gemma4 nvfp4 fix](../sources/prs/sglang/PR-22079.md), [[diffusion] Default NVFP4 to CUTLASS and add all-model shape benchmarks](../sources/prs/sglang/PR-22091.md), [[Diffusion] Add diffusion NVFP4 scaled-mm correctness test](../sources/prs/sglang/PR-22127.md), [[RL] Refactor NVFP4 shuffling/swizzling to in-place replacement](../sources/prs/sglang/PR-22204.md), [[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22574.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [[Diffusion] Add Wan2.2 ModelOpt NVFP4 support](../sources/prs/sglang/PR-22681.md), [[FlashInfer v0.6.11] [RL] Support FlashInfer per-token NVFP4 MoE](../sources/prs/sglang/PR-22918.md), [Reland Cute-DSL FP4 dense GEMM](../sources/prs/sglang/PR-23590.md), [Use Cute-DSL NVFP4 quantization kernels](../sources/prs/sglang/PR-23745.md), [perf(nvfp4): free unused source scales after weight processing](../sources/prs/sglang/PR-25107.md), [[codex] Update Wan2.2 ModelOpt CI checkpoints](../sources/prs/sglang/PR-25483.md), [Support FP4 gemm (1/2)](../sources/prs/sglang/PR-3899.md), [[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu](../sources/prs/sglang/PR-4953.md), [[1/2] Add Kernel support for Cutlass based Fused FP4 MoE](../sources/prs/sglang/PR-6093.md), [Support NVFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-7302.md), [FlashInfer NVFP4 MoE with EP & 2-stream shared expert](../sources/prs/sglang/PR-7327.md), [Qwen FP8/NVFP4 ModelOPT Quantization support](../sources/prs/sglang/PR-7912.md), [[Fix][Ready]Fix register spilling in cutlass nvfp4 gemm kernel on Blackwell](../sources/prs/sglang/PR-8127.md), [[NVIDIA] Add Low Latency NVFP4 decode kernels from Flashinfer](../sources/prs/sglang/PR-8552.md), [Faster weight processing (trtllm-gen moe nvfp4)](../sources/prs/sglang/PR-9162.md), [[NVIDIA] [3/N] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked ](../sources/prs/sglang/PR-9199.md), [[NVIDA] [1/N] Nvfp4 Masked Gemm: Add quant op for the flashinfer grouped gemm](../sources/prs/sglang/PR-9200.md), [[NVIDIA] [2/N] Optimize `silu_and_mul_scaled_fp4_grouped_quant` perf](../sources/prs/sglang/PR-9556.md), [Make fp4_quantize kernels work on sm103](../sources/prs/sglang/PR-9807.md), [[NVIDIA] Support nvfp4 quantization](../sources/prs/vllm/PR-12784.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[Hardware/NVIDIA/Kernel] [Functional Enablement] [1/N] Enable nvidia/DeepSeek-R1-FP4 Model](../sources/prs/vllm/PR-16362.md), [[Misc] Add compressed-tensors NVFP4A16 emulation support](../sources/prs/vllm/PR-17914.md), [Use NVFP4 Marlin for CompressedTensorsW4A16Fp4](../sources/prs/vllm/PR-18000.md), [[Quantization] Add compressed-tensors NVFP4 support](../sources/prs/vllm/PR-18312.md), [[Hardware][NVIDIA] FP4 MoE kernel optimization](../sources/prs/vllm/PR-19110.md), [[Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization](../sources/prs/vllm/PR-19500.md), [[Quantization] Add compressed-tensors emulations support for NVFP4](../sources/prs/vllm/PR-19879.md), [[Quantization] Add compressed-tensors NVFP4 MoE Support](../sources/prs/vllm/PR-19990.md), [Add ModelOpt Qwen3 nvfp4 support](../sources/prs/vllm/PR-20101.md), [[Bugfix] Fix some narrowing conversion warnings](../sources/prs/vllm/PR-20141.md), [[Kernel][Bugfix] Fixup some warnings in nvfp4_blockwise_moe when CUDA < 12.8](../sources/prs/vllm/PR-20324.md), [Support CUTLASS NVFP4 (w4a4) for Blackwell Geforce GPUs (SM120)](../sources/prs/vllm/PR-21309.md), [Support Tensorrt-LLM MoE fp4 for low-latency](../sources/prs/vllm/PR-21331.md), [Update flashinfer CUTLASS NVFP4 MoE Kernel to use per expert global scaling factor](../sources/prs/vllm/PR-21408.md), [[Bug] Fix Compressed Tensor NVFP4 `cutlass_fp4_group_mm` illegal memory access](../sources/prs/vllm/PR-21465.md), [[Feature] Add Flashinfer MoE Support for Compressed Tensor NVFP4](../sources/prs/vllm/PR-21639.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv NVFP4-out Attention Kernel](../sources/prs/vllm/PR-22703.md), [Fix nvfp4 swizzling](../sources/prs/vllm/PR-23140.md), [fix incompatibililty with non cuda platform for nvfp4](../sources/prs/vllm/PR-23478.md), [[Bugfix] Fix Marlin NVFP4 for modelopt](../sources/prs/vllm/PR-23659.md), [[NVIDIA] Support SiluMul + NVFP4 quant fusion](../sources/prs/vllm/PR-23671.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files](../sources/prs/vllm/PR-23727.md), [[BUGFIX ] fix undefined silu_and_mul_nvfp4_quant](../sources/prs/vllm/PR-23929.md), [[Transform] [Quantization] Add QuTLASS support to vLLM](../sources/prs/vllm/PR-24440.md), [[Bugfix] Fix accuracy issue for silu_mul + nvfp4 quant fusion kernel](../sources/prs/vllm/PR-24833.md), [[Compile] Fix Compile Warning for Ignoring `MIN_BLOCK_PER_SM`](../sources/prs/vllm/PR-25193.md), [Enable Fbgemm NVFP4 on Dense models](../sources/prs/vllm/PR-25609.md), [[Quantization/NVFP4] Speed up TRTLLM NVFP4 MOE weight loading and fix K/V scale loading for MLA Attn](../sources/prs/vllm/PR-25968.md), [[Bugfix] Allow skipping MoE in NVFP4 (fix for MTP)](../sources/prs/vllm/PR-25987.md), [[MoE] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked](../sources/prs/vllm/PR-25990.md), [[NVIDIA] Add support for cudnn fp4 gemm via flashinfer](../sources/prs/vllm/PR-26107.md), [[ModelOpt] Load w13/w2_input_scale for all experts, nvfp4](../sources/prs/vllm/PR-26135.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [Add TRTLLM MoE NVFP4 kernel to CompressedTensorsW4A4MoeMethod](../sources/prs/vllm/PR-28892.md), [[Feat] Support non-gated activations in NVFP4 modelopt path](../sources/prs/vllm/PR-29004.md), [[Kernel] Add NVFP4 MoE CUTLASS support for SM120](../sources/prs/vllm/PR-29242.md), [[Bugfix] Fix mismatched nvfp4 gemm output shape](../sources/prs/vllm/PR-29742.md), [[EPLB] Support EPLB w/ NVFP4](../sources/prs/vllm/PR-29804.md), [[Compressed-Tensors] Simplify NVFP4 Conditions, enable marlin support for NVFP4A16 MoEs](../sources/prs/vllm/PR-30881.md), [[Kernel][Performance] Enable smaller Scaling Factor tiling for NVFP4 small-batch decoding](../sources/prs/vllm/PR-30885.md), [[NVFP4][Perf] Tune NVFP4 input quant kernel for small batch size](../sources/prs/vllm/PR-30897.md), [ [FIX] Always support TP > 4 for FP4 Gemm](../sources/prs/vllm/PR-31099.md), [[Bugfix] Fix Broken ModelOpt NVFP4 MoE](../sources/prs/vllm/PR-31742.md), [[Perf] Fuse stride preparation for NVFP4 cutlass_moe](../sources/prs/vllm/PR-31837.md), [[Perf][Kernel] Optimize FP4 quantization kernels (SM100F)](../sources/prs/vllm/PR-32520.md), [Support compress-tensors with nvfp4 or fp8 weights and modelopt with nvfp4 weights on Turing](../sources/prs/vllm/PR-33076.md), [fix: Add SM120 (RTX Blackwell) support for FlashInfer CUTLASS NVFP4 MoE kernels](../sources/prs/vllm/PR-33417.md), [[Kernel] Support Flashinfer trtllm fused MoE non gated FP8 & NVFP4](../sources/prs/vllm/PR-33506.md), [[Bugfix] Fix DSV3.2 NVFP4](../sources/prs/vllm/PR-33932.md), [[ModelBash][DSR1 NVFp4] Avoid Bf16 Bias Cast](../sources/prs/vllm/PR-34298.md), [[Custom Ops] Add functional + out variant for scaled_fp4_quant](../sources/prs/vllm/PR-34389.md), [[BUGFIX] Fix accuracy regression for NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with TP>1](../sources/prs/vllm/PR-34476.md), [[Model] Add NVFP4 quantization support for Step3.5-Flash](../sources/prs/vllm/PR-34478.md), [[Bugfix] Rescale NVFP4 weight scales to fix BF16 dequant underflow](../sources/prs/vllm/PR-34577.md), [[Bugfix] Fix NVFP4 TRTLLM MoE non-gated support; add gsm8k for Nemotron-3-Nano FP8+NVFP4](../sources/prs/vllm/PR-34725.md), [[BugFix] Fix fp4 quant kernel on CUDA 12.8](../sources/prs/vllm/PR-35210.md), [[NVFP4] Support NVFP4 dense models from `modelopt` and `compressed-tensors` on AMD Instinct MI300, MI355X and Hopper through emulation](../sources/prs/vllm/PR-35733.md), [[Bugfix] Fix passing of activation_type to trtllm fused MoE NVFP4 and FP8](../sources/prs/vllm/PR-36017.md), [[Mamba] Flashinfer selective_state_update](../sources/prs/vllm/PR-36162.md), [[mla] Support fused FP8/NVFP4 output quantization in MLA attention (#35792)](../sources/prs/vllm/PR-36205.md), [[Bug][MoE] Fix TRTLLM NVFP4 Routing Kernel Precision](../sources/prs/vllm/PR-36725.md), [[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts](../sources/prs/vllm/PR-36728.md), [[MoE Refactor] Mxfp4 oracle rebased](../sources/prs/vllm/PR-37128.md), [Fix minimax m2.5 nvfp4 kv scales weight loading](../sources/prs/vllm/PR-37214.md), [[MoE/EPLB] Fix FlashInfer nvfp4 experts + EPLB correctness](../sources/prs/vllm/PR-37217.md), [[Kernel] Add non-gated support for NVFP4 CUTLASS MoE](../sources/prs/vllm/PR-37320.md), [[Bugfix] Fix EP weight filter breaking EPLB and NVFP4 accuracy](../sources/prs/vllm/PR-37322.md), [Add nvfp4 support to reshape_and_cache_flash](../sources/prs/vllm/PR-37332.md), [[Bugfix] Remove assertion for NVFP4 scale dynamic range](../sources/prs/vllm/PR-37465.md), [[Bugfix] Fix marlin nvfp4 rescaling](../sources/prs/vllm/PR-37502.md), [[4/n] Migrate FP4/W4A8 CUTLASS kernels to torch stable ABI](../sources/prs/vllm/PR-37503.md), [[Perf] Use torch compile to fuse pack topk in trtllm moe](../sources/prs/vllm/PR-37695.md), [[Bugfix] Preserve CUDA arch suffix (a/f) for SM12x — fixes NVFP4 NaN on desktop Blackwell](../sources/prs/vllm/PR-37725.md), [[MoE] Move FlashInfer CuteDSL experts into fused_moe/experts/](../sources/prs/vllm/PR-37759.md), [[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration](../sources/prs/vllm/PR-38050.md), [[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell](../sources/prs/vllm/PR-38083.md), [[Quantization] Add FlashInfer CuteDSL batched experts backend for NVFP4 MoE](../sources/prs/vllm/PR-38251.md), [[MoE] Add RoutingMethodType.Simulated to TRT-LLM FP8/NVFP4 kernel allowlists](../sources/prs/vllm/PR-38329.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[Compile] Fix nvfp4 compile warning](../sources/prs/vllm/PR-38573.md), [[Bugfix] Fix NVFP4+MTP crash: force unquantized mtp.fc for Qwen3.5](../sources/prs/vllm/PR-38832.md), [[MoE Refactor] Split up compressed_tensors_moe.py](../sources/prs/vllm/PR-38960.md), [[Refactor] Move NVFP4 GEMM management into NvFp4LinearKernel](../sources/prs/vllm/PR-39129.md), [[Feature] Batch invariant nvfp4 linear support](../sources/prs/vllm/PR-39322.md), [[Kernel] Support TRTLLM GEN NVFP4 MoE for non-512-aligned hidden dims via weight padding](../sources/prs/vllm/PR-39510.md), [[Bugfix] Reject non-nvfp4 dtypes when using the flashinfer_nvlink_one_sided all2all backend](../sources/prs/vllm/PR-39717.md), [[Bug] Fix batch invariance nvfp4 support](../sources/prs/vllm/PR-39820.md), [Add nvfp4 kv cache support](../sources/prs/vllm/PR-40177.md), [[Bugfix] Guard mxfp4_experts_quant bindings on ENABLE_NVFP4_SM100](../sources/prs/vllm/PR-40191.md), [[MoE] Move cutlass moe to fused_moe/experts/](../sources/prs/vllm/PR-40574.md), [[Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4](../sources/prs/vllm/PR-41050.md), [Add NVFP4 all-gather GEMM fusion for AsyncTP](../sources/prs/vllm/PR-41882.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement](../sources/prs/vllm/PR-42774.md), [NVFP4 and Block-Scaled Narrow Precision](../wiki/hardware/nvfp4.md), [Gated Dual GEMM (Gate-Up + SwiGLU Fusion)](../wiki/kernels/gated-dual-gemm.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Fine-Grained FP8/FP4 Quantization](../wiki/techniques/fine-grained-quantization.md) | | `pdl` | [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [Programmatic Dependent Launch / Grid Dependency Control](../wiki/hardware/pdl-gdc.md) | | `tcgen05` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Colfax CUTLASS Tutorial: GEMM Kernels Using Tensor Memory for Blackwell](../sources/blogs/colfax-cutlass-blackwell.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [FlashAttention-4 Blog](../sources/blogs/flash-attention-4.md), [FlashMLA — Multi-head Latent Attention](../sources/blogs/flashmla.md), [Writing High-Performance Matrix Multiplication Kernels for Blackwell with JAX Pallas](../sources/blogs/jax-pallas-blackwell-matmul.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [FlashAttention-4: Hardware-Friendly Attention on Blackwell](../sources/docs/flash-attention-4.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Tiled Flash Linear Attention (TFLA)](../sources/docs/tfla.md), [Triton 3.6.0 Release Notes — Blackwell (SM100) Lowering](../sources/docs/triton-3.6-blackwell.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [Support for GEMM-K=0 for Blackwell Grouped GEMMs](../sources/prs/cutlass/PR-2746.md), [[Cutlass gemm] Fix SM100 FP8 nosmem epilogue-fusion shape_div 'Divisibility Condition' for non-multiple-of-64 N tiles](../sources/prs/cutlass/PR-2946.md), [[CuTeDSL] Fix: SM100 block-scale gemm overlapping accumulator](../sources/prs/cutlass/PR-2995.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feature] Support tcgen5mma lowering for `.kind::i8`](../sources/prs/tilelang/PR-1764.md), [[CUDA] Support tcgen5mma gemm ts](../sources/prs/tilelang/PR-1866.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [[Bugfix] Fix default enable for CUTLASS MLA on SM100](../sources/prs/vllm/PR-22738.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [Two-SM Cooperative MMA](../wiki/hardware/2sm-cooperative.md), [tcgen05.mma — Blackwell MMA Instruction](../wiki/hardware/tcgen05-mma.md), [Tensor Memory (TMEM)](../wiki/hardware/tmem.md), [FlashAttention-4](../wiki/kernels/flash-attention-4.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [CUDA C++ for Blackwell Kernels](../wiki/languages/cuda-cpp.md), [CuTe DSL for Blackwell](../wiki/languages/cute-dsl.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Register Accumulators to TMEM](../wiki/migration/register-to-tmem.md), [Migrating from wgmma to tcgen05](../wiki/migration/wgmma-to-tcgen05.md), [Not Reaching Peak FLOPS](../wiki/patterns/compute-bound.md), [Pipeline Stalls](../wiki/patterns/pipeline-stalls.md), [Warp Specialization on Blackwell](../wiki/techniques/warp-specialization.md) | -| `tma` | [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [Writing High-Performance Matrix Multiplication Kernels for Blackwell with JAX Pallas](../sources/blogs/jax-pallas-blackwell-matmul.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [simveit effective_transpose](../sources/blogs/simveit-effective-transpose.md), [simveit load_and_store](../sources/blogs/simveit-load-and-store.md), [TFLOPS Gap: Why FP4 MoE Kernel Engineering Matters on Blackwell](../sources/blogs/tflops-gap-fp4-moe.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[TRTLLM-10022][feat] Add hopper xqa decode support for skip softmax attention](../sources/prs/TensorRT-LLM/PR-10264.md), [[https://nvbugs/5669671][fix] Support GuidedDecoder with sharded logits (pick #10698)](../sources/prs/TensorRT-LLM/PR-10742.md), [[TRTLLM-9831][perf] Use TMA.RED to improve effective memory bandwidth](../sources/prs/TensorRT-LLM/PR-10987.md), [[TRTLLM-11092][feat] add support for visual gen FA4 attention backend](../sources/prs/TensorRT-LLM/PR-11697.md), [[None][feat] Support sparse mqa/gqa attention](../sources/prs/TensorRT-LLM/PR-12470.md), [[None][feat] Trtllm-gen FMHA JIT support](../sources/prs/TensorRT-LLM/PR-12612.md), [[TRTLLM-11485][feat] Feature rework: Add SageAttention refreshed kernels (attentionOp only)](../sources/prs/TensorRT-LLM/PR-12937.md), [[None][perf] Drop cubin and Eliminate ~6s FMHA JIT recompile in eager generation by aligning kernel selection with CUDA graph warmup](../sources/prs/TensorRT-LLM/PR-13505.md), [[None][feat] Add DeepSeekV4 attention kernels](../sources/prs/TensorRT-LLM/PR-13652.md), [[None][feat] Update the logic of FMHA JIT path](../sources/prs/TensorRT-LLM/PR-14291.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[None][feat] GPT-OSS Sm120/Sm121 Support](../sources/prs/TensorRT-LLM/PR-7937.md), [[None][feat] Update TRTLLM MoE cubins; reduce mxfp4 weight padding requirement; tighten TMA bound](../sources/prs/TensorRT-LLM/PR-9025.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [[EVT] Add support for Row/Col broadcast PtrArray](../sources/prs/cutlass/PR-2033.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Hopper Grouped GEMM support for FP8 Accum](../sources/prs/cutlass/PR-2123.md), [Flash MLA support](../sources/prs/cutlass/PR-2130.md), [Flash MLA Support - Step 2](../sources/prs/cutlass/PR-2134.md), [Fix sm100 gemm wrong static constexpr that breaks compilation on Windows](../sources/prs/cutlass/PR-2167.md), [Fix SM90 beta=1 hang and stream-K launch errors](../sources/prs/cutlass/PR-2172.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [[ex77] fix mla split; add fwd lse; add bwd varlen](../sources/prs/cutlass/PR-2366.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [fix: examples/cute/tutorial/blackwell/04_mma_tma_2sm_sm100.cu GridDim miscalculated](../sources/prs/cutlass/PR-2492.md), [Support PDL for SM90 Array TMA GEMM](../sources/prs/cutlass/PR-2719.md), [[Bug Fix]Bypass launch grids for SM120 Kernel with SM90 Mainloop & SM100 TileScheduler](../sources/prs/cutlass/PR-2865.md), [new example with TMA prefetch feature targeting for DRAM latency boun…](../sources/prs/cutlass/PR-2881.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Replace std::min with cute::min in sm120 blockwise scaling device functions](../sources/prs/cutlass/PR-3055.md), [Small Tile N BlockScaled GEMM + Grouped GEMM on SM12x](../sources/prs/cutlass/PR-3176.md), [Add var-seq-len to FA3 fp16 / bf16 fwd](../sources/prs/flash-attention/PR-1072.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG](../sources/prs/flash-attention/PR-1173.md), [Add seqused_q in fwd / bwd and seqused_k in bwd in hopper FA.](../sources/prs/flash-attention/PR-1182.md), [Add local attention in Hopper FAv3](../sources/prs/flash-attention/PR-1233.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [Paged Attention support for FA3](../sources/prs/flash-attention/PR-1268.md), [FA3 paged attention: Readiness for Cutlass 3.6 / default value for block_table](../sources/prs/flash-attention/PR-1331.md), [Fix FA3 Varlen Performance regression](../sources/prs/flash-attention/PR-1361.md), [Support hdimQK != hdimV backward](../sources/prs/flash-attention/PR-1604.md), [Improve causal backward determinism perf with SPT schedule](../sources/prs/flash-attention/PR-1893.md), [[Cute,Sm100,Fwd] use correction warps for epi when not using TMA](../sources/prs/flash-attention/PR-2014.md), [[Cute,Fwd,Sm100] don't pass mask_fn to softmax_step generically](../sources/prs/flash-attention/PR-2026.md), [[Cute,Fwd] Extend score_mod to variable sequence length](../sources/prs/flash-attention/PR-2043.md), [Add score-mod bwd support ](../sources/prs/flash-attention/PR-2070.md), [[Cute,Fwd,Sm100] fp8 e4m3 and e5m2 support](../sources/prs/flash-attention/PR-2109.md), [[Fwd,Sm90] Add paged KV attention support (tma and cp.async)](../sources/prs/flash-attention/PR-2360.md), [[Cute,Sm100,Fwd] add MLA 64/512 with topk sparsity for MQA 128 heads](../sources/prs/flash-attention/PR-2441.md), [[hd256] Add TMA paged KV support to SM100 2CTA forward kernel](../sources/prs/flash-attention/PR-2489.md), [[FA4][hd256] Backward TMA bulk-store epilogue + LSE/dpsum coalesce](../sources/prs/flash-attention/PR-2497.md), [feat: Softmax free sampling](../sources/prs/flashinfer/PR-1035.md), [[nvidia] initial support for blackwell kernels](../sources/prs/flashinfer/PR-1039.md), [bugfix: adding lse output to blackwell fmha kernels](../sources/prs/flashinfer/PR-1071.md), [bugfix: follow user-specified sm_scale for blackwell cutlass fmha](../sources/prs/flashinfer/PR-1072.md), [bugfix: host-precomuted plan function for blackwell fmha](../sources/prs/flashinfer/PR-1106.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [feat: Fused temperature online softmax kernel](../sources/prs/flashinfer/PR-1153.md), [feat: logits processor fustion rule for temperature softmax](../sources/prs/flashinfer/PR-1170.md), [bugfix: softmax NaN results caused by large -inf masks](../sources/prs/flashinfer/PR-1178.md), [bugfix: fix blackwell fmha hanging issue for empty kv_len](../sources/prs/flashinfer/PR-1198.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [bugfix: fix fused-temperature softmax IMA issue](../sources/prs/flashinfer/PR-1596.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [Tune kernel compilation parameters for https://github.com/flashinfer-ai/flashinfer/pull/1850 ](../sources/prs/flashinfer/PR-1878.md), [Update trtllm-gen fused moe routing kernel and add more kernels](../sources/prs/flashinfer/PR-1955.md), [Feature: Add support for L40 FusedMoE in cutlass path](../sources/prs/flashinfer/PR-1973.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [perf: improve sampling/mask/softmax performance (part 1/2)](../sources/prs/flashinfer/PR-2044.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [update xqa license](../sources/prs/flashinfer/PR-2117.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [feat: MxInt4 x Bf16 TRT-LLM Gen MoE support](../sources/prs/flashinfer/PR-2159.md), [refactor: pull trtllm-gen batch-gemm/gemm headers from artifactory; update tma descriptor shape init](../sources/prs/flashinfer/PR-2235.md), [feat: add GDN Attention](../sources/prs/flashinfer/PR-2276.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [feat: update trtllm-gen MoE cubins](../sources/prs/flashinfer/PR-2416.md), [refactor: reduce hopper's gdn prefill compilation time and fix docstring.](../sources/prs/flashinfer/PR-2422.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [feat: Add TRTLLM-Gen Skip-Softmax kernels for prefill and decode](../sources/prs/flashinfer/PR-2477.md), [feat: Enable TRTLLM-Gen Skip-Softmax attention for MLA](../sources/prs/flashinfer/PR-2547.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor](../sources/prs/flashinfer/PR-2740.md), [Upgrade cutlass 4.2.1 -> 4.4.2](../sources/prs/flashinfer/PR-2798.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [feat(gdn): state checkpointing in chunk_gated_delta_rule](../sources/prs/flashinfer/PR-2908.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [[inductor][triton 3.3] Fix cpp_wrapper w/ TMA in triton 3.3](../sources/prs/pytorch/PR-149993.md), [[user triton] AOT inductor support for device-side TMA](../sources/prs/pytorch/PR-157241.md), [[kernel][moe] add moe topk fast](../sources/prs/sglang/PR-13969.md), [Move fa4 from sgl-kernel to jit kernel](../sources/prs/sglang/PR-17353.md), [Support skip-softmax attention](../sources/prs/sglang/PR-19089.md), [[Kernel] Fuse temperature + softmax in sampling for decode speedup](../sources/prs/sglang/PR-20501.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt](../sources/prs/sglang/PR-4515.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [[perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128](../sources/prs/sglang/PR-6929.md), [[1/n]: add cutlass W4A8 moe kernel for hopper architecture](../sources/prs/sglang/PR-7772.md), [[NVIDIA] Fix missing `get_col_major_tma_aligned_tensor` for Blackwell deepgemm in EpMoE](../sources/prs/sglang/PR-8955.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [[BugFix] Fix Hopper TMA lowering without warp specialization](../sources/prs/tilelang/PR-1840.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [[TMA] Support FP4 TensorMap TMA copies](../sources/prs/tilelang/PR-2107.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [Add: Support for Sparse24Bitmask Compressed Models](../sources/prs/vllm/PR-12097.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [Fix `numel()` downcast in vllm/csrc/moe/moe_align_sum_kernels.cu +2](../sources/prs/vllm/PR-17082.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for DeepSeek TP](../sources/prs/vllm/PR-20769.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Kernel] Add topk_sigmoid kernel](../sources/prs/vllm/PR-31246.md), [Add TMA support to fused_moe_lora kernel](../sources/prs/vllm/PR-32195.md), [[Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper](../sources/prs/vllm/PR-32619.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[Perf][GDN] Align TMA usage with upstream FLA](../sources/prs/vllm/PR-38981.md), [fix: clamp NaN/Inf in topk_softmax to prevent duplicate expert IDs](../sources/prs/vllm/PR-39391.md), [mbarrier (Memory Barrier Primitives)](../wiki/hardware/mbarrier.md), [Tensor Memory Accelerator (TMA)](../wiki/hardware/tma.md), [FlashAttention SM100 MLA TopK Sparse Forward](../wiki/kernels/flash-attention-sm100-mla-topk.md), [CuTe DSL for Blackwell](../wiki/languages/cute-dsl.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Pipeline Stalls](../wiki/patterns/pipeline-stalls.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md), [Software Pipelining and Multi-Stage Buffering](../wiki/techniques/pipeline-stages.md), [Shared Memory Swizzling](../wiki/techniques/swizzling.md) | +| `tma` | [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [Writing High-Performance Matrix Multiplication Kernels for Blackwell with JAX Pallas](../sources/blogs/jax-pallas-blackwell-matmul.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [simveit effective_transpose](../sources/blogs/simveit-effective-transpose.md), [simveit load_and_store](../sources/blogs/simveit-load-and-store.md), [TFLOPS Gap: Why FP4 MoE Kernel Engineering Matters on Blackwell](../sources/blogs/tflops-gap-fp4-moe.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[TRTLLM-10022][feat] Add hopper xqa decode support for skip softmax attention](../sources/prs/TensorRT-LLM/PR-10264.md), [[https://nvbugs/5669671][fix] Support GuidedDecoder with sharded logits (pick #10698)](../sources/prs/TensorRT-LLM/PR-10742.md), [[TRTLLM-9831][perf] Use TMA.RED to improve effective memory bandwidth](../sources/prs/TensorRT-LLM/PR-10987.md), [[TRTLLM-11092][feat] add support for visual gen FA4 attention backend](../sources/prs/TensorRT-LLM/PR-11697.md), [[None][feat] Support sparse mqa/gqa attention](../sources/prs/TensorRT-LLM/PR-12470.md), [[None][feat] Trtllm-gen FMHA JIT support](../sources/prs/TensorRT-LLM/PR-12612.md), [[TRTLLM-11485][feat] Feature rework: Add SageAttention refreshed kernels (attentionOp only)](../sources/prs/TensorRT-LLM/PR-12937.md), [[None][perf] Drop cubin and Eliminate ~6s FMHA JIT recompile in eager generation by aligning kernel selection with CUDA graph warmup](../sources/prs/TensorRT-LLM/PR-13505.md), [[None][feat] Add DeepSeekV4 attention kernels](../sources/prs/TensorRT-LLM/PR-13652.md), [[None][feat] Update the logic of FMHA JIT path](../sources/prs/TensorRT-LLM/PR-14291.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[None][feat] GPT-OSS Sm120/Sm121 Support](../sources/prs/TensorRT-LLM/PR-7937.md), [[None][feat] Update TRTLLM MoE cubins; reduce mxfp4 weight padding requirement; tighten TMA bound](../sources/prs/TensorRT-LLM/PR-9025.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [[EVT] Add support for Row/Col broadcast PtrArray](../sources/prs/cutlass/PR-2033.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Hopper Grouped GEMM support for FP8 Accum](../sources/prs/cutlass/PR-2123.md), [Flash MLA support](../sources/prs/cutlass/PR-2130.md), [Flash MLA Support - Step 2](../sources/prs/cutlass/PR-2134.md), [Fix sm100 gemm wrong static constexpr that breaks compilation on Windows](../sources/prs/cutlass/PR-2167.md), [Fix SM90 beta=1 hang and stream-K launch errors](../sources/prs/cutlass/PR-2172.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [[ex77] fix mla split; add fwd lse; add bwd varlen](../sources/prs/cutlass/PR-2366.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [fix: examples/cute/tutorial/blackwell/04_mma_tma_2sm_sm100.cu GridDim miscalculated](../sources/prs/cutlass/PR-2492.md), [Support PDL for SM90 Array TMA GEMM](../sources/prs/cutlass/PR-2719.md), [[Bug Fix]Bypass launch grids for SM120 Kernel with SM90 Mainloop & SM100 TileScheduler](../sources/prs/cutlass/PR-2865.md), [new example with TMA prefetch feature targeting for DRAM latency boun…](../sources/prs/cutlass/PR-2881.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Replace std::min with cute::min in sm120 blockwise scaling device functions](../sources/prs/cutlass/PR-3055.md), [Small Tile N BlockScaled GEMM + Grouped GEMM on SM12x](../sources/prs/cutlass/PR-3176.md), [Add var-seq-len to FA3 fp16 / bf16 fwd](../sources/prs/flash-attention/PR-1072.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG](../sources/prs/flash-attention/PR-1173.md), [Add seqused_q in fwd / bwd and seqused_k in bwd in hopper FA.](../sources/prs/flash-attention/PR-1182.md), [Add local attention in Hopper FAv3](../sources/prs/flash-attention/PR-1233.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [Paged Attention support for FA3](../sources/prs/flash-attention/PR-1268.md), [FA3 paged attention: Readiness for Cutlass 3.6 / default value for block_table](../sources/prs/flash-attention/PR-1331.md), [Fix FA3 Varlen Performance regression](../sources/prs/flash-attention/PR-1361.md), [Support hdimQK != hdimV backward](../sources/prs/flash-attention/PR-1604.md), [Improve causal backward determinism perf with SPT schedule](../sources/prs/flash-attention/PR-1893.md), [[Cute,Sm100,Fwd] use correction warps for epi when not using TMA](../sources/prs/flash-attention/PR-2014.md), [[Cute,Fwd,Sm100] don't pass mask_fn to softmax_step generically](../sources/prs/flash-attention/PR-2026.md), [[Cute,Fwd] Extend score_mod to variable sequence length](../sources/prs/flash-attention/PR-2043.md), [Add score-mod bwd support ](../sources/prs/flash-attention/PR-2070.md), [[Cute,Fwd,Sm100] fp8 e4m3 and e5m2 support](../sources/prs/flash-attention/PR-2109.md), [[Fwd,Sm90] Add paged KV attention support (tma and cp.async)](../sources/prs/flash-attention/PR-2360.md), [[Cute,Sm100,Fwd] add MLA 64/512 with topk sparsity for MQA 128 heads](../sources/prs/flash-attention/PR-2441.md), [[hd256] Add TMA paged KV support to SM100 2CTA forward kernel](../sources/prs/flash-attention/PR-2489.md), [[FA4][hd256] Backward TMA bulk-store epilogue + LSE/dpsum coalesce](../sources/prs/flash-attention/PR-2497.md), [feat: Softmax free sampling](../sources/prs/flashinfer/PR-1035.md), [[nvidia] initial support for blackwell kernels](../sources/prs/flashinfer/PR-1039.md), [bugfix: adding lse output to blackwell fmha kernels](../sources/prs/flashinfer/PR-1071.md), [bugfix: follow user-specified sm_scale for blackwell cutlass fmha](../sources/prs/flashinfer/PR-1072.md), [bugfix: host-precomuted plan function for blackwell fmha](../sources/prs/flashinfer/PR-1106.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [feat: Fused temperature online softmax kernel](../sources/prs/flashinfer/PR-1153.md), [feat: logits processor fustion rule for temperature softmax](../sources/prs/flashinfer/PR-1170.md), [bugfix: softmax NaN results caused by large -inf masks](../sources/prs/flashinfer/PR-1178.md), [bugfix: fix blackwell fmha hanging issue for empty kv_len](../sources/prs/flashinfer/PR-1198.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [bugfix: fix fused-temperature softmax IMA issue](../sources/prs/flashinfer/PR-1596.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [Tune kernel compilation parameters for https://github.com/flashinfer-ai/flashinfer/pull/1850 ](../sources/prs/flashinfer/PR-1878.md), [Update trtllm-gen fused moe routing kernel and add more kernels](../sources/prs/flashinfer/PR-1955.md), [Feature: Add support for L40 FusedMoE in cutlass path](../sources/prs/flashinfer/PR-1973.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [perf: improve sampling/mask/softmax performance (part 1/2)](../sources/prs/flashinfer/PR-2044.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [update xqa license](../sources/prs/flashinfer/PR-2117.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [feat: MxInt4 x Bf16 TRT-LLM Gen MoE support](../sources/prs/flashinfer/PR-2159.md), [refactor: pull trtllm-gen batch-gemm/gemm headers from artifactory; update tma descriptor shape init](../sources/prs/flashinfer/PR-2235.md), [feat: add GDN Attention](../sources/prs/flashinfer/PR-2276.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [feat: update trtllm-gen MoE cubins](../sources/prs/flashinfer/PR-2416.md), [refactor: reduce hopper's gdn prefill compilation time and fix docstring.](../sources/prs/flashinfer/PR-2422.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [feat: Add TRTLLM-Gen Skip-Softmax kernels for prefill and decode](../sources/prs/flashinfer/PR-2477.md), [feat: Enable TRTLLM-Gen Skip-Softmax attention for MLA](../sources/prs/flashinfer/PR-2547.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor](../sources/prs/flashinfer/PR-2740.md), [Upgrade cutlass 4.2.1 -> 4.4.2](../sources/prs/flashinfer/PR-2798.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [feat(gdn): state checkpointing in chunk_gated_delta_rule](../sources/prs/flashinfer/PR-2908.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [[inductor][triton 3.3] Fix cpp_wrapper w/ TMA in triton 3.3](../sources/prs/pytorch/PR-149993.md), [[user triton] AOT inductor support for device-side TMA](../sources/prs/pytorch/PR-157241.md), [[kernel][moe] add moe topk fast](../sources/prs/sglang/PR-13969.md), [Move fa4 from sgl-kernel to jit kernel](../sources/prs/sglang/PR-17353.md), [Support skip-softmax attention](../sources/prs/sglang/PR-19089.md), [[Kernel] Fuse temperature + softmax in sampling for decode speedup](../sources/prs/sglang/PR-20501.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt](../sources/prs/sglang/PR-4515.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [[perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128](../sources/prs/sglang/PR-6929.md), [[1/n]: add cutlass W4A8 moe kernel for hopper architecture](../sources/prs/sglang/PR-7772.md), [[NVIDIA] Fix missing `get_col_major_tma_aligned_tensor` for Blackwell deepgemm in EpMoE](../sources/prs/sglang/PR-8955.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [[BugFix] Fix Hopper TMA lowering without warp specialization](../sources/prs/tilelang/PR-1840.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [[TMA] Support FP4 TensorMap TMA copies](../sources/prs/tilelang/PR-2107.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [Add: Support for Sparse24Bitmask Compressed Models](../sources/prs/vllm/PR-12097.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [Fix `numel()` downcast in vllm/csrc/moe/moe_align_sum_kernels.cu +2](../sources/prs/vllm/PR-17082.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for DeepSeek TP](../sources/prs/vllm/PR-20769.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Kernel] Add topk_sigmoid kernel](../sources/prs/vllm/PR-31246.md), [Add TMA support to fused_moe_lora kernel](../sources/prs/vllm/PR-32195.md), [[Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper](../sources/prs/vllm/PR-32619.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[Perf][GDN] Align TMA usage with upstream FLA](../sources/prs/vllm/PR-38981.md), [fix: clamp NaN/Inf in topk_softmax to prevent duplicate expert IDs](../sources/prs/vllm/PR-39391.md), [mbarrier (Memory Barrier Primitives)](../wiki/hardware/mbarrier.md), [Tensor Memory Accelerator (TMA)](../wiki/hardware/tma.md), [FlashAttention SM100 MLA TopK Sparse Forward](../wiki/kernels/flash-attention-sm100-mla-topk.md), [CuTe DSL for Blackwell](../wiki/languages/cute-dsl.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86)](../wiki/migration/hopper-to-ampere-backport.md), [Pipeline Stalls](../wiki/patterns/pipeline-stalls.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md), [Software Pipelining and Multi-Stage Buffering](../wiki/techniques/pipeline-stages.md), [Shared Memory Swizzling](../wiki/techniques/swizzling.md) | | `tmem` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Colfax CUTLASS Tutorial: GEMM Kernels Using Tensor Memory for Blackwell](../sources/blogs/colfax-cutlass-blackwell.md), [FlashAttention-4 Blog](../sources/blogs/flash-attention-4.md), [FlashMLA — Multi-head Latent Attention](../sources/blogs/flashmla.md), [Writing High-Performance Matrix Multiplication Kernels for Blackwell with JAX Pallas](../sources/blogs/jax-pallas-blackwell-matmul.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [FlashAttention-4: Hardware-Friendly Attention on Blackwell](../sources/docs/flash-attention-4.md), [NVIDIA Blackwell Tuning Guide](../sources/docs/nvidia-blackwell-tuning-guide.md), [NVIDIA CUDA Toolkit 13.x for Blackwell](../sources/docs/nvidia-cuda-13.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [Triton 3.6.0 Release Notes — Blackwell (SM100) Lowering](../sources/docs/triton-3.6-blackwell.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [[TRTLLM-9992][perf] Enable PDL for CuteDSL kernels and overlap MoeOutputMemset](../sources/prs/TensorRT-LLM/PR-10043.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [Support for GEMM-K=0 for Blackwell Grouped GEMMs](../sources/prs/cutlass/PR-2746.md), [[CuTeDSL] Fix: SM100 block-scale gemm overlapping accumulator](../sources/prs/cutlass/PR-2995.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [[Bugfix] Fix default enable for CUTLASS MLA on SM100](../sources/prs/vllm/PR-22738.md), [tcgen05.mma — Blackwell MMA Instruction](../wiki/hardware/tcgen05-mma.md), [Tensor Memory (TMEM)](../wiki/hardware/tmem.md), [FlashAttention-4](../wiki/kernels/flash-attention-4.md), [Gated Dual GEMM (Gate-Up + SwiGLU Fusion)](../wiki/kernels/gated-dual-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [CUDA C++ for Blackwell Kernels](../wiki/languages/cuda-cpp.md), [CuTe DSL for Blackwell](../wiki/languages/cute-dsl.md), [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md), [Register Accumulators to TMEM](../wiki/migration/register-to-tmem.md), [Migrating from wgmma to tcgen05](../wiki/migration/wgmma-to-tcgen05.md), [Register Pressure — Low Occupancy](../wiki/patterns/register-pressure.md), [Double/Multi-Buffering Patterns](../wiki/techniques/double-buffering.md), [Epilogue Fusion](../wiki/techniques/epilogue-fusion.md), [Kernel Fusion](../wiki/techniques/kernel-fusion.md), [Ping-Pong Scheduling](../wiki/techniques/ping-pong-scheduling.md), [Warp Specialization on Blackwell](../wiki/techniques/warp-specialization.md) | -| `wgmma` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [simveit load_and_store](../sources/blogs/simveit-load-and-store.md), [Tiled Flash Linear Attention (TFLA)](../sources/docs/tfla.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [Add swizzle layout detection and automatic merging for layout conflicts](../sources/prs/tilelang/PR-1736.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Migrating from wgmma to tcgen05](../wiki/migration/wgmma-to-tcgen05.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md) | +| `wgmma` | [Microbenchmarking NVIDIA's Blackwell Architecture](../sources/blogs/blackwell-microbenchmarking.md), [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [DeepGEMM — FP8 GEMM Library](../sources/blogs/deepgemm.md), [simveit load_and_store](../sources/blogs/simveit-load-and-store.md), [Tiled Flash Linear Attention (TFLA)](../sources/docs/tfla.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [Add swizzle layout detection and automatic merging for layout conflicts](../sources/prs/tilelang/PR-1736.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86)](../wiki/migration/hopper-to-ampere-backport.md), [Migrating from wgmma to tcgen05](../wiki/migration/wgmma-to-tcgen05.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md) | diff --git a/queries/by-language.md b/queries/by-language.md index 7864341e7..53ebe7ef8 100644 --- a/queries/by-language.md +++ b/queries/by-language.md @@ -4,11 +4,11 @@ | Language | Guide | Related Pages | |----------|-------|--------------| -| `cuda-cpp` | [CUDA C++ for Blackwell Kernels](../wiki/languages/cuda-cpp.md) | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [NVIDIA Developer Code Samples](../sources/blogs/nvidia-code-samples.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [NVIDIA Blackwell Compatibility Guide](../sources/docs/blackwell-compatibility-guide.md), [Fix performance issue of m-grouped contiguous GEMMs.](../sources/prs/DeepGEMM/PR-168.md), [Fix multicast bug and optimize masked GEMM](../sources/prs/DeepGEMM/PR-193.md), [fix: use SM90ArchSpec instead of SM100ArchSpec in sm90_bf16_k_grouped_gemm](../sources/prs/DeepGEMM/PR-270.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Use swizzling instead of padding](../sources/prs/DeepGEMM/PR-86.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[None][feat] sm100 weight-only kernel](../sources/prs/TensorRT-LLM/PR-10190.md), [[TRTLLM-10022][feat] Add hopper xqa decode support for skip softmax attention](../sources/prs/TensorRT-LLM/PR-10264.md), [[https://nvbugs/5669671][fix] Support GuidedDecoder with sharded logits (pick #10698)](../sources/prs/TensorRT-LLM/PR-10742.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11165.md), [[None][feat] Optimize super-v3 nvfp4 for better perf](../sources/prs/TensorRT-LLM/PR-11273.md), [[None][feat] Optimize by fuse nvfp4_quant to layernorm_gated for mamba2_mixer](../sources/prs/TensorRT-LLM/PR-11473.md), [[None][feat] TRT-LLM Gen MoE finalize kernel optimization](../sources/prs/TensorRT-LLM/PR-11501.md), [[None][feat] Add support for expert_number<=2048 and K<=32](../sources/prs/TensorRT-LLM/PR-11510.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[TRTLLM-11119][feat] Blackwell SageAttention, Integrate into AttentionOp API](../sources/prs/TensorRT-LLM/PR-11718.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11733.md), [[None][feat] Add fused DiT QK Norm + RoPE CUDA kernel for FLUX](../sources/prs/TensorRT-LLM/PR-11869.md), [[TRTLLM-10421][perf] Add fused cat+fp8_quantize CUDA kernel for DSA indexer](../sources/prs/TensorRT-LLM/PR-11899.md), [[#11694][feat] AutoDeploy: Improve the piecewise CG memory usage](../sources/prs/TensorRT-LLM/PR-11993.md), [[https://nvbugs/5955188][fix] Fix harmony parsers and WAR routing PDL for agentic coding use cases](../sources/prs/TensorRT-LLM/PR-12046.md), [[TRTLLM-11540][feat] Add EAGLE3 dynamic tree speculative decoding support](../sources/prs/TensorRT-LLM/PR-12062.md), [[None][feat] Minimax RMS norm optimization](../sources/prs/TensorRT-LLM/PR-12163.md), [[None][feat] Add fused allreduce+RMSNorm op and optional residual in …](../sources/prs/TensorRT-LLM/PR-12201.md), [[None][feat] Support update weight for nvfp4](../sources/prs/TensorRT-LLM/PR-12320.md), [[None][feat] Temporally-Correlated Heuristic-guided Indexer TopK for Sparse Attention](../sources/prs/TensorRT-LLM/PR-12385.md), [[None][perf] add Dynamic SMEM block routing in MOE](../sources/prs/TensorRT-LLM/PR-12456.md), [[None][feat] Support sparse mqa/gqa attention](../sources/prs/TensorRT-LLM/PR-12470.md), [[https://nvbugs/5983390][perf] Split MLA DSA custom op for piecewise CUDA graph capture](../sources/prs/TensorRT-LLM/PR-12503.md), [[None][feat] Add Mamba2 MTP SSM cache CUDA kernel for tree-based speculative decoding](../sources/prs/TensorRT-LLM/PR-12537.md), [[https://nvbugs/5983390][perf] Multiple host perf optimizations for DSA part](../sources/prs/TensorRT-LLM/PR-12581.md), [[None][feat] Trtllm-gen FMHA JIT support](../sources/prs/TensorRT-LLM/PR-12612.md), [[None][feat] Optimize mamba SSD prefill and extend flashinfer dispatch](../sources/prs/TensorRT-LLM/PR-12731.md), [[TRTLLM-11485][feat] Feature rework: Add SageAttention refreshed kernels (attentionOp only)](../sources/prs/TensorRT-LLM/PR-12937.md), [[None][feat] Update rms_norm + fp4_qaunt kernel supporting more dim](../sources/prs/TensorRT-LLM/PR-13033.md), [[#12716][feat] Fused cross-head QK Norm + RoPE kernel for WAN](../sources/prs/TensorRT-LLM/PR-13052.md), [[None][feat] Optimize causal_conv1d prefill and decode kernels](../sources/prs/TensorRT-LLM/PR-13103.md), [[None][feat] Add FP4 residual quantization kernel without channel reo…](../sources/prs/TensorRT-LLM/PR-13117.md), [[https://nvbugs/5945047][fix] Fix cluster launch enablement for SM120 GPUs in allReduce fusion](../sources/prs/TensorRT-LLM/PR-13169.md), [[None][feat] Integrate FP4 indexer for DSA on Blackwell](../sources/prs/TensorRT-LLM/PR-13340.md), [[None][perf] Extend customMoeRouting kernel to support Qwen3.5](../sources/prs/TensorRT-LLM/PR-13433.md), [[None][perf] Scheme X L2-aware dispatcher and PDL launchers for sparse-attention GVR Top-K](../sources/prs/TensorRT-LLM/PR-13477.md), [[None][perf] Drop cubin and Eliminate ~6s FMHA JIT recompile in eager generation by aligning kernel selection with CUDA graph warmup](../sources/prs/TensorRT-LLM/PR-13505.md), [[None][feat] Fuse FP8 1x128 quantize + UE8M0 scale pack on SM100](../sources/prs/TensorRT-LLM/PR-13628.md), [[#13580][fix] AutoDeploy: Support Gemma3n/4 E2B variants](../sources/prs/TensorRT-LLM/PR-13630.md), [[None][feat] Add DeepSeekV4 attention kernels](../sources/prs/TensorRT-LLM/PR-13652.md), [[https://nvbugs/6108841][fix] add hidden_dim=6144 router GEMM instantiation for GLM-5](../sources/prs/TensorRT-LLM/PR-13740.md), [[None][perf] Optimize DeepSeek-V4 compressor BF16 input](../sources/prs/TensorRT-LLM/PR-13761.md), [[None][fix] Plumb swiglu_limit through DeepGEMM and TRTLLMGen FP8 fused MoE](../sources/prs/TensorRT-LLM/PR-13767.md), [[None][fix] Fix fused MHC for DeepSeek-V4-Pro hidden size](../sources/prs/TensorRT-LLM/PR-13771.md), [[None][feat] Indexer topk opt](../sources/prs/TensorRT-LLM/PR-13811.md), [[None][perf] mHC fused_hc kernel optimizations + DS-V4 entry-boundary RMSNorm fold-in](../sources/prs/TensorRT-LLM/PR-13892.md), [[None][perf] Add CUDA q_b norm for DeepSeek V4](../sources/prs/TensorRT-LLM/PR-13975.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [[None][feat] DSv4: enable GVR Heuristic Top-K for compress_ratio=4](../sources/prs/TensorRT-LLM/PR-14219.md), [[None][feat] Update the logic of FMHA JIT path](../sources/prs/TensorRT-LLM/PR-14291.md), [[None][chore] Update Claude Code agents and skills](../sources/prs/TensorRT-LLM/PR-14344.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[None][chore] Fix kernel launch param and add TRTLLM MoE backend test](../sources/prs/TensorRT-LLM/PR-7524.md), [[None][fix] Fix and add test for TRTLLM MoE backend](../sources/prs/TensorRT-LLM/PR-7755.md), [[TRTLLM-8637][feat] Optimize the routing kernel for DeepseekV3 (MoE CUTLASS backend); Add support for 384 experts (MoE TRTLLM backend)](../sources/prs/TensorRT-LLM/PR-7761.md), [[None][feat] GPT-OSS Sm120/Sm121 Support](../sources/prs/TensorRT-LLM/PR-7937.md), [[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP](../sources/prs/TensorRT-LLM/PR-8501.md), [[None][feat] Enable nvfp4 cuda core for sm120](../sources/prs/TensorRT-LLM/PR-8620.md), [[None][feat] Update TRTLLM MoE cubins; reduce mxfp4 weight padding requirement; tighten TMA bound](../sources/prs/TensorRT-LLM/PR-9025.md), [[None][fix] support topk autotuner input for expert slot per group larger than 32](../sources/prs/TensorRT-LLM/PR-9087.md), [[None][feat] TRT-LLM Gen MoE optimize DeepSeek Fp8 activation kernel](../sources/prs/TensorRT-LLM/PR-9175.md), [[None][feat] add fp4 gemm + allreduce](../sources/prs/TensorRT-LLM/PR-9729.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [[None][feat] Fused kernels (qknormrope + moe routing) and two-model MTP support for glm4moe](../sources/prs/TensorRT-LLM/PR-9852.md), [[None][feat] Port fp4 quantization kernel optimization from FlashInfer](../sources/prs/TensorRT-LLM/PR-9854.md), [[None][feat] Adding torch ext API for FusedAddRMSNormQuant kernel](../sources/prs/TensorRT-LLM/PR-9905.md), [[TRTLLM-9493][feat] Add helixPostProcessNative kernel for cp_dim=2](../sources/prs/TensorRT-LLM/PR-9924.md), [Experimental Python cooperative algorithms](../sources/prs/cccl/PR-1973.md), [fix thread-reduce performance regression](../sources/prs/cccl/PR-2944.md), [Fix scan / sm90 perf regression ](../sources/prs/cccl/PR-3236.md), [Fix the vectorized loading of BlockLoad](../sources/prs/cccl/PR-3517.md), [Add b200 tunings for scan.exclusive.sum](../sources/prs/cccl/PR-3559.md), [Fix SM100 histogram tunings](../sources/prs/cccl/PR-3691.md), [Split Optimize Warp Reduce PR - CUB part](../sources/prs/cccl/PR-4716.md), [Add nondeterministic reduce that uses atomics](../sources/prs/cccl/PR-4961.md), [CUB - Add internal integer utils and tests (Split `WarpReduce` PR)](../sources/prs/cccl/PR-5314.md), [Combine `block_reduce_warp_reduction_nondeterministic.cuh` specialization with original deterministic one ](../sources/prs/cccl/PR-5408.md), [Add dynamic CUB dispatch for segmented_sort](../sources/prs/cccl/PR-6069.md), [[CUB] Use `BlockLoadToShared` in `DeviceMerge`](../sources/prs/cccl/PR-6077.md), [Fix debug section around line 390 of dispatch_topk](../sources/prs/cccl/PR-6152.md), [Split fixed-size segmented reduce dispatch header](../sources/prs/cccl/PR-6597.md), [Integrate decoupled lookahead warpspeed scan](../sources/prs/cccl/PR-6811.md), [Use integer promotion for `warp_reduce`](../sources/prs/cccl/PR-6819.md), [Implement new tuning API arch dispatching](../sources/prs/cccl/PR-7093.md), [Two-phase reduction for fixed size segmented reduction for very large segment sizes](../sources/prs/cccl/PR-7114.md), [Implement the new tuning API for deterministic (rfa) reduce dispatch](../sources/prs/cccl/PR-7346.md), [Radix-selection based `BlockTopK` specialization](../sources/prs/cccl/PR-7384.md), [Implement the new tuning API for `DeviceRleDispatch`](../sources/prs/cccl/PR-7669.md), [Optimize non fixed size segmented reduce for small segments using max_segment_size](../sources/prs/cccl/PR-7718.md), [Add env SegmentedReduce (non fixed-size overloads)](../sources/prs/cccl/PR-7795.md), [Forward policy hub from `dispatch_streaming_arg_reduce_t` to `reduce::dispatch`](../sources/prs/cccl/PR-7805.md), [Implement the new tuning API for `detail::reduce::dispatch_streaming_arg_reduce_t`](../sources/prs/cccl/PR-7807.md), [Use the new tuning API internally for `detail::transform::dispatch`](../sources/prs/cccl/PR-7810.md), [[Backport branch/3.3.x] Forward policy hub from `dispatch_streaming_arg_reduce_t` to `reduce::dispatch`](../sources/prs/cccl/PR-7814.md), [Optimized Device-to-Device Tensor Copy (`cudax`)](../sources/prs/cccl/PR-7823.md), [Implement the new tuning API for `DispatchSegmentedRadixSort`](../sources/prs/cccl/PR-7844.md), [Implement the new tuning API for `DispatchSegmentedSort`](../sources/prs/cccl/PR-7874.md), [Implement the new tuning API for `DispatchTopK`](../sources/prs/cccl/PR-7928.md), [[cuda.compute]: Fix faulty pointer arithmetic calculation in CUB dispatch](../sources/prs/cccl/PR-7940.md), [Reduce usage of `cub::DispatchReduce`](../sources/prs/cccl/PR-7944.md), [Use the new tuning API for `detail::radix_sort::dispatch`](../sources/prs/cccl/PR-7949.md), [Adds support for non-fundamental types via decomposer to `DeviceTopK` ](../sources/prs/cccl/PR-8040.md), [Optimized Device-to-Device Tensor Copy (cudax) - Transpose Case](../sources/prs/cccl/PR-8125.md), [Avoid passing uninitialized values to scan_op](../sources/prs/cccl/PR-8184.md), [[STF] Move unstable_unique from STF to generic cudax utility](../sources/prs/cccl/PR-8190.md), [Expose `max_segment_size` guarantee in cuda.compute](../sources/prs/cccl/PR-8284.md), [Port `thrust::min|max_element` to CUB](../sources/prs/cccl/PR-8291.md), [Implement the new tuning API for `DispatchSelectIf`](../sources/prs/cccl/PR-8311.md), [simplify dispatch segmented reduce to use latest dispatch and new tunings API](../sources/prs/cccl/PR-8332.md), [Apply some random warpspeed tunings](../sources/prs/cccl/PR-8352.md), [[cub]: implement utilities for policy selection](../sources/prs/cccl/PR-8355.md), [Replace `detail::merge::dispatch` by CUB's public API](../sources/prs/cccl/PR-8381.md), [[CUB] Replace `Shuffle(Up|Down|Index)` with cuda::device::warp_shuffle - RadixSort only](../sources/prs/cccl/PR-8395.md), [Vectorize mbarrier initialization in warpspeed scan](../sources/prs/cccl/PR-8423.md), [[thrust] Single-pass `is_partitioned` via adjacent zip_iterator](../sources/prs/cccl/PR-8427.md), [Replace `detail::merge_sort::dispatch` by CUB's public API](../sources/prs/cccl/PR-8473.md), [Replace `detail::scan::dispatch` by CUB's public API](../sources/prs/cccl/PR-8495.md), [Implement the new tuning API for `detail::batched_topk::dispatch_batched_topk`](../sources/prs/cccl/PR-8538.md), [Replace `detail::for_each::dispatch` by CUB's public API](../sources/prs/cccl/PR-8565.md), [Replace `detail::segmented_reduce::dispatch` by the public API](../sources/prs/cccl/PR-8695.md), [Use the new tuning API internally for `detail::topk::dispatch`](../sources/prs/cccl/PR-8742.md), [Use the new tuning API internally for `detail::reduce_by_key::dispatch`](../sources/prs/cccl/PR-8756.md), [Use the new tuning API internally for `detail::reduce[_nd]::dispatch[_nd]`](../sources/prs/cccl/PR-8826.md), [Fix Warpspeed scan shifted output store](../sources/prs/cccl/PR-8839.md), [[cub] Simplify arch dispatch](../sources/prs/cccl/PR-8861.md), [Use the new tuning API internally for `detail::select::dispatch` and `DeviceSelect`](../sources/prs/cccl/PR-8880.md), [[STF] Add per-handle exec_place stream resources](../sources/prs/cccl/PR-8905.md), [Use the new tuning API internally for `detail::select|three_way_partition::dispatch` and `DevicePartition`](../sources/prs/cccl/PR-8925.md), [Use the new tuning API internally for `detail::segmented_radix_sort::dispatch`](../sources/prs/cccl/PR-8927.md), [[libcu++] Always suppress C++ extensions warnings in prologue](../sources/prs/cccl/PR-9019.md), [Fix segmented radix sort benchmark segment size type](../sources/prs/cccl/PR-9039.md), [[libcu++] Fix default make_shared_resource construction](../sources/prs/cccl/PR-9044.md), [Vectorize contiguous iterators in `cub::BlockLoad`/`Store`](../sources/prs/cccl/PR-9056.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [[EVT] Add support for Row/Col broadcast PtrArray](../sources/prs/cutlass/PR-2033.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Flash MLA support](../sources/prs/cutlass/PR-2130.md), [Flash MLA Support - Step 2](../sources/prs/cutlass/PR-2134.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [Fix sm100 gemm wrong static constexpr that breaks compilation on Windows](../sources/prs/cutlass/PR-2167.md), [Fix SM90 beta=1 hang and stream-K launch errors](../sources/prs/cutlass/PR-2172.md), [Set EpiTile correctly when TileN is not divisible by 32](../sources/prs/cutlass/PR-2220.md), [Use cudaMemcpyAsync in gemm grouped with kRequiresPrecomputation sche…](../sources/prs/cutlass/PR-2256.md), [war to fix blackwell grouped groupwise hang](../sources/prs/cutlass/PR-2267.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [Correct divmod order in example 77 (blackwell fmha)](../sources/prs/cutlass/PR-2291.md), [Handle get_masked_trip_count for small length in fmha example](../sources/prs/cutlass/PR-2292.md), [Fix epilogue::thread::Convert cannot be used with DefaultEpilogue](../sources/prs/cutlass/PR-2333.md), [[ex77] fix mla split; add fwd lse; add bwd varlen](../sources/prs/cutlass/PR-2366.md), [support fp16 accmulator for sm89 fp8 mma](../sources/prs/cutlass/PR-2378.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [fix: examples/cute/tutorial/blackwell/04_mma_tma_2sm_sm100.cu GridDim miscalculated](../sources/prs/cutlass/PR-2492.md), [DistGEMM bug fixes](../sources/prs/cutlass/PR-2713.md), [Support PDL for SM90 Array TMA GEMM](../sources/prs/cutlass/PR-2719.md), [Support for GEMM-K=0 for Blackwell Grouped GEMMs](../sources/prs/cutlass/PR-2746.md), [Blockscaled Ragged Contiguous Grouped Gemm for MoEs](../sources/prs/cutlass/PR-2790.md), [[Bug Fix]Bypass launch grids for SM120 Kernel with SM90 Mainloop & SM100 TileScheduler](../sources/prs/cutlass/PR-2865.md), [[cute] Add constexpr specifier to make_tiled_copy](../sources/prs/cutlass/PR-2875.md), [Fix incorrect tensor layout strides in Blackwell MMA tutorial comments](../sources/prs/cutlass/PR-2921.md), [[Cutlass gemm] Fix SM100 FP8 nosmem epilogue-fusion shape_div 'Divisibility Condition' for non-multiple-of-64 N tiles](../sources/prs/cutlass/PR-2946.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Replace std::min with cute::min in sm120 blockwise scaling device functions](../sources/prs/cutlass/PR-3055.md), [Small Tile N BlockScaled GEMM + Grouped GEMM on SM12x](../sources/prs/cutlass/PR-3176.md), [Add Snake activation functor for EVT](../sources/prs/cutlass/PR-3184.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [Fix FA3 Varlen Performance regression](../sources/prs/flash-attention/PR-1361.md), [Add sorting and head swizzle to varlen scheduler](../sources/prs/flash-attention/PR-1823.md), [feat: update decode attention APIs](../sources/prs/flashinfer/PR-1007.md), [misc: fix instrument code for mla profiler](../sources/prs/flashinfer/PR-1014.md), [add multi-item scoring](../sources/prs/flashinfer/PR-1015.md), [fix: add zero init for KV tiled copy](../sources/prs/flashinfer/PR-1029.md), [feat: add functional per-head FP8 quantization for FA3](../sources/prs/flashinfer/PR-1033.md), [feat: Softmax free sampling](../sources/prs/flashinfer/PR-1035.md), [[nvidia] initial support for blackwell kernels](../sources/prs/flashinfer/PR-1039.md), [fix: top_k_mask_logits hangs on -inf inputs](../sources/prs/flashinfer/PR-1050.md), [[nvidia] Add Blackwell FMHA decode kernel from TRT-LLM](../sources/prs/flashinfer/PR-1051.md), [Fix KV chunking for POD. ](../sources/prs/flashinfer/PR-1054.md), [bugfix: temporally disable split-kv in blackwell mla](../sources/prs/flashinfer/PR-1055.md), [Parameterize prefix mask call (needed by POD-Attention)](../sources/prs/flashinfer/PR-1059.md), [bugfix: adding lse output to blackwell fmha kernels](../sources/prs/flashinfer/PR-1071.md), [bugfix: follow user-specified sm_scale for blackwell cutlass fmha](../sources/prs/flashinfer/PR-1072.md), [perf: accelerate blackwell grouped gemm](../sources/prs/flashinfer/PR-1086.md), [bugfix: fix fp8 attention kernels aot compilation issue](../sources/prs/flashinfer/PR-1087.md), [comm: refactor and initialize `flashinfer.comm` module](../sources/prs/flashinfer/PR-1089.md), [feat: add trtllm all-reduce (non-MoE)](../sources/prs/flashinfer/PR-1096.md), [bugfix: host-precomuted plan function for blackwell fmha](../sources/prs/flashinfer/PR-1106.md), [feat: add trtllm moe_allreduce_fusion](../sources/prs/flashinfer/PR-1108.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [bugfix: Fix test and output shape of fp4 quantize](../sources/prs/flashinfer/PR-1114.md), [hotfix: fix the blackwell fmha stream](../sources/prs/flashinfer/PR-1116.md), [[Feature] Support PDL for batch Prefill and Decode](../sources/prs/flashinfer/PR-1117.md), [Fix pointer dtype bug in rope](../sources/prs/flashinfer/PR-1129.md), [feat: add trtllm all-reduce fusion](../sources/prs/flashinfer/PR-1131.md), [MNNVL MoE All-to-All Support](../sources/prs/flashinfer/PR-1134.md), [fix: negative zero by type trait --> binary value](../sources/prs/flashinfer/PR-1136.md), [[feat] add unified batch attention w/ correctness tests.](../sources/prs/flashinfer/PR-1137.md), [Fix FA2 and FA3 multi-item scoring and cuda illegal memory access error](../sources/prs/flashinfer/PR-1140.md), [feat: Fused temperature online softmax kernel](../sources/prs/flashinfer/PR-1153.md), [Add more logging to TRTLLM-GEN debug trace (NFC)](../sources/prs/flashinfer/PR-1158.md), [feat: add finalize_moe_allreduce from trtllm](../sources/prs/flashinfer/PR-1159.md), [feat: nvshmem python bindings](../sources/prs/flashinfer/PR-1160.md), [feat: update non-fused moe](../sources/prs/flashinfer/PR-1161.md), [feat: enable and update all-reduce fused quantization](../sources/prs/flashinfer/PR-1164.md), [bugfix: softmax NaN results caused by large -inf masks](../sources/prs/flashinfer/PR-1178.md), [update trtllm-gen decode attention kernel launcher](../sources/prs/flashinfer/PR-1189.md), [[feat] optimize persistent batch attention perf.](../sources/prs/flashinfer/PR-1200.md), [[fix] fix BatchAttention CTA_TILE_KV mask issue](../sources/prs/flashinfer/PR-1206.md), [Fix the issue with auxillary kernel launch and grid dim calculation](../sources/prs/flashinfer/PR-1208.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [[comm] TRT-LLM's Multi-Node NVLink All-Reduce Kernel](../sources/prs/flashinfer/PR-1213.md), [Feature/sm100 low latency nvfp4 kernels](../sources/prs/flashinfer/PR-1214.md), [Enable cudnn decode and add tests for the cudnn decode kernel](../sources/prs/flashinfer/PR-1221.md), [feat: add trtllm-gen mla cubin](../sources/prs/flashinfer/PR-1222.md), [Fix missing hash in the cudnn cubin path](../sources/prs/flashinfer/PR-1227.md), [feat: Add non-causal cudnn prefill kernels](../sources/prs/flashinfer/PR-1230.md), [bugfix: support uint8_t for vec_t class template](../sources/prs/flashinfer/PR-1234.md), [add trtllm-gen context attention](../sources/prs/flashinfer/PR-1239.md), [Patch fp8 cubin availability](../sources/prs/flashinfer/PR-1240.md), [feat: Support MXFP8 x MXFP4 CUTLASS grouped GEMM](../sources/prs/flashinfer/PR-1241.md), [Add trtllm-gen attention mha kernel with FP8 Q/K/V and FP8 output](../sources/prs/flashinfer/PR-1242.md), [Remove sm100+ requirment for trtllm allreduce kernels](../sources/prs/flashinfer/PR-1249.md), [Reduce the JIT compilation time of gen_gemm_sm100_module](../sources/prs/flashinfer/PR-1251.md), [TRT-LLM's Multi-Node NVLink AR + fused RMSNorm kernel](../sources/prs/flashinfer/PR-1255.md), [feat: enable trtllm-gen mla MTP](../sources/prs/flashinfer/PR-1258.md), [Made AR output optional + esthetic changes](../sources/prs/flashinfer/PR-1265.md), [Bug fix: fix duplicate launch in POD](../sources/prs/flashinfer/PR-1267.md), [Add shuffle matrix flag](../sources/prs/flashinfer/PR-1272.md), [Convert scale_factor from scalar to Tensor in trt_allreduce_fusion](../sources/prs/flashinfer/PR-1284.md), [fix multiCtasKvScratchPtr misalignment issue (new one)](../sources/prs/flashinfer/PR-1286.md), [Bug fix: guard fp8 e8m0 and e2m1 compile ](../sources/prs/flashinfer/PR-1287.md), [refactor: refactor trtllm-gen attention kernel integration code](../sources/prs/flashinfer/PR-1289.md), [[fix] fix integer overflow in FA2 customized_mask & add buffer overflow warning.](../sources/prs/flashinfer/PR-1290.md), [refactor: Improved metainfo for trtllm-gen fmha](../sources/prs/flashinfer/PR-1292.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [add cutlass backend for mm_fp4](../sources/prs/flashinfer/PR-1296.md), [feat: Add weight layout option for trtllm-gen fused moe](../sources/prs/flashinfer/PR-1297.md), [perfix: use lightweight API to query device property](../sources/prs/flashinfer/PR-1298.md), [[Feature] SM level profiler ](../sources/prs/flashinfer/PR-1305.md), [Fix the bug of the kernel-selection heuristic in trtllm-gen](../sources/prs/flashinfer/PR-1307.md), [Refactor Fused Moe Module](../sources/prs/flashinfer/PR-1309.md), [feat: support output nvfp4 in trtllm-gen function call.](../sources/prs/flashinfer/PR-1318.md), [Make Fp8 MoE routing_bias optional](../sources/prs/flashinfer/PR-1319.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [Optimizations for TRTLLM MNNVL Allreduce](../sources/prs/flashinfer/PR-1321.md), [feat: Add k_scale and v_scale to persistent attention ](../sources/prs/flashinfer/PR-1322.md), [feat: Support logits_soft_cap for Persistent attn; fix kv split limit](../sources/prs/flashinfer/PR-1324.md), [feat: Fused rope fp8 quantize kernel for MLA](../sources/prs/flashinfer/PR-1339.md), [fix: fix trtllm-gen mla error on new interface](../sources/prs/flashinfer/PR-1348.md), [feature: add fp4 mm using trtllm backend](../sources/prs/flashinfer/PR-1355.md), [support trtllm-gen prefill fp4 output](../sources/prs/flashinfer/PR-1360.md), [Support scale factor start index for fp4 mha prefill/decode](../sources/prs/flashinfer/PR-1363.md), [bugfix: fixed cutlass fused moe usage of FP4QuantizationSFLayout::SWIZZLED](../sources/prs/flashinfer/PR-1371.md), [bugfix: Add guard for fp4/fp8 related include headers](../sources/prs/flashinfer/PR-1376.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [feature: add cutlass as bmm_fp8 backend.](../sources/prs/flashinfer/PR-1397.md), [Fix trtllm moe launcher local_num_experts](../sources/prs/flashinfer/PR-1398.md), [fix shared memory alignment conflict in sampling.cuh](../sources/prs/flashinfer/PR-1402.md), [[bugfix] Fix compilation failure when compiling csrc/trtllm_moe_allreduce_fusion.cu](../sources/prs/flashinfer/PR-1410.md), [Fixes for Blackwell Tests](../sources/prs/flashinfer/PR-1434.md), [fix: remote redundant zero_init from trtllm-gen attn](../sources/prs/flashinfer/PR-1444.md), [Add alignment in MxFP8Quantization](../sources/prs/flashinfer/PR-1445.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [bugfix: Verify num_experts greater or equal to local_experts + offset](../sources/prs/flashinfer/PR-1469.md), [perf: add 1x4x1 cluster shape for fp8 bmm M<16 cases](../sources/prs/flashinfer/PR-1473.md), [tuner: Trtllm-gen Fp4 MoE Autotunner](../sources/prs/flashinfer/PR-1475.md), [perf: add fast path to TopPRenormProbKernel for top_p >= 1.0, significantly boosting SGLang workloads](../sources/prs/flashinfer/PR-1483.md), [feat: add pdl for trtllm-gen attn](../sources/prs/flashinfer/PR-1484.md), [feat: Support fp8 qkv, fp16/bf16 out MHA for trtllm-gen.](../sources/prs/flashinfer/PR-1490.md), [Perf: support scale_a/scale_b instead of combined scale in cutlass bmm_fp8](../sources/prs/flashinfer/PR-1491.md), [fix: Replace cub Max/Min with cuda::maximum/minimum for cuda 13 compatibility](../sources/prs/flashinfer/PR-1500.md), [feat: integrate xqa attention backend](../sources/prs/flashinfer/PR-1503.md), [update allreduce to match trtllm](../sources/prs/flashinfer/PR-1507.md), [Support cuda<12.8 built for trtllm_allreduce_fusion.](../sources/prs/flashinfer/PR-1508.md), [backend: Refactor trtllm-gen fmha metainfo loading](../sources/prs/flashinfer/PR-1518.md), [Fix linking errors with CUDA 13](../sources/prs/flashinfer/PR-1523.md), [Add GeGLU support to trtllm-gen NVFP4 Fused MoE Kernel](../sources/prs/flashinfer/PR-1525.md), [bugfix: Fix compile error for undefined swizzle enum.](../sources/prs/flashinfer/PR-1530.md), [bugfix: Fix Persistent kernel precision for masked output ](../sources/prs/flashinfer/PR-1533.md), [Remove cuda-python from dependency and check at runtime](../sources/prs/flashinfer/PR-1534.md), [feat: Integrate TRTLLM varlen kernel for deepseek R1 prefill ](../sources/prs/flashinfer/PR-1537.md), [fix trtllm_allreduce_fusion twoshot register problem.](../sources/prs/flashinfer/PR-1545.md), [perf: replace cudaGetDeviceProperties with cudaDeviceGetAttribute](../sources/prs/flashinfer/PR-1547.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [Add mnnvl_moe_alltoallv_prepare_without_allgather](../sources/prs/flashinfer/PR-1550.md), [bugfix: fix persistent attention kernel correctness on blackwell](../sources/prs/flashinfer/PR-1559.md), [fix: separate out fp4 lib into sm90 and sm100 versions, add oob checking in fused moe](../sources/prs/flashinfer/PR-1565.md), [Backend: downgrade trtllm-gen kernel to cuda-12](../sources/prs/flashinfer/PR-1567.md), [bugfix: fix cuda version guard macros](../sources/prs/flashinfer/PR-1571.md), [update trtllm-gen fp4 autotuner and routing](../sources/prs/flashinfer/PR-1573.md), [bugfix: update trtllm-gen gemm kernel names](../sources/prs/flashinfer/PR-1577.md), [bugfix: Fix arg passing to TORCH_CHECK and TORCH_WARN macros](../sources/prs/flashinfer/PR-1582.md), [fix: semaphoress must be at the fixed range in workspace buffer on trtllm_gen attention](../sources/prs/flashinfer/PR-1584.md), [bugfix: fix fused-temperature softmax IMA issue](../sources/prs/flashinfer/PR-1596.md), [bugfix: fix the register overflow issue for topk renorm kernels on blackwell](../sources/prs/flashinfer/PR-1597.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [feat: cutlass fp4 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1609.md), [feat: cutlass fp8 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1610.md), [bugfix: fix fp4 quantization with 8x4 scale factor layout](../sources/prs/flashinfer/PR-1611.md), [bugfix: fix merge_attention_state in BatchAttention w/ gqa-group-size in Qwen family](../sources/prs/flashinfer/PR-1614.md), [perf: Fix the tactic sorting in TrtllmGenBatchedGemmRunner::getValidConfigIndices](../sources/prs/flashinfer/PR-1615.md), [bugfix: collect all modules to aot](../sources/prs/flashinfer/PR-1622.md), [bugfix: trtllm-gen fmha sm101 and sm100 compatibility](../sources/prs/flashinfer/PR-1631.md), [perf&bugfix: skip kv-tile computation out of sliding window in FA2; fix __syncthreads in mergestate](../sources/prs/flashinfer/PR-1661.md), [Refactor Blackwell unit test scripts](../sources/prs/flashinfer/PR-1667.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [feat: Add `variant.OutputTransform()` to decode kernels](../sources/prs/flashinfer/PR-1670.md), [feat: Batch-size invariant FA2 Prefill & Decode](../sources/prs/flashinfer/PR-1675.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [Update TGV GEMM default kernel and TGV code cleanup.](../sources/prs/flashinfer/PR-1682.md), [perf: Port the separate reduce kernel mode from trtllm.](../sources/prs/flashinfer/PR-1685.md), [Support Kimi-K2 for TRT: templatize number of experts](../sources/prs/flashinfer/PR-1696.md), [Fix DeepSeek quality for TRTLLM fused MoE routing](../sources/prs/flashinfer/PR-1723.md), [bugfix: partially fix tests/test_trtllm_gen_fused_moe.py unit test failure](../sources/prs/flashinfer/PR-1724.md), [TVM: support TVM binding for GroupedGemm](../sources/prs/flashinfer/PR-1725.md), [fix: put sampling kernel launch into macro](../sources/prs/flashinfer/PR-1727.md), [fix: fix cannot import name 'cuda' from 'cuda' in CUDA13](../sources/prs/flashinfer/PR-1764.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [Masked batch nvfp4 quantization](../sources/prs/flashinfer/PR-1774.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [Bugfix: Fix data hazard in persistent reduce](../sources/prs/flashinfer/PR-1826.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [Update the routing for TRTLLMGEN to support kimi k2 and qwen](../sources/prs/flashinfer/PR-1831.md), [[Quantization] Add per-expert global scaling factor for fp4 batched quantize](../sources/prs/flashinfer/PR-1835.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [Bugfix: fix o_strides in persistent kernel ](../sources/prs/flashinfer/PR-1865.md), [Tune kernel compilation parameters for https://github.com/flashinfer-ai/flashinfer/pull/1850 ](../sources/prs/flashinfer/PR-1878.md), [feat: Add FP4 TRTLLM-Gen throughput MOE batched gemms](../sources/prs/flashinfer/PR-1882.md), [MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA](../sources/prs/flashinfer/PR-1924.md), [Add layernorm op for inputs of mixed dtype](../sources/prs/flashinfer/PR-1926.md), [silu_and_mul nvfp4 quanization fusion rework](../sources/prs/flashinfer/PR-1927.md), [Feature: Support Relu2 activation in fused MoE](../sources/prs/flashinfer/PR-1954.md), [Update trtllm-gen fused moe routing kernel and add more kernels](../sources/prs/flashinfer/PR-1955.md), [Fix: Verify scales are not None for Cutlass FP8 FusedMoE](../sources/prs/flashinfer/PR-1961.md), [feat: enable deepgemm jit for fp8 block-scale on SM90](../sources/prs/flashinfer/PR-1969.md), [feat: autotune tile_tokens_dim in trtllm-gen MOE](../sources/prs/flashinfer/PR-1980.md), [fix: correct PDL parameter handling in RopeQuantize kernel](../sources/prs/flashinfer/PR-1982.md), [minor fix for xqa](../sources/prs/flashinfer/PR-1994.md), [Bugfix: Change get() -> GetDLTensorPtr() in cutlass FusedMoE validations](../sources/prs/flashinfer/PR-1995.md), [feat: add xqa backend and completes NHD/HND coverage for trtllm-gen/xqa backend](../sources/prs/flashinfer/PR-2001.md), [Feature: Support non-gated activation in cutlass fused MoE nvfp4](../sources/prs/flashinfer/PR-2011.md), [[feat] Refactor trtllmgen MOE and add Bf16 trtllmgen moe](../sources/prs/flashinfer/PR-2014.md), [[DSV3] Optimized Router Gemm](../sources/prs/flashinfer/PR-2019.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [perf: Speed up fp4 quantization for small batch with swizzling for cutlass MoE](../sources/prs/flashinfer/PR-2025.md), [[NVIDIA] Thor & Spark Support](../sources/prs/flashinfer/PR-2028.md), [Enable renormalize(naive) routing for fp8 per-tensor](../sources/prs/flashinfer/PR-2030.md), [use scalar for kv_scale in xqa](../sources/prs/flashinfer/PR-2033.md), [feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA) ](../sources/prs/flashinfer/PR-2037.md), [perf: improve sampling/mask/softmax performance (part 1/2)](../sources/prs/flashinfer/PR-2044.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [Fix dtype of output scales from mnnvl_moe_alltoallv_prepare_without_allgather](../sources/prs/flashinfer/PR-2048.md), [[BUG] Fix trtllm-gen fp4 moe renormalize routing](../sources/prs/flashinfer/PR-2049.md), [Add support for topkPacked input in block-level renormalize](../sources/prs/flashinfer/PR-2051.md), [feat: add xqa mla backend](../sources/prs/flashinfer/PR-2053.md), [perf: Optimize helper max/minmax function in sampling.cuh](../sources/prs/flashinfer/PR-2058.md), [Fix moe fp8 failure for sm121](../sources/prs/flashinfer/PR-2061.md), [Fix: several bugs/issues with trtllm-gen attention kernels. ](../sources/prs/flashinfer/PR-2062.md), [perf: TRT-LLM MoE Block-FP8 activation optimization](../sources/prs/flashinfer/PR-2063.md), [feat: BF16 GEMM using CUTLASS backend for SM100](../sources/prs/flashinfer/PR-2070.md), [[Feature] Support batch prefill for POD Attention](../sources/prs/flashinfer/PR-2079.md), [enable xqa fp8 output](../sources/prs/flashinfer/PR-2081.md), [[API change] Allow using torch.Tensor for scales for trtllm-gen attention](../sources/prs/flashinfer/PR-2084.md), [refactor: pass hopper deepgemm include directory through python](../sources/prs/flashinfer/PR-2090.md), [perf: TRT-LLM Gen finalize kernel optimization](../sources/prs/flashinfer/PR-2092.md), [perf: enable pdl for cutlass fp4 gemm](../sources/prs/flashinfer/PR-2095.md), [Port TRT-LLM communication kernels to flashinfer](../sources/prs/flashinfer/PR-2102.md), [enable xqa speculative decoding](../sources/prs/flashinfer/PR-2105.md), [feat: support more head dim in RoPE kernel](../sources/prs/flashinfer/PR-2109.md), [add tensor scale input for xqa](../sources/prs/flashinfer/PR-2110.md), [refactor: update fa3 codebase and fix hopper unittest [part 1]](../sources/prs/flashinfer/PR-2111.md), [feature: make the LSE returned by MLA support base 2 or e #2113](../sources/prs/flashinfer/PR-2114.md), [update xqa license](../sources/prs/flashinfer/PR-2117.md), [Refactor trtllm_mnnvl_allreduce](../sources/prs/flashinfer/PR-2118.md), [perf: bunch of features and optimizations for top-k (sampling + sparse attention)](../sources/prs/flashinfer/PR-2119.md), [feat: support variable sequence length in decode kernel of trtllm-gen attention](../sources/prs/flashinfer/PR-2125.md), [fix flaky xqa test](../sources/prs/flashinfer/PR-2126.md), [make DeepGEMM swapAB available for linear gemm SM90](../sources/prs/flashinfer/PR-2131.md), [feat: add trtllm-gen per-tensor sparseMla kernels.](../sources/prs/flashinfer/PR-2138.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [fix xqa mha_sm90.cu](../sources/prs/flashinfer/PR-2157.md), [feat: MxInt4 x Bf16 TRT-LLM Gen MoE support](../sources/prs/flashinfer/PR-2159.md), [Add data type check for deepseek fp4 moe](../sources/prs/flashinfer/PR-2165.md), [Fix for moe on sm110](../sources/prs/flashinfer/PR-2190.md), [feat: unit-test and api change, w4a8 grouped-gemm fused MoE for SM90](../sources/prs/flashinfer/PR-2193.md), [Move the run function definition out of BatchedGemmInterface](../sources/prs/flashinfer/PR-2211.md), [feat: further optimize top-k and add fused top-k page construction kernels for DSA](../sources/prs/flashinfer/PR-2215.md), [feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe](../sources/prs/flashinfer/PR-2217.md), [fix: add DeepSeek routing for Bf16xBf16 and MxIntxBf16 TRT-LLM Gen MoE](../sources/prs/flashinfer/PR-2234.md), [refactor: pull trtllm-gen batch-gemm/gemm headers from artifactory; update tma descriptor shape init](../sources/prs/flashinfer/PR-2235.md), [[feat] Integrate SGLang concat_mla_k kernel into flashinfer](../sources/prs/flashinfer/PR-2237.md), [feat: RMSNorm/Fused RMSNorm + FP8 Quantization kernels](../sources/prs/flashinfer/PR-2243.md), [Remove cudaStreamSynchronize from gemm_groupwise_sm120.cuh for CUDA graph compatibility](../sources/prs/flashinfer/PR-2244.md), [feat: Support numLocalTokens=0 for moe All-to-all](../sources/prs/flashinfer/PR-2247.md), [feat: support non-contiguous query for trtllm-gen attention backend](../sources/prs/flashinfer/PR-2254.md), [fix: support int64 IdType for RoPE part argument in `rope_quantize_fp8_append_paged_kv_cache`](../sources/prs/flashinfer/PR-2255.md), [[TRTLLM-Gen Fmha] add optimized trtllm-gen decode kernels for high throughput + speculative decoding](../sources/prs/flashinfer/PR-2265.md), [[performance]optimize for nvfp4](../sources/prs/flashinfer/PR-2268.md), [feat: add GDN Attention](../sources/prs/flashinfer/PR-2276.md), [feat: IdType indices in sampling kernels](../sources/prs/flashinfer/PR-2281.md), [Selective State Update kernel (mamba)](../sources/prs/flashinfer/PR-2301.md), [[Perf][Feature] Add SM103-specific schedulers for NVFP4 CUTLASS kernels](../sources/prs/flashinfer/PR-2303.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [Fix: FilteredTopKUnifiedKernel read value out of length](../sources/prs/flashinfer/PR-2308.md), [[ML3] Optimized Router Gemm](../sources/prs/flashinfer/PR-2323.md), [bugfix: fix multi-cta top-k implementation when k value is different for different row](../sources/prs/flashinfer/PR-2325.md), [[perf] Improve gemm_fp8_nt_groupwise (cutlass backend) by 10-40% for batch sizes <= 32](../sources/prs/flashinfer/PR-2327.md), [fix: guard batchWarpReduceSum with ENABLE_FP8 to fix compilation without FP8](../sources/prs/flashinfer/PR-2328.md), [feat: expose swizzled_input_sf parameter for CUTLASS fused MOE](../sources/prs/flashinfer/PR-2330.md), [Optimize quantization function in large problem size](../sources/prs/flashinfer/PR-2343.md), [Enable fp16/bf16/f32 support for selective_state_update (mamba)](../sources/prs/flashinfer/PR-2366.md), [bugfix: hotfix of PR 2366 (mamba kernel)](../sources/prs/flashinfer/PR-2378.md), [fix: ensure each CTA processes full numHeadsQPerKv for trtllm decode kernel](../sources/prs/flashinfer/PR-2380.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [feat: cuteDSL fp4 moe for better DSR1 performance.](../sources/prs/flashinfer/PR-2398.md), [Remove cudaMalloc/Free in GDN prefill kernel](../sources/prs/flashinfer/PR-2415.md), [feat: update trtllm-gen MoE cubins](../sources/prs/flashinfer/PR-2416.md), [refactor: reduce hopper's gdn prefill compilation time and fix docstring.](../sources/prs/flashinfer/PR-2422.md), [refactor: refactoring cuda code to cute-dsl (part 1)](../sources/prs/flashinfer/PR-2428.md), [fix: Sampling: CUDA Graph fix](../sources/prs/flashinfer/PR-2432.md), [fix: Fix NaN output in mxfp8_quantize for very small input values](../sources/prs/flashinfer/PR-2441.md), [MTP for mamba ](../sources/prs/flashinfer/PR-2444.md), [bugfix: fix stub generation directory in fused_moe module](../sources/prs/flashinfer/PR-2445.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [fix: fix illegal memory access for NaN input in sampling kernels](../sources/prs/flashinfer/PR-2456.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [feat: Add MXFP8 GEMM mm_mxfp8 (cutlass)](../sources/prs/flashinfer/PR-2464.md), [feat: Add TRTLLM-Gen Skip-Softmax kernels for prefill and decode](../sources/prs/flashinfer/PR-2477.md), [fix: add support check for gemm config for cutlass moe](../sources/prs/flashinfer/PR-2495.md), [Feat: Trtllm-gen MxFP8 MoE integration](../sources/prs/flashinfer/PR-2505.md), [perf: cache cudaGetDeviceProperties in gdn_prefill to avoid per-call overhead](../sources/prs/flashinfer/PR-2509.md), [Support NVFP4 KV cache decode on SM120](../sources/prs/flashinfer/PR-2520.md), [misc: point triton blackwell-ptxas to local cuda ptxas](../sources/prs/flashinfer/PR-2543.md), [feat: Enable TRTLLM-Gen Skip-Softmax attention for MLA](../sources/prs/flashinfer/PR-2547.md), [[Bugfix][comm] Fix FP4 one-shot launch config instability in trtllm_allreduce_fusion](../sources/prs/flashinfer/PR-2557.md), [Add support for the combinations of allreduce, allgather, and reducescatter](../sources/prs/flashinfer/PR-2563.md), [fix: W4A8 autotune crash in cutlass_fused_moe profiler workspace](../sources/prs/flashinfer/PR-2564.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [feat: trtllm tinygemm2 in flashinfer as bf16 routergemm](../sources/prs/flashinfer/PR-2587.md), [Mamba SSU: better automatic kernel selection + algorithm selection optionally exposed to the user.](../sources/prs/flashinfer/PR-2591.md), [[bugfix] Fix FilteredTopK overflow correctness](../sources/prs/flashinfer/PR-2605.md), [fix: cute dsl nvfp4 moe routing index error](../sources/prs/flashinfer/PR-2629.md), [fix: add SM121 support to SM120 version guards](../sources/prs/flashinfer/PR-2631.md), [[fp8_blockwise]Fix int32 overflow in TRTLLM fused MoE activation kernel](../sources/prs/flashinfer/PR-2642.md), [feat: FP32 dtype output for BF16 matmuls (CUTLASS & cuDNN)](../sources/prs/flashinfer/PR-2644.md), [int16 Block-Scaled State and Stochastic Rounding for SSU (mamba)](../sources/prs/flashinfer/PR-2645.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [feat: implement deterministic topk](../sources/prs/flashinfer/PR-2661.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [fix: reduce smem allocation for tinygemm2 kernel in SM120](../sources/prs/flashinfer/PR-2670.md), [feat: add support for more MLA head dimensions](../sources/prs/flashinfer/PR-2677.md), [Add varlen and speculative decoding support to selective state update](../sources/prs/flashinfer/PR-2700.md), [Add NVFP4 KV cache quantization support for SM100](../sources/prs/flashinfer/PR-2702.md), [feat: Add support for TRTLLM MXFP8 non-gated MoE with ReLU2](../sources/prs/flashinfer/PR-2707.md), [Mamba2 SSD Combined Forward Pass (Blackwell CuTe DSL Kernel)](../sources/prs/flashinfer/PR-2709.md), [feat: Add DiT-oriented kernels where Qk (Bmm1) type can be reinterpreted into Int8 or BFloat16](../sources/prs/flashinfer/PR-2711.md), [fix: Add SM120 (RTX Blackwell desktop) support for NVFP4 MoE kernels](../sources/prs/flashinfer/PR-2725.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor](../sources/prs/flashinfer/PR-2740.md), [[feat] Add 2048 experts and 32 Top K ](../sources/prs/flashinfer/PR-2744.md), [[feat] Add air top-p algorithm](../sources/prs/flashinfer/PR-2752.md), [feat: Add FP4 KV cache quant/dequant kernels ](../sources/prs/flashinfer/PR-2757.md), [feat: Expose TRT-LLM FMHA style paged KV Cache and page table layout](../sources/prs/flashinfer/PR-2770.md), [feat: FP8 output support for CUTLASS MLA paged attention](../sources/prs/flashinfer/PR-2779.md), [feat: Support padding tokens with seqlen=0 for rope+quant+kv cache update fusion kernel](../sources/prs/flashinfer/PR-2792.md), [Upgrade cutlass 4.2.1 -> 4.4.2](../sources/prs/flashinfer/PR-2798.md), [[fmha-v2] Support HND and NHD paged KV cache layouts with conditional stride handling](../sources/prs/flashinfer/PR-2799.md), [fix: Autotuner _find_nearest_profile non-power-of-2 num_tokens, create launchers for all supported tileN in trtllm fused MoE](../sources/prs/flashinfer/PR-2821.md), [[Fmha] Sparse MLA decode kernel selection heuristics](../sources/prs/flashinfer/PR-2836.md), [[Perf] Add FMHAv2 to flashinfer_benchmark.py and eliminate unnecessary H2D](../sources/prs/flashinfer/PR-2841.md), [read real strides for kv and block scale](../sources/prs/flashinfer/PR-2844.md), [fix: int32 overflow in `trtllm_fp4_block_scale_moe` causing "Unsupported hidden state scale shape" for EP32+ configs](../sources/prs/flashinfer/PR-2853.md), [Add support for Relu2 in BF16 fused MoE](../sources/prs/flashinfer/PR-2864.md), [Mamba SSU: horizontal MTP kernel (+ DSTATE=96 support)](../sources/prs/flashinfer/PR-2865.md), [Fix silent bug with FP8 per tensor non-gated MoE](../sources/prs/flashinfer/PR-2882.md), [fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120](../sources/prs/flashinfer/PR-2898.md), [feat: add MXFP8 GEMM support for SM120](../sources/prs/flashinfer/PR-2902.md), [feat(gdn): state checkpointing in chunk_gated_delta_rule](../sources/prs/flashinfer/PR-2908.md), [feat: Add cuBLASLt backend for `mm_bf16` and enable multi-tactic autotuning for FP8/MXFP8 runners](../sources/prs/flashinfer/PR-2914.md), [feat: add Relu2 (squared ReLU) activation support in CUTLASS MoE backend](../sources/prs/flashinfer/PR-2926.md), [fix: use float instead of double in sampling binary search to avoid FP64 bottleneck on SM103](../sources/prs/flashinfer/PR-2945.md), [Update NVSHMEM interface to use NVSHMEM4Py instead of custom bindings](../sources/prs/flashinfer/PR-2960.md), [Improved `simple` mamba SSU kernel ](../sources/prs/flashinfer/PR-2962.md), [test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9](../sources/prs/flashinfer/PR-2963.md), [Add flashinfer.fused_rmsnorm_silu() with native kernel backend](../sources/prs/flashinfer/PR-2965.md), [Fused moe all-reduce routed scaling factor + quant support](../sources/prs/flashinfer/PR-2966.md), [fix: restore SM120 CUTLASS MoE tile candidate removed by #2927 (test_trtllm_cutlass_fused_moe.py)](../sources/prs/flashinfer/PR-2984.md), [[Fmha] support nvfp4 output keepsMmaAb generation kernels](../sources/prs/flashinfer/PR-2988.md), [fix: tinygemm2 hang issue due to barrier sync](../sources/prs/flashinfer/PR-2996.md), [perf: Optimize CUTLASS MoE helper kernels for small-batch decode workloads](../sources/prs/flashinfer/PR-3014.md), [fix: extend moe alltoall top-k specializations](../sources/prs/flashinfer/PR-3021.md), [[feat] Add routing_replay_out support to MoE kernels and Python API](../sources/prs/flashinfer/PR-3024.md), [perf: Port TRT-LLM SM120/SM121 FP4 CUTLASS GEMM optimizations. Add PDL](../sources/prs/flashinfer/PR-3026.md), [[feat] Trtllm-gen Per-token Nvfp4 MoE](../sources/prs/flashinfer/PR-3027.md), [fused_moe: pre-filter SM89 tactics with zero occupancy on SM120 Blackwell (fix review feedback on #2764)](../sources/prs/flashinfer/PR-3032.md), [Support lse in trtllm paged attn kernels](../sources/prs/flashinfer/PR-3058.md), [Support Allreduce + Norm + Per-token Group Fp8 Quant Fusion](../sources/prs/flashinfer/PR-3059.md), [Support NVFP4 KV for prefill and batch attention kernels](../sources/prs/flashinfer/PR-3097.md), [feat: Enable FP8 (E4M3/E5M2) in concat_mla_k for optimize long-context prefill performance and refactor type dispatch for BF16/FP16](../sources/prs/flashinfer/PR-3129.md), [perf: Add no-bias path for tinygemm_bf16](../sources/prs/flashinfer/PR-3151.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat: DiT layer norm fusions for WAN: flashinfer.diffusion_ops](../sources/prs/flashinfer/PR-3157.md), [feat: enable glm5 router gemm](../sources/prs/flashinfer/PR-3185.md), [[Bugfix] Fix fused MoE autotuning correctness issues by filtering clusterDimZ](../sources/prs/flashinfer/PR-3227.md), [perf: optimize per-token nvfp4 quantization kernel.](../sources/prs/flashinfer/PR-3237.md), [Update moe gemm](../sources/prs/flashinfer/PR-3239.md), [Add dynamic tokens-per-page TRTLLM-GEN GQA kernels](../sources/prs/flashinfer/PR-3259.md), [Update trtllm FMHA cubins](../sources/prs/flashinfer/PR-3317.md), [[feat] Add gemma RMS AR fusion](../sources/prs/flashinfer/PR-3322.md), [checkpointing_ssu kernel: fused replay + conditional state-write for Mamba2](../sources/prs/flashinfer/PR-3324.md), [feat(cute_dsl/moe): add `moe_output_memset_inplace` dense memset wrapper](../sources/prs/flashinfer/PR-3328.md), [perf: fix the iteration bound of SWA in FA2 prefill template](../sources/prs/flashinfer/PR-714.md), [bugfix: FusedAddRMSNorm kernels might require more than 48KB shared memory when d is large.](../sources/prs/flashinfer/PR-718.md), [Align KV chunk size binary search with actual KV chunk splitting.](../sources/prs/flashinfer/PR-728.md), [Change `apply_rope_with_cos_sin_cache` to accept `cos_sin_cache`](../sources/prs/flashinfer/PR-754.md), [feat: support deepseek prefill attention shape](../sources/prs/flashinfer/PR-765.md), [bugfix: Ensure Loop Termination by Enforcing IEEE-754 Compliance in Sampling Kernels](../sources/prs/flashinfer/PR-774.md), [perf: refactor fa2 prefill template](../sources/prs/flashinfer/PR-776.md), [bugfix: drop CTA_TILE_Q=32](../sources/prs/flashinfer/PR-785.md), [bugfix: MLA decode should multiply sm_scale by math::log2e](../sources/prs/flashinfer/PR-787.md), [fix rope logic in mla decoding](../sources/prs/flashinfer/PR-793.md), [feat: support f32 attention output in FA2 template](../sources/prs/flashinfer/PR-799.md), [feat: apply sm_scale at logits instead of q in FA2 template](../sources/prs/flashinfer/PR-801.md), [perf: memory efficient deepseek mla fused page-attention kernel](../sources/prs/flashinfer/PR-804.md), [bugfix: mla page-attention kernel for different page sizes](../sources/prs/flashinfer/PR-810.md), [feat: unlocking MLA for A100](../sources/prs/flashinfer/PR-812.md), [feat: unlock MLA attention for sm89 (L40/L40s/4090)](../sources/prs/flashinfer/PR-814.md), [bugfix: bugfix on sm89 MLA](../sources/prs/flashinfer/PR-821.md), [bugfix: fix the signature of `CutlassSegmentGEMMSM90`](../sources/prs/flashinfer/PR-827.md), [perf: MLA decode kernel implemented by CuTe targeted to SM80](../sources/prs/flashinfer/PR-844.md), [misc: Remove duplicate param set in MLA kernel](../sources/prs/flashinfer/PR-850.md), [Add POD-Attention to FlashInfer](../sources/prs/flashinfer/PR-858.md), [perf: dynamic split-k for MLA](../sources/prs/flashinfer/PR-863.md), [bugfix: fix the behavior of MLA kernel when kv-length is 0](../sources/prs/flashinfer/PR-868.md), [Naive Support for Hopper FP8 Prefill Kernel with Per-Head Quantization](../sources/prs/flashinfer/PR-869.md), [perf: FlashAttention-3 style MLA PageAttention](../sources/prs/flashinfer/PR-887.md), [feat - support mla kvcache store](../sources/prs/flashinfer/PR-888.md), [perf: fix MLA split-k performance bug](../sources/prs/flashinfer/PR-898.md), [perf: tweak the pipeline design of mla kernel](../sources/prs/flashinfer/PR-901.md), [feat: flashinfer intra-kernel profiler](../sources/prs/flashinfer/PR-913.md), [feat: experimenta support of PDL](../sources/prs/flashinfer/PR-930.md), [bugfix: fix potential issues of FA3 template loading nans for PageAttention](../sources/prs/flashinfer/PR-945.md), [perf: Use 2WG pipeline design for MLA implementation on Hopper](../sources/prs/flashinfer/PR-952.md), [[TVM] Added tvm binding for sampling kernel](../sources/prs/flashinfer/PR-958.md), [perf: Fix python API overhead when CUDAGraph is not enabled](../sources/prs/flashinfer/PR-969.md), [perf: dual pivot top-p/top-k renorm](../sources/prs/flashinfer/PR-974.md), [perf: prefetch page indices for mla kernel](../sources/prs/flashinfer/PR-991.md), [feat: SM-constraint Communication Kernels](../sources/prs/flashinfer/PR-994.md), [3rdparty: upgrade cutlass to 3.9](../sources/prs/flashinfer/PR-997.md), [ROCm SDPA: Ensure attn_mask has the same dtype with q](../sources/prs/pytorch/PR-144398.md), [Add release branch push triggers to inductor-rocm-mi300.yml](../sources/prs/pytorch/PR-149871.md), [[CUDA][avgpool2d] Fix backward launch bounds again for sm100, sm120](../sources/prs/pytorch/PR-150640.md), [[CUDA][avgpool2d] Fix backward launch bounds again for `sm100`, `sm120`](../sources/prs/pytorch/PR-150676.md), [[CUDA] Only use vec128 if CUDA version is newer than 12.8](../sources/prs/pytorch/PR-150705.md), [[ATen][CUDA] Optimize 128 bit vectorization](../sources/prs/pytorch/PR-152967.md), [Fix uint view copy (#151598)](../sources/prs/pytorch/PR-154121.md), [Fix macOS build with `USE_MPS=OFF`](../sources/prs/pytorch/PR-156932.md), [[user triton] AOT inductor support for device-side TMA](../sources/prs/pytorch/PR-157241.md), [[PowerPC] Fixed build issue for vsx vec256 complexfloat and scaled_mm_out_cpu ](../sources/prs/pytorch/PR-157422.md), [[cherry-pick][inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack](../sources/prs/pytorch/PR-158646.md), [CUDA 13.0 Windows Nvidia Driver Update to 580.88](../sources/prs/pytorch/PR-162501.md), [fix cpp extension distributed warning spew](../sources/prs/pytorch/PR-162764.md), [[Graph Partition] improve custom op output alias](../sources/prs/pytorch/PR-163380.md), [[graph partition] Add way to register custom rule (#163310)](../sources/prs/pytorch/PR-163395.md), [CUDA 13.0 Warning update for supported architectures](../sources/prs/pytorch/PR-163633.md), [[Cherry-Pick] [CD] CUDA 13 specific followup changes. Remove sm50-70 From CUDA 12.6 and CUDA 12.8 builds (#162455)](../sources/prs/pytorch/PR-163764.md), [[CD] CUDA 13.0 fix preload logic to include nvidia/cu13/lib/](../sources/prs/pytorch/PR-163766.md), [Move inductor jobs 3.9->3.10](../sources/prs/pytorch/PR-163954.md), [[cuDNN][SDPA] Disable dropout for cuDNN SDPA on 9.11 - 9.13](../sources/prs/pytorch/PR-164026.md), [[AARCH64][CD][CUDA13][Triton][PTXAS] Turn on BUILD_BUNDLE_PTXAS=1 ](../sources/prs/pytorch/PR-164236.md), [CUDA 13.0 builds fix on Amazon Linux 2023](../sources/prs/pytorch/PR-164893.md), [[Graph Partition] move custom rules to inductor config (#166458)](../sources/prs/pytorch/PR-166967.md), [[Graph Partition] fix graph partition input signature for fallback kernels](../sources/prs/pytorch/PR-166985.md), [[cuDNN][SDPA][Convolution] Expose cuDNN runtime version in CUDA hooks](../sources/prs/pytorch/PR-167327.md), [[ROCm] Enable shared memory based pruning for Triton configs](../sources/prs/pytorch/PR-170190.md), [[inductor] Fix cudagraph skip for index_put_ with boolean indices, gr…](../sources/prs/pytorch/PR-170884.md), [[ROCm] Make grouped GEMM CK opt‑in via env and default to fallback path](../sources/prs/pytorch/PR-171140.md), [[cherry-pick][CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds ](../sources/prs/pytorch/PR-171189.md), [[cherry-pick][cuDNN][SDPA] cuDNN SDPA off-by-default for cuDNN versions < 12.9 (#171627)](../sources/prs/pytorch/PR-171895.md), [Skip modded_nanogpt model in TorchInductor benchmark](../sources/prs/pytorch/PR-172141.md), [[Graph Partition] Improve support for mutation ops](../sources/prs/pytorch/PR-172577.md), [Update inductor expected accuracy files](../sources/prs/pytorch/PR-175096.md), [[benchmark] Skip pytorch_CycleGAN_and_pix2pix from inductor benchmarks](../sources/prs/pytorch/PR-175299.md), [[release-only] Remove +ptx from cuda 13.0 builds](../sources/prs/pytorch/PR-175567.md), [[release 2.12] Apply Release only changes to 2.12 branch](../sources/prs/pytorch/PR-180470.md), [Disable kernel cutlass_mla_decode on SM103](../sources/prs/sglang/PR-10058.md), [Optimize nvfp4 block scaled gemm kernel when M is small.](../sources/prs/sglang/PR-10101.md), [fix: resolve gb200 image link](../sources/prs/sglang/PR-10343.md), [Fix correction bias undefined behavior for nvfp4 models](../sources/prs/sglang/PR-10426.md), [Update CUTLASS. Refine KernelSchedule for fp8 (grouped) gemm.](../sources/prs/sglang/PR-10491.md), [[sgl-kernel] Optimize concat_mla_k kernel](../sources/prs/sglang/PR-10543.md), [Optimize cutlass int8 gemm kernel for large M on SM89 Ada GPU](../sources/prs/sglang/PR-10714.md), [disable sm100 for FlashMLA and fast-hadamard-transform in cuda12.6.1](../sources/prs/sglang/PR-11274.md), [[NVIDIA] Add new SMs support for Spark & Thor](../sources/prs/sglang/PR-11287.md), [[sgl-kernel][1/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-11432.md), [Improve Kernel Build Time](../sources/prs/sglang/PR-11508.md), [support cutlass fp4 kernel in sm120](../sources/prs/sglang/PR-11737.md), [Fixed aarch64 flash-mla](../sources/prs/sglang/PR-12009.md), [(1/n)support context parallel with deepseekv3.2-DSA](../sources/prs/sglang/PR-12065.md), [[sgl-kernel][4/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-12080.md), [[sgl-kernel] clean up fa fetch in CMakeLists.txt](../sources/prs/sglang/PR-12392.md), [[Fix] `concat_mla_absorb_q_kernel` fails for long inputs](../sources/prs/sglang/PR-12453.md), [[NVIDIA] Fix CUDA arch requirement in nvfp4 cast](../sources/prs/sglang/PR-12581.md), [Support moe topk sigmoid kernel](../sources/prs/sglang/PR-13049.md), [diffusion: enable fa4 for blackwell](../sources/prs/sglang/PR-13263.md), [[sgl-kernel][Feat][B200][1/N]Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-13731.md), [[kernel][moe] add moe topk fast](../sources/prs/sglang/PR-13969.md), [Add new moe wna16 marlin gemm](../sources/prs/sglang/PR-14122.md), [Opt moe align block size kernel](../sources/prs/sglang/PR-14133.md), [sync attention, deepseek doc](../sources/prs/sglang/PR-14335.md), [[CPU] Implement MXFP4 Gemm kernels for intel AMX to support GPT OSS series.](../sources/prs/sglang/PR-14385.md), [Add CUDA kernel size analysis tool for sgl-kernel optimization](../sources/prs/sglang/PR-14544.md), [[sgl-kernel][Feat][B200][2/N] Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-14640.md), [[NPU][eagle3] support qwen eagle3 on NPU](../sources/prs/sglang/PR-14820.md), [[sgl-kernel][1/2] Fused qk_norm_rope for GLM4.6](../sources/prs/sglang/PR-15141.md), [Add cache for flashinfer installation](../sources/prs/sglang/PR-15153.md), [[sgl-kernel] Update flashmla to include fp8 sparse_mla optimizations](../sources/prs/sglang/PR-15242.md), [Fix warp illegal instruction in kimi k2 thinking PCG](../sources/prs/sglang/PR-15306.md), [[diffusion] Add Sage Attention 3 Support for sm 120 (RTX5090)](../sources/prs/sglang/PR-15382.md), [[sgl-kernel][6/7]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-15471.md), [[Feature] JIT Fused QK norm + qk norm clean up](../sources/prs/sglang/PR-15835.md), [[JIT kernel] Apply jit per_tensor_quant_fp8 kernel](../sources/prs/sglang/PR-15836.md), [[Feature] add aligned_vector type for JIT kernel](../sources/prs/sglang/PR-16162.md), [Fix sgl-kernel jobs to skip when target_stage is specified](../sources/prs/sglang/PR-16308.md), [[diffusion] Fix RuntimeError in SageAttention3 on Nvidia Blackwell with Qwen-Image](../sources/prs/sglang/PR-16335.md), [[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell](../sources/prs/sglang/PR-16502.md), [[DeepSeek v3.2] Opt MTP decode cuda batch sizes and nsa implementation](../sources/prs/sglang/PR-16961.md), [Kernel: optimize decoding metadata in NSA multi-spec backend with fused kernels](../sources/prs/sglang/PR-17554.md), [Make flashMLA work on: Cu13, B300](../sources/prs/sglang/PR-17600.md), [Feature/support longcat flash lite](../sources/prs/sglang/PR-17838.md), [[Move sgl-kernel Kernel to JIT] Add JIT concat MLA kernels](../sources/prs/sglang/PR-17889.md), [[Diffsuion & JIT_kernel] QKNorm cross heads kernel](../sources/prs/sglang/PR-18073.md), [[Hicache & JIT_kernel] Support page first layout & mla jit kernel](../sources/prs/sglang/PR-18311.md), [docs: expand and update modelopt documentation](../sources/prs/sglang/PR-18479.md), [[FIX] Correct JIT kernel compilation on newer GPUs with outdated driver metadata.](../sources/prs/sglang/PR-18496.md), [Add claude skills for sgl-kernel and jit-kernel](../sources/prs/sglang/PR-18855.md), [Use single mma warp group for short q_len in FA to optimize decoding performance](../sources/prs/sglang/PR-18985.md), [[jit_kernel] Add fused_qknorm_rope JIT kernel](../sources/prs/sglang/PR-19059.md), [[DeepSeek-V3.2][JIT-kernel] Support nsa fuse store indexer k cache](../sources/prs/sglang/PR-19148.md), [[Kernel Slimming] Migrate NVFP4 kernels to JIT](../sources/prs/sglang/PR-19437.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [Add compile-time 256-bit vector guard for pre-Blackwell](../sources/prs/sglang/PR-19794.md), [[JIT Kernel][Feature] Support JIT custom all reduce (rewrite as v2)](../sources/prs/sglang/PR-19880.md), [[JIT Kernel] Reland NVFP4 kernels to JIT](../sources/prs/sglang/PR-20012.md), [[diffusion] Support nvfp4 for Flux.2](../sources/prs/sglang/PR-20137.md), [fix ci by removing nvidia-cutlass-dsl-libs-base and force reinstall n…](../sources/prs/sglang/PR-20380.md), [fix(docs): correct quantization documentation (#20301)](../sources/prs/sglang/PR-20619.md), [Fix(jit): support rmsnorm for hidden_size in {64, 128, 256}](../sources/prs/sglang/PR-20661.md), [[Feature][JIT Kernel] Fused TP QK norm For Minimax](../sources/prs/sglang/PR-20673.md), [CUTLASS FP8 Blockwise GEMM improvement of SM120](../sources/prs/sglang/PR-20887.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Tiny Fix] Fix IS_BLACKWELL env var empty string warning in rerun-ut workflow](../sources/prs/sglang/PR-20957.md), [ci: run Stage A CUDA tests as stage-a-test-small-1-gpu on 5090](../sources/prs/sglang/PR-20988.md), [[Whisper] Enable CUDA graph support and timestamp for whisper model](../sources/prs/sglang/PR-21190.md), [Split pr-test.yml: extract sgl-kernel, jit-kernel, and multimodal-gen tests into separate workflow files](../sources/prs/sglang/PR-21219.md), [CUTLASS NVFP4 GEMM improvement of SM120](../sources/prs/sglang/PR-21314.md), [[Diffusion] Add qknorm rope fuse kernel](../sources/prs/sglang/PR-21440.md), [fix: piecewise_cuda_graph get correct qo_indptr](../sources/prs/sglang/PR-21452.md), [fix: TRT-LLM MHA CUDA illegal address with EAGLE v2 + DP attention](../sources/prs/sglang/PR-21649.md), [[jit_kernel] Optimize fused_qknorm_rope: deduplicate sincosf for interleave RoPE ](../sources/prs/sglang/PR-21654.md), [Remove flashinfer wheel cache cleanup that deletes other versions](../sources/prs/sglang/PR-21711.md), [[Feature] JIT rmsnorm update (with claude)](../sources/prs/sglang/PR-21834.md), [[Bugfix] Fix CUDA graph replay issues in trtllm_mla draft_extend](../sources/prs/sglang/PR-21987.md), [[Diffusion] Fix weight scale swizzle and add large-M kernel config for FLUX.2-dev-NVFP4](../sources/prs/sglang/PR-22064.md), [[diffusion] Default NVFP4 to CUTLASS and add all-model shape benchmarks](../sources/prs/sglang/PR-22091.md), [Lazy import flash_attention_v4 to avoid loading flash_attn.cute at startup](../sources/prs/sglang/PR-22306.md), [[Docker] Fix Trivy CVEs, cubin download 403s, and kernels command order](../sources/prs/sglang/PR-22322.md), [[CI/Docker] Clean up redundant flashinfer cubin downloads](../sources/prs/sglang/PR-22491.md), [[Docker] Remove flashinfer cache copy](../sources/prs/sglang/PR-22653.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [[Diffusion] Add Wan2.2 ModelOpt NVFP4 support](../sources/prs/sglang/PR-22681.md), [[Fix] Fix accuracy bug in Flashmla sparse MLA kernel](../sources/prs/sglang/PR-22723.md), [Dual MoE CUDA graph capture for lora/nolora batches](../sources/prs/sglang/PR-22809.md), [[Fix/Kernel] Add JIT rmsnorm_hf kernel to fix transformers backend MMLU accuracy regression ](../sources/prs/sglang/PR-22931.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [[feat] Init true on policy with qwen_dense](../sources/prs/sglang/PR-23961.md), [Enable PDL for various kernels in DSV32/GLM5](../sources/prs/sglang/PR-23965.md), [[VLM] Optimize Gemma4 VLM with PCG and fuse RMSNorm + residual add + scalar](../sources/prs/sglang/PR-24048.md), [Refactor device timer, clean up metrics collector, and add fwd occupancy metric](../sources/prs/sglang/PR-24197.md), [[codex] Optimize hidden-size 512 RMSNorm dispatch](../sources/prs/sglang/PR-24710.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [Support draft extend cuda graph for tokenspeed_mla attention backend](../sources/prs/sglang/PR-25489.md), [[fp8] SM90 swap-AB scaled_mm dispatch (~1.16x kernel geomean, +5.8-18.5% end-to-end)](../sources/prs/sglang/PR-25532.md), [fix (jit kernel): elementwise activation C++ error](../sources/prs/sglang/PR-25695.md), [[Codex] Remove stale DeepSeek V4 JIT kernels](../sources/prs/sglang/PR-25764.md), [[Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename](../sources/prs/sglang/PR-25821.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [Support cutlass Int8 gemm](../sources/prs/sglang/PR-2752.md), [upgrade cutlass v3.7.0](../sources/prs/sglang/PR-2967.md), [feat: add flashinfer as 3rdparty and use rmsnorm as example](../sources/prs/sglang/PR-3033.md), [Support sm90 Int8 gemm](../sources/prs/sglang/PR-3035.md), [support w8a8 fp8 kernel with CUTLASS](../sources/prs/sglang/PR-3047.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [fix undefined symbol cudaGetDriverEntryPointByVersion](../sources/prs/sglang/PR-3372.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [[Feature] Apply Cublas Grouped Gemm kernel](../sources/prs/sglang/PR-3629.md), [Support FP4 gemm (1/2)](../sources/prs/sglang/PR-3899.md), [Add fast decode plan for flashinfer mla](../sources/prs/sglang/PR-3987.md), [[Revision] Add fast decode plan for flashinfer mla ](../sources/prs/sglang/PR-4012.md), [Support overlapping two batches](../sources/prs/sglang/PR-4068.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0](../sources/prs/sglang/PR-4231.md), [add THIRDPARTYNOTICES for DeepGEMM](../sources/prs/sglang/PR-4272.md), [Support Blackwell Block Scale FP8 Gemm](../sources/prs/sglang/PR-4278.md), [update deepgemm](../sources/prs/sglang/PR-4284.md), [Add deepseek style fused moe group gate selection kernel](../sources/prs/sglang/PR-4530.md), [[Fix] Fix raw_bs bug when using flashinfer mla and eagle](../sources/prs/sglang/PR-4557.md), [Support fp8 gemm for blackwell](../sources/prs/sglang/PR-4558.md), [avoid cudaStreamSynchronize in DeepSeekV2AttentionMLA](../sources/prs/sglang/PR-4577.md), [support cmake for sgl-kernel](../sources/prs/sglang/PR-4706.md), [Support (1 <= dp < tp) in the dp attention in DeepEP](../sources/prs/sglang/PR-4770.md), [[Feat] support deepgemm for cmake](../sources/prs/sglang/PR-4864.md), [[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu](../sources/prs/sglang/PR-4953.md), [update cutlass tag](../sources/prs/sglang/PR-5011.md), [fix deepgemm as well](../sources/prs/sglang/PR-5030.md), [support sgl-kernel on blackwell](../sources/prs/sglang/PR-5074.md), [reduce moe_align_block_size_kernel small batch mode overhead](../sources/prs/sglang/PR-5086.md), [Blackwell Cutlass MLA kernel](../sources/prs/sglang/PR-5142.md), [[1/2] Add FP8 Blockscale MoE CUTLASS kernel for Blackwell](../sources/prs/sglang/PR-5281.md), [fix: solve cu118 issue for cutlass mla](../sources/prs/sglang/PR-5331.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [chore: upgrade DeepGEMM](../sources/prs/sglang/PR-5395.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [feat: use flashinfer jit package](../sources/prs/sglang/PR-5547.md), [[2/2] Add python wrapper for CUTLASS FP8 Blockscale MoE Kernel. ](../sources/prs/sglang/PR-5694.md), [[PP] Add pipeline parallelism](../sources/prs/sglang/PR-5724.md), [cutlass 3.9 supported to improve fp8_blockwise_gemm](../sources/prs/sglang/PR-5820.md), [Add sm_120 for blackwell](../sources/prs/sglang/PR-5903.md), [chore: upgrade cutlass 3.9.2](../sources/prs/sglang/PR-6004.md), [chore: upgrade deepgemm](../sources/prs/sglang/PR-6073.md), [feat: mtp support dp-attention](../sources/prs/sglang/PR-6081.md), [[1/2] Add Kernel support for Cutlass based Fused FP4 MoE](../sources/prs/sglang/PR-6093.md), [Cutlass MLA: Disable split kv due to https://github.com/NVIDIA/cutlass/issues/2274](../sources/prs/sglang/PR-6101.md), [Upgrade CUTLASS 4.0](../sources/prs/sglang/PR-6336.md), [reduce torch.zeros overhead in moe align block size kernel](../sources/prs/sglang/PR-6369.md), [[Feature] Comprehensive Hybrid Parallelism Support](../sources/prs/sglang/PR-6389.md), [Fix bug of deepseek-v3 under DP+EP mode with large batchsize/seqlen](../sources/prs/sglang/PR-6449.md), [[EP] Add cuda kernel for moe_ep_pre_reorder](../sources/prs/sglang/PR-6699.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [Correctly abort the failed grammar requests & Improve the handling of abort](../sources/prs/sglang/PR-6803.md), [[EP] Add cuda kernel for moe_ep_post_reorder](../sources/prs/sglang/PR-6837.md), [Fix AWQ Dequant and Weight Loading of deepseek v2](../sources/prs/sglang/PR-6842.md), [fix ep_moe_reorder kernel bugs](../sources/prs/sglang/PR-6858.md), [Add a CUDA kernel for fusing mapping and weighted sum for MoE.](../sources/prs/sglang/PR-6916.md), [[sgl-kernel] Add cuda kernel for moe_ep_silu_and_mul](../sources/prs/sglang/PR-6919.md), [[perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128](../sources/prs/sglang/PR-6929.md), [Clean up server_args.py](../sources/prs/sglang/PR-7037.md), [Tiny fix cutlass_mla_get_workspace_size stub incorrect signature](../sources/prs/sglang/PR-7057.md), [Fix grammar abort & Minor style fixes](../sources/prs/sglang/PR-7204.md), [feat: support compatibility between MTP and two-batch-overlap](../sources/prs/sglang/PR-7225.md), [Add CUTLASS FP8 Blockscale MoE kernel for Hopper architecture](../sources/prs/sglang/PR-7278.md), [fix: resolve blackwell deepep image issue](../sources/prs/sglang/PR-7331.md), [Fuse sorted_token_ids padding to moe_align_block_size kernel](../sources/prs/sglang/PR-7437.md), [fix: fix apply_shuffle_mul_sum](../sources/prs/sglang/PR-7444.md), [[CMake] Fix sgl-kernel CMakeLists for Blackwell](../sources/prs/sglang/PR-7543.md), [Add dsv3 router gemm kernel](../sources/prs/sglang/PR-7627.md), [Add dsv3 fused a gemm to sgl-kernel](../sources/prs/sglang/PR-7630.md), [[Feature] CUDA Green Context Support](../sources/prs/sglang/PR-7649.md), [[1/n]: add cutlass W4A8 moe kernel for hopper architecture](../sources/prs/sglang/PR-7772.md), [[kernel] opt moe align block kernel by block/warp scan algorithm](../sources/prs/sglang/PR-7884.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [[Fix][Ready]Fix register spilling in cutlass nvfp4 gemm kernel on Blackwell](../sources/prs/sglang/PR-8127.md), [[sgl-kernel] Opt per_token_quant_fp8 with warp reduce](../sources/prs/sglang/PR-8130.md), [[2/N]Support DeepSeek-R1 w4a8 low latency deepep](../sources/prs/sglang/PR-8464.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/sglang/PR-8818.md), [chore: support blackwell cu129 image](../sources/prs/sglang/PR-8928.md), [[NVIDA] [1/N] Nvfp4 Masked Gemm: Add quant op for the flashinfer grouped gemm](../sources/prs/sglang/PR-9200.md), [[fix]: fix cutlass moe ut and and Opt H20 cutlass groupGemm performance](../sources/prs/sglang/PR-9272.md), [Support trtllm_allreduce_fusion in flashinfer for cuda<12.8](../sources/prs/sglang/PR-9339.md), [[sgl-kernel] feat: Support sm120 cutlass fp8 gemm kernel](../sources/prs/sglang/PR-9403.md), [[NVIDIA] [2/N] Optimize `silu_and_mul_scaled_fp4_grouped_quant` perf](../sources/prs/sglang/PR-9556.md), [Update CUTLASS 4.2 & Enable K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM](../sources/prs/sglang/PR-9559.md), [Make sm100 fp8 kernels available on sm103](../sources/prs/sglang/PR-9789.md), [Make fp4_quantize kernels work on sm103](../sources/prs/sglang/PR-9807.md), [CUTLASS fp8 blockwise gemm support of sm120](../sources/prs/sglang/PR-9969.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feat] profiler support cudagraph backend](../sources/prs/tilelang/PR-1658.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [[Feature] Support tcgen5mma lowering for `.kind::i8`](../sources/prs/tilelang/PR-1764.md), [[CUDA] Support tcgen5mma gemm ts](../sources/prs/tilelang/PR-1866.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[CUDA] Improve int4 GEMM lowering and packed codegen support](../sources/prs/tilelang/PR-2073.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[CUDA] Add native SM75 MMA GEMM support for FP16, INT8 and INT4](../sources/prs/tilelang/PR-2198.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support](../sources/prs/vllm/PR-10995.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [[Build] Only build 9.0a for scaled_mm and sparse kernels](../sources/prs/vllm/PR-12339.md), [[ROCm] Faster Custom Paged Attention kernels](../sources/prs/vllm/PR-12348.md), [[Attention] MLA decode optimizations](../sources/prs/vllm/PR-12528.md), [[Kernel] port sgl moe_align_block_size kernels](../sources/prs/vllm/PR-12574.md), [[Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3](../sources/prs/vllm/PR-12587.md), [[Attention] MLA with chunked prefill](../sources/prs/vllm/PR-12639.md), [[Perf] Mem align KV caches for CUDA devices (MLA perf improvement)](../sources/prs/vllm/PR-12676.md), [[Core][AMD] Migrate fully transparent sleep mode to ROCm platform](../sources/prs/vllm/PR-12695.md), [[Bugfix][Kernel] Fix per-token/per-channel quantization for Hopper scaled mm](../sources/prs/vllm/PR-12696.md), [[Kernel] Make rotary_embedding ops more flexible with input shape](../sources/prs/vllm/PR-12777.md), [[NVIDIA] Support nvfp4 quantization](../sources/prs/vllm/PR-12784.md), [Optimize moe_align_block_size for deepseek_v3](../sources/prs/vllm/PR-12850.md), [[Misc][Kernel]: Add GPTQAllSpark Quantization](../sources/prs/vllm/PR-12931.md), [[Kernel]Add streamK for block-quantized CUTLASS kernels](../sources/prs/vllm/PR-12978.md), [[Kernel] moe wna16 cuda kernel](../sources/prs/vllm/PR-13321.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[ROCm][MoE] mi300 mixtral8x7B perf for specific BS](../sources/prs/vllm/PR-13577.md), [[Kernel] FlashMLA integration](../sources/prs/vllm/PR-13747.md), [[Attention] MLA support for V1](../sources/prs/vllm/PR-13789.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[ROCm] Disable chunked prefill/prefix caching when running MLA on non-cuda platforms](../sources/prs/vllm/PR-13844.md), [[Attention] Flash MLA for V1](../sources/prs/vllm/PR-13867.md), [[V1] EP/TP MoE + DP Attention](../sources/prs/vllm/PR-13931.md), [[Kernel] CUTLASS grouped gemm fp8 MoE kernel](../sources/prs/vllm/PR-13972.md), [[Kernel] optimize performance of gptq marlin kernel when n is small](../sources/prs/vllm/PR-14138.md), [dynamic distpatch of fp8 kernels](../sources/prs/vllm/PR-14245.md), [[Attention] FlashAttn MLA](../sources/prs/vllm/PR-14258.md), [fix minor miscalled method](../sources/prs/vllm/PR-14327.md), [[Build/BugFix] Fix hopper 12.8 build](../sources/prs/vllm/PR-14354.md), [Add cutlass support for blackwell fp8 blockwise gemm](../sources/prs/vllm/PR-14383.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [[Kernel] moe wna16 marlin kernel](../sources/prs/vllm/PR-14447.md), [permute/unpermute kernel for moe optimization](../sources/prs/vllm/PR-14568.md), [[Attention] Flash Attention 3 - fp8](../sources/prs/vllm/PR-14570.md), [[BugFix/Build] Fix sparse kernels not getting built on hopper](../sources/prs/vllm/PR-14572.md), [[Kernel] GGUF MoE kernel](../sources/prs/vllm/PR-14613.md), [[Kernel] allow non-contiguous input for marlin kernel](../sources/prs/vllm/PR-14658.md), [[Bugfix][Kernel][CPU] Fix num_tokens in CPU rotary embedding kernel](../sources/prs/vllm/PR-14667.md), [[V1] Fully Transparent Implementation of CPU Offloading](../sources/prs/vllm/PR-15354.md), [[Kernel] Fix conflicting macro names for gguf kernels](../sources/prs/vllm/PR-15456.md), [[ROCM][KERNEL] Paged attention for V1](../sources/prs/vllm/PR-15720.md), [[Bugfix] fix use_atomic_add support of marlin kernel when using v1 engine](../sources/prs/vllm/PR-15946.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [[ROCM] Add gfx950 to the custom attention archs](../sources/prs/vllm/PR-16034.md), [Add FlexAttention to V1](../sources/prs/vllm/PR-16078.md), [[Kernel] support merge_attn_states CUDA kernel, 3x speedup](../sources/prs/vllm/PR-16173.md), [[Hardware/NVIDIA/Kernel] [Functional Enablement] [1/N] Enable nvidia/DeepSeek-R1-FP4 Model](../sources/prs/vllm/PR-16362.md), [Allocate kv_cache with stride order](../sources/prs/vllm/PR-16605.md), [[V1] V1 FlashInfer Attention](../sources/prs/vllm/PR-16684.md), [[misc] ignore marlin_moe_wna16 local gen codes](../sources/prs/vllm/PR-16760.md), [[Kernel] GGUF MoeVec kernel](../sources/prs/vllm/PR-16780.md), [[BugFix] Accuracy fix for llama4 int4 - improperly casted scales](../sources/prs/vllm/PR-16801.md), [[Kernel] some optimizations for dense marlin and moe marlin](../sources/prs/vllm/PR-16850.md), [Update PyTorch to 2.7.0](../sources/prs/vllm/PR-16859.md), [[Kernel] Add expert_map support to Cutlass FP8 MOE](../sources/prs/vllm/PR-16861.md), [[Attention] FA3 decode perf improvement - single mma warp group support for head dim 128](../sources/prs/vllm/PR-16864.md), [Update Qwen1.5-MoE-W4A16-compressed-tensors.yaml](../sources/prs/vllm/PR-16946.md), [[ROCm][Kernel][V1] Enable AMD Radeon GPU Custom Paged Attention on v1](../sources/prs/vllm/PR-17004.md), [Fix `numel()` downcast in vllm/csrc/moe/moe_align_sum_kernels.cu +2](../sources/prs/vllm/PR-17082.md), [[ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention](../sources/prs/vllm/PR-17139.md), [[NVIDIA] Support Cutlass w8a8 FP8 for Blackwell Geforce GPUs (sm120)](../sources/prs/vllm/PR-17280.md), [[Misc][ROCm] Exclude `cutlass_mla_decode` for ROCm build](../sources/prs/vllm/PR-17289.md), [[Attention] MLA move o_proj q_proj into cuda-graph region](../sources/prs/vllm/PR-17484.md), [[Attention] MLA move rotary embedding to cuda-graph region](../sources/prs/vllm/PR-17668.md), [[Kernel] fp4 marlin kernel](../sources/prs/vllm/PR-17687.md), [[Kernel] Have rotary embeddings support tensors](../sources/prs/vllm/PR-18046.md), [Fix Broken macro for cutlass moe](../sources/prs/vllm/PR-18049.md), [[Build] Supports CUDA 12.6 and 11.8 after Blackwell Update](../sources/prs/vllm/PR-18316.md), [Sm100 blockwise fp8 swap ab](../sources/prs/vllm/PR-18564.md), [[Kernel] Integrate CUTLASS MoE kernel with PPLX](../sources/prs/vllm/PR-18762.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-18778.md), [[BugFix] FA2 MLA Accuracy Issue](../sources/prs/vllm/PR-18807.md), [[Hardware][NVIDIA] FP4 MoE kernel optimization](../sources/prs/vllm/PR-19110.md), [[V1] Use FlashInfer by default on Blackwell GPUs](../sources/prs/vllm/PR-19118.md), [[Bugfix][V1] Allow manual FlashAttention for Blackwell](../sources/prs/vllm/PR-19492.md), [[Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization](../sources/prs/vllm/PR-19500.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [Only build CUTLASS MoE kernels on Hopper](../sources/prs/vllm/PR-19648.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [Fix FA2 fallback for Blackwell V1](../sources/prs/vllm/PR-19781.md), [[Bugfix] Build moe_data for both sm100 and sm90](../sources/prs/vllm/PR-20086.md), [[Bugfix] Fix some narrowing conversion warnings](../sources/prs/vllm/PR-20141.md), [Replace `multiply_add` with `homogeneous_multiply_add` to Address Clang Template Parameter Issue](../sources/prs/vllm/PR-20142.md), [[Bugfix] Fix topk_ids indices_type for CUTLASS w8a8 FP8 MoE](../sources/prs/vllm/PR-20166.md), [[Kernel][Bugfix] Fixup some warnings in nvfp4_blockwise_moe when CUDA < 12.8](../sources/prs/vllm/PR-20324.md), [Update PyTorch to 2.8.0](../sources/prs/vllm/PR-20358.md), [[Kernel] SM90 CUTLASS FP8 GEMM: add support for swap AB + kernel tuning](../sources/prs/vllm/PR-20396.md), [[feat]: add SM100 support for cutlass FP8 groupGEMM](../sources/prs/vllm/PR-20447.md), [[Performance] Performance improvements in non-blockwise fp8 CUTLASS MoE](../sources/prs/vllm/PR-20762.md), [SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for DeepSeek TP](../sources/prs/vllm/PR-20769.md), [[fix]: disable cutlass block scaled group gemm for EP](../sources/prs/vllm/PR-20781.md), [[Perf] Add swap_ab to SM90 FP8 non-block CUTLASS moe grouped gemm](../sources/prs/vllm/PR-20911.md), [Support mnnvl all2allv from Flashinfer](../sources/prs/vllm/PR-21003.md), [[Kernel] Flashinfer MLA (trtllm-gen) decode kernel integration](../sources/prs/vllm/PR-21078.md), [[Perf] Cuda Kernel for Per Token Group Quant](../sources/prs/vllm/PR-21083.md), [[perf] Add fused MLA QKV + strided layernorm](../sources/prs/vllm/PR-21116.md), [[Perf] Use FlashInfer RoPE for RotaryEmbedding.forward_cuda when available](../sources/prs/vllm/PR-21126.md), [[Feature][Kernel]FusedMoE LoRA](../sources/prs/vllm/PR-21229.md), [[v1] - Mamba1 Attention Metadata](../sources/prs/vllm/PR-21249.md), [Support CUTLASS NVFP4 (w4a4) for Blackwell Geforce GPUs (SM120)](../sources/prs/vllm/PR-21309.md), [[Bugfix][CUDA] fixes CUDA FP8 kv cache dtype supported](../sources/prs/vllm/PR-21420.md), [[Bug] Fix Compressed Tensor NVFP4 `cutlass_fp4_group_mm` illegal memory access](../sources/prs/vllm/PR-21465.md), [[Kernel] Improve machete memory bound perf](../sources/prs/vllm/PR-21556.md), [[BugFix] Fix IMA FlashMLA full cuda-graph and DP + Update FlashMLA](../sources/prs/vllm/PR-21691.md), [update flashinfer to v0.2.9rc2](../sources/prs/vllm/PR-21701.md), [Fix Flashinfer CUTLASS MOE Allgather](../sources/prs/vllm/PR-21963.md), [[Kernel] Add support for block FP8 on SM120 (NVIDIA 5090 and RTX PRO 6000)](../sources/prs/vllm/PR-22131.md), [Fp8 paged attention update](../sources/prs/vllm/PR-22222.md), [Upgrade FA3 for attention sink](../sources/prs/vllm/PR-22313.md), [[Attention] FA3 Attention Sinks Perf Boost](../sources/prs/vllm/PR-22478.md), [[Fix] enable swap_ab for pplx problem size computation](../sources/prs/vllm/PR-22991.md), [[Kernel] CUTLASS MoE FP8: Integrate cuda moe permute/unpermute](../sources/prs/vllm/PR-23045.md), [[V1] address post issues related to #20059 (part 1); cascade attention reenable by default](../sources/prs/vllm/PR-23046.md), [[kernel] Support W4A8 on Hopper](../sources/prs/vllm/PR-23198.md), [[Kernel] Add fused grouped_topk kernel for MoE](../sources/prs/vllm/PR-23274.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Compile] Fix Compile Warning SM100 Cutlass MLA](../sources/prs/vllm/PR-23287.md), [fix incompatibililty with non cuda platform for nvfp4](../sources/prs/vllm/PR-23478.md), [[Compile] Fix Compile Warning for `w4a8_mm_entry.cu`](../sources/prs/vllm/PR-23660.md), [[NVIDIA] Support SiluMul + NVFP4 quant fusion](../sources/prs/vllm/PR-23671.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files](../sources/prs/vllm/PR-23727.md), [[Feature] Support Decode Context Parallel (DCP) for MLA](../sources/prs/vllm/PR-23734.md), [[Kernel] cuda kernels for upcoming decode context parallel feature](../sources/prs/vllm/PR-23791.md), [[Kernel] Faster pre-processing time for W4A8](../sources/prs/vllm/PR-23972.md), [[Model] Add LongCat-Flash ](../sources/prs/vllm/PR-23991.md), [[Kernel] Support decode context parallelism on Blackwell with CUTLASS MLA](../sources/prs/vllm/PR-24385.md), [[NVIDIA] Blackwell Family](../sources/prs/vllm/PR-24673.md), [[Kernel][Quantization] add w4a8 support for marlin kernel](../sources/prs/vllm/PR-24722.md), [[Bugfix] Fix accuracy issue for silu_mul + nvfp4 quant fusion kernel](../sources/prs/vllm/PR-24833.md), [[Bugfix][B200] Fix `cutlass_mla` hang](../sources/prs/vllm/PR-24966.md), [Disable failing GPT-OSS Eval (Blackwell) for now](../sources/prs/vllm/PR-25107.md), [[Compile] Fix Compile Warning for Ignoring `MIN_BLOCK_PER_SM`](../sources/prs/vllm/PR-25193.md), [[Bugfix] [B200] cutlass_mla - ensure kv_split == 1 for batch size > 1](../sources/prs/vllm/PR-25509.md), [Fuse RoPE and MLA KV-cache write](../sources/prs/vllm/PR-25774.md), [Update launch_bounds_utils.h for correct compile on Multiple Cuda Arch - PTXAS out of range Warning](../sources/prs/vllm/PR-25843.md), [Fix INT8 quantization error on Blackwell GPUs (SM100+)](../sources/prs/vllm/PR-25935.md), [[Performance] Split FlashAttn attention and cache update](../sources/prs/vllm/PR-25954.md), [Fix undefined symbol: cutlass_moe_mm_sm100](../sources/prs/vllm/PR-26098.md), [[NVIDIA] [Perf] Update to leverage flashinfer trtllm FP4 MOE throughput kernel](../sources/prs/vllm/PR-26714.md), [[Attention] Tune CUTLASS MLA num_splits](../sources/prs/vllm/PR-26846.md), [[Perf] SM100 - add swap AB optimization to CUTLASS FP8 GEMM](../sources/prs/vllm/PR-27284.md), [Prefer FlashAttention MLA as default over FlashMLA](../sources/prs/vllm/PR-27363.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [[Performance] Fused blockwise quant RMS norm](../sources/prs/vllm/PR-27883.md), [[Kernel] Optimize rms_norm kernel](../sources/prs/vllm/PR-27931.md), [Update Flashinfer from `v0.4.1` to `v0.5.2`](../sources/prs/vllm/PR-27952.md), [[Perf][DeepSeek] Add sigmoid+bias fusion to fused_grouped_topk from TRTLLM](../sources/prs/vllm/PR-28124.md), [[Performance][B200] silu_mul_quant: pack scales in int32](../sources/prs/vllm/PR-28358.md), [[NVIDIA] Guard SM100 CUTLASS MoE macro to SM100 builds v2](../sources/prs/vllm/PR-28938.md), [chore: add RTX_PRO_6000 GLM4.6-FP8 kernel tuning](../sources/prs/vllm/PR-29240.md), [[Kernel] Add NVFP4 MoE CUTLASS support for SM120](../sources/prs/vllm/PR-29242.md), [Lora MoE Align Improvements](../sources/prs/vllm/PR-29257.md), [[Kernel][MoE] optimize `moe_align_block_size`](../sources/prs/vllm/PR-29642.md), [[Kernel]Support W4A8 Grouped GEMM on Hopper](../sources/prs/vllm/PR-29691.md), [[Perf] Improve fp8 quant in mla; replace ReduceSum with ReduceScatterSum](../sources/prs/vllm/PR-29795.md), [[Kernel][Quantization][MoE] add marlin kernel support for turing (sm75)](../sources/prs/vllm/PR-29901.md), [[Perf] Do FP4 quant before All gather on flashinfer trtllmgen MOE ](../sources/prs/vllm/PR-30014.md), [Add llmcompressor fp8 kv-cache quant (per-tensor and per-attn_head)](../sources/prs/vllm/PR-30141.md), [gptq marlin quantization support for fused moe with lora](../sources/prs/vllm/PR-30254.md), [[Feature] Add SM103 (Blackwell Ultra) Support to vLLM](../sources/prs/vllm/PR-30484.md), [OffloadingConnector: Support kernel_block_size != block_size](../sources/prs/vllm/PR-30692.md), [[NVFP4][Perf] Tune NVFP4 input quant kernel for small batch size](../sources/prs/vllm/PR-30897.md), [[Kernel] Add topk_sigmoid kernel](../sources/prs/vllm/PR-31246.md), [[Perf] Fuse stride preparation for NVFP4 cutlass_moe](../sources/prs/vllm/PR-31837.md), [[1/N][Attention] Restructure attention: move files](../sources/prs/vllm/PR-31916.md), [[Perf][Kernel] Optimize FP4 quantization kernels (SM100F)](../sources/prs/vllm/PR-32520.md), [fix: Add glm4_moe_lite to MLA detection](../sources/prs/vllm/PR-32614.md), [[Attention] FA4 integration](../sources/prs/vllm/PR-32974.md), [[Feature] Support CPU Offloading without Pytorch Pinned Memory that leads to doubled allocation](../sources/prs/vllm/PR-32993.md), [[Kernel] Apply 256bit LDG/STG To Activation Kernels](../sources/prs/vllm/PR-33022.md), [Add support for Mistral Large 3 inference with Flashinfer MoE](../sources/prs/vllm/PR-33174.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Kernel] Add enable_sm120_or_later for SM121 (DGX Spark) CUTLASS support](../sources/prs/vllm/PR-33517.md), [[Feature][Core] Support Fabric detection to adapt the MNNVL protocol for the GB series](../sources/prs/vllm/PR-33540.md), [[Bugfix]fix output Nan/Inf in marlin if dtype=float16](../sources/prs/vllm/PR-33972.md), [Reapply [Attention][FA3] Update FA3 to include new swizzle optimization](../sources/prs/vllm/PR-34043.md), [fix(cpu): fix mla_decode compilation on x86 without AVX512](../sources/prs/vllm/PR-34052.md), [[Kernel] Optimize grouped topk kernel](../sources/prs/vllm/PR-34206.md), [[ModelBash][DSV3] Add TRTLLM DSV3 Router GEMM kernel (6% B1 Speedup)](../sources/prs/vllm/PR-34302.md), [[Bugfix] Enforce DeepGEMM when using sparse_attn_indexer on CUDA](../sources/prs/vllm/PR-34374.md), [[Custom Ops] Add functional + out variant for scaled_fp4_quant](../sources/prs/vllm/PR-34389.md), [[Kernel] Integrate SM100 MXFP8 blockscaled grouped MM and quant kernels](../sources/prs/vllm/PR-34448.md), [[Model Bash] DeepSeek R1 BF16 Min Latency QKV A GEMM (0.5% E2E Speedup)](../sources/prs/vllm/PR-34758.md), [[Bugfix] Gate 256-bit instructions to CUDA 12.9+](../sources/prs/vllm/PR-34791.md), [[Attention][Perf][Kernel] Replace torch.cat with vectorized CUDA kernel MLA query concat - DeepSeek-V3.2](../sources/prs/vllm/PR-34917.md), [[Model Runner V2] Support attention group](../sources/prs/vllm/PR-35036.md), [[Performance] Cublas Bf16 Gate with Fp32 Output](../sources/prs/vllm/PR-35121.md), [[Bugfix] Fix DSV3 kernels breaking _C and _moe_C on unsupported arches](../sources/prs/vllm/PR-35123.md), [[Bugfix] Fix expert_ids padding values in moe_align_block_size kernel](../sources/prs/vllm/PR-35161.md), [[BugFix] Fix fp4 quant kernel on CUDA 12.8](../sources/prs/vllm/PR-35210.md), [[Feat] Add CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function](../sources/prs/vllm/PR-35271.md), [[Attention][Perf] Optimize cp_gather_and_upconvert_fp8_kv_cache - DeepSeek-v3.2](../sources/prs/vllm/PR-35290.md), [[Kernel] Add FlashInfer MoE A2A Kernel](../sources/prs/vllm/PR-36022.md), [Add 320 dimension size support to MLA](../sources/prs/vllm/PR-36161.md), [docs: fix wrong cc in int8.md](../sources/prs/vllm/PR-36209.md), [[Kernel] Fuse FP8 output quantization into merge_attn_states](../sources/prs/vllm/PR-36518.md), [Update Flashinfer to 0.6.6](../sources/prs/vllm/PR-36768.md), [[MTP][Sparse MLA] Take advantage of native MTP support in indexer when possible](../sources/prs/vllm/PR-36982.md), [[Kernel] Add gpt-oss Router GEMM kernel](../sources/prs/vllm/PR-37205.md), [[UX] Add flashinfer-cubin as CUDA default dep](../sources/prs/vllm/PR-37233.md), [[Perf] Set Flashinfer sparse MLA as default backend for FP8 kv cache](../sources/prs/vllm/PR-37252.md), [[Kernel] Add non-gated support for NVFP4 CUTLASS MoE](../sources/prs/vllm/PR-37320.md), [Add nvfp4 support to reshape_and_cache_flash](../sources/prs/vllm/PR-37332.md), [[torch.compile] Refactor Attention Quant Fusion Pass and Remove Boilerplate](../sources/prs/vllm/PR-37373.md), [[Perf][Kernel] Persistent TopK scheduler: unified CUDAGraph-safe kernel with dynamic per-row dispatch - DeepSeek-V3.2 DSA decode](../sources/prs/vllm/PR-37421.md), [[Kernel] Add MXFP4 W4A4 CUTLASS MoE kernel for SM100](../sources/prs/vllm/PR-37463.md), [[4/n] Migrate FP4/W4A8 CUTLASS kernels to torch stable ABI](../sources/prs/vllm/PR-37503.md), [refactor: abstract deepgemm support into platform](../sources/prs/vllm/PR-37519.md), [[Bugfix] Preserve CUDA arch suffix (a/f) for SM12x — fixes NVFP4 NaN on desktop Blackwell](../sources/prs/vllm/PR-37725.md), [[Kernel] Optimize SM120 CUTLASS blockwise FP8 GEMM](../sources/prs/vllm/PR-37970.md), [[Kernel] Add swapAB support for SM120 CUTLASS blockwise FP8 GEMM ](../sources/prs/vllm/PR-38325.md), [[CI Bugfix] Pre-download missing FlashInfer headers in Docker build](../sources/prs/vllm/PR-38391.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[Perf] Batch KV cache swap copies via cuMemcpyBatchAsync](../sources/prs/vllm/PR-38460.md), [[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity](../sources/prs/vllm/PR-38479.md), [[Compile] Fix nvfp4 compile warning](../sources/prs/vllm/PR-38573.md), [[FA4] Update flash-attention to latest upstream FA4](../sources/prs/vllm/PR-38690.md), [[Refactor] Improve indexer decode path metadata preparation](../sources/prs/vllm/PR-38865.md), [[Bugfix] Fix broken explicit unquantized kv cache dtype support](../sources/prs/vllm/PR-38922.md), [[Bugfix] Fix GDN FLA kernel crashes with NULL_BLOCK_ID=0 CUDA graph padding](../sources/prs/vllm/PR-39064.md), [[XPU] Quick fix for TritonMLA to remove cuda hardcode](../sources/prs/vllm/PR-39088.md), [perf(moe): add tuned fused_moe config for RTX PRO 6000 Blackwell Server Edition](../sources/prs/vllm/PR-39183.md), [Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps](../sources/prs/vllm/PR-39306.md), [Fix NUMA binding on non-CDMM Grace-Blackwell systems](../sources/prs/vllm/PR-39361.md), [fix: clamp NaN/Inf in topk_softmax to prevent duplicate expert IDs](../sources/prs/vllm/PR-39391.md), [[Perf] Fuse Zero Initializer for FP8 DeepGemm Block Quant Kernel](../sources/prs/vllm/PR-39547.md), [[Bugfix] Add Marlin kernel in block scaled mm kernel selection.](../sources/prs/vllm/PR-40105.md), [[Bugfix] moe lora align kernel grid](../sources/prs/vllm/PR-40131.md), [[Bugfix] Guard mxfp4_experts_quant bindings on ENABLE_NVFP4_SM100](../sources/prs/vllm/PR-40191.md), [[Performance][DSR1]: Fused RoPE+KVCache+q_concat for MLA](../sources/prs/vllm/PR-40392.md), [[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement](../sources/prs/vllm/PR-40408.md), [[GDN] Enable FI Blackwell GDN prefill kernel](../sources/prs/vllm/PR-40717.md), [[DSV4] Add silu clamp limit to shared expert](../sources/prs/vllm/PR-40950.md), [[DSV4] Fuse norm and router for low latency scenario](../sources/prs/vllm/PR-41263.md), [Faster per-token fp8 group quant packed kernel for blackwell](../sources/prs/vllm/PR-41326.md), [[Bugfix] Fix condition to clear persistent topk so that it can be captured regardless](../sources/prs/vllm/PR-41665.md), [[MLA Attention Backend] Add TOKENSPEED_MLA backend for DSR1/Kimi K25 prefill + decode on Blackwell](../sources/prs/vllm/PR-41778.md), [[CUDA][CUTLASS] Enable cutlass scaled mm for non-compatible sizes ](../sources/prs/vllm/PR-41868.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Perf] Use 2D-grid to eliminate divmod in W8W8 group quant](../sources/prs/vllm/PR-42153.md), [[Misc] add humming to dependencies](../sources/prs/vllm/PR-42540.md), [[6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued)](../sources/prs/vllm/PR-42663.md), [[Refactor] Remove dead cuda kernels](../sources/prs/vllm/PR-42767.md), [[Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement](../sources/prs/vllm/PR-42774.md), [[Perf][MLA] Enable FULL cudagraph capture for TRITON_MLA decode](../sources/prs/vllm/PR-42885.md), [[Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel ](../sources/prs/vllm/PR-7174.md), [[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin](../sources/prs/vllm/PR-7701.md), [[Bugfix] Fix Machete unittests failing with `NotImplementedError`](../sources/prs/vllm/PR-9218.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../wiki/kernels/flashmla.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Gated Delta Net — Linear Attention](../wiki/kernels/gated-delta-net.md), [Gated Dual GEMM (Gate-Up + SwiGLU Fusion)](../wiki/kernels/gated-dual-gemm.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md), [Sparse MLA (DeepSeek V3.2)](../wiki/kernels/sparse-mla.md), [TensorRT-LLM Blackwell FP4 DSA Indexer](../wiki/kernels/tensorrt-llm-blackwell-indexer.md), [CCCL CUB Memory Primitives For Selection And Scan](../wiki/techniques/cccl-memory-primitives.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md) | +| `cuda-cpp` | [CUDA C++ for Blackwell Kernels](../wiki/languages/cuda-cpp.md) | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Kernels](../sources/blogs/colfax-cutlass-kernels.md), [Modular: Matrix Multiplication on Blackwell](../sources/blogs/modular-blackwell-matmul.md), [NVIDIA Developer Code Samples](../sources/blogs/nvidia-code-samples.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [NVIDIA Blackwell Compatibility Guide](../sources/docs/blackwell-compatibility-guide.md), [NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6)](../sources/docs/nvidia-ampere-tuning-guide.md), [NVIDIA Ampere GA102 GPU Architecture Whitepaper (v2.1)](../sources/docs/nvidia-ga102-whitepaper.md), [Fix performance issue of m-grouped contiguous GEMMs.](../sources/prs/DeepGEMM/PR-168.md), [Fix multicast bug and optimize masked GEMM](../sources/prs/DeepGEMM/PR-193.md), [fix: use SM90ArchSpec instead of SM100ArchSpec in sm90_bf16_k_grouped_gemm](../sources/prs/DeepGEMM/PR-270.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [ Solving bank conflict via padding and TMA 3D store](../sources/prs/DeepGEMM/PR-78.md), [Use 1D TMA store instead of 3D](../sources/prs/DeepGEMM/PR-83.md), [Use swizzling instead of padding](../sources/prs/DeepGEMM/PR-86.md), [Support TMA multicast on B with m_grouped_gemm_contiguous.](../sources/prs/DeepGEMM/PR-88.md), [[None][feat] sm100 weight-only kernel](../sources/prs/TensorRT-LLM/PR-10190.md), [[TRTLLM-10022][feat] Add hopper xqa decode support for skip softmax attention](../sources/prs/TensorRT-LLM/PR-10264.md), [[https://nvbugs/5669671][fix] Support GuidedDecoder with sharded logits (pick #10698)](../sources/prs/TensorRT-LLM/PR-10742.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11165.md), [[None][feat] Optimize super-v3 nvfp4 for better perf](../sources/prs/TensorRT-LLM/PR-11273.md), [[None][feat] Optimize by fuse nvfp4_quant to layernorm_gated for mamba2_mixer](../sources/prs/TensorRT-LLM/PR-11473.md), [[None][feat] TRT-LLM Gen MoE finalize kernel optimization](../sources/prs/TensorRT-LLM/PR-11501.md), [[None][feat] Add support for expert_number<=2048 and K<=32](../sources/prs/TensorRT-LLM/PR-11510.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[TRTLLM-11119][feat] Blackwell SageAttention, Integrate into AttentionOp API](../sources/prs/TensorRT-LLM/PR-11718.md), [[https://nvbugs/5799917][fix] Recover from CUTLASS MoE doActivation perf regression for MXFP4/NVFP4 dtype](../sources/prs/TensorRT-LLM/PR-11733.md), [[None][feat] Add fused DiT QK Norm + RoPE CUDA kernel for FLUX](../sources/prs/TensorRT-LLM/PR-11869.md), [[TRTLLM-10421][perf] Add fused cat+fp8_quantize CUDA kernel for DSA indexer](../sources/prs/TensorRT-LLM/PR-11899.md), [[#11694][feat] AutoDeploy: Improve the piecewise CG memory usage](../sources/prs/TensorRT-LLM/PR-11993.md), [[https://nvbugs/5955188][fix] Fix harmony parsers and WAR routing PDL for agentic coding use cases](../sources/prs/TensorRT-LLM/PR-12046.md), [[TRTLLM-11540][feat] Add EAGLE3 dynamic tree speculative decoding support](../sources/prs/TensorRT-LLM/PR-12062.md), [[None][feat] Minimax RMS norm optimization](../sources/prs/TensorRT-LLM/PR-12163.md), [[None][feat] Add fused allreduce+RMSNorm op and optional residual in …](../sources/prs/TensorRT-LLM/PR-12201.md), [[None][feat] Support update weight for nvfp4](../sources/prs/TensorRT-LLM/PR-12320.md), [[None][feat] Temporally-Correlated Heuristic-guided Indexer TopK for Sparse Attention](../sources/prs/TensorRT-LLM/PR-12385.md), [[None][perf] add Dynamic SMEM block routing in MOE](../sources/prs/TensorRT-LLM/PR-12456.md), [[None][feat] Support sparse mqa/gqa attention](../sources/prs/TensorRT-LLM/PR-12470.md), [[https://nvbugs/5983390][perf] Split MLA DSA custom op for piecewise CUDA graph capture](../sources/prs/TensorRT-LLM/PR-12503.md), [[None][feat] Add Mamba2 MTP SSM cache CUDA kernel for tree-based speculative decoding](../sources/prs/TensorRT-LLM/PR-12537.md), [[https://nvbugs/5983390][perf] Multiple host perf optimizations for DSA part](../sources/prs/TensorRT-LLM/PR-12581.md), [[None][feat] Trtllm-gen FMHA JIT support](../sources/prs/TensorRT-LLM/PR-12612.md), [[None][feat] Optimize mamba SSD prefill and extend flashinfer dispatch](../sources/prs/TensorRT-LLM/PR-12731.md), [[TRTLLM-11485][feat] Feature rework: Add SageAttention refreshed kernels (attentionOp only)](../sources/prs/TensorRT-LLM/PR-12937.md), [[None][feat] Update rms_norm + fp4_qaunt kernel supporting more dim](../sources/prs/TensorRT-LLM/PR-13033.md), [[#12716][feat] Fused cross-head QK Norm + RoPE kernel for WAN](../sources/prs/TensorRT-LLM/PR-13052.md), [[None][feat] Optimize causal_conv1d prefill and decode kernels](../sources/prs/TensorRT-LLM/PR-13103.md), [[None][feat] Add FP4 residual quantization kernel without channel reo…](../sources/prs/TensorRT-LLM/PR-13117.md), [[https://nvbugs/5945047][fix] Fix cluster launch enablement for SM120 GPUs in allReduce fusion](../sources/prs/TensorRT-LLM/PR-13169.md), [[None][feat] Integrate FP4 indexer for DSA on Blackwell](../sources/prs/TensorRT-LLM/PR-13340.md), [[None][perf] Extend customMoeRouting kernel to support Qwen3.5](../sources/prs/TensorRT-LLM/PR-13433.md), [[None][perf] Scheme X L2-aware dispatcher and PDL launchers for sparse-attention GVR Top-K](../sources/prs/TensorRT-LLM/PR-13477.md), [[None][perf] Drop cubin and Eliminate ~6s FMHA JIT recompile in eager generation by aligning kernel selection with CUDA graph warmup](../sources/prs/TensorRT-LLM/PR-13505.md), [[None][feat] Fuse FP8 1x128 quantize + UE8M0 scale pack on SM100](../sources/prs/TensorRT-LLM/PR-13628.md), [[#13580][fix] AutoDeploy: Support Gemma3n/4 E2B variants](../sources/prs/TensorRT-LLM/PR-13630.md), [[None][feat] Add DeepSeekV4 attention kernels](../sources/prs/TensorRT-LLM/PR-13652.md), [[https://nvbugs/6108841][fix] add hidden_dim=6144 router GEMM instantiation for GLM-5](../sources/prs/TensorRT-LLM/PR-13740.md), [[None][perf] Optimize DeepSeek-V4 compressor BF16 input](../sources/prs/TensorRT-LLM/PR-13761.md), [[None][fix] Plumb swiglu_limit through DeepGEMM and TRTLLMGen FP8 fused MoE](../sources/prs/TensorRT-LLM/PR-13767.md), [[None][fix] Fix fused MHC for DeepSeek-V4-Pro hidden size](../sources/prs/TensorRT-LLM/PR-13771.md), [[None][feat] Indexer topk opt](../sources/prs/TensorRT-LLM/PR-13811.md), [[None][perf] mHC fused_hc kernel optimizations + DS-V4 entry-boundary RMSNorm fold-in](../sources/prs/TensorRT-LLM/PR-13892.md), [[None][perf] Add CUDA q_b norm for DeepSeek V4](../sources/prs/TensorRT-LLM/PR-13975.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [[None][feat] DSv4: enable GVR Heuristic Top-K for compress_ratio=4](../sources/prs/TensorRT-LLM/PR-14219.md), [[None][feat] Update the logic of FMHA JIT path](../sources/prs/TensorRT-LLM/PR-14291.md), [[None][chore] Update Claude Code agents and skills](../sources/prs/TensorRT-LLM/PR-14344.md), [feat: Add w4a8_mxfp4_fp8 quantization recipe.](../sources/prs/TensorRT-LLM/PR-4867.md), [[OMNIML-2336][feat] Add NVFP4 x FP8](../sources/prs/TensorRT-LLM/PR-6809.md), [[None][chore] Fix kernel launch param and add TRTLLM MoE backend test](../sources/prs/TensorRT-LLM/PR-7524.md), [[None][fix] Fix and add test for TRTLLM MoE backend](../sources/prs/TensorRT-LLM/PR-7755.md), [[TRTLLM-8637][feat] Optimize the routing kernel for DeepseekV3 (MoE CUTLASS backend); Add support for 384 experts (MoE TRTLLM backend)](../sources/prs/TensorRT-LLM/PR-7761.md), [[None][feat] GPT-OSS Sm120/Sm121 Support](../sources/prs/TensorRT-LLM/PR-7937.md), [[None][fix] Fix the performance issue of FP8 blockwise grouped GEMM when using attention DP](../sources/prs/TensorRT-LLM/PR-8501.md), [[None][feat] Enable nvfp4 cuda core for sm120](../sources/prs/TensorRT-LLM/PR-8620.md), [[None][feat] Update TRTLLM MoE cubins; reduce mxfp4 weight padding requirement; tighten TMA bound](../sources/prs/TensorRT-LLM/PR-9025.md), [[None][fix] support topk autotuner input for expert slot per group larger than 32](../sources/prs/TensorRT-LLM/PR-9087.md), [[None][feat] TRT-LLM Gen MoE optimize DeepSeek Fp8 activation kernel](../sources/prs/TensorRT-LLM/PR-9175.md), [[None][feat] add fp4 gemm + allreduce](../sources/prs/TensorRT-LLM/PR-9729.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [[None][feat] Fused kernels (qknormrope + moe routing) and two-model MTP support for glm4moe](../sources/prs/TensorRT-LLM/PR-9852.md), [[None][feat] Port fp4 quantization kernel optimization from FlashInfer](../sources/prs/TensorRT-LLM/PR-9854.md), [[None][feat] Adding torch ext API for FusedAddRMSNormQuant kernel](../sources/prs/TensorRT-LLM/PR-9905.md), [[TRTLLM-9493][feat] Add helixPostProcessNative kernel for cp_dim=2](../sources/prs/TensorRT-LLM/PR-9924.md), [Experimental Python cooperative algorithms](../sources/prs/cccl/PR-1973.md), [fix thread-reduce performance regression](../sources/prs/cccl/PR-2944.md), [Fix scan / sm90 perf regression ](../sources/prs/cccl/PR-3236.md), [Fix the vectorized loading of BlockLoad](../sources/prs/cccl/PR-3517.md), [Add b200 tunings for scan.exclusive.sum](../sources/prs/cccl/PR-3559.md), [Fix SM100 histogram tunings](../sources/prs/cccl/PR-3691.md), [Split Optimize Warp Reduce PR - CUB part](../sources/prs/cccl/PR-4716.md), [Add nondeterministic reduce that uses atomics](../sources/prs/cccl/PR-4961.md), [CUB - Add internal integer utils and tests (Split `WarpReduce` PR)](../sources/prs/cccl/PR-5314.md), [Combine `block_reduce_warp_reduction_nondeterministic.cuh` specialization with original deterministic one ](../sources/prs/cccl/PR-5408.md), [Add dynamic CUB dispatch for segmented_sort](../sources/prs/cccl/PR-6069.md), [[CUB] Use `BlockLoadToShared` in `DeviceMerge`](../sources/prs/cccl/PR-6077.md), [Fix debug section around line 390 of dispatch_topk](../sources/prs/cccl/PR-6152.md), [Split fixed-size segmented reduce dispatch header](../sources/prs/cccl/PR-6597.md), [Integrate decoupled lookahead warpspeed scan](../sources/prs/cccl/PR-6811.md), [Use integer promotion for `warp_reduce`](../sources/prs/cccl/PR-6819.md), [Implement new tuning API arch dispatching](../sources/prs/cccl/PR-7093.md), [Two-phase reduction for fixed size segmented reduction for very large segment sizes](../sources/prs/cccl/PR-7114.md), [Implement the new tuning API for deterministic (rfa) reduce dispatch](../sources/prs/cccl/PR-7346.md), [Radix-selection based `BlockTopK` specialization](../sources/prs/cccl/PR-7384.md), [Implement the new tuning API for `DeviceRleDispatch`](../sources/prs/cccl/PR-7669.md), [Optimize non fixed size segmented reduce for small segments using max_segment_size](../sources/prs/cccl/PR-7718.md), [Add env SegmentedReduce (non fixed-size overloads)](../sources/prs/cccl/PR-7795.md), [Forward policy hub from `dispatch_streaming_arg_reduce_t` to `reduce::dispatch`](../sources/prs/cccl/PR-7805.md), [Implement the new tuning API for `detail::reduce::dispatch_streaming_arg_reduce_t`](../sources/prs/cccl/PR-7807.md), [Use the new tuning API internally for `detail::transform::dispatch`](../sources/prs/cccl/PR-7810.md), [[Backport branch/3.3.x] Forward policy hub from `dispatch_streaming_arg_reduce_t` to `reduce::dispatch`](../sources/prs/cccl/PR-7814.md), [Optimized Device-to-Device Tensor Copy (`cudax`)](../sources/prs/cccl/PR-7823.md), [Implement the new tuning API for `DispatchSegmentedRadixSort`](../sources/prs/cccl/PR-7844.md), [Implement the new tuning API for `DispatchSegmentedSort`](../sources/prs/cccl/PR-7874.md), [Implement the new tuning API for `DispatchTopK`](../sources/prs/cccl/PR-7928.md), [[cuda.compute]: Fix faulty pointer arithmetic calculation in CUB dispatch](../sources/prs/cccl/PR-7940.md), [Reduce usage of `cub::DispatchReduce`](../sources/prs/cccl/PR-7944.md), [Use the new tuning API for `detail::radix_sort::dispatch`](../sources/prs/cccl/PR-7949.md), [Adds support for non-fundamental types via decomposer to `DeviceTopK` ](../sources/prs/cccl/PR-8040.md), [Optimized Device-to-Device Tensor Copy (cudax) - Transpose Case](../sources/prs/cccl/PR-8125.md), [Avoid passing uninitialized values to scan_op](../sources/prs/cccl/PR-8184.md), [[STF] Move unstable_unique from STF to generic cudax utility](../sources/prs/cccl/PR-8190.md), [Expose `max_segment_size` guarantee in cuda.compute](../sources/prs/cccl/PR-8284.md), [Port `thrust::min|max_element` to CUB](../sources/prs/cccl/PR-8291.md), [Implement the new tuning API for `DispatchSelectIf`](../sources/prs/cccl/PR-8311.md), [simplify dispatch segmented reduce to use latest dispatch and new tunings API](../sources/prs/cccl/PR-8332.md), [Apply some random warpspeed tunings](../sources/prs/cccl/PR-8352.md), [[cub]: implement utilities for policy selection](../sources/prs/cccl/PR-8355.md), [Replace `detail::merge::dispatch` by CUB's public API](../sources/prs/cccl/PR-8381.md), [[CUB] Replace `Shuffle(Up|Down|Index)` with cuda::device::warp_shuffle - RadixSort only](../sources/prs/cccl/PR-8395.md), [Vectorize mbarrier initialization in warpspeed scan](../sources/prs/cccl/PR-8423.md), [[thrust] Single-pass `is_partitioned` via adjacent zip_iterator](../sources/prs/cccl/PR-8427.md), [Replace `detail::merge_sort::dispatch` by CUB's public API](../sources/prs/cccl/PR-8473.md), [Replace `detail::scan::dispatch` by CUB's public API](../sources/prs/cccl/PR-8495.md), [Implement the new tuning API for `detail::batched_topk::dispatch_batched_topk`](../sources/prs/cccl/PR-8538.md), [Replace `detail::for_each::dispatch` by CUB's public API](../sources/prs/cccl/PR-8565.md), [Replace `detail::segmented_reduce::dispatch` by the public API](../sources/prs/cccl/PR-8695.md), [Use the new tuning API internally for `detail::topk::dispatch`](../sources/prs/cccl/PR-8742.md), [Use the new tuning API internally for `detail::reduce_by_key::dispatch`](../sources/prs/cccl/PR-8756.md), [Use the new tuning API internally for `detail::reduce[_nd]::dispatch[_nd]`](../sources/prs/cccl/PR-8826.md), [Fix Warpspeed scan shifted output store](../sources/prs/cccl/PR-8839.md), [[cub] Simplify arch dispatch](../sources/prs/cccl/PR-8861.md), [Use the new tuning API internally for `detail::select::dispatch` and `DeviceSelect`](../sources/prs/cccl/PR-8880.md), [[STF] Add per-handle exec_place stream resources](../sources/prs/cccl/PR-8905.md), [Use the new tuning API internally for `detail::select|three_way_partition::dispatch` and `DevicePartition`](../sources/prs/cccl/PR-8925.md), [Use the new tuning API internally for `detail::segmented_radix_sort::dispatch`](../sources/prs/cccl/PR-8927.md), [[libcu++] Always suppress C++ extensions warnings in prologue](../sources/prs/cccl/PR-9019.md), [Fix segmented radix sort benchmark segment size type](../sources/prs/cccl/PR-9039.md), [[libcu++] Fix default make_shared_resource construction](../sources/prs/cccl/PR-9044.md), [Vectorize contiguous iterators in `cub::BlockLoad`/`Store`](../sources/prs/cccl/PR-9056.md), [Improve sm90 mixed dtype kernel](../sources/prs/cutlass/PR-1883.md), [[EVT] Add support for Row/Col broadcast PtrArray](../sources/prs/cutlass/PR-2033.md), [Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2037.md), [Improvements for: Groupwise scaling along M for FP8 gemm](../sources/prs/cutlass/PR-2095.md), [Flash MLA support](../sources/prs/cutlass/PR-2130.md), [Flash MLA Support - Step 2](../sources/prs/cutlass/PR-2134.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Blockwise Improvement and Programmatic Dependent Launch](../sources/prs/cutlass/PR-2161.md), [Fix sm100 gemm wrong static constexpr that breaks compilation on Windows](../sources/prs/cutlass/PR-2167.md), [Fix SM90 beta=1 hang and stream-K launch errors](../sources/prs/cutlass/PR-2172.md), [Set EpiTile correctly when TileN is not divisible by 32](../sources/prs/cutlass/PR-2220.md), [Use cudaMemcpyAsync in gemm grouped with kRequiresPrecomputation sche…](../sources/prs/cutlass/PR-2256.md), [war to fix blackwell grouped groupwise hang](../sources/prs/cutlass/PR-2267.md), [hopper-blockwise-generalization-optimization](../sources/prs/cutlass/PR-2270.md), [Correct divmod order in example 77 (blackwell fmha)](../sources/prs/cutlass/PR-2291.md), [Handle get_masked_trip_count for small length in fmha example](../sources/prs/cutlass/PR-2292.md), [Fix epilogue::thread::Convert cannot be used with DefaultEpilogue](../sources/prs/cutlass/PR-2333.md), [[ex77] fix mla split; add fwd lse; add bwd varlen](../sources/prs/cutlass/PR-2366.md), [support fp16 accmulator for sm89 fp8 mma](../sources/prs/cutlass/PR-2378.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [fix: examples/cute/tutorial/blackwell/04_mma_tma_2sm_sm100.cu GridDim miscalculated](../sources/prs/cutlass/PR-2492.md), [DistGEMM bug fixes](../sources/prs/cutlass/PR-2713.md), [Support PDL for SM90 Array TMA GEMM](../sources/prs/cutlass/PR-2719.md), [Support for GEMM-K=0 for Blackwell Grouped GEMMs](../sources/prs/cutlass/PR-2746.md), [Blockscaled Ragged Contiguous Grouped Gemm for MoEs](../sources/prs/cutlass/PR-2790.md), [[Bug Fix]Bypass launch grids for SM120 Kernel with SM90 Mainloop & SM100 TileScheduler](../sources/prs/cutlass/PR-2865.md), [[cute] Add constexpr specifier to make_tiled_copy](../sources/prs/cutlass/PR-2875.md), [Fix incorrect tensor layout strides in Blackwell MMA tutorial comments](../sources/prs/cutlass/PR-2921.md), [[Cutlass gemm] Fix SM100 FP8 nosmem epilogue-fusion shape_div 'Divisibility Condition' for non-multiple-of-64 N tiles](../sources/prs/cutlass/PR-2946.md), [[Bug Fix]Set NumSplitsM to 1 when TileShapeM < 128 in sm90 fp8 blockwise scaling CollectiveMma](../sources/prs/cutlass/PR-2965.md), [Replace std::min with cute::min in sm120 blockwise scaling device functions](../sources/prs/cutlass/PR-3055.md), [Small Tile N BlockScaled GEMM + Grouped GEMM on SM12x](../sources/prs/cutlass/PR-3176.md), [Add Snake activation functor for EVT](../sources/prs/cutlass/PR-3184.md), [Fp8 kernel with "in-kernel" transpose of V in producer](../sources/prs/flash-attention/PR-1100.md), [FA3 kvcache + split kv + gqa parallelization](../sources/prs/flash-attention/PR-1236.md), [Fix FA3 Varlen Performance regression](../sources/prs/flash-attention/PR-1361.md), [Add sorting and head swizzle to varlen scheduler](../sources/prs/flash-attention/PR-1823.md), [feat: update decode attention APIs](../sources/prs/flashinfer/PR-1007.md), [misc: fix instrument code for mla profiler](../sources/prs/flashinfer/PR-1014.md), [add multi-item scoring](../sources/prs/flashinfer/PR-1015.md), [fix: add zero init for KV tiled copy](../sources/prs/flashinfer/PR-1029.md), [feat: add functional per-head FP8 quantization for FA3](../sources/prs/flashinfer/PR-1033.md), [feat: Softmax free sampling](../sources/prs/flashinfer/PR-1035.md), [[nvidia] initial support for blackwell kernels](../sources/prs/flashinfer/PR-1039.md), [fix: top_k_mask_logits hangs on -inf inputs](../sources/prs/flashinfer/PR-1050.md), [[nvidia] Add Blackwell FMHA decode kernel from TRT-LLM](../sources/prs/flashinfer/PR-1051.md), [Fix KV chunking for POD. ](../sources/prs/flashinfer/PR-1054.md), [bugfix: temporally disable split-kv in blackwell mla](../sources/prs/flashinfer/PR-1055.md), [Parameterize prefix mask call (needed by POD-Attention)](../sources/prs/flashinfer/PR-1059.md), [bugfix: adding lse output to blackwell fmha kernels](../sources/prs/flashinfer/PR-1071.md), [bugfix: follow user-specified sm_scale for blackwell cutlass fmha](../sources/prs/flashinfer/PR-1072.md), [perf: accelerate blackwell grouped gemm](../sources/prs/flashinfer/PR-1086.md), [bugfix: fix fp8 attention kernels aot compilation issue](../sources/prs/flashinfer/PR-1087.md), [comm: refactor and initialize `flashinfer.comm` module](../sources/prs/flashinfer/PR-1089.md), [feat: add trtllm all-reduce (non-MoE)](../sources/prs/flashinfer/PR-1096.md), [bugfix: host-precomuted plan function for blackwell fmha](../sources/prs/flashinfer/PR-1106.md), [feat: add trtllm moe_allreduce_fusion](../sources/prs/flashinfer/PR-1108.md), [Add CUTLASS fused moe kernels from TensorRT-LLM.](../sources/prs/flashinfer/PR-1113.md), [bugfix: Fix test and output shape of fp4 quantize](../sources/prs/flashinfer/PR-1114.md), [hotfix: fix the blackwell fmha stream](../sources/prs/flashinfer/PR-1116.md), [[Feature] Support PDL for batch Prefill and Decode](../sources/prs/flashinfer/PR-1117.md), [Fix pointer dtype bug in rope](../sources/prs/flashinfer/PR-1129.md), [feat: add trtllm all-reduce fusion](../sources/prs/flashinfer/PR-1131.md), [MNNVL MoE All-to-All Support](../sources/prs/flashinfer/PR-1134.md), [fix: negative zero by type trait --> binary value](../sources/prs/flashinfer/PR-1136.md), [[feat] add unified batch attention w/ correctness tests.](../sources/prs/flashinfer/PR-1137.md), [Fix FA2 and FA3 multi-item scoring and cuda illegal memory access error](../sources/prs/flashinfer/PR-1140.md), [feat: Fused temperature online softmax kernel](../sources/prs/flashinfer/PR-1153.md), [Add more logging to TRTLLM-GEN debug trace (NFC)](../sources/prs/flashinfer/PR-1158.md), [feat: add finalize_moe_allreduce from trtllm](../sources/prs/flashinfer/PR-1159.md), [feat: nvshmem python bindings](../sources/prs/flashinfer/PR-1160.md), [feat: update non-fused moe](../sources/prs/flashinfer/PR-1161.md), [feat: enable and update all-reduce fused quantization](../sources/prs/flashinfer/PR-1164.md), [bugfix: softmax NaN results caused by large -inf masks](../sources/prs/flashinfer/PR-1178.md), [update trtllm-gen decode attention kernel launcher](../sources/prs/flashinfer/PR-1189.md), [[feat] optimize persistent batch attention perf.](../sources/prs/flashinfer/PR-1200.md), [[fix] fix BatchAttention CTA_TILE_KV mask issue](../sources/prs/flashinfer/PR-1206.md), [Fix the issue with auxillary kernel launch and grid dim calculation](../sources/prs/flashinfer/PR-1208.md), [feat: trtllm-gen fp8 moe kernels](../sources/prs/flashinfer/PR-1212.md), [[comm] TRT-LLM's Multi-Node NVLink All-Reduce Kernel](../sources/prs/flashinfer/PR-1213.md), [Feature/sm100 low latency nvfp4 kernels](../sources/prs/flashinfer/PR-1214.md), [Enable cudnn decode and add tests for the cudnn decode kernel](../sources/prs/flashinfer/PR-1221.md), [feat: add trtllm-gen mla cubin](../sources/prs/flashinfer/PR-1222.md), [Fix missing hash in the cudnn cubin path](../sources/prs/flashinfer/PR-1227.md), [feat: Add non-causal cudnn prefill kernels](../sources/prs/flashinfer/PR-1230.md), [bugfix: support uint8_t for vec_t class template](../sources/prs/flashinfer/PR-1234.md), [add trtllm-gen context attention](../sources/prs/flashinfer/PR-1239.md), [Patch fp8 cubin availability](../sources/prs/flashinfer/PR-1240.md), [feat: Support MXFP8 x MXFP4 CUTLASS grouped GEMM](../sources/prs/flashinfer/PR-1241.md), [Add trtllm-gen attention mha kernel with FP8 Q/K/V and FP8 output](../sources/prs/flashinfer/PR-1242.md), [Remove sm100+ requirment for trtllm allreduce kernels](../sources/prs/flashinfer/PR-1249.md), [Reduce the JIT compilation time of gen_gemm_sm100_module](../sources/prs/flashinfer/PR-1251.md), [TRT-LLM's Multi-Node NVLink AR + fused RMSNorm kernel](../sources/prs/flashinfer/PR-1255.md), [feat: enable trtllm-gen mla MTP](../sources/prs/flashinfer/PR-1258.md), [Made AR output optional + esthetic changes](../sources/prs/flashinfer/PR-1265.md), [Bug fix: fix duplicate launch in POD](../sources/prs/flashinfer/PR-1267.md), [Add shuffle matrix flag](../sources/prs/flashinfer/PR-1272.md), [Convert scale_factor from scalar to Tensor in trt_allreduce_fusion](../sources/prs/flashinfer/PR-1284.md), [fix multiCtasKvScratchPtr misalignment issue (new one)](../sources/prs/flashinfer/PR-1286.md), [Bug fix: guard fp8 e8m0 and e2m1 compile ](../sources/prs/flashinfer/PR-1287.md), [refactor: refactor trtllm-gen attention kernel integration code](../sources/prs/flashinfer/PR-1289.md), [[fix] fix integer overflow in FA2 customized_mask & add buffer overflow warning.](../sources/prs/flashinfer/PR-1290.md), [refactor: Improved metainfo for trtllm-gen fmha](../sources/prs/flashinfer/PR-1292.md), [Update cutlass fp4 moe kernels](../sources/prs/flashinfer/PR-1294.md), [add cutlass backend for mm_fp4](../sources/prs/flashinfer/PR-1296.md), [feat: Add weight layout option for trtllm-gen fused moe](../sources/prs/flashinfer/PR-1297.md), [perfix: use lightweight API to query device property](../sources/prs/flashinfer/PR-1298.md), [[Feature] SM level profiler ](../sources/prs/flashinfer/PR-1305.md), [Fix the bug of the kernel-selection heuristic in trtllm-gen](../sources/prs/flashinfer/PR-1307.md), [Refactor Fused Moe Module](../sources/prs/flashinfer/PR-1309.md), [feat: support output nvfp4 in trtllm-gen function call.](../sources/prs/flashinfer/PR-1318.md), [Make Fp8 MoE routing_bias optional](../sources/prs/flashinfer/PR-1319.md), [Add blockwise-scaled FP8 GEMM via TRTLLM-Gen.](../sources/prs/flashinfer/PR-1320.md), [Optimizations for TRTLLM MNNVL Allreduce](../sources/prs/flashinfer/PR-1321.md), [feat: Add k_scale and v_scale to persistent attention ](../sources/prs/flashinfer/PR-1322.md), [feat: Support logits_soft_cap for Persistent attn; fix kv split limit](../sources/prs/flashinfer/PR-1324.md), [feat: Fused rope fp8 quantize kernel for MLA](../sources/prs/flashinfer/PR-1339.md), [fix: fix trtllm-gen mla error on new interface](../sources/prs/flashinfer/PR-1348.md), [feature: add fp4 mm using trtllm backend](../sources/prs/flashinfer/PR-1355.md), [support trtllm-gen prefill fp4 output](../sources/prs/flashinfer/PR-1360.md), [Support scale factor start index for fp4 mha prefill/decode](../sources/prs/flashinfer/PR-1363.md), [bugfix: fixed cutlass fused moe usage of FP4QuantizationSFLayout::SWIZZLED](../sources/prs/flashinfer/PR-1371.md), [bugfix: Add guard for fp4/fp8 related include headers](../sources/prs/flashinfer/PR-1376.md), [GPT-OSS Support: Add Blackwell MoE mxfp4 implementation from TRTLLM and Attention Sink](../sources/prs/flashinfer/PR-1389.md), [gpt-oss: Add MXFP8 x MXFP4 CUTLASS MOE for SM100 and BF16 x MXFP4 CUTLASS for SM90 + SwigluBias Activation](../sources/prs/flashinfer/PR-1396.md), [feature: add cutlass as bmm_fp8 backend.](../sources/prs/flashinfer/PR-1397.md), [Fix trtllm moe launcher local_num_experts](../sources/prs/flashinfer/PR-1398.md), [fix shared memory alignment conflict in sampling.cuh](../sources/prs/flashinfer/PR-1402.md), [[bugfix] Fix compilation failure when compiling csrc/trtllm_moe_allreduce_fusion.cu](../sources/prs/flashinfer/PR-1410.md), [Fixes for Blackwell Tests](../sources/prs/flashinfer/PR-1434.md), [fix: remote redundant zero_init from trtllm-gen attn](../sources/prs/flashinfer/PR-1444.md), [Add alignment in MxFP8Quantization](../sources/prs/flashinfer/PR-1445.md), [Remove getEnvEnablePDL in favor of enable_pdl parameter](../sources/prs/flashinfer/PR-1446.md), [bugfix: Verify num_experts greater or equal to local_experts + offset](../sources/prs/flashinfer/PR-1469.md), [perf: add 1x4x1 cluster shape for fp8 bmm M<16 cases](../sources/prs/flashinfer/PR-1473.md), [tuner: Trtllm-gen Fp4 MoE Autotunner](../sources/prs/flashinfer/PR-1475.md), [perf: add fast path to TopPRenormProbKernel for top_p >= 1.0, significantly boosting SGLang workloads](../sources/prs/flashinfer/PR-1483.md), [feat: add pdl for trtllm-gen attn](../sources/prs/flashinfer/PR-1484.md), [feat: Support fp8 qkv, fp16/bf16 out MHA for trtllm-gen.](../sources/prs/flashinfer/PR-1490.md), [Perf: support scale_a/scale_b instead of combined scale in cutlass bmm_fp8](../sources/prs/flashinfer/PR-1491.md), [fix: Replace cub Max/Min with cuda::maximum/minimum for cuda 13 compatibility](../sources/prs/flashinfer/PR-1500.md), [feat: integrate xqa attention backend](../sources/prs/flashinfer/PR-1503.md), [update allreduce to match trtllm](../sources/prs/flashinfer/PR-1507.md), [Support cuda<12.8 built for trtllm_allreduce_fusion.](../sources/prs/flashinfer/PR-1508.md), [backend: Refactor trtllm-gen fmha metainfo loading](../sources/prs/flashinfer/PR-1518.md), [Fix linking errors with CUDA 13](../sources/prs/flashinfer/PR-1523.md), [Add GeGLU support to trtllm-gen NVFP4 Fused MoE Kernel](../sources/prs/flashinfer/PR-1525.md), [bugfix: Fix compile error for undefined swizzle enum.](../sources/prs/flashinfer/PR-1530.md), [bugfix: Fix Persistent kernel precision for masked output ](../sources/prs/flashinfer/PR-1533.md), [Remove cuda-python from dependency and check at runtime](../sources/prs/flashinfer/PR-1534.md), [feat: Integrate TRTLLM varlen kernel for deepseek R1 prefill ](../sources/prs/flashinfer/PR-1537.md), [fix trtllm_allreduce_fusion twoshot register problem.](../sources/prs/flashinfer/PR-1545.md), [perf: replace cudaGetDeviceProperties with cudaDeviceGetAttribute](../sources/prs/flashinfer/PR-1547.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [Add mnnvl_moe_alltoallv_prepare_without_allgather](../sources/prs/flashinfer/PR-1550.md), [bugfix: fix persistent attention kernel correctness on blackwell](../sources/prs/flashinfer/PR-1559.md), [fix: separate out fp4 lib into sm90 and sm100 versions, add oob checking in fused moe](../sources/prs/flashinfer/PR-1565.md), [Backend: downgrade trtllm-gen kernel to cuda-12](../sources/prs/flashinfer/PR-1567.md), [bugfix: fix cuda version guard macros](../sources/prs/flashinfer/PR-1571.md), [update trtllm-gen fp4 autotuner and routing](../sources/prs/flashinfer/PR-1573.md), [bugfix: update trtllm-gen gemm kernel names](../sources/prs/flashinfer/PR-1577.md), [bugfix: Fix arg passing to TORCH_CHECK and TORCH_WARN macros](../sources/prs/flashinfer/PR-1582.md), [fix: semaphoress must be at the fixed range in workspace buffer on trtllm_gen attention](../sources/prs/flashinfer/PR-1584.md), [bugfix: fix fused-temperature softmax IMA issue](../sources/prs/flashinfer/PR-1596.md), [bugfix: fix the register overflow issue for topk renorm kernels on blackwell](../sources/prs/flashinfer/PR-1597.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [feat: cutlass fp4 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1609.md), [feat: cutlass fp8 gemm bringup for SM120 & SM121](../sources/prs/flashinfer/PR-1610.md), [bugfix: fix fp4 quantization with 8x4 scale factor layout](../sources/prs/flashinfer/PR-1611.md), [bugfix: fix merge_attention_state in BatchAttention w/ gqa-group-size in Qwen family](../sources/prs/flashinfer/PR-1614.md), [perf: Fix the tactic sorting in TrtllmGenBatchedGemmRunner::getValidConfigIndices](../sources/prs/flashinfer/PR-1615.md), [bugfix: collect all modules to aot](../sources/prs/flashinfer/PR-1622.md), [bugfix: trtllm-gen fmha sm101 and sm100 compatibility](../sources/prs/flashinfer/PR-1631.md), [perf&bugfix: skip kv-tile computation out of sliding window in FA2; fix __syncthreads in mergestate](../sources/prs/flashinfer/PR-1661.md), [Refactor Blackwell unit test scripts](../sources/prs/flashinfer/PR-1667.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [feat: Add `variant.OutputTransform()` to decode kernels](../sources/prs/flashinfer/PR-1670.md), [feat: Batch-size invariant FA2 Prefill & Decode](../sources/prs/flashinfer/PR-1675.md), [perf: improve attention of tcgen05 flash-attention](../sources/prs/flashinfer/PR-1681.md), [Update TGV GEMM default kernel and TGV code cleanup.](../sources/prs/flashinfer/PR-1682.md), [perf: Port the separate reduce kernel mode from trtllm.](../sources/prs/flashinfer/PR-1685.md), [Support Kimi-K2 for TRT: templatize number of experts](../sources/prs/flashinfer/PR-1696.md), [Fix DeepSeek quality for TRTLLM fused MoE routing](../sources/prs/flashinfer/PR-1723.md), [bugfix: partially fix tests/test_trtllm_gen_fused_moe.py unit test failure](../sources/prs/flashinfer/PR-1724.md), [TVM: support TVM binding for GroupedGemm](../sources/prs/flashinfer/PR-1725.md), [fix: put sampling kernel launch into macro](../sources/prs/flashinfer/PR-1727.md), [fix: fix cannot import name 'cuda' from 'cuda' in CUDA13](../sources/prs/flashinfer/PR-1764.md), [feat: add xqa fp8 mha and fp8 kv cache](../sources/prs/flashinfer/PR-1769.md), [Masked batch nvfp4 quantization](../sources/prs/flashinfer/PR-1774.md), [feat:enable fp8 blockscale moe for fused cultass for sm90](../sources/prs/flashinfer/PR-1819.md), [Bugfix: Fix data hazard in persistent reduce](../sources/prs/flashinfer/PR-1826.md), [feat: trtrllm-gen global scaled FP8 GEMMs](../sources/prs/flashinfer/PR-1829.md), [Update the routing for TRTLLMGEN to support kimi k2 and qwen](../sources/prs/flashinfer/PR-1831.md), [[Quantization] Add per-expert global scaling factor for fp4 batched quantize](../sources/prs/flashinfer/PR-1835.md), [Add head_dim=64 for tcgen05 tcgen05 flash-attention implementation](../sources/prs/flashinfer/PR-1850.md), [Bugfix: fix o_strides in persistent kernel ](../sources/prs/flashinfer/PR-1865.md), [Tune kernel compilation parameters for https://github.com/flashinfer-ai/flashinfer/pull/1850 ](../sources/prs/flashinfer/PR-1878.md), [feat: Add FP4 TRTLLM-Gen throughput MOE batched gemms](../sources/prs/flashinfer/PR-1882.md), [MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA](../sources/prs/flashinfer/PR-1924.md), [Add layernorm op for inputs of mixed dtype](../sources/prs/flashinfer/PR-1926.md), [silu_and_mul nvfp4 quanization fusion rework](../sources/prs/flashinfer/PR-1927.md), [Feature: Support Relu2 activation in fused MoE](../sources/prs/flashinfer/PR-1954.md), [Update trtllm-gen fused moe routing kernel and add more kernels](../sources/prs/flashinfer/PR-1955.md), [Fix: Verify scales are not None for Cutlass FP8 FusedMoE](../sources/prs/flashinfer/PR-1961.md), [feat: enable deepgemm jit for fp8 block-scale on SM90](../sources/prs/flashinfer/PR-1969.md), [feat: autotune tile_tokens_dim in trtllm-gen MOE](../sources/prs/flashinfer/PR-1980.md), [fix: correct PDL parameter handling in RopeQuantize kernel](../sources/prs/flashinfer/PR-1982.md), [minor fix for xqa](../sources/prs/flashinfer/PR-1994.md), [Bugfix: Change get() -> GetDLTensorPtr() in cutlass FusedMoE validations](../sources/prs/flashinfer/PR-1995.md), [feat: add xqa backend and completes NHD/HND coverage for trtllm-gen/xqa backend](../sources/prs/flashinfer/PR-2001.md), [Feature: Support non-gated activation in cutlass fused MoE nvfp4](../sources/prs/flashinfer/PR-2011.md), [[feat] Refactor trtllmgen MOE and add Bf16 trtllmgen moe](../sources/prs/flashinfer/PR-2014.md), [[DSV3] Optimized Router Gemm](../sources/prs/flashinfer/PR-2019.md), [update trtllm cutlass moe ](../sources/prs/flashinfer/PR-2020.md), [perf: Speed up fp4 quantization for small batch with swizzling for cutlass MoE](../sources/prs/flashinfer/PR-2025.md), [[NVIDIA] Thor & Spark Support](../sources/prs/flashinfer/PR-2028.md), [Enable renormalize(naive) routing for fp8 per-tensor](../sources/prs/flashinfer/PR-2030.md), [use scalar for kv_scale in xqa](../sources/prs/flashinfer/PR-2033.md), [feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA) ](../sources/prs/flashinfer/PR-2037.md), [perf: improve sampling/mask/softmax performance (part 1/2)](../sources/prs/flashinfer/PR-2044.md), [Rebase FP8 SM100 Cutlass FMHA Attention to main (original PR#1238)](../sources/prs/flashinfer/PR-2047.md), [Fix dtype of output scales from mnnvl_moe_alltoallv_prepare_without_allgather](../sources/prs/flashinfer/PR-2048.md), [[BUG] Fix trtllm-gen fp4 moe renormalize routing](../sources/prs/flashinfer/PR-2049.md), [Add support for topkPacked input in block-level renormalize](../sources/prs/flashinfer/PR-2051.md), [feat: add xqa mla backend](../sources/prs/flashinfer/PR-2053.md), [perf: Optimize helper max/minmax function in sampling.cuh](../sources/prs/flashinfer/PR-2058.md), [Fix moe fp8 failure for sm121](../sources/prs/flashinfer/PR-2061.md), [Fix: several bugs/issues with trtllm-gen attention kernels. ](../sources/prs/flashinfer/PR-2062.md), [perf: TRT-LLM MoE Block-FP8 activation optimization](../sources/prs/flashinfer/PR-2063.md), [feat: BF16 GEMM using CUTLASS backend for SM100](../sources/prs/flashinfer/PR-2070.md), [[Feature] Support batch prefill for POD Attention](../sources/prs/flashinfer/PR-2079.md), [enable xqa fp8 output](../sources/prs/flashinfer/PR-2081.md), [[API change] Allow using torch.Tensor for scales for trtllm-gen attention](../sources/prs/flashinfer/PR-2084.md), [refactor: pass hopper deepgemm include directory through python](../sources/prs/flashinfer/PR-2090.md), [perf: TRT-LLM Gen finalize kernel optimization](../sources/prs/flashinfer/PR-2092.md), [perf: enable pdl for cutlass fp4 gemm](../sources/prs/flashinfer/PR-2095.md), [Port TRT-LLM communication kernels to flashinfer](../sources/prs/flashinfer/PR-2102.md), [enable xqa speculative decoding](../sources/prs/flashinfer/PR-2105.md), [feat: support more head dim in RoPE kernel](../sources/prs/flashinfer/PR-2109.md), [add tensor scale input for xqa](../sources/prs/flashinfer/PR-2110.md), [refactor: update fa3 codebase and fix hopper unittest [part 1]](../sources/prs/flashinfer/PR-2111.md), [feature: make the LSE returned by MLA support base 2 or e #2113](../sources/prs/flashinfer/PR-2114.md), [update xqa license](../sources/prs/flashinfer/PR-2117.md), [Refactor trtllm_mnnvl_allreduce](../sources/prs/flashinfer/PR-2118.md), [perf: bunch of features and optimizations for top-k (sampling + sparse attention)](../sources/prs/flashinfer/PR-2119.md), [feat: support variable sequence length in decode kernel of trtllm-gen attention](../sources/prs/flashinfer/PR-2125.md), [fix flaky xqa test](../sources/prs/flashinfer/PR-2126.md), [make DeepGEMM swapAB available for linear gemm SM90](../sources/prs/flashinfer/PR-2131.md), [feat: add trtllm-gen per-tensor sparseMla kernels.](../sources/prs/flashinfer/PR-2138.md), [feat: TRTLLM FMHAv2 backend for ctx attention](../sources/prs/flashinfer/PR-2142.md), [fix xqa mha_sm90.cu](../sources/prs/flashinfer/PR-2157.md), [feat: MxInt4 x Bf16 TRT-LLM Gen MoE support](../sources/prs/flashinfer/PR-2159.md), [Add data type check for deepseek fp4 moe](../sources/prs/flashinfer/PR-2165.md), [Fix for moe on sm110](../sources/prs/flashinfer/PR-2190.md), [feat: unit-test and api change, w4a8 grouped-gemm fused MoE for SM90](../sources/prs/flashinfer/PR-2193.md), [Move the run function definition out of BatchedGemmInterface](../sources/prs/flashinfer/PR-2211.md), [feat: further optimize top-k and add fused top-k page construction kernels for DSA](../sources/prs/flashinfer/PR-2215.md), [feat: Support unpadded output hidden size for trtllm_fp4_block_scale_moe](../sources/prs/flashinfer/PR-2217.md), [fix: add DeepSeek routing for Bf16xBf16 and MxIntxBf16 TRT-LLM Gen MoE](../sources/prs/flashinfer/PR-2234.md), [refactor: pull trtllm-gen batch-gemm/gemm headers from artifactory; update tma descriptor shape init](../sources/prs/flashinfer/PR-2235.md), [[feat] Integrate SGLang concat_mla_k kernel into flashinfer](../sources/prs/flashinfer/PR-2237.md), [feat: RMSNorm/Fused RMSNorm + FP8 Quantization kernels](../sources/prs/flashinfer/PR-2243.md), [Remove cudaStreamSynchronize from gemm_groupwise_sm120.cuh for CUDA graph compatibility](../sources/prs/flashinfer/PR-2244.md), [feat: Support numLocalTokens=0 for moe All-to-all](../sources/prs/flashinfer/PR-2247.md), [feat: support non-contiguous query for trtllm-gen attention backend](../sources/prs/flashinfer/PR-2254.md), [fix: support int64 IdType for RoPE part argument in `rope_quantize_fp8_append_paged_kv_cache`](../sources/prs/flashinfer/PR-2255.md), [[TRTLLM-Gen Fmha] add optimized trtllm-gen decode kernels for high throughput + speculative decoding](../sources/prs/flashinfer/PR-2265.md), [[performance]optimize for nvfp4](../sources/prs/flashinfer/PR-2268.md), [feat: add GDN Attention](../sources/prs/flashinfer/PR-2276.md), [feat: IdType indices in sampling kernels](../sources/prs/flashinfer/PR-2281.md), [Selective State Update kernel (mamba)](../sources/prs/flashinfer/PR-2301.md), [[Perf][Feature] Add SM103-specific schedulers for NVFP4 CUTLASS kernels](../sources/prs/flashinfer/PR-2303.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron](../sources/prs/flashinfer/PR-2304.md), [Fix: FilteredTopKUnifiedKernel read value out of length](../sources/prs/flashinfer/PR-2308.md), [[ML3] Optimized Router Gemm](../sources/prs/flashinfer/PR-2323.md), [bugfix: fix multi-cta top-k implementation when k value is different for different row](../sources/prs/flashinfer/PR-2325.md), [[perf] Improve gemm_fp8_nt_groupwise (cutlass backend) by 10-40% for batch sizes <= 32](../sources/prs/flashinfer/PR-2327.md), [fix: guard batchWarpReduceSum with ENABLE_FP8 to fix compilation without FP8](../sources/prs/flashinfer/PR-2328.md), [feat: expose swizzled_input_sf parameter for CUTLASS fused MOE](../sources/prs/flashinfer/PR-2330.md), [Optimize quantization function in large problem size](../sources/prs/flashinfer/PR-2343.md), [Enable fp16/bf16/f32 support for selective_state_update (mamba)](../sources/prs/flashinfer/PR-2366.md), [bugfix: hotfix of PR 2366 (mamba kernel)](../sources/prs/flashinfer/PR-2378.md), [fix: ensure each CTA processes full numHeadsQPerKv for trtllm decode kernel](../sources/prs/flashinfer/PR-2380.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [feat: cuteDSL fp4 moe for better DSR1 performance.](../sources/prs/flashinfer/PR-2398.md), [Remove cudaMalloc/Free in GDN prefill kernel](../sources/prs/flashinfer/PR-2415.md), [feat: update trtllm-gen MoE cubins](../sources/prs/flashinfer/PR-2416.md), [refactor: reduce hopper's gdn prefill compilation time and fix docstring.](../sources/prs/flashinfer/PR-2422.md), [refactor: refactoring cuda code to cute-dsl (part 1)](../sources/prs/flashinfer/PR-2428.md), [fix: Sampling: CUDA Graph fix](../sources/prs/flashinfer/PR-2432.md), [fix: Fix NaN output in mxfp8_quantize for very small input values](../sources/prs/flashinfer/PR-2441.md), [MTP for mamba ](../sources/prs/flashinfer/PR-2444.md), [bugfix: fix stub generation directory in fused_moe module](../sources/prs/flashinfer/PR-2445.md), [feat: Add TRTLLM fmha_v2 library for SM90 attention with Skip-Softmax ](../sources/prs/flashinfer/PR-2446.md), [fix: fix illegal memory access for NaN input in sampling kernels](../sources/prs/flashinfer/PR-2456.md), [feat: Support Fused MoE non gated Relu2 NVFP4 & FP8 and support Nemotron, fixed](../sources/prs/flashinfer/PR-2462.md), [feat: Add MXFP8 GEMM mm_mxfp8 (cutlass)](../sources/prs/flashinfer/PR-2464.md), [feat: Add TRTLLM-Gen Skip-Softmax kernels for prefill and decode](../sources/prs/flashinfer/PR-2477.md), [fix: add support check for gemm config for cutlass moe](../sources/prs/flashinfer/PR-2495.md), [Feat: Trtllm-gen MxFP8 MoE integration](../sources/prs/flashinfer/PR-2505.md), [perf: cache cudaGetDeviceProperties in gdn_prefill to avoid per-call overhead](../sources/prs/flashinfer/PR-2509.md), [Support NVFP4 KV cache decode on SM120](../sources/prs/flashinfer/PR-2520.md), [misc: point triton blackwell-ptxas to local cuda ptxas](../sources/prs/flashinfer/PR-2543.md), [feat: Enable TRTLLM-Gen Skip-Softmax attention for MLA](../sources/prs/flashinfer/PR-2547.md), [[Bugfix][comm] Fix FP4 one-shot launch config instability in trtllm_allreduce_fusion](../sources/prs/flashinfer/PR-2557.md), [Add support for the combinations of allreduce, allgather, and reducescatter](../sources/prs/flashinfer/PR-2563.md), [fix: W4A8 autotune crash in cutlass_fused_moe profiler workspace](../sources/prs/flashinfer/PR-2564.md), [Implement `cutlass_fused_moe` mxfp8](../sources/prs/flashinfer/PR-2581.md), [feat: trtllm tinygemm2 in flashinfer as bf16 routergemm](../sources/prs/flashinfer/PR-2587.md), [Mamba SSU: better automatic kernel selection + algorithm selection optionally exposed to the user.](../sources/prs/flashinfer/PR-2591.md), [[bugfix] Fix FilteredTopK overflow correctness](../sources/prs/flashinfer/PR-2605.md), [fix: cute dsl nvfp4 moe routing index error](../sources/prs/flashinfer/PR-2629.md), [fix: add SM121 support to SM120 version guards](../sources/prs/flashinfer/PR-2631.md), [[fp8_blockwise]Fix int32 overflow in TRTLLM fused MoE activation kernel](../sources/prs/flashinfer/PR-2642.md), [feat: FP32 dtype output for BF16 matmuls (CUTLASS & cuDNN)](../sources/prs/flashinfer/PR-2644.md), [int16 Block-Scaled State and Stochastic Rounding for SSU (mamba)](../sources/prs/flashinfer/PR-2645.md), [[feat] trtllm-gen mxfp8 gemm](../sources/prs/flashinfer/PR-2653.md), [feat: implement deterministic topk](../sources/prs/flashinfer/PR-2661.md), [perf: Update trtllm-gen batched GEMM kernels - faster, more NVFP4 tile dims, MXFP8 with relu2 act](../sources/prs/flashinfer/PR-2667.md), [fix: reduce smem allocation for tinygemm2 kernel in SM120](../sources/prs/flashinfer/PR-2670.md), [feat: add support for more MLA head dimensions](../sources/prs/flashinfer/PR-2677.md), [Add varlen and speculative decoding support to selective state update](../sources/prs/flashinfer/PR-2700.md), [Add NVFP4 KV cache quantization support for SM100](../sources/prs/flashinfer/PR-2702.md), [feat: Add support for TRTLLM MXFP8 non-gated MoE with ReLU2](../sources/prs/flashinfer/PR-2707.md), [Mamba2 SSD Combined Forward Pass (Blackwell CuTe DSL Kernel)](../sources/prs/flashinfer/PR-2709.md), [feat: Add DiT-oriented kernels where Qk (Bmm1) type can be reinterpreted into Int8 or BFloat16](../sources/prs/flashinfer/PR-2711.md), [fix: Add SM120 (RTX Blackwell desktop) support for NVFP4 MoE kernels](../sources/prs/flashinfer/PR-2725.md), [Support for MXFP4 and NVFP4 group GEMMs on GeForce and Spark](../sources/prs/flashinfer/PR-2738.md), [misc: Update gemm/batched gemm cubins from trtllm-gen, gemm header refactor](../sources/prs/flashinfer/PR-2740.md), [[feat] Add 2048 experts and 32 Top K ](../sources/prs/flashinfer/PR-2744.md), [[feat] Add air top-p algorithm](../sources/prs/flashinfer/PR-2752.md), [feat: Add FP4 KV cache quant/dequant kernels ](../sources/prs/flashinfer/PR-2757.md), [feat: Expose TRT-LLM FMHA style paged KV Cache and page table layout](../sources/prs/flashinfer/PR-2770.md), [feat: FP8 output support for CUTLASS MLA paged attention](../sources/prs/flashinfer/PR-2779.md), [feat: Support padding tokens with seqlen=0 for rope+quant+kv cache update fusion kernel](../sources/prs/flashinfer/PR-2792.md), [Upgrade cutlass 4.2.1 -> 4.4.2](../sources/prs/flashinfer/PR-2798.md), [[fmha-v2] Support HND and NHD paged KV cache layouts with conditional stride handling](../sources/prs/flashinfer/PR-2799.md), [fix: Autotuner _find_nearest_profile non-power-of-2 num_tokens, create launchers for all supported tileN in trtllm fused MoE](../sources/prs/flashinfer/PR-2821.md), [[Fmha] Sparse MLA decode kernel selection heuristics](../sources/prs/flashinfer/PR-2836.md), [[Perf] Add FMHAv2 to flashinfer_benchmark.py and eliminate unnecessary H2D](../sources/prs/flashinfer/PR-2841.md), [read real strides for kv and block scale](../sources/prs/flashinfer/PR-2844.md), [fix: int32 overflow in `trtllm_fp4_block_scale_moe` causing "Unsupported hidden state scale shape" for EP32+ configs](../sources/prs/flashinfer/PR-2853.md), [Add support for Relu2 in BF16 fused MoE](../sources/prs/flashinfer/PR-2864.md), [Mamba SSU: horizontal MTP kernel (+ DSTATE=96 support)](../sources/prs/flashinfer/PR-2865.md), [Fix silent bug with FP8 per tensor non-gated MoE](../sources/prs/flashinfer/PR-2882.md), [fix: snap weight_scale_vec_size to handle block_scale_interleave padding for SM120](../sources/prs/flashinfer/PR-2898.md), [feat: add MXFP8 GEMM support for SM120](../sources/prs/flashinfer/PR-2902.md), [feat(gdn): state checkpointing in chunk_gated_delta_rule](../sources/prs/flashinfer/PR-2908.md), [feat: Add cuBLASLt backend for `mm_bf16` and enable multi-tactic autotuning for FP8/MXFP8 runners](../sources/prs/flashinfer/PR-2914.md), [feat: add Relu2 (squared ReLU) activation support in CUTLASS MoE backend](../sources/prs/flashinfer/PR-2926.md), [fix: use float instead of double in sampling binary search to avoid FP64 bottleneck on SM103](../sources/prs/flashinfer/PR-2945.md), [Update NVSHMEM interface to use NVSHMEM4Py instead of custom bindings](../sources/prs/flashinfer/PR-2960.md), [Improved `simple` mamba SSU kernel ](../sources/prs/flashinfer/PR-2962.md), [test: xfail cuDNN FP8 prefill on Blackwell with CUDA <= 12.9](../sources/prs/flashinfer/PR-2963.md), [Add flashinfer.fused_rmsnorm_silu() with native kernel backend](../sources/prs/flashinfer/PR-2965.md), [Fused moe all-reduce routed scaling factor + quant support](../sources/prs/flashinfer/PR-2966.md), [fix: restore SM120 CUTLASS MoE tile candidate removed by #2927 (test_trtllm_cutlass_fused_moe.py)](../sources/prs/flashinfer/PR-2984.md), [[Fmha] support nvfp4 output keepsMmaAb generation kernels](../sources/prs/flashinfer/PR-2988.md), [fix: tinygemm2 hang issue due to barrier sync](../sources/prs/flashinfer/PR-2996.md), [perf: Optimize CUTLASS MoE helper kernels for small-batch decode workloads](../sources/prs/flashinfer/PR-3014.md), [fix: extend moe alltoall top-k specializations](../sources/prs/flashinfer/PR-3021.md), [[feat] Add routing_replay_out support to MoE kernels and Python API](../sources/prs/flashinfer/PR-3024.md), [perf: Port TRT-LLM SM120/SM121 FP4 CUTLASS GEMM optimizations. Add PDL](../sources/prs/flashinfer/PR-3026.md), [[feat] Trtllm-gen Per-token Nvfp4 MoE](../sources/prs/flashinfer/PR-3027.md), [fused_moe: pre-filter SM89 tactics with zero occupancy on SM120 Blackwell (fix review feedback on #2764)](../sources/prs/flashinfer/PR-3032.md), [Support lse in trtllm paged attn kernels](../sources/prs/flashinfer/PR-3058.md), [Support Allreduce + Norm + Per-token Group Fp8 Quant Fusion](../sources/prs/flashinfer/PR-3059.md), [Support NVFP4 KV for prefill and batch attention kernels](../sources/prs/flashinfer/PR-3097.md), [feat: Enable FP8 (E4M3/E5M2) in concat_mla_k for optimize long-context prefill performance and refactor type dispatch for BF16/FP16](../sources/prs/flashinfer/PR-3129.md), [perf: Add no-bias path for tinygemm_bf16](../sources/prs/flashinfer/PR-3151.md), [Integrate CUTLASS Small Tile N Blockscaled GEMMs/Grouped GEMMs for SM120 and SM121](../sources/prs/flashinfer/PR-3152.md), [feat: DiT layer norm fusions for WAN: flashinfer.diffusion_ops](../sources/prs/flashinfer/PR-3157.md), [feat: enable glm5 router gemm](../sources/prs/flashinfer/PR-3185.md), [[Bugfix] Fix fused MoE autotuning correctness issues by filtering clusterDimZ](../sources/prs/flashinfer/PR-3227.md), [perf: optimize per-token nvfp4 quantization kernel.](../sources/prs/flashinfer/PR-3237.md), [Update moe gemm](../sources/prs/flashinfer/PR-3239.md), [Add dynamic tokens-per-page TRTLLM-GEN GQA kernels](../sources/prs/flashinfer/PR-3259.md), [Update trtllm FMHA cubins](../sources/prs/flashinfer/PR-3317.md), [[feat] Add gemma RMS AR fusion](../sources/prs/flashinfer/PR-3322.md), [checkpointing_ssu kernel: fused replay + conditional state-write for Mamba2](../sources/prs/flashinfer/PR-3324.md), [feat(cute_dsl/moe): add `moe_output_memset_inplace` dense memset wrapper](../sources/prs/flashinfer/PR-3328.md), [perf: fix the iteration bound of SWA in FA2 prefill template](../sources/prs/flashinfer/PR-714.md), [bugfix: FusedAddRMSNorm kernels might require more than 48KB shared memory when d is large.](../sources/prs/flashinfer/PR-718.md), [Align KV chunk size binary search with actual KV chunk splitting.](../sources/prs/flashinfer/PR-728.md), [Change `apply_rope_with_cos_sin_cache` to accept `cos_sin_cache`](../sources/prs/flashinfer/PR-754.md), [feat: support deepseek prefill attention shape](../sources/prs/flashinfer/PR-765.md), [bugfix: Ensure Loop Termination by Enforcing IEEE-754 Compliance in Sampling Kernels](../sources/prs/flashinfer/PR-774.md), [perf: refactor fa2 prefill template](../sources/prs/flashinfer/PR-776.md), [bugfix: drop CTA_TILE_Q=32](../sources/prs/flashinfer/PR-785.md), [bugfix: MLA decode should multiply sm_scale by math::log2e](../sources/prs/flashinfer/PR-787.md), [fix rope logic in mla decoding](../sources/prs/flashinfer/PR-793.md), [feat: support f32 attention output in FA2 template](../sources/prs/flashinfer/PR-799.md), [feat: apply sm_scale at logits instead of q in FA2 template](../sources/prs/flashinfer/PR-801.md), [perf: memory efficient deepseek mla fused page-attention kernel](../sources/prs/flashinfer/PR-804.md), [bugfix: mla page-attention kernel for different page sizes](../sources/prs/flashinfer/PR-810.md), [feat: unlocking MLA for A100](../sources/prs/flashinfer/PR-812.md), [feat: unlock MLA attention for sm89 (L40/L40s/4090)](../sources/prs/flashinfer/PR-814.md), [bugfix: bugfix on sm89 MLA](../sources/prs/flashinfer/PR-821.md), [bugfix: fix the signature of `CutlassSegmentGEMMSM90`](../sources/prs/flashinfer/PR-827.md), [perf: MLA decode kernel implemented by CuTe targeted to SM80](../sources/prs/flashinfer/PR-844.md), [misc: Remove duplicate param set in MLA kernel](../sources/prs/flashinfer/PR-850.md), [Add POD-Attention to FlashInfer](../sources/prs/flashinfer/PR-858.md), [perf: dynamic split-k for MLA](../sources/prs/flashinfer/PR-863.md), [bugfix: fix the behavior of MLA kernel when kv-length is 0](../sources/prs/flashinfer/PR-868.md), [Naive Support for Hopper FP8 Prefill Kernel with Per-Head Quantization](../sources/prs/flashinfer/PR-869.md), [perf: FlashAttention-3 style MLA PageAttention](../sources/prs/flashinfer/PR-887.md), [feat - support mla kvcache store](../sources/prs/flashinfer/PR-888.md), [perf: fix MLA split-k performance bug](../sources/prs/flashinfer/PR-898.md), [perf: tweak the pipeline design of mla kernel](../sources/prs/flashinfer/PR-901.md), [feat: flashinfer intra-kernel profiler](../sources/prs/flashinfer/PR-913.md), [feat: experimenta support of PDL](../sources/prs/flashinfer/PR-930.md), [bugfix: fix potential issues of FA3 template loading nans for PageAttention](../sources/prs/flashinfer/PR-945.md), [perf: Use 2WG pipeline design for MLA implementation on Hopper](../sources/prs/flashinfer/PR-952.md), [[TVM] Added tvm binding for sampling kernel](../sources/prs/flashinfer/PR-958.md), [perf: Fix python API overhead when CUDAGraph is not enabled](../sources/prs/flashinfer/PR-969.md), [perf: dual pivot top-p/top-k renorm](../sources/prs/flashinfer/PR-974.md), [perf: prefetch page indices for mla kernel](../sources/prs/flashinfer/PR-991.md), [feat: SM-constraint Communication Kernels](../sources/prs/flashinfer/PR-994.md), [3rdparty: upgrade cutlass to 3.9](../sources/prs/flashinfer/PR-997.md), [ROCm SDPA: Ensure attn_mask has the same dtype with q](../sources/prs/pytorch/PR-144398.md), [Add release branch push triggers to inductor-rocm-mi300.yml](../sources/prs/pytorch/PR-149871.md), [[CUDA][avgpool2d] Fix backward launch bounds again for sm100, sm120](../sources/prs/pytorch/PR-150640.md), [[CUDA][avgpool2d] Fix backward launch bounds again for `sm100`, `sm120`](../sources/prs/pytorch/PR-150676.md), [[CUDA] Only use vec128 if CUDA version is newer than 12.8](../sources/prs/pytorch/PR-150705.md), [[ATen][CUDA] Optimize 128 bit vectorization](../sources/prs/pytorch/PR-152967.md), [Fix uint view copy (#151598)](../sources/prs/pytorch/PR-154121.md), [Fix macOS build with `USE_MPS=OFF`](../sources/prs/pytorch/PR-156932.md), [[user triton] AOT inductor support for device-side TMA](../sources/prs/pytorch/PR-157241.md), [[PowerPC] Fixed build issue for vsx vec256 complexfloat and scaled_mm_out_cpu ](../sources/prs/pytorch/PR-157422.md), [[cherry-pick][inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack](../sources/prs/pytorch/PR-158646.md), [CUDA 13.0 Windows Nvidia Driver Update to 580.88](../sources/prs/pytorch/PR-162501.md), [fix cpp extension distributed warning spew](../sources/prs/pytorch/PR-162764.md), [[Graph Partition] improve custom op output alias](../sources/prs/pytorch/PR-163380.md), [[graph partition] Add way to register custom rule (#163310)](../sources/prs/pytorch/PR-163395.md), [CUDA 13.0 Warning update for supported architectures](../sources/prs/pytorch/PR-163633.md), [[Cherry-Pick] [CD] CUDA 13 specific followup changes. Remove sm50-70 From CUDA 12.6 and CUDA 12.8 builds (#162455)](../sources/prs/pytorch/PR-163764.md), [[CD] CUDA 13.0 fix preload logic to include nvidia/cu13/lib/](../sources/prs/pytorch/PR-163766.md), [Move inductor jobs 3.9->3.10](../sources/prs/pytorch/PR-163954.md), [[cuDNN][SDPA] Disable dropout for cuDNN SDPA on 9.11 - 9.13](../sources/prs/pytorch/PR-164026.md), [[AARCH64][CD][CUDA13][Triton][PTXAS] Turn on BUILD_BUNDLE_PTXAS=1 ](../sources/prs/pytorch/PR-164236.md), [CUDA 13.0 builds fix on Amazon Linux 2023](../sources/prs/pytorch/PR-164893.md), [[Graph Partition] move custom rules to inductor config (#166458)](../sources/prs/pytorch/PR-166967.md), [[Graph Partition] fix graph partition input signature for fallback kernels](../sources/prs/pytorch/PR-166985.md), [[cuDNN][SDPA][Convolution] Expose cuDNN runtime version in CUDA hooks](../sources/prs/pytorch/PR-167327.md), [[ROCm] Enable shared memory based pruning for Triton configs](../sources/prs/pytorch/PR-170190.md), [[inductor] Fix cudagraph skip for index_put_ with boolean indices, gr…](../sources/prs/pytorch/PR-170884.md), [[ROCm] Make grouped GEMM CK opt‑in via env and default to fallback path](../sources/prs/pytorch/PR-171140.md), [[cherry-pick][CUDA] Upgrade cuDNN to 9.15.1 for CUDA 13 builds ](../sources/prs/pytorch/PR-171189.md), [[cherry-pick][cuDNN][SDPA] cuDNN SDPA off-by-default for cuDNN versions < 12.9 (#171627)](../sources/prs/pytorch/PR-171895.md), [Skip modded_nanogpt model in TorchInductor benchmark](../sources/prs/pytorch/PR-172141.md), [[Graph Partition] Improve support for mutation ops](../sources/prs/pytorch/PR-172577.md), [Update inductor expected accuracy files](../sources/prs/pytorch/PR-175096.md), [[benchmark] Skip pytorch_CycleGAN_and_pix2pix from inductor benchmarks](../sources/prs/pytorch/PR-175299.md), [[release-only] Remove +ptx from cuda 13.0 builds](../sources/prs/pytorch/PR-175567.md), [[release 2.12] Apply Release only changes to 2.12 branch](../sources/prs/pytorch/PR-180470.md), [Disable kernel cutlass_mla_decode on SM103](../sources/prs/sglang/PR-10058.md), [Optimize nvfp4 block scaled gemm kernel when M is small.](../sources/prs/sglang/PR-10101.md), [fix: resolve gb200 image link](../sources/prs/sglang/PR-10343.md), [Fix correction bias undefined behavior for nvfp4 models](../sources/prs/sglang/PR-10426.md), [Update CUTLASS. Refine KernelSchedule for fp8 (grouped) gemm.](../sources/prs/sglang/PR-10491.md), [[sgl-kernel] Optimize concat_mla_k kernel](../sources/prs/sglang/PR-10543.md), [Optimize cutlass int8 gemm kernel for large M on SM89 Ada GPU](../sources/prs/sglang/PR-10714.md), [disable sm100 for FlashMLA and fast-hadamard-transform in cuda12.6.1](../sources/prs/sglang/PR-11274.md), [[NVIDIA] Add new SMs support for Spark & Thor](../sources/prs/sglang/PR-11287.md), [[sgl-kernel][1/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-11432.md), [Improve Kernel Build Time](../sources/prs/sglang/PR-11508.md), [support cutlass fp4 kernel in sm120](../sources/prs/sglang/PR-11737.md), [Fixed aarch64 flash-mla](../sources/prs/sglang/PR-12009.md), [(1/n)support context parallel with deepseekv3.2-DSA](../sources/prs/sglang/PR-12065.md), [[sgl-kernel][4/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-12080.md), [[sgl-kernel] clean up fa fetch in CMakeLists.txt](../sources/prs/sglang/PR-12392.md), [[Fix] `concat_mla_absorb_q_kernel` fails for long inputs](../sources/prs/sglang/PR-12453.md), [[NVIDIA] Fix CUDA arch requirement in nvfp4 cast](../sources/prs/sglang/PR-12581.md), [Support moe topk sigmoid kernel](../sources/prs/sglang/PR-13049.md), [diffusion: enable fa4 for blackwell](../sources/prs/sglang/PR-13263.md), [[sgl-kernel][Feat][B200][1/N]Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-13731.md), [[kernel][moe] add moe topk fast](../sources/prs/sglang/PR-13969.md), [Add new moe wna16 marlin gemm](../sources/prs/sglang/PR-14122.md), [Opt moe align block size kernel](../sources/prs/sglang/PR-14133.md), [sync attention, deepseek doc](../sources/prs/sglang/PR-14335.md), [[CPU] Implement MXFP4 Gemm kernels for intel AMX to support GPT OSS series.](../sources/prs/sglang/PR-14385.md), [Add CUDA kernel size analysis tool for sgl-kernel optimization](../sources/prs/sglang/PR-14544.md), [[sgl-kernel][Feat][B200][2/N] Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-14640.md), [[NPU][eagle3] support qwen eagle3 on NPU](../sources/prs/sglang/PR-14820.md), [[sgl-kernel][1/2] Fused qk_norm_rope for GLM4.6](../sources/prs/sglang/PR-15141.md), [Add cache for flashinfer installation](../sources/prs/sglang/PR-15153.md), [[sgl-kernel] Update flashmla to include fp8 sparse_mla optimizations](../sources/prs/sglang/PR-15242.md), [Fix warp illegal instruction in kimi k2 thinking PCG](../sources/prs/sglang/PR-15306.md), [[diffusion] Add Sage Attention 3 Support for sm 120 (RTX5090)](../sources/prs/sglang/PR-15382.md), [[sgl-kernel][6/7]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-15471.md), [[Feature] JIT Fused QK norm + qk norm clean up](../sources/prs/sglang/PR-15835.md), [[JIT kernel] Apply jit per_tensor_quant_fp8 kernel](../sources/prs/sglang/PR-15836.md), [[Feature] add aligned_vector type for JIT kernel](../sources/prs/sglang/PR-16162.md), [Fix sgl-kernel jobs to skip when target_stage is specified](../sources/prs/sglang/PR-16308.md), [[diffusion] Fix RuntimeError in SageAttention3 on Nvidia Blackwell with Qwen-Image](../sources/prs/sglang/PR-16335.md), [[Fix]Pin mooncake version to 0.3.7.post2 in grace blackwell](../sources/prs/sglang/PR-16502.md), [[DeepSeek v3.2] Opt MTP decode cuda batch sizes and nsa implementation](../sources/prs/sglang/PR-16961.md), [Kernel: optimize decoding metadata in NSA multi-spec backend with fused kernels](../sources/prs/sglang/PR-17554.md), [Make flashMLA work on: Cu13, B300](../sources/prs/sglang/PR-17600.md), [Feature/support longcat flash lite](../sources/prs/sglang/PR-17838.md), [[Move sgl-kernel Kernel to JIT] Add JIT concat MLA kernels](../sources/prs/sglang/PR-17889.md), [[Diffsuion & JIT_kernel] QKNorm cross heads kernel](../sources/prs/sglang/PR-18073.md), [[Hicache & JIT_kernel] Support page first layout & mla jit kernel](../sources/prs/sglang/PR-18311.md), [docs: expand and update modelopt documentation](../sources/prs/sglang/PR-18479.md), [[FIX] Correct JIT kernel compilation on newer GPUs with outdated driver metadata.](../sources/prs/sglang/PR-18496.md), [Add claude skills for sgl-kernel and jit-kernel](../sources/prs/sglang/PR-18855.md), [Use single mma warp group for short q_len in FA to optimize decoding performance](../sources/prs/sglang/PR-18985.md), [[jit_kernel] Add fused_qknorm_rope JIT kernel](../sources/prs/sglang/PR-19059.md), [[DeepSeek-V3.2][JIT-kernel] Support nsa fuse store indexer k cache](../sources/prs/sglang/PR-19148.md), [[Kernel Slimming] Migrate NVFP4 kernels to JIT](../sources/prs/sglang/PR-19437.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [Add compile-time 256-bit vector guard for pre-Blackwell](../sources/prs/sglang/PR-19794.md), [[JIT Kernel][Feature] Support JIT custom all reduce (rewrite as v2)](../sources/prs/sglang/PR-19880.md), [[JIT Kernel] Reland NVFP4 kernels to JIT](../sources/prs/sglang/PR-20012.md), [[diffusion] Support nvfp4 for Flux.2](../sources/prs/sglang/PR-20137.md), [fix ci by removing nvidia-cutlass-dsl-libs-base and force reinstall n…](../sources/prs/sglang/PR-20380.md), [fix(docs): correct quantization documentation (#20301)](../sources/prs/sglang/PR-20619.md), [Fix(jit): support rmsnorm for hidden_size in {64, 128, 256}](../sources/prs/sglang/PR-20661.md), [[Feature][JIT Kernel] Fused TP QK norm For Minimax](../sources/prs/sglang/PR-20673.md), [CUTLASS FP8 Blockwise GEMM improvement of SM120](../sources/prs/sglang/PR-20887.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Tiny Fix] Fix IS_BLACKWELL env var empty string warning in rerun-ut workflow](../sources/prs/sglang/PR-20957.md), [ci: run Stage A CUDA tests as stage-a-test-small-1-gpu on 5090](../sources/prs/sglang/PR-20988.md), [[Whisper] Enable CUDA graph support and timestamp for whisper model](../sources/prs/sglang/PR-21190.md), [Split pr-test.yml: extract sgl-kernel, jit-kernel, and multimodal-gen tests into separate workflow files](../sources/prs/sglang/PR-21219.md), [CUTLASS NVFP4 GEMM improvement of SM120](../sources/prs/sglang/PR-21314.md), [[Diffusion] Add qknorm rope fuse kernel](../sources/prs/sglang/PR-21440.md), [fix: piecewise_cuda_graph get correct qo_indptr](../sources/prs/sglang/PR-21452.md), [fix: TRT-LLM MHA CUDA illegal address with EAGLE v2 + DP attention](../sources/prs/sglang/PR-21649.md), [[jit_kernel] Optimize fused_qknorm_rope: deduplicate sincosf for interleave RoPE ](../sources/prs/sglang/PR-21654.md), [Remove flashinfer wheel cache cleanup that deletes other versions](../sources/prs/sglang/PR-21711.md), [[Feature] JIT rmsnorm update (with claude)](../sources/prs/sglang/PR-21834.md), [[Bugfix] Fix CUDA graph replay issues in trtllm_mla draft_extend](../sources/prs/sglang/PR-21987.md), [[Diffusion] Fix weight scale swizzle and add large-M kernel config for FLUX.2-dev-NVFP4](../sources/prs/sglang/PR-22064.md), [[diffusion] Default NVFP4 to CUTLASS and add all-model shape benchmarks](../sources/prs/sglang/PR-22091.md), [Lazy import flash_attention_v4 to avoid loading flash_attn.cute at startup](../sources/prs/sglang/PR-22306.md), [[Docker] Fix Trivy CVEs, cubin download 403s, and kernels command order](../sources/prs/sglang/PR-22322.md), [[CI/Docker] Clean up redundant flashinfer cubin downloads](../sources/prs/sglang/PR-22491.md), [[Docker] Remove flashinfer cache copy](../sources/prs/sglang/PR-22653.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [[Diffusion] Add Wan2.2 ModelOpt NVFP4 support](../sources/prs/sglang/PR-22681.md), [[Fix] Fix accuracy bug in Flashmla sparse MLA kernel](../sources/prs/sglang/PR-22723.md), [Dual MoE CUDA graph capture for lora/nolora batches](../sources/prs/sglang/PR-22809.md), [[Fix/Kernel] Add JIT rmsnorm_hf kernel to fix transformers backend MMLU accuracy regression ](../sources/prs/sglang/PR-22931.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [[feat] Init true on policy with qwen_dense](../sources/prs/sglang/PR-23961.md), [Enable PDL for various kernels in DSV32/GLM5](../sources/prs/sglang/PR-23965.md), [[VLM] Optimize Gemma4 VLM with PCG and fuse RMSNorm + residual add + scalar](../sources/prs/sglang/PR-24048.md), [Refactor device timer, clean up metrics collector, and add fwd occupancy metric](../sources/prs/sglang/PR-24197.md), [[codex] Optimize hidden-size 512 RMSNorm dispatch](../sources/prs/sglang/PR-24710.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [Support draft extend cuda graph for tokenspeed_mla attention backend](../sources/prs/sglang/PR-25489.md), [[fp8] SM90 swap-AB scaled_mm dispatch (~1.16x kernel geomean, +5.8-18.5% end-to-end)](../sources/prs/sglang/PR-25532.md), [fix (jit kernel): elementwise activation C++ error](../sources/prs/sglang/PR-25695.md), [[Codex] Remove stale DeepSeek V4 JIT kernels](../sources/prs/sglang/PR-25764.md), [[Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename](../sources/prs/sglang/PR-25821.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [Support cutlass Int8 gemm](../sources/prs/sglang/PR-2752.md), [upgrade cutlass v3.7.0](../sources/prs/sglang/PR-2967.md), [feat: add flashinfer as 3rdparty and use rmsnorm as example](../sources/prs/sglang/PR-3033.md), [Support sm90 Int8 gemm](../sources/prs/sglang/PR-3035.md), [support w8a8 fp8 kernel with CUTLASS](../sources/prs/sglang/PR-3047.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [add tensorrt_llm common and cutlass_extensions as 3rdparty](../sources/prs/sglang/PR-3216.md), [support blockwise fp8 matmul kernel](../sources/prs/sglang/PR-3267.md), [fix undefined symbol cudaGetDriverEntryPointByVersion](../sources/prs/sglang/PR-3372.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [[Feature] Apply Cublas Grouped Gemm kernel](../sources/prs/sglang/PR-3629.md), [Support FP4 gemm (1/2)](../sources/prs/sglang/PR-3899.md), [Add fast decode plan for flashinfer mla](../sources/prs/sglang/PR-3987.md), [[Revision] Add fast decode plan for flashinfer mla ](../sources/prs/sglang/PR-4012.md), [Support overlapping two batches](../sources/prs/sglang/PR-4068.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0](../sources/prs/sglang/PR-4231.md), [add THIRDPARTYNOTICES for DeepGEMM](../sources/prs/sglang/PR-4272.md), [Support Blackwell Block Scale FP8 Gemm](../sources/prs/sglang/PR-4278.md), [update deepgemm](../sources/prs/sglang/PR-4284.md), [Add deepseek style fused moe group gate selection kernel](../sources/prs/sglang/PR-4530.md), [[Fix] Fix raw_bs bug when using flashinfer mla and eagle](../sources/prs/sglang/PR-4557.md), [Support fp8 gemm for blackwell](../sources/prs/sglang/PR-4558.md), [avoid cudaStreamSynchronize in DeepSeekV2AttentionMLA](../sources/prs/sglang/PR-4577.md), [support cmake for sgl-kernel](../sources/prs/sglang/PR-4706.md), [Support (1 <= dp < tp) in the dp attention in DeepEP](../sources/prs/sglang/PR-4770.md), [[Feat] support deepgemm for cmake](../sources/prs/sglang/PR-4864.md), [[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu](../sources/prs/sglang/PR-4953.md), [update cutlass tag](../sources/prs/sglang/PR-5011.md), [fix deepgemm as well](../sources/prs/sglang/PR-5030.md), [support sgl-kernel on blackwell](../sources/prs/sglang/PR-5074.md), [reduce moe_align_block_size_kernel small batch mode overhead](../sources/prs/sglang/PR-5086.md), [Blackwell Cutlass MLA kernel](../sources/prs/sglang/PR-5142.md), [[1/2] Add FP8 Blockscale MoE CUTLASS kernel for Blackwell](../sources/prs/sglang/PR-5281.md), [fix: solve cu118 issue for cutlass mla](../sources/prs/sglang/PR-5331.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [chore: upgrade DeepGEMM](../sources/prs/sglang/PR-5395.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [feat: use flashinfer jit package](../sources/prs/sglang/PR-5547.md), [[2/2] Add python wrapper for CUTLASS FP8 Blockscale MoE Kernel. ](../sources/prs/sglang/PR-5694.md), [[PP] Add pipeline parallelism](../sources/prs/sglang/PR-5724.md), [cutlass 3.9 supported to improve fp8_blockwise_gemm](../sources/prs/sglang/PR-5820.md), [Add sm_120 for blackwell](../sources/prs/sglang/PR-5903.md), [chore: upgrade cutlass 3.9.2](../sources/prs/sglang/PR-6004.md), [chore: upgrade deepgemm](../sources/prs/sglang/PR-6073.md), [feat: mtp support dp-attention](../sources/prs/sglang/PR-6081.md), [[1/2] Add Kernel support for Cutlass based Fused FP4 MoE](../sources/prs/sglang/PR-6093.md), [Cutlass MLA: Disable split kv due to https://github.com/NVIDIA/cutlass/issues/2274](../sources/prs/sglang/PR-6101.md), [Upgrade CUTLASS 4.0](../sources/prs/sglang/PR-6336.md), [reduce torch.zeros overhead in moe align block size kernel](../sources/prs/sglang/PR-6369.md), [[Feature] Comprehensive Hybrid Parallelism Support](../sources/prs/sglang/PR-6389.md), [Fix bug of deepseek-v3 under DP+EP mode with large batchsize/seqlen](../sources/prs/sglang/PR-6449.md), [[EP] Add cuda kernel for moe_ep_pre_reorder](../sources/prs/sglang/PR-6699.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [Correctly abort the failed grammar requests & Improve the handling of abort](../sources/prs/sglang/PR-6803.md), [[EP] Add cuda kernel for moe_ep_post_reorder](../sources/prs/sglang/PR-6837.md), [Fix AWQ Dequant and Weight Loading of deepseek v2](../sources/prs/sglang/PR-6842.md), [fix ep_moe_reorder kernel bugs](../sources/prs/sglang/PR-6858.md), [Add a CUDA kernel for fusing mapping and weighted sum for MoE.](../sources/prs/sglang/PR-6916.md), [[sgl-kernel] Add cuda kernel for moe_ep_silu_and_mul](../sources/prs/sglang/PR-6919.md), [[perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128](../sources/prs/sglang/PR-6929.md), [Clean up server_args.py](../sources/prs/sglang/PR-7037.md), [Tiny fix cutlass_mla_get_workspace_size stub incorrect signature](../sources/prs/sglang/PR-7057.md), [Fix grammar abort & Minor style fixes](../sources/prs/sglang/PR-7204.md), [feat: support compatibility between MTP and two-batch-overlap](../sources/prs/sglang/PR-7225.md), [Add CUTLASS FP8 Blockscale MoE kernel for Hopper architecture](../sources/prs/sglang/PR-7278.md), [fix: resolve blackwell deepep image issue](../sources/prs/sglang/PR-7331.md), [Fuse sorted_token_ids padding to moe_align_block_size kernel](../sources/prs/sglang/PR-7437.md), [fix: fix apply_shuffle_mul_sum](../sources/prs/sglang/PR-7444.md), [[CMake] Fix sgl-kernel CMakeLists for Blackwell](../sources/prs/sglang/PR-7543.md), [Add dsv3 router gemm kernel](../sources/prs/sglang/PR-7627.md), [Add dsv3 fused a gemm to sgl-kernel](../sources/prs/sglang/PR-7630.md), [[Feature] CUDA Green Context Support](../sources/prs/sglang/PR-7649.md), [[1/n]: add cutlass W4A8 moe kernel for hopper architecture](../sources/prs/sglang/PR-7772.md), [[kernel] opt moe align block kernel by block/warp scan algorithm](../sources/prs/sglang/PR-7884.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [[Fix][Ready]Fix register spilling in cutlass nvfp4 gemm kernel on Blackwell](../sources/prs/sglang/PR-8127.md), [[sgl-kernel] Opt per_token_quant_fp8 with warp reduce](../sources/prs/sglang/PR-8130.md), [[2/N]Support DeepSeek-R1 w4a8 low latency deepep](../sources/prs/sglang/PR-8464.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/sglang/PR-8818.md), [chore: support blackwell cu129 image](../sources/prs/sglang/PR-8928.md), [[NVIDA] [1/N] Nvfp4 Masked Gemm: Add quant op for the flashinfer grouped gemm](../sources/prs/sglang/PR-9200.md), [[fix]: fix cutlass moe ut and and Opt H20 cutlass groupGemm performance](../sources/prs/sglang/PR-9272.md), [Support trtllm_allreduce_fusion in flashinfer for cuda<12.8](../sources/prs/sglang/PR-9339.md), [[sgl-kernel] feat: Support sm120 cutlass fp8 gemm kernel](../sources/prs/sglang/PR-9403.md), [[NVIDIA] [2/N] Optimize `silu_and_mul_scaled_fp4_grouped_quant` perf](../sources/prs/sglang/PR-9556.md), [Update CUTLASS 4.2 & Enable K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM](../sources/prs/sglang/PR-9559.md), [Make sm100 fp8 kernels available on sm103](../sources/prs/sglang/PR-9789.md), [Make fp4_quantize kernels work on sm103](../sources/prs/sglang/PR-9807.md), [CUTLASS fp8 blockwise gemm support of sm120](../sources/prs/sglang/PR-9969.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feat] profiler support cudagraph backend](../sources/prs/tilelang/PR-1658.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [[Feature] Support tcgen5mma lowering for `.kind::i8`](../sources/prs/tilelang/PR-1764.md), [[CUDA] Support tcgen5mma gemm ts](../sources/prs/tilelang/PR-1866.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[CUDA] Improve int4 GEMM lowering and packed codegen support](../sources/prs/tilelang/PR-2073.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[CUDA] Add native SM75 MMA GEMM support for FP16, INT8 and INT4](../sources/prs/tilelang/PR-2198.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support](../sources/prs/vllm/PR-10995.md), [[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling](../sources/prs/vllm/PR-11868.md), [[Build] Only build 9.0a for scaled_mm and sparse kernels](../sources/prs/vllm/PR-12339.md), [[ROCm] Faster Custom Paged Attention kernels](../sources/prs/vllm/PR-12348.md), [[Attention] MLA decode optimizations](../sources/prs/vllm/PR-12528.md), [[Kernel] port sgl moe_align_block_size kernels](../sources/prs/vllm/PR-12574.md), [[Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3](../sources/prs/vllm/PR-12587.md), [[Attention] MLA with chunked prefill](../sources/prs/vllm/PR-12639.md), [[Perf] Mem align KV caches for CUDA devices (MLA perf improvement)](../sources/prs/vllm/PR-12676.md), [[Core][AMD] Migrate fully transparent sleep mode to ROCm platform](../sources/prs/vllm/PR-12695.md), [[Bugfix][Kernel] Fix per-token/per-channel quantization for Hopper scaled mm](../sources/prs/vllm/PR-12696.md), [[Kernel] Make rotary_embedding ops more flexible with input shape](../sources/prs/vllm/PR-12777.md), [[NVIDIA] Support nvfp4 quantization](../sources/prs/vllm/PR-12784.md), [Optimize moe_align_block_size for deepseek_v3](../sources/prs/vllm/PR-12850.md), [[Misc][Kernel]: Add GPTQAllSpark Quantization](../sources/prs/vllm/PR-12931.md), [[Kernel]Add streamK for block-quantized CUTLASS kernels](../sources/prs/vllm/PR-12978.md), [[Kernel] moe wna16 cuda kernel](../sources/prs/vllm/PR-13321.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[ROCm][MoE] mi300 mixtral8x7B perf for specific BS](../sources/prs/vllm/PR-13577.md), [[Kernel] FlashMLA integration](../sources/prs/vllm/PR-13747.md), [[Attention] MLA support for V1](../sources/prs/vllm/PR-13789.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [[ROCm] Disable chunked prefill/prefix caching when running MLA on non-cuda platforms](../sources/prs/vllm/PR-13844.md), [[Attention] Flash MLA for V1](../sources/prs/vllm/PR-13867.md), [[V1] EP/TP MoE + DP Attention](../sources/prs/vllm/PR-13931.md), [[Kernel] CUTLASS grouped gemm fp8 MoE kernel](../sources/prs/vllm/PR-13972.md), [[Kernel] optimize performance of gptq marlin kernel when n is small](../sources/prs/vllm/PR-14138.md), [dynamic distpatch of fp8 kernels](../sources/prs/vllm/PR-14245.md), [[Attention] FlashAttn MLA](../sources/prs/vllm/PR-14258.md), [fix minor miscalled method](../sources/prs/vllm/PR-14327.md), [[Build/BugFix] Fix hopper 12.8 build](../sources/prs/vllm/PR-14354.md), [Add cutlass support for blackwell fp8 blockwise gemm](../sources/prs/vllm/PR-14383.md), [[BugFix] Illegal Memory Access in the blockwise cutlass fp8 GEMMs](../sources/prs/vllm/PR-14396.md), [[Kernel] moe wna16 marlin kernel](../sources/prs/vllm/PR-14447.md), [permute/unpermute kernel for moe optimization](../sources/prs/vllm/PR-14568.md), [[Attention] Flash Attention 3 - fp8](../sources/prs/vllm/PR-14570.md), [[BugFix/Build] Fix sparse kernels not getting built on hopper](../sources/prs/vllm/PR-14572.md), [[Kernel] GGUF MoE kernel](../sources/prs/vllm/PR-14613.md), [[Kernel] allow non-contiguous input for marlin kernel](../sources/prs/vllm/PR-14658.md), [[Bugfix][Kernel][CPU] Fix num_tokens in CPU rotary embedding kernel](../sources/prs/vllm/PR-14667.md), [[V1] Fully Transparent Implementation of CPU Offloading](../sources/prs/vllm/PR-15354.md), [[Kernel] Fix conflicting macro names for gguf kernels](../sources/prs/vllm/PR-15456.md), [[ROCM][KERNEL] Paged attention for V1](../sources/prs/vllm/PR-15720.md), [[Bugfix] fix use_atomic_add support of marlin kernel when using v1 engine](../sources/prs/vllm/PR-15946.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [[ROCM] Add gfx950 to the custom attention archs](../sources/prs/vllm/PR-16034.md), [Add FlexAttention to V1](../sources/prs/vllm/PR-16078.md), [[Kernel] support merge_attn_states CUDA kernel, 3x speedup](../sources/prs/vllm/PR-16173.md), [[Hardware/NVIDIA/Kernel] [Functional Enablement] [1/N] Enable nvidia/DeepSeek-R1-FP4 Model](../sources/prs/vllm/PR-16362.md), [Allocate kv_cache with stride order](../sources/prs/vllm/PR-16605.md), [[V1] V1 FlashInfer Attention](../sources/prs/vllm/PR-16684.md), [[misc] ignore marlin_moe_wna16 local gen codes](../sources/prs/vllm/PR-16760.md), [[Kernel] GGUF MoeVec kernel](../sources/prs/vllm/PR-16780.md), [[BugFix] Accuracy fix for llama4 int4 - improperly casted scales](../sources/prs/vllm/PR-16801.md), [[Kernel] some optimizations for dense marlin and moe marlin](../sources/prs/vllm/PR-16850.md), [Update PyTorch to 2.7.0](../sources/prs/vllm/PR-16859.md), [[Kernel] Add expert_map support to Cutlass FP8 MOE](../sources/prs/vllm/PR-16861.md), [[Attention] FA3 decode perf improvement - single mma warp group support for head dim 128](../sources/prs/vllm/PR-16864.md), [Update Qwen1.5-MoE-W4A16-compressed-tensors.yaml](../sources/prs/vllm/PR-16946.md), [[ROCm][Kernel][V1] Enable AMD Radeon GPU Custom Paged Attention on v1](../sources/prs/vllm/PR-17004.md), [Fix `numel()` downcast in vllm/csrc/moe/moe_align_sum_kernels.cu +2](../sources/prs/vllm/PR-17082.md), [[ROCm][FP8][Kernel] FP8 quantization fused into Custom Paged Attention](../sources/prs/vllm/PR-17139.md), [[NVIDIA] Support Cutlass w8a8 FP8 for Blackwell Geforce GPUs (sm120)](../sources/prs/vllm/PR-17280.md), [[Misc][ROCm] Exclude `cutlass_mla_decode` for ROCm build](../sources/prs/vllm/PR-17289.md), [[Attention] MLA move o_proj q_proj into cuda-graph region](../sources/prs/vllm/PR-17484.md), [[Attention] MLA move rotary embedding to cuda-graph region](../sources/prs/vllm/PR-17668.md), [[Kernel] fp4 marlin kernel](../sources/prs/vllm/PR-17687.md), [[Kernel] Have rotary embeddings support tensors](../sources/prs/vllm/PR-18046.md), [Fix Broken macro for cutlass moe](../sources/prs/vllm/PR-18049.md), [[Build] Supports CUDA 12.6 and 11.8 after Blackwell Update](../sources/prs/vllm/PR-18316.md), [Sm100 blockwise fp8 swap ab](../sources/prs/vllm/PR-18564.md), [[Kernel] Integrate CUTLASS MoE kernel with PPLX](../sources/prs/vllm/PR-18762.md), [[Perf] Tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-18778.md), [[BugFix] FA2 MLA Accuracy Issue](../sources/prs/vllm/PR-18807.md), [[Hardware][NVIDIA] FP4 MoE kernel optimization](../sources/prs/vllm/PR-19110.md), [[V1] Use FlashInfer by default on Blackwell GPUs](../sources/prs/vllm/PR-19118.md), [[Bugfix][V1] Allow manual FlashAttention for Blackwell](../sources/prs/vllm/PR-19492.md), [[Hardware][NVIDIA][kernel] Fp4 MOE quant kernel optimization](../sources/prs/vllm/PR-19500.md), [[Perf] Further tunings for SM100 FP8 CUTLASS kernel](../sources/prs/vllm/PR-19566.md), [Only build CUTLASS MoE kernels on Hopper](../sources/prs/vllm/PR-19648.md), [[feat]: CUTLASS block scaled group gemm for SM100](../sources/prs/vllm/PR-19757.md), [Fix FA2 fallback for Blackwell V1](../sources/prs/vllm/PR-19781.md), [[Bugfix] Build moe_data for both sm100 and sm90](../sources/prs/vllm/PR-20086.md), [[Bugfix] Fix some narrowing conversion warnings](../sources/prs/vllm/PR-20141.md), [Replace `multiply_add` with `homogeneous_multiply_add` to Address Clang Template Parameter Issue](../sources/prs/vllm/PR-20142.md), [[Bugfix] Fix topk_ids indices_type for CUTLASS w8a8 FP8 MoE](../sources/prs/vllm/PR-20166.md), [[Kernel][Bugfix] Fixup some warnings in nvfp4_blockwise_moe when CUDA < 12.8](../sources/prs/vllm/PR-20324.md), [Update PyTorch to 2.8.0](../sources/prs/vllm/PR-20358.md), [[Kernel] SM90 CUTLASS FP8 GEMM: add support for swap AB + kernel tuning](../sources/prs/vllm/PR-20396.md), [[feat]: add SM100 support for cutlass FP8 groupGEMM](../sources/prs/vllm/PR-20447.md), [[Performance] Performance improvements in non-blockwise fp8 CUTLASS MoE](../sources/prs/vllm/PR-20762.md), [SM100 Cutlass MLA decode with unrestricted num_heads (< 128) for DeepSeek TP](../sources/prs/vllm/PR-20769.md), [[fix]: disable cutlass block scaled group gemm for EP](../sources/prs/vllm/PR-20781.md), [[Perf] Add swap_ab to SM90 FP8 non-block CUTLASS moe grouped gemm](../sources/prs/vllm/PR-20911.md), [Support mnnvl all2allv from Flashinfer](../sources/prs/vllm/PR-21003.md), [[Kernel] Flashinfer MLA (trtllm-gen) decode kernel integration](../sources/prs/vllm/PR-21078.md), [[Perf] Cuda Kernel for Per Token Group Quant](../sources/prs/vllm/PR-21083.md), [[perf] Add fused MLA QKV + strided layernorm](../sources/prs/vllm/PR-21116.md), [[Perf] Use FlashInfer RoPE for RotaryEmbedding.forward_cuda when available](../sources/prs/vllm/PR-21126.md), [[Feature][Kernel]FusedMoE LoRA](../sources/prs/vllm/PR-21229.md), [[v1] - Mamba1 Attention Metadata](../sources/prs/vllm/PR-21249.md), [Support CUTLASS NVFP4 (w4a4) for Blackwell Geforce GPUs (SM120)](../sources/prs/vllm/PR-21309.md), [[Bugfix][CUDA] fixes CUDA FP8 kv cache dtype supported](../sources/prs/vllm/PR-21420.md), [[Bug] Fix Compressed Tensor NVFP4 `cutlass_fp4_group_mm` illegal memory access](../sources/prs/vllm/PR-21465.md), [[Kernel] Improve machete memory bound perf](../sources/prs/vllm/PR-21556.md), [[BugFix] Fix IMA FlashMLA full cuda-graph and DP + Update FlashMLA](../sources/prs/vllm/PR-21691.md), [update flashinfer to v0.2.9rc2](../sources/prs/vllm/PR-21701.md), [Fix Flashinfer CUTLASS MOE Allgather](../sources/prs/vllm/PR-21963.md), [[Kernel] Add support for block FP8 on SM120 (NVIDIA 5090 and RTX PRO 6000)](../sources/prs/vllm/PR-22131.md), [Fp8 paged attention update](../sources/prs/vllm/PR-22222.md), [Upgrade FA3 for attention sink](../sources/prs/vllm/PR-22313.md), [[Attention] FA3 Attention Sinks Perf Boost](../sources/prs/vllm/PR-22478.md), [[Fix] enable swap_ab for pplx problem size computation](../sources/prs/vllm/PR-22991.md), [[Kernel] CUTLASS MoE FP8: Integrate cuda moe permute/unpermute](../sources/prs/vllm/PR-23045.md), [[V1] address post issues related to #20059 (part 1); cascade attention reenable by default](../sources/prs/vllm/PR-23046.md), [[kernel] Support W4A8 on Hopper](../sources/prs/vllm/PR-23198.md), [[Kernel] Add fused grouped_topk kernel for MoE](../sources/prs/vllm/PR-23274.md), [[Perf] Use upstream CUTLASS for SM90 Block FP8 kernel](../sources/prs/vllm/PR-23280.md), [[Compile] Fix Compile Warning SM100 Cutlass MLA](../sources/prs/vllm/PR-23287.md), [fix incompatibililty with non cuda platform for nvfp4](../sources/prs/vllm/PR-23478.md), [[Compile] Fix Compile Warning for `w4a8_mm_entry.cu`](../sources/prs/vllm/PR-23660.md), [[NVIDIA] Support SiluMul + NVFP4 quant fusion](../sources/prs/vllm/PR-23671.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[Bugfix][Misc] Fix silu_and_mul_nvfp4_quant issue and extract common utils for nvfp4 kernel source files](../sources/prs/vllm/PR-23727.md), [[Feature] Support Decode Context Parallel (DCP) for MLA](../sources/prs/vllm/PR-23734.md), [[Kernel] cuda kernels for upcoming decode context parallel feature](../sources/prs/vllm/PR-23791.md), [[Kernel] Faster pre-processing time for W4A8](../sources/prs/vllm/PR-23972.md), [[Model] Add LongCat-Flash ](../sources/prs/vllm/PR-23991.md), [[Kernel] Support decode context parallelism on Blackwell with CUTLASS MLA](../sources/prs/vllm/PR-24385.md), [[NVIDIA] Blackwell Family](../sources/prs/vllm/PR-24673.md), [[Kernel][Quantization] add w4a8 support for marlin kernel](../sources/prs/vllm/PR-24722.md), [[Bugfix] Fix accuracy issue for silu_mul + nvfp4 quant fusion kernel](../sources/prs/vllm/PR-24833.md), [[Bugfix][B200] Fix `cutlass_mla` hang](../sources/prs/vllm/PR-24966.md), [Disable failing GPT-OSS Eval (Blackwell) for now](../sources/prs/vllm/PR-25107.md), [[Compile] Fix Compile Warning for Ignoring `MIN_BLOCK_PER_SM`](../sources/prs/vllm/PR-25193.md), [[Bugfix] [B200] cutlass_mla - ensure kv_split == 1 for batch size > 1](../sources/prs/vllm/PR-25509.md), [Fuse RoPE and MLA KV-cache write](../sources/prs/vllm/PR-25774.md), [Update launch_bounds_utils.h for correct compile on Multiple Cuda Arch - PTXAS out of range Warning](../sources/prs/vllm/PR-25843.md), [Fix INT8 quantization error on Blackwell GPUs (SM100+)](../sources/prs/vllm/PR-25935.md), [[Performance] Split FlashAttn attention and cache update](../sources/prs/vllm/PR-25954.md), [Fix undefined symbol: cutlass_moe_mm_sm100](../sources/prs/vllm/PR-26098.md), [[NVIDIA] [Perf] Update to leverage flashinfer trtllm FP4 MOE throughput kernel](../sources/prs/vllm/PR-26714.md), [[Attention] Tune CUTLASS MLA num_splits](../sources/prs/vllm/PR-26846.md), [[Perf] SM100 - add swap AB optimization to CUTLASS FP8 GEMM](../sources/prs/vllm/PR-27284.md), [Prefer FlashAttention MLA as default over FlashMLA](../sources/prs/vllm/PR-27363.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [[Performance] Fused blockwise quant RMS norm](../sources/prs/vllm/PR-27883.md), [[Kernel] Optimize rms_norm kernel](../sources/prs/vllm/PR-27931.md), [Update Flashinfer from `v0.4.1` to `v0.5.2`](../sources/prs/vllm/PR-27952.md), [[Perf][DeepSeek] Add sigmoid+bias fusion to fused_grouped_topk from TRTLLM](../sources/prs/vllm/PR-28124.md), [[Performance][B200] silu_mul_quant: pack scales in int32](../sources/prs/vllm/PR-28358.md), [[NVIDIA] Guard SM100 CUTLASS MoE macro to SM100 builds v2](../sources/prs/vllm/PR-28938.md), [chore: add RTX_PRO_6000 GLM4.6-FP8 kernel tuning](../sources/prs/vllm/PR-29240.md), [[Kernel] Add NVFP4 MoE CUTLASS support for SM120](../sources/prs/vllm/PR-29242.md), [Lora MoE Align Improvements](../sources/prs/vllm/PR-29257.md), [[Kernel][MoE] optimize `moe_align_block_size`](../sources/prs/vllm/PR-29642.md), [[Kernel]Support W4A8 Grouped GEMM on Hopper](../sources/prs/vllm/PR-29691.md), [[Perf] Improve fp8 quant in mla; replace ReduceSum with ReduceScatterSum](../sources/prs/vllm/PR-29795.md), [[Kernel][Quantization][MoE] add marlin kernel support for turing (sm75)](../sources/prs/vllm/PR-29901.md), [[Perf] Do FP4 quant before All gather on flashinfer trtllmgen MOE ](../sources/prs/vllm/PR-30014.md), [Add llmcompressor fp8 kv-cache quant (per-tensor and per-attn_head)](../sources/prs/vllm/PR-30141.md), [gptq marlin quantization support for fused moe with lora](../sources/prs/vllm/PR-30254.md), [[Feature] Add SM103 (Blackwell Ultra) Support to vLLM](../sources/prs/vllm/PR-30484.md), [OffloadingConnector: Support kernel_block_size != block_size](../sources/prs/vllm/PR-30692.md), [[NVFP4][Perf] Tune NVFP4 input quant kernel for small batch size](../sources/prs/vllm/PR-30897.md), [[Kernel] Add topk_sigmoid kernel](../sources/prs/vllm/PR-31246.md), [[Perf] Fuse stride preparation for NVFP4 cutlass_moe](../sources/prs/vllm/PR-31837.md), [[1/N][Attention] Restructure attention: move files](../sources/prs/vllm/PR-31916.md), [[Perf][Kernel] Optimize FP4 quantization kernels (SM100F)](../sources/prs/vllm/PR-32520.md), [fix: Add glm4_moe_lite to MLA detection](../sources/prs/vllm/PR-32614.md), [[Attention] FA4 integration](../sources/prs/vllm/PR-32974.md), [[Feature] Support CPU Offloading without Pytorch Pinned Memory that leads to doubled allocation](../sources/prs/vllm/PR-32993.md), [[Kernel] Apply 256bit LDG/STG To Activation Kernels](../sources/prs/vllm/PR-33022.md), [Add support for Mistral Large 3 inference with Flashinfer MoE](../sources/prs/vllm/PR-33174.md), [[Bugfix] Fix quant RMS norm fusion for quantization with TMA-aligned scales](../sources/prs/vllm/PR-33255.md), [[Kernel] Add enable_sm120_or_later for SM121 (DGX Spark) CUTLASS support](../sources/prs/vllm/PR-33517.md), [[Feature][Core] Support Fabric detection to adapt the MNNVL protocol for the GB series](../sources/prs/vllm/PR-33540.md), [[Bugfix]fix output Nan/Inf in marlin if dtype=float16](../sources/prs/vllm/PR-33972.md), [Reapply [Attention][FA3] Update FA3 to include new swizzle optimization](../sources/prs/vllm/PR-34043.md), [fix(cpu): fix mla_decode compilation on x86 without AVX512](../sources/prs/vllm/PR-34052.md), [[Kernel] Optimize grouped topk kernel](../sources/prs/vllm/PR-34206.md), [[ModelBash][DSV3] Add TRTLLM DSV3 Router GEMM kernel (6% B1 Speedup)](../sources/prs/vllm/PR-34302.md), [[Bugfix] Enforce DeepGEMM when using sparse_attn_indexer on CUDA](../sources/prs/vllm/PR-34374.md), [[Custom Ops] Add functional + out variant for scaled_fp4_quant](../sources/prs/vllm/PR-34389.md), [[Kernel] Integrate SM100 MXFP8 blockscaled grouped MM and quant kernels](../sources/prs/vllm/PR-34448.md), [[Model Bash] DeepSeek R1 BF16 Min Latency QKV A GEMM (0.5% E2E Speedup)](../sources/prs/vllm/PR-34758.md), [[Bugfix] Gate 256-bit instructions to CUDA 12.9+](../sources/prs/vllm/PR-34791.md), [[Attention][Perf][Kernel] Replace torch.cat with vectorized CUDA kernel MLA query concat - DeepSeek-V3.2](../sources/prs/vllm/PR-34917.md), [[Model Runner V2] Support attention group](../sources/prs/vllm/PR-35036.md), [[Performance] Cublas Bf16 Gate with Fp32 Output](../sources/prs/vllm/PR-35121.md), [[Bugfix] Fix DSV3 kernels breaking _C and _moe_C on unsupported arches](../sources/prs/vllm/PR-35123.md), [[Bugfix] Fix expert_ids padding values in moe_align_block_size kernel](../sources/prs/vllm/PR-35161.md), [[BugFix] Fix fp4 quant kernel on CUDA 12.8](../sources/prs/vllm/PR-35210.md), [[Feat] Add CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function](../sources/prs/vllm/PR-35271.md), [[Attention][Perf] Optimize cp_gather_and_upconvert_fp8_kv_cache - DeepSeek-v3.2](../sources/prs/vllm/PR-35290.md), [[Kernel] Add FlashInfer MoE A2A Kernel](../sources/prs/vllm/PR-36022.md), [Add 320 dimension size support to MLA](../sources/prs/vllm/PR-36161.md), [docs: fix wrong cc in int8.md](../sources/prs/vllm/PR-36209.md), [[Kernel] Fuse FP8 output quantization into merge_attn_states](../sources/prs/vllm/PR-36518.md), [Update Flashinfer to 0.6.6](../sources/prs/vllm/PR-36768.md), [[MTP][Sparse MLA] Take advantage of native MTP support in indexer when possible](../sources/prs/vllm/PR-36982.md), [[Kernel] Add gpt-oss Router GEMM kernel](../sources/prs/vllm/PR-37205.md), [[UX] Add flashinfer-cubin as CUDA default dep](../sources/prs/vllm/PR-37233.md), [[Perf] Set Flashinfer sparse MLA as default backend for FP8 kv cache](../sources/prs/vllm/PR-37252.md), [[Kernel] Add non-gated support for NVFP4 CUTLASS MoE](../sources/prs/vllm/PR-37320.md), [Add nvfp4 support to reshape_and_cache_flash](../sources/prs/vllm/PR-37332.md), [[torch.compile] Refactor Attention Quant Fusion Pass and Remove Boilerplate](../sources/prs/vllm/PR-37373.md), [[Perf][Kernel] Persistent TopK scheduler: unified CUDAGraph-safe kernel with dynamic per-row dispatch - DeepSeek-V3.2 DSA decode](../sources/prs/vllm/PR-37421.md), [[Kernel] Add MXFP4 W4A4 CUTLASS MoE kernel for SM100](../sources/prs/vllm/PR-37463.md), [[4/n] Migrate FP4/W4A8 CUTLASS kernels to torch stable ABI](../sources/prs/vllm/PR-37503.md), [refactor: abstract deepgemm support into platform](../sources/prs/vllm/PR-37519.md), [[Bugfix] Preserve CUDA arch suffix (a/f) for SM12x — fixes NVFP4 NaN on desktop Blackwell](../sources/prs/vllm/PR-37725.md), [[Kernel] Optimize SM120 CUTLASS blockwise FP8 GEMM](../sources/prs/vllm/PR-37970.md), [[Kernel] Add swapAB support for SM120 CUTLASS blockwise FP8 GEMM ](../sources/prs/vllm/PR-38325.md), [[CI Bugfix] Pre-download missing FlashInfer headers in Docker build](../sources/prs/vllm/PR-38391.md), [[NVIDIA] Bugfix NVFP4 DGX Spark and RTX50](../sources/prs/vllm/PR-38423.md), [[Perf] Batch KV cache swap copies via cuMemcpyBatchAsync](../sources/prs/vllm/PR-38460.md), [[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity](../sources/prs/vllm/PR-38479.md), [[Compile] Fix nvfp4 compile warning](../sources/prs/vllm/PR-38573.md), [[FA4] Update flash-attention to latest upstream FA4](../sources/prs/vllm/PR-38690.md), [[Refactor] Improve indexer decode path metadata preparation](../sources/prs/vllm/PR-38865.md), [[Bugfix] Fix broken explicit unquantized kv cache dtype support](../sources/prs/vllm/PR-38922.md), [[Bugfix] Fix GDN FLA kernel crashes with NULL_BLOCK_ID=0 CUDA graph padding](../sources/prs/vllm/PR-39064.md), [[XPU] Quick fix for TritonMLA to remove cuda hardcode](../sources/prs/vllm/PR-39088.md), [perf(moe): add tuned fused_moe config for RTX PRO 6000 Blackwell Server Edition](../sources/prs/vllm/PR-39183.md), [Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps](../sources/prs/vllm/PR-39306.md), [Fix NUMA binding on non-CDMM Grace-Blackwell systems](../sources/prs/vllm/PR-39361.md), [fix: clamp NaN/Inf in topk_softmax to prevent duplicate expert IDs](../sources/prs/vllm/PR-39391.md), [[Perf] Fuse Zero Initializer for FP8 DeepGemm Block Quant Kernel](../sources/prs/vllm/PR-39547.md), [[Bugfix] Add Marlin kernel in block scaled mm kernel selection.](../sources/prs/vllm/PR-40105.md), [[Bugfix] moe lora align kernel grid](../sources/prs/vllm/PR-40131.md), [[Bugfix] Guard mxfp4_experts_quant bindings on ENABLE_NVFP4_SM100](../sources/prs/vllm/PR-40191.md), [[Performance][DSR1]: Fused RoPE+KVCache+q_concat for MLA](../sources/prs/vllm/PR-40392.md), [[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement](../sources/prs/vllm/PR-40408.md), [[GDN] Enable FI Blackwell GDN prefill kernel](../sources/prs/vllm/PR-40717.md), [[DSV4] Add silu clamp limit to shared expert](../sources/prs/vllm/PR-40950.md), [[DSV4] Fuse norm and router for low latency scenario](../sources/prs/vllm/PR-41263.md), [Faster per-token fp8 group quant packed kernel for blackwell](../sources/prs/vllm/PR-41326.md), [[Bugfix] Fix condition to clear persistent topk so that it can be captured regardless](../sources/prs/vllm/PR-41665.md), [[MLA Attention Backend] Add TOKENSPEED_MLA backend for DSR1/Kimi K25 prefill + decode on Blackwell](../sources/prs/vllm/PR-41778.md), [[CUDA][CUTLASS] Enable cutlass scaled mm for non-compatible sizes ](../sources/prs/vllm/PR-41868.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[Perf] Use 2D-grid to eliminate divmod in W8W8 group quant](../sources/prs/vllm/PR-42153.md), [[Misc] add humming to dependencies](../sources/prs/vllm/PR-42540.md), [[6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued)](../sources/prs/vllm/PR-42663.md), [[Refactor] Remove dead cuda kernels](../sources/prs/vllm/PR-42767.md), [[Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement](../sources/prs/vllm/PR-42774.md), [[Perf][MLA] Enable FULL cudagraph capture for TRITON_MLA decode](../sources/prs/vllm/PR-42885.md), [[Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel ](../sources/prs/vllm/PR-7174.md), [[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin](../sources/prs/vllm/PR-7701.md), [[Bugfix] Fix Machete unittests failing with `NotImplementedError`](../sources/prs/vllm/PR-9218.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [FlashMLA — Multi-head Latent Attention](../wiki/kernels/flashmla.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Gated Delta Net — Linear Attention](../wiki/kernels/gated-delta-net.md), [Gated Dual GEMM (Gate-Up + SwiGLU Fusion)](../wiki/kernels/gated-dual-gemm.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md), [Sparse MLA (DeepSeek V3.2)](../wiki/kernels/sparse-mla.md), [TensorRT-LLM Blackwell FP4 DSA Indexer](../wiki/kernels/tensorrt-llm-blackwell-indexer.md), [CCCL CUB Memory Primitives For Selection And Scan](../wiki/techniques/cccl-memory-primitives.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md) | | `cute-dsl` | [CuTe DSL for Blackwell](../wiki/languages/cute-dsl.md) | [Colfax Article Source Kernels](../sources/blogs/colfax-article-source-kernels.md), [Colfax CUTLASS Tutorial: GEMM Kernels Using Tensor Memory for Blackwell](../sources/blogs/colfax-cutlass-blackwell.md), [FlashAttention-4 Blog](../sources/blogs/flash-attention-4.md), [NVFP4 GEMV and Improved NVFP4 GEMV](../sources/blogs/simon-nvfp4-gemv.md), [simveit effective_transpose](../sources/blogs/simveit-effective-transpose.md), [simveit load_and_store](../sources/blogs/simveit-load-and-store.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [CUTLASS Changelog: SM100/Blackwell Entries](../sources/docs/cutlass-changelog-sm100.md), [NVIDIA CUTLASS 4.x Blackwell Support](../sources/docs/nvidia-cutlass-blackwell.md), [[None][perf] Add more optimization options for MOE CuteDSL finalized kernel](../sources/prs/TensorRT-LLM/PR-10042.md), [[TRTLLM-9992][perf] Enable PDL for CuteDSL kernels and overlap MoeOutputMemset](../sources/prs/TensorRT-LLM/PR-10043.md), [[None][feat] CuteDSL MOE FC1 Enhancement](../sources/prs/TensorRT-LLM/PR-10088.md), [[TRTLLM-9457][feat] Add cute dsl fp8 gemm for Blackwell](../sources/prs/TensorRT-LLM/PR-10130.md), [[TRTLLM-9831][perf] Enable 2CTA with autotune for CuteDSL MoE and Grouped GEMM optimizations](../sources/prs/TensorRT-LLM/PR-10201.md), [[TRTLLM-10147][perf] Balanced random MoE workload generator for CuteDSL kernel UT, autotuner and layerwise benchmark](../sources/prs/TensorRT-LLM/PR-10279.md), [[TRTLLM-9661][chore] Further reduce tuning time for cuteDSL nvFP4 dense gemm.](../sources/prs/TensorRT-LLM/PR-10339.md), [[None] [feat] Add test script and raster M for gather fc1 kernel](../sources/prs/TensorRT-LLM/PR-10429.md), [[TRTLLM-10276][feat] Integrate cutedsl argmax kernel](../sources/prs/TensorRT-LLM/PR-10476.md), [[None] [feat] Add densegemm backend for MoE](../sources/prs/TensorRT-LLM/PR-10479.md), [[TRTLLM-9831][perf] Use TMA.RED to improve effective memory bandwidth](../sources/prs/TensorRT-LLM/PR-10987.md), [[None][feat] fuse shared to sparse experts in TRT-LLM Gen MoE](../sources/prs/TensorRT-LLM/PR-11143.md), [[https://nvbugs/5854860][fix] Fix cutedsl argmax on sm120](../sources/prs/TensorRT-LLM/PR-11181.md), [[TRTLLM-10004][feat] Enable GEMM -> AR with GEMM output in registered buffers](../sources/prs/TensorRT-LLM/PR-11589.md), [[TRTLLM-11092][feat] add support for visual gen FA4 attention backend](../sources/prs/TensorRT-LLM/PR-11697.md), [[https://nvbugs/5885070][fix] fix deepeplowlatency with cutedsl moe backend](../sources/prs/TensorRT-LLM/PR-11769.md), [[TRTLLM-10990][feat] Fuse SwiGLU and quant into shared expert](../sources/prs/TensorRT-LLM/PR-11897.md), [[TRTLLM-10407][feat] Integrate CuTE DSL top-k kernel for Blackwell](../sources/prs/TensorRT-LLM/PR-11900.md), [[TRTLLM-11289][feat] Integrate CuteDSL's bf16 dense GEMMs](../sources/prs/TensorRT-LLM/PR-12074.md), [[None][feat] CuteDSL MOE: Add raster along M/N support for blockscaled contiguous backbone kernel](../sources/prs/TensorRT-LLM/PR-12079.md), [[None][feat] Add DWDP (Distributed Weight Data Parallelism) support for MoE inference](../sources/prs/TensorRT-LLM/PR-12136.md), [[TRTLLM-10407][perf] Enable CuteDSL indexer_top_k in model](../sources/prs/TensorRT-LLM/PR-12236.md), [[TRTLLM-10407][perf] Add cute dsl single pass multi cta cluster topk](../sources/prs/TensorRT-LLM/PR-12354.md), [[None][feat] Add PDL support to CuTE DSL top-k kernels](../sources/prs/TensorRT-LLM/PR-12506.md), [[None][feat] Optimize mamba SSD prefill and extend flashinfer dispatch](../sources/prs/TensorRT-LLM/PR-12731.md), [[TRTLLM-11797][feat] Add cutedsl moe backend supporting for qwen3.5.](../sources/prs/TensorRT-LLM/PR-12799.md), [[TRTLLM-11585][feat] Add CUTEDSL moe backend for nemotron-h](../sources/prs/TensorRT-LLM/PR-12884.md), [[TRTLLM-34871][feat] Add cute dsl FP8 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13219.md), [[None][perf] FC2 DenseGEMM autotune: split-K, swap_ab, fine-grained tuning buckets](../sources/prs/TensorRT-LLM/PR-13833.md), [[TRTLLM-35237][feat] Add cute dsl FP4 paged MQA logits decode kernel](../sources/prs/TensorRT-LLM/PR-13929.md), [[None][feat] Enable EPLB for trtllm-gen and cutlass backend](../sources/prs/TensorRT-LLM/PR-8886.md), [[TRTLLM-9685] [feat] Add gather fc1 kernel by cuteDSL](../sources/prs/TensorRT-LLM/PR-9618.md), [Blockwise and Groupwise GEMM for Blackwell and Improvements for Hopper](../sources/prs/cutlass/PR-2139.md), [Example 77 add blackwell flash-attention bwd for MLA shape](../sources/prs/cutlass/PR-2466.md), [Add Blackwell MLA forward (shape: d=192, dv=128) implementation](../sources/prs/cutlass/PR-2472.md), [fix gqa issue for blackwell fmha.py](../sources/prs/cutlass/PR-2599.md), [Add tutorial fp16_gemm_1](../sources/prs/cutlass/PR-2750.md), [new example with TMA prefetch feature targeting for DRAM latency boun…](../sources/prs/cutlass/PR-2881.md), [[CuTeDSL] Fix: SM100 block-scale gemm overlapping accumulator](../sources/prs/cutlass/PR-2995.md), [[Cute-DSL] Add option for issue_clc_query without multicast](../sources/prs/cutlass/PR-3021.md), [[Hopper CuTeDSL] Add grouped GEMM kernel example](../sources/prs/cutlass/PR-3091.md), [[CLI] add cutedsl fp16 gemm tutorial from 2 to 6](../sources/prs/cutlass/PR-3106.md), [Update blackwell tutorial to be compatible with 4.5-dev version](../sources/prs/cutlass/PR-3130.md), [feat: Adding varlen support to cute-dsl sm80 bwd](../sources/prs/flash-attention/PR-1934.md), [[Cute,Fwd,Sm100] fp8 e4m3 and e5m2 support](../sources/prs/flash-attention/PR-2109.md), [[Cute,Flex,Fwd] Allow vectorized score_mod definitions](../sources/prs/flash-attention/PR-2236.md), [[Cute,Sm100,Fwd] add MLA 64/512 with topk sparsity for MQA 128 heads](../sources/prs/flash-attention/PR-2441.md), [feat: masked layout fp4 gemm using cute-dsl](../sources/prs/flashinfer/PR-1331.md), [Add python API for masked grouped gemm](../sources/prs/flashinfer/PR-1481.md), [fix: update cutedsl masked moe gemm](../sources/prs/flashinfer/PR-1488.md), [fix: update masked moe gemm fp4 tensor reshape](../sources/prs/flashinfer/PR-1495.md), [feat: scaling at fp4 gemm epilogue](../sources/prs/flashinfer/PR-1498.md), [Add benchmark for cutedsl gemm](../sources/prs/flashinfer/PR-1502.md), [bugfix: Fix stream handling in cutedsl gemm](../sources/prs/flashinfer/PR-1509.md), [refactor fp4 masked gemm cute-dsl implementation and add manual cache](../sources/prs/flashinfer/PR-1521.md), [feat: initial support for SM103, SM110, SM120, SM121](../sources/prs/flashinfer/PR-1608.md), [Support output signals for overlapping for cutedsl gemm](../sources/prs/flashinfer/PR-1677.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [fix: fix cannot import name 'cuda' from 'cuda' in CUDA13](../sources/prs/flashinfer/PR-1764.md), [tests: upgrade cutlass, fix import and skip non-SM100 cutedsl two shot allreduce](../sources/prs/flashinfer/PR-1812.md), [raise error for group_gemm_fp8_nt_groupwise then num_groups > 1 on sm120/121](../sources/prs/flashinfer/PR-1862.md), [enable sm103 moe dsl backend](../sources/prs/flashinfer/PR-2149.md), [Fix gemm allreduce two shot](../sources/prs/flashinfer/PR-2171.md), [feat: Fused RMSNorm + FP4 Quantization Kernels in CuTe-DSL](../sources/prs/flashinfer/PR-2233.md), [fix: Add global scale support and optional output allocation for RMSNorm+FP4Quant fusion kernels](../sources/prs/flashinfer/PR-2260.md), [[WIP] Refactor: simplify torch -> cute-dsl boilerplate and enable tvm-ffi for cute-dsl kernels](../sources/prs/flashinfer/PR-2279.md), [fix: In-place Residual Update for add_rmsnorm_fp4quant](../sources/prs/flashinfer/PR-2385.md), [feat: Add output_both_sf_layouts option to add_rmsnorm_fp4quant API](../sources/prs/flashinfer/PR-2395.md), [feat: cuteDSL fp4 moe for better DSR1 performance.](../sources/prs/flashinfer/PR-2398.md), [perf: improve gdn decode cute-dsl kernels](../sources/prs/flashinfer/PR-2405.md), [refactor: simplify fp4 rmsnorm](../sources/prs/flashinfer/PR-2421.md), [refactor: refactoring cuda code to cute-dsl (part 1)](../sources/prs/flashinfer/PR-2428.md), [Add cute-dsl backends to mxfp[8,4]_quantization for future refactor](../sources/prs/flashinfer/PR-2443.md), [Ameyn/gdn decode cutedsl kernel](../sources/prs/flashinfer/PR-2498.md), [refactor: Port upstream CUTLASS fixes and refactor grouped_gemm_nt_masked GEMM module location](../sources/prs/flashinfer/PR-2503.md), [[Bug] Fix spark unit test failures for test_add_rmsnorm_fp4_quant_cute_dsl](../sources/prs/flashinfer/PR-2573.md), [fix: cute dsl nvfp4 moe routing index error](../sources/prs/flashinfer/PR-2629.md), [feat: support mxfp4 & mxfp8 entrypoint for blackwell cutedsl dense gemm](../sources/prs/flashinfer/PR-2660.md), [Add cute dsl mla decode op](../sources/prs/flashinfer/PR-2743.md), [[CuTe DSL] Add modular FMHA prefill and MLA decode attention kernels](../sources/prs/flashinfer/PR-2805.md), [CuteDSL MoE fix redundant output buffer zeroing](../sources/prs/flashinfer/PR-2811.md), [feat: Add CuTe-DSL backend for NVFP4 quantization](../sources/prs/flashinfer/PR-2838.md), [feat: add pdl support for cute dsl mla decode kernel support](../sources/prs/flashinfer/PR-2901.md), [perf: Optimize CuTe-DSL fp4 and fp8 quantization kernels](../sources/prs/flashinfer/PR-2904.md), [feat: Add CuTe DSL grouped-gemm + combine fusion support](../sources/prs/flashinfer/PR-2944.md), [feat: add PDL support to rmsnorm_fp4quant and add_rmsnorm_fp4quant CuTe DSL kernels](../sources/prs/flashinfer/PR-3008.md), [Prevent MoE autotuner buffer overflow on large token buckets](../sources/prs/flashinfer/PR-3025.md), [feat: Add backend="b12x" for mm_fp4 on SM120](../sources/prs/flashinfer/PR-3051.md), [feat: Add b12x CuTe DSL fused MoE for SM120](../sources/prs/flashinfer/PR-3066.md), [cute-dsl fmha prefill (cubin integration): remove front-padding, add attention_sink, and pdl support](../sources/prs/flashinfer/PR-3181.md), [fix(sm12x): fix micro-kernel workspace sizing when routed_rows > num_local_experts](../sources/prs/flashinfer/PR-3191.md), [fix(cute_dsl/moe): make autotuner bucket configuration adapt to runtime input](../sources/prs/flashinfer/PR-3216.md), [Support Kimi K2.5 H64 CuTe DSL MLA decode](../sources/prs/flashinfer/PR-3235.md), [fix(cute_dsl/moe): unbias autotuner profiling for tile_size enumeration](../sources/prs/flashinfer/PR-3252.md), [feat(moe): add SM120 W4A16 b12x kernels](../sources/prs/flashinfer/PR-3271.md), [feat(cute_dsl/moe): deterministic balanced autotune profile inputs](../sources/prs/flashinfer/PR-3286.md), [feat(cute_dsl/moe): add `moe_output_memset_inplace` dense memset wrapper](../sources/prs/flashinfer/PR-3328.md), [Support single batch overlap](../sources/prs/sglang/PR-10422.md), [Replace [silu_and_mul_]scaled_fp4_group_quant by Flashinfer equivalent](../sources/prs/sglang/PR-12376.md), [[diffusion] kernel fusion: gated residual layernorm scale shift and layernorm scale shift kernel fusion for Qwen-Image, WAN and HunyuanVideo](../sources/prs/sglang/PR-14717.md), [[jit-kernel] Add CuTe DSL GDN Decode Kernel](../sources/prs/sglang/PR-15631.md), [Move fa4 from sgl-kernel to jit kernel](../sources/prs/sglang/PR-17353.md), [[diffusion] Diffusion norm fusion for z-image](../sources/prs/sglang/PR-18762.md), [[SGLang-Diffusion] Fix custom op fake impl missing eps default for torch.compile](../sources/prs/sglang/PR-19725.md), [[diffusion] fix bug of copy_if](../sources/prs/sglang/PR-20094.md), [[KDA] Support CuTeDSL KDA decode kernel](../sources/prs/sglang/PR-21203.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [Add dedicated FlashInferCuteDslMoE layer for standard-path FP4 MoE](../sources/prs/sglang/PR-21339.md), [[Bugfix] Lazy-import CuteDSL KDA kernel to fix AMD/ROCm startup crash](../sources/prs/sglang/PR-21428.md), [feat: Support flashinfer_cutedsl MoE runner with flashinfer alltoall backend](../sources/prs/sglang/PR-22669.md), [[Refactor] Refactor DeepEP dispatcher](../sources/prs/sglang/PR-22822.md), [Reland Cute-DSL FP4 dense GEMM](../sources/prs/sglang/PR-23590.md), [Use Cute-DSL NVFP4 quantization kernels](../sources/prs/sglang/PR-23745.md), [[MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner](../sources/prs/sglang/PR-25525.md), [[NVIDIA] [3/N] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked ](../sources/prs/sglang/PR-9199.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Bugfix] Fix CuTeDSL autotune cache invalid ELF header (#1967)](../sources/prs/tilelang/PR-1972.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [[MoE] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked](../sources/prs/vllm/PR-25990.md), [[FIX] Add NO_MUL activation support for modular kernel path](../sources/prs/vllm/PR-31528.md), [fix: Add SM120 (RTX Blackwell) support for FlashInfer CUTLASS NVFP4 MoE kernels](../sources/prs/vllm/PR-33417.md), [[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts](../sources/prs/vllm/PR-36728.md), [[MoE/EPLB] Fix FlashInfer nvfp4 experts + EPLB correctness](../sources/prs/vllm/PR-37217.md), [[MoE] Move FlashInfer CuteDSL experts into fused_moe/experts/](../sources/prs/vllm/PR-37759.md), [[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration](../sources/prs/vllm/PR-38050.md), [[Quantization] Add FlashInfer CuteDSL batched experts backend for NVFP4 MoE](../sources/prs/vllm/PR-38251.md), [[Bugfix] [Tests] Enforce `out` tensor device in `kernel/moe/test_cutedsl_moe.py`](../sources/prs/vllm/PR-39644.md), [[DSv4] Improved fused Indexer Q quant kernel](../sources/prs/vllm/PR-41428.md), [[DSv4] Improved dequant gather K cache kernel](../sources/prs/vllm/PR-42236.md), [[Perf] Re-enable flashinfer autotune by default and cleanup](../sources/prs/vllm/PR-42857.md), [add cutedsl dsv4 indexer fp8 kernel](../sources/prs/vllm/PR-42899.md), [[Model Refactoring] Move deepseek_v4_ops to models/deepseek_v4 [3/N]](../sources/prs/vllm/PR-43073.md), [FlashAttention-4](../wiki/kernels/flash-attention-4.md), [FlashAttention SM100 MLA TopK Sparse Forward](../wiki/kernels/flash-attention-sm100-mla-topk.md), [FP8 Block-Scale GEMM](../wiki/kernels/fp8-block-scale-gemm.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Gated Dual GEMM (Gate-Up + SwiGLU Fusion)](../wiki/kernels/gated-dual-gemm.md), [Grouped GEMM for MoE](../wiki/kernels/grouped-gemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [Sparse MLA (DeepSeek V3.2)](../wiki/kernels/sparse-mla.md), [External Source-Map Research For Kernel Edits](../wiki/techniques/external-source-map-research.md) | | `cutile` | | [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md) | | `jax-pallas` | | [Writing High-Performance Matrix Multiplication Kernels for Blackwell with JAX Pallas](../sources/blogs/jax-pallas-blackwell-matmul.md) | -| `ptx` | [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md) | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [NVIDIA Blackwell Compatibility Guide](../sources/docs/blackwell-compatibility-guide.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [misc: point triton blackwell-ptxas to local cuda ptxas](../sources/prs/flashinfer/PR-2543.md), [[AARCH64][CD][CUDA13][Triton][PTXAS] Turn on BUILD_BUNDLE_PTXAS=1 ](../sources/prs/pytorch/PR-164236.md), [[release-only] Remove +ptx from cuda 13.0 builds](../sources/prs/pytorch/PR-175567.md), [Add swizzle layout detection and automatic merging for layout conflicts](../sources/prs/tilelang/PR-1736.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [Update launch_bounds_utils.h for correct compile on Multiple Cuda Arch - PTXAS out of range Warning](../sources/prs/vllm/PR-25843.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md) | +| `ptx` | [PTX Instructions for SM100](../wiki/languages/ptx-sm100.md) | [Twelve Attempts at NVFP4 Batched GEMV](../sources/blogs/amandeep-nvfp4-attempts.md), [tcgen05 for dummies](../sources/blogs/tcgen05-tutorial.md), [Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation](../sources/blogs/tilus-nvidia.md), [Blackwell NVFP4 Kernel Hackathon Journey](../sources/blogs/yue-nvfp4-hackathon.md), [GPU Mode NVFP4 Hackathon - Problem 1: Batched GEMV](../sources/contests/gpu-mode-nvfp4/problem-1-gemv.md), [GPU Mode NVFP4 Hackathon - Problem 2: NVFP4 GEMM](../sources/contests/gpu-mode-nvfp4/problem-2-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 3: Gated Dual GEMM](../sources/contests/gpu-mode-nvfp4/problem-3-gated-dual-gemm.md), [GPU Mode NVFP4 Hackathon - Problem 4: Grouped GEMM](../sources/contests/gpu-mode-nvfp4/problem-4-grouped-gemm.md), [NVIDIA Blackwell Compatibility Guide](../sources/docs/blackwell-compatibility-guide.md), [PTX ISA Ampere (sm_80/sm_86) Instructions Reference](../sources/docs/nvidia-ptx-isa-ampere.md), [PTX ISA SM100 Instructions Reference](../sources/docs/nvidia-ptx-isa-sm100.md), [[Public release 26/04] Introducing Mega MoE, FP4 Indexer and other features/fixes](../sources/prs/DeepGEMM/PR-304.md), [Sync nv_dev with upstream #316 (Mega MoE optimizations & benchmarks)](../sources/prs/DeepGEMM/PR-328.md), [misc: point triton blackwell-ptxas to local cuda ptxas](../sources/prs/flashinfer/PR-2543.md), [[AARCH64][CD][CUDA13][Triton][PTXAS] Turn on BUILD_BUNDLE_PTXAS=1 ](../sources/prs/pytorch/PR-164236.md), [[release-only] Remove +ptx from cuda 13.0 builds](../sources/prs/pytorch/PR-175567.md), [Add swizzle layout detection and automatic merging for layout conflicts](../sources/prs/tilelang/PR-1736.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[Bugfix] Enable `.shared::cta` in TMA copy paths only on CUDA 12.8+](../sources/prs/tilelang/PR-2087.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [Update launch_bounds_utils.h for correct compile on Multiple Cuda Arch - PTXAS out of range Warning](../sources/prs/vllm/PR-25843.md), [DeepGEMM — FP8 GEMM with Fine-Grained Scaling](../wiki/kernels/deepgemm.md), [NVFP4 GEMM — 4-bit Floating Point Matrix Multiply](../wiki/kernels/nvfp4-gemm.md), [NVFP4 Batched GEMV](../wiki/kernels/nvfp4-gemv.md) | | `python` | | [cuTile Python DSL Reference](../sources/docs/cutile-python-dsl.md), [Performance: reducing the percentage of FFMA interleaving yields a sight performance gain, roughly 0.5%](../sources/prs/DeepGEMM/PR-42.md), [[None][feat] sm100 weight-only kernel](../sources/prs/TensorRT-LLM/PR-10190.md), [[TRTLLM-9798][feat] Change to use new DeepGEMM MQA sm100 kernel for MTP-3](../sources/prs/TensorRT-LLM/PR-10226.md), [[None][feat] MiniMax M2 support](../sources/prs/TensorRT-LLM/PR-10532.md), [[None][feat] Remove non flash attetnion style fmha_v2 kernel for hopper](../sources/prs/TensorRT-LLM/PR-11381.md), [[None][fix] Fix int4 awq for sm120/121](../sources/prs/TensorRT-LLM/PR-11561.md), [[None][fix] Fix SM120 issue for rms_norm with nvfp4_quant_fusion](../sources/prs/TensorRT-LLM/PR-11774.md), [[None][feat] GLM 5 support and DSA MTP fixes](../sources/prs/TensorRT-LLM/PR-11990.md), [[TRTLLM-11285][feat] Fuse indexer wk + weights_proj into single GEMM in TF32 for DS-V3.2](../sources/prs/TensorRT-LLM/PR-12055.md), [[https://nvbugs/5983390][fix] Remove redundant D2H sync to optimize perf](../sources/prs/TensorRT-LLM/PR-12445.md), [[#12634][feat] AutoDeploy: Support rank 256 MLA in flashinfer_mla](../sources/prs/TensorRT-LLM/PR-12519.md), [[https://nvbugs/5879577][fix] Fix KeyError in DeepSeekV3Lite FP8 MTP weight loading](../sources/prs/TensorRT-LLM/PR-12530.md), [[None][feat] Add bf16 trtllm-gen moe support through flashinfer.](../sources/prs/TensorRT-LLM/PR-12738.md), [[None][fix] Propagate init_load_balancer to DeepGemmFusedMoE in create_moe_backend](../sources/prs/TensorRT-LLM/PR-13207.md), [[TRTLLM-11127][feat] add W4A8_MXFP4_FP8 MoE unit test support](../sources/prs/TensorRT-LLM/PR-13401.md), [[TRTLLM-11285][perf] Force enable TF32 tensor cores for DSA indexer fused GEMM](../sources/prs/TensorRT-LLM/PR-13452.md), [[TRTLLM-12128][feat] enable SageAttention for Wan/FLUX (new commits)](../sources/prs/TensorRT-LLM/PR-13570.md), [[TRTLLM-12316][feat] Integrate FP4 indexer for DSv4](../sources/prs/TensorRT-LLM/PR-13575.md), [[None][feat] Enable EPLB for DeepSeek-V4](../sources/prs/TensorRT-LLM/PR-13595.md), [[None][feat] Add bf16 trtllm moe through flashinfer.](../sources/prs/TensorRT-LLM/PR-13689.md), [[None][fix] Use compressed lengths for DeepSeek-V4 indexer](../sources/prs/TensorRT-LLM/PR-13802.md), [[None][feat] Update FMHA cubins for head_dim 80](../sources/prs/TensorRT-LLM/PR-13808.md), [[TRTLLM-12503][feat] Parallel VAE independent scaling and fix arg passing](../sources/prs/TensorRT-LLM/PR-13873.md), [[None][feat] enable TRTLLM-Gen internal routing](../sources/prs/TensorRT-LLM/PR-13997.md), [[None][feat] Enable 2 DSv4 perf optimizations by default](../sources/prs/TensorRT-LLM/PR-14120.md), [[TRTLLM-12462][fix] Fix FP8 block scaling GEMM autotuner cache growth](../sources/prs/TensorRT-LLM/PR-14165.md), [[https://nvbugs/6025177][test] rcca tests using kimi k2.5 fp4](../sources/prs/TensorRT-LLM/PR-14172.md), [[https://nvbugs/6163147][fix] swap layer.mlp in place for Mixtral modelopt export](../sources/prs/TensorRT-LLM/PR-14179.md), [[None][fix] Avoid dp_size x ep_size double-count in MegaMoEDeepGemm SymmBuffer](../sources/prs/TensorRT-LLM/PR-14213.md), [[None][refactor] clean up AttentionForwardArgs](../sources/prs/TensorRT-LLM/PR-14244.md), [[None][fix] Handle unset attention_dp_relax in ADP routers](../sources/prs/TensorRT-LLM/PR-14276.md), [[https://nvbugs/6095421][fix] Update resolve_moe_backend](../sources/prs/TensorRT-LLM/PR-14282.md), [[None][chore] Update Claude Code agents and skills](../sources/prs/TensorRT-LLM/PR-14344.md), [[None][feat] GPT-OSS Sm120/Sm121 Support](../sources/prs/TensorRT-LLM/PR-7937.md), [[TRTLLM-8535][feat] Support DeepSeek V3.2 with FP8 + BF16 KV cache/NVFP4 + BF16 KV cache](../sources/prs/TensorRT-LLM/PR-8405.md), [[TRTLLM-8827] [feat] Enable low precision alltoall for Cutlass and TRTLLMGen backends](../sources/prs/TensorRT-LLM/PR-8675.md), [[TRTLLM-8958][feat] and [TRTLLM-8960]: create ConfigurableMoE and support TRTLLMGenFusedMoE as backend](../sources/prs/TensorRT-LLM/PR-9486.md), [[https://nvbugs/5726962][feat] Apply fusion for W4AFP8_AWQ MoE](../sources/prs/TensorRT-LLM/PR-9838.md), [Experimental Python cooperative algorithms](../sources/prs/cccl/PR-1973.md), [[cuda.compute]: Fix faulty pointer arithmetic calculation in CUB dispatch](../sources/prs/cccl/PR-7940.md), [Expose `max_segment_size` guarantee in cuda.compute](../sources/prs/cccl/PR-8284.md), [[cub]: implement utilities for policy selection](../sources/prs/cccl/PR-8355.md), [Hopper Grouped GEMM support for FP8 Accum](../sources/prs/cutlass/PR-2123.md), [fix gqa issue for blackwell fmha.py](../sources/prs/cutlass/PR-2599.md), [Add tutorial fp16_gemm_1](../sources/prs/cutlass/PR-2750.md), [new example with TMA prefetch feature targeting for DRAM latency boun…](../sources/prs/cutlass/PR-2881.md), [[Cute-DSL] Add option for issue_clc_query without multicast](../sources/prs/cutlass/PR-3021.md), [[Hopper CuTeDSL] Add grouped GEMM kernel example](../sources/prs/cutlass/PR-3091.md), [Support for Group GEMM in CUTLASS Profiler for GeForce and Spark](../sources/prs/cutlass/PR-3092.md), [[CLI] add cutedsl fp16 gemm tutorial from 2 to 6](../sources/prs/cutlass/PR-3106.md), [Update blackwell tutorial to be compatible with 4.5-dev version](../sources/prs/cutlass/PR-3130.md), [Small Tile N BlockScaled GEMM + Grouped GEMM on SM12x](../sources/prs/cutlass/PR-3176.md), [Add var-seq-len to FA3 fp16 / bf16 fwd](../sources/prs/flash-attention/PR-1072.md), [FA3 FP8 qkv descales + restore max offset for h128 causal + added sync for producer WG](../sources/prs/flash-attention/PR-1173.md), [Add seqused_q in fwd / bwd and seqused_k in bwd in hopper FA.](../sources/prs/flash-attention/PR-1182.md), [Add local attention in Hopper FAv3](../sources/prs/flash-attention/PR-1233.md), [Paged Attention support for FA3](../sources/prs/flash-attention/PR-1268.md), [FA3 paged attention: Readiness for Cutlass 3.6 / default value for block_table](../sources/prs/flash-attention/PR-1331.md), [Support hdimQK != hdimV backward](../sources/prs/flash-attention/PR-1604.md), [Improve causal backward determinism perf with SPT schedule](../sources/prs/flash-attention/PR-1893.md), [[Cute,Fwd,Sm100] Implement SplitKV](../sources/prs/flash-attention/PR-1940.md), [Blackwell FlashAttention-BWD (v1.0)](../sources/prs/flash-attention/PR-1945.md), [[Cute] Block sparse support Sm100](../sources/prs/flash-attention/PR-1985.md), [[Cute,Fwd,Sm100] Support `q_stage=1` for inference](../sources/prs/flash-attention/PR-1993.md), [[Cute,Fwd,Sm100] Support paged attention](../sources/prs/flash-attention/PR-1999.md), [[Cute,Sm100,Fwd] use correction warps for epi when not using TMA](../sources/prs/flash-attention/PR-2014.md), [[Cute,Fwd,Sm100] don't pass mask_fn to softmax_step generically](../sources/prs/flash-attention/PR-2026.md), [[Cute,Bwd,Sm100] enable deterministic mode for sm100 bwd and fix race conditions](../sources/prs/flash-attention/PR-2033.md), [[Cute,Fwd] Extend score_mod to variable sequence length](../sources/prs/flash-attention/PR-2043.md), [Add score-mod bwd support ](../sources/prs/flash-attention/PR-2070.md), [Add blocksparse support for bwd on blackwell](../sources/prs/flash-attention/PR-2085.md), [Fix IMA in fwd on m boundary](../sources/prs/flash-attention/PR-2091.md), [Add pack-gqa fwd support for sparse impl w/ broadcasted H dim](../sources/prs/flash-attention/PR-2098.md), [[Cute,Fwd,Sm100] distributed offset calculation for paged KV](../sources/prs/flash-attention/PR-2104.md), [[NVIDIA] Enable Jetson Thor FA4](../sources/prs/flash-attention/PR-2108.md), [[CUTE][SM90]Enable pack-gqa with broadcasted maskmods](../sources/prs/flash-attention/PR-2145.md), [[Cute][Flex]Add pack-gqa divmod](../sources/prs/flash-attention/PR-2180.md), [[Cute,Fwd,Sm100] support irregular qhead / kvhead ratios](../sources/prs/flash-attention/PR-2186.md), [[Ai-assisted] CLC work stealing](../sources/prs/flash-attention/PR-2218.md), [[Bwd,Sm120] Add SM120 backward pass support](../sources/prs/flash-attention/PR-2330.md), [Add SM120 varlen attention support](../sources/prs/flash-attention/PR-2333.md), [[Fwd,Sm90] Add paged KV attention support (tma and cp.async)](../sources/prs/flash-attention/PR-2360.md), [[Cute,Sm100,Bwd] refine bwd swizzle for deterministic](../sources/prs/flash-attention/PR-2390.md), [Feat([FA4][CUTE DSL]) Add head_dim=256 support (forward + backward)](../sources/prs/flash-attention/PR-2412.md), [Add CLC scheduler heuristic](../sources/prs/flash-attention/PR-2455.md), [[hd256] Improve forward kernel with exp2 FMA emulation (3% to 9% performance gain)](../sources/prs/flash-attention/PR-2488.md), [[hd256] Add TMA paged KV support to SM100 2CTA forward kernel](../sources/prs/flash-attention/PR-2489.md), [[FA4][hd256] Backward TMA bulk-store epilogue + LSE/dpsum coalesce](../sources/prs/flash-attention/PR-2497.md), [Fix ZeroDivisionError in num_splits_heuristic for empty Q workloads](../sources/prs/flash-attention/PR-2515.md), [bugfix: import wrapper of mla decode](../sources/prs/flashinfer/PR-1013.md), [Add fp4 quantization swizzling tests](../sources/prs/flashinfer/PR-1157.md), [feat: nvshmem python bindings](../sources/prs/flashinfer/PR-1160.md), [feat: logits processor fustion rule for temperature softmax](../sources/prs/flashinfer/PR-1170.md), [Expose fp4 blockscale swizzling kernel](../sources/prs/flashinfer/PR-1176.md), [[feat] support block sparse attention w/ variable block sizes and head-wise sparse patterns](../sources/prs/flashinfer/PR-1177.md), [bugfix: fix invalid blackwell fmha unittests](../sources/prs/flashinfer/PR-1181.md), [bugfix: fix blackwell fmha hanging issue for empty kv_len](../sources/prs/flashinfer/PR-1198.md), [Add DeepGEMM kernels](../sources/prs/flashinfer/PR-1209.md), [Fix test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-1211.md), [init add gemm fp8 using cudnn backend](../sources/prs/flashinfer/PR-1264.md), [feat: add masked deepgemm support and benchmarking](../sources/prs/flashinfer/PR-1266.md), [hotfix: fix deepgemm artifactory hash](../sources/prs/flashinfer/PR-1278.md), [fix: update trtllm-gen fmha benchmark](../sources/prs/flashinfer/PR-1280.md), [Unify groupwise fp8 GEMM test](../sources/prs/flashinfer/PR-1281.md), [add mm_fp4 use cudnn backend](../sources/prs/flashinfer/PR-1288.md), [Remove FAST_BUILD FLAG for MOE](../sources/prs/flashinfer/PR-1291.md), [Support loading autotuned results from json for cutlass fp4 moe backends](../sources/prs/flashinfer/PR-1310.md), [test qkvo quantization not equal to 1.](../sources/prs/flashinfer/PR-1314.md), [minor: add trtllm_gen_mla benchmark](../sources/prs/flashinfer/PR-1316.md), [Allow cudnn prefill kernels to be called natively](../sources/prs/flashinfer/PR-1317.md), [refactor: Improved metainfo for trtllm-gen kernels](../sources/prs/flashinfer/PR-1328.md), [add torch float4_e2m1fn_x2 check for cudnn fp4 backend](../sources/prs/flashinfer/PR-1333.md), [[Fix] remove torch 2.8 requirement for FP4 GEMM](../sources/prs/flashinfer/PR-1334.md), [Fix bench deepgemm setting](../sources/prs/flashinfer/PR-1344.md), [Support passing kv_data_type to MultiLevelCascadeAttentionWrapper.plan()](../sources/prs/flashinfer/PR-1350.md), [[fix] remove (view) transpose to keep consistent with majorness MN requirement.](../sources/prs/flashinfer/PR-1358.md), [hotfix: update mxfp4 groupwise-scaled gemm unittests](../sources/prs/flashinfer/PR-1359.md), [Update autotune results for the nvfp4 cutlass moe backends for v0.2.9](../sources/prs/flashinfer/PR-1361.md), [refactor: download trtllm gemm metadata from server](../sources/prs/flashinfer/PR-1378.md), [Allow BatchPrefillPagedWrapper to call cudnn API](../sources/prs/flashinfer/PR-1384.md), [Adding FP8 benchmark on attention and matmul testing](../sources/prs/flashinfer/PR-1390.md), [Add Mxfp4 trtllm-gen moe unit tests](../sources/prs/flashinfer/PR-1399.md), [feature: enable cublas for fp4 gemm when cudnn == 9.11.1 or >= 9.13](../sources/prs/flashinfer/PR-1405.md), [Faster weight processing (moe nvfp4)](../sources/prs/flashinfer/PR-1412.md), [refactor: Sink attention AoT](../sources/prs/flashinfer/PR-1427.md), [Fix redundant kernels in moe](../sources/prs/flashinfer/PR-1428.md), [bugfix: fix perf issue by using fp8 graph that can use cublaslt](../sources/prs/flashinfer/PR-1435.md), [feat: enable trtllm-gen attn speculative decoding verify by decode](../sources/prs/flashinfer/PR-1453.md), [Fix TRTLLM NVFP4-out attention kernel scale factor dim issue](../sources/prs/flashinfer/PR-1460.md), [feat: Enable multiple fused-moe backends](../sources/prs/flashinfer/PR-1472.md), [refactor: unify autotuner for bmm_fp8](../sources/prs/flashinfer/PR-1479.md), [fix missing enable_pdl argument in trtllm-gen fp4 moe](../sources/prs/flashinfer/PR-1480.md), [Add python API for masked grouped gemm](../sources/prs/flashinfer/PR-1481.md), [flashinfer_benchmark QoL Improvements and Attention FP8 Support](../sources/prs/flashinfer/PR-1512.md), [Remove cuda-python from dependency and check at runtime](../sources/prs/flashinfer/PR-1534.md), [Add sm check for sm100 only cutlass/trtllm kernel](../sources/prs/flashinfer/PR-1535.md), [feat: Add fp8-qkv, fp16/bf16 output MHA](../sources/prs/flashinfer/PR-1540.md), [perf: Enable SplitK and fix tile-scheduling for moe fp4 fused moe](../sources/prs/flashinfer/PR-1548.md), [feat: Support for inferring out_dtype from out.dtype for TRTLLM attention kernel](../sources/prs/flashinfer/PR-1578.md), [refactor: Expose calculate_tile_tokens_dim function](../sources/prs/flashinfer/PR-1581.md), [bugfix: Fix test_fp4_quantize test bug](../sources/prs/flashinfer/PR-1585.md), [fix: limit the number of nvcc threads for each kernel](../sources/prs/flashinfer/PR-1589.md), [fix: Improve TRTLLM attention kernel out_dtype unit test](../sources/prs/flashinfer/PR-1590.md), [bugfix: fix unittest test_fp8_quantize](../sources/prs/flashinfer/PR-1599.md), [feat: Enable MnnvlMemory (for alltoallv) on B200](../sources/prs/flashinfer/PR-1601.md), [feat: add support of fp4_batched_quantize](../sources/prs/flashinfer/PR-1633.md), [fix: pass workspace for trtllm-gen attention](../sources/prs/flashinfer/PR-1635.md), [test: pytest.mark.xfail on deepgemm](../sources/prs/flashinfer/PR-1636.md), [bugfix: Fix FLOPS calculation for bench_trtllm_gen_mla.py](../sources/prs/flashinfer/PR-1640.md), [fix: zero-init workspace buffer for trtllm-gen fmha](../sources/prs/flashinfer/PR-1643.md), [Added mx_fp4 support using the cudnn backend](../sources/prs/flashinfer/PR-1644.md), [Add benchmark for MLARopeQuantize](../sources/prs/flashinfer/PR-1656.md), [test: update fused_moe test to random scale factor](../sources/prs/flashinfer/PR-1665.md), [[Hotfix] `test_fp4_quantize.py` failure on sm103](../sources/prs/flashinfer/PR-1666.md), [TGV GEMM as a BF16 backend alternative to cuBLAS](../sources/prs/flashinfer/PR-1668.md), [test: better fp8 quantization init for fused_moe test](../sources/prs/flashinfer/PR-1674.md), [[misc] add a wrapper class for attention sink jit args](../sources/prs/flashinfer/PR-1679.md), [Update deepgemm backend for 103a](../sources/prs/flashinfer/PR-1694.md), [[cute_dsl] add gemm + all reduce (two_shot)](../sources/prs/flashinfer/PR-1695.md), [hotfix: Hotfix for `test_pod_kernels.py` on B300](../sources/prs/flashinfer/PR-1698.md), [feat: Benchmark mm_fp4 mxfp4 support and gemm autotune support. Restore mm_fp4 API behavior](../sources/prs/flashinfer/PR-1706.md), [bugfix: increase workspace to make trtllm gen attention unit test pass](../sources/prs/flashinfer/PR-1707.md), [test: skip the unsupported test cases for sm120/121](../sources/prs/flashinfer/PR-1710.md), [perf: Add tuning config for cutlass moe for a hardware](../sources/prs/flashinfer/PR-1716.md), [feat: port fast_decode_plan from sgl](../sources/prs/flashinfer/PR-1745.md), [tests: xfail attention sink UT for sliding window + non causal case](../sources/prs/flashinfer/PR-1752.md), [tests: xfail moe quantization classes mxfp8_bf16 UTs on sm103 ](../sources/prs/flashinfer/PR-1754.md), [Fix tests/test_trtllm_gen_attention.py::test_trtllm_batch_prefill, ::test_trtllm_batch_decode mismatch error](../sources/prs/flashinfer/PR-1755.md), [fix: should pass global_override_indptr_cpu in fast_decode_plan param list](../sources/prs/flashinfer/PR-1757.md), [Added xfail for mx_fp4 matmul on SM120](../sources/prs/flashinfer/PR-1766.md), [tests: skip non SM100/103 for grouped deepgemm](../sources/prs/flashinfer/PR-1767.md), [add test case for trtllm gen fused moe with kimi k2 problem sizes](../sources/prs/flashinfer/PR-1768.md), [Waive / disable test_mla_decode_kernel.py::test_mla_decode_kernel for not sm80 ](../sources/prs/flashinfer/PR-1771.md), [Support checks PoC](../sources/prs/flashinfer/PR-1809.md), [tests: Update support for tgv_gemm to SM100 only and add to ut](../sources/prs/flashinfer/PR-1810.md), [fix: fp4 moe on sm120](../sources/prs/flashinfer/PR-1817.md), [misc: fix some B200 GEMM bench](../sources/prs/flashinfer/PR-1883.md), [fix: Fix trtllm-gen prefill IMA when batch_size==1](../sources/prs/flashinfer/PR-1912.md), [Add realistic bench for persistent kernel ](../sources/prs/flashinfer/PR-1942.md), [fix: Add cutlass as an mm_fp4 backend in compute capability 12.0 in benchmark code](../sources/prs/flashinfer/PR-1959.md), [fix: ensure SM120/121 SFA/SFB contiguity](../sources/prs/flashinfer/PR-1963.md), [Feature: Add support for L40 FusedMoE in cutlass path](../sources/prs/flashinfer/PR-1973.md), [fix: Make attention microbenchmark correctly use page table](../sources/prs/flashinfer/PR-1976.md), [fix: Skipping attention sink Blackwell test outside of Blackwell](../sources/prs/flashinfer/PR-1978.md), [feat: Add backend='auto' to mm_fp4 and enable autotune for backend='cudnn'](../sources/prs/flashinfer/PR-1979.md), [unittest: Add head dim 256 test cases and mark as xfail](../sources/prs/flashinfer/PR-1999.md), [Fix trtllm-gen attention illegal memory access](../sources/prs/flashinfer/PR-2002.md), [fix: Enable SM121 for mm_fp4](../sources/prs/flashinfer/PR-2012.md), [feat: suitable_auto_backends to prune auto backends, bmm_fp8 refactor, heuristic_func intake](../sources/prs/flashinfer/PR-2029.md), [Added an initial implementation of Q and KV Cache in fp8 and to use t…](../sources/prs/flashinfer/PR-2035.md), [test: Skip test_fp8_quantize.py on Hopper](../sources/prs/flashinfer/PR-2052.md), [misc: Add XQA decode to microbenchmark for sm90 and sm120](../sources/prs/flashinfer/PR-2055.md), [[Test] Optimize test_trtllm_gen_fused_moe.py](../sources/prs/flashinfer/PR-2072.md), [unittest: improve the efficiency of xqa unittests](../sources/prs/flashinfer/PR-2075.md), [fix: fix test_trtllm_gen_attention when max_seq_len < page_size](../sources/prs/flashinfer/PR-2076.md), [Patch sm103 for 3xfp4 moe generation](../sources/prs/flashinfer/PR-2082.md), [refactor: update dpsk fused_moe test [1]](../sources/prs/flashinfer/PR-2088.md), [refactor: pass hopper deepgemm include directory through python](../sources/prs/flashinfer/PR-2090.md), [refactor: update dpsk fused_moe test [2]](../sources/prs/flashinfer/PR-2097.md), [[DSR1] Added MLA test](../sources/prs/flashinfer/PR-2100.md), [fix: Fix bench_mm_fp8.py](../sources/prs/flashinfer/PR-2129.md), [A unified API for the MNNVL and single-node/multi-GPU AllReduce kernels.](../sources/prs/flashinfer/PR-2130.md), [fix(trtllm): reset negative strideBatch to 0 for ragged KV layout to …](../sources/prs/flashinfer/PR-2134.md), [fix: some bugs of headDim 256 trtllm-gen fmha kernels. ](../sources/prs/flashinfer/PR-2137.md), [Enable Hopper FA3 FP8 attention in decode.py](../sources/prs/flashinfer/PR-2148.md), [refactor: Move mla code from decode.py to mla.py and add to documentation](../sources/prs/flashinfer/PR-2163.md), [fix: compile flags for trtllm fmha_v2 ](../sources/prs/flashinfer/PR-2175.md), [Rename noauxtc to fused_topk_deepseek](../sources/prs/flashinfer/PR-2181.md), [Permute page table in benchmarking](../sources/prs/flashinfer/PR-2194.md), [misc: support checks for gemm](../sources/prs/flashinfer/PR-2214.md), [Fp8 attention are now part of cuDNN 9.17.1](../sources/prs/flashinfer/PR-2241.md), [test: Fix MNNVL tests to skip when container lacks SYS_PTRACE capability](../sources/prs/flashinfer/PR-2245.md), [feat: Add support for bmm mxfp8](../sources/prs/flashinfer/PR-2256.md), [Fix CUTLASS FP8 gemm correctness issue on SM120/SM121 for shapes where N is not divisible by ScaleGranularityN.](../sources/prs/flashinfer/PR-2261.md), [test: use .float() in in F.cosine_similarity() in bmm_fp8 test](../sources/prs/flashinfer/PR-2266.md), [Tiny fix bench tgv gemm](../sources/prs/flashinfer/PR-2277.md), [fix: Decode benchmark's fa2_tc uses backend=fa2 in wrapper](../sources/prs/flashinfer/PR-2302.md), [Support both 3D and 4D kv_cache shapes in MLA APIs](../sources/prs/flashinfer/PR-2334.md), [Added the cudnn backend Ragged KV Cache wrapper](../sources/prs/flashinfer/PR-2352.md), [benchmarks: Add norm and quantization routines to microbenchmark harness.](../sources/prs/flashinfer/PR-2362.md), [feat: [Qwen3-Next] Add Cute DSL GDN decode kernel and tests](../sources/prs/flashinfer/PR-2370.md), [feat: BF16 GEMM using cuDNN backend](../sources/prs/flashinfer/PR-2376.md), [A Blackwell-optimized version of selective_state_update (decode)](../sources/prs/flashinfer/PR-2387.md), [perf: mm_fp4 heuristic prioritizes CUTLASS over cuDNN on SM103](../sources/prs/flashinfer/PR-2404.md), [perf: add fp4 GEMM tile configs and streamK scheduler for SM120](../sources/prs/flashinfer/PR-2460.md), [fix: blockscale moe routine supports non-DS routing](../sources/prs/flashinfer/PR-2476.md), [fix: Fix memory bandwidth calculation in MLA benchmarks](../sources/prs/flashinfer/PR-2479.md), [Feat/gdn decode pooled](../sources/prs/flashinfer/PR-2521.md), [feat: BF16 GEMM benchmarking support](../sources/prs/flashinfer/PR-2525.md), [pick fa2 for BatchDecodeWithPagedKVCacheWrapper auto backend](../sources/prs/flashinfer/PR-2530.md), [fix: include fp8_blockscale_gemm_90 in AOT jit-cache](../sources/prs/flashinfer/PR-2533.md), [fallback to fa2 (instead of fa3) for unsupported configuration (bf16 Q, Fp8 KV)](../sources/prs/flashinfer/PR-2536.md), [tests: bmm_fp8 for SM110](../sources/prs/flashinfer/PR-2538.md), [feat: cute dsl mmfp4 for blackwell](../sources/prs/flashinfer/PR-2540.md), [Add gen_gemm_sm100_module_cutlass_mxfp8 to jit-cache](../sources/prs/flashinfer/PR-2549.md), [fix: allow fmha_v2_prefill_deepseek on SM121 (DGX Spark)](../sources/prs/flashinfer/PR-2559.md), [fix: guard CUTLASS FMHA against SM12x and fix fmha_v2 SM121a check](../sources/prs/flashinfer/PR-2560.md), [feat: add is_sm12x_supported() helper for SM12x family detection](../sources/prs/flashinfer/PR-2574.md), [tests: add bias testing to nvfp4 moe](../sources/prs/flashinfer/PR-2585.md), [Perf: Optimize GDN decode pretranspose kernel for all batch sizes](../sources/prs/flashinfer/PR-2588.md), [support qk_nope_head_dim for 192 check for GLM-5](../sources/prs/flashinfer/PR-2607.md), [Ameyn/gdn bf16 tolerance parallel reduction](../sources/prs/flashinfer/PR-2610.md), [perf(gdn): optimize MTP kernel with ILP rows and SMEM v caching](../sources/prs/flashinfer/PR-2618.md), [feat: add pool+indices support to gated_delta_rule_decode_pretranspose (bf16 path) ](../sources/prs/flashinfer/PR-2619.md), [fix: trtllm_mxint4_block_scale_moe unit test to index output list](../sources/prs/flashinfer/PR-2627.md), [benchmark: Enable speculative decode microbenchmarking for paged decode](../sources/prs/flashinfer/PR-2628.md), [benchmark: Add MXFP4/MXFP8 quantization mode support to FP4 MoE benchmark](../sources/prs/flashinfer/PR-2635.md), [Enable sm120f compilation](../sources/prs/flashinfer/PR-2650.md), [fix: Add fused MOE and GEMM AOT modules for SM121](../sources/prs/flashinfer/PR-2654.md), [benchmarks: Add FP8 input / BF16 output in ragged prefill benchmark](../sources/prs/flashinfer/PR-2666.md), [feat(gdn): add BF16 state kernel with MTP support beyond T>4 with intermediate caching.](../sources/prs/flashinfer/PR-2679.md), [fix(jit): GEMM kernels produce NaN under concurrency — missing GDC flags cause PDL synchronization barriers to compile as no-ops](../sources/prs/flashinfer/PR-2716.md), [[gdn] support non-contiguous state for decoding](../sources/prs/flashinfer/PR-2727.md), [Support in-place update for `trtllm_fp8_block_scale_moe`](../sources/prs/flashinfer/PR-2739.md), [[Spark unit test debugging] Fix for tests/attention/test_trtllm_gen_mla.py](../sources/prs/flashinfer/PR-2750.md), [[Spark unit test debugging] Fix for tests/gemm/test_groupwise_scaled_gemm_fp8.py](../sources/prs/flashinfer/PR-2751.md), [perf: Performance tune cute dsl RMSNorm variants](../sources/prs/flashinfer/PR-2777.md), [fix(jit): enable GDC for CUTLASS GEMM PDL — SM100 flag only](../sources/prs/flashinfer/PR-2780.md), [tests: skip sliding window + fp8 to prevent hang in fmha_v2 unit tests](../sources/prs/flashinfer/PR-2781.md), [[fix] Bugfix 1367: fix VariableBlockSparseAttention buffer overflow by dynamically resizing kv_lens_buffer](../sources/prs/flashinfer/PR-2802.md), [feat(gdn): add padding index guard for bf16 decode kernel](../sources/prs/flashinfer/PR-2810.md), [[Spark unit test] Adjust tolerance for test_xqa, test_logits_processor](../sources/prs/flashinfer/PR-2828.md), [perf: Optimize GDN MTP decode kernel (v15) — eliminate ilp=1 fallback…](../sources/prs/flashinfer/PR-2842.md), [fix: fix cute dsl swap_ab tactic failure](../sources/prs/flashinfer/PR-2870.md), [fix: add cute dsl moe utils to AOT](../sources/prs/flashinfer/PR-2872.md), [[fix] bugfix 2856: Fix pre-allocated out shape check in trtllm_batch_decode_with_kv_cache_mla for q_len_per_req > 1](../sources/prs/flashinfer/PR-2876.md), [[NVIDIA] fix(jit): enable GDC for CUTLASS fused MoE PDL — prevent random crashes on SM12x](../sources/prs/flashinfer/PR-2913.md), [fix: Fix autotuner crash on meta-device tensor in trtllm_fp4_block_scale_routed_moe](../sources/prs/flashinfer/PR-2916.md), [feat: SM121 (GB10) tile filtering and autotuner robustness](../sources/prs/flashinfer/PR-2927.md), [CuTe DSL FP4 GEMM Heuristic](../sources/prs/flashinfer/PR-2940.md), [[Perf] Refactor MoE autotuning to set valid topk ids in routed MoE tuning](../sources/prs/flashinfer/PR-2942.md), [Only swizzle on v block scale; rename kv_block_scales to kv_cache_sf](../sources/prs/flashinfer/PR-2954.md), [Update NVSHMEM interface to use NVSHMEM4Py instead of custom bindings](../sources/prs/flashinfer/PR-2960.md), [test: skip unsupported mm_mxfp8 configurations on SM12x](../sources/prs/flashinfer/PR-2974.md), [feat(comm): add MOE Finalize/Reduction patterns to unified allreduce_fusion API](../sources/prs/flashinfer/PR-2982.md), [ Fix MXFP4/MXFP8 failures in SM120 FAST_BUILD and expand all_tiles[] ](../sources/prs/flashinfer/PR-2994.md), [[feat] Add blackwell GDN prefill kernel](../sources/prs/flashinfer/PR-3001.md), [fix: use sym_int64 for strides in rmsnorm CuTe DSL kernels to prevent int32 overflow](../sources/prs/flashinfer/PR-3007.md), [[chore] Install nvidia-cutlass-dsl[cu13] for cu130+](../sources/prs/flashinfer/PR-3017.md), [[feat] Add routing_replay_out support to MoE kernels and Python API](../sources/prs/flashinfer/PR-3024.md), [Fix/3170 dense blockscaled sm12x](../sources/prs/flashinfer/PR-3180.md), [test: enable bmm_mxfp8 cutlass backend coverage on SM12x](../sources/prs/flashinfer/PR-3183.md), [Include TinyGEMM into BF16 autotuner](../sources/prs/flashinfer/PR-3203.md), [feat(trace): embed runnable init() in every TraceTemplate](../sources/prs/flashinfer/PR-3221.md), [Ameyn/gdn bf16 dispatcher and 4d pool](../sources/prs/flashinfer/PR-3268.md), [fix(fmha_v2): fix FP8 V-scratch pipeline and varlen scheduler on SM90](../sources/prs/flashinfer/PR-3276.md), [Ep api design - Build Infra dependencies](../sources/prs/flashinfer/PR-3315.md), [feat: Separate QK/VO head dim dispatch for sm90 AOT](../sources/prs/flashinfer/PR-778.md), [bugfix: fix batch prefill attention kernel unittests](../sources/prs/flashinfer/PR-781.md), [bugfix: fix the behavior of mla plan function when provided with host tensors](../sources/prs/flashinfer/PR-816.md), [unittest: add MLA test cases where kv_len is evenly divided by page_size.](../sources/prs/flashinfer/PR-861.md), [perf: reduce torch.library dispatch overhead](../sources/prs/flashinfer/PR-968.md), [perf: Fix python API overhead when CUDAGraph is not enabled](../sources/prs/flashinfer/PR-969.md), [Update torch-xpu-ops commit pin](../sources/prs/pytorch/PR-144209.md), [[inductor][cpu] Fix bmm b_index for dynamic expressions in inductor autotuner](../sources/prs/pytorch/PR-144248.md), [Fix PythonMod printing](../sources/prs/pytorch/PR-144335.md), [Remove runtime dependency on packaging](../sources/prs/pytorch/PR-149125.md), [Add AOTI shim for _weight_int4pack_mm_cpu_tensor (#149031)](../sources/prs/pytorch/PR-149386.md), [op should NOT be static in aoti_torch_call_dispatcher](../sources/prs/pytorch/PR-149644.md), [Dont exclude constant_pad_nd in prologue fusion](../sources/prs/pytorch/PR-150145.md), [[inductor] Fix inductor windows linker error](../sources/prs/pytorch/PR-150447.md), [[Windows][inductor] fix blank space break windows file path](../sources/prs/pytorch/PR-150448.md), [[dynamo][super variable] Fix bug to use correct source](../sources/prs/pytorch/PR-152774.md), [[FlexAttention] Remove Old Constraint on lastdim strides](../sources/prs/pytorch/PR-153104.md), [Mark auto_functionalized HOPs as cacheable (#151194)](../sources/prs/pytorch/PR-153304.md), [[FlexAttention] explicilty create grad_q w/ strides](../sources/prs/pytorch/PR-153641.md), [[MPS] Switch Cholesky decomp to column wise](../sources/prs/pytorch/PR-158237.md), [Add warning about removed sm50 and sm60 arches](../sources/prs/pytorch/PR-158301.md), [[CD] CUDA 13 specific followup changes. Remove sm50-70 From CUDA 12.6 and CUDA 12.8 builds](../sources/prs/pytorch/PR-162455.md), [fix cpp extension distributed warning spew](../sources/prs/pytorch/PR-162764.md), [[Cherry Pick][Graph Partition] allow sharing default device context](../sources/prs/pytorch/PR-163097.md), [[Release 2.9] [cuDNN][SDPA][submodule] Roll-back cuDNN frontend upgrade, update Met…](../sources/prs/pytorch/PR-163265.md), [CUDA 13.0 Warning update for supported architectures](../sources/prs/pytorch/PR-163585.md), [fix pickling for BitwiseFn](../sources/prs/pytorch/PR-163861.md), [[SDPA] [MPS] Fixes regression in 2.8.0 for scaled_dot_product_attention using mps](../sources/prs/pytorch/PR-164364.md), [[Flex attention] Fix flex attention head broadcast](../sources/prs/pytorch/PR-164368.md), [[inductor] don't try to reorder loops for template](../sources/prs/pytorch/PR-166910.md), [[Dynamo] Don't guard data ptrs by default with mark_static_address](../sources/prs/pytorch/PR-166913.md), [[Inductor] No longer throw error in bmm out_dtype lowering due to tem…](../sources/prs/pytorch/PR-166922.md), [[GraphPartition] cache get_free_symbol_uses (#166338)](../sources/prs/pytorch/PR-166994.md), [[cuDNN][SDPA] Check-in test for #166211](../sources/prs/pytorch/PR-167121.md), [[Inductor] ExternKernelBenchmarkRequest best attempt](../sources/prs/pytorch/PR-170246.md), [[flex_attention] adds support for low precision K/V inputs in compiled mode with GPU](../sources/prs/pytorch/PR-170486.md), [[cherry-pick] Fix vllm issue for flex (#170499)](../sources/prs/pytorch/PR-170555.md), [Avoid closing random file handles in Inductor](../sources/prs/pytorch/PR-171150.md), [[xpu][fix][inductor] fallback bfloat16 atomics to eager](../sources/prs/pytorch/PR-171247.md), [[MPS] Fix 2-pass SDPA memory corruption by forcing float accumulators](../sources/prs/pytorch/PR-175580.md), [[CI] Update inductor CI jobs to CUDA 13.0](../sources/prs/pytorch/PR-175826.md), [[Inductor] Reject non-contiguous subnode fusion in mix-order reduction.](../sources/prs/pytorch/PR-176410.md), [[inductor] Fix Identity comparability and evalf recursion](../sources/prs/pytorch/PR-176783.md), [[Inductor] Don't unfuse addmm for bf16/fp16 to avoid precision loss](../sources/prs/pytorch/PR-177144.md), [[Inductor][MPS] Fix half-precision type mismatches in Metal shader codegen (#176436)](../sources/prs/pytorch/PR-177193.md), [[MPS] fix compiling of SDPA producing nan results](../sources/prs/pytorch/PR-178009.md), [feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA](../sources/prs/sglang/PR-10078.md), [[Feature] Add MLAProcess for DeepSeek MLA on NPU](../sources/prs/sglang/PR-10130.md), [Enable native ModelOpt quantization support (3/3)](../sources/prs/sglang/PR-10154.md), [Fix chunked prefix cache for nvfp4](../sources/prs/sglang/PR-10180.md), [Add support for bf16 x bf16 cutlass fused MoE](../sources/prs/sglang/PR-10275.md), [support qwen3_next blackwell](../sources/prs/sglang/PR-10403.md), [Fix cutlass moe accuracy drop caused by attention UB from DP padding mode](../sources/prs/sglang/PR-10414.md), [Support single batch overlap](../sources/prs/sglang/PR-10422.md), [Fix correction bias undefined behavior for nvfp4 models](../sources/prs/sglang/PR-10426.md), [feat: add dsv3 fp4 cutlass moe etp ut](../sources/prs/sglang/PR-10433.md), [Cache the result of `is_blackwell` platform check](../sources/prs/sglang/PR-10498.md), [Enable trtllm mla prefix extend](../sources/prs/sglang/PR-10526.md), [Fix bias handling in TritonMoeQuantInfo within quantization/mxfp4.py](../sources/prs/sglang/PR-10579.md), [support qwen3-next-fp8 deepep](../sources/prs/sglang/PR-10622.md), [[Auto Sync] Update modelopt_quant.py (20250920)](../sources/prs/sglang/PR-10688.md), [Unify SGL Kernel Releases](../sources/prs/sglang/PR-10701.md), [Fix MTP MoE weight loading with NVFP4 target model.](../sources/prs/sglang/PR-10758.md), [Fuse quantize and rope in trtllm_mla MTP](../sources/prs/sglang/PR-10779.md), [[2/2] Support MHA prefill with FlashAttention 4.](../sources/prs/sglang/PR-10937.md), [Quick Fix: fix Qwen3-VL launch failure caused by MRotaryEmbedding arg](../sources/prs/sglang/PR-10985.md), [chore: upgrade sgl-kernel 0.3.13](../sources/prs/sglang/PR-11056.md), [Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization](../sources/prs/sglang/PR-11081.md), [[NVIDIA] Add new SMs support for Spark & Thor](../sources/prs/sglang/PR-11287.md), [[AMD] Clean up vllm dependencies in moe_runner/triton.py](../sources/prs/sglang/PR-11349.md), [[sgl-kernel][1/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-11432.md), [[NVIDIA] FA3/FA4 Fix ](../sources/prs/sglang/PR-11606.md), [Support shared experts overlap in cutlass moe](../sources/prs/sglang/PR-11611.md), [[DeepseekV32] Enable flashmla_prefill kernel with fp8 kvcache](../sources/prs/sglang/PR-11655.md), [Use trtllm_mla decode kernel for draft extend in speculative decoding](../sources/prs/sglang/PR-11664.md), [Support running FP4 Deepseek on SM120.](../sources/prs/sglang/PR-11708.md), [[sgl-kernel] support flashmla libtorch](../sources/prs/sglang/PR-11717.md), [Change bf16 to fp8 for some gemms in attention for DeepSeek ckpt v2](../sources/prs/sglang/PR-11805.md), [Use cutlass fp4 gemm by default](../sources/prs/sglang/PR-11813.md), [Support nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8/NVFP4](../sources/prs/sglang/PR-11866.md), [DeepSeek-V3.2: Add Adaptive MHA Attention Pathway for Short-Sequence Prefill](../sources/prs/sglang/PR-11892.md), [chore: upgrade flashinfer 0.4.1](../sources/prs/sglang/PR-11933.md), [Feature/nano v2 offline modelopt fp8 and nvfp4](../sources/prs/sglang/PR-12018.md), [(1/n)support context parallel with deepseekv3.2-DSA](../sources/prs/sglang/PR-12065.md), [[Ascend] qwen optimization](../sources/prs/sglang/PR-12078.md), [[Ascend][feature] support L1+ L2 radixcache on ascend](../sources/prs/sglang/PR-12214.md), [[DeepseekV32]: use `_concat_mla_absorb_q_general` to replace `torch.cat`](../sources/prs/sglang/PR-12215.md), [[hotfix] missing `w13_weight_fp8` and `w2_weight_fp8` in UE8M0 requantization](../sources/prs/sglang/PR-12259.md), [[Deepseek V3.2] Enable flashmla_auto with MTP](../sources/prs/sglang/PR-12294.md), [fix seqlen bug for trtllm_mla's draft_extend](../sources/prs/sglang/PR-12295.md), [fix: Llama 4 BF16 load on Blackwell](../sources/prs/sglang/PR-12308.md), [fix: llama 4 + trtllm gen + fp8 kv cache incompatibility](../sources/prs/sglang/PR-12347.md), [Replace [silu_and_mul_]scaled_fp4_group_quant by Flashinfer equivalent](../sources/prs/sglang/PR-12376.md), [perf: trtllm mla performance minor improvements](../sources/prs/sglang/PR-12435.md), [Use sgl fp4 quant kernel by default](../sources/prs/sglang/PR-12482.md), [[Ascend] Support enable-mixed-chunk in non-MLA scenarios](../sources/prs/sglang/PR-12491.md), [chore: upgrade flashinfer 0.5.0](../sources/prs/sglang/PR-12523.md), [Enable Flashinfer TRTLLM-GEN-MoE FP8 blockwise kernel for Qwen3-Next on Blackwell](../sources/prs/sglang/PR-12543.md), [[CPU] Fix MoE layer support for DeepSeek-OCR models](../sources/prs/sglang/PR-12555.md), [feat: Add FP4 (E2M1) KV Cache Support for MHA](../sources/prs/sglang/PR-12612.md), [[NVIDIA] Fix wrong symmetric sizes for fp4 cases](../sources/prs/sglang/PR-12640.md), [[sgl-kernel][5/N]Support Expert Specialization Grouped GEMM](../sources/prs/sglang/PR-12666.md), [[fix] Only enable flashinfer all reduce fusion by default for single-node servers](../sources/prs/sglang/PR-12724.md), [[Bugfix] Fix illegal memory access](../sources/prs/sglang/PR-12758.md), [[Ascend] support Kimi-K2-Thinking](../sources/prs/sglang/PR-12759.md), [Update dsv3 quantization auto setting for sm100](../sources/prs/sglang/PR-12778.md), [ignore the deepgemm check when the model weight with nvfp4 and moe ba…](../sources/prs/sglang/PR-12782.md), [[DeepSeek-V3.2][NSA] Enable MHA Pathway for Short Sequence Prefill on B200 (SM100)](../sources/prs/sglang/PR-12788.md), [[Deepseek V3.2] Only skip Indexer logits computation when is_extend_without_speculative](../sources/prs/sglang/PR-12816.md), [Apply moe_reduce_sum kernel for fused_marlin_moe](../sources/prs/sglang/PR-12888.md), [[Deepseek V3.2] Use torch.compile to speed up torch.cat in nsa](../sources/prs/sglang/PR-13022.md), [Support moe topk sigmoid kernel](../sources/prs/sglang/PR-13049.md), [[sgl-kernel] support custom fp8 flashmla kernel](../sources/prs/sglang/PR-13087.md), [support mtp with deepseek r1 nvfp4 model](../sources/prs/sglang/PR-13115.md), [Aiter fp8 kv cache](../sources/prs/sglang/PR-13147.md), [Support internvl on Blackwell (which doesn't support fa3): add `SingletonCache` support to Vision{Sdpa|Triton|Ascend}Attention](../sources/prs/sglang/PR-13151.md), [[NPU]Optimization of `forward_npu` for `UnquantizedFusedMoEMethod`](../sources/prs/sglang/PR-13158.md), [Fix nan in global scaling factor for large scale nvfp4 EP](../sources/prs/sglang/PR-13162.md), [diffusion: enable fa4 for blackwell](../sources/prs/sglang/PR-13263.md), [[NVIDIA] Fix broken fp8 MoE of deepseek v3](../sources/prs/sglang/PR-13264.md), [[NVIDIA] Fix use case of SGLANG_ENABLE_FLASHINFER_GEMM](../sources/prs/sglang/PR-13274.md), [Support weight update for blackwell DeepGEMM](../sources/prs/sglang/PR-13324.md), [Flashinfer TRTLLM-GEN-MoE + Qwen3](../sources/prs/sglang/PR-13489.md), [Fix target MLA with eagle3 support for PD disaggregation](../sources/prs/sglang/PR-13555.md), [[BugFix] fix prefixcache performance and accuracy on ascend](../sources/prs/sglang/PR-13573.md), [[ROCM] Optimized deepseek-r1 fp8 model with + triton_gemm_a8w8 + batch_gemm_a8w8 + fused set_mla_kv_buffer kernel](../sources/prs/sglang/PR-13617.md), [[DeepSeekV3.2] Enable pure TP & Partial DP Attention](../sources/prs/sglang/PR-13646.md), [Fix EPLB + FP4 Quantization Compatibility Issue](../sources/prs/sglang/PR-13715.md), [[bugfix] fix TBO crashes when attn_tp_size > 1](../sources/prs/sglang/PR-13730.md), [[sgl-kernel][Feat][B200][1/N]Support MXFP8 Grouped GEMM in Blackwell](../sources/prs/sglang/PR-13731.md), [fix trtllm mla spec](../sources/prs/sglang/PR-13738.md), [[AMD] Support --enable-aiter-allreduce-fusion on AMD GPUs](../sources/prs/sglang/PR-13747.md), [[chore]Upgrade flashinfer to 0.5.3](../sources/prs/sglang/PR-13751.md), [[Feat][NVFP4] Enable NVFP4 MoE for Qwen series models (eg. Qwen3-Next) #13761](../sources/prs/sglang/PR-13761.md), [Support fp4 fp8 non gated moe](../sources/prs/sglang/PR-13794.md), [[NVIDIA] Enable TRTLLM BF16 MoE on Blackwell GPUs](../sources/prs/sglang/PR-13798.md), [update flashinfer_cubin==0.5.3](../sources/prs/sglang/PR-13848.md), [[BugFix] fix outplace_fused_experts missing is_gated](../sources/prs/sglang/PR-13864.md), [Feat: GLM-4.6 supports shared experts fusion](../sources/prs/sglang/PR-13873.md), [Fix update weight error for blackwell DeepGEMM](../sources/prs/sglang/PR-13910.md), [[DeepSeek v3.2] opt Context Parallelism: support fused moe, multi batch and fp8 kvcache](../sources/prs/sglang/PR-13959.md), [Use trtllm mha decode kernel for target_verify in speculative decoding](../sources/prs/sglang/PR-13976.md), [Support KTransformers for Qwen3-VL moe](../sources/prs/sglang/PR-13983.md), [Fix flashinfer cutlass MoE output shape for non-FP4-packed inputs](../sources/prs/sglang/PR-14028.md), [[LoRA][III] Add LoRA support for MoE layers and enable TP](../sources/prs/sglang/PR-14105.md), [Add new moe wna16 marlin gemm](../sources/prs/sglang/PR-14122.md), [Apply new moe wna16 marlin gemm](../sources/prs/sglang/PR-14125.md), [Apply new moe align block size kernel](../sources/prs/sglang/PR-14134.md), [Support checking fp8 params in weight_checker](../sources/prs/sglang/PR-14147.md), [fix: Increase FlashInfer workspace size for Qwen3VL models](../sources/prs/sglang/PR-14173.md), [Add Mistral Large 3 support.](../sources/prs/sglang/PR-14213.md), [[bug fix] fix ima with get_mla_kv_buffer_kernel overflow](../sources/prs/sglang/PR-14224.md), [Tiny use trtllm_mha as default when possible](../sources/prs/sglang/PR-14291.md), [[Fix] add block size logic for sm120 smem size](../sources/prs/sglang/PR-14311.md), [[FIX] trtllm-moe-fp4-renorm for Qwen series models](../sources/prs/sglang/PR-14350.md), [Support FP8 MLA prefill and 128k context.](../sources/prs/sglang/PR-14395.md), [[NPU] perf update with kvcache nz & w4a8 quant](../sources/prs/sglang/PR-14423.md), [Add Mistral Large 3 Eagle Support](../sources/prs/sglang/PR-14466.md), [Mistral Large 3 NVFP4 support](../sources/prs/sglang/PR-14485.md), [[diffusion] kernel fusion: gated residual layernorm scale shift and layernorm scale shift kernel fusion for Qwen-Image, WAN and HunyuanVideo](../sources/prs/sglang/PR-14717.md), [[NPU][eagle3] support qwen eagle3 on NPU](../sources/prs/sglang/PR-14820.md), [Apply back moe_sum_reduce for fused_marlin_moe](../sources/prs/sglang/PR-14829.md), [fix: trtllm mha attention auto-selection on sm120](../sources/prs/sglang/PR-14842.md), [Fix dsv3 dp accuracy issue when using bf16-kv](../sources/prs/sglang/PR-14897.md), [Fix accuracy issue when using a16w16 mla_decode_fwd](../sources/prs/sglang/PR-14936.md), [[AMD] Support fused_rms_mxfp4_quant in the prefill stage for DeepSeek-R1-MXFP4](../sources/prs/sglang/PR-14975.md), [add transformers version validation for glm-4.6v moe models](../sources/prs/sglang/PR-14998.md), [Mistral Large 3 NVFP4 TRTLLM MoE support](../sources/prs/sglang/PR-15049.md), [fix(attention): Prevent trtllm_mha auto-selection with eagle3 speculative decoding](../sources/prs/sglang/PR-15127.md), [[sgl-kernel][1/2] Fused qk_norm_rope for GLM4.6](../sources/prs/sglang/PR-15141.md), [[NVIDIA] upstream FA4](../sources/prs/sglang/PR-15182.md), [[NVIDIA] Fixes for NVFP4 all-gather with spec decoding](../sources/prs/sglang/PR-15280.md), [[Fix] A followup fix for TRTLLM BF16 MoE](../sources/prs/sglang/PR-15303.md), [Fix the accuracy issue when running mxfp4 dsv3 model and enable ep](../sources/prs/sglang/PR-15304.md), [feat: support bitsandbytes quantization algorithm](../sources/prs/sglang/PR-15325.md), [[distributed] Clean up MoE groups in destroy_model_parallel](../sources/prs/sglang/PR-15345.md), [[Tiny]Add warning for deepgemm on Blackwell](../sources/prs/sglang/PR-15352.md), [[NPU]mindspore model support moe](../sources/prs/sglang/PR-15363.md), [[NPU]DeepSeek-V3.2 support npu mlaprolog](../sources/prs/sglang/PR-15381.md), [[diffusion] Add Sage Attention 3 Support for sm 120 (RTX5090)](../sources/prs/sglang/PR-15382.md), [Super tiny add moe_ep_rank to prometheus labels](../sources/prs/sglang/PR-15407.md), [Flashinfer MOE FP8 support for Mistral Large 3.](../sources/prs/sglang/PR-15422.md), [Optimize MiMo-V2-Flash by flashinfer fused allreduce](../sources/prs/sglang/PR-15464.md), [[Perf] Add Flashinfer DeepGEMM SM90 for SwapAB Optimization](../sources/prs/sglang/PR-15514.md), [Optimize FP8 MLA KV cache writes with Triton kernel](../sources/prs/sglang/PR-15522.md), [Optimize Bailing-MoE with FlashInfer Fused All-Reduce](../sources/prs/sglang/PR-15526.md), [MoE: Skip SiLU/GELU activation for masked experts](../sources/prs/sglang/PR-15539.md), [Update flashinfer to 0.6.1](../sources/prs/sglang/PR-15551.md), [[sgl-kernel] Streamline kernel size report (Top 20 only) and clean up](../sources/prs/sglang/PR-15552.md), [Fix BatchMLAPagedAttentionWrapper query/qo_inptr mismatch for EAGLE](../sources/prs/sglang/PR-15601.md), [[jit-kernel] Add CuTe DSL GDN Decode Kernel](../sources/prs/sglang/PR-15631.md), [Add SwapAB Optimization for triton fused_moe_kernel on SM90.](../sources/prs/sglang/PR-15712.md), [[Perf] Eliminate the slice op for Flashinfer `trtllm_fp4_block_scale_moe`](../sources/prs/sglang/PR-15731.md), [Fix GLM-4.7 MoE Detector complex JSON Schema type parsing](../sources/prs/sglang/PR-15753.md), [Fix: Handle empty func_name and None values in GLM MoE detectors](../sources/prs/sglang/PR-15754.md), [[Feature] JIT Fused QK norm + qk norm clean up](../sources/prs/sglang/PR-15835.md), [[JIT kernel] Apply jit per_tensor_quant_fp8 kernel](../sources/prs/sglang/PR-15836.md), [[diffusion] model: support TurboWan2.1-T2V-1.3B/14B SLA](../sources/prs/sglang/PR-15888.md), [[fix]deepgemm precompile when warmup](../sources/prs/sglang/PR-15891.md), [Bugfix for ds-vl2](../sources/prs/sglang/PR-15894.md), [[NPU] NZ for non-quantized MOE, Qwen3 MOE double memory consumption fix](../sources/prs/sglang/PR-15904.md), [ Add tuned triton==3.5.1 h200 tp2, tp4 for qwen 3 next](../sources/prs/sglang/PR-15948.md), [Tiny fix cannot launch nvfp4 checkpoint with bf16 kv cache](../sources/prs/sglang/PR-15986.md), [[Performance] Force split_k=1 for MXFP4 Triton kernels on Hopper](../sources/prs/sglang/PR-16014.md), [Support fa4 decoding](../sources/prs/sglang/PR-16034.md), [optimize get_topk_ragged by fusing get k and k_scale triton kernel](../sources/prs/sglang/PR-16043.md), [[Diffusion] Flux support flashinfer rope](../sources/prs/sglang/PR-16055.md), [enhance accuracy for model kimi-vl-instruct-a3b](../sources/prs/sglang/PR-16076.md), [fix layer intermediate size](../sources/prs/sglang/PR-16084.md), [[Diffusion] Zimage opt with qknorm and flashinfer rope](../sources/prs/sglang/PR-16161.md), [[Feature] add aligned_vector type for JIT kernel](../sources/prs/sglang/PR-16162.md), [[VLM] Adopt jit qk_norm kernel in VLM](../sources/prs/sglang/PR-16171.md), [[NemotronH] Add latent MoE support](../sources/prs/sglang/PR-16227.md), [[Fix] Only add SM90 and SM100 to check for auto-enabling TRT Allreduce Fusion](../sources/prs/sglang/PR-16283.md), [[diffusion] Fix RuntimeError in SageAttention3 on Nvidia Blackwell with Qwen-Image](../sources/prs/sglang/PR-16335.md), [[Fix]Fix FA3 Performance in Diffusion Model ](../sources/prs/sglang/PR-16382.md), [Fix FP8 MoE NaN with DeepGEMM on Blackwell](../sources/prs/sglang/PR-16622.md), [[Rework] Add SwapAB Optimization for triton fused_moe_kernel on SM90.](../sources/prs/sglang/PR-16723.md), [[AMD] Support redundant expert with a2a moe in gfx95x.](../sources/prs/sglang/PR-16791.md), [[Fix] `flashinfer_trtllm` `intermediate_size` assertion with Qwen3 + TP=8](../sources/prs/sglang/PR-16824.md), [Support mxint4 flashinfer_trtllm moe gemm](../sources/prs/sglang/PR-16892.md), [[DeepSeek v3.2] Opt MTP decode cuda batch sizes and nsa implementation](../sources/prs/sglang/PR-16961.md), [[NPU]bugfix: fix for dsv3.2 and dsvl2](../sources/prs/sglang/PR-17007.md), [[MUSA][2/N] sgl-kernel build](../sources/prs/sglang/PR-17053.md), [Optimize GDN decode for Qwen3 Next](../sources/prs/sglang/PR-17094.md), [[diffusion] fix: fix using upstream flash_attn on blackwell](../sources/prs/sglang/PR-17111.md), [Enable XQA for SM90 and SM120](../sources/prs/sglang/PR-17115.md), [Inclusion of nvfp4 blockscale in EPLB Rebalance](../sources/prs/sglang/PR-17158.md), [[Fix] GLM 4.7 + NVFP4 + MTP](../sources/prs/sglang/PR-17166.md), [[GLM 4.7] Add RTX 6000 Pro aka sm120](../sources/prs/sglang/PR-17235.md), [[New Model] GLM4.7-Flash](../sources/prs/sglang/PR-17247.md), [[FIX] Always support TP > 4 for FP4 Gemm](../sources/prs/sglang/PR-17300.md), [Disable mla persistent kernel when not using fp8 kv_cache](../sources/prs/sglang/PR-17327.md), [Move fa4 from sgl-kernel to jit kernel](../sources/prs/sglang/PR-17353.md), [Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE](../sources/prs/sglang/PR-17449.md), [[NPU] enhance accuracy for model kimi-vl-a3b-instruct](../sources/prs/sglang/PR-17480.md), [Kernel: optimize decoding metadata in NSA multi-spec backend with fused kernels](../sources/prs/sglang/PR-17554.md), [[hotfix] Reenable all reduce fusion on sm100](../sources/prs/sglang/PR-17591.md), [[feat] Support nvfp4 quantized model of Qwen3-Next](../sources/prs/sglang/PR-17627.md), [Upgrade transformers==5.3.0](../sources/prs/sglang/PR-17784.md), [fix(quantization): add sgl_kernel fallback for FP4 quantize on Blackwell GPUs](../sources/prs/sglang/PR-17816.md), [Feature/support longcat flash lite](../sources/prs/sglang/PR-17838.md), [[Move sgl-kernel Kernel to JIT] Add JIT concat MLA kernels](../sources/prs/sglang/PR-17889.md), [Skipped warning on sm100](../sources/prs/sglang/PR-18000.md), [[Bugfix] Fix Mistral Large 3 NVFP4 TRTLLM MoE](../sources/prs/sglang/PR-18065.md), [Feat/add fi selective state update kernel call](../sources/prs/sglang/PR-18070.md), [[Diffsuion & JIT_kernel] QKNorm cross heads kernel](../sources/prs/sglang/PR-18073.md), [Fix nvfp4 weight update](../sources/prs/sglang/PR-18085.md), [[Blackwell] Make mxint4 flashinfer_trtllm moe gemm set by default on blackwell](../sources/prs/sglang/PR-18136.md), [[ModelOpt] Fix broken Qwen3-235B-A22B-Instruct-2507-NVFP4 launch](../sources/prs/sglang/PR-18189.md), [[ModelOPT] Support Qwen 3 Next Coder NVFP4](../sources/prs/sglang/PR-18224.md), [Support Qwen3 MoE context parallel](../sources/prs/sglang/PR-18233.md), [[ROCm] Optimize Deepseek R1 on MI300X](../sources/prs/sglang/PR-18242.md), [[Hicache & JIT_kernel] Support page first layout & mla jit kernel](../sources/prs/sglang/PR-18311.md), [[AMD] Support Qwen3-Coder-Next on AMD platform](../sources/prs/sglang/PR-18355.md), [[MUSA][10/N] Add GGUF support](../sources/prs/sglang/PR-18357.md), [feat(gdn): add FlashInfer K-last SSM layout support for GDN prefill and decode for Hopper](../sources/prs/sglang/PR-18361.md), [[Kimi-K2.5] Fix NVFP4 Kimi-K2.5 weight mapping and exclude list](../sources/prs/sglang/PR-18370.md), [Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4](../sources/prs/sglang/PR-18389.md), [[AMD] Update aiter to v0.1.10.post2](../sources/prs/sglang/PR-18423.md), [feat: add FA4 SM90 paged KV decode support & update attention docs](../sources/prs/sglang/PR-18442.md), [Tilelang sparse decode fwd for dsv32 mi355](../sources/prs/sglang/PR-18488.md), [[FIX] Correct JIT kernel compilation on newer GPUs with outdated driver metadata.](../sources/prs/sglang/PR-18496.md), [Fp8 prefill attn kernel integration](../sources/prs/sglang/PR-18528.md), [[AMD] Fix accuracy issue when running TP4 dsv3 model with mtp](../sources/prs/sglang/PR-18607.md), [[AMD] DSR1/V3 use fp8 bmm in MLA for MI300X](../sources/prs/sglang/PR-18624.md), [[sglang-miles] True on-policy training support for FSDP2](../sources/prs/sglang/PR-18639.md), [use flashinfer.sampling](../sources/prs/sglang/PR-18696.md), [[RL] Support per-layer mixed FP8/BF16 serving for FP8 checkpoints](../sources/prs/sglang/PR-18742.md), [fix: update Blackwell log/error messages to include SM12x](../sources/prs/sglang/PR-18751.md), [[diffusion] Diffusion norm fusion for z-image](../sources/prs/sglang/PR-18762.md), [fix: add SM110 (Jetson AGX Thor) to Blackwell capability check](../sources/prs/sglang/PR-18787.md), [Migrate renorm kernels from sgl-kernel to FlashInfer JIT](../sources/prs/sglang/PR-18854.md), [[Perf] ~9.5x faster Blackwell MXFP4 MoE weight loading](../sources/prs/sglang/PR-18858.md), [Migrate norm kernels to FlashInfer JIT implementation](../sources/prs/sglang/PR-18871.md), [[sgl-kernel] rebase FlashMLA 0217](../sources/prs/sglang/PR-18902.md), [Fix NSA FP8 KV cache path for both-trtllm MHA one-shot](../sources/prs/sglang/PR-18931.md), [[Qwen3.5] Enable nvfp4 checkpoint](../sources/prs/sglang/PR-18937.md), [[Sarvam] Add inference support for Sarvam MoE LLMs](../sources/prs/sglang/PR-18938.md), [[jit_kernel] Add fused_qknorm_rope JIT kernel](../sources/prs/sglang/PR-19059.md), [Support skip-softmax attention](../sources/prs/sglang/PR-19089.md), [feat: Support MXFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-19143.md), [[DeepSeek-V3.2][JIT-kernel] Support nsa fuse store indexer k cache](../sources/prs/sglang/PR-19148.md), [[NVIDIA] Integrate FlashInfer decode kernel (Blackwell) for Qwen3.5](../sources/prs/sglang/PR-19150.md), [Adjust padding size to improve triton_kernels moe performance](../sources/prs/sglang/PR-19174.md), [[AMD] Fix accuracy while using --enable-dp-attention](../sources/prs/sglang/PR-19247.md), [Fix nightly Mistral-Large-3 NVFP4 accuracy threshold](../sources/prs/sglang/PR-19402.md), [[AMD] Fix weight load shape mismatch for amd dsr1 0528 mxfp4](../sources/prs/sglang/PR-19425.md), [[Feature] add feature mla_ag_after_qlora for dsv3.2](../sources/prs/sglang/PR-19428.md), [Fix/nemotron mtp quantaized](../sources/prs/sglang/PR-19433.md), [[Kernel Slimming] Migrate NVFP4 kernels to JIT](../sources/prs/sglang/PR-19437.md), [[FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE](../sources/prs/sglang/PR-19537.md), [[NPU] bugs fix for Deepseek models](../sources/prs/sglang/PR-19544.md), [[diffusion][llm] macOS support](../sources/prs/sglang/PR-19549.md), [[miles] fix for glm5](../sources/prs/sglang/PR-19634.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [Support `triton_kernels` for GPT-OSS on SM120](../sources/prs/sglang/PR-19718.md), [Various SM120 improvements](../sources/prs/sglang/PR-19721.md), [[SGLang-Diffusion] Fix custom op fake impl missing eps default for torch.compile](../sources/prs/sglang/PR-19725.md), [Add compile-time 256-bit vector guard for pre-Blackwell](../sources/prs/sglang/PR-19794.md), [[JIT Kernel][Feature] Support JIT custom all reduce (rewrite as v2)](../sources/prs/sglang/PR-19880.md), [Use TRTLLM allreduce fusion for Qwen 3.5](../sources/prs/sglang/PR-19889.md), [Fix MLA decode path returning unwritten (padded) rows](../sources/prs/sglang/PR-19902.md), [[AMD] Fix Tensor Memory Aliasing ](../sources/prs/sglang/PR-19928.md), [[AMD] Fix FP8 assertion failure in aiter MLA decode by falling back to self.k_scale](../sources/prs/sglang/PR-19935.md), [[AMD] Tilelang sparse fwd for dsv32 mi355/mi300](../sources/prs/sglang/PR-19945.md), [[JIT Kernel] Reland NVFP4 kernels to JIT](../sources/prs/sglang/PR-20012.md), [[Bugfix] Work around FlashInfer unified transport issue on GB](../sources/prs/sglang/PR-20039.md), [Fix SM120 `triton_kernels` MXFP4 `block_k` for GPT-OSS](../sources/prs/sglang/PR-20040.md), [MiniMax-M2.5 - Support dp attention, dp reduce scatter, FP4 all gather, AR fusion in prepare_attn](../sources/prs/sglang/PR-20067.md), [Fix streaming session with paged KV cache (SWA/MLA)](../sources/prs/sglang/PR-20070.md), [Enable modelopt quantized FLUX deployment](../sources/prs/sglang/PR-20082.md), [[V32/GLM5] Change default setting of V32 nvfp4 on TP4](../sources/prs/sglang/PR-20086.md), [[diffusion] fix bug of copy_if](../sources/prs/sglang/PR-20094.md), [[diffusion] Support nvfp4 for Flux.2](../sources/prs/sglang/PR-20137.md), [[AMD] Fp8 prefill integration with radix cache path for dpsk models](../sources/prs/sglang/PR-20187.md), [[4/n jit_kernel restruct] speed up CI tests and add benchmark workflow](../sources/prs/sglang/PR-20268.md), [[AMD] Add 4-GPU test suite for MI325 runners](../sources/prs/sglang/PR-20294.md), [[Benchmark] use flashinfer bench_gpu_time instead of triton do_bench](../sources/prs/sglang/PR-20305.md), [[Fix] Add fallback for flashinfer allreduce fusion](../sources/prs/sglang/PR-20384.md), [[NVIDIA] Enable fp8 flashinfer_trtllm_routed MoE for MiniMax-M2.5](../sources/prs/sglang/PR-20394.md), [[AMD][Bug-fix] Fix gpu fault when run the test with dp-attention-enabled and max-concurrency is over 256](../sources/prs/sglang/PR-20399.md), [[Model] Support Nemotron 3 Super NVFP4](../sources/prs/sglang/PR-20407.md), [[AMD][AITER] Guard _use_mla_ps_kernel with self.use_mla in draft_extend_v2 paths](../sources/prs/sglang/PR-20409.md), [[GDN] Add benchmark for sglang gdn prefill](../sources/prs/sglang/PR-20428.md), [Support Triton MLA FP8 KV cache](../sources/prs/sglang/PR-20479.md), [[Kernel] Fuse temperature + softmax in sampling for decode speedup](../sources/prs/sglang/PR-20501.md), [[Diffusion] Clean upstream fa3 in hopper](../sources/prs/sglang/PR-20576.md), [Use Flashinfer for target_verify in GDN model for SM120](../sources/prs/sglang/PR-20604.md), [FIX: (NSA) Compute topk_indices_offset when NSA prefill flashmla_sparse is used with FP8 KV cache](../sources/prs/sglang/PR-20606.md), [[Diffusion] Add a benchmark for rmsnorm/fuse_add_rmsnorm](../sources/prs/sglang/PR-20632.md), [Fix(jit): support rmsnorm for hidden_size in {64, 128, 256}](../sources/prs/sglang/PR-20661.md), [[Feature][JIT Kernel] Fused TP QK norm For Minimax](../sources/prs/sglang/PR-20673.md), [[Diffusion] Fix compile graph broken by flashinfer rope](../sources/prs/sglang/PR-20699.md), [Add Mistral Small 4 (Pixtral) support](../sources/prs/sglang/PR-20708.md), [Use FlashInfer tinygemm for GPT-OSS MoE router on SM90+](../sources/prs/sglang/PR-20755.md), [fix: guard configure_deep_gemm_num_sms when JIT disabled](../sources/prs/sglang/PR-20868.md), [[JIT Kernel] Fix NVFP4 multi-arch compilation failure](../sources/prs/sglang/PR-20874.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Qwen3.5] Fuse split/reshape/cat ops in GDN projection with Triton kernel](../sources/prs/sglang/PR-21019.md), [[Chore] Clean up JIT compilation flags](../sources/prs/sglang/PR-21022.md), [fix: wrap _import_static_state in inference_mode to fix resume on Blackwell](../sources/prs/sglang/PR-21035.md), [perf: precompute FA3 scheduler_metadata to eliminate per-layer prepare_varlen_num_blocks](../sources/prs/sglang/PR-21104.md), [ci: remove IS_BLACKWELL env var; auto-detect Blackwell](../sources/prs/sglang/PR-21118.md), [[Not-Merge][AMD] GLM-5 performance optimization](../sources/prs/sglang/PR-21166.md), [[Whisper] Enable CUDA graph support and timestamp for whisper model](../sources/prs/sglang/PR-21190.md), [[NPU] bugfix for import sgl-kernel error](../sources/prs/sglang/PR-21200.md), [[KDA] Support CuTeDSL KDA decode kernel](../sources/prs/sglang/PR-21203.md), [[AMD]: Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5…](../sources/prs/sglang/PR-21213.md), [[refactor] Clean up duplicate flashinfer trtllm moe code](../sources/prs/sglang/PR-21233.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [[NVIDIA] Enable FP4 flashinfer trtllm routed moe](../sources/prs/sglang/PR-21240.md), [P2P Weight Update features for miles ](../sources/prs/sglang/PR-21278.md), [[RL] Support mxfp8 DeepSeek V3](../sources/prs/sglang/PR-21280.md), [[MUSA] apply_vocab_mask support musa device](../sources/prs/sglang/PR-21296.md), [CUTLASS NVFP4 GEMM improvement of SM120](../sources/prs/sglang/PR-21314.md), [[Kernel] Support FlashInfer TRTLLM-Gen fused MoE for non-gated FP4 & FP8 (Nemotron)](../sources/prs/sglang/PR-21321.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [Add dedicated FlashInferCuteDslMoE layer for standard-path FP4 MoE](../sources/prs/sglang/PR-21339.md), [[GDN] Fuse GDN kkt + solve_tril into one kernel](../sources/prs/sglang/PR-21411.md), [[Bugfix] Lazy-import CuteDSL KDA kernel to fix AMD/ROCm startup crash](../sources/prs/sglang/PR-21428.md), [fix nemotron capture for non attention layers](../sources/prs/sglang/PR-21436.md), [[Diffusion] Add qknorm rope fuse kernel](../sources/prs/sglang/PR-21440.md), [Add explicit disable flag for FlashInfer allreduce fusion](../sources/prs/sglang/PR-21446.md), [fix: piecewise_cuda_graph get correct qo_indptr](../sources/prs/sglang/PR-21452.md), [Migrate all callers from /get_server_info to /server_info](../sources/prs/sglang/PR-21463.md), [[AMD] Enable FP8 KV cache and FP8 attention kernel for NSA on MI300/MI355 with TileLang backend](../sources/prs/sglang/PR-21511.md), [test: point DSV3 int8 MLA CI models to lmsys Hugging Face org](../sources/prs/sglang/PR-21561.md), [[FlashInver v0.6.7] Integrate flashinfer_trtllm mxfp8 gemm](../sources/prs/sglang/PR-21576.md), [Change default mm-attention backend from triton_attn to fa4](../sources/prs/sglang/PR-21595.md), [fix: TRT-LLM MHA CUDA illegal address with EAGLE v2 + DP attention](../sources/prs/sglang/PR-21649.md), [[jit_kernel] Optimize fused_qknorm_rope: deduplicate sincosf for interleave RoPE ](../sources/prs/sglang/PR-21654.md), [[AMD] Use tgemm.mm for MoEGate router gemm in deepseek_v2.py](../sources/prs/sglang/PR-21657.md), [[XPU] Enable qwen3.5 on XPU](../sources/prs/sglang/PR-21668.md), [[AMD] Add GLM-5-FP8 nightly performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-21710.md), [Harden FlashInfer FP4 imports in standard dispatcher](../sources/prs/sglang/PR-21776.md), [[Fix] Fall back to triton MOE for GPT-OSS on Blackwell with driver >= 595](../sources/prs/sglang/PR-21780.md), [[DSA] Support trtllm sparse mla kernel for prefill batches ](../sources/prs/sglang/PR-21783.md), [Remove redundant test_moe_eval_accuracy_large](../sources/prs/sglang/PR-21787.md), [[Feature] JIT rmsnorm update (with claude)](../sources/prs/sglang/PR-21834.md), [ [GDN] Remove FlashInfer GDN decode + no_buffer guard and default to FlashInfer on SM100+ ](../sources/prs/sglang/PR-21861.md), [[server] Add --quantization unquant to explicitly opt out of quantization](../sources/prs/sglang/PR-21863.md), [[Misc] [MXFP8] Drop sm100 mxfp8 warning](../sources/prs/sglang/PR-21881.md), [fix pcg torch dynamo recompile in mxfp8 Triton path](../sources/prs/sglang/PR-21888.md), [[Bugfix] Temporarily skip TRTLLM attention on (G)B300 (SM103) to avoid high-concurrency hang](../sources/prs/sglang/PR-21906.md), [[DSA] Set trtllm kernels as default for Blackwell](../sources/prs/sglang/PR-21914.md), [[Bugfix] Fix CUDA graph replay issues in trtllm_mla draft_extend](../sources/prs/sglang/PR-21987.md), [Tiny fix trtllm_fp8_per_tensor_scale_moe_wrapper router_logits dtype](../sources/prs/sglang/PR-22006.md), [[NPU] enable mla prepare fused kernel only when being mla attn](../sources/prs/sglang/PR-22024.md), [[MUSA][9/N] Add FA3 attention backend support through MATE (MUSA AI Tensor Engine)](../sources/prs/sglang/PR-22051.md), [[Diffusion] Fix weight scale swizzle and add large-M kernel config for FLUX.2-dev-NVFP4](../sources/prs/sglang/PR-22064.md), [[nvidia] Gemma4 nvfp4 fix](../sources/prs/sglang/PR-22079.md), [[diffusion] Default NVFP4 to CUTLASS and add all-model shape benchmarks](../sources/prs/sglang/PR-22091.md), [[Diffusion] Add diffusion NVFP4 scaled-mm correctness test](../sources/prs/sglang/PR-22127.md), [[Hotfix] Fix router gemm on sm103](../sources/prs/sglang/PR-22134.md), [[Disagg][NIXL] Fix heterogeneous TP KV transfer for non-MLA models (same logic with mooncake, Step 1/2 for Qwen3.5 support)](../sources/prs/sglang/PR-22145.md), [[hisparse]: Adding ci for hisparse kvcache-swap-in jit-kernel](../sources/prs/sglang/PR-22155.md), [[HiSparse]: Add benchmark for hisparse kernel](../sources/prs/sglang/PR-22187.md), [[RL] Refactor NVFP4 shuffling/swizzling to in-place replacement](../sources/prs/sglang/PR-22204.md), [Reduce unnecessary kernels and copies in the NSA indexer](../sources/prs/sglang/PR-22232.md), [[AMD][HIP] NSA: bf16 passthrough from RMSNorm to eliminate FP8 dequantization](../sources/prs/sglang/PR-22258.md), [Lazy import flash_attention_v4 to avoid loading flash_attn.cute at startup](../sources/prs/sglang/PR-22306.md), [[AMD] Fix GLM-5 fp8 KV quant path dispatch on MI300](../sources/prs/sglang/PR-22314.md), [[Reland] DeepSeek-R1-0528-w4a8: DeepEP Low Latency Dispatch Adopts FP8 Communication](../sources/prs/sglang/PR-22316.md), [[Lora] Lora quat info re-factor and support deepseekv3 mla lora](../sources/prs/sglang/PR-22323.md), [[AMD] Add GLM-5.1-FP8 nightly accuracy and performance benchmarks for MI30x and MI35x](../sources/prs/sglang/PR-22336.md), [:sparkles: [diffusion][npu][quant] Add MXFP4 quantization support for Wan2.2 Diffusion on Ascend NPU](../sources/prs/sglang/PR-22338.md), [[Diffusion] modelopt diffusion fp8 support for flux1/flux2 and wan2.2](../sources/prs/sglang/PR-22365.md), [[DSA] Hopper FP8 FlashMLA KV padding](../sources/prs/sglang/PR-22372.md), [[Lora] Lora kimi support](../sources/prs/sglang/PR-22381.md), [[AMD] Use aiter CK layernorm2d for LayerNorm to reduce NSA indexer kernel launches](../sources/prs/sglang/PR-22424.md), [[Fix] Fix several bugs on DSA models](../sources/prs/sglang/PR-22430.md), [Upgrade sglang-torch-profiler-analysis SKILLS](../sources/prs/sglang/PR-22440.md), [[RL] Fix weight update for mxfp8 flashinfer_cutlass gemm backend](../sources/prs/sglang/PR-22484.md), [GLM-5/5.1 MXFP4 Checkpoint Inference Compatibility Fix](../sources/prs/sglang/PR-22543.md), [[Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22574.md), [diffusion: fix layerwise offload for ModelOpt quantized DiTs](../sources/prs/sglang/PR-22594.md), [feat: Support flashinfer_cutedsl MoE runner with flashinfer alltoall backend](../sources/prs/sglang/PR-22669.md), [reland [Diffusion] Add FLUX.1-dev ModelOpt NVFP4 support](../sources/prs/sglang/PR-22672.md), [[Diffusion] Add Wan2.2 ModelOpt NVFP4 support](../sources/prs/sglang/PR-22681.md), [[Step3p5] Optimize allreduce in MoE layers ](../sources/prs/sglang/PR-22773.md), [Dual MoE CUDA graph capture for lora/nolora batches](../sources/prs/sglang/PR-22809.md), [diffusion: add HunyuanVideo GroupNorm+SiLU fast path](../sources/prs/sglang/PR-22814.md), [[Refactor] Refactor DeepEP dispatcher](../sources/prs/sglang/PR-22822.md), [[FlashInfer v0.6.11] [RL] Support FlashInfer per-token NVFP4 MoE](../sources/prs/sglang/PR-22918.md), [[Fix/Kernel] Add JIT rmsnorm_hf kernel to fix transformers backend MMLU accuracy regression ](../sources/prs/sglang/PR-22931.md), [[codex] diffusion: enable group norm silu fuse by default](../sources/prs/sglang/PR-23148.md), [[BugFix] Resolve adaptive speculative decoding conflicts for Qwen3.5 (hybrid GDN)](../sources/prs/sglang/PR-23331.md), [[Diffusion][NPU]Add attention backends for diffusion models for Ascend NPU](../sources/prs/sglang/PR-23482.md), [Reland Cute-DSL FP4 dense GEMM](../sources/prs/sglang/PR-23590.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [Use Cute-DSL NVFP4 quantization kernels](../sources/prs/sglang/PR-23745.md), [feat: port SGLANG_JIT_DEEPGEMM_FAST_WARMUP to deepseek_v4 branch](../sources/prs/sglang/PR-23756.md), [[MoE] Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute](../sources/prs/sglang/PR-23760.md), [Optimize large GroupNorm SiLU apply](../sources/prs/sglang/PR-23938.md), [[feat] Init true on policy with qwen_dense](../sources/prs/sglang/PR-23961.md), [Enable PDL for various kernels in DSV32/GLM5](../sources/prs/sglang/PR-23965.md), [[VLM] Optimize Gemma4 VLM with PCG and fuse RMSNorm + residual add + scalar](../sources/prs/sglang/PR-24048.md), [Refactor device timer, clean up metrics collector, and add fwd occupancy metric](../sources/prs/sglang/PR-24197.md), [[KDA] Optimize prefill kernels with diagonal and recompute fuse](../sources/prs/sglang/PR-24271.md), [[diffusion] Fuse LTX2 split rotary embedding](../sources/prs/sglang/PR-24411.md), [Port MXFP4 Marlin MoE support to JIT kernel path](../sources/prs/sglang/PR-24490.md), [Fix performance regression on Deepseek V3 on `moe-runner-backend=triton` on SM90](../sources/prs/sglang/PR-24562.md), [[Codex] Opt Mistral Large performace ](../sources/prs/sglang/PR-24611.md), [Support spec v2 for FlashMLA speculative decoding](../sources/prs/sglang/PR-24640.md), [[Gemma4] Optimize Gemm4 with fused Q/K/V RMSNorm + per-expert FP8 ckpt loader](../sources/prs/sglang/PR-24696.md), [[codex] Optimize hidden-size 512 RMSNorm dispatch](../sources/prs/sglang/PR-24710.md), [Add FlashInfer SM90 cutlass MXFP4 MoE backend (W4A16) for GPT-OSS + DeepSeek-V4](../sources/prs/sglang/PR-24816.md), [[attn backend] Integrate tokenspeed_mla prefill/decode kernels (fp8 kv cache, blackwell)](../sources/prs/sglang/PR-24925.md), [Amd/deepseek v4 rebase main 0509](../sources/prs/sglang/PR-24933.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [perf(nvfp4): free unused source scales after weight processing](../sources/prs/sglang/PR-25107.md), [Fix AMX GQA extend attention](../sources/prs/sglang/PR-25180.md), [[MUSA][Diffusion] Improve wan model inference speed using torch.compile](../sources/prs/sglang/PR-25256.md), [Support Gemma4 Pipeline Parallelism](../sources/prs/sglang/PR-25284.md), [Fix EPLB mapping for TopK paths](../sources/prs/sglang/PR-25285.md), [[Gemma4]: Fix FP8 Triton scale layout](../sources/prs/sglang/PR-25286.md), [[NSA] Avoid repeated NSA MQA logits memory queries](../sources/prs/sglang/PR-25299.md), [[Intel GPU] Enable DeepSeek V4 Inference on XPU](../sources/prs/sglang/PR-25336.md), [[AMD] test(sgl-kernel): seed RNG on ROCm in test_moe_topk_sigmoid to fix tie-break flake](../sources/prs/sglang/PR-25356.md), [[AMD] Enable shared-experts fusion with new KIMI-K2.5-MXFP4 model.](../sources/prs/sglang/PR-25390.md), [[codex] Update Wan2.2 ModelOpt CI checkpoints](../sources/prs/sglang/PR-25483.md), [Support draft extend cuda graph for tokenspeed_mla attention backend](../sources/prs/sglang/PR-25489.md), [Update logging for inplace setting in MoE layer](../sources/prs/sglang/PR-25499.md), [[misc] Throw error when single batch overlap is enabled on Hopper ](../sources/prs/sglang/PR-25509.md), [Fix logging for inplace setting in the flashInfer-trtllm backend](../sources/prs/sglang/PR-25522.md), [[Bug Fix] Align glm4_moe_nextn NPU MTP loading with qwen3 MTP](../sources/prs/sglang/PR-25524.md), [[MoE Refactor] Migrate flashinfer_cutedsl + DeepEP to MoeRunner](../sources/prs/sglang/PR-25525.md), [Use DeepGEMM BF16 for unquantized DeepEP LL MoE](../sources/prs/sglang/PR-25540.md), [amd/deepseek_v4 27/N [fix] Reduce Triton autotune configs for faster first-time server launch](../sources/prs/sglang/PR-25554.md), [Use triton_attn as default vision attention on B300 (SM103)](../sources/prs/sglang/PR-25570.md), [[Benchmark] Add SGLANG_SIMULATE_UNIFORM_EXPERTS for balanced expert routing with dummy weights](../sources/prs/sglang/PR-25571.md), [Introduce SchedulerDPAttnAdapter to own DP-attention state](../sources/prs/sglang/PR-25611.md), [Move DP-attention adapter methods to SchedulerDPAttnAdapter](../sources/prs/sglang/PR-25612.md), [[SP] Fix runtime_max_tokens_per_rank for sequence parallelism](../sources/prs/sglang/PR-25685.md), [Add no_combine support to cutlass_moe_fp4](../sources/prs/sglang/PR-25688.md), [fix (jit kernel): elementwise activation C++ error](../sources/prs/sglang/PR-25695.md), [[diffusion] Fix GLM-Image /v1/images/edits support](../sources/prs/sglang/PR-25697.md), [[Codex] Remove stale DeepSeek V4 JIT kernels](../sources/prs/sglang/PR-25764.md), [[Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename](../sources/prs/sglang/PR-25821.md), [[Refactor] Pass PP start_layer via model constructor instead of forward_batch.token_to_kv_pool](../sources/prs/sglang/PR-25825.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [Allow local cutlass directory to be used in sgl-kernel build](../sources/prs/sglang/PR-3037.md), [sync the upstream updates of flashinfer](../sources/prs/sglang/PR-3051.md), [feat: integrate gemm_fp8 kernel into gemm](../sources/prs/sglang/PR-3056.md), [Apply sgl w8a8 fp8 kernel](../sources/prs/sglang/PR-3148.md), [integrate blockwise fp8 kernel](../sources/prs/sglang/PR-3529.md), [feat: support flashinfer mla attention for deepseek v3](../sources/prs/sglang/PR-3550.md), [update flashinfer-python](../sources/prs/sglang/PR-3557.md), [feat: support flashinfer mla with prefix cache](../sources/prs/sglang/PR-3643.md), [add control for cutlass fp8 blockwise gemm](../sources/prs/sglang/PR-3727.md), [Feature DeepSeek V3/R1 INT8 Quantization (block-wise)](../sources/prs/sglang/PR-3730.md), [[Feature] DeepSeek V3/R1 INT8 Quantization (channel-wise) ](../sources/prs/sglang/PR-3888.md), [Support FP4 gemm (1/2)](../sources/prs/sglang/PR-3899.md), [upgrade flashinfer v0.2.2.post1](../sources/prs/sglang/PR-3934.md), [[tools] add fp8 max/min constant in utils](../sources/prs/sglang/PR-3959.md), [Add fast decode plan for flashinfer mla](../sources/prs/sglang/PR-3987.md), [Hierarchical Caching supports MLA](../sources/prs/sglang/PR-4009.md), [[Revision] Add fast decode plan for flashinfer mla ](../sources/prs/sglang/PR-4012.md), [Support overlapping two batches](../sources/prs/sglang/PR-4068.md), [DeepGemm integrate to gemm](../sources/prs/sglang/PR-4165.md), [linear support deepgemm](../sources/prs/sglang/PR-4199.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [Clean up fp8 support](../sources/prs/sglang/PR-4230.md), [[Feature] Integrate DeepEP into SGLang](../sources/prs/sglang/PR-4232.md), [upgrade flashinfer 0.2.3](../sources/prs/sglang/PR-4317.md), [[FIX] fix incorrect output when enable both deepgemm and torch compile](../sources/prs/sglang/PR-4359.md), [[ROCm] fix dtype](../sources/prs/sglang/PR-4510.md), [Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt](../sources/prs/sglang/PR-4515.md), [Add deepseek style fused moe group gate selection kernel](../sources/prs/sglang/PR-4530.md), [[Fix] Fix raw_bs bug when using flashinfer mla and eagle](../sources/prs/sglang/PR-4557.md), [avoid cudaStreamSynchronize in DeepSeekV2AttentionMLA](../sources/prs/sglang/PR-4577.md), [[quantization] fix channelwise conversion with scalar weight scale](../sources/prs/sglang/PR-4596.md), [Set deepgemm to the default value in the hopper architecture.](../sources/prs/sglang/PR-4613.md), [Optimize Permute Kernel in DeepEP](../sources/prs/sglang/PR-4643.md), [Fix loading KV quantization scale; Enable modelopt kv cache](../sources/prs/sglang/PR-4686.md), [[Model] Adding Qwen3 and Qwen3MoE](../sources/prs/sglang/PR-4693.md), [support cmake for sgl-kernel](../sources/prs/sglang/PR-4706.md), [[Feature] Support DeepEP Low Latency](../sources/prs/sglang/PR-4767.md), [Support (1 <= dp < tp) in the dp attention in DeepEP](../sources/prs/sglang/PR-4770.md), [Introduce moe_dense_tp_size to fix dense layer errors in DeepSeek V3 + 4x8xH100](../sources/prs/sglang/PR-4836.md), [Feat/support encoder model (like bert)](../sources/prs/sglang/PR-4887.md), [Add DeepSeek V3/R1 shared experts fusion](../sources/prs/sglang/PR-4918.md), [[Fix] DeepEP Compatibility with Low Latency](../sources/prs/sglang/PR-5068.md), [reduce moe_align_block_size_kernel small batch mode overhead](../sources/prs/sglang/PR-5086.md), [Support MHA with chunked prefix cache for DeepSeek chunked prefill](../sources/prs/sglang/PR-5113.md), [Blackwell Cutlass MLA kernel](../sources/prs/sglang/PR-5142.md), [Add optimized native kernels in sgl-kernel](../sources/prs/sglang/PR-5150.md), [feat: add DeepGEMM build warning](../sources/prs/sglang/PR-5176.md), [feat: use fa3 mla by default on hopper](../sources/prs/sglang/PR-5210.md), [[Fix] Turn off DeepGEMM by default](../sources/prs/sglang/PR-5263.md), [[1/2] Add FP8 Blockscale MoE CUTLASS kernel for Blackwell](../sources/prs/sglang/PR-5281.md), [fix: use deepgemm only on hopper](../sources/prs/sglang/PR-5310.md), [Add Speculative Decoding Eagle3 topk > 1](../sources/prs/sglang/PR-5318.md), [fix: determine if flashinfer is installed](../sources/prs/sglang/PR-5336.md), [Fix DeepGEMM masked cannot be run on groups not being multiple or 4](../sources/prs/sglang/PR-5340.md), [[perf] experimental enhance fp8 per-tensor quant](../sources/prs/sglang/PR-5370.md), [apply fused moe gate in ds v3/r1](../sources/prs/sglang/PR-5371.md), [[PD Bug] fix MLA get_contiguous_buf_infos error](../sources/prs/sglang/PR-5384.md), [Add Cutlass MLA attention backend](../sources/prs/sglang/PR-5390.md), [[PD] Fix dynamic port support and MLA buffer for Mooncake](../sources/prs/sglang/PR-5415.md), [[Feat] upgrade pytorch2.6](../sources/prs/sglang/PR-5417.md), [BLackwell cutlass mla: Add check for bad page size/block num combinations](../sources/prs/sglang/PR-5431.md), [[perf] introduce deep gemm group_gemm_masked as gemm](../sources/prs/sglang/PR-5432.md), [Avoid computing lse in Ragged Prefill when there's no prefix.](../sources/prs/sglang/PR-5476.md), [Fix sampler nan check when calling top_k_top_p_sampling_from_probs](../sources/prs/sglang/PR-5546.md), [[feature] enable pre compile jit deep_gemm](../sources/prs/sglang/PR-5580.md), [[fix] force use deepgemm in compile_deep_gemm](../sources/prs/sglang/PR-5618.md), [ DeepEP normal support deepgemm-contiguous](../sources/prs/sglang/PR-5626.md), [Turn on DeepGemm By Default and Update Doc](../sources/prs/sglang/PR-5628.md), [[perf] dsv3 bmm fallback to bf16](../sources/prs/sglang/PR-5662.md), [[2/2] Add python wrapper for CUTLASS FP8 Blockscale MoE Kernel. ](../sources/prs/sglang/PR-5694.md), [[PP] Add pipeline parallelism](../sources/prs/sglang/PR-5724.md), [Fuse MLA set kv cache kernel](../sources/prs/sglang/PR-5748.md), [opt flashinfer mla cat](../sources/prs/sglang/PR-5822.md), [Cutlass MLA decode - fix dtype error](../sources/prs/sglang/PR-5868.md), [[Fix] Fix a bug for flashmla to run R1 model](../sources/prs/sglang/PR-5875.md), [Improve dp attention port assignment scheme](../sources/prs/sglang/PR-5889.md), [[qwen3] support qwen3 ep moe](../sources/prs/sglang/PR-5917.md), [[Feat] Enable PDL automatically on Hopper architecture](../sources/prs/sglang/PR-5981.md), [KV‑Cache (MHA, MLA): add missing start_layer / end_layer fields to MHATokenToKVPoolHost and MLATokenToKVPoolHost](../sources/prs/sglang/PR-6016.md), [feat: mtp support dp-attention](../sources/prs/sglang/PR-6081.md), [[1/2] Add Kernel support for Cutlass based Fused FP4 MoE](../sources/prs/sglang/PR-6093.md), [feat: add dp attention support for Qwen 2/3 MoE models, fixes #6088](../sources/prs/sglang/PR-6121.md), [Reduce MoE memory usage](../sources/prs/sglang/PR-6147.md), [enable auto-round quantization model](../sources/prs/sglang/PR-6226.md), [Enable FlashInfer support encoder models and add head_dim padding workaround](../sources/prs/sglang/PR-6230.md), [fix: fix MLA for ShardedModelLoader/RemoteModelLoader](../sources/prs/sglang/PR-6287.md), [[Fix] Improve dependencies for Blackwell image](../sources/prs/sglang/PR-6334.md), [reduce torch.zeros overhead in moe align block size kernel](../sources/prs/sglang/PR-6369.md), [[Feature] Comprehensive Hybrid Parallelism Support](../sources/prs/sglang/PR-6389.md), [Add fp8 fused_experts kernel for CPU in sgl-kernel and add UT](../sources/prs/sglang/PR-6404.md), [Fix bug of deepseek-v3 under DP+EP mode with large batchsize/seqlen](../sources/prs/sglang/PR-6449.md), [Fix topk inference performance reduce](../sources/prs/sglang/PR-6474.md), [[Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell](../sources/prs/sglang/PR-6479.md), [qwen3moe support two batch overlap](../sources/prs/sglang/PR-6598.md), [Refine pre_reorder_triton_kernel slightly to improve performance](../sources/prs/sglang/PR-6627.md), [[CPU] [BF16] Call fused_experts_cpu, weight_packed_linear and bmm_cpu kernel in DeepSeek model](../sources/prs/sglang/PR-6641.md), [Fix DeepEP error in Qwen 3 MoE models](../sources/prs/sglang/PR-6673.md), [[EP] Add cuda kernel for moe_ep_pre_reorder](../sources/prs/sglang/PR-6699.md), [Fix PP for Qwen3 MoE](../sources/prs/sglang/PR-6709.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [[CPU] add optimizations for INT8 and FP8 DeepSeek](../sources/prs/sglang/PR-6769.md), [[CPU] support the case where num_attention_heads or intermediate_size is not divisible by the TP size](../sources/prs/sglang/PR-6771.md), [Support token-level quantization for EP MoE](../sources/prs/sglang/PR-6782.md), [[PD] Add different TP sizes support for no-MLA models](../sources/prs/sglang/PR-6793.md), [Correctly abort the failed grammar requests & Improve the handling of abort](../sources/prs/sglang/PR-6803.md), [feat: integrate deepgemm into EPMoE](../sources/prs/sglang/PR-6821.md), [CPU: map changes from developing branch in sgl-kernel](../sources/prs/sglang/PR-6833.md), [[EP] Add cuda kernel for moe_ep_post_reorder](../sources/prs/sglang/PR-6837.md), [Fix AWQ Dequant and Weight Loading of deepseek v2](../sources/prs/sglang/PR-6842.md), [[DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model](../sources/prs/sglang/PR-6853.md), [Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa](../sources/prs/sglang/PR-6890.md), [Add a CUDA kernel for fusing mapping and weighted sum for MoE.](../sources/prs/sglang/PR-6916.md), [[sgl-kernel] Add cuda kernel for moe_ep_silu_and_mul](../sources/prs/sglang/PR-6919.md), [[perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128](../sources/prs/sglang/PR-6929.md), [[Feature] Support Flashinfer fmha on Blackwell](../sources/prs/sglang/PR-6930.md), [[sgl-kernel] update deepgemm](../sources/prs/sglang/PR-6942.md), [chore: upgrade flashinfer v0.2.6.post1 jit](../sources/prs/sglang/PR-6958.md), [Fuse routed scaling factor in deepseek](../sources/prs/sglang/PR-6970.md), [Fix cutlass MLA gets almost zero accuracy](../sources/prs/sglang/PR-6998.md), [Fix torchvision version for Blackwell](../sources/prs/sglang/PR-7015.md), [Update default settings for blackwell](../sources/prs/sglang/PR-7023.md), [Clean up server_args.py](../sources/prs/sglang/PR-7037.md), [Fix positional argument](../sources/prs/sglang/PR-7093.md), [feat: update blackwell setup](../sources/prs/sglang/PR-7119.md), [fix amd EP MoE FP8 issue](../sources/prs/sglang/PR-7125.md), [Enable ModelOpt Llama4 fp8 checkpoint deployment in SGLang](../sources/prs/sglang/PR-7129.md), [Enable native ModelOpt quantization support (1/3) ](../sources/prs/sglang/PR-7149.md), [[amd] Opt dsv3 moe](../sources/prs/sglang/PR-7160.md), [Fix Deepseek R1 0528 FP4 tensor name mismatch issue during weights loading.](../sources/prs/sglang/PR-7164.md), [Support new DeepGEMM](../sources/prs/sglang/PR-7172.md), [Tiny let DeepGEMM scale checks cover more cases](../sources/prs/sglang/PR-7182.md), [chore: upgrade sgl-kernel v0.1.8.post2](../sources/prs/sglang/PR-7186.md), [[AMD] Fail gracefully when AITER is unavailable gfx90a GPUs](../sources/prs/sglang/PR-7187.md), [Fix a minor bug related to DeepGEMM upgrade](../sources/prs/sglang/PR-7191.md), [Fix error when disabling new DeepGEMM](../sources/prs/sglang/PR-7198.md), [Fix grammar abort & Minor style fixes](../sources/prs/sglang/PR-7204.md), [feat: support compatibility between MTP and two-batch-overlap](../sources/prs/sglang/PR-7225.md), [Minor style and doc fix](../sources/prs/sglang/PR-7228.md), [[fix] fix DeepGEMM blackwell input quant & ut & fix style and log](../sources/prs/sglang/PR-7247.md), [[AMD] add aiter fused moe in DeepEP path](../sources/prs/sglang/PR-7268.md), [Support NVFP4 quantized dense models on AMD CDNA2/CDNA3 GPUs](../sources/prs/sglang/PR-7302.md), [Let EP prefill support new DeepGEMM](../sources/prs/sglang/PR-7310.md), [Kernels for efficient KV cache IO](../sources/prs/sglang/PR-7313.md), [FlashInfer NVFP4 MoE with EP & 2-stream shared expert](../sources/prs/sglang/PR-7327.md), [Fix MTP with Deepseek R1 Fp4](../sources/prs/sglang/PR-7376.md), [Quick fix for DeepGemm requant to also cover MTP.](../sources/prs/sglang/PR-7378.md), [Fix torch compile run](../sources/prs/sglang/PR-7391.md), [[AMD][Quantization] Add `int4fp8_moe` online quantization on ROCm](../sources/prs/sglang/PR-7392.md), [Fix CPU offloading for MLA memory pool](../sources/prs/sglang/PR-7409.md), [Fuse sorted_token_ids padding to moe_align_block_size kernel](../sources/prs/sglang/PR-7437.md), [Support non-contiguous query input for extend/decode attention](../sources/prs/sglang/PR-7462.md), [Add Tencent HunYuanMoEV1 model support](../sources/prs/sglang/PR-7549.md), [[b200] support trt-llm allreduce fuse rms_norm_add kernel](../sources/prs/sglang/PR-7621.md), [Add dsv3 router gemm kernel](../sources/prs/sglang/PR-7627.md), [Add dsv3 fused a gemm to sgl-kernel](../sources/prs/sglang/PR-7630.md), [[Feature] Layer-wise Prefill](../sources/prs/sglang/PR-7634.md), [Apply dsv3_fused_a_gemm kernel](../sources/prs/sglang/PR-7635.md), [[Feature] CUDA Green Context Support](../sources/prs/sglang/PR-7649.md), [chore: upgrade flashinfer v0.2.7 jit](../sources/prs/sglang/PR-7663.md), [Add fp4 quantize before all-gather for Flashinfer cutlass MoE DP (max throughput)](../sources/prs/sglang/PR-7667.md), [Integrate triton moe kernel](../sources/prs/sglang/PR-7689.md), [feat: support DeepSeek-R1-W4AFP8 model with ep-moe mode](../sources/prs/sglang/PR-7762.md), [[1/n]: add cutlass W4A8 moe kernel for hopper architecture](../sources/prs/sglang/PR-7772.md), [Qwen FP8/NVFP4 ModelOPT Quantization support](../sources/prs/sglang/PR-7912.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [[fix] fix modelopt fp4 on b200](../sources/prs/sglang/PR-8195.md), [[1/N]Support DeepSeek-R1 w4a8 normal deepep](../sources/prs/sglang/PR-8247.md), [Support triton kernels v3.4.0 for fused_moe](../sources/prs/sglang/PR-8258.md), [[2/N]Support DeepSeek-R1 w4a8 low latency deepep](../sources/prs/sglang/PR-8464.md), [Update cutlass_moe.py](../sources/prs/sglang/PR-8535.md), [Update cutlass_moe.py](../sources/prs/sglang/PR-8545.md), [[NVIDIA] Add Low Latency NVFP4 decode kernels from Flashinfer](../sources/prs/sglang/PR-8552.md), [TRTLLM-MLA FP8 path](../sources/prs/sglang/PR-8638.md), [feat: support cutlass_moe_fp8 kernel for fusedmoe in sm90](../sources/prs/sglang/PR-8678.md), [fuse allreduce and residual_rmsnorm](../sources/prs/sglang/PR-8731.md), [Fix mismatch between padded_scales shape and reshape dimensions in modelopt quantization](../sources/prs/sglang/PR-8766.md), [feat: add trtllm-gen mha from direct call](../sources/prs/sglang/PR-8782.md), [[Perf] Auto enable best flashinfer mxfp4 kernel in b200](../sources/prs/sglang/PR-8898.md), [Fix hopper launch gpt-oss model illegal memory](../sources/prs/sglang/PR-8908.md), [[NVIDIA] Fix missing `get_col_major_tma_aligned_tensor` for Blackwell deepgemm in EpMoE](../sources/prs/sglang/PR-8955.md), [optimize: reduce shulffle and quantization overhead in cutlass_moe sm90](../sources/prs/sglang/PR-8962.md), [[fix] fix enable_pdl for blackwell](../sources/prs/sglang/PR-9011.md), [[sgl-kernel] Support FlashInfer top_k_top_p_sampling_from_logits](../sources/prs/sglang/PR-9060.md), [Faster weight processing (trtllm-gen moe nvfp4)](../sources/prs/sglang/PR-9162.md), [[NVIDIA] [3/N] Nvfp4 Masked Gemm: Add flashinfer grouped_gemm_nt_masked ](../sources/prs/sglang/PR-9199.md), [[NVIDA] [1/N] Nvfp4 Masked Gemm: Add quant op for the flashinfer grouped gemm](../sources/prs/sglang/PR-9200.md), [[fix]: fix cutlass moe ut and and Opt H20 cutlass groupGemm performance](../sources/prs/sglang/PR-9272.md), [Support trtllm_allreduce_fusion in flashinfer for cuda<12.8](../sources/prs/sglang/PR-9339.md), [Fix FP4 inference corruption issue in glm4.5-air model](../sources/prs/sglang/PR-9346.md), [Support DP attention with GPT-OSS](../sources/prs/sglang/PR-9359.md), [[fix] Fix mxfp4 triton MoE tp bug](../sources/prs/sglang/PR-9473.md), [Optimize moe_sum_reduce_kernel](../sources/prs/sglang/PR-9477.md), [fix: blackwell dsv3 fp8 issue temporary solution](../sources/prs/sglang/PR-9530.md), [[NVIDIA] [2/N] Optimize `silu_and_mul_scaled_fp4_grouped_quant` perf](../sources/prs/sglang/PR-9556.md), [Update CUTLASS 4.2 & Enable K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM](../sources/prs/sglang/PR-9559.md), [Tiny fix wrong comments](../sources/prs/sglang/PR-9589.md), [Single Batch Overlap for MoE Models](../sources/prs/sglang/PR-9660.md), [fix mooncake store mla zero copy meta](../sources/prs/sglang/PR-9678.md), [move is_sm90_supported/is_sm100_supported to python/sglang/srt/utils.py](../sources/prs/sglang/PR-9679.md), [[ModelOpt] Fix Weight Loading for DSR1-FP4 Quantization](../sources/prs/sglang/PR-9712.md), [[CPU] Add FP8 Bmm support](../sources/prs/sglang/PR-9744.md), [[Model] Support Meituan LongCat-Flash && LongCat-Flash-MTP](../sources/prs/sglang/PR-9824.md), [perf: Avoid unnecessary data type conversions for DeepSeek-V3 on Blackwell](../sources/prs/sglang/PR-9834.md), [support using fa4 on deepseek on blackwell](../sources/prs/sglang/PR-9928.md), [[Fix] DeepSeek EP accuracy issue on B200 GPUs](../sources/prs/sglang/PR-9946.md), [Enable native ModelOpt quantization support (2/3)](../sources/prs/sglang/PR-9991.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [[Feature] Support tcgen5mma lowering for `.kind::i8`](../sources/prs/tilelang/PR-1764.md), [[BugFix] Fix Hopper TMA lowering without warp specialization](../sources/prs/tilelang/PR-1840.md), [[CUDA] Support tcgen5mma gemm ts](../sources/prs/tilelang/PR-1866.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Bugfix] Fix CuTeDSL autotune cache invalid ELF header (#1967)](../sources/prs/tilelang/PR-1972.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[CUDA] Improve int4 GEMM lowering and packed codegen support](../sources/prs/tilelang/PR-2073.md), [[TMA] Support FP4 TensorMap TMA copies](../sources/prs/tilelang/PR-2107.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md), [[Perf] Enable fast math in sparse MLA example](../sources/prs/tilelang/PR-2219.md), [[Core] Support fully transparent sleep mode](../sources/prs/vllm/PR-11743.md), [[ROCm][MoE] moe tuning support for rocm](../sources/prs/vllm/PR-12049.md), [[Kernel] Flash Attention 3 Support](../sources/prs/vllm/PR-12093.md), [Add: Support for Sparse24Bitmask Compressed Models](../sources/prs/vllm/PR-12097.md), [[Hardware][Gaudi][Feature] Enable Dynamic MoE for Mixtral](../sources/prs/vllm/PR-12303.md), [[Core] Optimizing cross-attention `QKVParallelLinear` computation](../sources/prs/vllm/PR-12325.md), [[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24](../sources/prs/vllm/PR-12417.md), [[Misc][MoE] add Deepseek-V3 moe tuning support](../sources/prs/vllm/PR-12558.md), [Expert Parallelism (EP) Support for DeepSeek Models](../sources/prs/vllm/PR-12583.md), [Apply torch.compile to fused_moe/grouped_topk](../sources/prs/vllm/PR-12637.md), [Disable chunked prefill and/or prefix caching when MLA is enabled ](../sources/prs/vllm/PR-12642.md), [[AMD][ROCm] Enable DeepSeek model on ROCm](../sources/prs/vllm/PR-12662.md), [Squelch MLA warning for Compressed-Tensors Models](../sources/prs/vllm/PR-12704.md), [[VLM] Add MLA with pure RoPE support for deepseek-vl2 models](../sources/prs/vllm/PR-12729.md), [[Misc] Update w2 scale loading for GPTQMarlinMoE](../sources/prs/vllm/PR-12757.md), [[Bugfix] Better FP8 supported defaults](../sources/prs/vllm/PR-12796.md), [[Attention] Use FA3 for MLA on Hopper](../sources/prs/vllm/PR-12807.md), [[Model] Deepseek GGUF support ](../sources/prs/vllm/PR-13167.md), [Expand MLA to support most types of quantization](../sources/prs/vllm/PR-13181.md), [[Quant][Perf] Use moe_wna16 kernel by default for MoEs with many experts](../sources/prs/vllm/PR-13236.md), [[Bugfix] Massage MLA's usage of flash attn for RoCM](../sources/prs/vllm/PR-13310.md), [[NVIDIA] Support nvfp4 tcgen05 gemm](../sources/prs/vllm/PR-13571.md), [[Bugfix] Fix max_num_batched_tokens for MLA](../sources/prs/vllm/PR-13620.md), [[Kernel] Optimize moe intermediate_cache usage](../sources/prs/vllm/PR-13625.md), [[ROCM] fix native attention function call](../sources/prs/vllm/PR-13650.md), [[BugFix] Illegal memory access for MoE On H20](../sources/prs/vllm/PR-13693.md), [[Bugfix] Support MLA for CompressedTensorsWNA16](../sources/prs/vllm/PR-13725.md), [[V1] V1 Enablement Oracle ](../sources/prs/vllm/PR-13726.md), [Fix CompressedTensorsWNA16MoE with grouped scales](../sources/prs/vllm/PR-13769.md), [Fix precommit fail in fused_moe intermediate_cache2 chunking](../sources/prs/vllm/PR-13772.md), [[Bugfix][Quantization] Fix FP8 + EP](../sources/prs/vllm/PR-13784.md), [add tcgen05 support for tcgen05 fp8 gemm](../sources/prs/vllm/PR-13798.md), [Fix mla prefill context performance](../sources/prs/vllm/PR-13897.md), [[Misc] Print FusedMoE detail info](../sources/prs/vllm/PR-13974.md), [[core] moe fp8 block quant tuning support](../sources/prs/vllm/PR-14068.md), [[V1] Implement sliding window attention in kv_cache_manager](../sources/prs/vllm/PR-14097.md), [[v1] Add comments to the new ragged paged attention Pallas kernel](../sources/prs/vllm/PR-14155.md), [[V1][TPU] TPU multimodal model support for ragged attention](../sources/prs/vllm/PR-14158.md), [[V1][TPU] Support V1 Sampler for ragged attention](../sources/prs/vllm/PR-14227.md), [[Hardware] Update the flash attn tag to support Blackwell](../sources/prs/vllm/PR-14244.md), [[BugFix] Fix prefix caching V0 MLA](../sources/prs/vllm/PR-14255.md), [[Misc] Add Qwen2MoeForCausalLM moe tuning support ](../sources/prs/vllm/PR-14276.md), [[Hardware][TPU]Enable ragged paged attention kernel and resolve recompilation issue](../sources/prs/vllm/PR-14310.md), [[Bug] Fix Attention when ignored in by quant_method](../sources/prs/vllm/PR-14313.md), [[ROCm] Enable chunked prefill/paged attention in MLA on ROCm](../sources/prs/vllm/PR-14316.md), [[Perf] Reduce MLA CPU overheads in V1](../sources/prs/vllm/PR-14384.md), [[ROCm][Kernel] MoE weights padding](../sources/prs/vllm/PR-14454.md), [[Bugfix] DeepSeek Accuracy](../sources/prs/vllm/PR-14476.md), [[Perf] Improve MLA on V1](../sources/prs/vllm/PR-14540.md), [[Quantization][FP8] Adding support for fp8 gemm layer input in fp8](../sources/prs/vllm/PR-14578.md), [[Model] Add support for Gemma 3](../sources/prs/vllm/PR-14660.md), [[Bugfix][IPEX] Add `VLLM_CPU_MOE_PREPACK` to allow disabling MoE prepack when CPU does not support it](../sources/prs/vllm/PR-14681.md), [[Kernel][CPU] CPU MLA](../sources/prs/vllm/PR-14744.md), [[Attention] MLA get rid of materialization](../sources/prs/vllm/PR-14770.md), [[Attention] Get rid of mla cache alignment](../sources/prs/vllm/PR-14842.md), [[V1][BugFix] Detect interleaved sliding window attention](../sources/prs/vllm/PR-14896.md), [[V1] Default MLA to V1](../sources/prs/vllm/PR-14921.md), [[FEAT][ROCm] Integrate Fused MoE Kernels from AITER](../sources/prs/vllm/PR-14967.md), [[FEAT] [ROCm]: Add AITER Block-Scaled GEMM Feature](../sources/prs/vllm/PR-14968.md), [[FEAT][ROCm] Integrate Paged Attention Kernel from AITER](../sources/prs/vllm/PR-15001.md), [[Bugfix] Fix incorrect qwen2.5-vl attention mask pre-computation](../sources/prs/vllm/PR-15200.md), [[Bugfix] Fix use_cascade_attention handling for Alibi-based models on vllm/v1](../sources/prs/vllm/PR-15211.md), [[Misc] Add attention mask pre-computation optimization back to Qwen2.5-VL](../sources/prs/vllm/PR-15273.md), [[Model] Add Qwen3 and Qwen3MoE](../sources/prs/vllm/PR-15289.md), [Fix non-contiguous input passed to Marlin kernel](../sources/prs/vllm/PR-15319.md), [[FEAT] [ROCm] Add AITER int8 scaled gemm kernel](../sources/prs/vllm/PR-15433.md), [Use Cache Hinting for fused_moe kernel](../sources/prs/vllm/PR-15511.md), [[moe][quant] add weight name case for offset](../sources/prs/vllm/PR-15515.md), [[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM](../sources/prs/vllm/PR-15587.md), [[TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU.](../sources/prs/vllm/PR-15732.md), [[V1] TPU - Fix fused MOE](../sources/prs/vllm/PR-15834.md), [[Bugfix] Fix cache block size calculation for CPU MLA](../sources/prs/vllm/PR-15848.md), [[FEAT][ROCm]: Support AITER MLA](../sources/prs/vllm/PR-15893.md), [[Hardware][Gaudi][BugFix] fix arguments of hpu fused moe](../sources/prs/vllm/PR-15945.md), [Add support to modelopt quantization of Mixtral model](../sources/prs/vllm/PR-15961.md), [[NVIDIA] Support Cutlass MLA for Blackwell GPUs](../sources/prs/vllm/PR-16032.md), [[Kernel] Use moe_wna16 kernel for compressed tensors wna16 moe models](../sources/prs/vllm/PR-16038.md), [[Model] use AutoWeightsLoader for phimoe,qwen2_moe,qwen3_moe](../sources/prs/vllm/PR-16203.md), [[Hardware][AMD] Improve OAM device ID + llama4 Maverick MOE tuning](../sources/prs/vllm/PR-16263.md), [[Llama4] Enable attention temperature tuning by default for long context (>32k)](../sources/prs/vllm/PR-16439.md), [[MLA] Simplification to batch P/D reordering](../sources/prs/vllm/PR-16673.md), [[ROCM] enable aiter fused moe kernel for llama4 bf16 checkpoints](../sources/prs/vllm/PR-16674.md), [[ROCm] Add aiter tkw1 kernel for Llama4 fp8](../sources/prs/vllm/PR-16727.md), [Support W8A8 INT8 MoE for compressed-tensors](../sources/prs/vllm/PR-16745.md), [[FEAT] [ROCm]: AITER Fused MOE V1 Support](../sources/prs/vllm/PR-16752.md), [[Bugfix] Fix moe weight losing all extra attrs after `process_weights_after_loading`.](../sources/prs/vllm/PR-16854.md), [[Bugfix] Add contiguous call inside rope kernel wrapper](../sources/prs/vllm/PR-17091.md), [[FEAT] [ROCm]: Add AITER CK 2 Stages MoE support](../sources/prs/vllm/PR-17110.md), [[Bugfix] gemma[2,3] interleaved attention when sliding window is disabled](../sources/prs/vllm/PR-17180.md), [[Bugfix] Get a specific type of layer from forward context](../sources/prs/vllm/PR-17222.md), [[BugFix] Fix vllm_flash_attn install issues](../sources/prs/vllm/PR-17267.md), [[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata must have shape (metadata_size)](../sources/prs/vllm/PR-17283.md), [[v1] AttentionMetadata for each layer](../sources/prs/vllm/PR-17394.md), [Fix noisy warning for uncalibrated q_scale/p_scale](../sources/prs/vllm/PR-17414.md), [[v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders](../sources/prs/vllm/PR-17483.md), [[BugFix] Fix mla cpu - missing 3 required positional arguments](../sources/prs/vllm/PR-17494.md), [[FEAT][ROCm]: Support AITER MLA on V1 Engine](../sources/prs/vllm/PR-17523.md), [[Bugfix][ROCm] Fix AITER MLA V1](../sources/prs/vllm/PR-17880.md), [[BugFix][AMD] Compatible patch for AITER lib after 04/20](../sources/prs/vllm/PR-17912.md), [[Misc] Add compressed-tensors NVFP4A16 emulation support](../sources/prs/vllm/PR-17914.md), [use ceil_div in cutlass block scaling shape check](../sources/prs/vllm/PR-17918.md), [[v1] Support multiple KV cache groups in GPU model runner](../sources/prs/vllm/PR-17945.md), [[BUG] [ROCm] [MLA] Fix variable name bug due to change in variable name in PR #17483](../sources/prs/vllm/PR-17961.md), [Use NVFP4 Marlin for CompressedTensorsW4A16Fp4](../sources/prs/vllm/PR-18000.md), [[Quantization] Add compressed-tensors NVFP4 support](../sources/prs/vllm/PR-18312.md), [[Model]: Fused MoE for nomic-embed-text-v2-moe](../sources/prs/vllm/PR-18321.md), [[Feature] Expert Parallelism Load Balancer (EPLB)](../sources/prs/vllm/PR-18343.md), [[Bug] Fix moe_sum signature](../sources/prs/vllm/PR-18440.md), [[V1] Support `LLM.apply_model`](../sources/prs/vllm/PR-18465.md), [[Hardware][AMD] integrate aiter chunked prefill into vllm](../sources/prs/vllm/PR-18596.md), [[P/D] Heterogeneous TP](../sources/prs/vllm/PR-18833.md), [[ROCm] [AITER] [Bugfix] Patch for AITER commit `648764942e552a8bb5fe16026703716a81f05374`](../sources/prs/vllm/PR-18990.md), [[Kernel] Support deep_gemm for linear methods](../sources/prs/vllm/PR-19085.md), [[Kernel] Apply torch.Tag.needs_fixed_stride_order only for torch==2.6.0](../sources/prs/vllm/PR-19346.md), [[Core] Support Local Chunked Attention for Hybrid KV Cache](../sources/prs/vllm/PR-19351.md), [[Kernels] Use empty for modular MoE workspaces](../sources/prs/vllm/PR-19667.md), [[Feature] Integrate new deepgemm](../sources/prs/vllm/PR-19820.md), [[Bugfix] Enable PP with AITER+V1](../sources/prs/vllm/PR-19822.md), [[Quantization] Add compressed-tensors emulations support for NVFP4](../sources/prs/vllm/PR-19879.md), [[Quantization] Add compressed-tensors NVFP4 MoE Support](../sources/prs/vllm/PR-19990.md), [Enable V1 for Hybrid SSM/Attention Models](../sources/prs/vllm/PR-20016.md), [[Attention] MLA - Flashinfer Ragged Prefill](../sources/prs/vllm/PR-20034.md), [Add ModelOpt Qwen3 nvfp4 support](../sources/prs/vllm/PR-20101.md), [[Bugfix] Mark 'hidden_states' as mutable in moe_forward registration.](../sources/prs/vllm/PR-20152.md), [[Bugfix] Fix Maverick correctness by filling zero to cache space in cutlass_moe](../sources/prs/vllm/PR-20167.md), [[Nixl] Heterogeneous TP support FlashInfer](../sources/prs/vllm/PR-20189.md), [[V1] [ROCm] Enable EP with AITER Fused MoE](../sources/prs/vllm/PR-20270.md), [Support Llama 4 for cutlass_moe_fp4](../sources/prs/vllm/PR-20453.md), [Support Llama 4 for fused_marlin_moe](../sources/prs/vllm/PR-20457.md), [[Perf] Reuse workspace for FP8+FP4 Marlin MoE](../sources/prs/vllm/PR-20500.md), [[Bugfix] Fix missing per_act_token parameter in compressed_tensors_moe](../sources/prs/vllm/PR-20509.md), [[feat] enable SM100 CUTLASS block scaled group gemm for smaller batch sizes](../sources/prs/vllm/PR-20640.md), [Integration SM100 FlashInfer fused allreduce RMSNorm](../sources/prs/vllm/PR-20691.md), [GLM-4.5 Model Support](../sources/prs/vllm/PR-20736.md), [[v1][core] Support for attention free models](../sources/prs/vllm/PR-20811.md), [[Feature][EPLB] Add eplb support for Qwen3](../sources/prs/vllm/PR-20815.md), [[Bugfix] Fix a couple PPLX+CUTLASS MoE bugs](../sources/prs/vllm/PR-20825.md), [[Bug] Fix DeepGemm for EP low latency case](../sources/prs/vllm/PR-20833.md), [[Model] Pooling models default to using chunked prefill & prefix caching if supported.](../sources/prs/vllm/PR-20930.md), [[Misc] Qwen MoE model supports LoRA](../sources/prs/vllm/PR-20932.md), [[Bugfix] Switch bailout logic for kv-cache-dtype with SM100 Flashinfer](../sources/prs/vllm/PR-20934.md), [Fall back if flashinfer comm module not found](../sources/prs/vllm/PR-20936.md), [[Bugfix] Fix Mistral3 support on SM100/SM120](../sources/prs/vllm/PR-20998.md), [Add FlashInfer allreduce RMSNorm Quant fusion](../sources/prs/vllm/PR-21069.md), [[Bugfix] Voxtral on Blackwell GPUs (RTX 50 series)](../sources/prs/vllm/PR-21077.md), [[Bugfix] Allocate less memory in non-batched CUTLASS MoE](../sources/prs/vllm/PR-21121.md), [[Attention] Optimize FlashInfer MetadataBuilder Build call](../sources/prs/vllm/PR-21137.md), [[Attention][DBO] Add support for "splitting" the CommonAttentionMetadata](../sources/prs/vllm/PR-21153.md), [[Feature][OCP MX] Support mxfp6 and mixed mxfp6-mxfp4](../sources/prs/vllm/PR-21166.md), [[Bug] DeepGemm: Fix TypeError: per_block_cast_to_fp8() missing 1 required positional argument: 'use_ue8m0' for SM100](../sources/prs/vllm/PR-21187.md), [Support encoder-only models without KV-Cache](../sources/prs/vllm/PR-21270.md), [Fix Flashinfer Allreduce+Norm enable disable calculation based on `fi_allreduce_fusion_max_token_num`](../sources/prs/vllm/PR-21325.md), [Support Tensorrt-LLM MoE fp4 for low-latency](../sources/prs/vllm/PR-21331.md), [[TPU][Bugfix] fix moe layer](../sources/prs/vllm/PR-21340.md), [[Quantization] Enable BNB support for more MoE models](../sources/prs/vllm/PR-21370.md), [Update flashinfer CUTLASS NVFP4 MoE Kernel to use per expert global scaling factor](../sources/prs/vllm/PR-21408.md), [[NVIDIA] Explicitly disable shuffled weights for flashinfer blockscale moe fp8 kernels](../sources/prs/vllm/PR-21411.md), [Updates to Flex + VLLm integration](../sources/prs/vllm/PR-21416.md), [[V1] Fix local chunked attention always disabled](../sources/prs/vllm/PR-21419.md), [[BugFix] Fix shared storage connector load kv only load attention layer](../sources/prs/vllm/PR-21428.md), [update flashinfer to v0.2.9rc1](../sources/prs/vllm/PR-21485.md), [[MoE] More balanced expert sharding](../sources/prs/vllm/PR-21497.md), [[NVIDIA] Fix Llama4 Scout FP4 functionality issues](../sources/prs/vllm/PR-21499.md), [Enable 4bit bnb prequant MOE](../sources/prs/vllm/PR-21548.md), [[Attention] Support multiple attention metadata builders per kv_cache_spec + proper local attention no hybrid kv cache fix](../sources/prs/vllm/PR-21588.md), [Override attention metadata for fast prefill in some KV sharing setups](../sources/prs/vllm/PR-21590.md), [[Feature] Add Flashinfer MoE Support for Compressed Tensor NVFP4](../sources/prs/vllm/PR-21639.md), [[xpu]support moe models on XPU platform](../sources/prs/vllm/PR-21643.md), [support `torch.compile` for bailing moe](../sources/prs/vllm/PR-21664.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv/out Attention Kernel](../sources/prs/vllm/PR-21716.md), [feat: Add Support GPTQ Quantization MOE on ROCM vllm serve](../sources/prs/vllm/PR-21733.md), [[Logs] Change flashinfer sampler logs to once](../sources/prs/vllm/PR-21759.md), [[Perf] Disable chunked local attention by default with llama4](../sources/prs/vllm/PR-21761.md), [[Bugfix] Check NVIDIA artifactory is accessible before using flashinfer cubin kernels](../sources/prs/vllm/PR-21893.md), [[Qwen3] Enable dual-chunk-attention support for Qwen3 models.](../sources/prs/vllm/PR-21924.md), [[BUGFIX] KeyError 'layers.14.mlp.gate.g_idx' for Qwen3-MoE with GPTQ on ROCm](../sources/prs/vllm/PR-22017.md), [[NVIDIA] Support Flashinfer TRT-LLM Prefill Attention Kernel](../sources/prs/vllm/PR-22095.md), [[EPLB] Support ernie4.5-moe](../sources/prs/vllm/PR-22100.md), [[fix] fix correct assertion syntax error in attention utils.](../sources/prs/vllm/PR-22154.md), [[bugfix] fix blackwell deepep installation](../sources/prs/vllm/PR-22255.md), [[Bugfix] Fix MoE BNB version](../sources/prs/vllm/PR-22260.md), [Support encoder_only attention for FlexAttention](../sources/prs/vllm/PR-22273.md), [[Bugfix] Fix 3D input passed into cutlass_scaled_mm](../sources/prs/vllm/PR-22278.md), [[ROCm] Add attention sink to use_rocm_custom_paged_attention](../sources/prs/vllm/PR-22329.md), [[gpt-oss] flashinfer mxfp4](../sources/prs/vllm/PR-22339.md), [Update `flashinfer-python==0.2.10`](../sources/prs/vllm/PR-22389.md), [[Bug] Fix B200 DeepGEMM E8M0 Accuracy Issue](../sources/prs/vllm/PR-22399.md), [[bugfix] Fix Llama3/4 issues caused by FlashInfer 0.2.10](../sources/prs/vllm/PR-22426.md), [[Quantization]: Support compressed-tensors mixed-precision model loading](../sources/prs/vllm/PR-22468.md), [Fix Llama4 FlashInfer FP4 MoE issues](../sources/prs/vllm/PR-22511.md), [[Model] Add Ernie4.5 VL Model Support](../sources/prs/vllm/PR-22514.md), [Quantization: support FP4 quantized models on AMD CDNA2/CDNA3 GPUs](../sources/prs/vllm/PR-22527.md), [Fix torch version check for SM100 mxfp4 ](../sources/prs/vllm/PR-22535.md), [Upgrade FlashInfer to v0.2.11](../sources/prs/vllm/PR-22613.md), [[Bugfix] Fix ModernBert load & Enable sliding window attention for bidirectional attention.](../sources/prs/vllm/PR-22637.md), [Support multiple attention groups for KV sharing](../sources/prs/vllm/PR-22672.md), [[Quantization] Expand compressed-tensors MoE matching logic to support NFP4 + FP8 MoEs](../sources/prs/vllm/PR-22674.md), [Force TRTLLM attention for gpt-oss on SM100](../sources/prs/vllm/PR-22678.md), [[Bugfix] Fix default enable for CUTLASS MLA on SM100](../sources/prs/vllm/PR-22738.md), [Fix GGUF loader for Qwen3 MoE.](../sources/prs/vllm/PR-22785.md), [[FIXBUG] Add return_success parameter to moe_wna16_weight_loader function](../sources/prs/vllm/PR-22797.md), [[Model] Modify the gate implementation of glm4_moe](../sources/prs/vllm/PR-22832.md), [[XPU] support data parallel for MoE models on XPU](../sources/prs/vllm/PR-22887.md), [[Kernel] Added flashinfer fp8 per-tensor gemms](../sources/prs/vllm/PR-22895.md), [[Bugfix] Fix DeepSeek MTP](../sources/prs/vllm/PR-22934.md), [Use Blackwell FlashInfer MXFP4 MoE by default if available ](../sources/prs/vllm/PR-23008.md), [[Bugfix gpt-oss] Fix float32 convert for flashinfer sink support](../sources/prs/vllm/PR-23016.md), [[Bugfix] fix qwen3 moe fp8 accuracy issue](../sources/prs/vllm/PR-23031.md), [[Core] Support weight_loader_v2 for `UnquantizedLinearMethod`](../sources/prs/vllm/PR-23036.md), [Add routed_scaling_factor to MoE grouped topk](../sources/prs/vllm/PR-23123.md), [[Bugfix] Fix accuracy issue when using flashinfer cutlass moe, TP=1 and modelopt.](../sources/prs/vllm/PR-23125.md), [Update to flashinfer-python==0.2.12 and disable AOT compile for non-release image](../sources/prs/vllm/PR-23129.md), [[Log] Warning Once for Cutlass MLA ](../sources/prs/vllm/PR-23137.md), [Fix nvfp4 swizzling](../sources/prs/vllm/PR-23140.md), [[CPU] add cpu fused moe pytorch native implementation](../sources/prs/vllm/PR-23146.md), [[XPU][Feature] fp8 online quantization support for XPU](../sources/prs/vllm/PR-23148.md), [Optimize input preparation for FlashInfer [2/N]](../sources/prs/vllm/PR-23174.md), [[Attention] Optimize make_local_attention_virtual_batches for Flash Attention](../sources/prs/vllm/PR-23185.md), [[Misc][qwen2_5_vl][torch.compile] Enable `supports_torch_compile` on generic nn.Module and demonstrate speedup on Qwen Vision model](../sources/prs/vllm/PR-23207.md), [[Core] Always use tensor cores for Flashinfer Decode Wrapper](../sources/prs/vllm/PR-23214.md), [[Perf] Small optimizations for silu_mul_fp8_quant_deep_gemm](../sources/prs/vllm/PR-23265.md), [[Kernels] Overlap shared experts with send/recv](../sources/prs/vllm/PR-23273.md), [[Bug] Fix R1 Accuracy 0 Bug](../sources/prs/vllm/PR-23294.md), [[Attention] Allow V1 flash_attn to support cross-attention](../sources/prs/vllm/PR-23297.md), [[Perf] Warmup FlashInfer attention during startup](../sources/prs/vllm/PR-23439.md), [[Attention][FA3] Update FA3 to include new swizzle optimization](../sources/prs/vllm/PR-23465.md), [fix(v1/kv_cache): resolve async KV transfer bug in cascade attention](../sources/prs/vllm/PR-23485.md), [[Bugfix] Fix Qwen3 MoE GPTQ inference](../sources/prs/vllm/PR-23490.md), [[V1][P/D]P2pNcclConnector supports flashinfer](../sources/prs/vllm/PR-23536.md), [Update Flashinfer to 0.2.14.post1](../sources/prs/vllm/PR-23537.md), [[Misc] Simplify FlashInfer attention metadata](../sources/prs/vllm/PR-23585.md), [DP/EP Support for gpt-oss with deepep-ht comm kernel on SM100](../sources/prs/vllm/PR-23608.md), [[Flashinfer] Support Flashinfer TRTLLM FP8-qkv BF16/FP16-out Attention Kernel](../sources/prs/vllm/PR-23647.md), [[Bugfix] Fix Marlin NVFP4 for modelopt](../sources/prs/vllm/PR-23659.md), [[v1] Add cross-attention KV cache support for encoder-decoder models](../sources/prs/vllm/PR-23664.md), [[Core/DBO][1/N] Add Dual-Batch Overlap mechanism to VLLM](../sources/prs/vllm/PR-23693.md), [[Kernel][tcgen05] nvfp4 fused tcgen05 moe](../sources/prs/vllm/PR-23696.md), [[FlashInfer] Cache hyper params in metadata builder](../sources/prs/vllm/PR-23732.md), [[BugFix][FlashInfer] Fix potential race condition for paged_kv_indptr_cpu](../sources/prs/vllm/PR-23737.md), [[Feat][EPLB] A novel static EPLB placement strategy for MoE models.](../sources/prs/vllm/PR-23745.md), [[Misc] add reorder_batch AttentionMetadataBuilder](../sources/prs/vllm/PR-23798.md), [[fix]: add Arm 4bit fused moe support](../sources/prs/vllm/PR-23809.md), [[Model][gpt-oss] Support DP+EP for GPT-OSS with FlashInfer trtllm-gen MoE](../sources/prs/vllm/PR-23819.md), [[BUGFIX ] fix undefined silu_and_mul_nvfp4_quant](../sources/prs/vllm/PR-23929.md), [Feature/vit attention unification# 23880](../sources/prs/vllm/PR-23978.md), [[BUGFIX] GPTQ quantization compatibility for Qwen3 MOE models (AutoGPTQ and AutoRound-GPTQ)](../sources/prs/vllm/PR-23994.md), [[PERF] Allreduce fusion. Support torch native matching. Tuning of the thresholds](../sources/prs/vllm/PR-24248.md), [[Transform] [Quantization] Add QuTLASS support to vLLM](../sources/prs/vllm/PR-24440.md), [[Feature] Disallow FlashMLA on Blackwell](../sources/prs/vllm/PR-24521.md), [[Performance] Move apply_w8a8_block_fp8_linear to an op class](../sources/prs/vllm/PR-24666.md), [[Model] Support Qwen3-VL Model Series](../sources/prs/vllm/PR-24727.md), [[Bug] Fix `is_flashmla_supported` Check Error](../sources/prs/vllm/PR-24774.md), [[DCP] Support Decode Context Parallel (DCP) for GQA with FlashAttention](../sources/prs/vllm/PR-24864.md), [[Attention][DCP] Support DCP with query length > 1 (MTP) with FA3](../sources/prs/vllm/PR-25049.md), [[Bug] Fix `returned_lse` not Defined issue](../sources/prs/vllm/PR-25106.md), [[ROCm] Small functional changes for gptoss](../sources/prs/vllm/PR-25201.md), [[BugFix] Fix MLA assert with CUTLASS MLA](../sources/prs/vllm/PR-25478.md), [feat: BF16 FlashInfer Fused Cutlass MOE for Hopper and Blackwell Expert Parallel](../sources/prs/vllm/PR-25503.md), [Enable Fbgemm NVFP4 on Dense models](../sources/prs/vllm/PR-25609.md), [[Flashinfer][gpt-oss] Support FP8-qkv Flashinfer TRTLLM Sinks Attention](../sources/prs/vllm/PR-25674.md), [[Bugfix] Enable padded FP4 quantization](../sources/prs/vllm/PR-25947.md), [[Quantization/NVFP4] Speed up TRTLLM NVFP4 MOE weight loading and fix K/V scale loading for MLA Attn](../sources/prs/vllm/PR-25968.md), [[Spec Decode] Enable efficient speculative decoding with FlashInfer-MLA](../sources/prs/vllm/PR-25984.md), [[Bugfix] Allow skipping MoE in NVFP4 (fix for MTP)](../sources/prs/vllm/PR-25987.md), [[NVIDIA] Add support for cudnn fp4 gemm via flashinfer](../sources/prs/vllm/PR-26107.md), [[ModelOpt] Load w13/w2_input_scale for all experts, nvfp4](../sources/prs/vllm/PR-26135.md), [[Performance] Dual stream execution of "shared_experts" and "selected_experts" inside FusedMoE](../sources/prs/vllm/PR-26440.md), [[Bugfix] Convert untraceable GroupShape to list for AMD impl](../sources/prs/vllm/PR-26535.md), [[ROCM] MoE fp4 CK kernel](../sources/prs/vllm/PR-26545.md), [support flashinfer_fp4 moe for 5090 gpu](../sources/prs/vllm/PR-26669.md), [[Bugfix] Fix gpt-oss w4a8 DP/EP on B200](../sources/prs/vllm/PR-26729.md), [Disable FlashInfer sampler by default](../sources/prs/vllm/PR-26859.md), [[Feature] Batch Invariant: Support DeepGEMM and Blackwell](../sources/prs/vllm/PR-27127.md), [[Kernels] Enable FlashInfer FP8 Blockscale on SM90 (for TEP DSR1)](../sources/prs/vllm/PR-27134.md), [[torch.compile] Enable silu_mul_fp8_quant fusion without custom ops enabled](../sources/prs/vllm/PR-27146.md), [[ROCM] Enable CompressedTensorsWNA16](../sources/prs/vllm/PR-27187.md), [[BUGFIX][ROCM] ViT FlashAttention on ROCm (no GFX9) and contiguous on qwen3vl ROCm TORCH_SDPA](../sources/prs/vllm/PR-27190.md), [Flashinfer_CUTLASS_MOE fuses quantization for TP](../sources/prs/vllm/PR-27223.md), [[Feature] Batch Invariant for R1 TP 8 on Blackwell](../sources/prs/vllm/PR-27229.md), [[Bugfix] Ensure calculated KV scales are applied in attention.](../sources/prs/vllm/PR-27232.md), [Bugfix: Cutlass FP8 FusedMoE bad scaling factors](../sources/prs/vllm/PR-27255.md), [Feature: Support Relu2 in FusedMoE fp8 cutlass path](../sources/prs/vllm/PR-27261.md), [[Misc] Make reorder batch also separate extends](../sources/prs/vllm/PR-27367.md), [[Performance] Support FP8 flashinfer TRTLLM MOE on Qwen3 and Qwen-3next](../sources/prs/vllm/PR-27492.md), [[Feature] Batch invariant torch.compile](../sources/prs/vllm/PR-27660.md), [[AMD] Use Decoupled Kernel Block Size to Support AITER MLA block_size=1](../sources/prs/vllm/PR-27715.md), [[Feature] Extend batch invariant torch.compile to B200](../sources/prs/vllm/PR-27856.md), [[Bug] Batch invariant: Fix flash attn MLA `RuntimeError: scheduler_metadata must have shape (metadata_size)`](../sources/prs/vllm/PR-27884.md), [[Performance][B200] Fix deepgemm prologue](../sources/prs/vllm/PR-27897.md), [[flashinfer][fix] do not check nvcc availability when using pre-downloaded cubins](../sources/prs/vllm/PR-27990.md), [[FlashInfer] Avoid FlashInfer block_size 16 + head_size 256 on blackwell](../sources/prs/vllm/PR-27994.md), [[ROCm][MLA] enable fp8 MLA decode on ROCm](../sources/prs/vllm/PR-28032.md), [[Model] Consolidate Deepseek-MoE implementation with DeepSeek-v2](../sources/prs/vllm/PR-28101.md), [[Mamba] - Consolidate Mambas Attention Logic](../sources/prs/vllm/PR-28133.md), [[flashinfer] fix FI all2all with FI cutlass moe](../sources/prs/vllm/PR-28166.md), [[Feature] Support recording expert indices for rollout router replay](../sources/prs/vllm/PR-28284.md), [[ROCm] Support for Whisper v1 with Aiter Unified Attention and Aiter Flash Attention](../sources/prs/vllm/PR-28376.md), [[Bugfix][EPLB] Disabled shared expert overlap when EPLB is enabled](../sources/prs/vllm/PR-28377.md), [[Bugfix] Fix SM100 gpt-oss regression due to faulty attn sink support](../sources/prs/vllm/PR-28561.md), [[Attention][Bugfix] Fix FA sink support](../sources/prs/vllm/PR-28660.md), [[Bugfix][Nixl] Fix kernel physical<>logical block_size issue ](../sources/prs/vllm/PR-28677.md), [[Performance] Reduce DeepGEMM N dim restriction from 128 to 64 multiplier ](../sources/prs/vllm/PR-28687.md), [[Feature] Prefill Context Parallel (PCP) basic support](../sources/prs/vllm/PR-28718.md), [[Bugfix] Fix GPT-OSS on AMD after #28603](../sources/prs/vllm/PR-28816.md), [bugfix: correct attn output with base 2 or e](../sources/prs/vllm/PR-28840.md), [[Bugfix] Fix GPT-OSS AR+NORM fusion](../sources/prs/vllm/PR-28841.md), [[Bugfix] Make compressed-tensors MoEs respect ignored layers](../sources/prs/vllm/PR-28878.md), [Add TRTLLM MoE NVFP4 kernel to CompressedTensorsW4A4MoeMethod](../sources/prs/vllm/PR-28892.md), [[BugFix] Fix async-scheduling + FlashAttn MLA](../sources/prs/vllm/PR-28990.md), [[Feat] Support non-gated activations in NVFP4 modelopt path](../sources/prs/vllm/PR-29004.md), [[DeepSeek + LMCache Multiprocess] handle MLA for deepseek model + LMCache Multiprocess connector](../sources/prs/vllm/PR-29039.md), [[Perf][Kernels] Enable FlashInfer DeepGEMM swapAB on SM90 (for W8A8 Linear Op)](../sources/prs/vllm/PR-29213.md), [[LoRA] Optimize 3D MoE logic](../sources/prs/vllm/PR-29222.md), [[Perf] Disable DeepGEMM MoE by default when TP=8 is used](../sources/prs/vllm/PR-29346.md), [Add unpermute-aware fused MoE path and small-batch fallback](../sources/prs/vllm/PR-29354.md), [[Bugfix] Fix grouped_topk pytorch impl when num_experts can't be grouped properly](../sources/prs/vllm/PR-29439.md), [[Attention] Cache attention metadata builds across hybrid KV-cache groups](../sources/prs/vllm/PR-29627.md), [[Bugfix] Defunctionalize TRTLLM AR+Norm op for avoiding extra clone kernel before it](../sources/prs/vllm/PR-29631.md), [[Attention] Make `split_decodes_and_prefills(..., require_uniform=True)` support padding](../sources/prs/vllm/PR-29644.md), [[perf] Use direct copy (broadcast) instead of cat for k_nope/k_pe in MLA prefill](../sources/prs/vllm/PR-29710.md), [[Quantization] Enable compressed-tensors AWQ for Turing GPU](../sources/prs/vllm/PR-29732.md), [[Bugfix] Fix mismatched nvfp4 gemm output shape](../sources/prs/vllm/PR-29742.md), [[MoE-FP8-modelopt] Add FlashInfer alignment padding for intermediate dimensions](../sources/prs/vllm/PR-29748.md), [Add Mistral Large 3 and Ministral 3](../sources/prs/vllm/PR-29757.md), [[ROCm] [Fused Moe EP] Use binary expert mask for aiter fused moe kernel](../sources/prs/vllm/PR-29773.md), [[ROCm][MXFP4] Infer w4a4 quant method in rocm aiter fused moe](../sources/prs/vllm/PR-29775.md), [[EPLB] Support EPLB w/ NVFP4](../sources/prs/vllm/PR-29804.md), [[Bugfix][Model] Support LoRA on Qwen3 Output Embedding](../sources/prs/vllm/PR-29816.md), [[SpecDecode] Simplified alternative padded-speculation acceptance rate fix](../sources/prs/vllm/PR-29845.md), [[Quantization] fix: overflow with static per-tensor scaling](../sources/prs/vllm/PR-29867.md), [[Bugfix] Fix FP8 MoE LoRA](../sources/prs/vllm/PR-29890.md), [[BugFix] Fix DBO assert `assert B_block_table == B_q`](../sources/prs/vllm/PR-29933.md), [[moe] Use enable_chunking func (to support disabling chunking)](../sources/prs/vllm/PR-29935.md), [[moe] Allow disabling DP chunking](../sources/prs/vllm/PR-29936.md), [[Bugfix] Fix flashinfer ar+norm kernel not available issue](../sources/prs/vllm/PR-29960.md), [[ROCm] add fallback for aiter fp8 decode mla](../sources/prs/vllm/PR-30005.md), [[Quantization] Support Quark int4-fp8 w4a8 for MoE](../sources/prs/vllm/PR-30071.md), [[Model][Quantization] Restore MoE + GGUF models support (incl. Qwen3 MoE) by allowing Sideload Parameters](../sources/prs/vllm/PR-30116.md), [[Model][Quantization] Override HF defaults to GGUF ones (incl. Qwen3 MoE)](../sources/prs/vllm/PR-30118.md), [Nvidia ModelOpt workaround for issue 28072](../sources/prs/vllm/PR-30164.md), [Add latent MoE support](../sources/prs/vllm/PR-30203.md), [[Bugfix]: Fix glm46 awq marlin moe wna16 compatibility](../sources/prs/vllm/PR-30210.md), [[LoRA] Reduce the loading time of MoE LoRA](../sources/prs/vllm/PR-30243.md), [[Bugfix] Fix DeepGEMM after #29546 ](../sources/prs/vllm/PR-30267.md), [[Model][Quantization] Fix / Add GGUF support for Qwen2 MoE models](../sources/prs/vllm/PR-30307.md), [[bugfix][quantization] fix quark qwen3 kv_cache quantization](../sources/prs/vllm/PR-30308.md), [[fix] fix SM check for Flashinfer TRTLLM MOE](../sources/prs/vllm/PR-30314.md), [[Bugfix] Fix fp8 DeepGemm compilation issues](../sources/prs/vllm/PR-30336.md), [[BugFix] Fix `AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight_scale'`](../sources/prs/vllm/PR-30399.md), [fix(gguf): Disable bfloat16 for GGUF on blackwell device](../sources/prs/vllm/PR-30408.md), [[ROCm][Bugfix] Add MLACommonMetadata to allowed attention types for speculative decoding](../sources/prs/vllm/PR-30430.md), [[Bugfix] Pass FA version in `MultiHeadAttention`](../sources/prs/vllm/PR-30575.md), [[Perf] Eliminate padding and slicing op for GPT-OSS with Flashinfer MXFP4 MXFP8 MoE](../sources/prs/vllm/PR-30647.md), [[Misc][LLaMa4] Compile LLaMa Vision Encoder](../sources/prs/vllm/PR-30709.md), [Update note comment for flashinfer attention warmup](../sources/prs/vllm/PR-30711.md), [[Perf] enable flashinfer rotary_embedding custom ops in DeepSeek rotary](../sources/prs/vllm/PR-30729.md), [[Bugfix] Fix broken ViT attention selection for Blackwell device](../sources/prs/vllm/PR-30731.md), [[SM100] Enable fp8 compute for prefill MLA](../sources/prs/vllm/PR-30746.md), [Add support for LoRA adapters in Nemotron-H models](../sources/prs/vllm/PR-30802.md), [[Kernels][FI] Skip trtllm attention when num_kv_heads=1](../sources/prs/vllm/PR-30842.md), [[Compressed-Tensors] Simplify NVFP4 Conditions, enable marlin support for NVFP4A16 MoEs](../sources/prs/vllm/PR-30881.md), [[Kernel][Performance] Enable smaller Scaling Factor tiling for NVFP4 small-batch decoding](../sources/prs/vllm/PR-30885.md), [[Feature]: Support NVIDIA ModelOpt HF FP8 variants FP8_PER_CHANNEL_PER_TOKEN and FP8_PB_WO in vLLM](../sources/prs/vllm/PR-30957.md), [[Mics] add pcp basic support to MoE model](../sources/prs/vllm/PR-31003.md), [[Bugfix] Fix GLM-4 MoE router logits dtype for data parallel chunking](../sources/prs/vllm/PR-31055.md), [ [FIX] Always support TP > 4 for FP4 Gemm](../sources/prs/vllm/PR-31099.md), [[BugFix] LoRA: Support loading base_layer of experts](../sources/prs/vllm/PR-31104.md), [[Bugfix][Hardware][AMD] Consolidate FP8 min/max values helper function](../sources/prs/vllm/PR-31106.md), [[Misc] Fix grammar errors in comments and messages](../sources/prs/vllm/PR-31115.md), [[Bugfix] Fix MoE LoRA bin/pt loading](../sources/prs/vllm/PR-31161.md), [[perf] Integrate flashinfer concat_mla_k](../sources/prs/vllm/PR-31171.md), [[Bugfix][Hardware][AMD] Fix exception types in AITER MLA FP8 check](../sources/prs/vllm/PR-31177.md), [[SM100] Resubmit FMHA FP8 prefill for MLA](../sources/prs/vllm/PR-31195.md), [[Bugfix][Hardware][AMD] Fix last_page_len calculation in AITER MLA decode](../sources/prs/vllm/PR-31282.md), [fix(rocm): add early return in get_flash_attn_version for ROCm](../sources/prs/vllm/PR-31286.md), [pin lora_b moe weights on cpu](../sources/prs/vllm/PR-31317.md), [[Misc] Fix Qwen2-MoE shared_expert_gate](../sources/prs/vllm/PR-31339.md), [[BugFix] add select_gemm_impl on CompressedTensorsWNA16MoEMethod to support LoRA](../sources/prs/vllm/PR-31453.md), [[Bugfix][ROCm] Fix Static Quant Issue](../sources/prs/vllm/PR-31502.md), [[ROCm][Bugfix] Fix accuracy issue on fmoe when `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS` enabled](../sources/prs/vllm/PR-31523.md), [Use the same memory for workspace13 and fused_output.](../sources/prs/vllm/PR-31531.md), [[Fix] Align fused moe lora_b shape with peft](../sources/prs/vllm/PR-31534.md), [[Bugfix] Fix Broken ModelOpt NVFP4 MoE](../sources/prs/vllm/PR-31742.md), [[Perf] Add opt-in SM100 Oink RMSNorm custom-op path](../sources/prs/vllm/PR-31828.md), [[MISC] Add strict contiguity check for FlashInfer attention tensors](../sources/prs/vllm/PR-32008.md), [[5/N][Attention] Finish eliminating `vllm/attention` folder](../sources/prs/vllm/PR-32064.md), [[BugFix] Fix DeepSeek-V3.1 + DeepGEMM incompatible scale shapes](../sources/prs/vllm/PR-32361.md), [[Model] Molmo2: Enable quantized weight mapping for vision backbone](../sources/prs/vllm/PR-32385.md), [[Hardware][SM100] Add TRTLLM Kernel for INT4 W4A16 Kernel.](../sources/prs/vllm/PR-32437.md), [[Perf] Create TMA-aligned input scale tensor for DeepGemm on Hopper](../sources/prs/vllm/PR-32619.md), [[Kernel] use flashinfer for gdn prefill](../sources/prs/vllm/PR-32846.md), [[Performance] Tune Mamba selective scan kernel for B200](../sources/prs/vllm/PR-32873.md), [[Bugfix] Fix FP8 MoE EP Weight Loading for ModelOpt Llama4](../sources/prs/vllm/PR-32886.md), [[Spec Decode] Unified Parallel Drafting](../sources/prs/vllm/PR-32887.md), [[ROCm][perf] Shuffle KV cache to use paged_attention_common](../sources/prs/vllm/PR-32914.md), [[NVIDIA] [feat] Integrate flashinfer Trtllmgen bf16 moe](../sources/prs/vllm/PR-32954.md), [Support compress-tensors with nvfp4 or fp8 weights and modelopt with nvfp4 weights on Turing](../sources/prs/vllm/PR-33076.md), [[Attention] Use `has_flashinfer` helper](../sources/prs/vllm/PR-33177.md), [[Bugfix] Disable TRTLLM attention when KV transfer is enabled](../sources/prs/vllm/PR-33192.md), [[Bugfix] Register fp8 cutlass_group_gemm as supported for only SM90+SM100](../sources/prs/vllm/PR-33285.md), [[PERF] Change GDN Attention State Layout from [N, HV, K, V] to [N, HV, V, K]](../sources/prs/vllm/PR-33291.md), [[Kernel] Support Flashinfer trtllm fused MoE non gated FP8 & NVFP4](../sources/prs/vllm/PR-33506.md), [[Perf] Disable clean_logits in deepgemm fp8_mqa_logits kernel](../sources/prs/vllm/PR-33568.md), [[Bugfix] Fix sparse MLA metadata building](../sources/prs/vllm/PR-33579.md), [[Bugfix] fix DeepSeek R1 with CUTLASS MLA Broken on B200](../sources/prs/vllm/PR-33637.md), [enable skipping of SW attention layers when using FP8 KV cache](../sources/prs/vllm/PR-33695.md), [[Bugfix] Fix DSV3.2 NVFP4](../sources/prs/vllm/PR-33932.md), [Adding support to Sarvam's MoE models](../sources/prs/vllm/PR-33942.md), [[Bugfix] Relax TRTLLM KV cache contiguity assertion for cross-layer layout](../sources/prs/vllm/PR-34158.md), [[Bugfix] Fix DP Attention Padding in Dummy Run](../sources/prs/vllm/PR-34187.md), [[ModelBash][DSR1 NVFp4] Avoid Bf16 Bias Cast](../sources/prs/vllm/PR-34298.md), [[CPU][Perf] Accelerate Attention head for s390x using vector intrinsics](../sources/prs/vllm/PR-34434.md), [[Llama4,Quantization] Simplify and generalize logic for Q/K permutations in quantized self-attn layers ](../sources/prs/vllm/PR-34471.md), [[BUGFIX] Fix accuracy regression for NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 with TP>1](../sources/prs/vllm/PR-34476.md), [[Model] Add NVFP4 quantization support for Step3.5-Flash](../sources/prs/vllm/PR-34478.md), [[Bugfix] Handle num_expert_group=None in flashinfer block-scale FP8 MoE](../sources/prs/vllm/PR-34494.md), [[BugFix] Add support for MTP num_speculative_tokens > 1 with sparse MLA](../sources/prs/vllm/PR-34552.md), [[Quantization] add humming quantization kernel](../sources/prs/vllm/PR-34556.md), [[Bugfix] Rescale NVFP4 weight scales to fix BF16 dequant underflow](../sources/prs/vllm/PR-34577.md), [[Update] Use FlashInfer fast_decode_plan directly instead of replication](../sources/prs/vllm/PR-34687.md), [[Bugfix] Fix MLA attention crash with AWQ/GPTQ quantized models](../sources/prs/vllm/PR-34695.md), [[torch.compile] Turn on silu+fp4 quant fusion by default for O1+](../sources/prs/vllm/PR-34718.md), [[Bugfix] Fix NVFP4 TRTLLM MoE non-gated support; add gsm8k for Nemotron-3-Nano FP8+NVFP4](../sources/prs/vllm/PR-34725.md), [[Attention] Use FA4 for MLA prefill](../sources/prs/vllm/PR-34732.md), [[Bugfix] Fix GDN attention crash with mixed decode/spec-decode batches](../sources/prs/vllm/PR-34871.md), [[Model Bash][DSR1] Add selective dynamic shape marking for CustomOp](../sources/prs/vllm/PR-34900.md), [[Perf] Enable FlashInfer DeepGEMM swapAB on SM90 by default](../sources/prs/vllm/PR-34924.md), [add mixed precision support for modelopt](../sources/prs/vllm/PR-35047.md), [Integrate flashinfer mm_mxfp8 in ModelOpt MXFP8](../sources/prs/vllm/PR-35053.md), [[Bug][DSV3.2] Always prepare metadata for DeepGEMM Sparse Attention](../sources/prs/vllm/PR-35075.md), [[BUGFIX][Qwen3.5] Hardcode `mlp.gate` as not quantizable ](../sources/prs/vllm/PR-35156.md), [[Linear Attention] fix bug for linear attention + prefix caching + reset_prefix_cache](../sources/prs/vllm/PR-35157.md), [[BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU](../sources/prs/vllm/PR-35219.md), [[Performance] Extract KV cache update op from flashinfer forward](../sources/prs/vllm/PR-35422.md), [[Bugfix] Fix KV Scale loading for MLA Models](../sources/prs/vllm/PR-35430.md), [[Quant][Feature] Support online MXFP8 quantization for MoE and dense models](../sources/prs/vllm/PR-35448.md), [[NVFP4] Support NVFP4 dense models from `modelopt` and `compressed-tensors` on AMD Instinct MI300, MI355X and Hopper through emulation](../sources/prs/vllm/PR-35733.md), [Fix routed experts capture for hybrid models (Mamba + Attention)](../sources/prs/vllm/PR-35744.md), [[MoE][Perf] Wrap DSV3 QKVAProj GEMM in custom op for torch.compile](../sources/prs/vllm/PR-35751.md), [[Mamba] Add stochastic rounding support](../sources/prs/vllm/PR-35753.md), [[Kernel] Add fused_sigmoid_gating_delta_rule_update kernel for Qwen3 Next](../sources/prs/vllm/PR-35777.md), [[Bugfix] Fix score layer quantization for sequence classification models - Qwen3 (VL) Reranker](../sources/prs/vllm/PR-35849.md), [[ROCm] Support MLA with nhead<16 and FP8 KV cache for TP=8 (Kimi K2.5/Linear)](../sources/prs/vllm/PR-35850.md), [[Perf] Support FP8 KV cache for Flashinfer MLA Sparse](../sources/prs/vllm/PR-35891.md), [[MoE] Move PF Methods to Folder](../sources/prs/vllm/PR-35927.md), [Add support for ModelOpt MXFP8 MoE models](../sources/prs/vllm/PR-35986.md), [[Bugfix] Fix passing of activation_type to trtllm fused MoE NVFP4 and FP8](../sources/prs/vllm/PR-36017.md), [[BugFix] Fallback from FA4->FA2 for Batch Invariance](../sources/prs/vllm/PR-36059.md), [[LMCache] Pass TP size in lookup for MLA multi-reader locking](../sources/prs/vllm/PR-36129.md), [[Bugfix] Disable FlashInfer TRTLLM BF16 path for non-gated MoE](../sources/prs/vllm/PR-36146.md), [[Mamba] Flashinfer selective_state_update](../sources/prs/vllm/PR-36162.md), [[Bugfix][MLA] Add logits size budget to sparse indexer prefill chunking](../sources/prs/vllm/PR-36178.md), [[mla] Support fused FP8/NVFP4 output quantization in MLA attention (#35792)](../sources/prs/vllm/PR-36205.md), [mla: don't update kv cache on dummy forwards](../sources/prs/vllm/PR-36282.md), [[Perf] Add TRTLLM FP8 MoE Modular Kernel](../sources/prs/vllm/PR-36307.md), [Disable cascade attention by default](../sources/prs/vllm/PR-36318.md), [Kimi k2.5 MLA based eagle3](../sources/prs/vllm/PR-36361.md), [feat(attention): extract KV-cache update from FlashAttentionDiffKV ba…](../sources/prs/vllm/PR-36466.md), [[Bugfix][Sparse MLA] report indexer CG support properly](../sources/prs/vllm/PR-36519.md), [[ROCm] Utilize persistent MLA kernel from AITER](../sources/prs/vllm/PR-36574.md), [[GDN] add a config for gdn kernel selection](../sources/prs/vllm/PR-36647.md), [[Misc][Attention] Clean up unused method in `CPU_ATTN`](../sources/prs/vllm/PR-36673.md), [[Bug] Fix FlashInfer MNNVL socket collisions under concurrent vLLM jobs](../sources/prs/vllm/PR-36674.md), [[ROCm][Perf] Allow MTP lens > 1 in Sparse MLA](../sources/prs/vllm/PR-36681.md), [fix(kv-cache): increase hybrid attention grouping threshold from 1.25 to 1.5](../sources/prs/vllm/PR-36684.md), [[ROCm] Attention selector reordering](../sources/prs/vllm/PR-36702.md), [[DSV3.2][MTP] Optimize Indexer MTP handling](../sources/prs/vllm/PR-36723.md), [[Bug][MoE] Fix TRTLLM NVFP4 Routing Kernel Precision](../sources/prs/vllm/PR-36725.md), [[ROCm] Fix KV copy methods and auto-select attention backend for ROCm](../sources/prs/vllm/PR-36845.md), [[ROCm] Validate block_size for explicitly selected attention backends](../sources/prs/vllm/PR-36846.md), [[Feat][Spec Decode] DFlash](../sources/prs/vllm/PR-36847.md), [[Bugfix] Fix FlashInfer GDN warmup ValueError on SM90 GPUs](../sources/prs/vllm/PR-36876.md), [[Feat][Bugfix] Enable additional dimension for Flashinfer MLA and fix routing dtype](../sources/prs/vllm/PR-36931.md), [[Bugfix] Disable cross-layer KV cache for MLA attention backends](../sources/prs/vllm/PR-37090.md), [[Benchmark] Improvements to attention benchmark script](../sources/prs/vllm/PR-37115.md), [[XPU] support MLA model on Intel GPU](../sources/prs/vllm/PR-37143.md), [Fix minimax m2.5 nvfp4 kv scales weight loading](../sources/prs/vllm/PR-37214.md), [[ROCM][Bugfix] Use correct stride in cp_mha_gather_cache_kernel for hybrid model (#37228)](../sources/prs/vllm/PR-37228.md), [[Bugfix] Expand quantization method support in perf metrics](../sources/prs/vllm/PR-37231.md), [[Attention] Support distinguishing between short extends and decodes](../sources/prs/vllm/PR-37303.md), [[Bugfix] Fix EP weight filter breaking EPLB and NVFP4 accuracy](../sources/prs/vllm/PR-37322.md), [[Model Runner V2] fix draft attention metadata generation](../sources/prs/vllm/PR-37364.md), [[Bugfix] Remove assertion for NVFP4 scale dynamic range](../sources/prs/vllm/PR-37465.md), [[BugFix] Allow qk_nope_head_dim=192 in FlashInfer MLA backend checks](../sources/prs/vllm/PR-37475.md), [[Bugfix] Fix marlin nvfp4 rescaling](../sources/prs/vllm/PR-37502.md), [Fix KV Offloading + MLA AssertionError by using num_kv_heads=1 in cpu…](../sources/prs/vllm/PR-37536.md), [[Performance] Remove unnecessary zero-fill of MLA decode output tensor in Aiter backend](../sources/prs/vllm/PR-37539.md), [[Bugfix][ROCm] Fix lru_cache on paged_mqa_logits_module](../sources/prs/vllm/PR-37547.md), [[Bugfix] Disable --calculate-kv-scales for hybrid GDN/Mamba+Attention…](../sources/prs/vllm/PR-37565.md), [[Bugfix] Disable monolithic TRTLLM MoE for Renormalize routing (#37591)](../sources/prs/vllm/PR-37605.md), [[ROCm][Bugfix] fix cache block size mismatch for aiter unified attention](../sources/prs/vllm/PR-37606.md), [[Perf] Use torch compile to fuse pack topk in trtllm moe](../sources/prs/vllm/PR-37695.md), [[Bug] Fix fp8 deepgemm batch invariant](../sources/prs/vllm/PR-37718.md), [[Test] Only Run MLA model when user explicitly set for batch invariance](../sources/prs/vllm/PR-37719.md), [[XPU] add gptq(int4) support](../sources/prs/vllm/PR-37844.md), [[Feature] Support per-draft-model MoE backend via `--speculative-config`](../sources/prs/vllm/PR-37880.md), [[ROCm][perf] fix Aiter sparse MLA with MTP>1](../sources/prs/vllm/PR-37887.md), [[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell](../sources/prs/vllm/PR-38083.md), [Fix NaN from stale FP4 scale padding in create_fp4_scale_tensor](../sources/prs/vllm/PR-38148.md), [[Model Runner V2] Rebuild attention metadata before eagle decode full…](../sources/prs/vllm/PR-38311.md), [[MoE] Add RoutingMethodType.Simulated to TRT-LLM FP8/NVFP4 kernel allowlists](../sources/prs/vllm/PR-38329.md), [[GDN] Eliminate GPU->CPU sync in prepare_chunk_indices during prefill](../sources/prs/vllm/PR-38361.md), [[QeRL] Fix online quantized reloading](../sources/prs/vllm/PR-38442.md), [[XPU] Fix spec-decode UTs under tests/v1/spec_decode](../sources/prs/vllm/PR-38491.md), [[Bugfix][MLA] Change default SM100 MLA prefill backend back to TRT-LLM](../sources/prs/vllm/PR-38562.md), [[ROCm] Fix aiter persistent mode mla with q/o nhead<16 for kimi-k2.5 tp8](../sources/prs/vllm/PR-38615.md), [Fix MLA runs when use_inductor_graph_partition=True](../sources/prs/vllm/PR-38631.md), [[Bugfix] Fix AWQ models batch invariance issues](../sources/prs/vllm/PR-38670.md), [[XPU] add xpu backend implementation of mxfp8 quant](../sources/prs/vllm/PR-38682.md), [[Bugfix] Restrict TRTLLM attention to SM100, fixing GB300 (SM103) hang](../sources/prs/vllm/PR-38730.md), [[Bugfix] Fix test mocks after SM100 restriction in #38730](../sources/prs/vllm/PR-38791.md), [[LMCache][MP] optimize save when mla enabled](../sources/prs/vllm/PR-38810.md), [[FlashAttention] Symlink FA4 instead of copying when using `VLLM_FLASH_ATTN_SRC_DIR`](../sources/prs/vllm/PR-38814.md), [[Quant] add CompressedTensorsW8A8Mxfp8 for linear and MoE layers](../sources/prs/vllm/PR-38815.md), [[Attention][MLA] Re-enable FA4 as default MLA prefill backend](../sources/prs/vllm/PR-38819.md), [[Bugfix] Fix NVFP4+MTP crash: force unquantized mtp.fc for Qwen3.5](../sources/prs/vllm/PR-38832.md), [[Attention] relax the head dim 512 and paged kv for sm90+FA4](../sources/prs/vllm/PR-38835.md), [[Bugfix] Re-enable Renormalize routing for TRT-LLM MoE experts](../sources/prs/vllm/PR-38859.md), [[Gemma4] Enable Fast Prefill Optimization](../sources/prs/vllm/PR-38879.md), [[MoE Refactor] Split up compressed_tensors_moe.py](../sources/prs/vllm/PR-38960.md), [[Perf][GDN] Align TMA usage with upstream FLA](../sources/prs/vllm/PR-38981.md), [[Bug] Fix routing bias dtype for trtllm per-block fp8 moe](../sources/prs/vllm/PR-38989.md), [[Bugfix][MoE] Fix 6-8% decode regression: prefer multi-stream shared expert overlap](../sources/prs/vllm/PR-38990.md), [[Perf] Change Trtllm fp8 MoE to use Shuffled Weights and BlockMajorK Layout](../sources/prs/vllm/PR-38993.md), [[Quantization] - Layerwise reloading of Attention/KV quantized models](../sources/prs/vllm/PR-38995.md), [[Bugfix] Fix FlashInfer crash with kv_cache_dtype_skip_layers](../sources/prs/vllm/PR-39002.md), [[Gemma4] Support quantized MoE ](../sources/prs/vllm/PR-39045.md), [[Bug] Fix Trtllm Fp8 MoE Weight Shuffle Memory Fragamentation](../sources/prs/vllm/PR-39054.md), [[ROCm] Align AiterFlashAttentionImpl attn_type check with backend](../sources/prs/vllm/PR-39119.md), [[Refactor] Move NVFP4 GEMM management into NvFp4LinearKernel](../sources/prs/vllm/PR-39129.md), [[Refactor] Move MXFP8 GEMM management into MxFp8LinearKernel](../sources/prs/vllm/PR-39205.md), [[Bug] Fix rocm sparse attn indexer issue](../sources/prs/vllm/PR-39225.md), [[Bugfix] FlashInfer MXINT4 MoE crashes, missing do_finalize](../sources/prs/vllm/PR-39315.md), [[Feature] Batch invariant nvfp4 linear support](../sources/prs/vllm/PR-39322.md), [[Model Runner V2] Fix flex attention kv blocks calculation issue](../sources/prs/vllm/PR-39353.md), [[Bugfix][CT] Fix KV cache scale handling](../sources/prs/vllm/PR-39418.md), [[MLA] Optimize mla indexer prepare uniform decode for MTP > 1](../sources/prs/vllm/PR-39458.md), [[Kernel] Support TRTLLM GEN NVFP4 MoE for non-512-aligned hidden dims via weight padding](../sources/prs/vllm/PR-39510.md), [[Bugfix] Fix tensor shape mismatch in sparse attention with speculative decoding](../sources/prs/vllm/PR-39542.md), [[Mooncake] Fix mixed MLA+Eagle block-size validation](../sources/prs/vllm/PR-39596.md), [[XPU] properly handle q_descale on XPU as quant query input not supported](../sources/prs/vllm/PR-39676.md), [[Bugfix] Fix mismatch between global and local attention heads in tensor-parallel mode for param2moe model](../sources/prs/vllm/PR-39707.md), [[Bugfix] Reject non-nvfp4 dtypes when using the flashinfer_nvlink_one_sided all2all backend](../sources/prs/vllm/PR-39717.md), [[Bugfix][NIXL] Fix `_logical_to_kernel_block_ids` conversion for non-mamba models](../sources/prs/vllm/PR-39724.md), [add warning when FP8 KV cache misses prefill query quantization](../sources/prs/vllm/PR-39752.md), [[Bugfix] add support for 'num_attention_groups' in ModelArchConfigConvertorBase for Step3p5](../sources/prs/vllm/PR-39796.md), [[Bug] Fix batch invariance nvfp4 support](../sources/prs/vllm/PR-39820.md), [[Bugfix] Disable FlashInfer CUTLASS MoE on SM121 (DGX Spark)](../sources/prs/vllm/PR-39825.md), [[Core] Replace routing replay with device cache and async D2H pipeline](../sources/prs/vllm/PR-39917.md), [[Attention] use diff kv backend for mimo v2 flash](../sources/prs/vllm/PR-40045.md), [[Bugfix] Temporarily disable B200 fp4 MoE layer tests](../sources/prs/vllm/PR-40057.md), [Add nvfp4 kv cache support](../sources/prs/vllm/PR-40177.md), [Fix MoE backend selection for LoRA (unquantized MoE)](../sources/prs/vllm/PR-40273.md), [[Kernel][Helion] Optimize Helion config parsing latency](../sources/prs/vllm/PR-40850.md), [[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided](../sources/prs/vllm/PR-40960.md), [[Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4](../sources/prs/vllm/PR-41050.md), [[Bugfix][Hybrid][NemotronH] Fix mamba_cache_mode=all + speculative decoding crash](../sources/prs/vllm/PR-41233.md), [[ROCm][Quantization][3/N] Refactor quark_moe w4a4 w/ oracle](../sources/prs/vllm/PR-41436.md), [[Quantization] Rework quantization_config to use QuantKey and allow for activation override](../sources/prs/vllm/PR-41566.md), [[MXFP4] Support for linear layers + compressed-tensors integration](../sources/prs/vllm/PR-41664.md), [fix: remove unused norm for dpskv4](../sources/prs/vllm/PR-41710.md), [[Spec Decode] Add Gemma4 MTP speculative decoding support](../sources/prs/vllm/PR-41745.md), [Add NVFP4 all-gather GEMM fusion for AsyncTP](../sources/prs/vllm/PR-41882.md), [[CPU] Add MXFP4 W4A16 MoE support](../sources/prs/vllm/PR-41922.md), [[Bugfix] Add swiglu limits to deepgemm fp8 methods](../sources/prs/vllm/PR-41986.md), [[Bugfix] Fix TRTLLM ragged MLA prefill workspace warmup](../sources/prs/vllm/PR-42112.md), [[LoRA] Support 2D and 3D MoE LoRA adapter at the same time](../sources/prs/vllm/PR-42242.md), [[Bugfix] mamba: run single-token extends as decodes](../sources/prs/vllm/PR-42430.md), [Refactor AWQ Marlin MoE onto modular WNA16 oracle](../sources/prs/vllm/PR-42483.md), [[UX] Add a persistent cache for FlashInfer autotuning](../sources/prs/vllm/PR-42537.md), [[Bugfix] fix swiglu limit issue for humming backend + deepseek v4](../sources/prs/vllm/PR-42541.md), [[CPU] Add fused GDN support for AMX CPU platform](../sources/prs/vllm/PR-42707.md), [Fix Weight loading for Qwen3.5-MTP and Qwen3-VL using runai_streamer](../sources/prs/vllm/PR-42716.md), [[CPU] Specify required KV cache layout for CPU attention backend](../sources/prs/vllm/PR-42740.md), [[ROCm] [Bugfix] Fix DeepSeek V4 Functionality and Accuracy](../sources/prs/vllm/PR-42810.md), [[Perf] Add do_not_specialize in fused FP8 RoPE kernel](../sources/prs/vllm/PR-42849.md), [[Model Refactoring] Migrate DeepSeek V4 to vllm/models/ [1/N] ](../sources/prs/vllm/PR-43004.md), [[XPU] update xpu graph usage](../sources/prs/vllm/PR-43043.md), [[CI failure] Temporarily disable using persistent cache for flashinfer autotune](../sources/prs/vllm/PR-43119.md), [FlashAttention SM100 MLA TopK Sparse Forward](../wiki/kernels/flash-attention-sm100-mla-topk.md), [TensorRT-LLM Blackwell FP4 DSA Indexer](../wiki/kernels/tensorrt-llm-blackwell-indexer.md) | | `tilelang` | | [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [Tilelang sparse decode fwd for dsv32 mi355](../sources/prs/sglang/PR-18488.md), [[AMD] Tilelang sparse fwd for dsv32 mi355/mi300](../sources/prs/sglang/PR-19945.md), [[AMD] Enable FP8 KV cache and FP8 attention kernel for NSA on MI300/MI355 with TileLang backend](../sources/prs/sglang/PR-21511.md), [Amd/deepseek v4 rebase main 0509](../sources/prs/sglang/PR-24933.md), [[Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename](../sources/prs/sglang/PR-25821.md), [[WIP] support more dtypes for tcgen05](../sources/prs/tilelang/PR-1229.md), [[Enhancement] add more dtype and fix mma.ws for fp16 for tcgen05](../sources/prs/tilelang/PR-1327.md), [[Feat] profiler support cudagraph backend](../sources/prs/tilelang/PR-1658.md), [[Feature] Support `cp.reduce.async.bulk.tensor`](../sources/prs/tilelang/PR-1667.md), [Add swizzle layout detection and automatic merging for layout conflicts](../sources/prs/tilelang/PR-1736.md), [[Feature] Support tcgen5mma lowering for `.kind::i8`](../sources/prs/tilelang/PR-1764.md), [fix(intrinsics): add missing _legalize_to_buffer_region in SM70 emitter](../sources/prs/tilelang/PR-1786.md), [[BugFix] Fix Hopper TMA lowering without warp specialization](../sources/prs/tilelang/PR-1840.md), [[CUDA] Support tcgen5mma gemm ts](../sources/prs/tilelang/PR-1866.md), [[Feature] Support cluster launch, query, synchronization and barrier operations](../sources/prs/tilelang/PR-1874.md), [[Feature] 2-SM support for TMA, TMEM and TCGEN5MMA on Blackwell](../sources/prs/tilelang/PR-1882.md), [[Feature] Add T.copy_cluster to support TMA multicast and SM-to-SM cluster copy](../sources/prs/tilelang/PR-1908.md), [[Feature] Add Producer-Consumer Warp Specialization and T.tma_copy() API](../sources/prs/tilelang/PR-1909.md), [[Feature] Block-scaled GEMM support for MXFP8 on Blackwell](../sources/prs/tilelang/PR-1945.md), [[Bugfix] Fix CuTeDSL autotune cache invalid ELF header (#1967)](../sources/prs/tilelang/PR-1972.md), [[Feature] Support TMA store in T.tma_copy()](../sources/prs/tilelang/PR-1981.md), [[Transform] Add InjectTcgen05Fence pass](../sources/prs/tilelang/PR-2003.md), [[Backend] Refactor gemm_sp](../sources/prs/tilelang/PR-2048.md), [[CUDA] Support int4 `T.gemm`](../sources/prs/tilelang/PR-2063.md), [[CUDA] Improve int4 GEMM lowering and packed codegen support](../sources/prs/tilelang/PR-2073.md), [[TMA] Support FP4 TensorMap TMA copies](../sources/prs/tilelang/PR-2107.md), [feat: auto-vectorize bf16/fp16 reduce with packed add2 intrinsics](../sources/prs/tilelang/PR-2112.md), [[CUDA][TMA] Add TMA tile::gather4 / tile::scatter4 support](../sources/prs/tilelang/PR-2129.md), [[codex] Split GEMM implementations by backend](../sources/prs/tilelang/PR-2153.md), [[TIR][IR] Update to use tirx](../sources/prs/tilelang/PR-2216.md), [[Python] Drop Python 3.9 support](../sources/prs/tilelang/PR-2218.md) | | `triton` | [Triton on Blackwell](../wiki/languages/triton-blackwell.md) | [Gated Delta Networks](../sources/blogs/gated-delta-net.md), [Native Sparse Attention (NSA)](../sources/blogs/nsa.md), [FlashInfer MLSys 2026 - Track A: Fused MoE FP8](../sources/contests/flashinfer-mlsys26/track-a-fused-moe.md), [FlashInfer MLSys 2026 - Track B: DeepSeek V3.2 Sparse Attention](../sources/contests/flashinfer-mlsys26/track-b-sparse-attention.md), [FlashInfer MLSys 2026 - Track C: Gated Delta Net](../sources/contests/flashinfer-mlsys26/track-c-gated-delta-net.md), [Triton 3.6.0 Release Notes — Blackwell (SM100) Lowering](../sources/docs/triton-3.6-blackwell.md), [[None][fix] impl fused triton kernel for e8m0 resmooth to reduce memory footprint](../sources/prs/TensorRT-LLM/PR-10327.md), [[#11694][feat] AutoDeploy: Improve the piecewise CG memory usage](../sources/prs/TensorRT-LLM/PR-11993.md), [[https://nvbugs/5983390][perf] Kernel fusions in _gather_k_cache_for_chunk of Indexer in DSA](../sources/prs/TensorRT-LLM/PR-12322.md), [[https://nvbugs/5983390][perf] Multiple host perf optimizations for DSA part](../sources/prs/TensorRT-LLM/PR-12581.md), [[None][feat] Add triton paged attention for AutoDeploy](../sources/prs/TensorRT-LLM/PR-12642.md), [[#12784][feat] AutoDeploy: Optimize DeepSeek-R1 model performance](../sources/prs/TensorRT-LLM/PR-12946.md), [[#13580][fix] AutoDeploy: Support Gemma3n/4 E2B variants](../sources/prs/TensorRT-LLM/PR-13630.md), [[None][feat] Keep DSv4 o_a_proj as FP8, and port vLLM's fused_inv_rope_fp8_quant](../sources/prs/TensorRT-LLM/PR-13938.md), [[https://nvbugs/6152892][fix] Fix Triton MOE memory free when no swizzling enabled](../sources/prs/TensorRT-LLM/PR-14069.md), [[None][feat] Add chunked prefill support for Gemma4 (text + vision multimodal)](../sources/prs/TensorRT-LLM/PR-14134.md), [[None][fix] Add SPDX Apache-2.0 headers to auto_deploy test files](../sources/prs/TensorRT-LLM/PR-14193.md), [[None][fix] Update the OSS headers in derived FLA ops and AD modeling code](../sources/prs/TensorRT-LLM/PR-14281.md), [[None][chore] Update Claude Code agents and skills](../sources/prs/TensorRT-LLM/PR-14344.md), [feat: ragged tensor padding kernel for blackwell kernel alignment](../sources/prs/flashinfer/PR-1025.md), [[nvidia] initial support for blackwell kernels](../sources/prs/flashinfer/PR-1039.md), [benchmark: trtllm-gen mha with sink, add benchmark args](../sources/prs/flashinfer/PR-1415.md), [refactor: update fa3 codebase and fix hopper unittest [part 1]](../sources/prs/flashinfer/PR-2111.md), [Selective State Update kernel (mamba)](../sources/prs/flashinfer/PR-2301.md), [MTP for mamba ](../sources/prs/flashinfer/PR-2444.md), [misc: point triton blackwell-ptxas to local cuda ptxas](../sources/prs/flashinfer/PR-2543.md), [Mamba SSU: better automatic kernel selection + algorithm selection optionally exposed to the user.](../sources/prs/flashinfer/PR-2591.md), [int16 Block-Scaled State and Stochastic Rounding for SSU (mamba)](../sources/prs/flashinfer/PR-2645.md), [Add varlen and speculative decoding support to selective state update](../sources/prs/flashinfer/PR-2700.md), [Mamba2 SSD Combined Forward Pass (Blackwell CuTe DSL Kernel)](../sources/prs/flashinfer/PR-2709.md), [fix(sm12x): fix micro-kernel workspace sizing when routed_rows > num_local_experts](../sources/prs/flashinfer/PR-3191.md), [checkpointing_ssu kernel: fused replay + conditional state-write for Mamba2](../sources/prs/flashinfer/PR-3324.md), [SM-constraint-GEMM by triton persistent kernel](../sources/prs/flashinfer/PR-982.md), [Triton `rms_norm` kernels](../sources/prs/flashinfer/PR-983.md), [[inductor] Fix profiler tests with latest Triton](../sources/prs/pytorch/PR-149059.md), [[inductor][triton 3.3] Fix cpp_wrapper w/ TMA in triton 3.3](../sources/prs/pytorch/PR-149993.md), [Fix uint view copy (#151598)](../sources/prs/pytorch/PR-154121.md), [[user triton] AOT inductor support for device-side TMA](../sources/prs/pytorch/PR-157241.md), [[release] Triton pin update to 3.4](../sources/prs/pytorch/PR-157752.md), [[cherry-pick][inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack](../sources/prs/pytorch/PR-158646.md), [[Inductor][Intel GPU] Save `threads_per_warp` from tirton compiled kernel for launching kernel correctly in cpp wrapper.](../sources/prs/pytorch/PR-163388.md), [[2.9 cherry pick][triton] update 3.5 pin to bbb06c0334a6772b92d24bde54956e675c8c6604 (#163382)](../sources/prs/pytorch/PR-163583.md), [[AARCH64][CD][CUDA13][Triton][PTXAS] Turn on BUILD_BUNDLE_PTXAS=1 ](../sources/prs/pytorch/PR-164236.md), [[Minor][Inductor] move some combo kernel log from warning to debug](../sources/prs/pytorch/PR-167020.md), [[RELEASE 2.10] Release only changes](../sources/prs/pytorch/PR-170112.md), [[ROCm] Enable shared memory based pruning for Triton configs](../sources/prs/pytorch/PR-170190.md), [[Inductor] Fix constants handling for Triton constexpr (triton#8248)](../sources/prs/pytorch/PR-171129.md), [[RELEASE 2.11] Release only changes](../sources/prs/pytorch/PR-175091.md), [[inductor] avoid multi-stage for mix-order-red by default (#176228)](../sources/prs/pytorch/PR-176495.md), [[release 2.12] Apply Release only changes to 2.12 branch](../sources/prs/pytorch/PR-180470.md), [Add support for bf16 x bf16 cutlass fused MoE](../sources/prs/sglang/PR-10275.md), [support qwen3_next blackwell](../sources/prs/sglang/PR-10403.md), [Fix bias handling in TritonMoeQuantInfo within quantization/mxfp4.py](../sources/prs/sglang/PR-10579.md), [Fix MTP MoE weight loading with NVFP4 target model.](../sources/prs/sglang/PR-10758.md), [Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization](../sources/prs/sglang/PR-11081.md), [[AMD] Clean up vllm dependencies in moe_runner/triton.py](../sources/prs/sglang/PR-11349.md), [Support shared experts overlap in cutlass moe](../sources/prs/sglang/PR-11611.md), [[Ascend] qwen optimization](../sources/prs/sglang/PR-12078.md), [Enable Flashinfer TRTLLM-GEN-MoE FP8 blockwise kernel for Qwen3-Next on Blackwell](../sources/prs/sglang/PR-12543.md), [[Bugfix] Fix illegal memory access](../sources/prs/sglang/PR-12758.md), [Support internvl on Blackwell (which doesn't support fa3): add `SingletonCache` support to Vision{Sdpa|Triton|Ascend}Attention](../sources/prs/sglang/PR-13151.md), [Fix nan in global scaling factor for large scale nvfp4 EP](../sources/prs/sglang/PR-13162.md), [[ROCM] Optimized deepseek-r1 fp8 model with + triton_gemm_a8w8 + batch_gemm_a8w8 + fused set_mla_kv_buffer kernel](../sources/prs/sglang/PR-13617.md), [[Feat][NVFP4] Enable NVFP4 MoE for Qwen series models (eg. Qwen3-Next) #13761](../sources/prs/sglang/PR-13761.md), [Support fp4 fp8 non gated moe](../sources/prs/sglang/PR-13794.md), [[NVIDIA] Enable TRTLLM BF16 MoE on Blackwell GPUs](../sources/prs/sglang/PR-13798.md), [[BugFix] fix outplace_fused_experts missing is_gated](../sources/prs/sglang/PR-13864.md), [Feat: GLM-4.6 supports shared experts fusion](../sources/prs/sglang/PR-13873.md), [[LoRA][III] Add LoRA support for MoE layers and enable TP](../sources/prs/sglang/PR-14105.md), [Apply new moe wna16 marlin gemm](../sources/prs/sglang/PR-14125.md), [Apply new moe align block size kernel](../sources/prs/sglang/PR-14134.md), [Add Mistral Large 3 support.](../sources/prs/sglang/PR-14213.md), [[Fix] add block size logic for sm120 smem size](../sources/prs/sglang/PR-14311.md), [[FIX] trtllm-moe-fp4-renorm for Qwen series models](../sources/prs/sglang/PR-14350.md), [Apply back moe_sum_reduce for fused_marlin_moe](../sources/prs/sglang/PR-14829.md), [Mistral Large 3 NVFP4 TRTLLM MoE support](../sources/prs/sglang/PR-15049.md), [Optimize FP8 MLA KV cache writes with Triton kernel](../sources/prs/sglang/PR-15522.md), [MoE: Skip SiLU/GELU activation for masked experts](../sources/prs/sglang/PR-15539.md), [Update flashinfer to 0.6.1](../sources/prs/sglang/PR-15551.md), [Add SwapAB Optimization for triton fused_moe_kernel on SM90.](../sources/prs/sglang/PR-15712.md), [ Add tuned triton==3.5.1 h200 tp2, tp4 for qwen 3 next](../sources/prs/sglang/PR-15948.md), [[Performance] Force split_k=1 for MXFP4 Triton kernels on Hopper](../sources/prs/sglang/PR-16014.md), [optimize get_topk_ragged by fusing get k and k_scale triton kernel](../sources/prs/sglang/PR-16043.md), [fix layer intermediate size](../sources/prs/sglang/PR-16084.md), [[NemotronH] Add latent MoE support](../sources/prs/sglang/PR-16227.md), [[Rework] Add SwapAB Optimization for triton fused_moe_kernel on SM90.](../sources/prs/sglang/PR-16723.md), [[Fix] `flashinfer_trtllm` `intermediate_size` assertion with Qwen3 + TP=8](../sources/prs/sglang/PR-16824.md), [Support mxint4 flashinfer_trtllm moe gemm](../sources/prs/sglang/PR-16892.md), [[GLM 4.7] Add RTX 6000 Pro aka sm120](../sources/prs/sglang/PR-17235.md), [Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE](../sources/prs/sglang/PR-17449.md), [[AMD] Update aiter to v0.1.10.post2](../sources/prs/sglang/PR-18423.md), [[sglang-miles] True on-policy training support for FSDP2](../sources/prs/sglang/PR-18639.md), [Adjust padding size to improve triton_kernels moe performance](../sources/prs/sglang/PR-19174.md), [Fix/nemotron mtp quantaized](../sources/prs/sglang/PR-19433.md), [[FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE](../sources/prs/sglang/PR-19537.md), [[diffusion][llm] macOS support](../sources/prs/sglang/PR-19549.md), [[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)](../sources/prs/sglang/PR-19652.md), [Support `triton_kernels` for GPT-OSS on SM120](../sources/prs/sglang/PR-19718.md), [Fix SM120 `triton_kernels` MXFP4 `block_k` for GPT-OSS](../sources/prs/sglang/PR-20040.md), [[Benchmark] use flashinfer bench_gpu_time instead of triton do_bench](../sources/prs/sglang/PR-20305.md), [Support Triton MLA FP8 KV cache](../sources/prs/sglang/PR-20479.md), [Add SGLang CUDA crash API logging inspired by FlashInfer](../sources/prs/sglang/PR-20910.md), [[Qwen3.5] Fuse split/reshape/cat ops in GDN projection with Triton kernel](../sources/prs/sglang/PR-21019.md), [[refactor] Clean up duplicate flashinfer trtllm moe code](../sources/prs/sglang/PR-21233.md), [Refactor JIT kernel CI to use run_suite.py registration system](../sources/prs/sglang/PR-21239.md), [[RL] Support mxfp8 DeepSeek V3](../sources/prs/sglang/PR-21280.md), [[misc] clean up kernel API](../sources/prs/sglang/PR-21325.md), [Change default mm-attention backend from triton_attn to fa4](../sources/prs/sglang/PR-21595.md), [[XPU] Enable qwen3.5 on XPU](../sources/prs/sglang/PR-21668.md), [[Fix] Fall back to triton MOE for GPT-OSS on Blackwell with driver >= 595](../sources/prs/sglang/PR-21780.md), [fix pcg torch dynamo recompile in mxfp8 Triton path](../sources/prs/sglang/PR-21888.md), [[nvidia] Gemma4 nvfp4 fix](../sources/prs/sglang/PR-22079.md), [[Lora] Lora quat info re-factor and support deepseekv3 mla lora](../sources/prs/sglang/PR-22323.md), [diffusion: add HunyuanVideo GroupNorm+SiLU fast path](../sources/prs/sglang/PR-22814.md), [[codex] diffusion: enable group norm silu fuse by default](../sources/prs/sglang/PR-23148.md), [Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-23686.md), [Optimize large GroupNorm SiLU apply](../sources/prs/sglang/PR-23938.md), [[diffusion] Fuse LTX2 split rotary embedding](../sources/prs/sglang/PR-24411.md), [Port MXFP4 Marlin MoE support to JIT kernel path](../sources/prs/sglang/PR-24490.md), [Fix performance regression on Deepseek V3 on `moe-runner-backend=triton` on SM90](../sources/prs/sglang/PR-24562.md), [[Codex] Opt Mistral Large performace ](../sources/prs/sglang/PR-24611.md), [[rebase]Deepseek_v4 support w4(mxfp4)a16 on hopper](../sources/prs/sglang/PR-24986.md), [[MUSA][Diffusion] Improve wan model inference speed using torch.compile](../sources/prs/sglang/PR-25256.md), [[Gemma4]: Fix FP8 Triton scale layout](../sources/prs/sglang/PR-25286.md), [[Intel GPU] Enable DeepSeek V4 Inference on XPU](../sources/prs/sglang/PR-25336.md), [Update logging for inplace setting in MoE layer](../sources/prs/sglang/PR-25499.md), [Fix logging for inplace setting in the flashInfer-trtllm backend](../sources/prs/sglang/PR-25522.md), [amd/deepseek_v4 27/N [fix] Reduce Triton autotune configs for faster first-time server launch](../sources/prs/sglang/PR-25554.md), [Add DeepSeekV4 fused MoE Triton autotune support](../sources/prs/sglang/PR-25569.md), [Use triton_attn as default vision attention on B300 (SM103)](../sources/prs/sglang/PR-25570.md), [[diffusion] Fix GLM-Image /v1/images/edits support](../sources/prs/sglang/PR-25697.md), [[Refactor] Rename NSA → DSA: user-facing aliases, file/class/import rename](../sources/prs/sglang/PR-25821.md), [[Test] Stage-a sanity kits; consolidate core/ + models_e2e/ tests](../sources/prs/sglang/PR-25831.md), [Feature DeepSeek V3/R1 INT8 Quantization (block-wise)](../sources/prs/sglang/PR-3730.md), [[Feature] DeepSeek V3/R1 INT8 Quantization (channel-wise) ](../sources/prs/sglang/PR-3888.md), [Add fast decode plan for flashinfer mla](../sources/prs/sglang/PR-3987.md), [[Revision] Add fast decode plan for flashinfer mla ](../sources/prs/sglang/PR-4012.md), [Accelerate FP8 CUDA Kernel by 20-28%](../sources/prs/sglang/PR-4215.md), [Feat/support encoder model (like bert)](../sources/prs/sglang/PR-4887.md), [Add DeepSeek V3/R1 shared experts fusion](../sources/prs/sglang/PR-4918.md), [reduce moe_align_block_size_kernel small batch mode overhead](../sources/prs/sglang/PR-5086.md), [Support tuning moe for llama 4 model](../sources/prs/sglang/PR-6042.md), [feat: mtp support dp-attention](../sources/prs/sglang/PR-6081.md), [enable auto-round quantization model](../sources/prs/sglang/PR-6226.md), [fix: enable multi-GPU Triton fused MoE tuning](../sources/prs/sglang/PR-6295.md), [reduce torch.zeros overhead in moe align block size kernel](../sources/prs/sglang/PR-6369.md), [Refine pre_reorder_triton_kernel slightly to improve performance](../sources/prs/sglang/PR-6627.md), [[CPU] [BF16] Call fused_experts_cpu, weight_packed_linear and bmm_cpu kernel in DeepSeek model](../sources/prs/sglang/PR-6641.md), [Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled](../sources/prs/sglang/PR-6736.md), [[CPU] add optimizations for INT8 and FP8 DeepSeek](../sources/prs/sglang/PR-6769.md), [[CPU] support the case where num_attention_heads or intermediate_size is not divisible by the TP size](../sources/prs/sglang/PR-6771.md), [[DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model](../sources/prs/sglang/PR-6853.md), [Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa](../sources/prs/sglang/PR-6890.md), [chore: upgrade flashinfer v0.2.6.post1 jit](../sources/prs/sglang/PR-6958.md), [Fuse routed scaling factor in deepseek](../sources/prs/sglang/PR-6970.md), [Update default settings for blackwell](../sources/prs/sglang/PR-7023.md), [Fix positional argument](../sources/prs/sglang/PR-7093.md), [Enable ModelOpt Llama4 fp8 checkpoint deployment in SGLang](../sources/prs/sglang/PR-7129.md), [Support new DeepGEMM](../sources/prs/sglang/PR-7172.md), [Fix grammar abort & Minor style fixes](../sources/prs/sglang/PR-7204.md), [FlashInfer NVFP4 MoE with EP & 2-stream shared expert](../sources/prs/sglang/PR-7327.md), [Fix MTP with Deepseek R1 Fp4](../sources/prs/sglang/PR-7376.md), [Fix torch compile run](../sources/prs/sglang/PR-7391.md), [Add fp4 quantize before all-gather for Flashinfer cutlass MoE DP (max throughput)](../sources/prs/sglang/PR-7667.md), [Integrate triton moe kernel](../sources/prs/sglang/PR-7689.md), [[feat] Support tp mode for DeepSeek-R1-W4AFP8](../sources/prs/sglang/PR-8118.md), [Support triton kernels v3.4.0 for fused_moe](../sources/prs/sglang/PR-8258.md), [[NVIDIA] Add Low Latency NVFP4 decode kernels from Flashinfer](../sources/prs/sglang/PR-8552.md), [fuse allreduce and residual_rmsnorm](../sources/prs/sglang/PR-8731.md), [[Perf] Auto enable best flashinfer mxfp4 kernel in b200](../sources/prs/sglang/PR-8898.md), [[fix] Fix mxfp4 triton MoE tp bug](../sources/prs/sglang/PR-9473.md), [Optimize moe_sum_reduce_kernel](../sources/prs/sglang/PR-9477.md), [Single Batch Overlap for MoE Models](../sources/prs/sglang/PR-9660.md), [[Kernel] add triton fused moe kernel for gptq/awq](../sources/prs/vllm/PR-12185.md), [[AMD][Quantization] Add TritonScaledMMLinearKernel since int8 is broken for AMD](../sources/prs/vllm/PR-12282.md), [[Attention] MLA decode optimizations](../sources/prs/vllm/PR-12528.md), [[Attention] Deepseek v3 MLA support with FP8 compute](../sources/prs/vllm/PR-12601.md), [[Attention] MLA with chunked prefill](../sources/prs/vllm/PR-12639.md), [[Perf] Mem align KV caches for CUDA devices (MLA perf improvement)](../sources/prs/vllm/PR-12676.md), [[core] Perf improvement for DSv3 on AMD GPUs](../sources/prs/vllm/PR-13718.md), [[Attention] MLA support for V1](../sources/prs/vllm/PR-13789.md), [[Attention] Flash MLA for V1](../sources/prs/vllm/PR-13867.md), [dynamic distpatch of fp8 kernels](../sources/prs/vllm/PR-14245.md), [[BugFix] MLA + V1, illegal memory access and accuracy issues](../sources/prs/vllm/PR-14253.md), [[BugFix][TritonMLA] Process weights after model loading for GGUF](../sources/prs/vllm/PR-14555.md), [[BugFix] Fix nightly MLA failure (FA2 + MLA chunked prefill, i.e. V1, producing bad results)](../sources/prs/vllm/PR-15492.md), [[ROCM][KERNEL] Paged attention for V1](../sources/prs/vllm/PR-15720.md), [Modularize fused experts and integrate PPLX kernels](../sources/prs/vllm/PR-15956.md), [[Kernel][Bugfix] Re-fuse triton moe weight application](../sources/prs/vllm/PR-16071.md), [Upstream Llama4 Support to Main](../sources/prs/vllm/PR-16113.md), [[Bug] [ROCm] Fix Llama 4 Enablement Bug on ROCm: V0 ROCmFlashAttentionImpl and Triton Fused MoE bugs](../sources/prs/vllm/PR-16198.md), [[Kernel] Support W8A8 channel-wise weights and per-token activations in triton fused_moe_kernel](../sources/prs/vllm/PR-16366.md), [[Perf]Optimize rotary_emb implementation to use Triton operator for improved inference performance](../sources/prs/vllm/PR-16457.md), [Enable PTPC FP8 for CompressedTensorsW8A8Fp8MoEMethod (triton fused_moe)](../sources/prs/vllm/PR-16537.md), [[torch.compile][ROCm] Fuse quantization onto attention using a torch.compile pass](../sources/prs/vllm/PR-16756.md), [[Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode](../sources/prs/vllm/PR-16828.md), [[Bugfix] Triton FA function takes no keyword arguments](../sources/prs/vllm/PR-16902.md), [[Attention] MLA move o_proj q_proj into cuda-graph region](../sources/prs/vllm/PR-17484.md), [fix amd triton mla path](../sources/prs/vllm/PR-17871.md), [[Bugfix][ROCm] Use `chunked_prefill_paged_decode` as fallback for V1 attention on ROCm](../sources/prs/vllm/PR-18093.md), [[Kernel] Integrate CUTLASS MoE kernel with PPLX](../sources/prs/vllm/PR-18762.md), [[Kernel] Enable fp8 support for pplx and BatchedTritonExperts.](../sources/prs/vllm/PR-18864.md), [[Kernels] Add activation chunking logic to FusedMoEModularKernel](../sources/prs/vllm/PR-19168.md), [[Bugfix] Don't attempt to use triton if no driver is active](../sources/prs/vllm/PR-19561.md), [ [Feature] Integrate SM100 DeepGEMM support](../sources/prs/vllm/PR-20087.md), [[Kernel] Optimize Prefill Attention in Unified Triton Attention Kernel](../sources/prs/vllm/PR-20308.md), [[Misc] DP : Add ExpertTokensMetadata](../sources/prs/vllm/PR-20332.md), [[Perf] Use Triton instead of Torch for DeepGEMM Per Token Group Quant](../sources/prs/vllm/PR-20841.md), [[Kernel] DeepGemm MoE : Integrate triton permute / unpermute kernels ](../sources/prs/vllm/PR-20903.md), [[v1] Add Whisper model support (encoder-decoder)](../sources/prs/vllm/PR-21088.md), [[Attention] Clean up iRoPE in V1](../sources/prs/vllm/PR-21188.md), [[Kernel] Enable Hybrid Model Support in Triton Unified Attention Kernel](../sources/prs/vllm/PR-21197.md), [[Feature][Kernel]FusedMoE LoRA](../sources/prs/vllm/PR-21229.md), [[v1][attention] Support Hybrid Allocator + FlashInfer](../sources/prs/vllm/PR-21412.md), [[Bugfix] Add proper comparison for package versions](../sources/prs/vllm/PR-22314.md), [[BugFix] Fix triton compile error in `kernel_unified_attention_2/3d` caused by attention sinks](../sources/prs/vllm/PR-22368.md), [[gpt-oss] triton kernel mxfp4](../sources/prs/vllm/PR-22421.md), [[NVIDIA] Support Flashinfer TRTLLM FP8-q/kv NVFP4-out Attention Kernel](../sources/prs/vllm/PR-22703.md), [fp8 kv cache support fix for torch.compile](../sources/prs/vllm/PR-22758.md), [[Misc] Add @tdoublep as a maintainer of hybrid model and Triton-attention related code](../sources/prs/vllm/PR-23122.md), [[ROCm][Aiter] Add triton fp8 bmm kernel for mla](../sources/prs/vllm/PR-23264.md), [[Bugfix] Fixing division by zero in triton_attn if query_heads/kv_heads > 16 ](../sources/prs/vllm/PR-23424.md), [[Feature] Add Hopper DeepGEMM E8M0 for DeepSeekV3.1 scale_fmt](../sources/prs/vllm/PR-23666.md), [[Feature] Support Decode Context Parallel (DCP) for MLA](../sources/prs/vllm/PR-23734.md), [[Bug] Fix Shape Validation for Fallback while Enabling E8M0 for DeepGEMM](../sources/prs/vllm/PR-26322.md), [Move query quantization to attention layer for Flashinfer & Triton.](../sources/prs/vllm/PR-26534.md), [[Attention] Use sparse prefill kernel for fp8 kv-cache in DeepSeek-v3.2](../sources/prs/vllm/PR-27532.md), [fix cross attention](../sources/prs/vllm/PR-28346.md), [[Model] Add support for openPangu moe model](../sources/prs/vllm/PR-28775.md), [[Feature] Batch invariant: Enable `TRITON_MLA` without prefix-caching](../sources/prs/vllm/PR-29125.md), [[Bugfix] Only use triton_kernels for MXFP4 on SM90 and SM100](../sources/prs/vllm/PR-29339.md), [[LoRA] Support Quantized Adapters](../sources/prs/vllm/PR-30286.md), [[ROCm][Quantization] GPT OSS Upstream MoE wmxfp4_afp8 with static scales](../sources/prs/vllm/PR-30357.md), [[Perf] Set split_k to 1 for triton_kernels](../sources/prs/vllm/PR-30528.md), [[Bugfix] Fix Triton FusedMoE LoRA](../sources/prs/vllm/PR-30585.md), [Triton Attention: Support cross-layers blocks](../sources/prs/vllm/PR-30687.md), [fused_moe_lora PDL improvements](../sources/prs/vllm/PR-30716.md), [[Bugfix] [Kernel] Triton attention kernels: mask out V blocks that fall outside sliding window](../sources/prs/vllm/PR-30887.md), [[Bugfix] Fix incorrect tiles creation for mm prefix triton attention](../sources/prs/vllm/PR-30974.md), [Use aiter triton fused_add_rmsnorm_pad for gpt-oss](../sources/prs/vllm/PR-30976.md), [[Bugfix][ROCm]Fix Qwen3-Next-80B-A3B-Thinking inference and optimize non-standard block size (544) support under rocm_atten](../sources/prs/vllm/PR-31380.md), [fixed mypy warnings for files vllm/v1/attention with TEMPORARY workaround](../sources/prs/vllm/PR-31465.md), [[FIX] Add NO_MUL activation support for modular kernel path](../sources/prs/vllm/PR-31528.md), [[LoRA]Disable linear LoRA kernel PDL](../sources/prs/vllm/PR-31777.md), [[1/N][Attention] Restructure attention: move files](../sources/prs/vllm/PR-31916.md), [[4/N][Attention] Move MLA common to model_executor](../sources/prs/vllm/PR-32060.md), [Add TMA support to fused_moe_lora kernel](../sources/prs/vllm/PR-32195.md), [[Bugfix][Attention] Explicitly report support for kv_cache_dtype bfloat16](../sources/prs/vllm/PR-32795.md), [Triton MLA perf fixes](../sources/prs/vllm/PR-33529.md), [[Kernel] Add FP8 KV cache support to Triton MLA decode attention](../sources/prs/vllm/PR-34597.md), [fix(mxfp4): return is_monolithic=False when LoRA is enabled for Triton backend](../sources/prs/vllm/PR-35382.md), [[XPU] Support block fp8 moe by fallback to TritonExpert on XPU](../sources/prs/vllm/PR-36458.md), [[Kernel] Fuse FP8 output quantization into merge_attn_states](../sources/prs/vllm/PR-36518.md), [[Bugfix] Warm up Triton autotuner for GDN layers during V1 profiling](../sources/prs/vllm/PR-36599.md), [[Bugfix] Fix KV scales inconsistency in fp8 MLA & FlashInfer kv_cache_dtype "auto" leading to gibberish](../sources/prs/vllm/PR-37054.md), [[MoE Refactor] Mxfp4 oracle rebased](../sources/prs/vllm/PR-37128.md), [[Misc] Add `float16` to `CacheDType`](../sources/prs/vllm/PR-37199.md), [[NIXL][BUG] Fix Triton heterogeneous TP](../sources/prs/vllm/PR-37940.md), [[Perf] triton bilinear_pos_embed kernel for ViT](../sources/prs/vllm/PR-37948.md), [[Perf] FP8 FlashInfer Attn for ViT](../sources/prs/vllm/PR-38065.md), [[Bugfix] Enable batch-invariant Triton matmul on all Ampere GPUs (SM 8x) ](../sources/prs/vllm/PR-38427.md), [[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity](../sources/prs/vllm/PR-38479.md), [[Kernels][MoE] Fix legacy_routing to use bitmatrix-based routing path](../sources/prs/vllm/PR-38504.md), [[Perf] Reduce H2D pageable memory copies](../sources/prs/vllm/PR-38794.md), [[MoE] Move GPT OSS Triton kernel experts into fused_moe/experts/](../sources/prs/vllm/PR-39007.md), [[XPU] Quick fix for TritonMLA to remove cuda hardcode](../sources/prs/vllm/PR-39088.md), [[MoE] Move cutlass moe to fused_moe/experts/](../sources/prs/vllm/PR-40574.md), [[Attention][TurboQuant] Share dequant buffers, eliminate float16_copy](../sources/prs/vllm/PR-40941.md), [[MoE] Move various experts classes to fused_moe/experts/](../sources/prs/vllm/PR-41979.md), [[feat] Add FP8 per-tensor Q scale support to Triton attention backend](../sources/prs/vllm/PR-42080.md), [[Perf] Wire silu_and_mul_per_block_quant into TritonFP8MoE (MiniMax-M2) ](../sources/prs/vllm/PR-42497.md), [[Kernel] Pack topk id/weights triton kernel](../sources/prs/vllm/PR-42527.md), [[Perf][MLA] Enable FULL cudagraph capture for TRITON_MLA decode](../sources/prs/vllm/PR-42885.md), [Fused MoE — FP8 Block-Scale Routing + Dual GEMM](../wiki/kernels/fused-moe.md), [Gated Delta Net — Linear Attention](../wiki/kernels/gated-delta-net.md), [Native Sparse Attention (NSA)](../wiki/kernels/nsa.md) | diff --git a/references/primer.md b/references/primer.md index f1662a271..14cdbb7b4 100644 --- a/references/primer.md +++ b/references/primer.md @@ -101,6 +101,23 @@ All page IDs below resolve via `get_page.py `. All paths are relative to the --- +## Ampere Extension (SM80 / SM86 backport target) + +Added for running this wiki's knowledge on Ampere hardware (A100, RTX 3090-class rigs). Start at `migration-hopper-to-ampere` when the question is "how do I do X from this wiki on a 3090/A100". + +| Topic | Page ID | Path | Notes | +|---|---|---|---| +| Hopper/Blackwell → Ampere backport map | `migration-hopper-to-ampere` | `wiki/migration/hopper-to-ampere-backport.md` | Instruction replacement table, capacity re-planning, scheduling paradigm shift, ncu checklist. THE entry point. | +| cp.async (async global→shared) | `hw-cp-async` | `wiki/hardware/cp-async.md` | What TMA becomes on sm_8x: per-thread 16 B copies, commit-groups, zfill OOB, multistage pipeline snippet. | +| mma.sync + ldmatrix | `hw-mma-sync-ampere` | `wiki/hardware/mma-sync-ampere.md` | Warp-scope synchronous tensor core; register economics; GA10x FP32-accumulate half-rate quirk; no stmatrix. | +| Ampere memory model | `hw-ampere-memory-model` | `wiki/hardware/ampere-memory-model.md` | sm_80 vs sm_86 capacity card (99 KB SMEM/block, 1536 thr/SM, 6 MB L2 on 3090), L2 persistence window, roofline position. | + +Ampere source docs: `doc-ampere-tuning-guide`, `doc-ga102-whitepaper`, `doc-ptx-isa-ampere`, `doc-cutlass-ampere`. + +Techniques that transfer to Ampere as-is: swizzling, double-buffering, pipeline-stages (cp.async flavor), persistent-kernel (atomic queue instead of CLC), tile-scheduling (more important — small L2), vectorized-loads, cache-policy, register-budgeting, kernel-fusion. Techniques that do NOT transfer: anything tagged tcgen05/tmem/clc/2sm-cooperative/tma-multicast — consult the migration table for each. + +--- + ## Source Repositories (PR coverage) | Repo | PR pages | Ledger | @@ -150,6 +167,11 @@ When the user types one of these, match to the canonical term shown: | MLA, multi-head latent attention | `mla` | | GDN, GatedDeltaNet, gated delta rule | `gated-delta-net` | | NSA, native sparse attention | `sparse-attention` | +| Ampere, A100, A800, GA100, SM80 | `sm80` | +| GA102, RTX 3090, RTX 3090 Ti, RTX 3080, RTX A6000, A40, A10, SM86 | `sm86` | +| cp.async, LDGSTS, async copy, cuda::memcpy_async | `cp-async` | +| mma.sync, HMMA, m16n8k16, warp MMA | `mma-sync` | +| L2 persistence, cudaAccessPolicyWindow, L2 set-aside | `l2-persistence` | The `query.py` tool applies these automatically when scoring and when using `--tag`. diff --git a/scripts/query.py b/scripts/query.py index a7f303494..2e19aeeb3 100755 --- a/scripts/query.py +++ b/scripts/query.py @@ -119,10 +119,13 @@ def score_keyword_match(fm, body, keywords): """ score = 0 title_text = str(fm.get("title", "")).lower() + def _as_list(v): + return v if isinstance(v, list) else ([v] if v else []) tag_text = " ".join( str(v) for k in ("tags", "techniques", "hardware_features", "kernel_types", - "languages", "aliases", "symptoms") - for v in (fm.get(k) or []) + "languages", "aliases", "symptoms", "architectures", + "from_arch", "to_arch") + for v in _as_list(fm.get(k)) ).lower() body_lower = body.lower() for kw in keywords: diff --git a/sources/docs/cutlass-ampere-support.md b/sources/docs/cutlass-ampere-support.md new file mode 100644 index 000000000..c692b27ae --- /dev/null +++ b/sources/docs/cutlass-ampere-support.md @@ -0,0 +1,51 @@ +--- +id: doc-cutlass-ampere +title: "CUTLASS Ampere (SM80/SM86) Support: MmaMultistage, CuTe SM80 Atoms, 3xTF32" +url: https://docs.nvidia.com/cutlass/ +source_category: official-doc +architectures: [sm80, sm86] +tags: [cp-async, mma-sync, ldmatrix, swizzling, double-buffering, pipeline-stages] +retrieved_at: 2026-08-03 +--- + +# CUTLASS Ampere (SM80/SM86) Support + +## Overview + +CUTLASS is the reference implementation for high-performance Ampere GEMM. Its SM80 code path is the canonical public example of the full Ampere idiom: multi-stage `cp.async` software pipeline feeding warp-scope `mma.sync` tensor core ops via `ldmatrix`, with swizzled shared-memory layouts. Anyone backporting Hopper/Blackwell kernel structure to sm_86 should read the CUTLASS SM80 mainloop first — it IS the target shape of the port. + +## The SM80 mainloop: MmaMultistage + +CUTLASS 2.x introduced `cutlass::gemm::threadblock::MmaMultistage` — the Ampere replacement for the Volta/Turing 2-stage `MmaPipelined`: + +- **N-stage circular SMEM buffer** (typically `Stages = 3..5`, bounded by SMEM: 99 KB/block on sm_86, 163 KB on sm_80). +- Global→shared operand movement issued with `cp.async` (`CacheOperation::Global` → `.cg` 16B for main operands, `.ca` for small fragments). +- Completion via `cp.async.wait_group`/commit-group counting; no mbarriers in the classic mainloop. +- Warp-level tile compute in `cutlass::gemm::warp::MmaTensorOp`, which lowers to `mma.sync.aligned.m16n8k16` (arch `cutlass::arch::Mma, 32, ...>`) with `ldmatrix`-based operand fetch (`cutlass::arch::LdMatrix`). +- Shared-memory layouts use XOR-swizzles (`TensorOpMultiplicandCrosswise` etc.) sized to keep `ldmatrix` bank-conflict-free. + +## CuTe (C++) SM80 atoms + +CUTLASS 3.x re-expresses the same hardware in CuTe: + +- MMA atoms: `SM80_16x8x16_F32F16F16F32_TN`, `SM80_16x8x16_F32BF16BF16F32_TN`, `SM80_16x8x8_F32TF32TF32F32_TN`, INT8 variants — all wrap `mma.sync`. +- Copy atoms: `SM80_CP_ASYNC_CACHEALWAYS` / `SM80_CP_ASYNC_CACHEGLOBAL` (cp.async .ca/.cg), `SM75_U32x4_LDSM_N` / `SM75_U16x8_LDSM_T` (ldmatrix, non-transposed/transposed). +- Tutorial `examples/cute/tutorial/sgemm_sm80.cu` walks the full pattern; `cute::cp_async_fence()` / `cute::cp_async_wait()` expose commit-group semantics. +- CollectiveBuilder path: SM80 collectives use the multistage schedule (no warp specialization — contrast with SM90 `KernelTmaWarpSpecialized*` schedules). + +## Ampere-specific examples in the CUTLASS tree + +| Example | What it demonstrates | +|---|---| +| `14_ampere_tf32_tensorop_gemm` | TF32 tensor-core GEMM on FP32 data | +| `15_ampere_sparse_tensorop_gemm` | 2:4 structured sparsity (`mma.sp`) | +| `27_ampere_3xtf32_fast_accurate_tensorop_gemm` | **3xTF32**: emulate FP32 GEMM with 3 TF32 MMAs (error-compensated), ~2x+ over FP32 CUDA cores at near-FP32 accuracy | +| `sgemm_sm80.cu` (CuTe tutorial) | Minimal readable multistage cp.async + mma.sync mainloop | + +Note: the newer Python CuTe DSL examples tree is Hopper/Blackwell-first; check the current `examples/python/CuTeDSL/` tree before assuming SM80 DSL coverage. The mature, performance-proven Ampere path is the C++ one above. + +## Practical notes for sm_86 specifically + +- CUTLASS kernels compiled for sm_80 run on sm_86, but tile shapes tuned for A100's 164 KB SMEM will fail to launch or spill on GA10x's 99 KB — pick smaller `Stages` or tile sizes (the profiler's sm_86 presets do this). +- Threadblock shapes that assume 2048 threads/SM occupancy targets should be re-tuned for 1536. +- FP16 accumulation (`ElementAccumulator = half_t`) doubles tensor throughput on GA10x (see doc-ga102-whitepaper) at the usual numerical risk; CUTLASS supports it natively. diff --git a/sources/docs/nvidia-ampere-tuning-guide.md b/sources/docs/nvidia-ampere-tuning-guide.md new file mode 100644 index 000000000..2872d8a26 --- /dev/null +++ b/sources/docs/nvidia-ampere-tuning-guide.md @@ -0,0 +1,76 @@ +--- +id: doc-ampere-tuning-guide +title: "NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6)" +url: https://docs.nvidia.com/cuda/ampere-tuning-guide/ +source_category: official-doc +architectures: [sm80, sm86] +tags: [cuda-cpp, cp-async, l2-persistence, mbarrier] +retrieved_at: 2026-08-03 +--- + +# NVIDIA Ampere GPU Architecture Tuning Guide (CC 8.0 / 8.6) + +## Overview + +The official NVIDIA Ampere tuning guide covers both Ampere compute capabilities: **8.0 (GA100: A100/A800)** and **8.6 (GA10x: RTX 3090/3080, RTX A6000, A40, A10)**. The two are NOT the same architecture from a tuning standpoint — occupancy ceilings, shared memory capacity, and FP32 issue rate all differ. This page records the facts that matter when writing or backporting kernels for sm_86. + +## CC 8.0 vs CC 8.6 — the table that matters + +| Resource | CC 8.0 (A100) | CC 8.6 (GA10x) | +|---|---|---| +| Max concurrent warps / SM | 64 (2048 threads) | **48 (1536 threads)** | +| Max thread blocks / SM | 32 | **16** | +| Unified L1/SMEM capacity / SM | 192 KB | **128 KB** | +| Max SMEM carveout / SM | 164 KB | **100 KB** | +| Max SMEM per thread block | 163 KB | **99 KB** | +| FP32 ops per cycle per SM | 64 | **128 (2x)** | + +Implications: + +- **Occupancy math changes on sm_86.** A block size that yields 100% occupancy on A100 (e.g., 2048 threads via 2x1024) caps at 75% of the sm_80 figure on GA10x. Blocks of 256/512 threads with ≤3 blocks resident hit the 1536-thread ceiling exactly. +- **SMEM-heavy Hopper-style kernels do not fit.** 99 KB per block is the hard ceiling on sm_86 (vs 163 KB on A100, 227 KB on H100). Multi-stage pipelines must shrink stage count or tile size accordingly. +- **The 2x FP32 pipe** (both datapaths can issue FP32; on GA100 one is FP32-only, the other INT32) means non-tensor-core elementwise/epilogue code is comparatively cheap on GA10x. + +## Asynchronous Data Copies (cp.async) + +Ampere introduces asynchronous copy from global to shared memory, bypassing the register file and (optionally) L1: + +- `cp.async.ca.shared.global` — 4, 8, or 16 bytes, caches in L1. +- `cp.async.cg.shared.global` — 16 bytes only, caches only in L2 (bypasses L1). Recommended for tensor-core operand staging. +- Completion managed either via **commit-group semantics** (`cp.async.commit_group` / `cp.async.wait_group N`) or via **asynchronous barriers** (`cuda::barrier`, PTX `mbarrier`, available since sm_80). +- The C++ surface is `cuda::memcpy_async` + `cuda::pipeline` (libcudax / ``). + +This is the Ampere ancestor of Hopper TMA: per-thread addressing, no bulk tensor descriptors, no multicast, no swizzle-on-the-fly — but the same "load directly to SMEM without burning registers" principle. + +## L2 Persistence (Access Policy Window) + +CC 8.0+ can set aside a portion of L2 for **persisting** accesses via `cudaAccessPolicyWindow` (per-stream or per-graph-node attribute): + +```cpp +cudaStreamAttrValue attr; +attr.accessPolicyWindow.base_ptr = kv_cache_ptr; +attr.accessPolicyWindow.num_bytes = window_bytes; // <= cudaLimitMaxL2FetchGranularity window +attr.accessPolicyWindow.hitRatio = 0.6f; // fraction treated as persisting +attr.accessPolicyWindow.hitProp = cudaAccessPropertyPersisting; +attr.accessPolicyWindow.missProp = cudaAccessPropertyStreaming; +cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &attr); +``` + +`cudaDeviceProp::persistingL2CacheMaxSize` reports the maximum set-aside. Useful for small hot operands re-read across many CTAs (e.g., decode-time KV heads, router weights). Streaming accesses can be marked `evict_first` via PTX cache policies to avoid polluting the persisting region. + +## Other sm_80+ features recorded by the guide + +- **Asynchronous barriers** (`cuda::barrier`): arrive/wait split, enables producer/consumer overlap without `__syncthreads()` full-block convergence. +- **Warp reduce instructions** (`redux.sync`) for int operands. +- **Improved L2 residency management + larger L2** vs Volta/Turing. +- **Third-generation tensor cores**: BF16 and TF32 inputs; `mma.sync` shapes up to m16n8k16 (FP16/BF16) and m16n8k8 (TF32). + +## What Ampere does NOT have (forward-looking) + +Recorded here because this KB is used for Hopper/Blackwell backports; all of the following first appear in sm_90 or sm_100: + +- No TMA (`cp.async.bulk.tensor`), no multicast loads. +- No thread block clusters, no distributed shared memory (DSMEM). +- No `wgmma` / `tcgen05` asynchronous tensor core instructions — only synchronous warp-scope `mma.sync`. +- No TMEM, no `setmaxnreg` register reallocation, no `stmatrix` (sm_90+), no PDL/GDC kernel-launch overlap. +- No FP8/FP6/FP4 tensor core datatypes (INT8/INT4 via `mma.sync` are available). diff --git a/sources/docs/nvidia-ga102-whitepaper.md b/sources/docs/nvidia-ga102-whitepaper.md new file mode 100644 index 000000000..d4270fa27 --- /dev/null +++ b/sources/docs/nvidia-ga102-whitepaper.md @@ -0,0 +1,52 @@ +--- +id: doc-ga102-whitepaper +title: "NVIDIA Ampere GA102 GPU Architecture Whitepaper (v2.1)" +url: https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.1.pdf +source_category: official-doc +architectures: [sm86] +tags: [mma-sync, cuda-cpp, fp8] +retrieved_at: 2026-08-03 +--- + +# NVIDIA Ampere GA102 GPU Architecture Whitepaper (v2.1) + +## Overview + +The GA102 whitepaper is the official spec source for consumer/workstation Ampere (sm_86): GeForce RTX 3090/3090 Ti/3080, RTX A6000, A40. The single most important fact for kernel work it documents: **on GA10x, FP16 tensor core math with FP32 accumulation runs at HALF the rate of FP16 accumulation** — a product segmentation choice that GA100 (A100) does not have. Every GEMM/attention kernel that accumulates in FP32 (i.e., all numerically-safe LLM kernels) pays this on RTX 3090. + +## RTX 3090 key specs (from the whitepaper tables) + +| Spec | RTX 3090 | +|---|---| +| SMs | 82 | +| CUDA cores | 10496 (128 FP32/SM) | +| Tensor cores | 328 (3rd gen, 4/SM) | +| Boost clock | 1.70 GHz | +| Memory | 24 GB GDDR6X, 384-bit, 936 GB/s | +| L2 cache | 6 MB | +| RT cores | 82 (irrelevant for compute) | + +## Tensor core peak rates, RTX 3090 (dense / sparse) + +| Path | Dense TFLOPS/TOPS | With 2:4 sparsity | +|---|---|---| +| FP16 in, **FP16 accumulate** | 142 | 284 | +| FP16 in, **FP32 accumulate** | **71** | 142 | +| BF16 in, FP32 accumulate | 71 | 142 | +| TF32 | 35.6 | 71 | +| INT8 | 284 | 568 | +| INT4 | 568 | 1136 | + +Notes: + +- The FP32-accumulate half-rate applies to the whole GA10x line including RTX A6000/A40 — it is a GA10x trait, not a GeForce-driver limitation. +- A100 (GA100, sm_80) runs FP16→FP32 accumulate at full rate (312 TFLOPS dense). +- BF16 inputs REQUIRE FP32 accumulation in `mma.sync` PTX, so BF16 on GA10x is always on the half-rate path. FP16 with FP16 accumulate is the only full-rate floating path — usable for attention P·V under careful scaling, risky for general GEMM. +- INT8 (284 TOPS) is 4x the FP32-accumulate FP16 rate — why weight-only INT8/INT4 (Marlin-style) and SmoothQuant-style W8A8 shine on 3090-class cards. + +## Architectural notes relevant to kernels + +- GA10x SM: 4 processing blocks (warp schedulers), 128 KB unified L1/SMEM, 64K 32-bit registers per SM (255 regs/thread max), 4 tensor cores. +- Datapath: both 64-lane FP32 pipes usable → 128 FP32 FMA/clk/SM; one pipe shares with INT32. +- PCIe 4.0 x16 host link. RTX 3090/3090 Ti, RTX A6000, and A40 support a 2-way NVLink bridge; RTX 3080 and below do not. Multi-GPU rigs without bridges fall back to P2P over PCIe. +- 2:4 structured sparsity supported by the sparse `mma.sp` path. diff --git a/sources/docs/nvidia-ptx-isa-ampere.md b/sources/docs/nvidia-ptx-isa-ampere.md new file mode 100644 index 000000000..1f745fef1 --- /dev/null +++ b/sources/docs/nvidia-ptx-isa-ampere.md @@ -0,0 +1,99 @@ +--- +id: doc-ptx-isa-ampere +title: "PTX ISA Ampere (sm_80/sm_86) Instructions Reference" +url: https://docs.nvidia.com/cuda/parallel-thread-execution/ +source_category: official-doc +architectures: [sm80, sm86] +tags: [ptx, cp-async, mma-sync, ldmatrix, mbarrier, cache-policy] +retrieved_at: 2026-08-03 +--- + +# PTX ISA Ampere (sm_80/sm_86) Instructions Reference + +## Overview + +PTX ISA 7.0+ introduces the sm_80-generation instructions that define Ampere kernel programming: asynchronous global→shared copies (`cp.async`), warp-scope synchronous tensor core MMA (`mma.sync` with the m16n8kX shapes), shared-memory matrix loads (`ldmatrix`), and shared-memory barriers (`mbarrier`). This page summarizes syntax and constraints as documented in the PTX ISA, for use when writing sm_86 kernels or backporting sm_90/sm_100 code. + +## cp.async — asynchronous copy (sm_80+) + +```asm +cp.async.ca.shared.global [smem_dst], [gmem_src], cp-size{, src-size}; // cp-size ∈ {4, 8, 16} +cp.async.cg.shared.global [smem_dst], [gmem_src], 16{, src-size}; // 16 bytes only +cp.async.commit_group; +cp.async.wait_group N; // wait until ≤ N groups in flight +cp.async.wait_all; +``` + +- `.ca` allocates in L1 and L2; `.cg` allocates in L2 only (use for tensor-core operand tiles). +- Optional `src-size` (< cp-size) reads fewer bytes and **zero-fills** the remainder of the shared destination — the OOB-guard idiom without branches. +- An optional cache-policy operand (`createpolicy.fractional.L2::evict_first` etc.) can be attached (`cp.async.ca.shared.global.L2::cache_hint`). +- Completion alternatives: commit-group counting (above) or `cp.async.mbarrier.arrive.shared.b64 [mbar];` which makes an mbarrier track the async group. +- Destination must be `.shared` state space, addresses must be naturally aligned to cp-size. + +## mma.sync — warp-scope tensor core MMA (shapes per dtype) + +```asm +// FP16 inputs, FP32 accumulator (the LLM workhorse): +mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 + {%f0,%f1,%f2,%f3}, // D: 4x f32 per thread + {%r0,%r1,%r2,%r3}, // A: 4x b32 (8x f16) per thread + {%r4,%r5}, // B: 2x b32 (4x f16) per thread + {%f0,%f1,%f2,%f3}; // C (accumulate in place) +``` + +Key shapes available on sm_80/sm_86: + +| Inputs | Accumulator | Shapes | +|---|---|---| +| f16 | f16 or f32 | m16n8k8, m16n8k16 | +| bf16 | **f32 only** | m16n8k8, m16n8k16 | +| tf32 | f32 | m16n8k4, m16n8k8 | +| s8/u8 | s32 | m16n8k16, m16n8k32 | +| s4/u4 | s32 | m16n8k32, m16n8k64 | +| f64 | f64 | m8n8k4 | + +- All operands live in **registers**, fragmented across the 32 lanes of one warp in a fixed documented layout; accumulators stay in registers between iterations (this is what TMEM replaces on Blackwell). +- `mma.sync` is **synchronous at warp scope**: the issuing warp stalls until the result lands — there is no `wgmma.mma_async`/`tcgen05.mma` style asynchronous tensor op before sm_90. +- Sparse variant `mma.sp` implements 2:4 structured sparsity with a metadata operand. +- Legacy `wmma.*` (m16n16k16) remains available but the m16n8kX `mma.sync` family is the canonical high-performance path (what CUTLASS emits). + +## ldmatrix — shared→register matrix fragment load (sm_75+) + +```asm +ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%r0,%r1,%r2,%r3}, [smem_addr]; +ldmatrix.sync.aligned.m8n8.x4.trans.shared.b16 {%r0,%r1,%r2,%r3}, [smem_addr]; +``` + +- Loads 1/2/4 8x8 b16 tiles per instruction into the exact register fragment layout `mma.sync` expects; `.trans` transposes at load (needed for column-major B operands). +- Each of the 32 lanes supplies a row-start address (lane i → row i of the tiles); addresses typically computed with a swizzle to avoid bank conflicts. +- **`stmatrix` (register→shared store) is sm_90+.** On Ampere the epilogue writes accumulators to shared/global with ordinary `st` instructions — one of the notable asymmetries versus Hopper. + +## mbarrier — shared-memory barrier (sm_80+) + +```asm +mbarrier.init.shared.b64 [mbar], count; +mbarrier.arrive.shared.b64 %r0, [mbar]; +mbarrier.test_wait.shared.b64 %p, [mbar], %r0; // poll +mbarrier.try_wait.shared.b64 %p, [mbar], %r0; // blocking-ish (sm_90 adds .parity forms used with TMA) +``` + +- Ampere mbarriers support init/arrive/test_wait/invalidate plus `cp.async.mbarrier.arrive` integration; the C++ view is `cuda::barrier`. +- **`mbarrier.arrive.expect_tx` and transaction-count tracking are sm_90+** (designed for TMA byte counting) — an Ampere backport must count arrivals, not bytes. + +## Cache policy / eviction hints (sm_80+) + +```asm +createpolicy.fractional.L2::evict_first.b64 %pol, 1.0; +ld.global.L2::cache_hint.f32 %f0, [gaddr], %pol; +ld.global.L1::no_allocate.v4.f32 {...}, [gaddr]; // streaming loads that skip L1 +``` + +- `evict_first` / `evict_last` / `no_allocate` policies let streaming tensors bypass or minimally pollute caches — pairs with the L2 persistence window set on the host side. +- `ld.global.nc` (`__ldg`) non-coherent constant-cache path remains available. + +## Other sm_80-relevant instructions + +- `redux.sync.op.u32` — warp-level reduction in one instruction (int only on Ampere). +- `cvt` with `.rn` packed conversions f32↔bf16x2 (`cvt.rn.bf16x2.f32`) for epilogue downcasts. +- `bar.warp.sync`, `shfl.sync`, standard since Volta, unchanged. +- NO `cp.async.bulk*`, NO `tensormap.*`, NO `wgmma.*`, NO `tcgen05.*`, NO `stmatrix`, NO `setmaxnreg`, NO `clusterlaunchcontrol.*` on sm_8x — attempting to compile any of these for sm_86 is an immediate ptxas error (the fastest way to inventory what a Hopper kernel needs replaced). diff --git a/wiki/hardware/ampere-memory-model.md b/wiki/hardware/ampere-memory-model.md new file mode 100644 index 000000000..da93e88cd --- /dev/null +++ b/wiki/hardware/ampere-memory-model.md @@ -0,0 +1,89 @@ +--- +id: hw-ampere-memory-model +title: "Ampere Memory Model: SMEM/L2/Occupancy on sm_80 vs sm_86" +type: hardware +architectures: [sm80, sm86] +tags: [cp-async, l2-persistence, cache-policy, shared-memory-optimization, mbarrier] +confidence: source-reported +reproducibility: snippet +related: [hw-cp-async, hw-mma-sync-ampere, technique-tile-scheduling, technique-cache-policy, technique-swizzling, migration-hopper-to-ampere] +sources: [doc-ampere-tuning-guide, doc-ga102-whitepaper, doc-ptx-isa-ampere] +aliases: ["L2 persistence", cudaAccessPolicyWindow, "shared memory carveout", "GA102 memory"] +blackwell_relevance: "Capacity planning is where Hopper/Blackwell kernel configs break first on Ampere: 99 KB SMEM/block (vs 227 KB) and single-digit-MB L2 (vs 50–126 MB) invalidate tile shapes and scheduling assumptions before any instruction is translated." +--- + +# Ampere Memory Model: SMEM/L2/Occupancy on sm_80 vs sm_86 + +## Overview + +Two different "Amperes" exist. GA100 (A100, sm_80) is the datacenter chip most papers mean; GA10x (RTX 3090/3080, RTX A6000, A40 — sm_86) is what consumer rigs run. They differ exactly where kernel configuration lives: SMEM capacity, occupancy ceilings, L2 size, and DRAM technology. This page is the capacity card for both, plus the L2-persistence and async-barrier features shared across Ampere. + +## Capacity card + +| Resource | A100 (sm_80) | RTX 3090 (sm_86) | H100 (sm_90, for contrast) | +|---|---|---|---| +| SMs | 108 | 82 | 132 | +| Max threads / SM | 2048 (64 warps) | **1536 (48 warps)** | 2048 | +| Max blocks / SM | 32 | **16** | 32 | +| Registers | 64K × 32-bit / SM, 255/thread | same | same | +| Unified L1/SMEM | 192 KB | **128 KB** | 256 KB | +| Max SMEM carveout / SM | 164 KB | **100 KB** | 228 KB | +| Max SMEM / block (opt-in) | 163 KB | **99 KB** | 227 KB | +| L2 | 40 MB | **6 MB** | 50 MB | +| DRAM | HBM2e, ~1.6–2.0 TB/s | GDDR6X, 936 GB/s | HBM3, 3.35 TB/s | +| BF16 tensor peak (f32 acc) | 312 TFLOPS | **71 TFLOPS** | 989 TFLOPS | + +Deriving the two headline constraints for sm_86 backports: + +1. **SMEM budget: 99 KB per block.** A Hopper kernel using 200+ KB (FA3-style Q/K/V staging, 4–6 pipeline stages) must shrink stage count, tile size, or both. Practical Ampere configs: 2–4 stages, ≤48 KB static + dynamic opt-in via `cudaFuncAttributeMaxDynamicSharedMemorySize`. +2. **Occupancy: 1536 threads/SM.** Three 512-thread blocks or six 256-thread blocks fill an SM. A 2×1024 config (fine on A100) strands 25% of the sm_86 thread budget. + +## Roofline position: why 3090 optimization priorities differ + +FLOP/byte knee = tensor peak ÷ DRAM bandwidth: + +- RTX 3090 (bf16, f32 acc): 71e12 / 936e9 ≈ **76 FLOP/B** +- A100 (bf16): 312e12 / 2.0e12 ≈ 156 FLOP/B +- H100 (bf16): 989e12 / 3.35e12 ≈ 295 FLOP/B + +The 3090 is *bandwidth-rich relative to its tensor throughput*: kernels cross into compute-bound territory at ~4× lower arithmetic intensity than on H100. Consequences: + +- GEMM-class kernels saturate tensor cores earlier — operand-delivery tricks (deep pipelining, giant tiles) pay off less than on Hopper; hitting the 71/142 TFLOPS ceiling is the game. +- Memory-bound kernels (decode GEMV, elementwise, KV-cache scans) sit closer to parity with datacenter cards than the TFLOPS gap suggests — this is why weight-only-quantized decode on GA10x is competitive. +- The small 6 MB L2 makes **tile scheduling for L2 reuse** (technique-tile-scheduling) *more* important than on A100/H100, not less: a 128×128 bf16 output tile's operand footprint already competes for the whole cache. + +## L2 persistence window (sm_80+) + +Pin a hot region (router weights, decode KV heads, quant scales) into a set-aside L2 slice: + +```cpp +// host side — compilable, CUDA 11+ +#include +void pin_l2(cudaStream_t stream, void* ptr, size_t bytes) { + cudaDeviceProp prop; cudaGetDeviceProperties(&prop, 0); + size_t setaside = prop.persistingL2CacheMaxSize; // e.g. up to 75% of L2 + cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, setaside); + cudaStreamAttrValue v{}; + v.accessPolicyWindow.base_ptr = ptr; + v.accessPolicyWindow.num_bytes = bytes; // ≤ accessPolicyMaxWindowSize + v.accessPolicyWindow.hitRatio = 1.0f; + v.accessPolicyWindow.hitProp = cudaAccessPropertyPersisting; + v.accessPolicyWindow.missProp = cudaAccessPropertyStreaming; + cudaStreamSetAttribute(stream, cudaStreamAttributeAccessPolicyWindow, &v); +} +``` + +On a 6 MB-L2 part this is a scarce resource — reserve it for operands re-read by *many* CTAs, and mark streaming tensors `evict_first` (PTX cache hints, technique-cache-policy) so they don't fight the window. + +## Async barriers (sm_80+) + +`cuda::barrier` (PTX `mbarrier`) enables arrive/wait split-phase sync — producer warps arrive after issuing `cp.async`, consumer warps wait without a full `__syncthreads()`. Ampere supports init/arrive/test_wait and `cp.async.mbarrier.arrive`; it does NOT support sm_90's `expect_tx` byte-count tracking (count arrivals instead — see hw-cp-async and migration-hopper-to-ampere). + +## SMEM banking (unchanged since Volta, still decisive) + +32 banks × 4 B; `ldmatrix` issues 32 row-reads per instruction, so unswizzled row-major tiles with power-of-two leading dimensions conflict catastrophically. Standard fix: XOR-swizzle the store index of `cp.async` destinations (technique-swizzling). GA10x and GA100 behave identically here — swizzle math ports from Hopper kernels unchanged, minus the TMA hardware swizzle modes (do it in index arithmetic). + +## GDDR6X behavior notes (sm_86) + +- 936 GB/s is the pin-rate ceiling; sustained achievable in well-coalesced kernels is ≈85–90% of it, and random-access patterns degrade harder than on HBM (narrower effective row buffer locality). +- Latency is higher than HBM — deeper cp.async pipelines (or higher occupancy) are needed to cover it, in tension with the smaller SMEM. This is THE central tuning trade-off on GA10x: stages × tile SMEM ≤ 99 KB while still covering GDDR6X latency. diff --git a/wiki/hardware/cp-async.md b/wiki/hardware/cp-async.md new file mode 100644 index 000000000..2a8843bca --- /dev/null +++ b/wiki/hardware/cp-async.md @@ -0,0 +1,147 @@ +--- +id: hw-cp-async +title: "cp.async — Asynchronous Global→Shared Copy (Ampere)" +type: hardware +architectures: [sm80, sm86] +tags: [cp-async, mbarrier, pipeline-stages, double-buffering] +confidence: source-reported +reproducibility: snippet +related: [hw-tma, hw-mbarrier, technique-pipeline-stages, technique-double-buffering, migration-hopper-to-ampere, hw-ampere-memory-model] +sources: [doc-ptx-isa-ampere, doc-ampere-tuning-guide, doc-cutlass-ampere] +aliases: [cp.async, LDGSTS, "async copy", "cuda::memcpy_async"] +blackwell_relevance: "cp.async remains available on sm_90/sm_100 for small non-bulk operands, but on Ampere it is the ONLY asynchronous load path — every TMA-based Hopper/Blackwell design backported to sm_8x lands on cp.async." +--- + +# cp.async — Asynchronous Global→Shared Copy (Ampere) + +## Overview + +`cp.async` (SASS: LDGSTS) is Ampere's asynchronous copy instruction: data moves from global memory directly into shared memory **without staging through the register file**, and the issuing warp continues executing. It is the mechanism behind every multi-stage software pipeline on sm_80/sm_86 — the CUTLASS `MmaMultistage` mainloop, FlashAttention-2's K/V staging, and Triton's `num_stages > 2` pipelining all lower to it. + +It is also the semantic ancestor of Hopper's TMA. When backporting TMA-based kernels, `cp.async` is what TMA becomes — with the address generation and tile bookkeeping moving back into the kernel code. + +## Key Properties + +| Property | Detail | +|---|---| +| Direction | GMEM → SMEM only (no store direction, unlike TMA) | +| Granularity | 4 / 8 / 16 bytes **per thread** (`.ca`); 16 bytes only (`.cg`) | +| Addressing | Each thread computes its own src/dst addresses (vs TMA descriptor) | +| Cache behavior | `.ca` allocates L1+L2; `.cg` bypasses L1 (use for operand tiles) | +| OOB handling | Optional `src-size` operand zero-fills the tail (no branches needed) | +| Completion | Commit-group counting (`commit_group`/`wait_group N`) or mbarrier arrive | +| Issue cost | One instruction per 16 B per thread — a 128x64 bf16 tile = 128 issues/warp vs 1 TMA descriptor op | +| Register cost | No data registers consumed; address registers still needed | + +## Programming Surfaces (pick one level) + +**PTX (full control):** + +```cuda +__device__ __forceinline__ void cp_async_cg_16(void* smem_dst, const void* gmem_src) { + unsigned dst = (unsigned)__cvta_generic_to_shared(smem_dst); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" :: "r"(dst), "l"(gmem_src)); +} +__device__ __forceinline__ void cp_async_commit() { asm volatile("cp.async.commit_group;\n"); } +template +__device__ __forceinline__ void cp_async_wait_group() { asm volatile("cp.async.wait_group %0;\n" :: "n"(N)); } +``` + +**CUDA intrinsics (``)** — `__pipeline_memcpy_async(dst, src, 16)`, `__pipeline_commit()`, `__pipeline_wait_prior(N)` map 1:1 to the PTX above. + +**libcu++** — `cuda::memcpy_async(group, dst, src, size, pipe)` with `cuda::pipeline`; adds mbarrier-backed completion and cleaner multi-stage structure at slight codegen-trust cost. + +**CuTe atoms** — `SM80_CP_ASYNC_CACHEGLOBAL` / `SM80_CP_ASYNC_CACHEALWAYS` + `cp_async_fence()` / `cp_async_wait()`. + +## Canonical Multi-Stage Pipeline (compilable, sm_80/sm_86) + +The universal Ampere mainloop shape — `STAGES` SMEM buffers, prologue fills `STAGES-1`, steady state overlaps one load with one compute: + +```cuda +// nvcc -arch=sm_86 -c cp_async_pipeline.cu +#include +#include + +constexpr int STAGES = 3; +constexpr int TILE = 4096; // bytes per stage per block +constexpr int THREADS = 128; +constexpr int PER_THR = TILE / THREADS / 16; // 16B copies per thread per stage + +extern __shared__ uint8_t smem[]; // STAGES * TILE bytes + +// Placeholder compute — a real kernel runs its ldmatrix/mma.sync schedule here +__device__ void consume_tile(const uint8_t* stage_base, float* acc) { + *acc += (float)stage_base[threadIdx.x]; +} + +__global__ void pipelined_kernel(const uint8_t* __restrict__ g_in, int num_tiles, float* out) { + float acc = 0.f; + auto stage_ptr = [&](int s) { return smem + s * TILE; }; + auto issue_stage = [&](int s, int tile_idx) { + const uint8_t* src = g_in + (size_t)tile_idx * TILE + threadIdx.x * 16; + uint8_t* dst = stage_ptr(s) + threadIdx.x * 16; + #pragma unroll + for (int i = 0; i < PER_THR; i++) + __pipeline_memcpy_async(dst + i * THREADS * 16, src + i * THREADS * 16, 16); + __pipeline_commit(); + }; + + // Prologue: fill STAGES-1 buffers + for (int s = 0; s < STAGES - 1 && s < num_tiles; s++) + issue_stage(s, s); + + for (int t = 0; t < num_tiles; t++) { + // Wait until the oldest in-flight group (tile t) has landed: + __pipeline_wait_prior(STAGES - 2); + __syncthreads(); // make stage visible to all warps + consume_tile(stage_ptr(t % STAGES), &acc); + __syncthreads(); // stage fully consumed, safe to overwrite + int next = t + STAGES - 1; + if (next < num_tiles) + issue_stage(next % STAGES, next); // refill the buffer just freed + } + if (threadIdx.x == 0) out[blockIdx.x] = acc; +} +``` + +Notes on the pattern: + +- `__pipeline_wait_prior(STAGES - 2)` = "at most STAGES-2 groups still in flight" — the oldest stage is complete, the rest keep loading. +- Both `__syncthreads()` are load-bearing: cp.async completion is only observed by the *issuing* thread's wait; other warps need the barrier (or an mbarrier scheme) before reading the stage. +- 16-byte alignment of both addresses is mandatory for the 16 B variant; misalignment is an illegal-instruction trap, not a slowdown. + +## Zero-Fill OOB Idiom + +For boundary tiles, instead of predicating loads, issue the full 16 B copy with a clamped `src-size` — the hardware zero-fills the remainder: + +```cuda +__device__ __forceinline__ void cp_async_cg_zfill(void* smem_dst, const void* gmem_src, int valid_bytes) { + unsigned dst = (unsigned)__cvta_generic_to_shared(smem_dst); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16, %2;\n" + :: "r"(dst), "l"(gmem_src), "r"(valid_bytes)); +} +``` + +This replaces the `zfill` role of TMA's built-in OOB clamping in backports (TMA clamps per tensor descriptor; here you clamp per 16 B transaction). + +## Performance Rules + +1. **Use `.cg` (16 B) for operand tiles.** `.ca` pollutes L1 that the compute path wants for other data. CUTLASS uses `.cg` for A/B tiles. +2. **3–5 stages** is the practical range. More stages hide more latency but eat SMEM (99 KB/block ceiling on sm_86 — see hw-ampere-memory-model) and raise `__syncthreads()` pressure. +3. **Issue density matters.** Unlike TMA (one thread issues a whole tile), cp.async spends warp issue slots — interleave issues with compute rather than bursting all copies back-to-back when the mainloop is issue-bound. +4. **Swizzle SMEM destinations** (XOR patterns) so subsequent `ldmatrix` reads are bank-conflict-free; cp.async has no hardware swizzle-on-store (TMA does). +5. **Don't mix commit-group and mbarrier tracking** for the same transfers; pick one completion scheme per pipeline. + +## vs TMA (backport view) + +| Concern | TMA (sm_90+) | cp.async (sm_8x) | +|---|---|---| +| Who computes addresses | Hardware, from descriptor | Every thread, in registers | +| Issue cost | 1 op / tile | 1 op / 16 B / thread | +| OOB | Descriptor clamp | Per-copy `src-size` zfill | +| Swizzle | On the fly (32/64/128 B) | Do it yourself in the dst index math | +| Multicast to CTAs | Yes (cluster) | No — each CTA loads its own copy (L2 absorbs the overlap) | +| SMEM→GMEM direction | Yes (`cp.async.bulk` store) | No — plain `st.global` epilogue | +| Completion | mbarrier expect_tx (bytes) | Commit-group count or mbarrier arrivals | + +The deltas in this table are exactly the work items of a Hopper→Ampere port; see migration-hopper-to-ampere. diff --git a/wiki/hardware/mma-sync-ampere.md b/wiki/hardware/mma-sync-ampere.md new file mode 100644 index 000000000..5c457e478 --- /dev/null +++ b/wiki/hardware/mma-sync-ampere.md @@ -0,0 +1,129 @@ +--- +id: hw-mma-sync-ampere +title: "mma.sync + ldmatrix — Ampere Tensor Core Programming" +type: hardware +architectures: [sm80, sm86] +tags: [mma-sync, ldmatrix, register-budgeting, swizzling] +confidence: source-reported +reproducibility: snippet +related: [hw-cp-async, technique-register-budgeting, technique-swizzling, migration-hopper-to-ampere, hw-tcgen05-mma, hw-ampere-memory-model] +sources: [doc-ptx-isa-ampere, doc-ga102-whitepaper, doc-cutlass-ampere] +aliases: [mma.sync, HMMA, m16n8k16, "warp MMA", ldmatrix] +blackwell_relevance: "mma.sync is what wgmma (sm_90) and tcgen05.mma (sm_100) replaced. Backporting any Hopper/Blackwell tensor-core kernel to Ampere means re-expressing its MMAs in this warp-scope synchronous form with register-resident accumulators." +--- + +# mma.sync + ldmatrix — Ampere Tensor Core Programming + +## Overview + +On sm_80/sm_86 the tensor core is driven by **`mma.sync`** — a *synchronous, warp-scope* instruction: 32 threads collectively hold the operand fragments in registers, issue the op, and stall until the result is back in their registers. There is no asynchronous tensor-core path on Ampere. Everything Hopper/Blackwell later added — warpgroup-scope async (`wgmma`), SMEM-sourced operands, TMEM accumulators, single-thread issue — exists precisely to remove the costs this page describes. Backports must add those costs back in. + +Operand fragments are loaded from shared memory with **`ldmatrix`**, which materializes the exact per-lane register layout `mma.sync` expects. + +## The workhorse shapes + +| Inputs | Acc | Shape (M×N×K) | Per-warp per-instr FLOPs | +|---|---|---|---| +| f16/bf16 | f32 | 16×8×16 | 4096 | +| f16 | f16 | 16×8×16 | 4096 | +| tf32 | f32 | 16×8×8 | 2048 | +| s8 | s32 | 16×8×32 | 8192 (OPs) | + +A warp-level GEMM tile (e.g., 64×64) is built by tiling these primitives: 64×64 from m16n8k16 = 4×8 = 32 `mma.sync` issues per K-step of 16. + +## Register economics (the defining constraint) + +All three operand classes live in registers, fragmented across 32 lanes: + +- **Accumulator**: an M×N f32 warp tile costs `M*N/32` registers per thread. A 64×64 tile = **128 registers/thread** — half the 255 budget before A/B fragments, addresses, and indices. +- **A fragment** (m16n8k16, f16): 4× b32 regs/thread per 16×16 tile; **B**: 2× b32 per 16×8. +- Consequences: Ampere warp tiles top out around 64×64; register spills show up as `LDL`/`STL` in SASS and destroy throughput; `-maxrregcount`/`__launch_bounds__` tuning is a first-class knob (see technique-register-budgeting). + +This is the direct contrast with sm_90 (`wgmma` reads A/B from SMEM, accumulator still registers but per-warpgroup) and sm_100 (`tcgen05` moves accumulators to TMEM entirely). + +## GA10x quirk: FP32 accumulation runs at half rate + +On sm_86 GeForce/workstation silicon (RTX 3090, RTX A6000, A40), FP16-input MMA with **FP32 accumulate runs at half the rate** of FP16 accumulate (71 vs 142 dense TFLOPS on RTX 3090). BF16 always requires an f32 accumulator, so BF16 is always on the slow path. A100 (sm_80) does not have this segmentation — both run at full rate. + +| Consequence | Practical guidance | +|---|---| +| BF16 GEMM ceiling on 3090 = 71 TFLOPS | Don't chase A100-style BF16 efficiency numbers on GA10x | +| f16×f16→f16 acc is the only full-rate FP path | Usable for attention P·V and short-K accumulations with scaling discipline; risky for deep-K GEMM | +| INT8→s32 runs at 284 TOPS (4× the f32-acc path) | Weight-only INT4/INT8 (Marlin-style) and W8A8 kernels are disproportionately rewarded on GA10x | +| TF32 = 35.6 TFLOPS | Still ~2× FP32 CUDA cores; 3xTF32 emulation (CUTLASS ex. 27) buys near-FP32 accuracy | + +## ldmatrix (and the missing stmatrix) + +```asm +ldmatrix.sync.aligned.m8n8.x4.shared.b16 {d0,d1,d2,d3}, [addr]; // 4 tiles of 8x8xb16 +ldmatrix.sync.aligned.m8n8.x2.trans.shared.b16 {d0,d1}, [addr]; // transposed (col-major B) +``` + +- Each lane passes a row address; the instruction shuffles data into the canonical fragment layout. Pair with an XOR-swizzled SMEM layout to make all 32 row reads bank-conflict-free (see technique-swizzling). +- `.trans` handles the B-operand transpose at load time — no separate transpose pass. +- **`stmatrix` does not exist on Ampere** (sm_90+). Epilogues write accumulators out with plain `st.shared`/`st.global.v4`; getting coalesced stores from the mma fragment layout requires a register→SMEM→register shuffle roundtrip (CUTLASS `Epilogue` does exactly this) — budget SMEM and cycles for it in backports. + +## Minimal compilable warp-tile fragment (sm_80/sm_86) + +One 16×16 A tile × 16×8 B tile accumulating a 16×8 f32 result — the primitive every Ampere GEMM mainloop repeats: + +```cuda +// nvcc -arch=sm_86 -c mma_fragment.cu +#include +#include + +__device__ __forceinline__ void ldmatrix_x4(uint32_t (&r)[4], const void* smem) { + unsigned a = (unsigned)__cvta_generic_to_shared(smem); + asm volatile("ldmatrix.sync.aligned.m8n8.x4.shared.b16 {%0,%1,%2,%3}, [%4];\n" + : "=r"(r[0]), "=r"(r[1]), "=r"(r[2]), "=r"(r[3]) : "r"(a)); +} +__device__ __forceinline__ void ldmatrix_x2_trans(uint32_t (&r)[2], const void* smem) { + unsigned a = (unsigned)__cvta_generic_to_shared(smem); + asm volatile("ldmatrix.sync.aligned.m8n8.x2.trans.shared.b16 {%0,%1}, [%2];\n" + : "=r"(r[0]), "=r"(r[1]) : "r"(a)); +} +__device__ __forceinline__ void mma_m16n8k16_f32(float (&d)[4], const uint32_t (&a)[4], + const uint32_t (&b)[2], const float (&c)[4]) { + asm volatile( + "mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32 " + "{%0,%1,%2,%3}, {%4,%5,%6,%7}, {%8,%9}, {%10,%11,%12,%13};\n" + : "=f"(d[0]), "=f"(d[1]), "=f"(d[2]), "=f"(d[3]) + : "r"(a[0]), "r"(a[1]), "r"(a[2]), "r"(a[3]), "r"(b[0]), "r"(b[1]), + "f"(c[0]), "f"(c[1]), "f"(c[2]), "f"(c[3])); +} + +// Per-lane addresses for the canonical fragment layouts: +__global__ void warp_tile_mma(const half* __restrict__ /*unused*/, float* out) { + __shared__ __align__(16) uint8_t sA[16 * 16 * 2]; // 16x16 f16, row-major + __shared__ __align__(16) uint8_t sB[16 * 8 * 2]; // 16x8 f16, col-major view + int lane = threadIdx.x & 31; + + uint32_t a[4], b[2]; + float acc[4] = {0.f, 0.f, 0.f, 0.f}; + // A: lanes 0..15 address rows 0..15 of the left 16x8 half, lanes 16..31 the right half + ldmatrix_x4(a, sA + ((lane & 15) * 16 + (lane >> 4) * 8) * 2); + // B: lanes 0..15 address the 16 rows of the 16x8 tile (transposed load) + ldmatrix_x2_trans(b, sB + (lane & 15) * 8 * 2); + mma_m16n8k16_f32(acc, a, b, acc); + + // D fragment layout (PTX ISA): lane = 4*row_group + col_pair; acc[0..1] cover + // rows 0..7, acc[2..3] rows 8..15 of the 16x8 tile. Real epilogues remap via SMEM. + out[lane * 4 + 0] = acc[0]; out[lane * 4 + 1] = acc[1]; + out[lane * 4 + 2] = acc[2]; out[lane * 4 + 3] = acc[3]; +} +``` + +(Address math for `ldmatrix` source rows follows the PTX ISA fragment tables; real kernels wrap it in a swizzle function.) + +## vs wgmma / tcgen05 (what a backport gives up) + +| Property | tcgen05 (sm_100) | wgmma (sm_90) | mma.sync (sm_8x) | +|---|---|---|---| +| Scope | 1 thread issues | warpgroup (128 thr) | warp (32 thr) | +| Async | Yes (mbarrier) | Yes (commit/wait) | **No — warp stalls** | +| A/B source | SMEM/TMEM | SMEM (desc) | **Registers (via ldmatrix)** | +| Accumulator | TMEM | Registers | **Registers** | +| Max shape | 128×512×k | 64×256×k | 16×8×16 | +| Latency hiding | Hardware | Warpgroup switch | **Occupancy + ILP only** | + +Because the warp blocks on every `mma.sync`, Ampere kernels hide tensor-core latency with *other warps* (occupancy) and instruction-level interleaving of ldmatrix/cp.async/mma — not with intra-warp async. This is why warp specialization pays off far less on Ampere than on Hopper (see migration-hopper-to-ampere). diff --git a/wiki/migration/hopper-to-ampere-backport.md b/wiki/migration/hopper-to-ampere-backport.md new file mode 100644 index 000000000..4fa965a44 --- /dev/null +++ b/wiki/migration/hopper-to-ampere-backport.md @@ -0,0 +1,108 @@ +--- +id: migration-hopper-to-ampere +title: "Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86)" +type: migration +from_arch: sm90 +to_arch: sm86 +tags: [cp-async, mma-sync, tma, wgmma, pipeline-stages, warp-specialization] +related: [hw-cp-async, hw-mma-sync-ampere, hw-ampere-memory-model, hw-tma, hw-tcgen05-mma, hw-mbarrier, technique-warp-specialization, technique-pipeline-stages, technique-double-buffering, technique-register-budgeting, technique-tile-scheduling] +sources: [doc-ptx-isa-ampere, doc-ampere-tuning-guide, doc-ga102-whitepaper, doc-cutlass-ampere] +blackwell_relevance: "Reverse-direction guide: maps every Hopper/Blackwell-only feature referenced across this wiki to its sm_8x replacement, so Blackwell-first pages remain actionable on Ampere hardware (A100, RTX 3090-class rigs)." +confidence: source-reported +reproducibility: pseudocode +--- + +# Backporting Hopper/Blackwell Kernels to Ampere (sm_80/sm_86) + +## Overview + +This wiki is Blackwell-first, but a large fleet of real hardware is Ampere — A100s in clouds and GA10x (RTX 3090/4-GPU rigs, A6000, A40) on desks. This page is the systematic reverse map: given a kernel design expressed in Hopper/Blackwell idioms (TMA + mbarrier + warp specialization + wgmma/tcgen05), produce the equivalent sm_8x design without rediscovering Ampere lore. + +A backport is three separate problems, in order: + +1. **Instructions** — mechanical: every sm_90+/sm_100 instruction has a defined replacement or workaround (table below). ptxas failures enumerate this list for you. +2. **Capacity** — arithmetic: SMEM 99 KB/block (sm_86), 1536 threads/SM, small L2, register-resident accumulators. Tile/stage configs must be re-derived, not copied. +3. **Scheduling paradigm** — conceptual: async-everything producer/consumer designs collapse back to synchronous multi-stage pipelines where *all* warps compute. Blindly keeping warp specialization usually loses. + +## 1. Instruction replacement table + +| Hopper/Blackwell construct | sm_8x replacement | Notes | +|---|---|---| +| `cp.async.bulk.tensor` (TMA load) | `cp.async.cg` per-thread 16 B copies | Address math + tile iteration in-kernel; see hw-cp-async | +| TMA OOB clamping | `cp.async` `src-size` zero-fill per copy | Clamp per 16 B transaction, not per tensor | +| TMA swizzle-on-store (32/64/128 B) | XOR-swizzle in destination index math | Same swizzle functions, applied manually | +| TMA multicast (cluster) | None — each CTA loads its own tile | L2 absorbs most of the duplicate traffic; schedule tiles for L2 reuse | +| `cp.async.bulk` SMEM→GMEM store | Plain `st.global` epilogue | No async store path | +| `wgmma.mma_async` (sm_90) | `mma.sync.m16n8k16` + `ldmatrix` | Warp-scope, synchronous, operands via registers; see hw-mma-sync-ampere | +| `tcgen05.mma` + TMEM (sm_100) | Same as above; accumulators live in registers | Register budget becomes the binding constraint | +| `stmatrix` | `st.shared` + layout shuffle in epilogue | Costs SMEM roundtrip + cycles | +| `mbarrier.arrive.expect_tx` (byte counting) | Commit-group counting (`cp.async.wait_group N`) or mbarrier arrival counting | Count *arrivals*, not bytes; `cuda::barrier` works on sm_80 | +| `setmaxnreg` (register realloc) | None — static `__launch_bounds__` / `-maxrregcount` | All warps keep equal register budgets | +| Thread block clusters / DSMEM | None — single-CTA designs + L2 | Cross-CTA sharing only via L2/global atomics | +| CLC (cluster launch control) | Persistent kernels + global atomic work queue | technique-persistent-kernels works fine on Ampere | +| PDL / GDC (launch overlap) | Streams, CUDA Graphs, kernel fusion | No intra-launch overlap primitive | +| FP8 (E4M3/E5M2) tensor ops | INT8 (`mma.sync` s8→s32, 284 TOPS on 3090) or FP16 | No FP8 tensor cores on Ampere; Marlin-style weight-only INT4/INT8 is the idiomatic substitute | +| NVFP4 / block-scaled MMA | Dequant-to-FP16 in-kernel + FP16 MMA | Scales applied in registers before/after MMA | +| TMEM double-buffering | Register double-buffering is impossible at Blackwell tile sizes — shrink the warp tile | 64×64/warp is the practical ceiling | + +## 2. Capacity re-planning (sm_86 numbers) + +Re-derive, don't copy, using the hw-ampere-memory-model card: + +- **Pipeline depth:** SMEM/block ceiling is 99 KB. A bf16 128×64 A-tile + 64×128 B-tile stage = 32 KB → 3 stages fit (96 KB); a Hopper 5-stage 200 KB config does not. Formula: `stages ≤ 99KB / stage_bytes`, keep ≥2. +- **Occupancy:** target divisors of 1536 (3×512, 6×256, 12×128). Hopper 2×1024 configs strand 25%. +- **Registers:** M×N warp-tile accumulator costs `M*N/32` regs/thread (64×64 f32 = 128). Budget: accums + A/B fragments + addresses ≤ 255, ideally ≤ 168 for 3-block residency. Watch `LDL/STL` in SASS — any spill in the mainloop is disqualifying. +- **L2 (GA10x only, 6 MB):** output-tile rasterization order (technique-tile-scheduling) decides operand re-reads; grid-stride swizzled tile order that keeps a K-panel resident is worth more on 3090 than on A100/H100. +- **GA10x FP32-accumulate half-rate:** ceiling for bf16 GEMM on 3090 is 71 TFLOPS, not the 142 the FP16-acc spec suggests. Set expectations (and roofline math) accordingly before profiling. + +## 3. Scheduling paradigm: async producer/consumer → synchronous multistage + +Hopper/Blackwell designs split warps into producers (issue TMA) and consumers (issue wgmma/tcgen05, which are *asynchronous*), with mbarrier handoff and `setmaxnreg` shifting registers to consumers. On Ampere all three enablers are missing: + +1. `mma.sync` **stalls the issuing warp** — a "consumer" warp can't overlap its own MMAs with anything; +2. accumulators are **pinned to the warp's registers** — you can't hand a tile's accumulation to another warp without an SMEM roundtrip; +3. no register reallocation — idle producer warps waste a full register share. + +The idiomatic Ampere shape (CUTLASS `MmaMultistage`, FlashAttention-2) is therefore: **every warp both loads and computes**. Each iteration: issue `cp.async` for stage `t+STAGES-1`, `wait_group` for stage `t`, `__syncthreads()`, run the warp-tile `ldmatrix`+`mma.sync` schedule, repeat. Latency is hidden by (a) 2–4 SMEM stages in flight and (b) other resident warps, not by intra-CTA role split. + +Warp specialization on Ampere is justified only when the "producer" work is *not* plain operand streaming — e.g., decompression/dequant of weights, sparse index gather, or top-k selection feeding a dense compute stage. Even then, measure against the multistage baseline first. + +## 4. Worked example: attention mainloop, both dialects + +Hopper/Blackwell (as written across this wiki): + +```text +producer warps: consumer warpgroups: + tma.load K[i] -> smem, mbar.expect_tx wait mbar(K[i]) + tma.load V[i] -> smem, mbar.expect_tx S = wgmma(Q, K[i]) # async, regs + softmax_update(S) # overlap with next wgmma + O += wgmma(P, V[i]) +``` + +Ampere backport (FlashAttention-2 shape): + +```text +all warps, per iteration i: + cp.async K[i+2] -> smem_stage[(i+2)%3]; commit_group # prefetch 2 ahead + cp.async.wait_group(1); __syncthreads() # K[i] resident + S_frag = mma.sync(Q_frag, ldmatrix(K[i])) # warp stalls per-issue; + m,l update; P_frag = exp(S_frag - m) # hidden by other warps + O_frag += mma.sync(P_frag, ldmatrix(V[i])) # f16 P·V may use f16 acc (GA10x full rate) + __syncthreads() # stage consumed +``` + +Deltas to notice: prefetch distance is explicit (`STAGES-1`); softmax can't overlap MMA within a warp (rely on occupancy); Q stays in registers both ways; the f16-accumulate escape hatch for P·V is a GA10x-specific win worth its numerics review. + +## 5. Backport verification checklist (ncu) + +After a port, confirm the Ampere idiom actually engaged: + +- `sm__inst_executed_pipe_tensor` active — tensor pipe busy % is the primary KPI; on 3090 bf16, 60–75% of the 71 TFLOPS ceiling is a strong result. +- `smsp__sass_inst_executed_op_ldsm` present (ldmatrix in use), zero `LDL/STL` in the mainloop (no spills). +- `l1tex__data_bank_conflicts_pipe_lsu` ≈ 0 — swizzle survived the port. +- `sm__warps_active.avg.pct_of_peak` ≥ ~50% — occupancy is your latency-hiding now; a Hopper-style 1-block config that was fine with async engines will crater here. +- Long-scoreboard stall % dominant → raise stages/occupancy; `selected`/`math pipe throttle` dominant → you're compute-bound, stop tuning loads. + +## Reference backport in the wild + +FlashAttention-2 (sm_80-era) vs FlashAttention-3/4 (sm_90/sm_100) is the canonical public before/after of this exact migration, in reverse: same algorithm, the FA3→FA2 deltas are precisely this page's tables. When porting an attention-family kernel, diff your plan against FA2's choices first.