bw24 — from-scratch LLM inference for sm_120a (RTX 50-series Blackwell)

From-scratch LLM inference engine in Rust + CUDA, built for one machine: an RTX 5090 Laptop (Blackwell sm_120a, 24 GB). No frameworks, no ggml — every kernel written and tuned against measured hardware limits, with llama.cpp as the benchmark to beat on the same rig.

The headline capability is MTP speculative decoding: 1.1-1.9x over llama.cpp's best spec config on every supported Qwen model, with trimmed drafter heads published ready-to-use (huggingface.co/Avifenesh/bw24-bench) — behind a drop-in OpenAI-compatible server. Exactness is the contract: speculative output is gated token-identical to plain decode, so the speedup never changes what the model says.

Running bw24 on your own rig — desktop 50-series, older NVIDIA, anything? A hardware validation report is the fastest way to help: 50-series reports bless the rest of the family, older-card reports map the compatibility floor.

Current standing: six supported models, all fully gated. Qwen leads llama.cpp on every cell (plain 1.06-1.08x, spec 1.1-1.9x). Gemma leads decisively where llama lacks the capability or the depth (31B spec 1.7k 1.16x, E4B spec ≥1.23x, E4B plain 1.10x) and sits at 0.99-1.06x elsewhere under the strictest best-vs-best pairing (2026-07-15 re-audit). Every number below is a same-session, same-prompt, interleaved measurement against llama.cpp's best config; exactness is gated (argmax match + speculative self-consistency) on every kernel change, so speed never buys different outputs.

Model support

Tier	Models	State
Supported	Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE (NVFP4/IQ4_XS); Gemma-4 26B-A4B MoE, 31B dense, E4B (QAT Q4_0 + MTP drafters)	Board-published, fully gated, exactness-first; margins per model in the tables below
In progress	Hy3-REAP50, MiniMax-M3 REAP50 (safetensors, VRAM→RAM→NVMe spill)	Loads + generates; hybrid/sigmoid-router arcs still open

Quick start

Prebuilt Linux x86_64 binaries (sm_120a) ship with each release — or build from source:

cargo build --release
./target/release/kernel-check                     # every kernel vs CPU reference
BW24_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."
BW24_SPEC_K=3 ./target/release/run-spec /path/to/qwen36-27b.gguf   # MTP speculative
./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"        # auto-download from HF (needs `hf` CLI)
./target/release/frspec-owngen model.gguf trim.gguf --validate     # build + validate an FR-Spec draft trim from the model's OWN generations
./target/release/bw24-server                      # OpenAI-compatible /v1

Expected output — kernel-check ends with:

ALL GREEN: kernels match CPU reference.

and run-gen prints its correctness gate before any generation:

verify-prefill argmax=N  decode argmax=N  logit maxdiff=...  MATCH

Tuned paths are the defaults — no flags needed. Flags exist only for runtime parameters, machine config, and rollback seams (docs/FLAGS.md). run-gen prints a prefill/decode argmax gate before timing anything; a MISMATCH line voids the numbers after it.

Performance — Qwen (NVFP4 / IQ4_XS)

Measured 2026-07-13 on the target rig (RTX 5090 Laptop, N=2+ medians, both engines interleaved in the same thermal window on the same rig, same exact prompts, validity-gated cold starts, no flags (tuned paths are defaults). Full per-run logs: research/tune-data/ (Qwen) and research/gemma4-bringup/ (Gemma) — every win and every loss) against llama.cpp built on the same machine, same exact prompts, both engines re-baselined the same day. Boards move with the tuning campaign — research/tune-data/rig5090.jsonl is the running record; the README is refreshed with every board-moving merge.

Plain decode (no speculation, tg128 at 512-token context):

Model	bw24 plain	llama.cpp plain	Ratio
Qwen3.5-9B NVFP4 (GGUF)	135.7	126.7	1.07x
Qwen3.6-27B NVFP4 (GGUF)	48.4	44.9	1.08x
Qwen3.6-35B-A3B MoE (IQ4_XS)	178.2	167.8	1.06x

Depth is part of the contract: at 6.3k-token context every lead holds (1.02-1.09x).

Speculative decoding (MTP head, both engines at their measured best):

Model	bw24 spec	llama.cpp spec-best	Ratio
Qwen3.5-9B (K=3 + own-gen trimmed draft)	281.0 / 211.7 / 187.1	122.2 / 121.5 / 117.7	2.30x / 1.74x / 1.59x
Qwen3.6-27B (K=3 + own-gen trimmed draft)	116.4 / 101.2 / 86.0	91.7 / 93.3 / 81.5	1.27x / 1.08x / 1.06x
Qwen3.6-35B-A3B (K=2 + own-gen trimmed draft)	280.6 / 259.6 / 258.0	236.5 / 174.6 / 173.5	1.19x / 1.49x / 1.49x

The three columns are three prompt classes: short code / medium code (both greedy) / long agentic, sampled at temp 0.7 with distribution-exact rejection sampling. Every spec row uses one trimmed draft file built by the standard regime — the model's own-generation FR-Spec ranks, byte-verbatim MTP extraction, NVFP4 head + Q4_K_M block (docs/DRAFT-REGIME.md — the three laws and why each was paid for).

Drafts: use ours or build your own. Prebuilt per-model drafts (exact pipeline, exact published bytes) live at huggingface.co/Avifenesh/bw24-bench under drafts/<model>/ — recommended for the board models. For any other model, requant, or finetune, build one in two commands (a finetune's distribution moved, so its draft must too):

./target/release/frspec-owngen model.gguf ranks.gguf 32768        # ranks from the model's OWN generations
tools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf  # extract + trim + quantize

Exact prompts and configs also in the bench repo; llama.cpp flags in docs/COMPETITOR-SETUP.md.

Performance — Gemma-4 26B-A4B (QAT Q4_0)

Same protocol, own campaign log (research/gemma4-bringup/rig5090-gemma4.jsonl). Contexts are real prompt depths — the FP8 KV cache makes the lead widen as context grows:

Cell	bw24	llama.cpp	Ratio
plain, short ctx	199.7	187.6	1.06x
plain, 1.7k ctx	183.1	173.1	1.06x
plain, 4.9k ctx	162.6	142.0	1.14x (fa-off bar — re-pair pending)
MTP spec, short ctx (K=6)	267	271	0.99x
MTP spec, 1.7k ctx (K=6 + FR trim)	298	286	1.04x

Spec rows re-paired 2026-07-15 under the best-vs-best protocol (llama server MTP at its swept best -fa 1 --spec-draft-n-max 3 --spec-draft-n-min 1, exact-token-id prompts, serialized same-window arms — co-resident runs OOM-spill and read 10x low). Earlier published spec margins (1.37-1.54x) rode bars and acceptance states that no longer reproduce — including with era binaries — and are retired; the campaign log carries the full archaeology (rig5090-gemma4.jsonl 2026-07-15).

llama's spec side is --spec-type draft-mtp, warm, at its measured best on the same box. What buys the margin: an FP8 (e4m3) KV cache on both layer classes (half the bytes of llama's f16 KV at near-zero dequant cost), occupancy-tuned attention tiles, wide-load Q4_0 expert dots, and an FR-Spec drafter-head trim (150 MB → 18 MB at unchanged 0.91-0.94 acceptance). Correctness is structural, not statistical: decode, verify, and graph replay launch the same kernel symbols (the parity law), so the verify gate reads a bit-exact 0.000e0 logit maxdiff at every context depth. The same FP8-KV lever is available for Qwen behind BW24_KV_FP8 (correctness-proven; ~45% smaller KV for long-context serving).

Performance — Gemma-4 31B / E4B (QAT Q4_0)

Cell	bw24	llama.cpp	Ratio
31B dense plain, short	40.8	40.2	1.02x — parity is not the bar
31B dense plain, 1.7k	38.4	37.4	1.03x (DRAM-duty arc + streaming W loads, 2026-07-14)
31B MTP spec, short (K=3)	98.0	92.3	1.06x (re-paired 2026-07-15; the published 167.5/112.1 pair does not reproduce — see jsonl)
31B MTP spec, 1.7k (K=6 + FR trim)	97.3	83.9	1.16x (t-batched globals fa + bar re-pair + DRAM-duty arc, 2026-07-14)
E4B MTP spec (K=6 assistant)	248	n/a — llama's 2026-06-30 build cannot serve the E4B MTP drafter (arch + fattn crash, fixed upstream later); vs its plain 181.0	≥1.23x
E4B plain, short	199.9	181.0	1.10x (PDL + weight-prefetch + softcap-skip, 2026-07-13)

The 31B spec jump (0.79x → 1.09x, 2026-07-12) came from a serving-mode config, not a new kernel: the FP8 (e4m3) windowed KV cache — a win for plain decode at depth — turns out to GUT the MTP drafter's acceptance (its single sliding-window attention reads that cache, and e4m3 noise flips its argmaxes: acceptance 0.758 → 1.000 on short chat once the windowed layers went back to q8_0/q5_1). Spec serving now defaults the windowed cache to q8_0/q5_1 automatically (BW24_DRAFT set ⇒ fp8-windows off; plain serving keeps fp8). The same clean cache also flipped the FR-Spec drafter-head trim positive on the 31B (its earlier acceptance loss WAS the fp8 noise pushing drafter argmaxes off the 32k set): own-generation ranks now ride at identical acceptance for a pure head-read win (+2-4%). Same-window interleaved pairs, N=2 each side.

Reproducing (both Gemma sections): same protocol as Qwen — exact-token-id prompts and llama.cpp's swept-best flags in docs/COMPETITOR-SETUP.md; every row's raw run (and every retired number's archaeology) in research/gemma4-bringup/rig5090-gemma4.jsonl.

Known gaps

• Prefill trails llama.cpp (0.59-0.78x), root-caused: llama benches NVFP4 prefill at W4A4 (FP4 activations), a numeric class bw24's exactness gates reject — bw24's in-tree W4A4 arm beats llama but forks argmax on long prompts (docs/FLAGS.md §5). Output quality outranks the prefill column.
• Gemma plain margins are thin where both engines sit at the DRAM wall: 31B plain 1.02x short / 1.03x at 1.7k, 26B plain 1.07x short / 1.06x at 1.7k (best-vs-best -fa 1 interleaved pairs, 2026-07-15; identical GGUF bytes both engines, best single kernel = 91% of wall, e2e 87-89%). Every measured mechanism class — ours plus llama/vLLM/SGLang current releases — is shipped or carries a falsification row (research/gemma4-bringup/rig5090-gemma4.jsonl). Spec under the 2026-07-15 best-vs-best re-audit: 31B 1.7k 1.16x and E4B ≥1.23x lead; 31B short 1.06x, 26B 0.99x/1.04x are open — the earlier 1.37-1.54x margins rode stale bars/acceptance states that no longer reproduce (era-binary-verified; jsonl).
• Safetensors runs checkpoints llama.cpp cannot (NVIDIA NVFP4 ST, 121 GB spilled MoEs) but GGUF is the published format — ST showed seed-sensitive long-context repetition (research/tune-data/27b-st-vs-gguf-final.md).

What's inside

• NVFP4 / Q4_0 decode — split-plane repacked matvecs, warp-level dp4a, int8 W4A8 tensor-core prefill GEMM, per-shape auto-dispatch.
• MTP speculative decoding — embedded draft head, one batched K+1 verify, zero-sync async rounds, adaptive draft depth; K=1..8 self-consistency gate.
• MoE on 24 GB — expert-major CSR batching, decode-once dequant, SLRU expert residency with VRAM→host→disk spill.
• Quantized-KV attention — fused prefill/decode FlashAttention-class kernels (q8_0/q5_1 or FP8-e4m3 KV per layer class), split-K, device-length counters for graph replay.
• CUDA-graph decode — one graph replay per token, 4 bytes/token host traffic.
• Hybrid + sigmoid-router architectures — gated-delta-net mixes (Qwen3.6), MiniMax/DeepSeek-style routing.
• Safetensors loader — modelopt NVFP4 repacks byte-exact; FP8/BF16 re-encode at load; disk-tier expert streaming.

Correctness discipline

Every kernel change passes, in order: kernel-check (CPU reference), the run-gen argmax gate, run-spec K=1..8 self-consistency — one command: tools/local-ci.sh. FP summation order is part of the contract — "faster" kernels that reduce in a different order get rejected when they flip argmax at tight margins (research/tune-data/).

Exactness gates are structurally blind to numeric shifts where decode and verify move together — that class silently cost half a spec margin across ~40 green commits in July 2026. The local perf CI (tools/local-ci.sh --perf) closes it: every published cell re-measured per engine-touching push, speculative acceptance and tokens/round tracked per cell against a rolling baseline (research/tune-data/perf-ci.jsonl), enforced by the pre-push hook. Upstream engines are swept weekly for portable decode mechanisms (tools/upstream-sweep.sh → research/upstream-sweeps.md).

Workspace layout

Crate	What it does
bw24-engine	CUDA kernels (cu/), forward passes, speculative decoding, MoE cache, graph decode
bw24-gguf	GGUF parser + tensor loading (memory-mapped)
bw24-tokenizer	BPE tokenizer + chat templates from GGUF metadata
bw24-runtime	CUDA device/stream/memory primitives over cudarc
bw24-server	OpenAI-compatible HTTP server (axum)
bw24-probe	Standalone hardware microbenches

Requirements

• NVIDIA Blackwell consumer GPU (sm_120a); primary target RTX 5090 Laptop.
• CUDA 13.1 (BW24_NVCC overrides the nvcc path), Rust edition 2024, cudarc 0.19.
• A model: GGUF or HF safetensors directory (pass either path).

Limitations

• Built for sm_120a only; tuning assumes this exact memory/compute ratio. On any other GPU, use llama.cpp (broadest hardware coverage) or mistral.rs (multi-platform Rust) instead — an arch/sm89-l40s branch exists for Ada but is untuned.
• Single GPU, single stream; no tensor parallelism or continuous batching.
• Moving research codebase; APIs and flags change without notice.

Docs

• ARCHITECTURE.md — tech stack + sm_120a feasibility ledger.
• HANDOVER.md — living state-of-work (standings, laws, open lanes).
• docs/decisions/ — design decision records.
• docs/COMPETITOR-SETUP.md — competitor engines at their peak on this box.
• research/tune-data/ + research/gemma4-bringup/ — every experiment as JSONL, wins and losses both.
• research/benchmarks.md — the A/B measurement protocol.

Contributing

Issues and PRs welcome — see CONTRIBUTING.md.

License

MIT — see LICENSE.