Lid cuda kernel

[spencer-zaid]

Overview

1. deepseek4 : Wire ggml_lightning_indexer into build_lid_top_k
2. ggml-cuda : add GGML_OP_LIGHTNING_INDEXER kernel (current implementation is CPU-only)

Additional information

Testing: token-identical vs unfused path at short context, needle-in-haystack retrieval correct at 10/50/90% depth (256K) plus spot-checked at 512K/1M's 50% depth, TDR/driver-timeout stress-tested at true near-max depth (253,952 tokens) with no crash, measured prefill/decode across 256K/512K/1M
Performance/testing coming in comment from tester with a mixed 5 gpu setup

Performance (rtx 5090 + 9950x3d + 96gb ddr5 6200 cl30)

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768, ~9gb at ubatch 2048 expected)

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Claude Sonnet for research/education, documentation, and assistance with debugging including some code it rewrote itself

[fairydreaming]

Thanks, but llama.cpp maintainers made it very clear that I should add CPU implementation first and implementation for other backends (CUDA etc) in separate following PRs, so I'm not going to include a CUDA implementation (mine, yours or any) in my current CPU lightning indexer PR branch. Sorry, but you can try later after it's merged. Wish your luck!

[EugeoSynthesisThirtyTwo]

This is such an amazing work

The master fork of llama.cpp

1. is much slower for prompt processing
2. can't handle more than 64k context on a 5090
3. has amnesia (Eval bug: Deepseek V4 Flash forgets everything ggml-org/llama.cpp#25171 & Eval bug: DeepSeek V4 Flash forgot context when hit cache or use checkpoint in long context ggml-org/llama.cpp#25259)

Your work fixes all of them

[spencer-zaid]

Thanks, but llama.cpp maintainers made it very clear that I should add CPU implementation first and implementation for other backends (CUDA etc) in separate following PRs, so I'm not going to include a CUDA implementation (mine, yours or any) in my current CPU lightning indexer PR branch. Sorry, but you can try later after it's merged. Wish your luck!

Got it - wishing you luck on them accepting your cpu indexer pr soon then!

[jonearth]

Report: works multi-GPU (2×3090, 1M ctx) — but llama-server defaults crash-loop on ANY hardware (-np 1 fixes it)

Built deepseek-lid-cuda @ 6164d85 on Linux (Ubuntu 24.04, CUDA 13.3 toolkit /
13.0 driver, arch 86). 2× RTX 3090 layer-split + CPU expert offload, 64GB DDR5,
antirez IQ2XXS chat-v2 quant, F16 KV.

Bench (all clean exits, llama-batched-bench -npl 1):

Context	ubatch	Prefill 2K/16K/131K (t/s)	Decode 2K/131K (t/s)	GPU0/GPU1 peak
256K	2048	343 / 277 / 114	13.8 / 10.9	23.7 / 21.2 GiB
512K	1024	265 / 224 / 103	13.7 / 10.9	22.3 / 21.0 GiB
1M	512	182 / 162 / 87	13.8 / 10.9	23.9 / 23.0 GiB

Flat decode 256K→1M reproduces. No kernel issues across the layer-split boundary.

The finding: llama-server with default --parallel -1 (auto slots) crash-loops
at startup — graph reserve goes down the unfused indexer path:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 398314.19 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 417662695680
graph_reserve: failed to allocate compute buffers

(256K ctx, ub2048.) Not multi-GPU-related — reproduced with CUDA_VISIBLE_DEVICES=0
(single card, 64K ctx): same fallback, ~231 GiB reserve attempt. So default
llama-server fails on any hardware; every validation so far (mine included) used
llama-cli/llama-bench, which are single-sequence and never hit it.

Workaround: -np 1 → server reserves the fused path (~3.2 GiB) and works —
running behind Open WebUI at 13.8 t/s now. Looks like the fused wiring in
deepseek4.cpp doesn't cover multi-sequence ubatches, so multi-slot worst-case
reserve falls back to the unfused chain. Happy to test a fix or provide more logs.

Two multi-GPU notes for the docs: the -ot preset needs GPU expert blocks straddling
both cards' contiguous layer ranges (lowest-N-blocks piles everything onto device 0
→ OOM at load), and each card allocates the full compute buffer from your formula,
not half.

[fairydreaming]

@jonearth Indeed I see weird things happening with multiple parallel sequences (while -np 1 runs fine). Thanks for bringing this to my attention.