DeepSeek V4 #24162
Conversation
github-actions
Bot
added
model
|
@am17an I wonder what's the purpose of f32 casts and conts after mulmats here? Removed them and got the same logits. |
|
@fairydreaming it's an artifact of debugging, you can push your changes to this branch (I added you as collaborator) |
|
Played with flash attention this weekend, here's my experimental patch: With FA enabled and added lightning indexer GGML OP compute buffers memory usage got really low, I think processing 1M tokens is achievable on a single RTX PRO 6000 Max-Q with CPU expert offloading (f16 cache type) even with 8k ubatch size. Some performance numbers (Epyc 9374F + RTX PRO 6000 Max-Q): Max memory usage I saw in nvidia-smi was 60836MiB / 97887MiB. Edit: forgot about Pro benchmark results, aborted in the middle but it got to: |
|
@am17an Any specific reason you went with DEEPSEEK_V4_FLASH/deepseek-v4-flash/deepseek_v4_flash when naming things instead of simply DEEPSEEK4/deepseek4/deepseek4? I mean this convention is a bit inconsistent with existing names and the flash part is confusing (sounds like flash only while pro uses this architecture too), maybe it would be better to change it now before it spreads? (I noticed that even the architecture name in GGUF is deepseek-v4-flash, so we'd have to update it in existing GGUF files or reconvert). |
|
I'm going to work on making graph reuse work across various compression boundaries and also make multi-sequence work, along with fixing a couple of issues. After that I think a round of simple optimization + running some evals and then this should be ready for review. Since it's a large PR it may make sense to separate out conversion, chat and then the model into separate PRs. In parallel #24231 + FA can be included when they're ready |
|
@am17an Sounds good, I stared at tensor values for the last few days comparing them with the DeepSeek inference code but haven't found any obvious problems. |
|
For anyone interested I have this PR with various optimizations (#24231+CUDA, #24011, FA changes) in my repo: https://github.com/fairydreaming/llama.cpp/tree/dsv4 PP is the same as reported above, TG is ~70% faster. |
Thanks! I tried but failed. It looks like antirez's gguf is not yet supported? |
@rujialiu Unfortunately there are multiple naming differences for model parameters and tensors that prevent antirez GGUFs from working with this PR. |
|
@am17an On the other hand maybe it's a good idea to unify the naming with antirez GGUFs? From what I see in files there's only a single difference in tensor shapes - in attention output tensor - [4096, 1024, 8] vs [4096, 8192, 1]. I can try to fix this it in the meantime, what do you think? |
Thanks for the reply. I'm especially interested in trying this REAP version in antirez's format, which (hopefully) is small enough for lower-end machines with only 64GB RAM: |
|
@fairydreaming sure, I think it makes sense to support already existing GGUFs. BTW can you check the latest commit for any perf improvements on your setup? Graph reuse was added across CSA boundaries |
|
@am17an Merged the changes and I see an improvement, TG in Flash now exceeds 20 t/s for short prompts (was around 18): |
|
@rujialiu OK, this is weird. I made a patch that allows antirez DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf that I downloaded some time ago to work in this PR, but your DeepSeek-V4-Flash-Spark-Mini-Q2-REAP-ds4.gguf for some reason causes |
|
Not sure if it's too early for this but I'm noticing a consistently reproducible issue where the model outputs malformed JSX tags during long responses as follows: Happens with both the raw unquantized Q8 GGUF and the quantized Q3 GGUF that I normally use but isn't reproducible with responses over the web/API. Doesn't happen with short responses. Repo used:
Command used for HF -> GGUF: Command used for Quantization: Launch command:
Prompt used for this:
My Setup: 2x RTX 3080 20GB |
@fairydreaming Thanks! I tried that REAP version with cchuter's branch i.e. https://github.com/cchuter/llama.cpp/tree/feat/v4-port-cuda which works with that DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf on my machine (tg ~4 tok/s, pp even slower). I also got: I can't check whether this REAP gguf works with antirez's ds4 because ds4 doesn't support native Windows. I had good experience running Minimax 2.5 REAP with llama.cpp, but I don't have any way to ensure that gguf is sane (or at least works with official ds4). Sorry about that. |
|
@fairydreaming OK, I found that cchuter's branch works with that REAP gguf (actually I tried a slightly larger 180B REAP gguf instead) with |
|
@Lowkey-Loki-SN I think it is something to do with tokenization, it messes up even small JAX templates for me. Mostly extra whitespace. |
Glad to hear it's reproducible on your end too! And yes, it is always either extra whitespace or newlines when it happens on my end |
|
@rujialiu From what I see the problem is that expert indices read from tid2eid tensors during Edit: @am17an is right, I disabled expert offloading (so all CUDA now) and now it works on with ubatch 8 but fails with ubatch 9. |
|
@fairydreaming ubatch 32 is when the offload would kick in, so probably something in cuda backend |
GGUF, llama.cpp command, prompt and response? |
|
Thank you! It works for me. But unfourtunatly it is not usable for agentic tasks, because the context window size very huge. For example, for model Qwen3.5 397b Q6 (or Nex N2 Pro) I can put to my local machine context window = 262 000 with -ub 14000 -b 14000 that gives me pp=400 t/s, tg=11.5 t/s |
|
@am17an My branch with merged HC ops: without HC ops: So there are big gains in TG rate, smaller but still noticeable in PP. |
|
Yeah, we really need a standard way to introduce new ggml ops without falling back to the CPU. I think if #24646 is accepted then it could be a good way to add lightning index + these ops. |
|
I opened a new issue here #25171 but perhaps it's best to discuss it here. When I load Deepseek-V4-Flash, then generate a new assistant turn during a conversation (using Sillytavern in chat-completion mode, for instance), it works great. The model remembers the conversation and the system prompt. However, after the model successfully generated its first answer, if I swipe to generate a new answer, or simply write another user turn, the model will have forgotten everything: the chat logs and the system prompt. |
|
@EugeoSynthesisThirtyTwo perhaps best to post a screen share of llama-ui with the exact problem you face |
|
When using antirez's gguf, it seems that the jinja template is lost? Because it never produces reasoning blocks. I passed "--chat-template-file models\templates\deepseek-ai-DeepSeek-V4.jinja" at the end of the command line but it doesn't seem to work. |
|
I've uploaded 2.73 BPW and 3.15 BPW GGUFs, seems to retain a good amount of quality: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF |
The reasoning defaults to "off" in the template because "enable_thinking" is not defined (missing). Solution for llama-server in browser: Otherwise, start the server with "--reasoning on" and a couple of other params, e.g. thinking token budget. Hope this helps. |
If you view the model info in the webui (by clicking on the model name) it will show the chat template. |
|
@fairydreaming Probably false alarm as I couldn't create a publicly reproducible case. I compared dsv4 branch 5a09621 with 6f4f53f it's based on, however, the results are different with the same seed, but neither of them seem inferior. Are they supposed to be different though? 
|
|
@am17an I didn't know there was a llama-ui ^^' really cool ui though ! I filled the system prompt with this: First generation (working): Details
Details
Details
Details
And if I try to ask the model a specific info (the password), it doesn't remember it: Details
And finally, if I then restart llama-server and ask the model the password again, it remembers it, because it's the first generation: Details
|
|
Experienced something weird today. Ran and after hour or two of continuous inference (send 8 prompts, wait for inference to end, repeat) I got this: I see that on Reddit someone posted the same error (and got downvoted). |
@Nekotekina That's normal I guess, layer outputs will be slightly different due to different order of calculations (since there are now several fused CUDA GGML OPs in my branch that replace sequences of multiple primitive GGML OPs from the current master implementation), differences grow over time from layer to layer, at some point a different expert is selected, this accelerates the result divergence and finally even the first generated token may be different. |
|
Do you think it's because of that? When I click "regenerate", the prompt history is exactly the same. To be sure I added a log debug to see the incoming request in /v1/chat/completions. Both requests are exactly the same, so it shouldn't discard 192 tokens. |
@joesixpaq Thanks! The bulb button works. I didn't try to click it because it looks like microphone 😆 |
@dfriehs Thanks! I tried and confirmed that antirez's gguf already contains the chat template and I just need to enable thinking in llama-ui via the "bulb" button. |
|
DeepSeek V4 Flash has been going OOM for me and crashing with Server args: Maybe there is something wrong with how I am invoking the server, but just posting here since I did not see anyone else affected yet and Update, immediately after I discovered the |
|
Anyone got this working with SYCL or Vulkan? No problems with CUDA, but on 4xB70 the model won't load even though there's plenty of VRAM available. -v logs this: Next tried using the cards over RPC, similar error there: ggml_backend_sycl_buffer_type_alloc_buffer: can't allocate 1165734400 Bytes of memory on device |
Try --cpu-moe instead, and if it does not fills up VRAM, experiment with --n-cpu-moe N. |
|
Getting the same error on two machines, each with 192GB VRAM, one with P40s and the other running Mi50s. The error is: ..../llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (blk.0.attn_output_a.weight (reshaped)) in a buffer (ROCm0_Split) that cannot run the operation (RESHAPE) Both are running fdb1db8, latest as of this writing. |
|
Following the bug I found on KV cache corruption... I confirm that the model works fine when I reprocess the full prompt, for instance if I pass "cache_prompt: false" in the request, it works fine. I came to the conclusion that Deepseek V4 needs checkpoints for its cache, like Qwen 3.5 (correct me if I am wrong). I have been trying to make Deepseek V4 compatible with the argument --ctx-checkpoints N for hours now, but I don't know where to start. Can someone help please ? |
|
It already uses checkpoints |
I encountered the same issue: after the initial response, subsequent dialogue turns fail to reference previous context for answering. I also tried modifying the Jinja template to forcibly retain all thinking content and tested by directly passing multiple conversation turns via the API. On the first call to llama-server at startup, the model could answer based on the preset multi-turn conversation history. However, when sending the same content to llama-server a second time, the model’s response showed complete amnesia. Additionally, from the logs, I found that my multi-turn conversation content exceeds 10,000 tokens. Yet, during the second request to llama-server, the log indicates that the context checkpoint size is clearly too small: "I slot operator(): id 3 | task 4295 | restored context checkpoint (pos_min = 11517, pos_max = 11517, n_tokens = 11518, n_past = 11518, size = 17.021 MiB)" Backend: Vulkan |
|
This is not helpful unless you provide a proper reproduction. You can file an issue with the exact steps to reproduce the problem |
In this issue, I have provided complete reproduction steps and test data. |
|
I experienced the same "context exceeded" bug as @fairydreaming. Was just running it under pi and it broke after ~14% of the 262144 configured context was reached. |
20 participants
Overview
This PR adds support for the deepseek-v4 models. The most novel part of this architecture is the compressed attention. There are two types:
CSA (Compressed Sparse Attn) - it is a variant of DSA (introduced in DeepseekV3.2), it operates on the same principle of the lightning indexer to get top-k tokens to attend to, except tokens are "compressed tokens". A compressed token in CSA is every 4 tokens compressed into 1. It maintains a window of the last 8 tokens and does this at every 4 token boundary
HCA (Heavily Compressed Attn) - it is like normal attention over compressed tokens plus SWA, the compression being large at 128 tokens.
This PR handles this by creating compression plans (
comp_planin the code) which are created by the context and executed on the GPU. There are some extra writes to maintain graph topology for graph reuse.These two caches are
llama_kv_cacheobjects but they are always non-unified (i.e. stream aware). The slots are managed by the context.Every layer also has a SWA cache, we use a
llama_kv_cache_iswawrapper for this to expose only the SWA. So attention is[swa entries | compressed block entries]Perf on a DGX Spark:
./build/bin/llama-bench -m DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf -dio 1 -fa 1 -ub 2048 -p 2048 -n 32TODOs
Mainly performance improvements
-sm tensorCredits
Thanks to @pwilkin for the correct chat template + debugging help
Thanks to @fairydreaming for his help in debugging + contributing fixes
Additional information
Requirements