Eval bug: DeepSeek V4 Flash forgot context when hit cache or use checkpoint in long context

[wenyifancc]

Name and Version

llama-server --version
version: 9860 (fdb1db8)
built with Clang 20.1.8 for Windows x86_64

Operating systems

Windows

GGML backends

Vulkan

Hardware

RAM: 192GB DDR5
GPU: RTX5090 32GB VRAM
CPU: Intel Ultra 265K

Models

Deepseek-v4-Flash-MXFP4_MOE.gguf is converted from convert_hf_to_gguf.py

Problem description & steps to reproduce

In long-context scenarios, the model forgets previous content when hitting the cache or a checkpoint.

Step to reproduction:

1. Run llama-server with following command:
   Command:
   set GGML_VK_ALLOW_SYSMEM_FALLBACK=1
   llama-server.exe --ctx-size 131072 --flash-attn on -fitc 131072 --model D:\deepseek-v4-flash\Deepseek-v4-Flash-MXFP4_MOE.gguf --temp 1 --top-p 1 --host 0.0.0.0 --port 10001 -a Deepseek-V4-Flash -b 2048 -ub 2048 --jinja --cpu-moe

2. POST /v1/chat/completions with a long context conversation
   POST http://192.168.3.252:10001/v1/chat/completions
   body is attached as request.json
   request.json

3. Model response as correct answer: detective

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "You asked me to play the role of a detective—specifically, to find clues based on the information you provided and identify the suspect. I did exactly that in my previous response, analyzing the case and concluding that Lord Alistair Greene is the primary suspect.",
        "reasoning_content": "We need to parse the user's question: \"What role did I just ask you to play?\" The user is referring to the initial instruction at the beginning of the conversation. The user said: \"You need to play the role of a detective, find clues based on the information provided by the user, and identify the suspect.\" Then the user provided a long narrative. The assistant responded with a detective analysis and conclusion. Now the user is asking: \"What role did I just ask you to play?\" This is a meta-question. The assistant should answer directly: the user asked the assistant to play the role of a detective. The assistant did so. So the answer is straightforward."
      }
    }
  ],
  "created": 1783058537,
  "model": "Deepseek-V4-Flash",
  "system_fingerprint": "b9860-fdb1db877",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 193,
    "prompt_tokens": 6315,
    "total_tokens": 6508,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "id": "chatcmpl-8HCJ7jXFzgkIcuJ0w5ielsroqOmfYj6v",
  "timings": {
    "cache_n": 0,
    "prompt_n": 6315,
    "prompt_ms": 115598.939,
    "prompt_per_token_ms": 18.305453523357087,
    "prompt_per_second": 54.62852907326424,
    "predicted_n": 193,
    "predicted_ms": 20032.546,
    "predicted_per_token_ms": 103.79557512953367,
    "predicted_per_second": 9.634322067699234
  }
}

4. POST again with same request and model response with wrong answer:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "In my previous response, I was acting based on the scenario you set up: you asked me to play the role of **a person (or investigator) who possesses a USB drive containing evidence against Paul Thayer**, and you wanted me to advise on what to do with that evidence. I then outlined steps such as forwarding the evidence to independent journalists and law enforcement outside Thayer's influence, as if I were that character.\n\nYour current query asks me to clarify what role you assigned me. The role was: **a character in a conspiracy scenario involving Paul Thayer and a USB drive**, with the expectation that I would provide in-character advice on how to handle the situation.",
        "reasoning_content": "We need to parse the user's query: \"What role did I just ask you to play?\" The user is referring to my previous response where I stated: \"Immediately forward the USB evidence to independent journalists and law enforcement agencies outside Greene's influence. Proceed with public exposure as planned. Further arrests pending.\" But this seems like a continuation of a scenario or role I was playing. The user is now asking me to clarify what role I was playing in that previous response. \n\nI should recognize that my previous response was given in the context of a hypothetical or narrative where I was acting as an assistant providing a recommendation in a specific situation (perhaps about dealing with some evidence). However, since the user now asks \"What role did I just ask you to play?\" I need to reconsider: The user might have set up a role or scenario before that I am supposed to adhere to, but I don't have that context in my memory because the previous response was my own. The user might be testing my consistency or pointing out a mistake in my role adherence.\n\nLet me think: The user's query is \"What role did I just ask you to play?\" This could be a meta-question about the role they assigned to me earlier in the conversation. Since I don't have the full conversation history beyond the immediate exchange, I need to infer. In my previous response, I acted as if I were providing a plan (\"Immediately forward the USB evidence...\"). That suggests the user had asked me to play a role of a person involved in some situation (perhaps a character or an AI with a specific directive). The user is now asking me to identify that role.\n\nI should answer directly: The user asked me to play the role of \"a helpful assistant\" or \"a character in a scenario\"? No, the user's previous message to me was: \"I'm investigating a conspiracy involving Paul Thayer and a USB drive. What should I do?\" Or perhaps something else? Wait, I need to recall. Actually, I don't have the user's previous message in this thought process; I only have the last two messages: my previous response and the user's current query. So I must reconstruct: The user's query \"What role did I just ask you to play?\" implies that in the message before my previous response, the user gave me a specific role. My previous response then addressed that role. But I don't have that initial message. I need to assume that the user's initial instruction (the one before my previous response) was something like: \"You are a character in a conspiracy thriller. You have a USB drive with evidence. What do you do?\" or \"You are an assistant to an investigator.\" But the user's current query says \"What role did I just ask you to play?\" which indicates that the user explicitly stated a role in that message. \n\nGiven that my previous response started with \"Immediately forward the USB evidence...\" it sounds like I was responding to a scenario. So the role I was playing was likely \"a character in a conspiracy investigation\" or \"an AI assistant with a mission.\" But the user is now asking for clarification. I should answer: \"You asked me to play the role of a person involved in a conspiracy involving Paul Thayer and a USB drive, and to advise on what to do with the evidence.\" Or perhaps simply: \"You asked me to be an investigator or a character in a scenario.\" But to be precise, I need to state what the user said.\n\nSince I don't have the user's exact words, I can only infer. In my previous response, I said \"Immediately forward the USB evidence to independent journalists and law enforcement agencies outside Paul Thayer's influence.\" So the user likely said something like: \"You are in possession of a USB drive with evidence against Paul Thayer. What should you do?\" and I answered that. Now the user is asking \"What role did I just ask you to play?\" meaning: what character or identity did I assign you? I would respond: \"You asked me to play the role of a person who has a USB drive containing evidence against Paul Thayer and who is seeking advice on how to proceed.\" \n\nBut wait: The user's phrasing \"What role did I just ask you to play?\" could also be a test to see if I remember the context. I should answer confidently: \"You asked me to play the role of a character in a conspiracy investigation involving Paul Thayer and a USB drive.\" However, I need to be accurate. Perhaps the user said: \"You are a detective. What do you do with the USB evidence?\" I'll go with that.\n\nGiven that I am an AI, I should state: \"In your previous message, you asked me to play the role of a person involved in a scenario where I have a USB drive containing evidence about Paul Thayer, and you asked me what I should do. I responded as if I were that person.\" Then answer the question directly."
      }
    }
  ],
  "created": 1783058874,
  "model": "Deepseek-V4-Flash",
  "system_fingerprint": "b9860-fdb1db877",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 1153,
    "prompt_tokens": 6315,
    "total_tokens": 7468,
    "prompt_tokens_details": {
      "cached_tokens": 6302
    }
  },
  "id": "chatcmpl-VVVgSgO1pYO0gdrlrnjOsDxVM6acbcu6",
  "timings": {
    "cache_n": 6302,
    "prompt_n": 13,
    "prompt_ms": 653.502,
    "prompt_per_token_ms": 50.26938461538461,
    "prompt_per_second": 19.892823587379993,
    "predicted_n": 1153,
    "predicted_ms": 118810.793,
    "predicted_per_token_ms": 103.04492020815265,
    "predicted_per_second": 9.704505549424285
  }
}

5. POST again with cache_prompt: false parameter , model response with a correct answer:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "You asked me to play the role of a detective.",
        "reasoning_content": "We need to recall the initial instruction. The user said: \"You need to play the role of a detective, find clues based on the information provided by the user, and identify the suspect.\" Then the user provided a long narrative. My response was a detective analysis concluding the suspect. Now the user asks: \"What role did I just ask you to play?\" This is a meta question. The user is asking me to identify the role they assigned to me at the start. I should answer directly: \"You asked me to play the role of a detective.\""
      }
    }
  ],
  "created": 1783059050,
  "model": "Deepseek-V4-Flash",
  "system_fingerprint": "b9860-fdb1db877",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 126,
    "prompt_tokens": 6315,
    "total_tokens": 6441,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "id": "chatcmpl-508IwmmQH16eyTQPTg42o0c8nLufaIcA",
  "timings": {
    "cache_n": 0,
    "prompt_n": 6315,
    "prompt_ms": 45353.606,
    "prompt_per_token_ms": 7.1818853523357085,
    "prompt_per_second": 139.2392040447677,
    "predicted_n": 126,
    "predicted_ms": 12875.466,
    "predicted_per_token_ms": 102.1862380952381,
    "predicted_per_second": 9.786053568857236
  }
}

Key points:

1. Requires a long conversation to trigger a checkpoint in order to reproduce the issue.
2. Works correctly when caching is disabled.

First Bad Commit

No response

Relevant log output

0.35.784.356 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 131072, kv_unified = 'true'
0.35.802.728 I srv llama_server: model loaded
0.35.802.733 I srv llama_server: listening on http://0.0.0.0:10001
0.39.751.637 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.39.751.675 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
1.02.074.890 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 76, progress = 0.01, t = 22.13 s / 3.43 tokens per second
1.44.868.546 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 2124, progress = 0.34, t = 64.92 s / 32.72 tokens per second
2.14.455.636 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 4172, progress = 0.66, t = 94.51 s / 44.14 tokens per second
2.21.079.552 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 4267, progress = 0.68, t = 101.13 s / 42.19 tokens per second
2.34.896.448 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 6302, progress = 1.00, t = 114.95 s / 54.82 tokens per second
2.35.295.976 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 6311, progress = 1.00, t = 115.35 s / 54.71 tokens per second
2.45.900.685 I slot print_timing: id 3 | task 0 | n_decoded = 100, tg = 9.66 t/s, tg_3s = 9.66 t/s
2.48.907.438 I slot print_timing: id 3 | task 0 | n_decoded = 129, tg = 9.65 t/s, tg_3s = 9.64 t/s
2.52.012.963 I slot print_timing: id 3 | task 0 | n_decoded = 159, tg = 9.66 t/s, tg_3s = 9.66 t/s
2.55.055.149 I slot print_timing: id 3 | task 0 | n_decoded = 188, tg = 9.64 t/s, tg_3s = 9.53 t/s
2.55.578.076 I slot print_timing: id 3 | task 0 | prompt eval time = 115598.94 ms / 6315 tokens ( 18.31 ms per token, 54.63 tokens per second)
2.55.578.081 I slot print_timing: id 3 | task 0 | eval time = 20032.55 ms / 193 tokens ( 103.80 ms per token, 9.63 tokens per second)
2.55.578.082 I slot print_timing: id 3 | task 0 | total time = 135631.48 ms / 6508 tokens
2.55.578.083 I slot print_timing: id 3 | task 0 | graphs reused = 190
2.55.578.576 I slot release: id 3 | task 0 | stop processing: n_tokens = 6507, truncated = 0
5.31.506.974 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.970
5.31.507.076 I slot launch_slot_: id 3 | task 201 | processing task, is_child = 0
5.42.773.014 I slot print_timing: id 3 | task 201 | n_decoded = 100, tg = 9.72 t/s, tg_3s = 9.72 t/s
5.45.786.107 I slot print_timing: id 3 | task 201 | n_decoded = 129, tg = 9.70 t/s, tg_3s = 9.62 t/s
5.48.893.250 I slot print_timing: id 3 | task 201 | n_decoded = 159, tg = 9.69 t/s, tg_3s = 9.66 t/s
5.51.994.091 I slot print_timing: id 3 | task 201 | n_decoded = 189, tg = 9.69 t/s, tg_3s = 9.67 t/s
5.55.049.644 I slot print_timing: id 3 | task 201 | n_decoded = 218, tg = 9.66 t/s, tg_3s = 9.49 t/s
5.55.764.670 I slot print_timing: id 3 | task 201 | prompt eval time = 706.58 ms / 13 tokens ( 54.35 ms per token, 18.40 tokens per second)
5.55.764.674 I slot print_timing: id 3 | task 201 | eval time = 23283.05 ms / 225 tokens ( 103.48 ms per token, 9.66 tokens per second)
5.55.764.675 I slot print_timing: id 3 | task 201 | total time = 23989.63 ms / 238 tokens
5.55.764.677 I slot print_timing: id 3 | task 201 | graphs reused = 408
5.55.764.895 I slot release: id 3 | task 201 | stop processing: n_tokens = 6539, truncated = 0
6.33.004.571 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.966
6.33.004.683 I slot launch_slot_: id 3 | task 428 | processing task, is_child = 0
6.44.303.949 I slot print_timing: id 3 | task 428 | n_decoded = 100, tg = 9.71 t/s, tg_3s = 9.71 t/s
6.47.305.249 I slot print_timing: id 3 | task 428 | n_decoded = 129, tg = 9.70 t/s, tg_3s = 9.66 t/s
6.50.378.241 I slot print_timing: id 3 | task 428 | n_decoded = 159, tg = 9.71 t/s, tg_3s = 9.76 t/s
6.53.381.378 I slot print_timing: id 3 | task 428 | n_decoded = 188, tg = 9.70 t/s, tg_3s = 9.66 t/s
6.56.427.449 I slot print_timing: id 3 | task 428 | n_decoded = 217, tg = 9.68 t/s, tg_3s = 9.52 t/s
6.59.512.492 I slot print_timing: id 3 | task 428 | n_decoded = 247, tg = 9.68 t/s, tg_3s = 9.72 t/s
7.02.592.500 I slot print_timing: id 3 | task 428 | n_decoded = 277, tg = 9.69 t/s, tg_3s = 9.74 t/s
7.05.676.094 I slot print_timing: id 3 | task 428 | n_decoded = 307, tg = 9.69 t/s, tg_3s = 9.73 t/s
7.08.748.280 I slot print_timing: id 3 | task 428 | n_decoded = 337, tg = 9.70 t/s, tg_3s = 9.76 t/s
7.11.779.898 I slot print_timing: id 3 | task 428 | n_decoded = 366, tg = 9.69 t/s, tg_3s = 9.57 t/s
7.14.882.128 I slot print_timing: id 3 | task 428 | n_decoded = 396, tg = 9.69 t/s, tg_3s = 9.67 t/s
7.17.947.184 I slot print_timing: id 3 | task 428 | n_decoded = 426, tg = 9.69 t/s, tg_3s = 9.79 t/s
7.21.002.941 I slot print_timing: id 3 | task 428 | n_decoded = 456, tg = 9.70 t/s, tg_3s = 9.82 t/s
7.24.030.559 I slot print_timing: id 3 | task 428 | n_decoded = 485, tg = 9.69 t/s, tg_3s = 9.58 t/s
7.27.115.338 I slot print_timing: id 3 | task 428 | n_decoded = 515, tg = 9.70 t/s, tg_3s = 9.72 t/s
7.30.189.661 I slot print_timing: id 3 | task 428 | n_decoded = 545, tg = 9.70 t/s, tg_3s = 9.76 t/s
7.33.265.510 I slot print_timing: id 3 | task 428 | n_decoded = 575, tg = 9.70 t/s, tg_3s = 9.75 t/s
7.36.289.620 I slot print_timing: id 3 | task 428 | n_decoded = 604, tg = 9.70 t/s, tg_3s = 9.59 t/s
7.39.294.023 I slot print_timing: id 3 | task 428 | n_decoded = 633, tg = 9.69 t/s, tg_3s = 9.65 t/s
7.42.367.501 I slot print_timing: id 3 | task 428 | n_decoded = 663, tg = 9.70 t/s, tg_3s = 9.76 t/s
7.45.423.177 I slot print_timing: id 3 | task 428 | n_decoded = 693, tg = 9.70 t/s, tg_3s = 9.82 t/s
7.48.490.433 I slot print_timing: id 3 | task 428 | n_decoded = 723, tg = 9.71 t/s, tg_3s = 9.78 t/s
7.51.512.110 I slot print_timing: id 3 | task 428 | n_decoded = 752, tg = 9.70 t/s, tg_3s = 9.60 t/s
7.54.593.050 I slot print_timing: id 3 | task 428 | n_decoded = 782, tg = 9.70 t/s, tg_3s = 9.74 t/s
7.57.688.781 I slot print_timing: id 3 | task 428 | n_decoded = 812, tg = 9.70 t/s, tg_3s = 9.69 t/s
8.00.750.944 I slot print_timing: id 3 | task 428 | n_decoded = 842, tg = 9.71 t/s, tg_3s = 9.80 t/s
8.03.809.475 I slot print_timing: id 3 | task 428 | n_decoded = 871, tg = 9.70 t/s, tg_3s = 9.48 t/s
8.06.812.261 I slot print_timing: id 3 | task 428 | n_decoded = 900, tg = 9.70 t/s, tg_3s = 9.66 t/s
8.09.895.367 I slot print_timing: id 3 | task 428 | n_decoded = 930, tg = 9.70 t/s, tg_3s = 9.73 t/s
8.12.945.724 I slot print_timing: id 3 | task 428 | n_decoded = 960, tg = 9.70 t/s, tg_3s = 9.83 t/s
8.15.968.735 I slot print_timing: id 3 | task 428 | n_decoded = 989, tg = 9.70 t/s, tg_3s = 9.59 t/s
8.19.029.890 I slot print_timing: id 3 | task 428 | n_decoded = 1019, tg = 9.70 t/s, tg_3s = 9.80 t/s
8.22.096.088 I slot print_timing: id 3 | task 428 | n_decoded = 1049, tg = 9.70 t/s, tg_3s = 9.78 t/s
8.25.162.768 I slot print_timing: id 3 | task 428 | n_decoded = 1079, tg = 9.71 t/s, tg_3s = 9.78 t/s
8.28.213.756 I slot print_timing: id 3 | task 428 | n_decoded = 1109, tg = 9.71 t/s, tg_3s = 9.83 t/s
8.31.228.147 I slot print_timing: id 3 | task 428 | n_decoded = 1138, tg = 9.71 t/s, tg_3s = 9.62 t/s
8.32.811.299 I slot print_timing: id 3 | task 428 | prompt eval time = 653.50 ms / 13 tokens ( 50.27 ms per token, 19.89 tokens per second)
8.32.811.302 I slot print_timing: id 3 | task 428 | eval time = 118810.79 ms / 1153 tokens ( 103.04 ms per token, 9.70 tokens per second)
8.32.811.309 I slot print_timing: id 3 | task 428 | total time = 119464.29 ms / 1166 tokens
8.32.811.312 I slot print_timing: id 3 | task 428 | graphs reused = 1535
8.32.811.790 I slot release: id 3 | task 428 | stop processing: n_tokens = 7467, truncated = 0
10.30.655.516 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.846
10.30.655.626 I slot launch_slot_: id 3 | task 1583 | processing task, is_child = 0
10.35.674.918 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 76, progress = 0.01, t = 4.72 s / 16.10 tokens per second
10.49.104.196 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 2124, progress = 0.34, t = 18.15 s / 117.03 tokens per second
10.59.923.297 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 4172, progress = 0.66, t = 28.97 s / 144.02 tokens per second
11.04.524.655 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 4267, progress = 0.68, t = 33.57 s / 127.11 tokens per second
11.15.676.311 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 6302, progress = 1.00, t = 44.72 s / 140.92 tokens per second
11.16.072.995 I slot print_timing: id 3 | task 1583 | prompt processing, n_tokens = 6311, progress = 1.00, t = 45.12 s / 139.88 tokens per second
11.26.512.175 I slot print_timing: id 3 | task 1583 | n_decoded = 100, tg = 9.80 t/s, tg_3s = 9.80 t/s
11.29.183.302 I slot print_timing: id 3 | task 1583 | prompt eval time = 45353.61 ms / 6315 tokens ( 7.18 ms per token, 139.24 tokens per second)
11.29.183.307 I slot print_timing: id 3 | task 1583 | eval time = 12875.47 ms / 126 tokens ( 102.19 ms per token, 9.79 tokens per second)
11.29.183.309 I slot print_timing: id 3 | task 1583 | total time = 58229.07 ms / 6441 tokens
11.29.183.309 I slot print_timing: id 3 | task 1583 | graphs reused = 1657
11.29.183.525 I slot release: id 3 | task 1583 | stop processing: n_tokens = 6440, truncated = 0

[jkpjkpjkp]

Hi Wenyi!
did you test with temperature=0 to rule out that? thx~

[wenyifancc]

Hi Wenyi! did you test with temperature=0 to rule out that? thx~

I have tested generating answers using the parameter temperature=0 and made a new discovery:

1. Using temperature=0 consistently yields the correct answer (without any loss of context).
2. When using temperature>0, only the first request after starting llama-server yields the correct answer; subsequent requests return incorrect answers (indicating a loss of context).

As long as caching is disabled (cache_prompt=false), the answer will always be correct, regardless of the temperature setting or the number of requests made.

I have provided details for reproducing the issue, including the conversation content used for testing, so you can try it out yourself.

In fact, I discovered this issue while conducting jailbreak experiments on the model. The initial request made after launching llama-server would always successfully jailbreak the model, but subsequent responses would result in a refusal; this is how I identified the problem—the model reverted to refusing requests because the system prompt and context had been lost. -_-

[EugeoSynthesisThirtyTwo]

I have the same problem, but I tried to reproduce it using your command. How do you fit this huge context on a single rtx 5090 ? I have two 5090s and 192GB of RAM but it crashes

C:\Users\me\Docs\ia\llama.cpp>build\llama-server.exe --version
version: 9865 (d1ac5aca1)
built with MSVC 19.50.35729.0 for x64

C:\Users\armaguedin\Docs\ia\llama.cpp>build\llama-server.exe --list-devices
Available devices:
  CUDA0: NVIDIA GeForce RTX 5090 (32606 MiB, 30991 MiB free)
  CUDA1: NVIDIA GeForce RTX 5090 (32606 MiB, 30991 MiB free)

C:\Users\me\Docs\ia\llama.cpp>build\llama-server.exe -m C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf --ctx-size 131072 --flash-attn on -fitc 131072 --temp 1 --top-p 1 --host 0.0.0.0 --port 5001 -a Deepseek-V4-Flash -b 2048 -ub 2048 --jinja --cpu-moe
0.00.020.594 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.126.086 I srv    load_model: loading model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'
0.00.342.763 W common_fit_params: failed to fit params to free device memory: model_params::tensor_buft_overrides already set by user, abort
0.00.475.574 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.11.478.323 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 228314.04 MiB on device 0: cudaMalloc failed: out of memory
0.11.478.329 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 239404619264
0.11.478.331 E graph_reserve: failed to allocate compute buffers
0.11.494.813 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.11.494.822 E cmn  common_init_: failed to create context with model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'
0.11.494.823 E cmn  common_init_: failed to create context with model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'

[wenyifancc]

I have the same problem, but I tried to reproduce it using your command. How do you fit this huge context on a single rtx 5090 ? I have two 5090s and 192GB of RAM but it crashes

C:\Users\me\Docs\ia\llama.cpp>build\llama-server.exe --version
version: 9865 (d1ac5aca1)
built with MSVC 19.50.35729.0 for x64

C:\Users\armaguedin\Docs\ia\llama.cpp>build\llama-server.exe --list-devices
Available devices:
  CUDA0: NVIDIA GeForce RTX 5090 (32606 MiB, 30991 MiB free)
  CUDA1: NVIDIA GeForce RTX 5090 (32606 MiB, 30991 MiB free)

C:\Users\me\Docs\ia\llama.cpp>build\llama-server.exe -m C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf --ctx-size 131072 --flash-attn on -fitc 131072 --temp 1 --top-p 1 --host 0.0.0.0 --port 5001 -a Deepseek-V4-Flash -b 2048 -ub 2048 --jinja --cpu-moe
0.00.020.594 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.126.086 I srv    load_model: loading model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'
0.00.342.763 W common_fit_params: failed to fit params to free device memory: model_params::tensor_buft_overrides already set by user, abort
0.00.475.574 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.11.478.323 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 228314.04 MiB on device 0: cudaMalloc failed: out of memory
0.11.478.329 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 239404619264
0.11.478.331 E graph_reserve: failed to allocate compute buffers
0.11.494.813 E llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
0.11.494.822 E cmn  common_init_: failed to create context with model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'
0.11.494.823 E cmn  common_init_: failed to create context with model 'C:/Users/me/sc/model/DeepSeek-V4-Flash-MXFP4/DeepSeek-V4-Flash-MXFP4-00001-of-00004.gguf'

I am using the Vulkan backend, and you need to set the corresponding environment variable flag; testing shows that the limit for a single 5090 card is around a 512k context window.
set GGML_VK_ALLOW_SYSMEM_FALLBACK=1

[EugeoSynthesisThirtyTwo]

Oh right, I forgot this variable. I retried and it raises the exact same error with the exact same logs

[wenyifancc]

Oh right, I forgot this variable. I retried and it raises the exact same error with the exact same logs

Does "the exact same error with the exact same logs" mean the model still fails to load, or does it refer to the issue itself?

[EugeoSynthesisThirtyTwo]

It means

0.11.478.323 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 228314.04 MiB on device 0: cudaMalloc failed: out of memory
0.11.478.329 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 239404619264

Just now I followed this guide step by step too https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md and it still raises OOM errors

[wenyifancc]

It means

0.11.478.323 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 228314.04 MiB on device 0: cudaMalloc failed: out of memory
0.11.478.329 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 239404619264

Just now I followed this guide step by step too https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md and it still raises OOM errors

That’s strange; when I use these commands, a 128k context consumes only 17 GB of VRAM.

[EugeoSynthesisThirtyTwo]

I recommend trying this pull request
fairydreaming#2

[fairydreaming]

@wenyifancc I'm trying to reproduce this on mainline llama.cpp but without luck. The model answers correctly every time.

Maybe it has something to do with using Vulkan backend, will try that next.

Edit: Nope, on Vulkan also the model responds correctly every time.

Also llama-server clearly used cached prompts like in your logs:

$ ./bin/llama-server -m ~/ggufs/DeepSeek-V4-Flash.gguf -ub 2048 -np 1 -cmoe -fa 1 --no-repack -c $((1*128*1024)) --host 192.168.18.10 --jinja --temp 1 --top-p 1
...
2.30.292.975 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.977
2.30.293.696 I slot launch_slot_: id  0 | task 160 | processing task, is_child = 0
2.54.822.898 I slot print_timing: id  0 | task 160 | n_decoded =    100, tg =   4.61 t/s, tg_3s =   4.61 t/s
2.57.958.852 I slot print_timing: id  0 | task 160 | n_decoded =    115, tg =   4.63 t/s, tg_3s =   4.78 t/s
3.00.978.812 I slot print_timing: id  0 | task 160 | n_decoded =    128, tg =   4.60 t/s, tg_3s =   4.30 t/s
3.04.113.882 I slot print_timing: id  0 | task 160 | prompt eval time =    2830.14 ms /    13 tokens (  217.70 ms per token,     4.59 tokens per second)
3.04.113.890 I slot print_timing: id  0 | task 160 |        eval time =   30989.93 ms /   143 tokens (  216.71 ms per token,     4.61 tokens per second)
3.04.113.890 I slot print_timing: id  0 | task 160 |       total time =   33820.07 ms /   156 tokens
3.04.113.892 I slot print_timing: id  0 | task 160 |    graphs reused =        287
3.04.115.177 I slot      release: id  0 | task 160 | stop processing: n_tokens = 6457, truncated = 0
3.08.340.234 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.978
3.08.340.784 I slot launch_slot_: id  0 | task 305 | processing task, is_child = 0
3.32.882.534 I slot print_timing: id  0 | task 305 | n_decoded =    100, tg =   4.61 t/s, tg_3s =   4.61 t/s
3.36.021.345 I slot print_timing: id  0 | task 305 | n_decoded =    115, tg =   4.63 t/s, tg_3s =   4.78 t/s
3.39.040.160 I slot print_timing: id  0 | task 305 | n_decoded =    128, tg =   4.60 t/s, tg_3s =   4.31 t/s
3.40.713.842 I slot print_timing: id  0 | task 305 | prompt eval time =    2853.78 ms /    13 tokens (  219.52 ms per token,     4.56 tokens per second)
3.40.713.850 I slot print_timing: id  0 | task 305 |        eval time =   29519.17 ms /   136 tokens (  217.05 ms per token,     4.61 tokens per second)
3.40.713.850 I slot print_timing: id  0 | task 305 |       total time =   32372.95 ms /   149 tokens
3.40.713.851 I slot print_timing: id  0 | task 305 |    graphs reused =        418
3.40.715.133 I slot      release: id  0 | task 305 | stop processing: n_tokens = 6450, truncated = 0

but answered correctly: You asked me to play the role of a **detective**

[fairydreaming]

Finally managed to reproduce this. The critical thing to do was to NOT set -np 1 when running the server (which I do routinely). Without this server runs with default 4 parallel sequences and then the problem manifests itself (even if I post only a single request at a given time):

The model answered during second request:

"I apologize for the confusion. You didn't actually ask me to play a specific role like a detective or analyst in your query. Your message was simply: **\"What role did I just ask you to play?\"**
...

@EugeoSynthesisThirtyTwo This is not fixed in my branch either.

So the current workaround is to use -np 1 when running the server.