Misc. bug: ui: "Stream connection lost and could not be resumed" with slow models after 1a87dcdc

[sdroege]

Name and Version

./llama-cli --version
version: 9843 (86b94708f)
built with GNU 16.1.1 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server --port 8012 --models-max 1 --sleep-idle-seconds 600 --models-preset ./models.ini --host 0.0.0.0 --webui-mcp-proxy --threads-http 2 -np 1 --threads 8 --tools get_datetime

Relevant models.ini entry:

[unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M]
mmap = true
fit = on
fit-target = 0
no-mmproj = true
spec-type = ngram-mod

c = 131072
temp = 0.15

Problem description & steps to reproduce

When using slow models from the llama-server web UI, after 1a87dcd it errors out in the UI after a couple of seconds during prompt processing. For me this only happens with Mistral Small 4 or Qwen 3.5 122B-A10B.

Error message in the UI is "Stream connection lost and could not be resumed". Below some more details from the Firefox devtools.

Reverting 1a87dcd on latest git master makes the problem disappear again.

Firefox console:

[AgenticStore] Starting agentic flow with 12 tools [bundle.BibSbayV.js:226:12667](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)
Failed to connect with service worker for getRuleList msg undefined [content.js:1:34437](moz-extension://d07b1982-a590-4189-90ef-886c3b98d90b/content.js)
XHR
GET
[http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3::unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M?from=563](http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3%3A%3Aunsloth%2FMistral-Small-4-119B-2603-GGUF%3AQ4_K_M?from=563)
[HTTP/1.1 404 Not Found 16000ms]

Error in sendMessage: Error: Stream connection lost and could not be resumed
    Immutable 32
[bundle.BibSbayV.js:181:20875](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)
Streaming error: Error: Stream connection lost and could not be resumed
    Immutable 32
[bundle.BibSbayV.js:226:38847](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)
XHR
GET
[http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3::unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M?from=0](http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3%3A%3Aunsloth%2FMistral-Small-4-119B-2603-GGUF%3AQ4_K_M?from=0)
[HTTP/1.1 404 Not Found 0ms]

attachServerStream replay got HTTP 404 for conv 8635c7a5-73f8-44ff-acbf-ffe3f11572c3 [bundle.BibSbayV.js:226:23762](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)
XHR
GET
[http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3::unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M?from=0](http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3%3A%3Aunsloth%2FMistral-Small-4-119B-2603-GGUF%3AQ4_K_M?from=0)
[HTTP/1.1 404 Not Found 0ms]

attachServerStream replay got HTTP 404 for conv 8635c7a5-73f8-44ff-acbf-ffe3f11572c3 [bundle.BibSbayV.js:226:23762](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)
XHR
GET
[http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3::unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M?from=0](http://127.0.0.1:8012/v1/stream/8635c7a5-73f8-44ff-acbf-ffe3f11572c3%3A%3Aunsloth%2FMistral-Small-4-119B-2603-GGUF%3AQ4_K_M?from=0)
[HTTP/1.1 404 Not Found 0ms]

attachServerStream replay got HTTP 404 for conv 8635c7a5-73f8-44ff-acbf-ffe3f11572c3 [bundle.BibSbayV.js:226:23762](http://127.0.0.1:8012/_app/immutable/bundle.BibSbayV.js)

​```

Firefox devtools network view:

First Bad Commit

1a87dcd , i.e. #23226

Relevant log output

Details

[33837] cmd_child_to_router:state:{"state":"ready","payload":{"id":"unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M","aliases":["unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M"],"tags":[],"object":"model","created":1782798344,"owned_by":"llamacpp","meta":{"vocab_type":2,"n_vocab":131072,"n_ctx":131072,"n_ctx_train":1048576,"n_embd":4096,"n_params":118972826624,"size":73755267072}}}
0.44.639.211 I srv  proxy_reques: proxying request to model unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M on port 33837
[33837] 0.07.838.432 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
[33837] 0.07.838.653 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
[33837] 0.25.434.913 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   2048, progress = 0.27, t =  17.60 s / 116.39 tokens per second
1.02.245.290 I srv  proxy_reques: proxying request to model unsloth/Mistral-Small-4-119B-2603-GGUF:Q4_K_M on port 33837
[33837] 0.41.149.423 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   4096, progress = 0.54, t =  33.31 s / 122.96 tokens per second
1.17.958.880 E srv    operator(): http client error: Connection handling canceled
[33837] 0.56.458.303 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   6144, progress = 0.81, t =  48.62 s / 126.37 tokens per second
[33837] 1.11.568.477 I slot print_timing: id  0 | task 0 | prompt eval time =   63158.38 ms /  7572 tokens (    8.34 ms per token,   119.89 tokens per second)
[33837] 1.11.568.480 I slot print_timing: id  0 | task 0 |        eval time =     571.41 ms /    10 tokens (   57.14 ms per token,    17.50 tokens per second)
[33837] 1.11.568.481 I slot print_timing: id  0 | task 0 |       total time =   63729.79 ms /  7582 tokens
[33837] 1.11.568.485 I slot print_timing: id  0 | task 0 |    graphs reused =          9
[33837] 1.11.568.817 I slot      release: id  0 | task 0 | stop processing: n_tokens = 7581, truncated = 0

[sdroege]

#25177 also doesn't fix this.

CC @ServeurpersoCom who wrote the original changes that broke it.

[ServeurpersoCom]

Right! The child's wake-up/loading time (LOADING/SLEEPING) needs to be included in the window during which the router agrees to route the stream, not just the LOADED state. Here is a fix for you to test while I try to reproduce the issue on my end; even my 1xxB models load too quickly, so I think missed this edge case during testing :

diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
index b0d957f1dc..99896749aa 100644
--- a/tools/server/server-models.cpp
+++ b/tools/server/server-models.cpp
@@ -1619,7 +1619,7 @@ static std::optional<server_model_meta> resolve_child_for_conv(
         return std::nullopt;
     }
     auto meta = models.get_meta(*tracked);
-    if (meta.has_value() && meta->is_ready()) {
+    if (meta.has_value() && meta->is_running()) {
         return meta;
     }
     return std::nullopt;

[sdroege]

Thanks for looking into it. That unfortunately seems to fail the same way still.

[ServeurpersoCom]

Yes, there are plenty of edge cases. If I hit F5 while a model is loading, it breaks the context.
How do you reproduce yours? I imagine it just breaks when loading a very large model, right?

[sdroege]

I just start a new chat with a larger model while it's still loading, and some time during pp usually this error happens.

[ServeurpersoCom]

I'm going to run some tests focusing on slow speed, for instance, trying to load a 100B MoE model onto an HDD and running it on the CPU.

[aa956]

Run into this or something very similar trying DeepSeek v4 flash with cpu-moe (after #24162 was merged).

Model that I've tried to use on single RTX 3090 is 2-bit DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf from https://huggingface.co/antirez/deepseek-v4-gguf/tree/main

I'm also using external proxy/router, llama-swap, but that's probably not related.

Command line:

LLAMA_ATTN_ROT_DISABLE=1 ./bin/llama-server \
--jinja \
--spec-type ngram-mod \
--cpu-moe \
--fit-ctx 131072 \
--parallel 1 \
--model ./models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf

No need for HDD-s, it's probably enough to use the default --mmap to have it slow enough even from NVMe.

[ServeurpersoCom]

Very interesting.

The crazy thing is, I tested the PR for this model without issue a few days ago, and it vastly exceeds my VRAM:

[MoE-MiniMax-M3-426B]
no-mmap = off
m = unsloth/MiniMax-M3-GGUF/MiniMax-M3-UD-Q2_K_XL-00001-of-00004.gguf
n-cpu-moe = 28
c = 131072

[ServeurpersoCom]

And I also run the following on my Raspberry Pi 5 16GB/SSD without issue:

[*]
fit = off               ; Disable automatic memory fitting (default: on)
ctk = q8_0              ; KV cache key quantization
ctv = q8_0              ; KV cache value quantization
fa = on                 ; Enable flash attention
np = 1                  ; Parallel request batching (default: 4)
kvu = off               ; Unified KV cache buffer (default: on)
b = 128                 ; Logical maximum batch size (default: 2048)
ub = 512                ; Physical maximum batch size (default: 512)
webui-mcp-proxy = on    ; Enabled for llama-server issue debugging, not exposed to the internet

[MoE-GPT-OSS-20B]
m = lmstudio-community/gpt-oss-20b-GGUF/gpt-oss-20b-MXFP4.gguf
c = 4096

[MoE-Vision-Qwen3.5-35B-A3B]
m = unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q3_K_XL.gguf
c = 4096

[MoE-Gemma-4-E2B-IT]
m = unsloth/gemma-4-E2B-it-GGUF/gemma-4-E2B-it-Q8_0.gguf
mm = unsloth/gemma-4-E2B-it-GGUF/mmproj-BF16.gguf
c = 4096

[MoE-Gemma-4-E4B-IT]
m = unsloth/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q8_0.gguf
c = 4096

So the bug lies in the difference. My static content is hosted on Apache2. everywhere!

[sdroege]

Let me know if there's any logs or other information I could provide to help tracking this down :)

[ServeurpersoCom]

Why --threads-http 2 ? Can you try without ? It might be HTTP thread starvation !

The SSE replay buffer holds two long-lived connections per generation (base stream + resume GET), so 2 HTTP threads get fully starved and everything else queues until it times out: "could not be resumed". Default is at least 5. Try dropping the flag or --threads-http 8.

If that fixes it, a real fix can be : "the resume path shouldn't need a second blocking worker".