[Bug] Remote `/v1/chat/completions` requests with a `model` field fail to load the chat model on demand after a cold start

[0fflineuser]

Summary

After a cold start with autoswapmode: true + nomodel: true, remote
OpenAI-compatible chat requests to /v1/chat/completions that include a
non-empty model field ALL fail with finish_reason:"error",
prompt_tokens:0 — the chat model never loads from these requests no matter
how many you send. The model can only be loaded by manual intervention: either
sending one request with an empty/absent model field, or chatting from the
bundled koboldcpp webui. After that, requests with a model field work fine.

This is a hard blocker for any client that requires a non-empty model name
(e.g., Home Assistant's conversation agent) — after a koboldcpp restart, such
clients are completely non-functional until manual intervention.

Only chat is affected. STT, TTS, and embeddings remote requests load
their models on demand fine after a cold start (verified). Image generation
was not tested. The bug is specific to /v1/chat/completions ) with
a non-empty model field.

Problem

I run koboldcpp as an all-in-one server on a 12GB AMD RX 6700 XT, hosting
multiple model types (chat, embeddings, STT, TTS, image) with autoswapmode: true, routermode: true, nomodel: true.

autoswapmode loads each model type on demand and swaps between them, which
is great for VRAM efficiency. However, after a cold start (boot or restart):

• Remote OpenAI-compatible chat requests to /v1/chat/completions that
  include a non-empty model field ALL fail with finish_reason:"error" and
  prompt_tokens:0 — not just the first one. The chat model never loads
  from these requests no matter how many you send.
• The chat model can ONLY be loaded by one of these manual interventions:
  • Send one request with an empty/absent model field, OR
  • Chat from the bundled koboldcpp webui (KoboldAI Lite).
• After either of the above, remote /v1/chat/completions requests WITH a
  model field work fine.
• Remote requests for the OTHER model types load their models on demand fine
  after a cold start — only chat is broken. Verified: STT
  (/v1/audio/transcriptions), TTS (/v1/audio/speech), and embeddings
  (/v1/embeddings) all work remotely with a model field set. Image
  generation was not tested.
• Setting nomodel: false does not help — the configured model_param is
  not loaded on startup when autoswapmode: true is set.

Clients that require a non-empty model name (e.g., Home Assistant's
conversation agent via the hass_local_openai_llm integration) are
hard-blocked after a koboldcpp restart: all their requests fail silently with
no recovery path, since they cannot send an empty model field and cannot
access the bundled webui. The server only becomes usable once someone
manually loads the chat model via the workarounds above.

Minimal reproduction setup

Self-contained Docker Compose setup to reproduce the bug. Requires an AMD GPU
with ROCm/Vulkan drivers on the host. Uses Gemma3-1B as the chat model —
small enough to test quickly.

docker-compose.yml:

services:
  koboldcpp:
    image: koboldai/koboldcpp:latest
    container_name: koboldcpp
    ports:
      - "5555:5001"
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    environment:
      - HSA_OVERRIDE_GFX_VERSION=10.3.0
      - KCPP_DONT_TUNNEL=true
      - KCPP_DONT_UPDATE=true
      - KCPP_ARGS=--config /workspace/configs/all-in-one.kcpps --admindir /workspace/configs --downloaddir /workspace
    volumes:
      - ./config:/workspace/configs:ro
      - koboldcpp-data:/workspace
    group_add:
      - 989 # render
      - 985 # video
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5001/api/extra/version"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s

volumes:
  koboldcpp-data:

config/all-in-one.kcpps:

{
  "model_param": "https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf?download=true",
  "whispermodel": "https://huggingface.co/koboldcpp/whisper/resolve/main/whisper-tiny.en-q5_1.bin",
  "ttsmodel": "https://huggingface.co/koboldcpp/tts/resolve/main/Kokoro_no_espeak_Q4.gguf",
  "contextsize": 8192,
  "gpulayers": 99,
  "threads": 4,
  "jinja": true,
  "jinjatools": true,
  "admin": true,
  "routermode": true,
  "autoswapmode": true,
  "nomodel": true,
  "host": "0.0.0.0",
  "port": 5001
}

Then:

docker compose up -d
# wait for the container to become healthy (~30s after model download)
docker compose logs -f koboldcpp

The curl commands in "How to reproduce" below run against localhost:5555
exposed by this compose. For step 1, use "model":"gemma-3-1b-it-Q4_K_M" (or
any non-empty string — the bug reproduces regardless of the model name).

How to reproduce

After the container is up and healthy:

1. Fails — /v1/chat/completions with a non-empty model field
   (reproduces the bug; repeated requests all fail the same way):

   curl -s http://localhost:5555/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"gemma-3-1b-it-Q4_K_M","messages":[{"role":"user","content":"hi"}]}'

   Returns finish_reason:"error", prompt_tokens:0. Repeat → still fails.

2. Manual workaround — one request with an empty/absent model field
   loads the chat model:

   curl -s http://localhost:5555/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"messages":[{"role":"user","content":"hi"}]}'

   The chat model loads and the request succeeds.

3. Chat from the bundled webui (open http://localhost:5555 in a browser
   and send a message) also loads the chat model.

4. Now works — after step 2 or 3, the request from step 1 (with a non-empty
   model field) succeeds:

   curl -s http://localhost:5555/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"gemma-3-1b-it-Q4_K_M","messages":[{"role":"user","content":"hi"}]}'

   Returns a real response with prompt_tokens > 0.

5. Works — other model types load on demand from remote requests (only
   chat is broken; STT, TTS, and embeddings confirmed working remotely with a
   model field after a cold start). The .kcpps above includes
   whisper-tiny.en-q5_1 (STT) and Kokoro_no_espeak_Q4 (TTS) so these can be
   tested directly:

   # STT — transcribe a short WAV file (loads whisper on demand)
   curl -s http://localhost:5555/v1/audio/transcriptions \
     -H 'Content-Type: multipart/form-data' \
     -F 'model=whisper-tiny.en-q5_1' \
     -F 'file=@/path/to/test.wav'
   
   # TTS — generate audio from text (loads Kokoro on demand)
   curl -s http://localhost:5555/v1/audio/speech \
     -H 'Content-Type: application/json' \
     -d '{"model":"Kokoro_no_espeak_Q4","input":"hello world","voice":"af"}'
   
   # Embeddings (requires an `embeddingsmodel` field in the `.kcpps`)
   curl -s http://localhost:5555/v1/embeddings \
     -H 'Content-Type: application/json' \
     -d '{"model":"<model name>","input":"test"}'

Expected behavior

Remote /v1/chat/completions requests with a non-empty model field should
load the chat model on demand, just like the bundled webui and the other
model types (embeddings, STT, TTS, image) already do. Today these requests
fail persistently after a cold start until the model is loaded by manual
intervention (an empty-model request or webui chat).

Impact

Home Assistant's conversation agent (via the hass_local_openai_llm
integration) requires a non-empty model name and does not retry on failure.
After every koboldcpp restart, HA Assist is completely non-functional — all
chat requests fail silently with finish_reason:"error",
prompt_tokens:0 and there is no recovery path without manual intervention
(an empty-model curl or chatting from the bundled webui). This makes the
autoswapmode + nomodel setup unusable for unattended/headless deployments
serving clients that require a model name.

Environment

• KoboldCpp 1.118.1
• Linux x86_64 (NixOS)
• AMD RX 6700 XT (12GB VRAM, ROCm/Vulkan)
• Config: nomodel: true, autoswapmode: true, routermode: true,
  jinja: true, jinjatools: true, adminunloadtimeout: 0

[0fflineuser]

The issue also seem to comme back if using an adminunloadtimeout with a non zero value.

[LostRuins]

@Anai-Guo has fixed the issue. Rolling builds will be available in 2 hours