[bug] Image upgrade doesn't detect stale backend venv → cryptic 'Engine core init failed'; install.sh needs uv but nvidia-l4t image has neither uv nor pip

[pos-ei-don]

Summary

Two related bugs I hit today on the nvidia-l4t-arm64-cuda-13 variant when upgrading LocalAI from v4.6.0 to v4.6.2 while keeping the persistent /backends/ volume:

1) Silent ABI mismatch between container image and backend venv

After the image swap (keeping -v localai-backends:/backends), model loads fail with:

ERROR Failed to load model modelID="coder"
  error=failed to load model with internal loader: could not load model (no success):
  Unexpected err=RuntimeError('Engine core initialization failed. See root cause above.
  Failed core proc(s): {}'), type(err)=<class 'RuntimeError'>
  backend="cuda13-nvidia-l4t-arm64-vllm"

The backend subprocess exits with exitCode=0 in ~10s. The Failed core proc(s): {} set is empty, no Python traceback surfaces in docker logs, and there is no obvious hint that the backend venv on the persistent volume is incompatible with the newer LocalAI image.

The LocalAI startup log DOES contain:

INFO Backend upgrade available (new build) backend="cuda13-nvidia-l4t-arm64-vllm"

…but that line looks informational (as opposed to causal), sits among unrelated "backend upgrade available" lines for other backends, and has no severity elevation. Without prior knowledge of the mechanism it's very hard to connect this to the load failure.

2) Recovery path via local-ai backends install is broken on the minimal image

The documented recovery — reinstalling the backend so the venv is rebuilt for the new image ABI — fails because the minimal nvidia-l4t-arm64-cuda-13 container has neither uv nor pip:

$ docker exec localai /local-ai backends install cuda13-nvidia-l4t-arm64-vllm
downloading backend cuda13-nvidia-l4t-arm64-vllm 100% |...|  DONE

$ docker exec localai bash /backends/cuda13-nvidia-l4t-arm64-vllm/install.sh
Initializing libbackend for cuda13-nvidia-l4t-arm64-vllm
Using portable Python
./common/libbackend.sh: line 378: uv: command not found

Result: OCI-layer download succeeds, backend.py and requirements-*.txt land in the backend dir, but venv/ never gets built. Backend is permanently broken until I manually restore a pre-upgrade snapshot from outside the container.

Both pip and uv are absent inside the image, so there is no way to bootstrap the missing package manager from within.

Repro

1. Run LocalAI v4.6.0 (nvidia-l4t-arm64-cuda-13), install a backend and load a model successfully.
2. Stop container, docker rm, and start LocalAI v4.6.2 with the same volumes (localai-backends, localai-models, …).
3. POST /v1/chat/completions for any model on that backend → 500 with the empty Failed core proc(s): {} message above.
4. Try to recover with docker exec localai /local-ai backends install cuda13-nvidia-l4t-arm64-vllm → OCI download completes but venv is not rebuilt.
5. Try bash /backends/cuda13-nvidia-l4t-arm64-vllm/install.sh inside the container → uv: command not found.

Expected behavior

At least one, ideally all:

• Diagnose ABI/version mismatch and surface it clearly: e.g. Backend cuda13-nvidia-l4t-arm64-vllm was installed with LocalAI v4.6.0 and may be incompatible with v4.6.2; run 'local-ai backends upgrade' before serving.
• Elevate "Backend upgrade available" to WARN with a "reinstall required" hint when the delta likely breaks ABI.
• Pipe backend-subprocess stderr into docker logs so the real Python-side traceback isn't lost behind the empty Failed core proc(s): {} set. The current wrapper hides the actual error.
• Bundle uv (or fall back to pip) in every nvidia-l4t-arm64-cuda-* image so that local-ai backends install actually completes end-to-end. Alternatively make install.sh bootstrap uv itself when missing.

Right now the documented recovery path is impossible from within the container.

Version

LocalAI v4.6.2 (variant nvidia-l4t-arm64-cuda-13). Reproducible.

Environment

• Host: NVIDIA DGX Spark (GB10, ARM64, 121 GB unified memory)
• Image: localai/localai:v4.6.2-nvidia-l4t-arm64-cuda-13
• Backend venv persistent on localai-backends volume

[pos-ei-don]

Follow-up after another cycle on this — image update 2026-07-08 to master-nvidia-l4t-arm64-cuda-13 @ revision a0ed395c (which contains PR #10742 for "backend logs in single mode"):

On the ABI-mismatch detection (items 1 + 2 in the original report):
Still reproducible. My practical workaround remains bypassing the reinstall path entirely — I keep a backend snapshot dir (/backends/cuda13-nvidia-l4t-arm64-vllm.pre-<image>-<ts>) and if a load fails post-update, I rm -rf the current backend dir and mv the snapshot back. Works, but is manual and requires knowing the pattern in advance. A clear "backend venv appears incompatible with current LocalAI image (built against , current )" warning would save the discovery step.

On capturing subprocess stderr (item 3):
PR #10742 has landed in a0ed395c but does not solve the vLLM-backend case in my setup — the second cold-load fail after the image update (~10 s, "Failed core proc(s): {}") was just as opaque in container logs as before. I suspect "single mode" in that PR means something orthogonal to LOCALAI_SINGLE_ACTIVE_BACKEND=true, but I did not chase this.

What did work for me was a small local patch in the vLLM backend's run.sh:

# stderr-capture (added just after 'set -x')
_STDERR_LOG="${EDIR:-$(dirname $0)}/vllm-stderr.log"
exec 2> >(tee -a "$_STDERR_LOG" >&2)
export PYTHONFAULTHANDLER=${PYTHONFAULTHANDLER:-1}
export VLLM_LOGGING_LEVEL=${VLLM_LOGGING_LEVEL:-INFO}

After this patch, four previously-invisible failure modes surfaced in one debugging session (KV-cache-init errors, VLLM_ENGINE_READY_TIMEOUT_S=600 timeouts, FlashInfer JIT compile errors, gRPC/abseil mutex assertions) — none of which had been visible in LocalAI's container log before, even with LOCALAI_LOG_LEVEL=debug. Filed the timeout finding separately upstream at vllm-project/vllm#48031.

Happy to submit this as a PR against the backend run.sh template if you'd like — it's ~4 lines and platform-neutral. Let me know which subdir is the canonical source (the current run.sh under /backends/cuda13-nvidia-l4t-arm64-vllm/ looks like it's fetched at install time, so I want to hit the right template repo).

On bundling uv / pip (item 4):
No update from my side — I didn't retest the local-ai backends install path because the snapshot rollback covers my case. Leaving that one to whoever runs into it on a machine without a snapshot.