@johny-mnemonic's stack — heterogeneous GPU cluster

[jaylfc]

Opening this as @johny-mnemonic's home thread. He's the first user genuinely outside my own hardware bubble (Intel N100 iGPU + multi-AMD-GPU plans + remote Ollama + LiteLLM in front of self-hosted endpoints + planning llama.cpp RPC for distributed AMD inference) and the gaps his setup hits aren't ones I'd otherwise surface for a while. So consider this a rolling space rather than a debug ticket.

How this works

• Post here freely — bug, friction, half-formed idea, "have you considered X", whatever. No need to ladder up to a formal issue first.
• I'll cross-link any PR or issue that touches this stack so you can weigh in early.
• When you say "this is what's blocking me next," that genuinely shifts what I prioritise. The Pi-first roadmap doesn't naturally surface multi-vendor gaps; you do.
• Pinned so anyone arriving with similar hardware can find this thread.

Already in flight from #312

• bug(hardware): runtime Vulkan detection misses iGPUs when vulkan-tools not installed #354 — runtime Vulkan detection misses iGPUs without vulkan-tools installed (the reason llama-cpp wasn't appearing in your Store)
• ux(store): IncompatibleToggle is unfindable when nothing is compatible #355 — Store's incompatibility toggle is unfindable when nothing matches your tier
• bug(agents): model picker doesn't surface network-category providers #356 — agent model picker doesn't surface network-category providers (the reason your remote Ollama models don't show up under Agent → deploy)
• feat(providers): support API key on local providers (llama.cpp, LiteLLM with --api-key) #349 — API key field for local providers (LiteLLM, llama.cpp with --api-key) — already merged in feat(providers): API key for local + OpenAI-compatible custom endpoint (#349 #350) #352
• feat(providers): add OpenAI-compatible custom endpoint (editable base URL + key) #350 — OpenAI-compatible cloud provider type with editable URL + key — already merged in feat(providers): API key for local + OpenAI-compatible custom endpoint (#349 #350) #352
• refactor: consolidate provider-type lists across config / providers / llm_proxy / ProvidersApp #351 — consolidate provider-type lists (refactor groundwork so the next provider type is a one-liner) — open as refactor(providers): canonical type schema + /api/providers/schema endpoint (#351) #353

Hardware/setup snapshot

Filling this in so anyone scrolling has context — please correct anything wrong:

• Host: Intel N100 mini-PC, 3.4GB RAM, Ubuntu 25.10
• Discrete GPUs (planned cluster): 2× AMD + 1× ?, intended to drive distributed inference via llama.cpp RPC (since EXO doesn't support AMD ROCm)
• Existing infra: LiteLLM proxy in front of network-side endpoints; Ollama on a Windows gaming PC

Floor for me to call out: with 3.4GB RAM on the N100, even with Vulkan working, you're going to be tight on what runs locally. The realistic local story is small Q4-quantised models (Qwen3 1.7B, SmolLM2-360M, etc.). The bigger-model story is the distributed AMD cluster once #354 + the Vulkan story + AMD-side install scripts catch up. I'll factor that in when prioritising.

[jaylfc]

Quick steer on your 3.4GB box from the live test we just ran in a Debian 12 LXC.

4GB total works fine for taOS as a dedicated memory layer — running the embedder + reranker + memory store with no local agent containers. That's the role the N100 is right-sized for in your cluster.

Where it breaks: trying to deploy an agent (OpenClaw / Hermes etc.) on the controller itself. Reproduced this in the test — at 4GB the nested agent container can't get enough RAM to launch, the deploy reports "failed" with no actionable error, and there's nothing in the wizard today that warns you before you click Deploy. Filing two issues from the test: a pre-flight check that'd surface the constraint up-front, and (longer-term) the wizard hint that says "this device is good for memory only, add a worker for agents".

So for your stack: keep agents on your gaming PCs (or the AMD cluster once that's wired) and let the N100 be the memory + cluster controller. The cross-machine agent-on-worker path is a feature gap we're working through but the architecture supports it.

Also from the same test: discovered three install-script gaps that would have hit you on a fresh Debian-family install (apt install incus failing on Debian 12, no incus admin init --auto after install, LiteLLM not auto-installed). All filed + being fixed in the next batch — your install was cleaner than this on Ubuntu 25.10 because incus is in default Ubuntu 25.10 repos, but Debian 12 / Ubuntu 22.04 LTS users would have hit them. Worth knowing in case you ever spin up another node.

[johny-mnemonic]

OK, will increase VM memory from 4GB to 8GB and will see if it will be OK for single running agent. I can go up to 12GB safely I think...

I don't want to install workers on other PCs for now. I am not even sure it will work on Windows. I have seen only Mac/Linux support mentioned.
They will just run LLMs on their powerful GPUs using either Ollama or llama.cpp and the rest should run on the taOS on the N100.

[johny-mnemonic]

I was busy with other stuff, but yesterday I updated taOS and tried to deploy my first agent again.
It succeeded as I was finally able to select a model and also the memory seems to be working.

Unfortunately when I tried to interact with the agent I have found quite a few issues.
In the "Agents" app basically any button I click either does nothing or it opens new window that says Connecting to shortcut..., but almost immediately ends with these two lines:

WebSocket connection error

Connection closed

This happens on all these: Hermes chat, Hermes doctor and Container shell. The rest of the buttons does nothing.

I tried to open "Messages" app and chat with the agent, but it successfully processed only single message and when I asked again it was thinking for a while and then produced message: [hermes error: ] and since then I can see it is sending prompts to my llama.cpp again and again, while I have no clue what it is doing as I am unable to see agent logs or interact with it at all as described above.

I have noticed that "Web Search" skill is enabled for the agent and it says it works using SearXNG or Perplexica. So I went to the Store and tried to install SearXNG, but it produces error saying "docker is not available on this controller". Seems like the container engine (incus) we installed previously is not used for some reason and it want's to use docker instead🤷‍♂️

I have also tried to get to the console of the agent using incus console command, but it is asking for login and password, which I have no clue about...

[jaylfc]

Hey, sorry for the slow reply, and thanks for the detailed writeup. Good news: most of what you ran into has been fixed since.

• The "Connecting to shortcut... WebSocket connection error / Connection closed" on Hermes chat, Hermes doctor and Container shell was a real bug (the shortcut was redeeming against an address your browser couldn't reach, and the container shell used the wrong incus exec flag). Both are fixed now. Pull the latest, open the agent's detail panel, and those shortcuts should connect. The Container shell there is also the intended way into the agent's container, so you won't need incus console or any login.
• SearXNG failing with "docker is not available on this controller": that was because your install predates a change to the installer. The installer now sets up Docker by default (alongside incus, with the networking configured so they coexist), since Store apps like SearXNG run as Docker containers. Re-run the installer / update to the latest and SearXNG should install fine.
• The [hermes error: ] in Messages and the agent looping prompts to your llama.cpp: I've been running Hermes in the chat app here without hitting that, so it may already be resolved, but it could also be specific to your provider/model/llama.cpp setup. Could you update and retry, and if it still happens, open the (now-working) Container shell on the agent, grab the recent logs, and paste the full error if there's more after [hermes error: ]? That'll let me pin down whether it's the endpoint, the model, or the framework.

Appreciate you sticking with the heterogeneous setup. Let me know how it goes after updating.