Initial setup

[johny-mnemonic]

I have successfully installed TAOS and logged into Web Desktop, but there is no model to select and I see no way to add model providers. There was no "setup wizard" on first launch. I guess it is not implemented yet...

There is a reference to "Settings → Providers → Add Provider" but I don't see such section in the Settings.

There is a "Advanced" section in the Settings, but when I try to add for example one of my Ollama endpoints it produces Save failed (404) error and the "Validate" button seems to do nothing. In the log it says "PUT /api/settings/config HTTP/1.1" 404 Not Found , so I guess this is also not implemented yet and I will have to find where the config file is and modify it on CLI, right?

I have modified the defaults there to look like this:

server:
  port: 6969
  host: 0.0.0.0

agents:
  max_concurrent: 5
  default_model: qwen3.5:4b

providers:
  - name: ollama
    url: http://ollama1.mydomain:11434

Hope this is correct.

I already have LiteLLM running in my network. Not sure if that can be used instead of internal LiteLLM referenced in the docs.

[jaylfc]

@johny-mnemonic apologies for the confusion — the docs were out of date. Provider configuration lives in the Providers app (top-level app in the dock, not a section inside Settings). The line in docs/taos-agent-manual.md that said "Settings → Providers → Add Provider" was wrong; just fixed in 56602cb.

Adding your Ollama endpoint via the UI:

1. Open the Providers app from the dock (icon next to Models / Cluster).
2. Click + Add Provider → pick ollama from the list.
3. URL = http://ollama1.mydomain:11434 (no trailing slash, no /v1).
4. Save. Click Test to verify reachability.
5. Once connected, the Models app will list every model your Ollama instance is serving. Pin one as your default and Agents → Add will pick it up.

About your manual YAML edit:

The providers: key you added under data/config.yaml won't be read — providers are persisted in a different store and configured through /api/providers. The Settings → Advanced 404 you hit (PUT /api/settings/config 404) is a real gap on our side; that screen shouldn't have a Providers-shaped form at all. Filed as #314.

About a setup wizard:

You're right, there isn't one yet — first-launch wizard is on the roadmap but not built. A model has to be configured through the Providers app before Agents can deploy. Tracking that gap.

External LiteLLM:

taOS bundles its own LiteLLM proxy that's automatically configured from your Providers entries — that's the path of least friction. Pointing taOS at a separate LiteLLM you already run elsewhere isn't supported as a first-class option today; it'd require pointing the agents directly at your LiteLLM URL and skipping the Providers layer entirely. Possible but undocumented. If your existing LiteLLM has providers configured already, the simpler path is to add those same providers to taOS's Providers app — taOS's bundled LiteLLM will route through them and you keep one source of truth in the UI.

Thanks for the detailed report — it's the kind of feedback that closes onboarding gaps fast.

[johny-mnemonic]

@jaylfc Thanks for help. I have discovered the Providers app basically at the same time you replied here 😃

In the end I had to destroy my Ubuntu 25.04 VM and start fresh as I was unable to make the iGPU passthrough working.
When I started fresh with Ubuntu 25.10 I succeeded and TAOS server-install succeeded as well, so you have another tested distro😉

Unfortunately seems like I will need help with few other things.

1. Same as you discuss in the Fresh install test — clean Orange Pi 5 Plus #2 I can see the installer finds the iGPU and reports everything is OK, but it does not even try to install llama.cpp and in the TAOS store, there is only rkllama, but no standard llama.cpp, so I am not sure how to install it.

2. When I use "Providers" app I can easily add Ollama endpoint, but when trying to add my LightLLM or llama.cpp (both of which have API keys) there seems to be no way to add them as local definition does not have API key field and the cloud providers does not allow to specify the API endpoint (it is hardcoded to api.openai.com in case of OpenAI).
   Would it be possible to change the cloud provider API fields to editable, with the default URL prefilled, instead of greyed non-editable ones?

3. Quick start guide says to start with SmolAgents framework, but seems like only OpenClaw is supported now. Or is it possible to use other frameworks , but they are just not tested yet and hence marked as Alpha?

4. Even though I have added the Ollama provider successfully and the "Providers" app shows all the available models Ollama offers, when I try to deploy any Agent the model selection is blank, saying "No models available".

Also I have noticed you have support for EXO, but that does not work with AMD ROCm and 2 out of 3 of my GPUs are AMD, so I am planning to go llama.cpp RPC route instead, where it should work. That should work with TAOS natively as any other llama.cpp endpoint, right? It will just lack the knowledge it talks to distributed LLM provider.

[jaylfc]

@johny-mnemonic Glad 25.10 worked. Tackling these in order.

llama.cpp not in the Store

It is in the catalog (app-catalog/services/llama-cpp/manifest.yaml), tagged for x86-vulkan-* and cpu-only tiers, so it should show up. Two likely reasons it isn't:

1. Stale catalog. The resolver and new service entries landed in PR feat(catalog): manifest dependency resolver — schema, resolver, recursive install dispatcher #325 a few days ago. If you installed before that you don't have them yet. git pull origin master in your install dir then sudo systemctl restart tinyagentos.service should bring them in.
2. The Store's "compatible with my device" filter is hiding it. Toggle that off and see if it appears.

If neither helps, I need:

• exact iGPU vendor and model (lspci | grep -E "VGA|3D")
• the GPU detection lines from your install-server.sh run (the intel gpu: or amdgpu: line)
• the hardware section from System Info, or curl -s -u user:pass http://localhost:6969/api/system | jq '.hardware'

That'll show which tier the resolver's placed you in and whether it matches what llama-cpp expects.

Providers app

You're right on both, both are real holes. Filed:

• feat(providers): support API key on local providers (llama.cpp, LiteLLM with --api-key) #349 for the API key field on local providers (llama.cpp, LiteLLM with --api-key). The form intentionally hides the input for the local category, which made sense back when "local" meant only Ollama.
• feat(providers): add OpenAI-compatible custom endpoint (editable base URL + key) #350 for an OpenAI-compatible cloud provider type with editable base URL plus key, basically the LiteLLM pattern. The current OpenAI/Anthropic URLs are hardcoded read-only chips, no way to point them somewhere else.

Workaround until both ship: put auth-required endpoints behind your existing LiteLLM and point taOS at LiteLLM as a single Ollama-shaped provider (LiteLLM exposes an Ollama-compatible surface). Not great but unblocks you.

llama.cpp RPC

Should work as a regular llama.cpp endpoint. RPC distributes layers across GPUs but exposes the same HTTP server (/completion, /v1/chat/completions) as single-node llama.cpp. taOS sees one endpoint and doesn't need to know it's distributed. Model layout, GPU mapping, node coordination all live inside the RPC server config. taOS just talks to the head node.

Caveats: the API key thing from above will still block you if your RPC server has auth, taOS won't see the underlying GPU topology (one provider, one model list as far as it's concerned), and yes EXO doesn't work with ROCm so RPC is the right path.

Want a more direct loop?

Most of taOS so far has been built around the hardware I run, Orange Pi 5 Plus with the RK3588 NPU, single node, no auth in front of inference. Your setup is basically the opposite end of the spectrum and it surfaces gaps I'd otherwise hit way later when other people with similar hardware showed up.

I want to spin up a dedicated discussion thread for your stack. Not a debug ticket, just an ongoing space where you can post whenever, raise things, share ideas, ask questions, complain about something half-broken, whatever. I'd treat you as the canonical voice for the multi-vendor and distributed-inference path. First person testing this stuff for real outside my own bubble.

How that'd actually work:

• you post into it freely, no need to file a formal issue first
• I cross-link PRs and issues touching your stack so you can weigh in before they merge
• when you tell me what's blocking you next, that genuinely shifts what I prioritise
• I pin the thread so anyone else arriving with similar hardware can find your journey

Say no and I'll keep replying here piecemeal, totally fine. But the offer's there if you want it. Either way the multi-vendor story is exactly where taOS needs to grow and you're closer to it than anyone else who's tried it so far.

If you're up for the thread, I'll spin it up and link from here.

[johny-mnemonic]

I don't see any kind of filtering in the Store, so no clue how/where to disable "compatible with my device" filter.

Here are the outputs you requested:

# lspci | grep -E "VGA|3D"
01:00.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

[server-install] intel gpu: present (Vulkan via Mesa, no separate driver install needed on most distros)

The curl command failed on authentication so I used the URL in browser:

"hardware": {
    "cpu": {
      "arch": "x86_64",
      "model": "Intel(R) N100",
      "cores": 2,
      "soc": ""
    },
    "ram_mb": 3382,
    "npu": {
      "type": "none",
      "device": "",
      "tops": 0,
      "cores": 0
    },
    "gpu": {
      "type": "none",
      "model": "",
      "vram_mb": 0,
      "vulkan": false,
      "cuda": false,
      "rocm": false
    },
    "disk": {
      "total_gb": 9,
      "free_gb": 4,
      "type": "ssd"
    },
    "os": {
      "distro": "ubuntu",
      "version": "25.10",
      "kernel": "6.17.0-23-generic"
    },
    "profile_id": "x86-cpu-3gb"
  }

Looks like the GPU detection failed and I think this output from lspci might be actually more useful:

01:00.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-N [UHD Graphics] [8086:46d1]

I do have also Ollama on one of my Windows gaming PCs, so I can use that till I would be able to add providers with API, but so far models from Ollama are not visible as I mentioned in previous comment.
Any help with that?

This discussion thread is fine by me. Thanks for asking AI agent😉

[jaylfc]

@johny-mnemonic Spotted, fair cop 🙂. Quick honesty note: I read every comment myself and the technical calls (what to fix, what to file, what to push back on) are mine. I delegate the typing on longer replies because I'm one person and there's a lot going on. The thinking and direction are still me; if anything in a reply ever feels off or like it's not actually engaging with what you said, call it out and I'll re-do it directly.

Onto the new findings, all genuinely useful.

Vulkan detection bug

This is a real bug, not a config thing on your end. The bash install-script correctly detected your iGPU (intel gpu: present (Vulkan via Mesa)), but the Python runtime hardware probe at tinyagentos/hardware.py:437 only checks Vulkan if the vulkaninfo binary is on PATH. Ubuntu 25.10 doesn't ship it by default. So your iGPU goes undetected at runtime → you're placed in the x86-cpu-3gb tier instead of x86-vulkan-* → the resolver hides most of the catalog (including llama-cpp) from the Store.

Workaround right now: sudo apt install vulkan-tools then restart taOS. Your /api/system should flip to vulkan: true and the Store should populate.

Real fix tracked in #354 — install-server.sh should pull vulkan-tools (and mesa-vulkan-drivers if missing) when it sees an Intel iGPU, plus a runtime fallback that infers Vulkan from /sys/class/drm/ when vulkaninfo is absent.

One thing to flag: with 3.4GB RAM you'll still be tight for most modern LLMs — most variants in the catalog have 4GB+ RAM floors. Once Vulkan's detected you'll see more options but the Q4-quantised small models (Qwen3 1.7B, SmolLM2, etc.) are realistically what you'll be running on this box. Bigger stuff will need to live on your Ollama Windows machine or eventually the AMD RPC cluster.

Store filter UX

The "compatible with my device" filter exists but the UI surface is the Show/Hide N models that won't run on the selected devices toggle at the bottom of each section. It only renders when count > 0. If your visible area is mostly empty (which it is right now, given the GPU detection issue above), the toggle is at the top with no surrounding cards and is easy to miss entirely. Filed as #355 — when the compatible list is empty, the toggle should be more prominent and labelled differently ("Nothing compatible with your device — show everything anyway?").

Ollama models invisible in agent picker

Real bug, filed as #356. The Providers app has three categories — local, network, cloud — but the agent's model picker only surfaces three tabs — local, worker, cloud. A remote Ollama on your Windows box almost certainly lands in the network category, which the picker doesn't show. So the models exist, are listed in Providers, but the picker can't reach them.

Workaround: if you can run Ollama directly on the same machine as taOS (instead of a remote Windows box), the models should show under local. Not ideal — the right fix is for the agent picker to handle network-category providers properly.

Dedicated thread

Spun up: #357. I've seeded it with a snapshot of your hardware/setup and links to every issue from this conversation, so you've got a single home that's already populated with what we're working through. From here forward, that's your thread — drop into it freely.

[johny-mnemonic]

No probs. Just that the last part of your message was so obviously AI generated I couldn't resist commenting it 😇

I have installed the vulkainfo and it correclty shows my GPU, but restarting taOS haven't changed the profile or the "vulkan": false.
Maybe reinstalling will fix it, not sure.

I am planning to run only tiny embedding models for memory on the taOS server and I have another 8+GB of free memory on my Proxmox, so I can easily increase it. I just started small to see how it will behave. If you have recommendation for good embedding model for QMD that can fit into 2-5GB, I am all ears. That should be good fit for this iGPU.

Models for agents will run on much faster GPUs I have in my home network (like two gaming PCs)...

I have found the filter in the Store. It is visible only in "Models" section.
Somehow the llama.cpp appeared in the "LLM Runtime" section so I installed it and wanted to install some model, but all I tried said: needs XY MB of disk, device has 0 MB free.
Maybe it correlates with this error from the service log:

May 07 15:03:18 taos python[6355]: disk quota monitor failed to initialise — disk routes will still work

I see another error in the service log, right below this one:

May 07 15:03:18 taos python[6355]: RuntimeError: No container backend configured. Call set_backend() first.

Not sure if it is an issue.
Also seems I will need to fix the TZ issue as it is 2 hour off...

We can move to #357 anytime. Best maybe once we tackle the current issue.

[jaylfc]

Haha fair play mate. I'm on the road today so I'm delegating a lot to claude and I'm just double checking and testing what I can etc, I'll have it look into your issues in a mo. Perfect use case for taOS, having your gaming PC's as workers, exactly what I do. I'm actually running full sweeps of benchmarks right now for taOSmd using different size models so I'll let you know how it goes! thanks again!

[jaylfc]

Fair shot, that closing was painted on a bit thick 😄

Vulkan still false after installing vulkan-tools

That's a separate cache bug. taOS writes data/hardware.json at first boot and reads it back on every restart, so once it's decided "no Vulkan" it won't re-probe just from a service restart. Workaround: sudo rm /home/$USER/tinyagentos/data/hardware.json and restart — next boot will re-probe and you should flip to vulkan: true and an x86-vulkan-3gb tier.

The proper fix is either always re-probe on startup or add a "Refresh hardware" button in Settings. Filed as #366.

Embedding model — let taOSmd decide

Worth zooming out here, because the way you're thinking about it ("pick the right embedding model for my hardware") is exactly what we're trying to take off the user's hands.

The memory layer behind taOS is taOSmd. It's not a separate thing you opt into — every agent you deploy is preconfigured with it. Embedding, retrieval, ranking, query expansion are all handled inside that layer with model defaults that have been benchmarked end-to-end (currently 97% Judge accuracy on LongMemEval-S with the default stack). You don't pick the embedding model per agent; agents inherit whatever taOSmd is configured to use, and that picks based on what hardware is available.

For your N100 right now: once Vulkan flips on and you're in an x86-vulkan-* tier, taOSmd will already steer you toward something like bge-m3 (~2.3GB) or bge-large-en-v1.5 (~1.3GB) for the embedder slot. If RAM stays tight it'll fall back to smaller models automatically.

Where this is going, and the part that's relevant to your multi-machine setup: the memory system is designed to be moved around the cluster — embeddings on one node, ranking on another, archive on a third — so you can put the heavier stuff on your gaming PCs (with their real GPUs) while the controller-side store stays light. That's not all wired up yet, but it's the direction. Your stack is exactly the cluster shape that work needs.

Bottom line: install bge-m3 from the Store if you want a manual pick, but the proper play is letting taOSmd's defaults handle it once your hardware tier is right.

Store filter only in "Models" section

Real UX gap. The compatibility filter should apply across Models, LLM Runtime, MCP, Services — anywhere the catalog lists installable things. Today the filter being absent in LLM Runtime is exactly why you ended up trying to install llama-cpp despite the resolver thinking it was incompatible. Filed as #367.

"device has 0 MB free" + disk quota monitor init failure

Same bug. The disk quota subsystem failed to initialise (per your log: disk quota monitor failed to initialise — disk routes will still work) and despite the message claiming routes still work, the install dispatcher reads from the same broken accounting and gets back 0. The "still work" assertion is wrong. Filed as #368 — and fixing the message so it's accurate.

"No container backend configured. Call set_backend() first."

Probably the underlying reason your Fedora-worker showed up with empty hardware too. The container backend (Incus or Docker) isn't installed or detected on your Ubuntu host, so anything that tries to spin up a worker container raises this. Two questions: do you have Incus or Docker installed on the Ubuntu box, and any preference between them? taOS supports both; auto-detect should pick whichever's present. If neither, the install script should be installing Incus by default (lighter than Docker on a home server). Filed as #369.

TZ off by 2 hours

Almost certainly not a taOS bug — taOS reads system time. Check timedatectl and sudo timedatectl set-timezone Europe/Prague (or wherever you're in). If it's still off after that, ping me and we'll dig.

Bigger picture

Almost everything you're hitting falls into the same bucket: install-time gaps where taOS detects something but doesn't fully wire it up, or relies on a host-side dependency we should be bundling. Vulkan-tools missing on Ubuntu, container backend not auto-detected, hardware cache not refreshing, etc. We're going to do an install-hardening pass — the bar is "install script finishes, taOS works fully without you apt installing anything else." Filed the umbrella as #370 with your stack as one of the named test platforms.

Thread move

Sounds good — once Vulkan's unblocked and the disk quota / container backend stuff is filed (already done above), I'll seed #357 with the open threads and we can keep using it from there.

[johny-mnemonic]

Yep, that helped. Now it says: "profile_id": "x86-vulkan-3gb".

So for running the embedding model directly on the taOS I don't have to do anything and taOSmd will run the llama.cpp with the best model itself? How do I at least check it is the case? So far I don't see llama.cpp running even though it says "Installed" in the Store.

I haven't installed docker or any other container backend. It is clean OS install just for taOS.
I don't remember seeing any such installation during the server-install process and both incus or docker are not present.

Yep, TZ was my issue, not taOS. I just mentioned it to explain the timestamps that were off. Already fixed.

OK, will wait for the disk space fix and will continue testing then.

[johny-mnemonic]

@jaylfc After recent fixes I managed to get a bit further while trying to add first agent.
Now the issue is with the memory. It offers three models but whichever I choose it complains that Ollama is not reachable. Not sure why it is looking for Ollama as I did not install Ollama but llama.cpp...

I tried to look into Store, and that now behaves a bit weird. I do see a lot of stuff as installed even though I haven't install them and when I click the "Uninstall" button it says it is not installed. For example Ollama and lot of other stuff (mostly models, but even Open WebUI as well) are like that, so I can't install Ollama even if I wanted to have it running on taOS (which I don't).

Also it says rkllama.cpp is installed, but that makes no sense as I don't have Rockchip CPU. Also in providers there are at least two rkllama providers registered which even do not have delete button, so I can't get rid of them.
And even though I have installed llama.cpp I don't see it running or installed.

[jaylfc]

Thanks for the detailed re-test — that's a few separate things tangled together. Let me work through them.

Local provider models missing from picker

Confirmed bug. The picker's classifier had only two lanes (cloud + worker-attached) and your locally-configured llama.cpp / ollama fell through the gap. Just merged a fix in #441 — adds a "controller" lane so non-cloud, non-worker providers surface in the picker's local tab alongside catalog-installed models. Pull master + restart and your llama.cpp models should appear in the agent wizard.

Memory step says "Ollama not reachable"

Separate bug. The memory wizard defaults to ollama as the embedder backend and doesn't yet detect that you've installed llama.cpp instead. Filing as a follow-up — fix coming in a separate PR. In the meantime, if you want to unblock yourself: install nomic-embed-text-v1.5 or bge-m3 from the Store and point the memory step at it.

Store shows things installed that aren't (Ollama, Open WebUI etc.) + can't uninstall

Side effect of a recent registry-fallback fix (#429). That change made the catalog surface anything in installed.json as "installed" so backend-installed models would show up — but stale entries are leaking through too. Need a consistency check between the registry and what's actually on disk. Filing as a follow-up.

rk-llama.cpp shows installed on non-rockchip + duplicate rkllama providers with no delete

auto_register_from_manifest doesn't gate on hardware compat (so rockchip-only services register on x86) and the providers list isn't deduping. Both filed and queued.

llama.cpp doesn't show running

Likely fixed by #441 once you pull master. If not, screenshot what you see after the pull and we'll dig.

---

Jay is asleep right now so this reply was written by Claude Opus 4.7.

[jaylfc]

Re-tested all five things you reported. Each has a shipped fix on master now:

Issue	PR
Local provider models not in picker (your #356 update)	#441 adds a "controller" lane in the picker so non-cloud, non-worker providers (your local llama.cpp) surface alongside catalog models
rk-llama.cpp showing as installed on non-rockchip hardware	#442 auto-register now gates on hardware tier compatibility before adding entries to config.backends
Store showing things as installed that you never installed (Ollama, Open WebUI, etc), Uninstall says "not installed"	#443 the installed flag now reflects state. Stale registry entries (services whose backend is unreachable) drop out of the installed list and become re-installable. Uninstall route also accepts the joined view so it can clean stale breadcrumbs
Memory step says "Ollama is not reachable" when llama.cpp is the only backend	#444 memory wizard now goes through the catalog resolver and picks whichever backend is actually available (rkllama, llama-cpp, sentence-transformers, ollama if installed). No more hardcoded Ollama dependency
Two rkllama provider entries with no delete button	#445 startup sweeps config.backends for duplicates (by name and by (type, url)); Edit/Delete buttons now show on stopped local providers, only hidden during active starting/stopping transitions

"llama.cpp doesn't show as running" should fall out of #441 (its models will surface in the picker once master propagates). If it still doesn't show after the pull, screenshot the Activity and Providers panels and we'll dig.

Pull master + restart. Curious how the Memory step goes for you specifically since the resolver path is new, let me know if it picks the right backend.

[jaylfc]

Vulkan tier looks right.

Embedder

You don't install anything yourself. taOSmd self-configures when you deploy an agent: pick a framework (OpenClaw or Hermes), the wizard's memory step asks which device runs the memory layer and what tier (Lite/Standard/Heavy), and the embedder + reranker + runtime get installed and started for you. After deploy, the Activity app (left sidebar) shows the loaded models with their host/port — that's how you confirm it's actually serving.

If you haven't deployed an agent yet, nothing memory-side will be running yet — that's by design. Standalone memory-without-an-agent isn't a first-class flow today; it's coming as part of the resource-scheduling work where the memory pipeline can move around the cluster independently of agents.

Container backend

Nothing got installed because that part of install-server.sh landed after your install ran. Re-running the script today installs Incus and inits the storage pool automatically. Without re-running, sudo apt install incus && sudo incus admin init --auto gets you there. Memory-only flows don't need it; agent deploys do.

Since the last comment

Postgres is now part of the install so per-agent virtual keys actually work (no more DB not connected warning). openai-compatible providers show up under Cloud in the deploy wizard. Hardware re-probes on every startup. Store compat filter applies across Models, LLM Runtime, MCP, etc. OpenClaw and Hermes both get taOSmd's librarian rules injected into AGENTS.md at deploy.

One thing in flight that's relevant for you: when you wire up the gaming PCs as workers, the install-targets endpoint now matches incus remote ↔ worker by URL host, so a name mismatch between the remote name and the worker registration name doesn't show up as "unknown hardware" anymore.

Will move ongoing chat to #357 once an agent's deployed and the memory layer's serving.