Run your own models, on your own hardware — fully offline, no API key, no outside service.

---

The OpenScenes Agent is the local-inference companion to OpenScenes. It serves local GGUF models over HTTP, so your chats can run entirely on your own GPU or CPU. With the agent, you depend on no cloud provider at all: your models, your machine, your data, zero outside calls.

OpenScenes talks to it exactly the way it talks to any cloud provider — the agent is just one more backend you can pick per chat. It knows nothing about characters, personas, or styles; it consumes a ready-made messages payload and returns a reply.

The whole agent/ folder is relocatable — every path resolves relative to agent.py, so you can move it to another drive or machine and it boots without code edits.

---

Install

The agent needs Python 3.12 (the version llama-cpp-python==0.3.4 ships wheels for). Run the setup script from inside agent/:

cd agent
python setup.py

It creates a virtualenv, installs Flask, then installs llama-cpp-python. If an Nvidia GPU is detected, it offers the CUDA build for GPU-accelerated inference; otherwise it installs the CPU build. (Detection means a GPU driver is present — the CUDA wheel bundles its own runtime libraries.)

---

Run

python agent.py

Drop your .gguf files into agent/llm/ — the agent treats that folder as the source of truth. Each file is registered automatically (under its filename), and removing a file removes its model. Then, in OpenScenes, set the agent's address in Settings → Connections and register a model whose Model matches one the agent lists.

---

Configuration

setup.py creates config.ini from the committed config.ini.example template, so it's ready after install. The live file holds your API_KEY and is gitignored — never commit it. Edit it as needed (flat key=value):

HOST=127.0.0.1
PORT=8090
API_KEY=

• HOST — bind address. Keep 127.0.0.1 for single-machine use. Only set 0.0.0.0 to allow other hosts on your LAN after you set an API_KEY — otherwise you expose an unauthenticated inference server.
• PORT — TCP port.
• API_KEY — Bearer token required on every request when set. Empty means no authentication — safe only on 127.0.0.1.

Keep config.ini free of real secrets in anything you commit or share.

---

Wire contract

All requests require Authorization: Bearer <API_KEY> when an API key is configured.

POST /v1/complete — run a completion against a registered model.

{ "name": "model-name", "messages": [ ... ], "max_tokens": 1024, "temperature": 0.8, "top_p": 0.95 }
→ 200 { "content": "..." }

GET /v1/llms — list registered models (synced to the llm/ folder on each call). POST /v1/llms — register a model ({name, filename}) against a GGUF already in llm/. DELETE /v1/llms/<id> — deregister a model and delete its GGUF.

Errors come back as {"error": "..."} with an appropriate status code (400 malformed/unknown model, 401 bad key, 500 generation failure).