feat(ollama): one container, several models, each its own priced app

.github/workflows/images.yml

@@ -16,6 +16,7 @@ on:
       - "hello-world/**"
       - "echo/**"
       - "tiles/**"
+      - "realtime-transcription/**"
       - ".github/workflows/images.yml"
   pull_request:
     paths: *image_paths
@@ -34,7 +35,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        example: [hello-world, echo, tiles]
+        example: [hello-world, echo, tiles, realtime-transcription]
     steps:
       - uses: actions/checkout@v7
 

README.md

@@ -29,21 +29,23 @@ flowchart LR
 The orchestrator is a **transparent reverse proxy**: every endpoint you expose is passed through to your app unchanged, so you write an ordinary service and it runs on the network as-is. The transports supported today:
 
 - **HTTP** request/response — the common case. (`hello-world`, `tiles`, `api-proxy`)
-- **HTTP + SSE** — streamed / token responses. (`vllm`)
+- **HTTP + SSE** — streamed / token responses. (`vllm`, `ollama`)
 - **Trickle** — continuous realtime video in/out. (`echo`)
-- **WebSocket** — long-lived bidirectional sessions. (external: `scope`)
+- **WebSocket** — long-lived bidirectional sessions. (`realtime-transcription`)
 
 Need a schema that isn't here? [Open an issue](https://github.com/livepeer/runner-app-examples/issues).
 
 ## Examples
 
-| Example                        | Goal                                                                                  | Registration | Mode                               | Transport         | Pricing |
-| ------------------------------ | ------------------------------------------------------------------------------------- | ------------ | ---------------------------------- | ----------------- | ------- |
-| [`hello-world`](./hello-world) | The simplest app: one request, one response                                           | dynamic      | single-shot                        | HTTP (JSON)       | fixed   |
-| [`tiles`](./tiles)             | Capacity fan-out — one call per tile                                                  | dynamic      | single-shot                        | HTTP (base64 PNG) | fixed   |
-| [`api-proxy`](./api-proxy)     | Pass calls through to a hosted API — the operator holds the key, callers pay per call | static       | single-shot                        | HTTP (JPEG bytes) | fixed   |
-| [`echo`](./echo)               | Realtime video, transformed and echoed back                                           | dynamic      | persistent                         | trickle           | hour    |
-| [`vllm`](./vllm)               | Drop-in OpenAI API; the client stays unmodified                                       | static       | persistent (single-shot by nature) | HTTP + SSE        | hour    |
+| Example                                              | Goal                                                                                  | Registration | Mode        | Transport         | Pricing |
+| ---------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------ | ----------- | ----------------- | ------- |
+| [`hello-world`](./hello-world)                       | The simplest app: one request, one response                                           | dynamic      | single-shot | HTTP (JSON)       | fixed   |
+| [`tiles`](./tiles)                                   | Capacity fan-out — one call per tile                                                  | dynamic      | single-shot | HTTP (base64 PNG) | fixed   |
+| [`api-proxy`](./api-proxy)                           | Pass calls through to a hosted API — the operator holds the key, callers pay per call | static       | single-shot | HTTP (JPEG bytes) | fixed   |
+| [`echo`](./echo)                                     | Realtime video, transformed and echoed back                                           | dynamic      | persistent  | trickle           | hour    |
+| [`vllm`](./vllm)                                     | Drop-in OpenAI API; the client stays unmodified                                       | static       | single-shot | HTTP + SSE        | hour    |
+| [`realtime-transcription`](./realtime-transcription) | Audio up, transcripts back, on one socket                                             | dynamic      | persistent  | WebSocket         | hour    |
+| [`ollama`](./ollama)                                 | One container, several models, each its own priced app                                | dynamic      | single-shot | HTTP + SSE        | hour    |
 
 Start with `hello-world` (the smallest end-to-end path); the others each layer on one new idea. More will follow, including a full example that exercises every feature. Each is self-contained and runs **offchain** (free, no wallet); most also run **on-chain** (paid) — see each README.
 
@@ -53,9 +55,11 @@ This set stays **minimal and curated**: it covers each value of the axes above (
 
 How the app attaches to the orchestrator:
 
-- **Dynamic** — the app self-registers via the SDK (`register_runner`) and heartbeats; the orchestrator drops it when heartbeats stop. Best for apps that come and go. (`hello-world`, `echo`)
+- **Dynamic** — the app self-registers via the SDK (`register_runner`) and heartbeats; the orchestrator drops it when heartbeats stop. Best for apps that come and go. (`hello-world`, `echo`, `realtime-transcription`, `ollama`)
 - **Static** — the orchestrator is configured with the app's URL in a `runners.json` and health-polls it; the app needs no SDK. Best for fixed, long-running deployments. (`vllm`, `api-proxy`)
 
+Both forms also take an optional **`metadata`** string: up to 1 KB of app-controlled UTF-8, echoed back in `/discovery` and never read by the orchestrator. Use it to pass callers app-specific detail the protocol doesn't model, such as a context window or the languages a model handles; clients read it off the discovered runner, as `cursor.candidates[0].raw["metadata"]` after `runner_selector`, or `session.runner.raw["metadata"]` after `reserve_session`. Anything a caller **selects or pays differently for** belongs in the app id instead, which is the only key discovery can filter — none of the examples here need `metadata` for that reason.
+
 The arrow flips — dynamic, the app announces itself; static, the orchestrator is told about a passive app:
 
 ```mermaid
@@ -74,18 +78,15 @@ flowchart LR
 
 Chosen _at_ registration (above); **defaults to `persistent`** — set on both `register_runner(...)` and in `runners.json`. The examples set it explicitly.
 
-- **Persistent** — a held-open session the client reserves and releases, billed per second of wall-clock (or once, with fixed pricing). Best for realtime / streaming. (`echo`, `vllm`)
-- **Single-shot** — one request in, one response out; the orchestrator reserves a session per call and releases it when the response returns, so the client manages no session at all. Best for batch / request-response. (`hello-world`, `tiles`, `api-proxy`)
-
-> [!NOTE]
-> The `vllm` example is single-shot by nature but stays **persistent** for now: it meters per second across a reserved session, and true per-token billing is brokerage for the gateway/signer layer.
+- **Persistent** — a held-open session the client reserves and releases, billed per second of wall-clock (or once, with fixed pricing). Best for realtime / streaming. (`echo`, `realtime-transcription`)
+- **Single-shot** — one request in, one response out; the orchestrator reserves a session per call and releases it when the response returns, so the client manages no session at all. Best for batch / request-response. With metered pricing the call pays for as long as it runs, so the work need not be short. (`hello-world`, `tiles`, `api-proxy`, `vllm`, `ollama`)
 
 ## Calling your app
 
 The client side depends on the runner's mode:
 
-- **Single-shot** — **discover → call**: find the app via `runner_selector`, then one `call_runner`. The orchestrator reserves a session for the call and releases it when the response returns; on the paid path `call_runner` answers the 402 payment challenge inline. (`hello-world`, `tiles`, `api-proxy`)
-- **Persistent** — **discover → reserve → call → release**: reserve a session (`reserve_session`), call it — `call_runner`, streamed frames, or a WebSocket, depending on transport — then release it (`stop_runner_session`), which settles payment on-chain. (`echo`, `vllm`)
+- **Single-shot** — **discover → call**: find the app via `runner_selector`, then one `call_runner`. The orchestrator reserves a session for the call and releases it when the response returns; on the paid path `call_runner` answers the 402 payment challenge inline. (`hello-world`, `tiles`, `api-proxy`, `vllm`, `ollama`)
+- **Persistent** — **discover → reserve → call → release**: reserve a session (`reserve_session`), call it — `call_runner`, streamed frames, or a WebSocket, depending on transport — then release it (`stop_runner_session`), which settles payment on-chain. (`echo`, `realtime-transcription`)
 
 Each example's `client.py` shows its exact calls — grep `# Livepeer:` to find them.
 

ollama/.env.example

@@ -0,0 +1,37 @@
+# Copy to .env (gitignored) and fill in. Never commit secrets.
+# Keystore dirs: absolute paths OUTSIDE this repo, mounted read-only.
+
+# Models to pull, space separated. Which models exist is config; which models get
+# advertised is discovered from Ollama by the registrar.
+MODELS=qwen2.5:0.5b llama3.2:1b
+# Concurrent generations the container can really run. The registrar splits this
+# across the models, so the advertised total matches the hardware (go-livepeer#4015).
+OLLAMA_NUM_PARALLEL=2
+OLLAMA_KEEP_ALIVE=1h
+
+# Per-model price in USD/hour: operator policy, so it is config rather than
+# discovered. Models not listed here register free.
+PRICES=qwen2.5:0.5b=0.01,llama3.2:1b=0.02
+
+# --- On-chain (paid) only below; offchain ignores these. ---
+
+NETWORK=arbitrum-one-mainnet
+ETH_RPC_URL=https://arb1.arbitrum.io/rpc
+
+# Signer (payer): needs an on-chain deposit + reserve.
+SIGNER_KEYSTORE_DIR=/absolute/path/to/signer-keystore
+SIGNER_ETH_ACCT=0xYourSignerAddress
+SIGNER_ETH_PASSWORD=your-signer-keystore-password
+
+# Orchestrator operating key (split-key): needs ETH for gas to redeem tickets.
+ORCH_KEYSTORE_DIR=/absolute/path/to/operator-keystore
+ORCH_ETH_ACCT=0xYourOperatorAddress
+ORCH_ETH_PASSWORD=your-operator-keystore-password
+# Registered orch = ticket recipient (-ethOrchAddr); empty = use the operating key.
+ORCH_ONCHAIN_ADDR=0xYourRegisteredOrchestrator
+
+# The runner's price lives in runners.json (static runner): USD per hour,
+# converted to wei via the price feed and metered per second.
+# Signer's max-price cap (payer side) is per billing unit, here one second, so
+# it must exceed the runners.json price / 3600 (0.000111USD is ~0.40 USD/hour).
+MAX_PRICE_PER_UNIT=0.000111USD

ollama/Dockerfile

@@ -0,0 +1,20 @@
+# Registrar sidecar: the only Livepeer code in this example. Ollama itself is the
+# stock upstream image with nothing added to it.
+FROM python:3.12-slim
+
+# Flush stdout/stderr immediately so output isn't block-buffered in `docker logs`.
+ENV PYTHONUNBUFFERED=1
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git \
+    && rm -rf /var/lib/apt/lists/*
+
+# livepeer-gateway SDK isn't on PyPI yet; install from Git.
+RUN pip install --no-cache-dir \
+    aiohttp \
+    "livepeer-gateway @ git+https://github.com/livepeer/livepeer-python-gateway@ja/live-runner"
+
+WORKDIR /app
+COPY registrar.py ./
+
+CMD ["python", "registrar.py"]

ollama/README.md

@@ -0,0 +1,103 @@
+# Ollama app (one container, several priced models)
+
+One Ollama container serving several models, each registered as **its own Live Runner app** with its own price. This is the example about **multiple capabilities from one process**: every other example here registers exactly once. Ollama itself is the stock upstream image with no Livepeer code in it — a **registrar sidecar** does the registering, which is what wrapping software you did not write looks like.
+
+|              |                                                  |
+| ------------ | ------------------------------------------------ |
+| App ids      | `ollama/<model>`, one per pulled model           |
+| Runner mode  | single-shot (one session per call)               |
+| Registration | dynamic (a sidecar self-registers via the SDK)   |
+| Transport    | HTTP + SSE (OpenAI `/v1/chat/completions`)       |
+| Pricing      | hour (metered per second of the call), per model |
+| Port         | 11434 (Ollama), 8080 (gateway)                   |
+
+Runs on GPU or CPU — drop the `deploy` block in `compose.yml` for CPU-only, and expect it to be slow. Prerequisites (Docker, `uv`, the not-yet-released SDK) and the shared on-chain/payment setup are in the [repo README](../README.md).
+
+## How it's wired
+
+Three moving parts, and only one of them knows about Livepeer:
+
+- **Ollama** — the stock image. Never modified, never aware of anything.
+- **`registrar.py`** — a sidecar that asks Ollama what it has (`GET /api/tags`) and calls `register_runner` once per model, each pointing at the same Ollama URL.
+- **`gateway.py`** — a host-side OpenAI endpoint that maps the OpenAI `model` field onto an app id and forwards. Any OpenAI client works against it unchanged.
+
+**Which models exist is discovered; what they cost is configured.** The registrar never has a model list of its own: `ollama pull` something and it appears on the network at the next start. Prices are operator policy, so they come from `PRICES` in `.env`.
+
+**One model, one app id, one price.** A runner carries exactly one `mode` and one `price_info`, and discovery can only filter on `app`, so several capabilities means several registrations. `qwen2.5:0.5b` and `llama3.2:1b` are genuinely different products at different prices, and a caller picks between them the same way they pick between orchestrators.
+
+**The app id is the model name, verbatim.** `llama3.2:1b` registers as `ollama/llama3.2:1b` — go-livepeer only requires an app id be non-empty and trimmed, so there is no reason to slug it. Keeping it exact makes the mapping reversible in both directions, which is why nothing here needs a `metadata` field to restate the name: what you discover is what you send.
+
+**Listing models is free.** `GET /v1/models` is answered from `/discovery`, a plain GET with no session and no payment, so it reports what **the network** offers rather than what one container holds. Only the forward reserves a session — and because the runners are single-shot and metered, that call pays for as long as the generation runs.
+
+## Capacity, and why it is hand-sized
+
+`OLLAMA_NUM_PARALLEL` says how many generations the container will really run at once. The registrar divides that across the models it registers, so the **sum** of the advertised capacities equals what the hardware can do.
+
+That arithmetic is manual on purpose. Each registration carries its own capacity counter, and **the orchestrator does not know they share a GPU** — see [Improvements this example is waiting on](#improvements-this-example-is-waiting-on). Register two models at `capacity: 1` each on a box that can only run one generation, and the orchestrator will cheerfully admit two sessions; both callers pay and both get contention instead of the 503 that would have been correct.
+
+Both models stay resident under `OLLAMA_KEEP_ALIVE`, so a request never waits for a swap.
+
+## Run offchain (free)
+
+```sh
+docker compose up -d --build   # pulls the models, then registers one app per model
+curl -sk https://localhost:8935/discovery | jq '.[].runners[] | {app, capacity, price_info}'
+uv run gateway.py --discovery https://localhost:8935/discovery &   # OpenAI endpoint on :8080
+uv run client.py --model qwen2.5:0.5b --prompt "In one sentence, what is Livepeer?"
+uv run client.py --model llama3.2:1b --prompt "write a haiku about GPUs"
+kill %1; docker compose down    # stop the gateway, then the stack
+```
+
+Ask the network what it serves, with a stock OpenAI client:
+
+```sh
+curl -s http://localhost:8080/v1/models | jq '.data[].id'
+# "llama3.2:1b"
+# "qwen2.5:0.5b"
+```
+
+Add a model without touching any config:
+
+```sh
+docker compose exec ollama ollama pull gemma3:270m
+docker compose restart registrar     # picks it up from /api/tags
+```
+
+It registers at the fallback price, since `PRICES` does not mention it.
+
+## Run on-chain (paid)
+
+Layer `compose.onchain.yml` to add a remote signer and run the orchestrator on-chain. Needs an Ethereum RPC, a funded signer wallet (deposit + reserve), and an orchestrator wallet — see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README.
+
+```sh
+cp .env.example .env   # fill in RPC, network, keystore paths, accounts, prices
+docker compose -f compose.yml -f compose.onchain.yml up -d --build
+uv run gateway.py --signer http://localhost:7936 --discovery https://localhost:8935/discovery &
+uv run client.py --model llama3.2:1b --prompt "In one sentence, what is Livepeer?"
+kill %1; docker compose -f compose.yml -f compose.onchain.yml down
+```
+
+There is no `runners.json` here: registration is dynamic, so each model's price comes from the registrar's `PRICES`. Pick a costlier model and the caller pays more, which is the whole reason each model is its own app.
+
+## Improvements this example is waiting on
+
+Two gaps are tracked in [go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015), and both are visible here:
+
+- **Capacity is tracked per registration.** Nothing tells the orchestrator that these runners share one GPU, so the advertised total is whatever the registrations happen to add up to. This example sizes that sum by hand from `OLLAMA_NUM_PARALLEL`; a `pool` field on the heartbeat would let the orchestrator enforce it instead, and refuse the surplus with a 503.
+- **Only one GPU can be described.** `LiveRunnerGPU` is a single struct, so a two-card host advertises one card and half its VRAM.
+
+A third piece is needed for the version that tracks capacity **dynamically** rather than by arithmetic — flipping every registration's `status` off a shared semaphore as work starts and finishes. That works today in principle, but the SDK exposes neither a public setter for `status` nor a way to force an immediate heartbeat, so it would mean reaching into private attributes. Not something an example should teach.
+
+## Run without Docker
+
+Start an orchestrator built from go-livepeer `v0.9.0` or newer (see [Build from source](https://docs.livepeer.org/v1/orchestrators/guides/install-go-livepeer#build-from-source)) and an Ollama server, then the registrar, gateway, and client directly:
+
+```sh
+./livepeer -orchestrator -useLiveRunners -serviceAddr localhost:8935 -orchSecret abcdef -v 6
+ollama serve &
+ollama pull qwen2.5:0.5b
+uv run registrar.py --orchestrator https://localhost:8935 --orchSecret abcdef \
+  --ollama-url http://localhost:11434 &
+uv run gateway.py &
+uv run client.py --model qwen2.5:0.5b --prompt "Hello!"
+```