feat(ollama): one container, several models, each its own priced app

[rickstaa]

Every example in the repo registers exactly once, so nothing showed the shape people actually deploy an LLM server in: one process, many models. A builder asking "can one deployment serve several models, billed separately?" had no answer here, and would probably guess wrong — a runner carries exactly one mode and one price_info, and discovery keys on app, so it necessarily means several registrations under distinct app ids.

Ollama is the stock upstream image with no Livepeer code. A registrar sidecar asks it what it has (GET /api/tags) and registers one app per model, all pointing at the same URL. That is what wrapping software you did not write looks like, and it sits between the two registration modes the repo already shows: dynamic registration of a container that has no idea Livepeer exists.

Which models exist is discovered; what they cost is configured. ollama pull something and it appears on the network. Prices are operator policy, so they live in PRICES.

The app id is the model name, verbatim

llama3.2:1b registers as ollama/llama3.2:1b. go-livepeer only requires an app id be non-empty and trimmed, so there is no reason to slug it, and keeping it exact makes the mapping reversible in both directions.

That matters because the first draft did slug it — lowercasing and turning : into - — and then used metadata to carry back the string it had just destroyed. That is a denormalization repairing self-inflicted lossiness, not data the network was missing, and it bent the rule #61 writes down. Removing the slug removed the metadata field, a JSON parse, its fallback branch, and the helper holding them. Nothing in this example needs metadata, which keeps the repo's rule intact rather than breaking it in the first example that followed.

Capacity has to be sized by hand

Each registration carries its own counter and the orchestrator cannot know they share a GPU (go-livepeer#4015), so the sum is derived from OLLAMA_NUM_PARALLEL. capacity: 0 is not expressible — the orchestrator coerces it to 1 — so with more models than parallel slots the total unavoidably overshoots. The registrar warns loudly rather than quietly advertising capacity the GPU does not have. The README carries an "Improvements this example is waiting on" section pointing at the same issue.

Listing is free. GET /v1/models is answered from /discovery, a plain GET with no session and no payment, so it reports what the network offers rather than what one container holds. Contrast vllm, where forwarding that GET costs a session.

Verified end to end (RTX 3090, offchain)

The test box had four models in its Ollama volume, two of them unrelated leftovers — which made verification better than planned.

Registration and discovery, one app per discovered model, names exact:

ollama/qwen2.5:0.5b
ollama/llama3.2:1b
ollama/gemma4-coder:12b
ollama/hf.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF:Q4_K_M

That last one is the proof the exact-id approach holds: capitals, a colon, and several slashes, round-tripping through registration, discovery, exact-match filtering, and forwarding. A generation against it returned HTTP 200.

The overcommit warning fired, because four models against OLLAMA_NUM_PARALLEL=2 cannot each hold a slot:

WARNING advertising 4 slots across 4 models but the container runs 2 at once:
raise OLLAMA_NUM_PARALLEL to at least the model count, or pull fewer models.

A real bug caught during verification — the first draft did that division silently.

Generation works buffered and streaming, each --model reaching the right backend. Capacity refusal: three concurrent requests against a capacity: 1 model gave one 200 and two JSON 503s.

Notes

Deliberately not added to images.yml: the only Dockerfile here builds the registrar, not a runner, and the CI convention publishes runner-example-<name>. Publishing a sidecar under that name would be misleading.

This adds a dimension the axis table does not have — capabilities per process — while its four axis values are all already covered. Worth deciding whether the table grows a column or this is filed as the exception.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new Ollama example demonstrating a realistic LLM deployment shape: one upstream container serving multiple models, with a registrar sidecar dynamically registering one Live Runner app per discovered model (per-model pricing) and a local OpenAI-compatible gateway that maps model → app id and forwards calls through the orchestrator.

Changes:

• Add ollama/ example: registrar sidecar (register_runner per model), OpenAI-compatible gateway, and stock OpenAI client.
• Add Docker/Compose setup for offchain + on-chain overlay, plus documentation for running and capacity sizing.
• Update repo-level README to include the new Ollama example in the comparison table and capability lists.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
README.md	Adds ollama to the examples matrix and capability lists.
ollama/registrar.py	Sidecar that discovers models via Ollama /api/tags and registers one app per model with pricing/capacity.
ollama/README.md	Documents the multi-model design, capacity sizing rationale, and run instructions (offchain/on-chain).
ollama/pyproject.toml	Adds Python project metadata/deps for the example tooling.
ollama/gateway.py	Local OpenAI-compatible gateway: lists models from discovery and forwards requests to selected runners.
ollama/Dockerfile	Builds the registrar sidecar image.
ollama/compose.yml	Offchain compose: orchestrator + Ollama + puller + registrar.
ollama/compose.onchain.yml	On-chain overlay: adds signer and on-chain orchestrator wiring.
ollama/client.py	Stock OpenAI client example for buffered and streaming calls.
ollama/.env.example	Example environment configuration for offchain/on-chain runs.

Suppressed comments (1)

ollama/.env.example:39

• These comments describe the runner price as coming from runners.json, but in this example pricing is dynamic (registrar PRICES/--price). The MAX_PRICE_PER_UNIT guidance should reference the metered USD/hour price configured via PRICES instead of runners.json.

# The runner's price lives in runners.json (static runner): USD per hour,
# converted to wei via the price feed and metered per second.
# Signer's max-price cap (payer side) is per billing unit, here one second, so
# it must exceed the runners.json price / 3600 (0.000111USD is ~0.40 USD/hour).
MAX_PRICE_PER_UNIT=0.000111USD

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

ollama/.env.example:8

• These vLLM-specific env vars/comments appear to be copy/paste leftovers and are misleading for the Ollama example (models are discovered from Ollama tags, not configured via a single VLLM_MODEL). Remove them so the .env template only documents Ollama-related config.

# Model vLLM serves (must match the client's --model).
VLLM_MODEL=Qwen/Qwen2.5-0.5B-Instruct
# Models to pull, space separated. Which models exist is config; which models get
# advertised is discovered from Ollama by the registrar.
MODELS=qwen2.5:0.5b llama3.2:1b

ollama/.env.example:39

• This section references pricing coming from runners.json, but this example uses dynamic registration where per-model pricing comes from PRICES in the registrar. Update this to avoid confusing operators about where pricing is set.

# The runner's price lives in runners.json (static runner): USD per hour,
# converted to wei via the price feed and metered per second.
# Signer's max-price cap (payer side) is per billing unit, here one second, so
# it must exceed the runners.json price / 3600 (0.000111USD is ~0.40 USD/hour).
MAX_PRICE_PER_UNIT=0.000111USD

ollama/registrar.py:53

• _parse_prices() will raise a ValueError with a stack trace on malformed input (e.g. a non-float). Since this is operator-provided config, it should fail fast with a clear error message indicating which entry is invalid.

    for item in raw.split(","):
        if "=" in item:
            name, _, value = item.partition("=")
            prices[name.strip()] = float(value)

ollama/registrar.py:89

• When Ollama isn't ready yet, /api/tags can return non-200 responses or non-JSON bodies. resp.json() can raise exceptions that are not aiohttp.ClientError, which would abort the retry loop immediately. Consider raising for status and also retrying on JSON decode errors.

            async with session.get(f"{base_url.rstrip('/')}/api/tags") as resp:
                data = await resp.json()

ollama/registrar.py:109

• If --parallel/OLLAMA_NUM_PARALLEL is set to 0 (or negative), per_model will still be forced to 1 and the registrar will advertise capacity the container cannot run. Validate --parallel up front and exit with an explicit error if it's < 1.

    per_model = max(1, args.parallel // len(models))

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

ollama/gateway.py:156

• _forward_or_error() only JSON-wraps LivepeerHTTPError, but other expected failures (discovery/selection errors, and the web.HTTPBadRequest/web.HTTPNotFound raised by _forward()) will return aiohttp’s default non-JSON responses. OpenAI clients typically expect JSON error bodies, so these should be normalized here too.

        try:
            return await _forward(request)
        except LivepeerHTTPError as exc:
            return web.json_response(
                {"error": {"message": str(exc), "type": "livepeer_error"}},

ollama/registrar.py:54

• _parse_prices() will raise a ValueError traceback on malformed PRICES entries (e.g., "model=", non-numeric values, or stray whitespace), which makes misconfiguration hard to diagnose. It’s better to validate each entry and fail with a clear error message (or explicitly skip empty items) so the registrar doesn’t crash obscurely.

    for item in raw.split(","):
        if "=" in item:
            name, _, value = item.partition("=")
            prices[name.strip()] = float(value)
    return prices

ollama/registrar.py:90

• _installed_models() doesn’t set a request timeout or check HTTP status before parsing JSON. If /api/tags hangs, returns non-JSON, or returns a non-2xx response, the registrar can block indefinitely or crash, rather than retrying cleanly within the 5-minute wait window.

            async with session.get(f"{base_url.rstrip('/')}/api/tags") as resp:
                data = await resp.json()
            models = [m["name"] for m in data.get("models", []) if m.get("name")]
            if models:
                return sorted(models)

ollama/gateway.py:112

• In _forward(), invalid JSON bodies will currently bubble up as an unhandled exception (producing a 500/HTML error), and an empty runner_selector result will raise IndexError on cursor.candidates[0]. Both cases break OpenAI-client compatibility and should return a structured 4xx instead.

        payload = await request.json() if request.can_read_body else {}
        runner_path = request.path  # e.g. /v1/chat/completions
        model = str(payload.get("model", "")).strip()
        if not model:
            raise web.HTTPBadRequest(text="request must name a model")

ollama/gateway.py:32

• _forward_or_error() should also handle non-HTTP Livepeer selection/discovery failures (e.g., NoRunnerAvailableError / NoOrchestratorAvailableError), but gateway.py only imports LivepeerHTTPError. Importing LivepeerGatewayError allows returning JSON errors instead of aiohttp’s default HTML 500.

This issue also appears on line 152 of the same file.

from livepeer_gateway.errors import LivepeerHTTPError

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (8)

ollama/gateway.py:144

• Same as the streaming branch: forward the inbound HTTP method to call_runner() so the gateway doesn’t accidentally turn non-POST requests into POSTs.

        result = await call_runner(  # Livepeer: 3
            runner=runner,  # discovery metadata tells call_runner the price unit
            runner_url=runner_url,
            payload=payload,
            signer_url=signer_url,

ollama/registrar.py:99

• --parallel/OLLAMA_NUM_PARALLEL can be set to 0 (or negative), but the capacity math later forces per-model capacity to at least 1, causing unintended over-advertising. Fail fast with a clear error when parallel < 1.

    args = _parse_args()
    prices = _parse_prices(args.prices)

ollama/registrar.py:108

• This comment claims the advertised total "matches the hardware". With the current floor division (parallel // len(models)), the advertised total can be less than --parallel when it isn’t evenly divisible (e.g., parallel=3, models=2 => advertised=2). Either adjust the wording here (and in the README) or adjust the capacity allocation logic.

    # Split the container's real concurrency across the models it serves, so the
    # advertised total matches the hardware instead of multiplying by model count.
    # capacity 0 is not expressible (the orchestrator coerces it to 1), so with more
    # models than parallel slots the total unavoidably overshoots -- say so loudly
    # rather than quietly advertising capacity the GPU does not have.

ollama/README.md:34

• The README says the sum of advertised capacities "equals" what the hardware can do, but registrar.py uses floor division, so the sum may be lower when OLLAMA_NUM_PARALLEL isn’t divisible by the model count. Update this sentence to match the actual behavior (or update the registrar to distribute the remainder).

`OLLAMA_NUM_PARALLEL` says how many generations the container will really run at once. The registrar divides that across the models it registers, so the **sum** of the advertised capacities equals what the hardware can do.

ollama/gateway.py:112

• cursor.candidates[0] will raise IndexError when no runners match the requested model, causing a 500 instead of a readable OpenAI-style JSON error. Guard against empty candidate lists and return a 503/404 style JSON response.

        cursor = await runner_selector(  # Livepeer: 2
            discovery_url=args.discovery,  # omit if the signer does discovery itself
            app=_app_id(model),
        )
        runner = cursor.candidates[0]

ollama/registrar.py:54

• _parse_prices() can crash with an unhelpful ValueError on invalid PRICES values (e.g., empty value or non-float), and silently ignores malformed entries without '='. Since PRICES is operator-supplied policy, validate it and exit with a clear message so misconfiguration is obvious.

    for item in raw.split(","):
        if "=" in item:
            name, _, value = item.partition("=")
            prices[name.strip()] = float(value)
    return prices

ollama/registrar.py:90

• _installed_models() retries for up to 5 minutes, but each GET has no timeout and it doesn’t check HTTP status / JSON parse errors. A hung connection or a non-JSON error response can block longer than intended or crash the registrar; add a request timeout, gate on 200 responses, and treat JSON decode failures as retryable.

            async with session.get(f"{base_url.rstrip('/')}/api/tags") as resp:
                data = await resp.json()
            models = [m["name"] for m in data.get("models", []) if m.get("name")]
            if models:
                return sorted(models)

ollama/gateway.py:125

• The gateway registers a catch-all route for /v1/*, but call_runner() is invoked without passing through the original HTTP method. This can mis-forward non-POST requests (e.g., OPTIONS/HEAD) by defaulting to call_runner’s default method. Pass method=request.method for parity with vllm/gateway.py.

This issue also appears on line 140 of the same file.

            async with await call_runner(  # Livepeer: 3 (streaming)
                runner=runner,  # discovery metadata tells call_runner the price unit
                runner_url=runner_url,
                payload=payload,
                signer_url=signer_url,