From 2a613e8f0b7549a0f6d39cc3017630c35a86a06c Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Fri, 14 Aug 2026 12:01:33 +0200 Subject: [PATCH 1/7] feat(api-proxy): offer two priced models instead of one The README claimed a second model was "more config, not code" without showing it. Run nginx.conf.template twice with a different MODEL, give each its own runners.json entry, app id, and price, and let the client pick with --app. FLUX.1 schnell is cheaper than SD3 medium, so the two prices differ for a reason a caller can see. Each entry keeps its own capacity: the work happens upstream, so the capabilities do not contend. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- README.md | 16 ++++++------ api-proxy/.env.example | 9 ++++--- api-proxy/README.md | 47 +++++++++++++++++++++-------------- api-proxy/client.py | 16 +++++++++--- api-proxy/compose.yml | 24 ++++++++++++++---- api-proxy/nginx.conf.template | 3 +++ api-proxy/runners.json | 13 ++++++++-- 7 files changed, 87 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 69c1555..30646c8 100644 --- a/README.md +++ b/README.md @@ -37,14 +37,14 @@ Need a schema that isn't here? [Open an issue](https://github.com/livepeer/runne ## Examples -| Example | Goal | Registration | Mode | Transport | Pricing | -| ---------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------ | ----------- | ----------------- | ------- | -| [`hello-world`](./hello-world) | The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed | -| [`tiles`](./tiles) | Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed | -| [`api-proxy`](./api-proxy) | Pass calls through to a hosted API — the operator holds the key, callers pay per call | static | single-shot | HTTP (JPEG bytes) | fixed | -| [`echo`](./echo) | Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour | -| [`vllm`](./vllm) | Drop-in OpenAI API; the client stays unmodified | static | single-shot | HTTP + SSE | hour | -| [`realtime-transcription`](./realtime-transcription) | Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour | +| Example | Goal | Registration | Mode | Transport | Pricing | +| ---------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------ | ----------- | ----------------- | ------- | +| [`hello-world`](./hello-world) | The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed | +| [`tiles`](./tiles) | Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed | +| [`api-proxy`](./api-proxy) | Pass calls through to hosted APIs — the operator holds the key, one capability per model | static | single-shot | HTTP (JPEG bytes) | fixed | +| [`echo`](./echo) | Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour | +| [`vllm`](./vllm) | Drop-in OpenAI API; the client stays unmodified | static | single-shot | HTTP + SSE | hour | +| [`realtime-transcription`](./realtime-transcription) | Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour | Start with `hello-world` (the smallest end-to-end path); the others each layer on one new idea. More will follow, including a full example that exercises every feature. Each is self-contained and runs **offchain** (free, no wallet); most also run **on-chain** (paid) — see each README. diff --git a/api-proxy/.env.example b/api-proxy/.env.example index 78e3d6a..e8173b8 100644 --- a/api-proxy/.env.example +++ b/api-proxy/.env.example @@ -22,9 +22,10 @@ ORCH_ETH_PASSWORD=your-operator-keystore-password # Registered orch = ticket recipient (-ethOrchAddr); empty = use the operating key. ORCH_ONCHAIN_ADDR=0xYourRegisteredOrchestrator -# The runner's price lives in runners.json (static runner): USD billed once per -# call (fixed pricing). Keep it under ~0.0019: the signer signs at most 100 +# Each runner's price lives in runners.json (static runner): USD billed once per +# call (fixed pricing). Keep each under ~0.0019: the signer signs at most 100 # tickets per payment and the demo orchestrator runs -ticketEV=1e10. -# Signer's max-price cap (payer side) is per billing unit, here one call, so it -# must exceed the runners.json price. +# Signer's max-price cap (payer side) is per billing unit, here one call, and it +# is one cap for every capability, so it must exceed the highest runners.json +# price (sd3, not flux). MAX_PRICE_PER_UNIT=0.000111USD diff --git a/api-proxy/README.md b/api-proxy/README.md index b6eb198..7ee478b 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -1,50 +1,59 @@ # API-proxy app (a runner that is pure config) -The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers a hosted model ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) by default) as a paid capability with two config files. +The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers two hosted models ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) and [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell)) as **two separately priced capabilities**, with nothing but config. -| | | -| ------------ | -------------------------------------------- | -| App id | `livepeer-example/stable-diffusion-3-medium` | -| Runner mode | single-shot | -| Registration | static (orchestrator config + health poll) | -| Transport | HTTP (HF payload in, JPEG bytes out) | -| Pricing | fixed (one price per call) | -| Port | 8989 | +| | | +| ------------ | ------------------------------------------------------------------------------- | +| App ids | `livepeer-example/stable-diffusion-3-medium`, `livepeer-example/flux-1-schnell` | +| Runner mode | single-shot | +| Registration | static (orchestrator config + health poll) | +| Transport | HTTP (HF payload in, JPEG bytes out) | +| Pricing | fixed, per capability (0.0001 and 0.00004 USD per image) | +| Port | 8989 (each nginx service) | Prerequisites (Docker, `uv`, and the [`livepeer-gateway` SDK](https://pypi.org/project/livepeer-gateway/)) and the shared on-chain/payment setup live in the [repo README](../README.md). The demo upstream additionally needs a **Hugging Face API token** (`HF_TOKEN`, from [huggingface.co → settings → tokens](https://huggingface.co/settings/tokens)) with inference-provider credits. ## How it's wired -The app is attached as a **static runner**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price — and health-polls `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. +The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls each `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. + +**One service, one model, one capability.** `compose.yml` runs the same `nginx.conf.template` twice with a different `MODEL`, and `runners.json` gives each its own app id and price. `--app` on the client picks which one to call. ## Offering an API as a capability — what this shows -Everything is operator-side config. `runners.json` names the capability and sets the **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the offered model. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover the capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. +Everything is operator-side config. `runners.json` names each capability and sets its **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the models offered. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover a capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. -The app id names the model, not the proxy, because that is what callers discover: they match it exactly, so it has to say what they get. Swapping `MODEL` means renaming the app id with it. +**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a `MODEL` means renaming its app id with it, and offering a third model is the same change once more — one `runners.json` entry, one nginx service. -Offering a second model is more config, not code: one more `runners.json` entry (its own app id and price) plus one more nginx service with a different `MODEL`. +**Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). **Fixed pricing** is the natural fit: one call is one bounded unit of work, so the runner bills one flat price per call instead of metering time. -> [!NOTE] -> Registration can also be **dynamic**: an operator tool can `register_runner` several API endpoints at runtime, each as its own priced capability, without touching the orchestrator config. See [livepeer/api-proxy](https://github.com/livepeer/api-proxy) for an example of dynamic endpoint registration, with key storage and request stats for orchestrator operators. +## Pinned here, dynamic in livepeer/api-proxy + +Both capabilities here are fixed at deploy time: two nginx services and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable. + +Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is one process that serves many endpoints, adding each at runtime with a CLI or a dashboard — one `register_runner` per endpoint, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. + +Same idea, opposite ends of the same axis: pin two and read the config, or run one process and enable endpoints as you go. ## Run offchain (free) ```sh cp .env.example .env # fill in HF_TOKEN; ignore the on-chain block docker compose up -d -curl -sk https://localhost:8935/discovery | jq '.[].runners[].app' # confirm livepeer-example/stable-diffusion-3-medium registered +curl -sk https://localhost:8935/discovery | jq '.[].runners[] | {app, price_info}' # both capabilities, each with its price uv run client.py --prompt "a watercolor painting of a llama writing code" +uv run client.py --app livepeer-example/flux-1-schnell \ + --prompt "a watercolor painting of a llama writing code" --output flux-out.jpg docker compose down ``` -`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the nginx runner. The client sends one prompt through the orchestrator and writes `api-proxy-out.jpg`. +`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the two nginx runners. The client sends one prompt through the orchestrator per call and writes the image; `--app` chooses the model and `--output` keeps the two results apart. ## Run on-chain (paid) -Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README. +Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises for the capability called. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README. ```sh cp .env.example .env # fill in HF_TOKEN, RPC, network, keystore paths, accounts @@ -55,4 +64,4 @@ uv run client.py --prompt "a watercolor painting of a llama writing code" \ docker compose -f compose.yml -f compose.onchain.yml down ``` -Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. +Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. Call the cheaper capability with `--app livepeer-example/flux-1-schnell` and the payment is smaller, which is the whole reason each model is its own app. The signer's `MAX_PRICE_PER_UNIT` is a single cap across capabilities, so it has to clear the **highest** price in `runners.json`. diff --git a/api-proxy/client.py b/api-proxy/client.py index 02524bd..86144bd 100644 --- a/api-proxy/client.py +++ b/api-proxy/client.py @@ -5,6 +5,10 @@ ({"inputs": ""}); the runner forwards it verbatim and the image comes back as raw JPEG bytes in `result.content`. +The operator offers two models, so `--app` picks which one to call. Discovery +matches app ids exactly, which is the whole reason each model has its own: it is +what a caller selects and pays for. + Livepeer integration (grep `# Livepeer:`): 1. runner_selector() — discover orchestrators advertising the app 2. call_runner() — call the app through the orchestrator; a non-JSON response @@ -25,7 +29,8 @@ from livepeer_gateway.selection import runner_selector DEFAULT_DISCOVERY = "https://localhost:8935/discovery" -APP_ID = "livepeer-example/stable-diffusion-3-medium" +DEFAULT_APP = "livepeer-example/stable-diffusion-3-medium" +FLUX_APP = "livepeer-example/flux-1-schnell" # the other one this demo offers DEFAULT_OUTPUT = "api-proxy-out.jpg" log = logging.getLogger("api-proxy-client") @@ -37,6 +42,11 @@ def _parse_args() -> argparse.Namespace: "--prompt", default="a watercolor painting of a llama writing code" ) parser.add_argument("--output", default=DEFAULT_OUTPUT, help="output image path") + parser.add_argument( + "--app", + default=DEFAULT_APP, + help=f"app id to call; this demo also offers {FLUX_APP}", + ) parser.add_argument("--discovery", default=DEFAULT_DISCOVERY) parser.add_argument( "--signer", default="", help="Remote signer base URL (on-chain/paid path)." @@ -52,10 +62,10 @@ async def main() -> None: try: cursor = await runner_selector( # Livepeer: 1 discovery_url=args.discovery, # omit if the signer does discovery itself - app=APP_ID, + app=args.app, ) runner = cursor.candidates[0] - log.info("app_url=%s", runner.url) + log.info("app=%s app_url=%s", args.app, runner.url) result = await call_runner( # Livepeer: 2 runner=runner, # discovery metadata tells call_runner the price unit diff --git a/api-proxy/compose.yml b/api-proxy/compose.yml index 420a937..ea63653 100644 --- a/api-proxy/compose.yml +++ b/api-proxy/compose.yml @@ -1,6 +1,7 @@ -# End-to-end offchain demo: orchestrator + the api-proxy "app" — a stock nginx -# attached as a static runner via runners.json (-liveRunnerConfig). There is no -# app code at all: nginx.conf.template is the whole runner. +# End-to-end offchain demo: orchestrator + the api-proxy "app" — two stock nginx +# services attached as static runners via runners.json (-liveRunnerConfig). There +# is no app code at all: nginx.conf.template is the whole runner, and each service +# is one capability because it pins one model. # # Needs HF_TOKEN (the upstream credential nginx injects) in a local .env — copy # .env.example. Once up, call it from the host with the SDK: @@ -28,9 +29,11 @@ services: volumes: - ./runners.json:/config/runners.json:ro - app: + # One service per capability. Both run the same template and the same + # credential; only MODEL differs, and runners.json prices them separately. + sd3: image: nginx:1.27-alpine - container_name: example_apps_api_proxy + container_name: example_apps_api_proxy_sd3 environment: # The operator-held upstream credential (huggingface.co → settings → tokens). - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} @@ -39,3 +42,14 @@ services: - MODEL=stabilityai/stable-diffusion-3-medium-diffusers volumes: - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro + + flux: + image: nginx:1.27-alpine + container_name: example_apps_api_proxy_flux + environment: + - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} + # A faster, cheaper model than sd3 — which is why it is a separate + # capability at a separate price rather than a flag on the first one. + - MODEL=black-forest-labs/FLUX.1-schnell + volumes: + - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro diff --git a/api-proxy/nginx.conf.template b/api-proxy/nginx.conf.template index f6498ac..56ec5d4 100644 --- a/api-proxy/nginx.conf.template +++ b/api-proxy/nginx.conf.template @@ -2,6 +2,9 @@ # model URL and injects the operator's token. The orchestrator health-polls # /health and reverse-proxies caller requests to /proxy. HF_TOKEN and MODEL are # substituted from the environment by the nginx image's template mechanism. +# +# One template, one model, one capability. compose.yml runs this same file twice +# with a different MODEL, which is how the operator offers two. server { listen 8989; diff --git a/api-proxy/runners.json b/api-proxy/runners.json index 7f52f63..6e71d9e 100644 --- a/api-proxy/runners.json +++ b/api-proxy/runners.json @@ -1,13 +1,22 @@ { "runners": [ { - "label": "api-proxy", + "label": "api-proxy-sd3", "app": "livepeer-example/stable-diffusion-3-medium", - "runner_url": "http://app:8989", + "runner_url": "http://sd3:8989", "health_url": "/health", "mode": "single-shot", "capacity": 1, "price_info": { "price": 0.0001, "unit": "fixed" } + }, + { + "label": "api-proxy-flux", + "app": "livepeer-example/flux-1-schnell", + "runner_url": "http://flux:8989", + "health_url": "/health", + "mode": "single-shot", + "capacity": 1, + "price_info": { "price": 0.00004, "unit": "fixed" } } ] } From 0304f5e341dce578972be81b42a41145915688ce Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Fri, 14 Aug 2026 19:01:09 +0200 Subject: [PATCH 2/7] refactor(api-proxy): serve both models from one nginx Two containers implied a capability is a process. It is a registration: one nginx now carries a route per model, and each runners.json entry points its app id at its own path, which the orchestrator preserves when it forwards. Keeps the repo consistent with ollama, where one container serves several capabilities. Health stays shared. `return 200` only reported that nginx is up, never that a model or the token is good, so a copy per capability would claim a precision it does not have. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/README.md | 20 +++++++++++--------- api-proxy/compose.yml | 34 ++++++++++++---------------------- api-proxy/nginx.conf.template | 32 +++++++++++++++++++++++++------- api-proxy/runners.json | 8 ++++---- 4 files changed, 52 insertions(+), 42 deletions(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index 7ee478b..8e28976 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -1,6 +1,6 @@ # API-proxy app (a runner that is pure config) -The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers two hosted models ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) and [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell)) as **two separately priced capabilities**, with nothing but config. +The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to its route's pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers two hosted models ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) and [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell)) as **two separately priced capabilities**, with nothing but config. | | | | ------------ | ------------------------------------------------------------------------------- | @@ -9,21 +9,23 @@ The live runner can also **pass calls through to an API that runs somewhere else | Registration | static (orchestrator config + health poll) | | Transport | HTTP (HF payload in, JPEG bytes out) | | Pricing | fixed, per capability (0.0001 and 0.00004 USD per image) | -| Port | 8989 (each nginx service) | +| Port | 8989 | Prerequisites (Docker, `uv`, and the [`livepeer-gateway` SDK](https://pypi.org/project/livepeer-gateway/)) and the shared on-chain/payment setup live in the [repo README](../README.md). The demo upstream additionally needs a **Hugging Face API token** (`HF_TOKEN`, from [huggingface.co → settings → tokens](https://huggingface.co/settings/tokens)) with inference-provider credits. ## How it's wired -The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls each `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. +The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls `/health` (an nginx `return 200`). Each `/proxy` location proxies to its pinned model URL (`MODEL_SD3` and `MODEL_FLUX` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. -**One service, one model, one capability.** `compose.yml` runs the same `nginx.conf.template` twice with a different `MODEL`, and `runners.json` gives each its own app id and price. `--app` on the client picks which one to call. +**A capability is a registration, not a container.** One nginx serves both: each `runners.json` entry points its app id at its own path (`http://app:8989/sd3`), and the orchestrator preserves that path when it forwards, so the two registrations land on different `location` blocks with different pinned models and different prices. `--app` on the client picks which one to call. + +The shared `/health` is deliberate. `return 200` only ever reported that nginx is up, never that an upstream model or the token is still good, so a copy per capability would claim a precision it does not have. ## Offering an API as a capability — what this shows Everything is operator-side config. `runners.json` names each capability and sets its **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the models offered. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover a capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. -**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a `MODEL` means renaming its app id with it, and offering a third model is the same change once more — one `runners.json` entry, one nginx service. +**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a `MODEL_*` means renaming its app id with it, and offering a third model is the same change once more — one `location` block, one `runners.json` entry. **Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). @@ -31,11 +33,11 @@ Everything is operator-side config. `runners.json` names each capability and set ## Pinned here, dynamic in livepeer/api-proxy -Both capabilities here are fixed at deploy time: two nginx services and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable. +Both capabilities here are fixed at deploy time: two `location` blocks and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable. -Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is one process that serves many endpoints, adding each at runtime with a CLI or a dashboard — one `register_runner` per endpoint, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. +Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is the same one-process-many-endpoints shape, except endpoints are added at runtime with a CLI or a dashboard — one `register_runner` each, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. -Same idea, opposite ends of the same axis: pin two and read the config, or run one process and enable endpoints as you go. +Same idea, opposite ends of the same axis: pin the routes and read the config, or enable endpoints as you go. ## Run offchain (free) @@ -49,7 +51,7 @@ uv run client.py --app livepeer-example/flux-1-schnell \ docker compose down ``` -`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the two nginx runners. The client sends one prompt through the orchestrator per call and writes the image; `--app` chooses the model and `--output` keeps the two results apart. +`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the nginx runner, which registers as two capabilities. The client sends one prompt through the orchestrator per call and writes the image; `--app` chooses the model and `--output` keeps the two results apart. ## Run on-chain (paid) diff --git a/api-proxy/compose.yml b/api-proxy/compose.yml index ea63653..da4ea6b 100644 --- a/api-proxy/compose.yml +++ b/api-proxy/compose.yml @@ -1,7 +1,7 @@ -# End-to-end offchain demo: orchestrator + the api-proxy "app" — two stock nginx -# services attached as static runners via runners.json (-liveRunnerConfig). There -# is no app code at all: nginx.conf.template is the whole runner, and each service -# is one capability because it pins one model. +# End-to-end offchain demo: orchestrator + the api-proxy "app" — a stock nginx +# attached as two static runners via runners.json (-liveRunnerConfig). There is +# no app code at all: nginx.conf.template is the whole runner, and it serves two +# capabilities from one process because a capability is a registration. # # Needs HF_TOKEN (the upstream credential nginx injects) in a local .env — copy # .env.example. Once up, call it from the host with the SDK: @@ -29,27 +29,17 @@ services: volumes: - ./runners.json:/config/runners.json:ro - # One service per capability. Both run the same template and the same - # credential; only MODEL differs, and runners.json prices them separately. - sd3: + app: image: nginx:1.27-alpine - container_name: example_apps_api_proxy_sd3 + container_name: example_apps_api_proxy environment: # The operator-held upstream credential (huggingface.co → settings → tokens). - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} - # The one model this runner offers; swap it here, and rename the app id - # in runners.json to match (the app id is what callers discover). - - MODEL=stabilityai/stable-diffusion-3-medium-diffusers - volumes: - - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro - - flux: - image: nginx:1.27-alpine - container_name: example_apps_api_proxy_flux - environment: - - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} - # A faster, cheaper model than sd3 — which is why it is a separate - # capability at a separate price rather than a flag on the first one. - - MODEL=black-forest-labs/FLUX.1-schnell + # One model per route in nginx.conf.template. Swap one here and rename its + # app id in runners.json to match (the app id is what callers discover). + - MODEL_SD3=stabilityai/stable-diffusion-3-medium-diffusers + # Faster and cheaper than sd3 — which is why it is a separate capability + # at a separate price rather than a flag on the first one. + - MODEL_FLUX=black-forest-labs/FLUX.1-schnell volumes: - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro diff --git a/api-proxy/nginx.conf.template b/api-proxy/nginx.conf.template index 56ec5d4..e870254 100644 --- a/api-proxy/nginx.conf.template +++ b/api-proxy/nginx.conf.template @@ -1,13 +1,19 @@ -# The whole runner: a stock nginx forwards each call to one pinned Hugging Face +# The whole runner: a stock nginx forwards each call to a pinned Hugging Face # model URL and injects the operator's token. The orchestrator health-polls -# /health and reverse-proxies caller requests to /proxy. HF_TOKEN and MODEL are -# substituted from the environment by the nginx image's template mechanism. +# /health and reverse-proxies caller requests to /proxy. HF_TOKEN and the +# MODEL_* vars are substituted from the environment by the nginx image's +# template mechanism. # -# One template, one model, one capability. compose.yml runs this same file twice -# with a different MODEL, which is how the operator offers two. +# One process, one route per capability. runners.json points each app id at its +# own path (http://app:8989/sd3), and the orchestrator preserves that path when +# it forwards, so the two registrations land on different location blocks. A +# capability is a registration, not a container. server { listen 8989; + # One health endpoint for both capabilities: `return 200` only ever reported + # that nginx is up, never that an upstream model or the token is still good, + # so a per-capability copy would claim a precision it does not have. location = /health { default_type application/json; return 200 '{"status":"ok"}'; @@ -17,10 +23,10 @@ server { # be spent on exactly this model, callers choose nothing but the payload. # Method and query string are part of "nothing": nginx would forward both # upstream as the caller sent them, so pin the method and drop the args. - location = /proxy { + location = /sd3/proxy { limit_except POST { deny all; } set $args ''; - proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL}; + proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL_SD3}; proxy_set_header Authorization "Bearer ${HF_TOKEN}"; # Hugging Face content-negotiates: with the SDK's Accept: # application/json it answers with a base64 PNG in a JSON string @@ -29,4 +35,16 @@ server { proxy_ssl_server_name on; proxy_read_timeout 120s; # a hosted diffusion model can take tens of seconds } + + # The same block against a different model. Offering one more capability is + # one more location here plus one more runners.json entry. + location = /flux/proxy { + limit_except POST { deny all; } + set $args ''; + proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL_FLUX}; + proxy_set_header Authorization "Bearer ${HF_TOKEN}"; + proxy_set_header Accept "image/jpeg"; + proxy_ssl_server_name on; + proxy_read_timeout 120s; + } } diff --git a/api-proxy/runners.json b/api-proxy/runners.json index 6e71d9e..9caa1a9 100644 --- a/api-proxy/runners.json +++ b/api-proxy/runners.json @@ -3,8 +3,8 @@ { "label": "api-proxy-sd3", "app": "livepeer-example/stable-diffusion-3-medium", - "runner_url": "http://sd3:8989", - "health_url": "/health", + "runner_url": "http://app:8989/sd3", + "health_url": "http://app:8989/health", "mode": "single-shot", "capacity": 1, "price_info": { "price": 0.0001, "unit": "fixed" } @@ -12,8 +12,8 @@ { "label": "api-proxy-flux", "app": "livepeer-example/flux-1-schnell", - "runner_url": "http://flux:8989", - "health_url": "/health", + "runner_url": "http://app:8989/flux", + "health_url": "http://app:8989/health", "mode": "single-shot", "capacity": 1, "price_info": { "price": 0.00004, "unit": "fixed" } From 8dec638d1f5f880b54ee15eae1bedc5cb2f662c2 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Tue, 18 Aug 2026 10:41:00 +0200 Subject: [PATCH 3/7] refactor(api-proxy): pin the models in the config that serves them The MODEL_* variables only moved a literal from one file to another, and swapping a model costs the same two edits either way. Pinning the URLs in nginx.conf.template puts each model beside the route that serves it, and makes "the pinned URL is the security model" true of the file you read rather than of the environment at runtime. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/README.md | 4 ++-- api-proxy/compose.yml | 6 ------ api-proxy/nginx.conf.template | 10 +++++----- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index 8e28976..9e7053e 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -15,7 +15,7 @@ Prerequisites (Docker, `uv`, and the [`livepeer-gateway` SDK](https://pypi.org/p ## How it's wired -The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls `/health` (an nginx `return 200`). Each `/proxy` location proxies to its pinned model URL (`MODEL_SD3` and `MODEL_FLUX` in [compose.yml](compose.yml)) with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. +The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls `/health` (an nginx `return 200`). Each `/proxy` location proxies to its pinned model URL with `Authorization: Bearer ` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": ""}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls. **A capability is a registration, not a container.** One nginx serves both: each `runners.json` entry points its app id at its own path (`http://app:8989/sd3`), and the orchestrator preserves that path when it forwards, so the two registrations land on different `location` blocks with different pinned models and different prices. `--app` on the client picks which one to call. @@ -25,7 +25,7 @@ The shared `/health` is deliberate. `return 200` only ever reported that nginx i Everything is operator-side config. `runners.json` names each capability and sets its **fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the models offered. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover a capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost. -**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a `MODEL_*` means renaming its app id with it, and offering a third model is the same change once more — one `location` block, one `runners.json` entry. +**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a model means renaming its app id with it, and offering a third is the same change once more: one `location` block, one `runners.json` entry. **Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). diff --git a/api-proxy/compose.yml b/api-proxy/compose.yml index da4ea6b..51c0520 100644 --- a/api-proxy/compose.yml +++ b/api-proxy/compose.yml @@ -35,11 +35,5 @@ services: environment: # The operator-held upstream credential (huggingface.co → settings → tokens). - HF_TOKEN=${HF_TOKEN:?set HF_TOKEN in .env (copy .env.example)} - # One model per route in nginx.conf.template. Swap one here and rename its - # app id in runners.json to match (the app id is what callers discover). - - MODEL_SD3=stabilityai/stable-diffusion-3-medium-diffusers - # Faster and cheaper than sd3 — which is why it is a separate capability - # at a separate price rather than a flag on the first one. - - MODEL_FLUX=black-forest-labs/FLUX.1-schnell volumes: - ./nginx.conf.template:/etc/nginx/templates/default.conf.template:ro diff --git a/api-proxy/nginx.conf.template b/api-proxy/nginx.conf.template index e870254..fb5a12f 100644 --- a/api-proxy/nginx.conf.template +++ b/api-proxy/nginx.conf.template @@ -1,8 +1,8 @@ # The whole runner: a stock nginx forwards each call to a pinned Hugging Face # model URL and injects the operator's token. The orchestrator health-polls -# /health and reverse-proxies caller requests to /proxy. HF_TOKEN and the -# MODEL_* vars are substituted from the environment by the nginx image's -# template mechanism. +# /health and reverse-proxies caller requests to /proxy. Only HF_TOKEN is +# substituted from the environment, by the nginx image's template mechanism; the +# models are pinned here, in the same file as the routes that serve them. # # One process, one route per capability. runners.json points each app id at its # own path (http://app:8989/sd3), and the orchestrator preserves that path when @@ -26,7 +26,7 @@ server { location = /sd3/proxy { limit_except POST { deny all; } set $args ''; - proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL_SD3}; + proxy_pass https://router.huggingface.co/hf-inference/models/stabilityai/stable-diffusion-3-medium-diffusers; proxy_set_header Authorization "Bearer ${HF_TOKEN}"; # Hugging Face content-negotiates: with the SDK's Accept: # application/json it answers with a base64 PNG in a JSON string @@ -41,7 +41,7 @@ server { location = /flux/proxy { limit_except POST { deny all; } set $args ''; - proxy_pass https://router.huggingface.co/hf-inference/models/${MODEL_FLUX}; + proxy_pass https://router.huggingface.co/hf-inference/models/black-forest-labs/FLUX.1-schnell; proxy_set_header Authorization "Bearer ${HF_TOKEN}"; proxy_set_header Accept "image/jpeg"; proxy_ssl_server_name on; From ec68b02af5a924c837411c0ba85ce79d71b26659 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Tue, 18 Aug 2026 10:46:20 +0200 Subject: [PATCH 4/7] docs(api-proxy): say why health cannot check the upstream Hugging Face exposes no readiness endpoint, only metadata, and the orchestrator polls health every 5s with a 3s timeout while a failed poll also releases live sessions. Record that in the comment so the next reader does not repeat the investigation. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/nginx.conf.template | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/api-proxy/nginx.conf.template b/api-proxy/nginx.conf.template index fb5a12f..07b797d 100644 --- a/api-proxy/nginx.conf.template +++ b/api-proxy/nginx.conf.template @@ -1,19 +1,16 @@ -# The whole runner: a stock nginx forwards each call to a pinned Hugging Face -# model URL and injects the operator's token. The orchestrator health-polls -# /health and reverse-proxies caller requests to /proxy. Only HF_TOKEN is -# substituted from the environment, by the nginx image's template mechanism; the -# models are pinned here, in the same file as the routes that serve them. +# The whole runner: a stock nginx pins one Hugging Face model per route and +# injects the operator's token. HF_TOKEN is the only value substituted from the +# environment, by the nginx image's template mechanism. # -# One process, one route per capability. runners.json points each app id at its -# own path (http://app:8989/sd3), and the orchestrator preserves that path when -# it forwards, so the two registrations land on different location blocks. A -# capability is a registration, not a container. +# runners.json points each app id at its own path (http://app:8989/sd3), which +# the orchestrator preserves when forwarding, so each lands on its own block. server { listen 8989; - # One health endpoint for both capabilities: `return 200` only ever reported - # that nginx is up, never that an upstream model or the token is still good, - # so a per-capability copy would claim a precision it does not have. + # One endpoint for both capabilities. `return 200` only reports that nginx is + # up, never that a model or the token is good. Hugging Face exposes no + # readiness endpoint, only metadata, and the orchestrator polls this every 5s + # with a 3s timeout, so a real upstream check does not belong here. location = /health { default_type application/json; return 200 '{"status":"ok"}'; @@ -36,8 +33,7 @@ server { proxy_read_timeout 120s; # a hosted diffusion model can take tens of seconds } - # The same block against a different model. Offering one more capability is - # one more location here plus one more runners.json entry. + # One more capability is one more location here plus one runners.json entry. location = /flux/proxy { limit_except POST { deny all; } set $args ''; From 68d44bde53900930d0210dd4ceb0f60495be4f65 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Tue, 18 Aug 2026 10:49:54 +0200 Subject: [PATCH 5/7] docs(api-proxy): stop restating why the prices differ The on-chain run section re-argued the point the capability section already makes, inside a passage that should only say what to type. Keep the signer cap, which is the part unique to running on-chain. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index 9e7053e..d4e4a8e 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -66,4 +66,4 @@ uv run client.py --prompt "a watercolor painting of a llama writing code" \ docker compose -f compose.yml -f compose.onchain.yml down ``` -Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. Call the cheaper capability with `--app livepeer-example/flux-1-schnell` and the payment is smaller, which is the whole reason each model is its own app. The signer's `MAX_PRICE_PER_UNIT` is a single cap across capabilities, so it has to clear the **highest** price in `runners.json`. +Each call is one paid single-shot session: the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. `--app` picks which price you pay. The signer's `MAX_PRICE_PER_UNIT` is one cap across both capabilities, so it has to clear the **highest** price in `runners.json`. From 2fcaa641cc0f67bdf4a43db40488789280e81124 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Tue, 18 Aug 2026 10:51:41 +0200 Subject: [PATCH 6/7] docs(api-proxy): say each thing once The health reasoning now lives in nginx.conf.template, where it also records that Hugging Face exposes no readiness endpoint, so the README points at it rather than restating a weaker version. The static versus dynamic section loses its closing summary, which only repeated the two paragraphs above it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index d4e4a8e..5e78c1d 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -19,7 +19,7 @@ The apps are attached as **static runners**: the orchestrator reads [runners.jso **A capability is a registration, not a container.** One nginx serves both: each `runners.json` entry points its app id at its own path (`http://app:8989/sd3`), and the orchestrator preserves that path when it forwards, so the two registrations land on different `location` blocks with different pinned models and different prices. `--app` on the client picks which one to call. -The shared `/health` is deliberate. `return 200` only ever reported that nginx is up, never that an upstream model or the token is still good, so a copy per capability would claim a precision it does not have. +The shared `/health` is deliberate: it reports that nginx is up and nothing more. [nginx.conf.template](nginx.conf.template) says why a real upstream check does not belong there. ## Offering an API as a capability — what this shows @@ -35,9 +35,7 @@ Everything is operator-side config. `runners.json` names each capability and set Both capabilities here are fixed at deploy time: two `location` blocks and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable. -Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is the same one-process-many-endpoints shape, except endpoints are added at runtime with a CLI or a dashboard — one `register_runner` each, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. - -Same idea, opposite ends of the same axis: pin the routes and read the config, or enable endpoints as you go. +Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is the same one-process-many-endpoints shape, except endpoints are added at runtime with a CLI or a dashboard: one `register_runner` each, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs. ## Run offchain (free) From 0b918b072ee88925868548f3f655716d2fa2b6e7 Mon Sep 17 00:00:00 2001 From: Rick Staa Date: Tue, 18 Aug 2026 11:19:20 +0200 Subject: [PATCH 7/7] fix(api-proxy): stop serialising callers a proxy never blocked capacity was 1, so three of four concurrent callers got a 503 while nothing on the machine was busy: the work happens upstream and nginx only holds a socket. Measured, four concurrent calls all get through at 4. For a proxy capacity is exposure control, capping paid-but-unfulfilled calls against the operator's credential, so it belongs to the upstream quota rather than to this machine. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV --- api-proxy/README.md | 2 +- api-proxy/runners.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/api-proxy/README.md b/api-proxy/README.md index 5e78c1d..7178d04 100644 --- a/api-proxy/README.md +++ b/api-proxy/README.md @@ -27,7 +27,7 @@ Everything is operator-side config. `runners.json` names each capability and set **A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a model means renaming its app id with it, and offering a third is the same change once more: one `location` block, one `runners.json` entry. -**Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). +**Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Capacity is exposure control rather than a local limit, since payment is taken when the session is reserved: it caps how many paid calls can be in flight against the operator's credential, so size it to the upstream quota rather than to this machine. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)). **Fixed pricing** is the natural fit: one call is one bounded unit of work, so the runner bills one flat price per call instead of metering time. diff --git a/api-proxy/runners.json b/api-proxy/runners.json index 9caa1a9..eaad64e 100644 --- a/api-proxy/runners.json +++ b/api-proxy/runners.json @@ -6,7 +6,7 @@ "runner_url": "http://app:8989/sd3", "health_url": "http://app:8989/health", "mode": "single-shot", - "capacity": 1, + "capacity": 4, "price_info": { "price": 0.0001, "unit": "fixed" } }, { @@ -15,7 +15,7 @@ "runner_url": "http://app:8989/flux", "health_url": "http://app:8989/health", "mode": "single-shot", - "capacity": 1, + "capacity": 4, "price_info": { "price": 0.00004, "unit": "fixed" } } ]