feat(api-proxy): offer two priced models instead of one - #72
Open
rickstaa wants to merge 2 commits into
Open
Conversation
The README claimed a second model was "more config, not code" without showing it. Run nginx.conf.template twice with a different MODEL, give each its own runners.json entry, app id, and price, and let the client pick with --app. FLUX.1 schnell is cheaper than SD3 medium, so the two prices differ for a reason a caller can see. Each entry keeps its own capacity: the work happens upstream, so the capabilities do not contend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
Two containers implied a capability is a process. It is a registration: one nginx now carries a route per model, and each runners.json entry points its app id at its own path, which the orchestrator preserves when it forwards. Keeps the repo consistent with ollama, where one container serves several capabilities. Health stays shared. `return 200` only reported that nginx is up, never that a model or the token is good, so a copy per capability would claim a precision it does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
rickstaa
force-pushed
the
rs/api-proxy-two-models
branch
from
August 17, 2026 10:14
f3bc8a8 to
0304f5e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The api-proxy README claimed that offering a second model is "more config, not code", but the example only ever showed one. Now it shows two, served from the one nginx.
nginx.conf.templategets a route per capability,/sd3/proxyand/flux/proxy, each pinning its own model.runners.jsongets a second entry with its own app id and price, each pointing at its own path (http://app:8989/sd3), which the orchestrator preserves when it forwards. FLUX.1 schnell is faster and cheaper than SD3 medium, so the two prices differ for a reason a caller can see.client.pygains--app, which is what actually demonstrates that discovery filters on app id, matched exactly.livepeer/api-proxybecomes a section: the static versus dynamic contrast only lands once there are two pinned capabilities to hold against runtime enablement.One process, two registrations, because a capability is a registration and not a container. That also keeps this consistent with the ollama example, where one container serves several capabilities.
Verified offchain end to end: both register healthy, both appear in
/discoveryas separate capabilities, and a call through the orchestrator to each app id reserves a session and lands on the right pinned model. The upstream returns 401 on a dummy token, which is as far as this goes without anHF_TOKENcarrying inference-provider credits, so no real generation was run.🤖 Generated with Claude Code
https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV