fix(bifrost): give the gateway a real cost, and a file to configure it

[elronbandel]

The bug

Every provider-call span in the results store carried gen_ai.usage.cost=0 — all ~2,100 of them — which is why the dashboard's cost column was blank across the entire result history.

bifrost prices each call from its bundled pricing.json, keyed on the model name. The handles we route are the upstream proxy's own (azure/gpt-5.5, aws/claude-opus-5, rits/google/gemma-4-31B), not litellm's public catalog names, so the lookup missed and it wrote 0. There is no fallback layer.

The upstream does know the cost — it returns x-litellm-response-cost on every response — but that's an HTTP header, and bifrost's span carries the cost it computed itself. Its otel plugin has no header-capture knob.

The fix (commit 1)

A global wildcard governance.pricing_overrides entry giving bifrost the rate, so its own arithmetic comes out right. One flat rate is correct here because the upstream bills one: probing it with gpt-5.4 / gpt-5.5 / gpt-4.1-mini / claude-opus-5 / claude-sonnet-5 / gemini-3-flash / gemma-4-31B returns the same cost for the same token counts. Solving two samples gives $2.50/1M in, $15.00/1M out, which then predicts x-litellm-response-cost to 1e-9 on every model tried. Overridable via EVAL_COST_INPUT_PER_TOKEN / EVAL_COST_OUTPUT_PER_TOKEN.

Per-model prices (commit 2)

One flat rate is right for our proxy, but the image ships standalone. EVAL_PRICING_FILE (default /opt/gateway/data/pricing.tsv, absent from the image) is read at startup and rendered into one exact override per model:

```
docker run -e EVAL_MODEL=azure/gpt-5.5 \
-v ./pricing.tsv:/opt/gateway/data/pricing.tsv:ro \
... ghcr.io/exgentic/models/bifrost:latest
```

```

model in/token out/token

azure/gpt-5.5 0.0000050 0.0000300
aws/claude-opus-5 0.0000150 0.0000750
```

Those entries are layered alongside the wildcard, not in place of it, so an incomplete list falls back to EVAL_COST_* rather than pricing at 0 — which was the original bug. Specificity decides; array order is irrelevant. A malformed line exits 2 with the line number (rule 22).

Two traps, documented in three places

• bifrost's exact match sees the BARE requested model (azure/gpt-5.5), not the provider-prefixed key it uses to index its own pricing.json (openai/azure/gpt-5.5). A prefixed pattern matches nothing and silently leaves the model on the fallback. Verified empirically both ways.
• Prices are per token, not per million.

Both are called out in the env contract, the .example file, and a static test assertion.

Why not just replace pricing.json?

It works, but it's a trap: any model absent from the replacement gets cost 0 with no fallback, reintroducing exactly this bug. Confirmed on the cluster.

Why whitespace-separated, not JSON?

The alpine image has neither jq nor python, and it must run under plain docker run (rule 20). Hence awk, hence the simple format.

pricing.tsv.example is copied in as documentation only — start reads pricing.tsv, never the example — so no price is baked into the image (rule 1).

Test fixture repairs

The --ignored gateway suite was 7/15 green before this branch, and the failures are pre-existing (reproduced on clean origin/main with my changes stashed):

• The fixtures set EVAL_MODEL=openai/azure/gpt-5.4 for every flavor, but bifrost and litellm take a bare handle — the prefixed string went upstream verbatim and 403'd with team_model_access_denied. portkey is the opposite: it eats the first path segment (MODEL_NAME=${EVAL_MODEL#*/}) and needs the prefix. Reconciled with a per-flavor eval_model().
• Two OTel tests entering start_pod_with_otel on the same clock tick collided on otelcol-<nanos> ("name is already in use"). Fixed with an atomic counter in the suffix.

Tests

• static_bifrost_pricing_example_parses_as_the_start_script_reads_it — 3 fields per row, per-token magnitude (catches an undivided per-1M figure), and no wire-prefixed model names.
• otel_bifrost_span_cost_is_nonzero — the regression guard for the actual bug.
• otel_bifrost_prices_a_listed_model_from_the_mounted_price_file — the file's rate wins over the flat fallback (the two differ by six orders of magnitude, so the resulting cost is unambiguous about which applied).
• otel_bifrost_falls_back_when_the_price_file_omits_the_model — a partial file leaves other models on EVAL_COST_*, and one model's entry doesn't leak onto another.

Asserting which rate applied, never that cost is under a threshold — a cost SLO is forbidden (tests/run/gateways/RULES.md rule 12).

Results: --ignored suite 17/17 (was 7/15); static tier 17/17; cargo fmt --check clean; clippy adds no new warnings. upstream_portkey_openai flaked once under --test-threads=2 with an upstream fetch failed inside portkey and passed alone — unrelated to pricing, portkey has no pricing path.

Also verified by hand against the real upstream, all four paths: baked file applies its rate, mounted file via EVAL_PRICING_FILE applies its rate, malformed file exits 2 loudly, no file at all falls back and still serves 200.

Dashboard

No change needed — app/results.py already reads gen_ai.usage.cost. It was reading a real attribute that was really 0.

🤖 Generated with Claude Code

[elronbandel]

Pushed 71df3cd5 — no price is baked into the image any more.

The first pass left $2.50/$15 as a shell default in start, which made our IBM proxy's rate a property of the gateway image for everyone who runs it. Prices depend on provider, contract, and account, and they change, so the image now carries none.

Both knobs are optional and independent:

EVAL_COST_*_PER_TOKEN	flat rate for every model (* wildcard)
EVAL_PRICING_FILE	per-model rates, one exact override each

Set both and the file wins for the models it names while the flat rate catches the rest. Set neither and we render no pricing_overrides at all, leaving bifrost's native bundled-catalog lookup exactly as it behaves out of the box. That path warns rather than exits — an unconfigured gateway is a working deployment, not a misconfiguration — but it does warn, because "models absent from that catalog report cost 0" is otherwise invisible until someone reads a blank cost column.

Two things now fail loud instead of quietly pricing wrong (rule 22):

• one flat knob without the other — would have priced that half at 0, which is worse than no pricing because the number looks real
• an EVAL_PRICING_FILE path that doesn't exist — a typo in -v previously booted fine and priced everything from the catalog

Our own rate moved to deploy/values-openshift.yaml through the chart's existing gatewayExtraEnv hook, so changing it is an edit to a values file, not a rebuild. Worth noting that this is also the first thing that ever set it: the chart's gateway container had no cost env at all, so production was running purely on the image default. src/RULES.md 12 already puts platform-specific settings in a composable values overlay.

Verified by rendering all four combinations: flat-only, file-only (correctly strips the leading comma — this is why the new no-config test exists, an empty array is easy to get subtly wrong and invalid JSON stops bifrost booting), both, and neither. Both misconfigurations exit 2. Suite is 18/18 ignored + 17/17 static.

One thing I did not touch: containers/models/gpt-5.4/config.yaml has its own baked input_cost_per_token: 0.0000025 / output_cost_per_token: 0.000010 for the litellm flavor. That's the same class of hardcode, and its output figure disagrees with what I measured on the proxy ($10/1M vs $15/1M). Out of scope here — flagging it rather than changing it silently.