Skip to content

fix(token-spy): price Claude 3.x models instead of billing them at zero - #2511

Open
Hoang130203 wants to merge 1 commit into
Osmantic:mainfrom
Hoang130203:fix/token-spy-claude3-pricing
Open

fix(token-spy): price Claude 3.x models instead of billing them at zero#2511
Hoang130203 wants to merge 1 commit into
Osmantic:mainfrom
Hoang130203:fix/token-spy-claude3-pricing

Conversation

@Hoang130203

Copy link
Copy Markdown

Summary

get_model_pricing() matches model ids against COST_TABLE by substring. The
table is keyed on Anthropic's post-Claude-4 spelling, where the family comes
before the generation:

"claude-haiku-4-5": {...}   # matches claude-haiku-4-5-20251001

Claude 3.x ids put the generation first, so none of them match anything and
every lookup falls through to the zero default. Run against origin/main:

  claude-opus-4-5-20251101       in=5.0    out=25.0
  claude-sonnet-4-5-20250929     in=3.0    out=15.0
  claude-haiku-4-5-20251001      in=1.0    out=5.0
  claude-3-5-sonnet-20241022     in=0.0    out=0.0     <-- ZERO COST
  claude-3-5-haiku-20241022      in=0.0    out=0.0     <-- ZERO COST
  claude-3-opus-20240229         in=0.0    out=0.0     <-- ZERO COST
  claude-3-haiku-20240307        in=0.0    out=0.0     <-- ZERO COST

Token Spy is the service that tells an operator what their agents cost.
Reporting $0.00 for a real Claude 3.5 Sonnet session is not a missing
feature — it reads as "this was free", which is the one answer a cost tracker
must never give wrongly. And it fails quietly: nothing in the UI distinguishes
"a local model, genuinely free" from "we could not price this".

The give-away that this is a bug and not a policy choice is already in the
table:

"claude-haiku-3-5": {"input": 0.80, "output": 4.0, "cache_read": 0.08, "cache_write": 1.0},

0.80 / 4.00 is exactly Claude 3.5 Haiku's rate — someone priced that tier
deliberately, wrote it in a spelling Anthropic never shipped, and it has been
unreachable ever since.

Fix

  1. Normalise generation-first ids onto the family-first form before the
    lookup. claude-3-5-haiku-20241022claude-haiku-3-5-20241022, which
    makes the row above reachable without inventing a price. Claude 4+ ids are
    left byte-identical.
  2. Add rows for the tiers that had none: sonnet-3-7, sonnet-3-5,
    opus-3, haiku-3.
  3. haiku-3 gets its own row so the bare claude-haiku catch-all does not
    swallow it. That catch-all is priced at Haiku 3.5's rate, so without an
    explicit row the fix would have billed Haiku 3 at 0.80/4.00 instead of
    0.25/1.25 — a 3.2× overstatement. Turning a silent undercount into a silent
    overcount would not be an improvement.

After:

  claude-3-7-sonnet-20250219     in=3.0    out=15.0
  claude-3-5-sonnet-20241022     in=3.0    out=15.0
  claude-3-5-haiku-20241022      in=0.8    out=4.0
  claude-3-opus-20240229         in=15.0   out=75.0
  claude-3-haiku-20240307        in=0.25   out=1.25
  gpt-4o                         in=0.0    out=0.0     (still a miss, correctly)

Test

providers/ had no test coverage — nothing under
extensions/services/token-spy/tests/ referenced it. Adds
tests/test_provider_pricing.py: 52 assertions over every shipped Claude id,
both spellings resolving to the same row, output ≥ input and cache_read < input
as typo guards, the Haiku 3 / 3.5 boundary, that non-Claude ids still miss, and
that calculate_cost uses the row the lookup returned. Wired into the existing
token-spy CI job.

AI Assistance

AI assisted with the normalisation regex, drafting the test matrix, and wording
this description. I found the mismatch by running the shipped table against
real model ids, and the four prices I added are the published rates listed
below for you to check.

Release Lane

  • Stable hotfix targeting release/2.6.x
  • Mainline change targeting main
  • Next-minor work targeting the next feature/minor release
  • Not sure; reviewer should help classify

Stable hotfix reason:

n/a

Changed Surface

  • Docs only
  • Tests only
  • Dashboard UI
  • Dashboard API / host agent
  • Installer / bootstrap / lifecycle
  • Docker Compose / service manifests
  • Model routing / Hermes / capabilities
  • Network exposure / auth / proxy
  • Dependencies / runtime wiring

(token-spy is a bundled extension service. One provider module and its new
tests; CI config is also touched.)

Risk And Validation

  • Risk level: Low
  • Validation run:
    • git diff --check
    • Markdown/link sanity for docs
    • Focused tests listed below
    • Dashboard lint/test/build
    • Extension audit / compose validation
    • Release-grade fleet or scoped hardware validation
    • Stable-lane patch validation, if targeting release/2.6.x

Commands/results:

$ python3 -m py_compile extensions/services/token-spy/providers/anthropic.py
py OK

$ python3 -m pytest tests/test_provider_pricing.py -q
52 passed

$ python3 -m pytest tests/ -q        # whole token-spy suite
72 passed, 1 skipped

# the new suite against origin/main's provider:
19 failed, 33 passed

Caveat — the numbers I added. These are the published per-1M-token rates I
used, and they are the part of this PR I would most like a second pair of eyes
on, since I am asserting data rather than fixing logic:

Row input output cache read cache write
claude-sonnet-3-7 3.00 15.00 0.30 3.75
claude-sonnet-3-5 3.00 15.00 0.30 3.75
claude-opus-3 15.00 75.00 1.50 18.75
claude-haiku-3 0.25 1.25 0.03 0.30

Sonnet 3.5/3.7 and Opus 3 reuse the rates already in the table for their
same-priced siblings (claude-sonnet-4, claude-opus-4), so only the Haiku 3
row is a genuinely new figure. If your pricing source says otherwise, change
the numbers — the normalisation and the tests stand independently of them.

Operational Change Check

get_model_pricing() runs inside the token-spy container when computing cost
for a recorded event. The change is additive: ids that already matched still
match the same row (asserted by
test_family_first_ids_are_untouched and by the OpenAI-table test), and ids
that matched nothing now match. No request path, schema, or database change —
though historical rows already stored with cost 0 stay 0; this fixes new
events, not the backfill.

  • This is not an operational change.
  • This is an operational change and validation is recorded above.
  • This is an operational change and validation is intentionally deferred for:

Notes For Reviewers

The deeper issue is that a pricing miss is invisible. Zero is both "free
local model" and "we don't know this one", and the UI cannot tell them apart. A
follow-up worth considering: have get_model_pricing report whether it matched,
and surface unpriced models in the usage view instead of summing them as $0. I
did not do it here because it touches the storage and the dashboard, and this
PR is about the models that should have priced all along. Say the word and I
will send it.

The bare claude-haiku catch-all is now nearly dead — every real Haiku id
has an explicit row. I left it rather than remove it, since it is the only
thing that would catch a future Haiku released before someone adds a row. It is
priced at 3.5 rates, so it would guess low for a newer model; if you would
rather it be removed so a new model reports zero and gets noticed, that is a
one-line change.

get_model_pricing() matched model ids against COST_TABLE by substring.
The table is keyed on Anthropic's post-Claude-4 spelling, where the
family comes before the generation:

    claude-haiku-4-5    matches  claude-haiku-4-5-20251001

Claude 3.x ids put the generation first, so none of them matched
anything and every lookup fell through to the zero default:

    claude-3-5-sonnet-20241022   ->  in=0.0  out=0.0
    claude-3-5-haiku-20241022    ->  in=0.0  out=0.0
    claude-3-opus-20240229       ->  in=0.0  out=0.0
    claude-3-haiku-20240307      ->  in=0.0  out=0.0

Token Spy is the service that tells an operator what their agents cost.
Reporting $0.00 for a real Claude 3.5 Sonnet session is not a missing
feature — it reads as "this was free", which is the one answer the
service must never give wrongly.

The table already carried a `claude-haiku-3-5` row priced at 0.80/4.00,
which is exactly Claude 3.5 Haiku's rate — written in a spelling
Anthropic never shipped, so it could never be reached.

Normalise generation-first ids onto the family-first form before the
lookup, which makes that existing row reachable, and add rows for the
tiers that had none: sonnet-3-7, sonnet-3-5, opus-3, haiku-3. Claude 4+
ids are unchanged by the normalisation.

haiku-3 gets its own row so the bare `claude-haiku` catch-all does not
price it at Haiku 3.5's rate — 0.25/1.25 against 0.80/4.00, a 3.2x
overstatement.

Adds tests/test_provider_pricing.py (52 assertions over every shipped
Claude id, both spellings, the catch-all boundary, and that non-Claude
models still miss) and runs it in the token-spy CI job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant