fix(token-spy): price Claude 3.x models instead of billing them at zero - #2511
Open
Hoang130203 wants to merge 1 commit into
Open
fix(token-spy): price Claude 3.x models instead of billing them at zero#2511Hoang130203 wants to merge 1 commit into
Hoang130203 wants to merge 1 commit into
Conversation
get_model_pricing() matched model ids against COST_TABLE by substring.
The table is keyed on Anthropic's post-Claude-4 spelling, where the
family comes before the generation:
claude-haiku-4-5 matches claude-haiku-4-5-20251001
Claude 3.x ids put the generation first, so none of them matched
anything and every lookup fell through to the zero default:
claude-3-5-sonnet-20241022 -> in=0.0 out=0.0
claude-3-5-haiku-20241022 -> in=0.0 out=0.0
claude-3-opus-20240229 -> in=0.0 out=0.0
claude-3-haiku-20240307 -> in=0.0 out=0.0
Token Spy is the service that tells an operator what their agents cost.
Reporting $0.00 for a real Claude 3.5 Sonnet session is not a missing
feature — it reads as "this was free", which is the one answer the
service must never give wrongly.
The table already carried a `claude-haiku-3-5` row priced at 0.80/4.00,
which is exactly Claude 3.5 Haiku's rate — written in a spelling
Anthropic never shipped, so it could never be reached.
Normalise generation-first ids onto the family-first form before the
lookup, which makes that existing row reachable, and add rows for the
tiers that had none: sonnet-3-7, sonnet-3-5, opus-3, haiku-3. Claude 4+
ids are unchanged by the normalisation.
haiku-3 gets its own row so the bare `claude-haiku` catch-all does not
price it at Haiku 3.5's rate — 0.25/1.25 against 0.80/4.00, a 3.2x
overstatement.
Adds tests/test_provider_pricing.py (52 assertions over every shipped
Claude id, both spellings, the catch-all boundary, and that non-Claude
models still miss) and runs it in the token-spy CI job.
23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_model_pricing()matches model ids againstCOST_TABLEby substring. Thetable is keyed on Anthropic's post-Claude-4 spelling, where the family comes
before the generation:
Claude 3.x ids put the generation first, so none of them match anything and
every lookup falls through to the zero default. Run against
origin/main:Token Spy is the service that tells an operator what their agents cost.
Reporting $0.00 for a real Claude 3.5 Sonnet session is not a missing
feature — it reads as "this was free", which is the one answer a cost tracker
must never give wrongly. And it fails quietly: nothing in the UI distinguishes
"a local model, genuinely free" from "we could not price this".
The give-away that this is a bug and not a policy choice is already in the
table:
0.80 / 4.00 is exactly Claude 3.5 Haiku's rate — someone priced that tier
deliberately, wrote it in a spelling Anthropic never shipped, and it has been
unreachable ever since.
Fix
lookup.
claude-3-5-haiku-20241022→claude-haiku-3-5-20241022, whichmakes the row above reachable without inventing a price. Claude 4+ ids are
left byte-identical.
sonnet-3-7,sonnet-3-5,opus-3,haiku-3.haiku-3gets its own row so the bareclaude-haikucatch-all does notswallow it. That catch-all is priced at Haiku 3.5's rate, so without an
explicit row the fix would have billed Haiku 3 at 0.80/4.00 instead of
0.25/1.25 — a 3.2× overstatement. Turning a silent undercount into a silent
overcount would not be an improvement.
After:
Test
providers/had no test coverage — nothing underextensions/services/token-spy/tests/referenced it. Addstests/test_provider_pricing.py: 52 assertions over every shipped Claude id,both spellings resolving to the same row, output ≥ input and cache_read < input
as typo guards, the Haiku 3 / 3.5 boundary, that non-Claude ids still miss, and
that
calculate_costuses the row the lookup returned. Wired into the existingtoken-spy CI job.
AI Assistance
AI assisted with the normalisation regex, drafting the test matrix, and wording
this description. I found the mismatch by running the shipped table against
real model ids, and the four prices I added are the published rates listed
below for you to check.
Release Lane
release/2.6.xmainStable hotfix reason:
Changed Surface
(
token-spyis a bundled extension service. One provider module and its newtests; CI config is also touched.)
Risk And Validation
git diff --checkrelease/2.6.xCommands/results:
Caveat — the numbers I added. These are the published per-1M-token rates I
used, and they are the part of this PR I would most like a second pair of eyes
on, since I am asserting data rather than fixing logic:
claude-sonnet-3-7claude-sonnet-3-5claude-opus-3claude-haiku-3Sonnet 3.5/3.7 and Opus 3 reuse the rates already in the table for their
same-priced siblings (
claude-sonnet-4,claude-opus-4), so only the Haiku 3row is a genuinely new figure. If your pricing source says otherwise, change
the numbers — the normalisation and the tests stand independently of them.
Operational Change Check
get_model_pricing()runs inside thetoken-spycontainer when computing costfor a recorded event. The change is additive: ids that already matched still
match the same row (asserted by
test_family_first_ids_are_untouchedand by the OpenAI-table test), and idsthat matched nothing now match. No request path, schema, or database change —
though historical rows already stored with cost 0 stay 0; this fixes new
events, not the backfill.
Notes For Reviewers
The deeper issue is that a pricing miss is invisible. Zero is both "free
local model" and "we don't know this one", and the UI cannot tell them apart. A
follow-up worth considering: have
get_model_pricingreport whether it matched,and surface unpriced models in the usage view instead of summing them as $0. I
did not do it here because it touches the storage and the dashboard, and this
PR is about the models that should have priced all along. Say the word and I
will send it.
The bare
claude-haikucatch-all is now nearly dead — every real Haiku idhas an explicit row. I left it rather than remove it, since it is the only
thing that would catch a future Haiku released before someone adds a row. It is
priced at 3.5 rates, so it would guess low for a newer model; if you would
rather it be removed so a new model reports zero and gets noticed, that is a
one-line change.