fix(token-spy): stop billing OpenAI cached tokens twice

[Hoang130203]

Summary

LLMProvider.calculate_cost() bills every *_tokens field independently:

return (
    usage.get("input_tokens", 0) * rates.get("input", 0) / 1_000_000 +
    usage.get("output_tokens", 0) * rates.get("output", 0) / 1_000_000 +
    usage.get("cache_read_tokens", 0) * rates.get("cache_read", 0) / 1_000_000 +
    usage.get("cache_write_tokens", 0) * rates.get("cache_write", 0) / 1_000_000
)

so the fields a provider reports must be disjoint. Anthropic's API returns
them that way — input_tokens excludes cache_read_input_tokens — and the
Anthropic provider passes them straight through.

OpenAI's prompt_tokens is a total that includes
prompt_tokens_details.cached_tokens, and the provider reported it verbatim:

"input_tokens": usage.get("prompt_tokens", 0),
"cache_read_tokens": usage.get("prompt_tokens_details", {}).get("cached_tokens", 0),

Every cached token was therefore charged twice — once at the full input rate,
once again at the cache-read rate.

Measured on gpt-4o, 1M prompt tokens at a 90% cache hit rate:

OpenAI response usage: {'input_tokens': 1000000, 'cache_read_tokens': 900000}
  reported cost = $3.6250
  correct cost  = $1.3750   (uncached 100k @ 2.5, cached 900k @ 1.25)
  overstated by 2.64x

Anthropic response usage: {'input_tokens': 100000, 'cache_read_tokens': 900000}
  reported cost = $0.5700
  correct cost  = $0.5700  <- matches

The same codebase getting it right for one provider and wrong for the other is
what convinced me this is a bug rather than a convention I had misread.

The error grows with how well caching is working. Prompt caching pays off
precisely when the hit rate is high, so a user who tunes their prompts for cache
reuse watches their reported spend increase. For a service whose whole purpose
is telling you what your agents cost, that is the wrong direction to be wrong
in — and it is wrong on the cheap path, not the expensive one.

Fix

A small _split_cached() helper subtracts the cached count from the total, in
both the response and stream paths, so the two fields stop overlapping. The
count is clamped to 0..total, so an impossible or non-numeric cached_tokens
cannot produce a negative input count or a negative bill.

after:
  usage:  {'input_tokens': 100000, 'output_tokens': 0, 'cache_read_tokens': 900000}
  cost:   $1.3750
  stream: {'input_tokens': 200, 'output_tokens': 50, 'cache_read_tokens': 800}
  no-cache response: input_tokens=500, cache_read_tokens=0   (unchanged)

Test

Adds tests/test_cached_token_accounting.py — 13 assertions: the split, a
hand-computed bill, the streaming path, the no-cache and null-details paths
asserted unchanged, impossible cached counts (1500 of 1000, -5,
"many", None), a missing usage block, and a cross-provider check that
OpenAI and Anthropic now report the same disjoint numbers for the same
conversation. Wired into the token-spy CI job.

AI Assistance

AI assisted with the clamping helper, the test matrix, and wording this
description. I found it by reading calculate_cost and asking what each
provider's input_tokens actually contains, then computing the bill both ways
against the shipped code.

Release Lane

• Stable hotfix targeting release/2.6.x
• Mainline change targeting main
• Next-minor work targeting the next feature/minor release
• Not sure; reviewer should help classify

Stable hotfix reason:

n/a

Changed Surface

• Docs only
• Tests only
• Dashboard UI
• Dashboard API / host agent
• Installer / bootstrap / lifecycle
• Docker Compose / service manifests
• Model routing / Hermes / capabilities
• Network exposure / auth / proxy
• Dependencies / runtime wiring

(token-spy is a bundled extension service. One provider module and its new
tests; CI config is also touched.)

Risk And Validation

• Risk level: Low
• Validation run:
  • git diff --check
  • Markdown/link sanity for docs
  • Focused tests listed below
  • Dashboard lint/test/build
  • Extension audit / compose validation
  • Release-grade fleet or scoped hardware validation
  • Stable-lane patch validation, if targeting release/2.6.x

Commands/results:

$ python3 -m py_compile extensions/services/token-spy/providers/openai.py
py OK

$ python3 -m pytest tests/test_cached_token_accounting.py -q
13 passed

$ python3 -m pytest tests/ -q          # whole token-spy suite
33 passed, 1 skipped

# the new suite against origin/main's provider:
FAILED test_cached_tokens_are_excluded_from_input
FAILED test_input_plus_cache_reconstructs_the_reported_total
FAILED test_cost_matches_a_hand_computed_bill
FAILED test_stream_usage_excludes_cached_from_input
FAILED test_null_prompt_tokens_details_is_tolerated
FAILED test_impossible_cached_counts_never_produce_negative_input[1500]
FAILED test_impossible_cached_counts_never_produce_negative_input[-5]
FAILED test_impossible_cached_counts_never_produce_negative_input[many]
FAILED test_both_providers_report_disjoint_fields
9 failed, 4 passed

The four that pass either way are the no-cache paths, which is the point — this
must not disturb a response without cache details.

Operational Change Check

extract_usage_from_response / extract_usage_from_stream run inside the
token-spy container when recording a completion. The change alters the
recorded input_tokens for OpenAI-compatible responses that carry cache
details, and therefore the computed cost — downward, to the correct figure.
Responses without prompt_tokens_details are byte-identical to before.

Two things worth knowing:

• Rows already stored keep their old numbers. This fixes new events; it is
  not a backfill. Historical OpenAI-with-cache costs remain overstated.

• Anyone reconstructing a prompt total from stored rows must now add
  input_tokens + cache_read_tokens. That was already true for Anthropic
  rows, so this makes the two consistent rather than introducing a new rule —
  but if a dashboard query assumes input_tokens is the total, it will now
  under-report for OpenAI. I grepped the token-spy dashboard and report code and
  did not find such a query; worth a second look from someone who knows those
  views better than I do.

• This is not an operational change.

• This is an operational change and validation is recorded above.

• This is an operational change and validation is intentionally deferred for:

Notes For Reviewers

cache_write stays 0 for OpenAI — the API does not expose a cache-write
count, and the table prices it at 0.0 for every OpenAI row, so nothing changes
there.

DeepSeek and Moonshot go through this same provider and both report
OpenAI-shaped usage with cache fields. DeepSeek's prompt_cache_hit_tokens /
prompt_cache_miss_tokens are a different pair of keys that this code does
not read at all, so DeepSeek caching is currently invisible rather than
double-counted. That is a separate gap and I did not fix it here — say the word
and I will send it, since the pricing rows for deepseek-chat and
deepseek-reasoner already carry cache_read rates that nothing can currently
reach.

Related: #2511 fixes the Claude 3.x rows in the Anthropic pricing table.
Different file, no overlap.