OpenAI cost calculation ignores cached input tokens (over-costs cache hits)

[eeshsaxena]

Summary

calculateOpenAICost in core/llm/utils/calculateRequestCost.ts bills the full promptTokens at the standard input rate and never accounts for cached input tokens, so requests that hit OpenAI's prompt cache are over-costed. The Anthropic branch in the same file already handles cache tokens; the OpenAI branch does not.

Detail

OpenAI reports cached input as usage.prompt_tokens_details.cached_tokens, and prompt_tokens includes those cached tokens. Cached input is billed at a discount (for example gpt-4o cached input is half the standard input rate). The cost function charges every prompt token at full rate:

const inputCost = (usage.promptTokens / 1_000_000) * modelPricing.input;
// no use of usage.promptTokensDetails.cachedTokens

Compare calculateAnthropicCost, which reads usage.promptTokensDetails and prices cachedTokens / cacheWriteTokens at their own rates.

Effect

For an OpenAI request with cached input (common with long, stable system prompts), the reported cost is higher than the actual OpenAI charge — the cached portion is billed at full price instead of the cache-read discount.

Suggested direction

Give the OpenAI pricing table a cachedInput rate and subtract the cached tokens from the full-rate input, pricing them separately, the way the Anthropic branch does:

const cachedTokens = usage.promptTokensDetails?.cachedTokens ?? 0;
const uncachedInput = Math.max(0, usage.promptTokens - cachedTokens);
const inputCost = (uncachedInput / 1_000_000) * modelPricing.input
                + (cachedTokens / 1_000_000) * modelPricing.cachedInput;

Happy to open a PR with the cached-input rates for the models already listed if that direction sounds right.

[bunnysayzz]

Confirmed this in the code. calculateOpenAICost in core/llm/utils/calculateRequestCost.ts bills every prompt token at the full input rate and never reads usage.promptTokensDetails.cachedTokens, while calculateAnthropicCost in the same file already prices cache reads and writes separately. Since OpenAI reports prompt_tokens including the cached portion, cache hits are over-costed.

I am implementing the fix now on a fork branch: cached input priced at the cache read rate, with the remaining prompt tokens at the full input rate, plus a Cache Read line in the breakdown. Will post the branch and test results here once it is ready.

[bunnysayzz]

The fix is ready on the fork: branch fix/openai-cached-token-cost-13104 (commit ede9c61).

What changed in core/llm/utils/calculateRequestCost.ts:

• The OpenAI pricing table now includes a cachedInput rate for every listed model (50% of the input rate, matching OpenAI's published pricing for these models: gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo variants, gpt-4).
• The cost calculation reads usage.promptTokensDetails.cachedTokens, subtracts it from the full-rate input (Math.max(0, ...) guards against the cached count ever exceeding prompt_tokens), and prices the cached portion at the cache read rate.
• The breakdown now shows a Cache Read line, same style as the Anthropic branch.

Tests: 4 new cases added to calculateRequestCost.vitest.ts (partial cache hit on gpt-4o and gpt-4o-mini, fully cached input, and cached tokens exceeding prompt tokens). vitest run passes: 26/26. The changed files are type-clean under tsc --noEmit.

Happy to open the PR if this direction looks good.

[bunnysayzz]

While verifying the calc fix I found the deeper reason the cached tokens were never priced: OpenAI usage never reaches the cost calculation at all. Every conversion path dropped it:

• fromChatResponse ignored response.usage
• fromChatCompletionChunk ignored the final usage-carrying stream chunk
• the Responses API handlers ignored both response.completed and the final response usage

So interaction.end.usage was always undefined for the openai provider, the GUI showed no cost for OpenAI sessions, and cachedTokens never got as far as calculateOpenAICost. The branch now carries a second commit (3081820) that fixes the plumbing:

• New toOpenAIUsage helper maps both the Chat Completions shape (prompt_tokens / prompt_tokens_details.cached_tokens) and the Responses shape (input_tokens / input_tokens_details.cached_tokens) onto the Usage type.
• All three conversion paths now attach usage (non-stream response, final stream chunk, response.completed, final Responses response).
• The OpenAI provider requests stream_options.include_usage so the final streamed chunk actually carries usage. Gated to the OpenAI provider only: Azure gates it behind an api-version, and the OpenAI-compatible providers (Fireworks, Groq, DeepSeek, etc.) extend the same class with their own providerName so they are excluded.

Tests: 6 new cases in openaiTypeConverters.test.ts (both usage shapes, all three attachment paths). jest openaiTypeConverters.test.ts passes 30/30, and vitest run calculateRequestCost.vitest.ts still passes 26/26. Changed files are type-clean under tsc --noEmit.

Together the two commits make OpenAI cost tracking work end to end: usage now flows, cached input is priced at the cache read rate instead of full rate, and the breakdown shows the Cache Read line.

[bunnysayzz]

Third commit (cdb4ca6) covers the CLI surface, where the same over-costing was live. recordStreamTelemetry in extensions/cli/src/stream/streamChatResponse.helpers.ts read cached tokens from prompt_tokens_details.cache_read_tokens, a field neither OpenAI nor Anthropic sends: OpenAI reports prompt_tokens_details.cached_tokens, Anthropic reports cache_read_input_tokens at the usage top level. So cachedTokens was always undefined and cache hits were billed at the full input rate in the CLI cost display and telemetry.

The mapping now accepts both shapes (plus the Responses API input_tokens/output_tokens fields) and feeds the normalized values into both calculateRequestCost and the cacheRead/cacheCreation telemetry records. Regression tests added for the OpenAI, Anthropic, and Responses usage shapes.

One honest caveat: the CLI vitest suite could not run in this checkout because the CLI workspace install is incomplete (missing vitest, winston, diff, dotenv, and more workspace links). The changed file is type-clean under tsc --noEmit, and the cost math it delegates to is covered by the core suites (vitest 26/26, jest 30/30).