Skip to content

Fix Bedrock Converse input_tokens double-subtracting cache tokens#832

Merged
crmne merged 3 commits into
crmne:mainfrom
jmangel:fix/bedrock-converse-input-tokens-cache
Jul 3, 2026
Merged

Fix Bedrock Converse input_tokens double-subtracting cache tokens#832
crmne merged 3 commits into
crmne:mainfrom
jmangel:fix/bedrock-converse-input-tokens-cache

Conversation

@jmangel

@jmangel jmangel commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #828.

Problem

Converse::Chat#input_tokens subtracts cacheReadInputTokens and cacheWriteInputTokens from inputTokens. But AWS Bedrock's inputTokens is already the non-cached count. The cache buckets are reported separately, not folded in. Per the AWS prompt-caching docs:

When prompt caching is enabled, the inputTokens field represents only the non-cached input tokens... total input tokens = inputTokens + cacheReadInputTokens + cacheWriteInputTokens

So the subtraction removes tokens that were never there, understating input on every cache hit and flooring it to 0 whenever the cached prefix exceeds the fresh input (the common multi-turn case). Any cost or usage tracking on message.input_tokens under-reports.

Why pass-through is correct

The Anthropic provider already handles the same Claude models correctly: anthropic/chat.rb passes input_tokens through raw and reads the cache buckets separately. Bedrock serves those same models with the same semantics, so this just brings Converse into line. The same fix landed in LiteLLM (#15292) and was reported in LangSmith (#1858).

Change

  • input_tokens returns usage['inputTokens'] unchanged. cached_tokens and cache_creation_tokens are still parsed separately, so nothing is lost. Streaming delegates to this method, so both paths are covered.
  • Updated the converse chat spec. The previous test asserted the subtraction on a payload that cannot occur under AWS semantics (inputTokens already excludes cache), so its fixture is corrected to a realistic one, and a regression is added for the floor-to-zero case.

Verified: bundle exec rspec spec/ruby_llm/protocols/converse/ plus spec/ruby_llm/providers/bedrock_spec.rb green (39 examples), rubocop clean.

jmangel and others added 2 commits July 2, 2026 22:00
AWS Bedrock reports inputTokens as already non-cached; cacheReadInputTokens
and cacheWriteInputTokens are separate buckets, not folded in. Subtracting
them (as inclusive providers like OpenAI/Gemini require) understated input
and floored it to zero on cache-heavy turns. Pass inputTokens through, as
the Anthropic provider already does for the same Claude models.

Updates the converse chat spec: the prior test encoded the subtraction on a
payload that can't occur (inputTokens already excludes cache per AWS), plus a
regression for the floor-to-zero case.
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.17%. Comparing base (5e252ea) to head (bf4bbfb).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #832      +/-   ##
==========================================
- Coverage   82.18%   82.17%   -0.02%     
==========================================
  Files         170      170              
  Lines        8029     8027       -2     
  Branches     1347     1347              
==========================================
- Hits         6599     6596       -3     
- Misses        890      891       +1     
  Partials      540      540              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@crmne crmne merged commit dc79ce6 into crmne:main Jul 3, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Bedrock double-subtracts cache tokens from input_tokens (AWS inputTokens already excludes cache)

2 participants