Skip to content

feat: inject anthropic prompt cache control#162

Merged
shudonglin merged 3 commits into
mainfrom
feat/anthropic-cache-control-injection
Jun 27, 2026
Merged

feat: inject anthropic prompt cache control#162
shudonglin merged 3 commits into
mainfrom
feat/anthropic-cache-control-injection

Conversation

@shudonglin

Copy link
Copy Markdown

Summary

Adds gateway-side Anthropic automatic prompt-cache control for native Anthropic requests when the client has not already supplied cache_control.

Changes

  • Adds ANTHROPIC_PROMPT_CACHE_TTL support with off, 5m, 1h, and auto policies
  • Applies top-level cache_control on typed and raw Anthropic Messages paths, streaming paths, and Anthropic Responses requests
  • Preserves any client-authored cache_control marker anywhere in the request body
  • Strips gateway-only prompt-cache policy headers before Anthropic passthrough forwarding
  • Adds focused unit tests for injection, preservation, opt-out, auto workload handling, and raw body mutation

Type of change

  • Bug fix
  • Feature
  • Refactor
  • Documentation
  • Chore/CI

Affected areas

  • Core (Go)
  • Transports (HTTP)
  • Providers/Integrations
  • Plugins
  • UI (React)
  • Docs

How to test

go test ./core/providers/anthropic -run 'TestApplyAnthropicPromptCacheControl'

Expected outcome: the Anthropic provider tests pass and cover top-level injection, client override preservation, header opt-out, auto workload TTL selection, and raw-body injection.

New configuration:

ANTHROPIC_PROMPT_CACHE_TTL=off|5m|1h|auto
x-anthropic-prompt-cache-ttl: off|5m|1h|auto
x-anthropic-prompt-cache-workload: eval|evaluation|benchmark|bench|batch|pipeline|long|long-running

Screenshots/Recordings

Not applicable; backend provider behavior only.

Breaking changes

  • Yes
  • No

Related issues

None

Security considerations

The request headers used to control gateway policy are stripped before Anthropic passthrough forwarding. Client-provided cache_control remains authoritative and is not overwritten.

Checklist

  • I read docs/contributing/README.md and followed the guidelines
  • I added/updated tests where appropriate
  • I updated documentation where needed
  • I verified builds succeed (Go and UI)
  • I verified the CI pipeline passes locally if applicable

@shudonglin shudonglin marked this pull request as ready for review June 27, 2026 16:57
@shudonglin

Copy link
Copy Markdown
Author

E2E cache validation

Ran on 2026-06-28 from branch feat/anthropic-cache-control-injection:

go test -count=1 ./core/providers/anthropic -run 'TestAnthropicProviderChatCompletionE2EPromptCacheControl|TestApplyAnthropicPromptCacheControl'

Result: passed (ok github.com/maximhq/bifrost/core/providers/anthropic 1.001s).

The E2E test uses a mock Anthropic upstream, so it validates gateway request/response behavior without spending real Anthropic tokens. It verifies top-level cache_control: {"type":"ephemeral","ttl":"1h"} is sent for long/eval workloads, gateway-only cache policy headers are stripped before upstream, client-supplied cache control is preserved, and Anthropic cache usage fields flow back through response handling.

@shudonglin shudonglin merged commit 821d9cd into main Jun 27, 2026
5 checks passed
@shudonglin shudonglin deleted the feat/anthropic-cache-control-injection branch June 27, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant