Proposal: cache-aware conversational sentinel support (Anthropic prompt cache)

[edwinow]

Preface: not opening PR as this is not merge ready / sharing early as a working starting point for the Anthropic cache path, not a complete cross-provider abstraction

branch on my fork
feat(sentinels): cache-aware conversational sentinel support

If easier to diff and you'd like me to open a PR anyways then let me know!

---

Summary

Adds cache-aware conversational sentinel support: a send-time message
transform that engages Anthropic prompt-cache reads across fires, a
chunkedWindow trim strategy that keeps the cached prefix stable
between shift events, and cost accounting that surfaces cache hits
through the existing sentinel.output event payload.

On release/alpha today, conversational sentinels get zero cache
reads — every fire pays full input-token price for the entire
retained history. This PR approaches the theoretical ~10× ceiling at
production scale (measured cached cost vs cost-model-projected uncached
baseline; see "Empirical evidence" below).

Sharing as draft / WIP: the Anthropic path is empirically verified
and locally complete, but I have not designed or tested Gemini/Google
cache semantics and have not run a live regression matrix across the
other Pi providers. See "What is and isn't covered" for the honest
scope.

Why this matters

Conversational sentinels are by definition multi-fire — same observer
re-running over many events. Without cache engagement, every fire pays
the full input-token cost for the entire retained history. With cache
engagement, fires past the first read the prefix at cache-read pricing
(10% of input on Anthropic), which is the single biggest cost lever
available for this class of workload.

Empirical evidence

Production-scale measurement (M=70, S=20, 100 fires, ~1000
tokens/turn, real Anthropic API, Haiku 4.5):

	Cost	Notes
with this PR (lastMessage)	$0.553	97/100 fires hit cache; real-API, fully measured
release/alpha today (no cache reads)	~$5.50	projected from input-token pricing at the same workload

Net: ~10× cheaper than release/alpha at production scale.

Honest scope of that headline: the cached side ($0.553) is real-API,
fully measured. The uncached baseline (~$5.50) is a cost-model
projection at the same workload — I did not run an uncached 100-fire
M=70 production-scale comparison separately (would have cost another
~$5–6 of API spend). Treat the ratio as "measured cached vs projected
uncached," not "measured both sides."

What's in the PR

Three logical commits on top of release/alpha:

1. feat(sentinels): schema widening for cache-aware conversational sentinels — type and schema changes only, no runtime behavior change. providerOptions plumbing matches AI SDK v5's SharedV2ProviderOptions; chunkedWindow added as third branch in trimmingStrategy discriminated union; conversational.cache config block with breakpoint: "lastMessage"; new cachedInputTokens / cacheCreationInputTokens fields on the sentinel.output event payload.

2. feat(sentinels): cache-aware conversational sentinel runtime + tests — the runtime. applyAnthropicCacheControlBreakpoint send-time helper converts retained history to AI-SDK-v5 parts-form and stamps cache_control on the LAST historical user/assistant message before the live current user is pushed. chunkedWindow trim strategy in history-manager.ts. Cache-aware computeCost helper used across all four sentinel.ts LLM call sites. 84-test unit suite covering placement, parts-form conversion, chunkedWindow shift semantics, cost math, and providerOptions spread-survives at all four AI-SDK call surfaces. Integration test (describe.skip by default) verifies the providerMetadata.anthropic.cacheCreationInputTokens extraction path against the live API.

3. docs(examples): cache-aware conversational sentinel demo — runnable example under learning/examples/cache-aware-conversational-sentinel/ with the 4-cell experimental matrix, a dry-run estimator (no API spend by default), and an opt-in --real-api path. The example's results/ folder is gitignored.

What is and isn't covered

Covered

• Anthropic cache-control path — deterministic unit + integration tests, live-API verification, real cost numbers above.
• OpenAI-style providerOptions shape — schema matches AI SDK v5's SharedV2ProviderOptions. Spread-survives regression tests at all four construction sites in sentinel.ts (conv text, non-conv text, conv structured, non-conv structured).
• Backwards compatibility — existing maxTurns and maxTokens trimming strategies unchanged. Existing implementation-observer.json example unchanged. Schema regeneration is additive only.

Intentionally not covered (WIP scope boundary)

• Gemini / Google cache semantics — not designed, not tested. Gemini's cache works differently (separate cached_content resource with explicit lifecycle) and probably needs its own provider plug-in rather than the same cache_control-style annotation.
• Cross-provider cache abstraction — this is not a unified caching API; it's the minimum needed to make Anthropic caching actually engage for conversational sentinels.
• Pi provider live-regression matrix beyond pi/openai-codex/gpt-5.5 — local shim validation passes and Anthropic live-API works, but I have not run a full live matrix across the other Pi providers.
• Parallel-fire cache-race within a single sentinel — not implemented. Between different sentinels there's no interference (independent cache keys). But if multiple parallel fires of the same sentinel race on a fresh prefix, they could each pay cache_write before any one populates the cache. The maximally-cache-efficient pattern would be "fire the first alone, wait for it to complete, then fan out the rest in parallel"; we did not implement this throttle. Niche enough that I judged it not worth the added complexity for this PR. Happy to add if maintainers want.
• eta tags in sentinel system prompts — would silently break caching (template renders differently per fire → different prefix → no cache hit). README documents this as a known footgun. Example uses a static system prompt.
• Haiku 4.5's 4096-token cache minimum — demo's natural history clears the floor after ~4 turns at 250 tokens/turn. Documented in README.

Test status

• bun run tc clean (Type check passed ✅)
• bun run lint clean (Checked 253 files. No fixes applied.)
• bun run build clean (CLI bundle 2129.55 KB minified, type declarations generated)
• Cache-aware unit suite: 84 pass / 0 fail / 175 expect() calls / 1.49s across cache-control.test.ts, history-manager.test.ts, sentinel-cost.test.ts, sentinel-manager.test.ts, sentinel-provider-options.test.ts.
• tests/integration/sentinel-cache.test.ts is describe.skip by default; un-skipping requires ANTHROPIC_API_KEY. Passes against live API: cacheCreationInputTokens > 0 on fire 1, cachedInputTokens > 0 on fire 2.