Skip to content

Add cacheflow prompt-cache simulator - #9

Merged
ron2111 merged 1 commit into
masterfrom
feature/cacheflow-simulator
Jul 11, 2026
Merged

Add cacheflow prompt-cache simulator#9
ron2111 merged 1 commit into
masterfrom
feature/cacheflow-simulator

Conversation

@ron2111

@ron2111 ron2111 commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a new dependency-free cacheflow package for token-aware prompt-cache planning and trace simulation.

This intentionally moves cache APIs out of the root omnitoken package because cache simulation can grow independently from the tokenizer core.

What changed

  • Added cacheflow/ package.
  • Moved single-prompt boundary planning into cacheflow.NewAligner / cacheflow.AlignTokenCount.
  • Added JSONL prompt trace support with cacheflow.ReadJSONL.
  • Added stable token-prefix simulation with cacheflow.Simulate.
  • Added best-effort cache-breaker hints for timestamps, UUID/request IDs, and dynamic JSON metadata.
  • Added omni cache-sim CLI.
  • Updated omni cache to use cacheflow.
  • Added cacheflow/README.md and linked it from README/CLI docs.
  • Removed old root cache API and docs/cache.md per the requested no-backward-compatibility cleanup.

Issues

Addresses the design and MVP direction for:

Verification

  • go test ./...
  • go vet ./...
  • adapter module tests for Gemini, Hugging Face, Llama3, Mistral, OSS, and Anthropic
  • CLI smoke tests:
    • omni cache
    • omni cache-sim -input prompts.jsonl -breakers -json

Notes

No new dependencies were added. Cacheflow remains a local planning/simulation tool and does not claim provider billing parity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant