Skip to content

Add bounded cache-aware Hugging Face routing - #599

Draft
KirschBluteX wants to merge 1 commit into
huggingface:mainfrom
KirschBluteX:codex/issue-572-hf-cache-routing
Draft

Add bounded cache-aware Hugging Face routing#599
KirschBluteX wants to merge 1 commit into
huggingface:mainfrom
KirschBluteX:codex/issue-572-hf-cache-routing

Conversation

@KirschBluteX

Copy link
Copy Markdown

Summary

Tau's existing Hugging Face routing pins the first successful backing provider for a session. That preserves locality, but it can also keep an automatic session on a route that repeatedly reports no effective cache reuse for a long, stable prefix.

This adds a fixed, bounded evaluator for automatic Hugging Face routes:

  • requests become eligible at 4,096 prompt tokens;
  • the first eligible request on each route is a cold warm-up;
  • two later append-only, comparable responses without positive reported reuse trigger another candidate;
  • any positive reported cache read retains the route;
  • evaluation stops after three routes or nine eligible requests.

Candidates come from the model API's live conversational mappings and are ordered deterministically. Discovery failures, unavailable candidates, and exhausted budgets leave the current route usable and surface a diagnostic instead of looping.

The policy remains in tau_coding. tau_agent gains only a provider-neutral flag that preserves whether a cache-read counter was reported. Automatic state and transitions use the existing session index and custom-entry mechanisms, /session reports the evaluator state, and route events render in transcript, final-text, JSON, and TUI output. Configured or extension-selected routes remain locked; /route automatic explicitly resets evaluation. Existing pinned records without route-source metadata remain explicit.

If route-state persistence fails after a completed response, Tau keeps the previous route and runtime, records a diagnostic, and still settles the completed turn. Route-specific pricing remains separate in #574.

Closes #572.

Validation

  • uv run pytest -q on Linux - 1533 passed, 3 skipped
  • focused affected suite - 162 passed
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run mypy on Linux - no issues in 102 source files
  • hugo --minify with Hugo 0.152.2 - 33 pages built
  • pagefind --site public with Pagefind 1.5.2 - 28 pages and 3,255 words indexed
  • git diff origin/main...HEAD --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adapt Hugging Face routing when a backend lacks effective prompt caching

1 participant