Feat/prompt caching - #137
Open
YoussefFadh wants to merge 2 commits into
Open
Conversation
The sourcehunt ReAct loop re-sends system + tool schemas + the full growing message history on every turn (hunter.py:1452) and never prunes, so the same large prefix is billed as fresh input tokens dozens of times per deep hunt. Nothing set a cache directive today (native.py:573). Add an opt-in, default-off cache policy to AsyncLLMClient.achat / achat_stream (cache_prefix, prompt_cache_key), plumbed through aask_text/aask_json and preserved across the reasoning/max_tokens retry rebuilds. _mark_cache_prefix places one rolling cache_control="ephemeral" breakpoint on the last message (a copy — caller objects are never mutated, so markers can't accumulate past Anthropic's 4-breakpoint cap) and NativeHunter.arun opts in with a per-hunt stable prompt_cache_key. This is a transport/billing hint only: the model receives byte-identical input, so findings are unchanged by construction. Inert on providers without caching (genai_pyo3 maps the string per-provider). Cached-token accounting was already wired (budget.py:389). Default-off means every existing caller keeps sending identical requests. Co-Authored-By: Claude <noreply@anthropic.com>
Reconstructing the last ChatMessage to attach cache_control dropped the
reasoning value paired with an assistant turn's thought_signature, so
Anthropic rejected the follow-up turn ("thinking blocks require one
reasoning value per thought signature"). A benchmark run surfaced this
on turn 2 of a deep hunt.
Mutate only the cache_control field in place instead: it is the one
field that is never part of the tokenized prompt, so toggling it is
invisible to the model and cannot drop reasoning_content,
thought_signatures, or raw_content_json. Clear stale markers on earlier
messages so the reused hunter history never exceeds one breakpoint.
Verified against the gateway: turn 1 writes cache, turn 2 reads it, no
400. Adds test_native_cache_prefix.py.
Co-Authored-By: Claude <noreply@anthropic.com>
jorge-garcia-le
approved these changes
Aug 10, 2026
jorge-garcia-le
left a comment
jorge-garcia-le
left a comment
Contributor
There was a problem hiding this comment.
This will require some extra works for fireworks it seems: https://docs.fireworks.ai/guides/prompt-caching
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add prompt caching to the sourcehunt hunt loop
Sourcehunt's hunt is a many-turn conversation, and every turn re-sent the whole context (system prompt, tool schemas, and the full running history) at full input price. That cost compounds fast on a deep hunt.
This change caches the repeated prefix so each turn reads it back from the model cache instead of paying to send it again. It's a transport-level hint only: the model still receives identical input, so it has no effect on what the hunt finds. It's opt-in and defaults off, so existing callers are unchanged, and it's inert on providers without caching. All four multi-turn loops (hunt, elaboration, exploitation, reverse-engineering) pick it up through arun.
Impact (measured before/after on our ground-truth CVEs, real Bedrock billing):
Changes: one helper plus two optional params in AsyncLLMClient (clearwing/llm/native.py), the hunt loop opting in (clearwing/sourcehunt/hunter.py), and tests.