When I run an AI agent (Hermes) with lots of tools, every turn feels like starting over. The server re-processes the whole prompt — system message, tool list, and chat history — even when only the latest message changed. That can take 10+ seconds of prefill on a ~8K–20K token prompt.
It gets worse because one user action triggers several model calls (main chat, tool results, title generation, etc.). They share the same tools but different conversation text, so the cache keeps getting wiped and rebuilt.
Lucebox already has prefix caching, but it treats the prompt as one big block. At 128K context, snapshot caching also failed on 24GB GPUs because snapshots tried to live in VRAM and ran out of memory — so caching often never kicked in at all.
What I think the solution is
Split the prompt into two cached parts:
- Anchor — system + tools + history (the big, mostly-static part) — restore from snapshot instead of recomputing
- Delta — the new user message or tool result — prefill only this small tail
Pin tool schemas in their own snapshot slot (keyed by tool fingerprint) so they survive when conversation cache slots change. Restore both with RESTORE_CHAIN, then prefill the delta. Store snapshots in CPU RAM at the right size so 128K context works without OOM.
On dual RTX 3090s this cut warm-turn prefill from ~11s to ~0.5s (~20×) and wall time from ~44s to ~4–5s on agent tool sessions.
PR - Anchored Prefix Caching for multi-turn agent workloads: #492
When I run an AI agent (Hermes) with lots of tools, every turn feels like starting over. The server re-processes the whole prompt — system message, tool list, and chat history — even when only the latest message changed. That can take 10+ seconds of prefill on a ~8K–20K token prompt.
It gets worse because one user action triggers several model calls (main chat, tool results, title generation, etc.). They share the same tools but different conversation text, so the cache keeps getting wiped and rebuilt.
Lucebox already has prefix caching, but it treats the prompt as one big block. At 128K context, snapshot caching also failed on 24GB GPUs because snapshots tried to live in VRAM and ran out of memory — so caching often never kicked in at all.
What I think the solution is
Split the prompt into two cached parts:
Pin tool schemas in their own snapshot slot (keyed by tool fingerprint) so they survive when conversation cache slots change. Restore both with
RESTORE_CHAIN, then prefill the delta. Store snapshots in CPU RAM at the right size so 128K context works without OOM.On dual RTX 3090s this cut warm-turn prefill from ~11s to ~0.5s (~20×) and wall time from ~44s to ~4–5s on agent tool sessions.
PR - Anchored Prefix Caching for multi-turn agent workloads: #492