Problem
Alluka stores a complete session transcript as sessions.messages_json and replaces that JSON value at the end of every run. Session rendering also loads and validates the full history before displaying the chat. Long-lived sessions therefore grow read, validation, serialization, write, and first-render work together.
The chat route converts the client-provided full transcript into model messages. Older context is never compacted or budgeted, so prompt cost and latency increase continuously as a session grows.
Performance improvement
Make session history and model context bounded, incremental, and inspectable.
Session hydration should initially present a recent history window with clear older-history availability. Large artifact payloads should not be included in routine transcript hydration. Older messages can be requested intentionally without forcing a cold load of the complete session.
Model context should follow a deterministic budget that retains the original objective, recent exchange, and relevant durable constraints while compacting older history into an explicitly identified context record. Context metrics distinguish measured/provider-reported values from estimates.
Expected behavior
- Opening a long-lived session does not require transferring, parsing, validating, or rendering its full transcript before the latest conversation is usable.
- Users can load older messages predictably and retain chronological continuity.
- Session updates do not rewrite unrelated historical content.
- The model receives bounded context whose composition can be explained after the run.
- Original user goals and recent exchanges are preserved when older context is compacted.
- Context compaction is deterministic and does not silently call another model to summarize history.
- Historical artifacts remain retrievable without embedding large payloads in normal chat responses.
Why this matters
This protects responsiveness and inference cost as agent sessions become durable records instead of disposable chats. It also establishes the storage foundation for branching, checkpoints, timelines, and reliable pagination.
References
Problem
Alluka stores a complete session transcript as
sessions.messages_jsonand replaces that JSON value at the end of every run. Session rendering also loads and validates the full history before displaying the chat. Long-lived sessions therefore grow read, validation, serialization, write, and first-render work together.The chat route converts the client-provided full transcript into model messages. Older context is never compacted or budgeted, so prompt cost and latency increase continuously as a session grows.
Performance improvement
Make session history and model context bounded, incremental, and inspectable.
Session hydration should initially present a recent history window with clear older-history availability. Large artifact payloads should not be included in routine transcript hydration. Older messages can be requested intentionally without forcing a cold load of the complete session.
Model context should follow a deterministic budget that retains the original objective, recent exchange, and relevant durable constraints while compacting older history into an explicitly identified context record. Context metrics distinguish measured/provider-reported values from estimates.
Expected behavior
Why this matters
This protects responsiveness and inference cost as agent sessions become durable records instead of disposable chats. It also establishes the storage foundation for branching, checkpoints, timelines, and reliable pagination.
References