[FEATURE] Conversation compaction to manage context limits

[jwesleye]

Feature Description

Automatically compact or summarize older parts of long conversations to stay within context window limits while preserving important information.

Problem/Motivation

AWS Strands agents have context window limits (e.g., 200K tokens for Claude Sonnet 4.5). During long sessions:

• Conversations can exceed the context limit
• Users may lose access to older parts of the conversation
• No warning when approaching limits
• No automatic management of conversation length

Proposed Solution

Core Features

1. Context monitoring - Track current conversation token usage
2. Limit warnings - Alert users when approaching context limits (80%, 90%, 95%)
3. Manual compaction - compact command to summarize older messages
4. Auto-compaction - Optional automatic compaction when reaching threshold
5. Smart summarization - Preserve key information while reducing token count

Implementation Options

Option A: Rolling Window

• Keep last N messages in full detail
• Summarize or remove older messages
• Simple, predictable behavior

Option B: Intelligent Summarization

• Use the agent itself to summarize old conversation chunks
• Preserve important context (code, decisions, key facts)
• More sophisticated but higher quality

Option C: Hybrid

• Keep recent messages (last 20-30) in full
• Summarize middle sections in chunks
• Drop very old messages beyond threshold

User Experience

# During conversation - warning appears
⚠️  Context usage: 180K / 200K tokens (90%)
Consider using 'compact' command to summarize older messages.

# Manual compaction
You: compact
Compacting conversation (keeping last 30 messages, summarizing 50 older messages)...
✓ Reduced from 180K to 95K tokens (47% reduction)
✓ Preserved 30 recent messages + summary of earlier conversation

# View context status
You: context
Current usage: 95K / 200K tokens (47%)
Messages: 80 total (30 full + 1 summary block)
Oldest message: 2 hours ago

Configuration

# In ~/.chatrc
context:
  max_tokens: 200000          # Model's context limit
  warning_thresholds: [0.8, 0.9, 0.95]  # Show warnings at 80%, 90%, 95%
  auto_compact: false         # Enable automatic compaction
  auto_compact_threshold: 0.85  # Compact at 85% if auto enabled
  preserve_recent: 30         # Always keep last N messages in full
  compaction_method: hybrid   # rolling | summarize | hybrid

Benefits

• ✅ Never hit context limits unexpectedly
• ✅ Maintain usable conversation history
• ✅ Clear visibility into context usage
• ✅ User control over compaction strategy
• ✅ Preserve important information

Related Commands

• context - Show current context usage and statistics
• compact - Manually trigger compaction
• compact --preview - Preview what would be compacted
• compact --method=<rolling|summarize> - Choose compaction strategy

Technical Considerations

Token Counting:

• Need accurate token counting for current conversation
• Use tiktoken or similar for Claude/GPT models
• Track running total as conversation progresses

Summarization Quality:

• Test different summarization prompts
• Ensure key information preserved (code, decisions, facts)
• Include metadata (timestamp, message count) in summaries

Backward Compatibility:

• Make all features opt-in initially
• Graceful degradation if token counting unavailable
• Don't break existing sessions

Edge Cases:

• Very first message after compaction (context reset)
• Session resume after compaction
• Saving/loading compacted sessions

Priority

• Critical
• High - Prevents frustrating context limit errors
• Medium
• Low

Dependencies

• Token counting library (tiktoken, anthropic tokenizer)
• Optional: summarization prompt engineering
• Configuration system (already exists)

Testing Plan

1. Test with conversations of various lengths
2. Verify token counting accuracy
3. Test summarization quality
4. Ensure session save/resume works with compaction
5. Performance test with very long conversations

Future Enhancements

• Semantic chunking (group related messages)
• Important message pinning (never compact)
• Compaction history tracking
• Export compacted conversations with summaries

[jwesleye]

Implementation Update - v1.6.0

We've successfully implemented the core session compaction and resume functionality! 🎉

✅ Completed Features (v1.6.0)

Manual Compaction:

• ✅ compact command to save current session and continue with summary
• ✅ Smart summarization using the agent itself
• ✅ Hybrid approach: Progressive summarization with n-1 chain model
• ✅ Auto-generate structured summaries on exit when auto_save: true
• ✅ Session resume using summaries instead of query replay (fast, token-efficient)
• ✅ Graceful error handling for missing summaries, agent mismatches

User Experience:

# Compact current session
You: compact

📝 Generating session summary...
💾 Saved session: myagent_20251230_143022 (15 queries, 12.5K tokens)
🔄 Starting new session with summary...

MyAgent: I've reviewed our conversation about the REST API.
We built Flask routes and database models. Ready to continue!

# Resume old session
You: sessions
You: resume 1

📋 Loading session...
✓ Found: MyAgent - Jan 26, 14:30 (15 queries, 12.5K tokens)
🔄 Restoring context...

MyAgent: I've reviewed our previous conversation...

Technical Implementation:

• Auto-summary generation on save (Phase 1)
• compact command for user-triggered compaction (Phase 2)
• Summary-based resume command (Phase 3)
• HTML markers for summary extraction: <!-- SESSION_SUMMARY_START/END -->
• Progressive summarization: condense old context, detail current session
• Session chaining tracked in markdown files

See compaction_and_resume_plan.md for full implementation details.

🔮 Future Enhancements (Still TODO)

These features from the original issue remain for future implementation:

Context Monitoring:

• Track current conversation token usage in real-time
• context command to show usage statistics
• Warning alerts at thresholds (80%, 90%, 95%)
• Display context usage in status bar

Auto-Compaction:

• Optional automatic compaction when reaching threshold
• Configuration for auto-compact behavior
• auto_compact_threshold setting

Advanced Features:

• compact --preview to preview what would be compacted
• Semantic chunking (group related messages)
• Important message pinning (never compact)
• Compaction history tracking

Configuration:

# Future configuration options
context:
  max_tokens: 200000
  warning_thresholds: [0.8, 0.9, 0.95]
  auto_compact: false
  auto_compact_threshold: 0.85
  preserve_recent: 30

📊 Current Status

Done: Manual compaction with compact command, auto-summarization, summary-based resume
Next: Context monitoring, warnings, auto-compact
Priority: The core functionality is complete; remaining features are enhancements for better UX

Keeping this issue open to track the remaining context monitoring and auto-compaction features.

[jwesleye]

Update: Scope Refinement

After reviewing the remaining features, we're implementing the core context monitoring features while marking some advanced features as out-of-scope for a "basic chat loop" tool.

✅ Implementing Now (v1.6.1)

Context Monitoring & Awareness:

• #context command - Display current token usage statistics
• Context percentage in status bar - Always-visible token usage
• Warning alerts at thresholds - Configurable warnings (80%, 90%, 95%)
• Configuration options - User-customizable limits and thresholds

Implementation Notes:

• Token estimates may not be 100% accurate (framework overhead, counting differences)
• Warnings will be advisory/informational, not hard blocks
• User-configurable max_tokens override in case metadata is incorrect
• Soft language ("estimated", "approximately") to acknowledge uncertainty

🚫 Not Planned (Out of Scope)

These features add significant complexity beyond "basic chat loop" functionality:

• ❌ Auto-compaction - Automatic compaction adds complexity in deciding when/what to compact
• ❌ #compact --preview - Preview mode requires dry-run logic and complex formatting
• ❌ Semantic chunking - Understanding message relationships is complex
• ❌ Message pinning - Requires new data structures, UI, and persistence
• ❌ Compaction history - Tracking and displaying history adds overhead

Rationale: The core compaction functionality is complete with the #compact command. These advanced features would be better suited for a more sophisticated conversation management tool or IDE integration.

📊 Final Status

Issue Resolution: Will remain open until context monitoring features are implemented, then close as complete.

Core Goal Achieved: ✅ Users can manage long conversations without hitting context limits

• Manual compaction works
• Summary-based resume works
• Context awareness coming soon

[jwesleye]

✅ Completed - Context Monitoring Features

This issue has been partially implemented with the core context monitoring features. The following features have been completed and merged:

Implemented Features

1. ✅ #context Command

• Displays detailed token usage statistics
• Shows total, input, and output tokens with formatted display (K/M suffixes)
• Shows percentage of context used when max_tokens is known
• Displays session time and query count
• Includes configurable warning messages at threshold percentages

2. ✅ Context Percentage in Status Bar

• Status bar now shows context percentage when max_tokens is available
• Example: 45.2K tokens (23%) or 180.5K tokens (90%)
• Dynamically formats large numbers (K for thousands, M for millions)

3. ✅ Configurable Warning Thresholds

• Configurable via context.warning_thresholds in .chatrc
• Default thresholds: [80, 90, 95]
• Highest threshold (95%) shows error with #compact suggestion
• Second-highest (90%) shows warning message
• Lower thresholds (80%) show informational messages
• Supports per-agent customization

4. ✅ Context Configuration System

• Added context section to config DEFAULTS
• Works with config hierarchy (global, project, agent overrides)
• Easy to customize warning behavior per agent or globally

Configuration Example

# Default configuration
context:
  warning_thresholds: [80, 90, 95]

# Custom per-agent
agents:
  'Memory Heavy Agent':
    context:
      warning_thresholds: [60, 80, 90, 95]  # Warn earlier

Bug Fix: AWS Strands Token Counting

Also fixed a critical bug discovered during implementation:

• AWS Strands accumulated_usage is cumulative across entire session
• Implemented delta calculation to track only per-query token usage
• Prevents impossible token counts (user reported "70.6M tokens" in status bar)
• Added (usage_dict, is_accumulated) tuple return from _extract_token_usage()

Documentation

• ✅ Updated CHANGELOG.md
• ✅ Added #context to README.md commands table
• ✅ Added comprehensive context section to docs/CONFIG.md

---

🚫 Out of Scope (Not Planned)

The following features from the original proposal are not planned for implementation as they exceed the scope of a "basic" chat loop:

Not Implemented

• ❌ Auto-compaction at thresholds - Adds significant complexity
• ❌ Context usage preview before commands - Over-engineering for basic use
• ❌ Semantic chunking for summaries - Too sophisticated for basic tool
• ❌ Message pinning - Feature creep beyond basic functionality

Rationale

This project is intentionally kept as a basic agent chat loop. The implemented features provide:

1. ✅ Visibility into context usage (#context command, status bar percentage)
2. ✅ Warnings when approaching limits (configurable thresholds)
3. ✅ Manual control (#compact command already exists from v1.6.0)

Users who need advanced features like auto-compaction, semantic chunking, or message pinning should consider more full-featured chat interfaces or build custom solutions on top of this basic framework.

---

Summary

What you get:

• Real-time context monitoring in status bar
• Detailed statistics via #context command
• Configurable warnings at customizable thresholds
• Manual compaction via existing #compact command
• Accurate token counting (including AWS Strands bug fix)

What's out of scope:

• Automatic compaction
• Advanced summarization strategies
• Message pinning and semantic features

This strikes a balance between useful functionality and maintaining the "basic" nature of the tool.

Closing this issue as the core context monitoring features are complete. 🎉

[jwesleye]

Core context monitoring features implemented. Advanced features marked as out of scope for basic chat loop.