feat: analytics, LLM feedback, session intelligence, and compact output#12
Merged
feat: analytics, LLM feedback, session intelligence, and compact output#12
Conversation
Track every analyze, analyze-change, and MCP get_context call in history.sqlite with file count, blast radius, confidence, and duration. New `contextception gain` command shows usage dashboard with summary, daily/weekly/monthly breakdowns, and JSON/CSV export. Phase 1 of RTK-inspired analytics: deterministic usage tracking that answers "is contextception being used and how?"
New `rate_context` MCP tool lets LLMs submit structured feedback after using get_context: which files were useful, unnecessary, or missing. New `contextception accuracy` command computes must-read precision/recall and likely-modify accuracy from this feedback data. Phase 2 of RTK-inspired analytics: LLM feedback loop for measuring and improving recommendation quality.
New session parser scans Claude Code JSONL session files to extract Edit/Write tool calls and contextception MCP usage. Two new commands: - `contextception discover` finds files edited without get_context - `contextception session` shows adoption rates across sessions Phase 3 of RTK-inspired analytics: session intelligence for measuring contextception adoption and finding missed opportunities.
New compact text formatter produces ~60-75% fewer tokens than JSON output while preserving all essential context information. Available on both analyze and analyze-change commands via --compact flag. Hook-context output now uses the compact formatter by default, providing richer context (likely-modify, warnings) in fewer tokens. Phase 4 of RTK-inspired analytics: token optimization for LLM context efficiency.
The discover command was showing 0% coverage because hook-context runs as a PreToolUse hook whose output (additionalContext) is not recorded in Claude Code session JSONL files. Now cross-references the usage_log table in history.sqlite to detect hook-injected context alongside explicit MCP get_context calls.
- gain: separate context analyses from change analyses; store NULL confidence for analyze-change so AVG excludes it; top files only shows files whose context was analyzed, not PR changed files - gain: deterministic top-files sort (secondary sort by path) - accuracy: deduplicate likely_modify denominator by distinct usage_id so multiple feedbacks on the same analysis don't inflate the count - feedback: use JSON-quoted path in usage_id lookup to prevent false matches (e.g. "db.go" won't match "db_test.go") - discover: filter test files regardless of context count, not just when context is zero - session: cross-reference usage_log for hook-injected context (same fix as discover) - session: compare calendar dates instead of hours elapsed for relative date display
linkage and compact direction display - GetUsageByPeriod: exclude analyze_change from SUM(file_count) so daily/weekly/monthly periods show context-analyzed files, not PR changed files - FilesWithUsage: filter to context analysis tools so discover/session don't count analyze_change files as "context requested" - RecordFeedback: prefer analyze/get_context entries over analyze_change when linking feedback to usage_log - FormatCompact: handle same_package direction and unknown directions instead of silently dropping them Found during rigorous output validation against ground truth.
…utput Update every documentation file to reflect the 4 new CLI commands (gain, accuracy, discover, session), the rate_context MCP tool, and the --compact flag: - README.md: add analytics section, update CLI/MCP tool lists, add --compact flag - CLAUDE.md: fix command count to 16, add session/ package to structure - docs/ARCHITECTURE.md: add rate_context tool, history/session subsystem docs - docs/mcp-tutorial.md: add rate_context to tool reference - integrations/README.md: update to nine tools, add rate_context - integrations/claude-code/CLAUDE.md: add rate_context workflow step - integrations/codex/agents.md: add rate_context tool docs and workflow step - AGENTS.md: update tool count, add history/session packages - CHANGELOG.md: add [Unreleased] section with all Phase 1-4 changes
…ck-token-optimization
Deterministic compact output via sorted map keys, SQL injection safety (LIKE → instr), lazy history store pooling in MCP server, session --since flag, unused parameter cleanup, and RateContext protocol documentation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Analytics and token optimization for contextception, implemented in 4 phases:
Phase 1 — Usage Analytics: Tracks every
analyze,analyze-change, and MCPget_contextcall inhistory.sqlite. Newcontextception gaincommand shows usage dashboard with top files, blast radius trends, and daily activity. Supports--daily/weekly/monthly,--format json|csvexport.Phase 2 — LLM Feedback Loop: New
rate_contextMCP tool lets LLMs submit structured feedback (usefulness rating, which files were useful/unnecessary/missing/modified). Newcontextception accuracycommand computes must-read precision/recall and likely-modify accuracy from this feedback.Phase 3 — Session Intelligence: Parses Claude Code session JSONL files to find files edited without context being checked. New
contextception discovercommand reports missed opportunities with coverage percentages. Newcontextception sessioncommand shows per-session adoption rates. Cross-referencesusage_logto detect hook-injected context invisible in session files.Phase 4 — Token Optimization: New
--compactflag onanalyzeandanalyze-changeproduces a text summary at ~60-75% fewer tokens than JSON. Hook-context output now uses the compact formatter by default, providing richer context in fewer tokens.New commands (4 CLI + 1 MCP)
contextception gain— usage analytics dashboardcontextception accuracy— recommendation quality metricscontextception discover— find files edited without contextcontextception session— adoption rates per sessionrate_contextMCP tool — structured LLM feedbackNew files
internal/analyzer/compact.go— compact text formatterinternal/cli/gain.go,accuracy.go,discover.go,session_cmd.go— new commandsinternal/session/— Claude Code session JSONL parserKey design decisions
history.sqlite(no new databases)analyze_changesoAVG()excludes it naturallydiscover/sessioncross-referenceusage_logfor hook-injected context (invisible in session JSONL)Test plan
make check)