diff --git a/docs/guides/hermes-monitor.md b/docs/guides/hermes-monitor.md index 4aa9128..5500b2c 100644 --- a/docs/guides/hermes-monitor.md +++ b/docs/guides/hermes-monitor.md @@ -53,6 +53,39 @@ Then read the generated Markdown report for today and send a short Chinese summa ) ``` +## Quick savings summary (lightweight) + +If you just want to answer "how many tokens did ContextPilot save?", use the +lightweight `scripts/contextpilot_savings.py` command instead of this monitor or +the analyzer below. It reads **only** the metadata-only telemetry file, imports +no Hermes internals, and prints a one-screen summary: + +```bash +python scripts/contextpilot_savings.py # last 24h +python scripts/contextpilot_savings.py --all-time # everything +python scripts/contextpilot_savings.py --format json +# or, after Hermes plugin install: +python ~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py +``` + +It reports events, chars saved, estimated tokens saved, the window, and average +tokens per event. This is the right tool for ordinary users; the monitor in this +guide (which also reads `state.db` metadata) and the content-aware analyzer below +are for deeper investigation. + +### Ask Hermes for savings + +To let users get this summary without typing the command, the repo ships a +narrow, read-only Hermes skill at `skills/contextpilot-savings/SKILL.md`. Copy +or install it into your Hermes skills, then ask Hermes something like "show +ContextPilot token savings" or "how much did ContextPilot save all time?". +Hermes locates `scripts/contextpilot_savings.py` (in the repo/plugin checkout or +at `~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py`), runs it +with the right `--since-hours` / `--all-time` / `--format json` options, and +summarizes the output. The skill is observe-only — it reads only the +metadata-only telemetry through this script and makes no code, config, or +scheduling changes. + ## Opportunity scanning `scripts/analyze_hermes_context_opportunities.py` is a companion scanner meant diff --git a/docs/guides/hermes.md b/docs/guides/hermes.md index d13f293..03a7a56 100644 --- a/docs/guides/hermes.md +++ b/docs/guides/hermes.md @@ -64,6 +64,47 @@ print(engine.get_status()) # {'engine': 'contextpilot', 'contextpilot_chars_saved': 18420, ...} ``` +## See token savings + +Once ContextPilot has run for a bit, you can see how many tokens it saved with a +single command from the ContextPilot repo or plugin directory: + +```bash +python scripts/contextpilot_savings.py +# or, after Hermes plugin install: +python ~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py +``` + +``` +ContextPilot token savings (last 24h) + Events: 117 + Chars saved: 6,147,074 + Estimated tokens saved: ~1,536,728 + Avg tokens/event: ~13,134 + Telemetry file: /root/.hermes/contextpilot/telemetry.jsonl +``` + +Useful options: + +- `--all-time` — total savings since you enabled ContextPilot. +- `--since-hours N` — only the last `N` hours (default `24`). +- `--format json` — stable, machine-readable output for dashboards. + +This reads only the metadata-only telemetry file +(`~/.hermes/contextpilot/telemetry.jsonl`); it never touches your conversations, +prompts, or tool output. If it reports no savings, make sure ContextPilot is +enabled, restart Hermes, and run a workload that reads the same content more than +once — savings only happen when content repeats across turns. + +### Ask Hermes instead + +If you'd rather not run the command yourself, install or copy the bundled +read-only Hermes skill at `skills/contextpilot-savings/SKILL.md` and just ask +Hermes ("how much did ContextPilot save?"). Hermes finds the script, runs it +with the right window/format, and summarizes the result. The skill is +observe-only: it reads nothing but the metadata-only telemetry via this script. +See [`hermes-monitor.md`](./hermes-monitor.md#ask-hermes-for-savings) for details. + ## Disabling ```bash diff --git a/scripts/contextpilot_savings.py b/scripts/contextpilot_savings.py new file mode 100644 index 0000000..2403d38 --- /dev/null +++ b/scripts/contextpilot_savings.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +"""Show how many tokens ContextPilot saved — a simple, user-facing summary. + +Reads only the metadata-only telemetry file ContextPilot writes +(``~/.hermes/contextpilot/telemetry.jsonl`` by default) and prints a concise +savings summary. It never reads conversation messages, system prompts, +reasoning, or tool payloads — the telemetry file only contains numeric counters. + +This is the lightweight companion to the heavier +``analyze_hermes_context_opportunities.py`` analyzer: no Hermes DB access, no +content scanning, no Hermes internals imported. Just "how much did I save?". + +Examples:: + + python scripts/contextpilot_savings.py # last 24h + python scripts/contextpilot_savings.py --all-time # everything + python scripts/contextpilot_savings.py --since-hours 1 # last hour + python scripts/contextpilot_savings.py --format json # machine readable +""" +from __future__ import annotations + +import argparse +import datetime as dt +import json +from pathlib import Path +from typing import Any, Dict + +DEFAULT_TELEMETRY_FILE = Path.home() / ".hermes" / "contextpilot" / "telemetry.jsonl" +DEFAULT_SINCE_HOURS = 24 + + +def summarize_telemetry( + telemetry_path: Path, + *, + since_hours: float | None, +) -> Dict[str, Any]: + """Aggregate the metadata-only telemetry file into a savings summary. + + ``since_hours`` of ``None`` means all-time (no time filtering). Only numeric + counters are read; no conversation/tool/system content can be present in the + file, and this function never emits raw text regardless. + + Malformed JSONL lines (and non-dict / non-numeric records) are skipped and + counted under ``skipped_lines``. + """ + result: Dict[str, Any] = { + "telemetry_file": str(telemetry_path), + "file_exists": telemetry_path.exists(), + "all_time": since_hours is None, + "since_hours": since_hours, + "window_start_iso": None, + "events": 0, + "chars_saved": 0, + "tokens_saved": 0, + "avg_tokens_per_event": None, + "skipped_lines": 0, + } + + if not telemetry_path.exists(): + return result + + if since_hours is None: + cutoff = None + else: + cutoff = dt.datetime.now(dt.timezone.utc).timestamp() - since_hours * 3600 + result["window_start_iso"] = ( + dt.datetime.fromtimestamp(cutoff, dt.timezone.utc).isoformat() + ) + + events = 0 + chars = 0 + tokens = 0 + skipped = 0 + with telemetry_path.open("r", encoding="utf-8", errors="replace") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + record = json.loads(line) + except (ValueError, TypeError): + skipped += 1 + continue + if not isinstance(record, dict): + skipped += 1 + continue + ts = record.get("ts") + if cutoff is not None: + if not isinstance(ts, (int, float)): + skipped += 1 + continue + if ts < cutoff: + continue + cs = record.get("chars_saved") + if not isinstance(cs, (int, float)) or cs < 0: + skipped += 1 + continue + saved_tokens = record.get("tokens_saved") + if isinstance(saved_tokens, (int, float)) and saved_tokens < 0: + skipped += 1 + continue + events += 1 + chars += int(cs) + tokens += ( + int(saved_tokens) + if isinstance(saved_tokens, (int, float)) + else int(cs) // 4 + ) + + result["events"] = events + result["chars_saved"] = chars + result["tokens_saved"] = tokens + result["skipped_lines"] = skipped + if events > 0: + result["avg_tokens_per_event"] = round(tokens / events, 1) + return result + + +def _format_window(summary: Dict[str, Any]) -> str: + if summary["all_time"]: + return "all time" + hours = summary["since_hours"] + if isinstance(hours, float) and hours.is_integer(): + hours = int(hours) + return f"last {hours}h" + + +def render_text(summary: Dict[str, Any]) -> str: + window = _format_window(summary) + path = summary["telemetry_file"] + + if not summary["file_exists"]: + return ( + "No ContextPilot telemetry found.\n" + f" Looked for: {path}\n\n" + "To start seeing token savings:\n" + " 1. Enable ContextPilot in Hermes " + "(hermes plugins → toggle contextpilot).\n" + " 2. Restart Hermes.\n" + " 3. Run a workload that reads the same content more than once " + "(e.g. read a file, then read it again).\n" + "ContextPilot only saves tokens when content repeats across turns." + ) + + if summary["events"] == 0: + return ( + f"No ContextPilot savings recorded in the {window} window.\n" + f" Telemetry file: {path}\n\n" + "If you expected savings:\n" + " - Make sure ContextPilot is enabled and Hermes was restarted.\n" + " - Try --all-time to widen the window.\n" + " - Run a workload with repeated content (read the same file " + "twice); savings only fire when content repeats across turns." + ) + + lines = [ + f"ContextPilot token savings ({window})", + f" Events: {summary['events']}", + f" Chars saved: {summary['chars_saved']:,}", + f" Estimated tokens saved: ~{summary['tokens_saved']:,}", + ] + if summary["avg_tokens_per_event"] is not None: + lines.append( + f" Avg tokens/event: ~{summary['avg_tokens_per_event']:,}" + ) + lines.append(f" Telemetry file: {path}") + if summary["skipped_lines"]: + lines.append( + f" (skipped {summary['skipped_lines']} malformed telemetry line(s))" + ) + return "\n".join(lines) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Show how many tokens ContextPilot saved (metadata-only).", + ) + parser.add_argument( + "--telemetry-file", + type=Path, + default=DEFAULT_TELEMETRY_FILE, + help="metadata-only ContextPilot telemetry file " + "(default: ~/.hermes/contextpilot/telemetry.jsonl)", + ) + parser.add_argument( + "--since-hours", + type=float, + default=DEFAULT_SINCE_HOURS, + help="only count savings in the last N hours (default: 24)", + ) + parser.add_argument( + "--all-time", + action="store_true", + help="count all savings, ignoring --since-hours", + ) + parser.add_argument( + "--format", + choices=["text", "json"], + default="text", + help="output format (default: text)", + ) + args = parser.parse_args(argv) + + since_hours = None if args.all_time else args.since_hours + summary = summarize_telemetry(args.telemetry_file, since_hours=since_hours) + + if args.format == "json": + print(json.dumps(summary, ensure_ascii=False, indent=2)) + else: + print(render_text(summary)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/contextpilot-savings/SKILL.md b/skills/contextpilot-savings/SKILL.md new file mode 100644 index 0000000..bd8211b --- /dev/null +++ b/skills/contextpilot-savings/SKILL.md @@ -0,0 +1,140 @@ +--- +name: contextpilot-savings +description: Use when a user asks how many tokens (or how much context/cost) ContextPilot has saved, or wants a ContextPilot savings status/summary inside Hermes Agent — e.g. "how much did ContextPilot save?", "show ContextPilot token savings", "ContextPilot status", "savings in the last 24h / all time", or "give me the raw savings JSON". This skill runs the metadata-only `scripts/contextpilot_savings.py` reporter and summarizes its output in plain language. It is read-only and observe-only — it never changes ContextPilot code or config, never enables routing/dropping/summarization, never creates cron jobs, branches, or pull requests, and never reads conversation messages, tool outputs, system prompts, reasoning, or raw session ids. +version: 1.0.0 +author: ContextPilot +license: MIT +metadata: + hermes: + tags: [contextpilot, hermes, telemetry, token-savings, observability, read-only] + related_skills: [] + category: observability + safety: read-only +--- + +# ContextPilot Savings (Hermes) + +This is a small, **read-only** skill: when a user wants to know how many tokens +ContextPilot has saved, run the lightweight savings reporter and explain the +result in plain language. That is the entire job — measure and report, nothing +else. + +> Scope: **show savings/status only.** This skill never modifies ContextPilot +> code or config, never enables context routing/dropping/summarization, never +> schedules background jobs, and never edits the repository or opens code-change +> requests. If a user wants any of that, this is the wrong tool — stop and tell +> them so. + +## When to use this skill + +- "How many tokens did ContextPilot save?" +- "Show me ContextPilot token savings" / "ContextPilot status". +- "Savings in the last 24 hours" / "savings all time". +- "Give me the raw savings data / JSON." + +## Privacy boundary (read this first) + +The only thing this skill is allowed to read is the **metadata-only** telemetry +file (`~/.hermes/contextpilot/telemetry.jsonl`), and it reads it **only** +through `scripts/contextpilot_savings.py`. That file contains nothing but +numeric counters (timestamps, chars saved, tokens saved). + +Do **not**, under any circumstances: + +- read `~/.hermes/state.db`, conversation `messages`, or message content, +- read system prompts, skill prompts, or reasoning fields, +- read raw tool-call payloads or tool outputs, +- read or print raw session ids. + +If you are ever tempted to inspect any of the above to "explain" the savings, +don't — the script's metadata summary is the complete, safe answer. + +## How to run it + +### Step 1 — Locate the script + +Use whichever of these exists (check in this order): + +1. `scripts/contextpilot_savings.py` — when you are in a ContextPilot repo or + plugin checkout (current working directory). +2. `~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py` — when + ContextPilot was installed as a Hermes plugin. + +Pick the first one that exists and use that path as `