Feat/hermes contextpilot monitoring

docs/guides/hermes-monitor.md

@@ -53,6 +53,39 @@ Then read the generated Markdown report for today and send a short Chinese summa
 )
 ```
 
+## Quick savings summary (lightweight)
+
+If you just want to answer "how many tokens did ContextPilot save?", use the
+lightweight `scripts/contextpilot_savings.py` command instead of this monitor or
+the analyzer below. It reads **only** the metadata-only telemetry file, imports
+no Hermes internals, and prints a one-screen summary:
+
+```bash
+python scripts/contextpilot_savings.py            # last 24h
+python scripts/contextpilot_savings.py --all-time # everything
+python scripts/contextpilot_savings.py --format json
+# or, after Hermes plugin install:
+python ~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py
+```
+
+It reports events, chars saved, estimated tokens saved, the window, and average
+tokens per event. This is the right tool for ordinary users; the monitor in this
+guide (which also reads `state.db` metadata) and the content-aware analyzer below
+are for deeper investigation.
+
+### Ask Hermes for savings
+
+To let users get this summary without typing the command, the repo ships a
+narrow, read-only Hermes skill at `skills/contextpilot-savings/SKILL.md`. Copy
+or install it into your Hermes skills, then ask Hermes something like "show
+ContextPilot token savings" or "how much did ContextPilot save all time?".
+Hermes locates `scripts/contextpilot_savings.py` (in the repo/plugin checkout or
+at `~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py`), runs it
+with the right `--since-hours` / `--all-time` / `--format json` options, and
+summarizes the output. The skill is observe-only — it reads only the
+metadata-only telemetry through this script and makes no code, config, or
+scheduling changes.
+
 ## Opportunity scanning
 
 `scripts/analyze_hermes_context_opportunities.py` is a companion scanner meant

docs/guides/hermes.md

@@ -64,6 +64,47 @@ print(engine.get_status())
 # {'engine': 'contextpilot', 'contextpilot_chars_saved': 18420, ...}
 ```
 
+## See token savings
+
+Once ContextPilot has run for a bit, you can see how many tokens it saved with a
+single command from the ContextPilot repo or plugin directory:
+
+```bash
+python scripts/contextpilot_savings.py
+# or, after Hermes plugin install:
+python ~/.hermes/plugins/ContextPilot/scripts/contextpilot_savings.py
+```
+
+```
+ContextPilot token savings (last 24h)
+  Events:                117
+  Chars saved:           6,147,074
+  Estimated tokens saved: ~1,536,728
+  Avg tokens/event:      ~13,134
+  Telemetry file:        /root/.hermes/contextpilot/telemetry.jsonl
+```
+
+Useful options:
+
+- `--all-time` — total savings since you enabled ContextPilot.
+- `--since-hours N` — only the last `N` hours (default `24`).
+- `--format json` — stable, machine-readable output for dashboards.
+
+This reads only the metadata-only telemetry file
+(`~/.hermes/contextpilot/telemetry.jsonl`); it never touches your conversations,
+prompts, or tool output. If it reports no savings, make sure ContextPilot is
+enabled, restart Hermes, and run a workload that reads the same content more than
+once — savings only happen when content repeats across turns.
+
+### Ask Hermes instead
+
+If you'd rather not run the command yourself, install or copy the bundled
+read-only Hermes skill at `skills/contextpilot-savings/SKILL.md` and just ask
+Hermes ("how much did ContextPilot save?"). Hermes finds the script, runs it
+with the right window/format, and summarizes the result. The skill is
+observe-only: it reads nothing but the metadata-only telemetry via this script.
+See [`hermes-monitor.md`](./hermes-monitor.md#ask-hermes-for-savings) for details.
+
 ## Disabling
 
 ```bash

scripts/contextpilot_savings.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""Show how many tokens ContextPilot saved — a simple, user-facing summary.
+
+Reads only the metadata-only telemetry file ContextPilot writes
+(``~/.hermes/contextpilot/telemetry.jsonl`` by default) and prints a concise
+savings summary. It never reads conversation messages, system prompts,
+reasoning, or tool payloads — the telemetry file only contains numeric counters.
+
+This is the lightweight companion to the heavier
+``analyze_hermes_context_opportunities.py`` analyzer: no Hermes DB access, no
+content scanning, no Hermes internals imported. Just "how much did I save?".
+
+Examples::
+
+    python scripts/contextpilot_savings.py                 # last 24h
+    python scripts/contextpilot_savings.py --all-time       # everything
+    python scripts/contextpilot_savings.py --since-hours 1  # last hour
+    python scripts/contextpilot_savings.py --format json    # machine readable
+"""
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+from pathlib import Path
+from typing import Any, Dict
+
+DEFAULT_TELEMETRY_FILE = Path.home() / ".hermes" / "contextpilot" / "telemetry.jsonl"
+DEFAULT_SINCE_HOURS = 24
+
+
+def summarize_telemetry(
+    telemetry_path: Path,
+    *,
+    since_hours: float | None,
+) -> Dict[str, Any]:
+    """Aggregate the metadata-only telemetry file into a savings summary.
+
+    ``since_hours`` of ``None`` means all-time (no time filtering). Only numeric
+    counters are read; no conversation/tool/system content can be present in the
+    file, and this function never emits raw text regardless.
+
+    Malformed JSONL lines (and non-dict / non-numeric records) are skipped and
+    counted under ``skipped_lines``.
+    """
+    result: Dict[str, Any] = {
+        "telemetry_file": str(telemetry_path),
+        "file_exists": telemetry_path.exists(),
+        "all_time": since_hours is None,
+        "since_hours": since_hours,
+        "window_start_iso": None,
+        "events": 0,
+        "chars_saved": 0,
+        "tokens_saved": 0,
+        "avg_tokens_per_event": None,
+        "skipped_lines": 0,
+    }
+
+    if not telemetry_path.exists():
+        return result
+
+    if since_hours is None:
+        cutoff = None
+    else:
+        cutoff = dt.datetime.now(dt.timezone.utc).timestamp() - since_hours * 3600
+        result["window_start_iso"] = (
+            dt.datetime.fromtimestamp(cutoff, dt.timezone.utc).isoformat()
+        )
+
+    events = 0
+    chars = 0
+    tokens = 0
+    skipped = 0
+    with telemetry_path.open("r", encoding="utf-8", errors="replace") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                record = json.loads(line)
+            except (ValueError, TypeError):
+                skipped += 1
+                continue
+            if not isinstance(record, dict):
+                skipped += 1
+                continue
+            ts = record.get("ts")
+            if cutoff is not None:
+                if not isinstance(ts, (int, float)):
+                    skipped += 1
+                    continue
+                if ts < cutoff:
+                    continue
+            cs = record.get("chars_saved")
+            if not isinstance(cs, (int, float)) or cs < 0:
+                skipped += 1
+                continue
+            saved_tokens = record.get("tokens_saved")
+            if isinstance(saved_tokens, (int, float)) and saved_tokens < 0:
+                skipped += 1
+                continue
+            events += 1
+            chars += int(cs)
+            tokens += (
+                int(saved_tokens)
+                if isinstance(saved_tokens, (int, float))
+                else int(cs) // 4
+            )
+
+    result["events"] = events
+    result["chars_saved"] = chars
+    result["tokens_saved"] = tokens
+    result["skipped_lines"] = skipped
+    if events > 0:
+        result["avg_tokens_per_event"] = round(tokens / events, 1)
+    return result
+
+
+def _format_window(summary: Dict[str, Any]) -> str:
+    if summary["all_time"]:
+        return "all time"
+    hours = summary["since_hours"]
+    if isinstance(hours, float) and hours.is_integer():
+        hours = int(hours)
+    return f"last {hours}h"
+
+
+def render_text(summary: Dict[str, Any]) -> str:
+    window = _format_window(summary)
+    path = summary["telemetry_file"]
+
+    if not summary["file_exists"]:
+        return (
+            "No ContextPilot telemetry found.\n"
+            f"  Looked for: {path}\n\n"
+            "To start seeing token savings:\n"
+            "  1. Enable ContextPilot in Hermes "
+            "(hermes plugins → toggle contextpilot).\n"
+            "  2. Restart Hermes.\n"
+            "  3. Run a workload that reads the same content more than once "
+            "(e.g. read a file, then read it again).\n"
+            "ContextPilot only saves tokens when content repeats across turns."
+        )
+
+    if summary["events"] == 0:
+        return (
+            f"No ContextPilot savings recorded in the {window} window.\n"
+            f"  Telemetry file: {path}\n\n"
+            "If you expected savings:\n"
+            "  - Make sure ContextPilot is enabled and Hermes was restarted.\n"
+            "  - Try --all-time to widen the window.\n"
+            "  - Run a workload with repeated content (read the same file "
+            "twice); savings only fire when content repeats across turns."
+        )
+
+    lines = [
+        f"ContextPilot token savings ({window})",
+        f"  Events:                {summary['events']}",
+        f"  Chars saved:           {summary['chars_saved']:,}",
+        f"  Estimated tokens saved: ~{summary['tokens_saved']:,}",
+    ]
+    if summary["avg_tokens_per_event"] is not None:
+        lines.append(
+            f"  Avg tokens/event:      ~{summary['avg_tokens_per_event']:,}"
+        )
+    lines.append(f"  Telemetry file:        {path}")
+    if summary["skipped_lines"]:
+        lines.append(
+            f"  (skipped {summary['skipped_lines']} malformed telemetry line(s))"
+        )
+    return "\n".join(lines)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        description="Show how many tokens ContextPilot saved (metadata-only).",
+    )
+    parser.add_argument(
+        "--telemetry-file",
+        type=Path,
+        default=DEFAULT_TELEMETRY_FILE,
+        help="metadata-only ContextPilot telemetry file "
+        "(default: ~/.hermes/contextpilot/telemetry.jsonl)",
+    )
+    parser.add_argument(
+        "--since-hours",
+        type=float,
+        default=DEFAULT_SINCE_HOURS,
+        help="only count savings in the last N hours (default: 24)",
+    )
+    parser.add_argument(
+        "--all-time",
+        action="store_true",
+        help="count all savings, ignoring --since-hours",
+    )
+    parser.add_argument(
+        "--format",
+        choices=["text", "json"],
+        default="text",
+        help="output format (default: text)",
+    )
+    args = parser.parse_args(argv)
+
+    since_hours = None if args.all_time else args.since_hours
+    summary = summarize_telemetry(args.telemetry_file, since_hours=since_hours)
+
+    if args.format == "json":
+        print(json.dumps(summary, ensure_ascii=False, indent=2))
+    else:
+        print(render_text(summary))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())