feat(dashboard): first-class persistent observability UI — what value is context-guru providing?

[OsherElhadad]

Problem statement

context-guru has no dashboard. Its entire observability surface is GET /stats (proxy/proxy.go:538-549) returning a single JSON snapshot of an in-memory Aggregator (metrics/metrics.go:70 — one mutex-guarded struct, ~267 lines). Nothing is persisted; restart the proxy and every number is gone. There is no HTML, no JS, no embedded assets anywhere in the repo (verified: zero go:embed directives).

A user therefore cannot answer the question the product exists to answer: what value is context-guru providing?

Motivation

Two benchmarks (docs/results/comparison.md, docs/results/terminal-bench-comparison.md) established that this workload's economics are counterintuitive: the request is ~99.95% cached, cache-write costs 11.5× a cache-read, unique token removal is 0.02–0.13% of the billed total, and cost tracks agent steps at r = 0.95. A tool whose savings are reported as "tokens removed" is therefore reporting the least important number — and, per docs/results/improvement-plan.md, we are currently blind on our largest cost line. Every other issue in this batch needs somewhere to show its numbers.

Both reference implementations were studied. Their lessons are concrete and are folded into the requirements below:

• headroom has a genuinely strong dashboard (2,755-line single HTML, Alpine + Tailwind + htmx from CDN, 5 s polling, three view modes). Its best ideas: four different savings denominators each with a stated justification; cache-miss attribution bucketed ttl_expiry | prefix_change | unknown | cold_start with TTL winning ties; reporting the cost of its own safety mechanisms alongside their benefit (compression_vs_cache, prefix_freeze) — which is what makes its numbers credible; token_accounting_status: complete|partial|missing per row; model-aware bust detection (first request per model is a cold start, never a bust); uncompressed_requests reason buckets answering "why didn't you compress this?"; a ?cached=1 5 s snapshot with the request tail refreshed on top; prose description strings inside the payload stating each layer's scope; bounded persistent state with percentages derived at read time; cost stored at write time so history does not reprice.
  Its weaknesses to beat: CDN assets (breaks offline, in a product with an air-gapped install story); the richest attribution data (compressions_by_strategy, tokens_saved_by_strategy) computed, serialized, and rendered nowhere; no histograms in /metrics so p95 is uncomputable; no search, filter, or pagination anywhere; no session list at all despite sessions existing internally; per-request history is a 10k in-memory deque; three tabs with three different definitions of "saved"; and its headline metric contradicts its own design note.
• gateway is not a reference for features — it has no database, no charts, no filters, no session concept, and its 100-request in-memory ring is lost on restart. It is a near-perfect reference for the delivery pattern: one static HTML file read at boot and returned from one route, plus SSE for live updates, with no frontend build step at all. Also worth porting: allowlist-based redaction (blanket-redact all headers by key, allowlist config keys), out-of-band capture so logging can never add latency or fail a request, per-attempt rows, and a write timeout + dead-client eviction on the SSE fan-out. Its one genuinely missing view is the one we most need: it carries finalUntransformedRequest.body and transformedRequest.body side by side and never diffs them.

Desired behavior

A production-quality dashboard that makes the optimization understandable, not just numeric. Within seconds a user should see: tokens saved, dollars saved, gross vs honest/net, cost without context-guru vs with, which components produced the savings, what overhead we added, whether compaction caused restoration/expand events, whether cache behavior improved or regressed, and — where benchmark data exists — whether quality/reward changed.

Overall metrics

total requests · total sessions · tokens before · tokens after · gross saved · unique saved · adjusted/net saved · savings % · estimated dollars saved · baseline estimated cost · actual estimated cost · context-guru's own LLM cost · expand/restoration tokens · bounce/expand count · expand rate · reverts · passthroughs · average context-guru latency · upstream latency · cache-read / cache-write / fresh-input / output tokens.

Several of these already exist in metrics/metrics.go:198-221 (tokens_before, saved_tokens, saved_tokens_unique, overcount_ratio, wasted_tokens, bounces, adjusted_saved, cg_added_ms_avg, upstream_ms_avg, llm_*); the rest need adding.

Time-series graphs

token savings · dollar savings · baseline vs context-guru cost · context size over conversation turns · cache reads · cache writes · context-guru latency · upstream latency · expand/restoration rate · request volume.

The important one: cumulative cost without context-guru vs with, the area between them being money saved.

Honest savings waterfall

baseline cost → deterministic savings → LLM-based savings → cache savings → restoration penalty → context-guru's own LLM cost → final net cost → net savings.

Per-component observability

executions · acted · mutated · skipped · reverted · gross tokens saved · unique tokens saved · adjusted savings · average and total latency · effectiveness · restoration behavior where attributable · economic value · cost of running the component. It must be easy to see which components earn their place — on current evidence cacheinject is inert, extract_llm is ~8× underwater, and failed_run never fires on the benchmark config. The dashboard should make that obvious without reading a doc.

Sessions

Searchable/filterable session list, which today does not exist in any form. Per session: id · model/provider · application/agent · preset/config · start/end · turns · tokens before/after/saved · dollars saved · cache reads/writes · expands · context-guru latency. Clicking a session shows its requests/turns.

Request/turn details

original size · compacted size · savings · which components ran · exactly when each activated · component timings · restores/expands · configuration in effect · cache behavior · before/after content where appropriate.

Diffs — the headline feature

Useful diffs for: original tool output/context, the transformed/compacted version, and restored content when expansion happens. Git-style rendering for code-oriented transformations. It must be easy to inspect: what did context-guru actually remove or rewrite? This is the view both reference implementations have the data for and neither built.

Configuration visibility

active preset · full effective configuration (resolved, not as typed) · enabled components · thresholds · model used by LLM components · cache policy · operating mode · changes over time where possible. Ideally compare configurations or sessions.

Benchmark views

Ingest and display benchmark results: baseline · context-guru · headroom · rtk, with reward · billed cost · token categories · steps · latency · tool cost, including cost-vs-reward visualizations. The rows-*.json + summary.json produced by deploy/harbor/*.py are the natural input format.

Search and filtering

By time range · session · model · provider · agent/application · preset · component · mode · benchmark · task · repository/project where available.

UX quality

Not a Prometheus/Grafana clone — a polished product. Study both references and improve on them.

Relevant code locations

• proxy/proxy.go:110-142 — the route table (/healthz, /stats, /expand, /compact, the chat handlers); where new routes attach.
• proxy/proxy.go:538-549 — the /stats handler; :547 merges cheapmodel.Usage() at serve time.
• proxy/proxy.go:355-383 — where compaction runs and RecordAddedLatency fires; the natural capture point.
• proxy/proxy.go:421-500 — upstream round-trip, RecordUpstreamLatency, the expand loop and RecordExpand.
• metrics/metrics.go:70 (single mutex), :88-100 (per-component stats incl. saved_tokens_unique, overcount_ratio, duration_ms), :196-221 (Snapshot shape).
• components/pipeline.go — per-component run/revert accounting; the source of "which components ran and when".
• apply/apply.go:115-126 (fail-open backstop), :152-173 (Ctx construction, session id, CacheAware, MaxCachedIdx) — where per-request facts are known.
• config/config.go — presets, for the effective-config view.
• deploy/harbor/*.py + docs/results/*.md — benchmark data to ingest.
• session/ — session resolution.

Proposed architecture

Grounded in what the references actually do, and deliberately boring.

1. Storage: SQLite via a pure-Go driver (modernc.org/sqlite, no CGO beyond what tree-sitter already forces). headroom's total lack of durable per-request history is its defining limitation; do not copy it. Two tables:
   • requests: id, session_id, ts (epoch ms — not a locale string, which is gateway's mistake), model, provider, agent, preset, mode, status, tokens_before, tokens_after, fresh/cache_read/cache_write/output, cost_usd, baseline_cost_usd, cg_llm_cost_usd, cg_latency_ms, upstream_ms, expands, reverts, token_accounting_status.
   • request_components: request_id, component, ran, acted, mutated, reverted, saved_gross, saved_unique, duration_ms.
   • Optional request_content: request_id, before, after (compressed, size-capped, opt-out flag) — required for the diff view.
     Indexes on (ts DESC), (session_id, ts). Retention by age and size. No rollup tables — bucket at query time (ts/:bucket*:bucket GROUP BY 1); SQLite handles millions of rows and pre-aggregation is the speculative complexity to skip until a query is measurably slow.
2. Capture off the hot path. The handler hands a struct to a buffered channel (drop + count when full); one writer goroutine batches inserts in a transaction and fans the summary to SSE clients. This preserves gateway's best property — logging can never add latency or fail a request — while adding durability it lacks. The existing Aggregator stays as the fast in-process counter.
3. Frontend: one embedded HTML file via go:embed, no npm, no build step (gateway's delivery model), but with CSS custom properties and a dark-mode block from line one (headroom had to bolt on ~25 light-mode overrides because it hardcoded hex). Vendor every asset into the embed FS — no CDN, since context-guru ships into VPC/air-gapped contexts. A small no-dependency chart library embedded as bytes (uPlot-class, ~45 KB) covers the time series.
4. API: history and live, not live-only. GET /api/requests with server-side filters + keyset pagination; GET /api/requests/{id} returning the request plus its component rows and before/after; GET /api/sessions; GET /api/stats?bucket=&since=; GET /api/config (effective, redacted); GET /api/benchmarks; GET /api/events (SSE, summary rows only, honoring Last-Event-ID so a reconnect backfills instead of silently losing the gap).
5. Redaction before the DB, not on read. Allowlist config keys, blanket-redact headers by key. Secrets must never reach disk. Size-cap content capture and make it opt-out.
6. Access gating (headroom's _request_can_view_dashboard_metadata): per-request content and effective config for loopback or an explicit trusted-CIDR allowlist; aggregates open. Cheap, and the right default for a proxy people bind to 0.0.0.0.

Metric-semantics requirements (non-negotiable)

• Report multiple denominators, each labeled with what it divides by. A whole-request ratio recounts the transcript every turn, so a 200-turn session reads ~0%. Ship at least: saved / attempted-to-compress, and saved / provider-billed-new-input (headroom's new_input_savings_percent — the cleverest number in either tool), guarded so a provider with no cache usage data cannot divide savings by themselves and report ~100%.
• Report the cost of our own safety mechanisms next to their benefit — tokens saved by compaction vs tokens lost to cache busts, and freeze/tail-only's foregone compaction. A compaction proxy that only shows tokens removed is unfalsifiable.
• Never count fake savings. Gross, unique, and adjusted must be visibly distinct; overcount_ratio already exists and should be surfaced, not hidden.
• Cache-miss attribution with a cold-start bucket that is not a failure.
• token_accounting_status per row — never render a partially-instrumented request as exact.
• "Why didn't you compact this?" as a first-class reason bucket.

Alternatives considered

• Prometheus + Grafana only. Explicitly rejected by the requirement. Also cannot show diffs or per-request detail.
• Keep in-memory, add an HTML view. Cheapest, but a restart erases everything and no historical analysis is possible — the same limitation that makes headroom's Session tab a footnote.
• A real SPA with a build step. Rejected: toolchain rot in a Go binary's release artifact, and gateway proves a single embedded file is sufficient at this scale.
• Write to JSONL instead of SQLite. Simpler, but filtering and pagination then move into the client, which is exactly the gap in headroom.

Storage / data-model implications

New dependency (modernc.org/sqlite) and a new on-disk artifact. Needs: a documented default path, a schema version with a discard-and-preserve path for a mismatch (headroom's approach), bounded growth, and a no-op/in-memory mode so the proxy still runs where the path is unwritable. Every percentage derived at read time; cost stored at write time so history does not reprice when a model's rate changes.

Backward compatibility

/stats must keep its current shape — the benchmark harnesses (deploy/harbor/*.py) parse it, and breaking it invalidates the reproduction path. Add fields; do not rename or remove. The dashboard is additive; persistence must be disableable.

Configuration design

--dashboard / env to enable, a DB path, a retention setting, a content-capture toggle, and a trusted-CIDR list. Sensible defaults; do not copy headroom's "disable observability in production" gate — that is backwards for a tool whose value is observability.

Testing plan

• Unit: schema migration, retention pruning, keyset pagination, every filter dimension, query-time bucketing.
• Unit: redaction — a known secret in a header/config never appears in a stored row.
• Unit: the capture channel drops rather than blocks when full, and the drop is counted.
• Unit: SSE fan-out with a hung client (write timeout + eviction).
• Unit: /stats shape unchanged (golden test against the current payload).
• go test -race on the capture path and the SSE hub — mandatory.
• Playwright: see below.

Real-world benchmark plan

This is dashboard work, so the requirement is real runtime data, not fixtures. Run 2–3 SWE-bench and 2–3 Terminal-Bench tasks plus a real Claude Code session through the proxy with the dashboard on, then validate against that populated data. Also ingest a full existing benchmark run (rows-*.json + summary.json) and verify per-task drill-down. Record: commit SHA, config, model, task ids, all four token tiers, per-component cost, total billed cost, steps, reward, latency, expands, reverts, savings. Measure and report the capture overhead — if the dashboard costs measurable request latency, that is a regression in a tool that sells latency awareness.

UI verification (Phase 5)

Playwright/browser automation against the actual rendered app, with screenshots of: Overview · savings/cost graph · component metrics · sessions · request detail · content/Git diff · configuration · observe-mode visualization · benchmark comparison · filters/search. Check desktop and a smaller viewport, empty states, loading states, large datasets, long session ids and long content, charts, tooltips, filtering, navigation. Follow headroom's one clearly right practice here: data-testid on every stat tile, asserted in tests, so the visual layer is regression-tested.

Acceptance criteria

• Persistent storage with schema versioning, bounded retention, and a working no-persistence mode.
• Capture is off the hot path; measured added latency reported; drops counted.
• Secrets never reach disk; redaction unit-tested.
• Dashboard served from embedded assets with no network fetches (verifiable offline).
• Every metric group above present; multiple labeled savings denominators; safety-mechanism costs shown beside benefits; overcount_ratio and token_accounting_status surfaced.
• Baseline-vs-actual cumulative cost chart with the saved area.
• Honest savings waterfall.
• Per-component economics view making it obvious which components earn their place.
• Session list + drill-down to requests; request detail with component timings and before/after diff.
• Effective-configuration view.
• Benchmark ingestion + cost-vs-reward view for all four arms.
• Server-side search/filter across every listed dimension, with pagination.
• Dark mode from the start; empty/loading/error states; small-viewport behavior.
• Playwright screenshots in the PR and in the docs.
• /stats shape unchanged (golden test).
• go test -race green.

Documentation updates

New docs/dashboard.md (with screenshots), docs/how-to/measure-savings.md rewritten around the dashboard, docs/reference/routes.md (new API routes), docs/reference/config.md, docs/design.md (the observability/event/store architecture), mkdocs.yml nav, and a README section. The GitHub Pages site must build and render.

Dependencies

Consumes metrics from #25 (frozen hit/miss/flip), #26 (restoration activity), #27 (xdedup attribution), #28 (extraction economics), #29 (per-family cmdfilter savings), and must render #31's three modes with unmistakably distinct semantics. Land the storage/event architecture early so the others can emit into it; the per-issue panels can follow. This is the largest issue in the batch — consider splitting the PR into storage+API, then UI, then benchmark ingestion.