feat(dash): first-class persistent observability dashboard - #57
Closed
OsherElhadad wants to merge 5 commits into
Closed
feat(dash): first-class persistent observability dashboard#57OsherElhadad wants to merge 5 commits into
OsherElhadad wants to merge 5 commits into
Conversation
added 5 commits
August 10, 2026 17:29
…he honest-metrics API
context-guru's entire observability surface was `GET /stats`: one in-memory Aggregator
snapshot, gone on restart. A user could not answer the question the product exists to
answer -- what value is context-guru providing? -- and nothing could show history,
per-request detail, or what compaction actually removed.
This adds the storage, capture and API layer. The UI lands next.
Capture is off the hot path, and that is the load-bearing property. The handler builds
one struct from values the request path already computed and hands it to a buffered
channel with a `default:` branch; a full queue DROPS the event and increments a counter
the dashboard itself displays. Measured cost on the request goroutine is ~175ns
(BenchmarkRecord), with a regression test that fails above 50us -- so moving I/O onto
the capture path breaks CI. An observability layer that can add latency to, or fail, a
request has no business existing in a tool that sells latency awareness.
One writer goroutine owns the database, batching inserts in a transaction and fanning
summaries to SSE. The existing Aggregator is untouched and remains the fast in-process
counter behind /stats.
Storage is SQLite via modernc.org/sqlite (pure Go -- no C toolchain beyond the one
tree-sitter already forces), WAL mode, four tables: requests, request_components,
request_content (gzip, size-capped, skippable) and the ingested bench_runs/bench_tasks.
Timestamps are epoch MILLISECONDS everywhere; a formatted locale string cannot be
range-queried, sorted portably, or bucketed. Retention is bounded by age AND size --
age alone cannot bound a burst, size alone silently erases a quiet week. A schema
version mismatch renames the old file aside and starts fresh: a dashboard is a derived
view, so discarding history beats refusing to boot and renaming beats deleting.
An unopenable path degrades to in-memory with a warning; the proxy's job is to proxy.
No rollup tables. Series are bucketed in SQL at query time (ts/bucket*bucket GROUP BY 1),
so any bucket width works with no migration. Pagination is KEYSET, not OFFSET, so page
500 of a busy proxy's history costs the same as page 1 and cannot skip or duplicate a row
while new requests arrive. p95 is computed exactly with ORDER BY + OFFSET -- answerable
at all, unlike a deployment with no histogram.
The metric semantics are the point, not the plumbing:
* Four labelled savings denominators, each carrying its own prose description in the
payload. A whole-request ratio recounts the transcript every turn, so a 200-turn
session reads ~0% however well compaction performs; a compressible-only ratio
flatters by excluding what we chose not to touch. Both are true. Neither is "the"
number, so each ships with the divisor named.
* The new-input ratio is guarded on the BILLED figure, not the sum: with no provider
usage data the denominator would be `saved` alone and the ratio would read ~100%.
It reports unavailable instead.
* Gross, unique and adjusted are separate columns and overcount_ratio is surfaced.
* The cost of our own safety mechanisms sits beside their benefit: tokens frozen for
cache safety, restorations after a premature offload, reverted runs, and
context-guru's own latency and LLM spend.
* token_accounting per row (complete|partial|missing). A request is priced only when
the provider reported all four tiers AND the model's rates are known; otherwise cost
reads unknown, never zero and never exact.
* Cache-miss attribution with cold_start as a NON-failure -- the first request of a
session, or the first for a model, has nothing to hit -- and TTL expiry winning ties
against a changed prefix, because a prefix that changed after the entry had already
expired was not the cause.
* "Why didn't you compact this?" as a first-class reason bucket rather than an absence
of data.
Redaction happens BEFORE the database, never on read: headers are blanket-redacted by
key against a short allowlist (a denylist fails the moment a gateway invents a new auth
header), config keys are allowlisted with credential-named keys always withheld, and
captured content is pattern-scrubbed then size-capped -- in that order, so a secret at
the truncation boundary cannot survive by being cut into an unmatchable prefix. A test
writes a canary through the real capture path and reads every stored column back.
Two subtleties the allowlist got wrong on the first pass, both fixed with tests:
`components` is keyed by user-chosen component names and cannot be allowlisted, so
redacting the subtree wholesale made the config view show nothing; and a substring match
on "token" also swallows max_tokens/min_tokens, redacting every threshold in that same
view. Safety that destroys the feature is not safety.
Cost is computed at WRITE time so history does not reprice when a published rate changes;
percentages are derived at READ time so a filter change needs no rebuild.
Supporting changes:
* apply.BodyTrace: BodyFull now delegates to it, so the rewrite is byte-identical
whether or not anyone is looking. The Trace carries the resolved session, the
RunReport, the cache-awareness facts (AttemptedTokens/FrozenTokens -- the honest
denominator and the cost of cache safety) and each rewritten message's before/after
text: the same material CONTEXT_GURU_DUMP writes to a file, handed to a caller.
* internal/modelinfo gains per-token pricing (four tiers), and a real bug is fixed:
the LiteLLM map was decoded into one typed map[string]struct, which fails on the
document's prose-filled `sample_spec` entry and on the handful of models that spell
an integer as a float -- yielding NOTHING. Window lookups have therefore silently
never resolved in production. Per-entry decode keeps the ~2,900 good rows; the
resolved rates match the benchmark harness's published price vector exactly.
* metrics.Snapshot gains the four billed token tiers plus attempted/frozen tokens and
the two ratios derived from them. Additive only -- deploy/harbor parses this payload.
Assisted-By: Claude Opus 5
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Serves the dashboard at /dashboard/ and mounts the API at /api/*, both only when
--dashboard is passed. Without the flag the route table is byte-identical to before and
every dashboard path 404s, verified by a test.
The UI is three files -- index.html, style.css, app.js -- embedded with go:embed. No npm,
no bundler, no framework, and deliberately NO CDN: context-guru ships into VPC and
air-gapped contexts, so a script tag pointing at unpkg is a shipping bug rather than a
nit. A test fetches each asset and fails on any external origin, and the assets are served
under `default-src 'self'` so a CDN tag added later breaks loudly in the browser instead of
silently breaking the offline install.
Charts are hand-drawn SVG rather than a vendored chart library. SVG path/rect/text is a
native browser feature, the series are small, and 45KB of dependency would buy a tooltip
that is fifteen lines here.
Design tokens are CSS custom properties from line one, with dark mode redefining only the
tokens -- both under prefers-color-scheme and under an explicit [data-theme], so the toggle
wins in both directions. (The reference implementation we studied hardcoded hex and had to
bolt on ~25 light-mode overrides afterwards.)
Views: Overview (20 stat tiles, the four labelled denominators with their prose
descriptions, the baseline-vs-actual cumulative cost chart with the saved area shaded, the
honest savings waterfall, safety-mechanism costs, cache-miss attribution, uncompressed
reasons, accounting confidence, five time series, and an SSE live feed), Components,
Sessions, Requests with the request drawer, Benchmarks with a cost-vs-reward scatter and
per-task drill-down, and Config.
The diff view is the headline feature -- the view both reference implementations carry the
data for and neither built. It is an LCS diff with common head/tail trimmed first (agent
transcripts share long identical stretches), rendered Git-style with line numbers and
collapsed unchanged runs, plus side-by-side and after-only modes. Blocks are ordered
biggest-saving first and the largest opens, because leading with an unchanged block buries
the answer to "what did context-guru actually remove?".
Every stat tile and panel carries a data-testid, asserted by a Go test so a rename fails
before the browser checks and the docs screenshots silently go stale.
Nothing renders through innerHTML. A tool output in a transcript is attacker-influenced
text, so every value goes through textContent or the el() helper, which throws on a raw
`html` prop. The same reasoning keeps the strict style-src CSP: inline style ATTRIBUTES
are blocked, so el() routes styles through the CSSOM, which is exempt and equally
expressive.
Proxy wiring:
* proxy/usage.go reads the four billed token tiers from a response in either dialect.
Anthropic's input_tokens already excludes the cached tiers; OpenAI's prompt_tokens
INCLUDES its cached_tokens, so fresh is the difference -- getting that backwards
double-counts the whole transcript every turn, which is exactly the kind of error a
savings number conceals.
* Streamed responses are sniffed through a BOUNDED head+tail window as the bytes go by,
never buffered: head because Anthropic reports the input tiers in the first SSE event
and the output count in the last, tail because OpenAI reports everything in a final
chunk. A newline separates the halves so a truncated line cannot fuse two events into
one unparseable one.
* The capture is finished in a defer AFTER the client's response is complete, so
redaction, gzip and the insert are all off the critical path.
* context-guru's own LLM spend is attributed per request as the delta of the cheap-model
counters across it.
Two real bugs the tests caught, both in the usage path and both silent under-reporting:
an output-only usage block (Anthropic's streaming message_delta, which carries the FINAL
completion count) parsed as "no usage at all"; and the sniffer dropped its tail window
whenever both halves saturated, losing usage for every large streamed response.
A golden test pins the exact key set of /stats and of each per-component object, in both
directions -- a lost field fails, and a NEW field fails until it is recorded in the golden
list. deploy/harbor/*.py parses that payload to produce every published benchmark result,
so a rename would invalidate the reproduction path silently: the harness would keep
running and report zeros, which is worse than a build break.
Also guards a class of bug that shipped a blank dashboard during development: a dropped
closing paren in app.js. Go compiled, every Go test passed, the HTML served 200, and the
page rendered nothing. A test now parses app.js with node when present and falls back to a
bracket-balance scan (strings, template literals and comments skipped) otherwise.
Assisted-By: Claude Opus 5
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…cture note New docs/dashboard.md: what each view shows, the five honest-metrics rules, the capture/store architecture with the measured overhead, the storage and retention model, the access-gating table, the flags and the API -- with screenshots taken from the real app driven by browser automation against live captured traffic, not fixtures. docs/how-to/measure-savings.md is rewritten around the dashboard and around one question it previously ducked: WHICH savings number should you quote? It now leads with a table mapping the question you are actually asking to the denominator that answers it, explains why a whole-request ratio trends to ~0% on a long session (the denominator grows quadratically as the agent re-sends its history), and explains why cost and token savings diverge -- a cache write bills ~11.5x a read, so removing unique tokens moves 0.02-0.13% of the billed total. /stats keeps its own section as the scriptable snapshot, with the stability contract stated. docs/design.md gains "Observability: the dashboard store", framed as five decisions and what each one refuses. docs/reference/routes.md documents every new route, every filter parameter and the access gating. docs/reference/config.md documents the flags, including why there is deliberately no "disable observability in production" switch: for a tool whose value IS observability, that would be backwards. Screenshots are cropped and JPEG-encoded to keep the whole set to ~2.5MB rather than the 10MB the raw 2x captures came to, and each one was scanned -- byte-level against the live credential values and pattern-wise, then again with OCR -- before being committed. `mkdocs build --strict` is clean. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…quest path
Three reviewer blockers on the dashboard, each with the regression test that
would have caught it.
**net_dollars_saved was inflated 13.1x.** `Event.Price` built the baseline from
`Saved()` — tokens_before − tokens_after for THIS turn — and the agent re-sends
its whole transcript every turn, so one compaction was paid for once per
remaining turn. The dashboard computed that exact factor and rendered it as
`overcount_ratio` in the tile beside the dollar figure, without applying it. On
63 replayed requests: $7.00 reported against $0.54 unique-based.
Two errors on one line, and the second multiplies the first: non-unique removed
tokens would have been served as cache READS, not writes, so pricing them at
CacheWrite (11.5x a read) inflates on top of the overcount. Now: unique savings
at the write rate, the re-sent remainder at the read rate, clamped so a
per-component unique figure exceeding a request's own gross saving cannot make
the remainder term negative.
Every downstream figure — session rows, the savings waterfall, the cumulative
baseline chart — sums the per-row `baseline_cost_usd`, so they all follow from
this one fix. `TestDollarsDeriveFromUniqueNotGrossSavings` builds a
deliberately-overcounted fixture (one 1,000-token compaction re-sent over ten
turns, overcount_ratio 10) and pins the aggregate dollar figure against it: the
old formula reports $0.0250 where $0.0043 is correct, and fails.
**Content redaction cost ~53 ms on the request goroutine.** `capture.finish`
runs from `serve`'s defer, which executes before the handler RETURNS, so nine
regexes over up to 48 blobs were paid by the next request on a keep-alive
connection — every real agent. The documented "0.000002%" was really ~25%.
`BenchmarkRecord`'s ~175 ns was honest about the channel send and blind to the
rest, because it calls `Record` directly and never `finish`.
Redaction moves to the writer goroutine, which already owns the event. Secrets
still never reach disk: it now runs immediately before the INSERT instead of at
the capture site, so what changed is which goroutine pays. The guard is an
end-to-end handler-latency test with content capture ON over one keep-alive
connection, paired and interleaved against a dashboard-off handler and compared
on medians (measuring one config fully and then the other attributes machine
drift to the dashboard: consecutive runs of that shape disagreed 4x). Measured:
−3 ms, i.e. within noise. Restoring redaction to the request path: +87 ms, test
fails.
**Content redaction leaked 11 of 22 realistic credential shapes.** Worst was
`Authorization: Bearer <token>`: the pattern matched `\S+` after the colon, so
it redacted the word "Bearer" and left the credential rendered in the diff view.
Also missing: JSON-form `{"api_key": …}` (the leading `\b[A-Z0-9_]*` could not
cross the quote, which also let GCP service-account keys through), basic-auth
URLs, Azure connection strings, glpat-/sk_live_/hf_/AIza/ya29 prefixes.
Fixed, and the 22 shapes are now a table-driven test: auth headers match to
end-of-line rather than one token; URL userinfo has its own rule that replaces
only the password, so the host stays readable; the prefix families collapse into
one alternation instead of seven passes; the assignment rule accepts an optional
quote before the name and treats `;` as a value terminator. The config path gets
the other half of the same gap — an allowlisted key like `anthropic_upstream` had
its VALUE passed through verbatim, and an upstream URL is exactly where a
`user:password@` credential lives.
But 22/22 passing does not make a denylist complete, and content is arbitrary
agent output that cannot be allowlisted the way headers and config keys are. So
`--dashboard-content` now defaults OFF. The diff view is still the best thing
here; it is opt-in.
Also, from the same review:
- `Open(":memory:")` used `file::memory:?cache=shared`, where the NAME
identifies the database — so every in-memory dashboard in the process was the
same one. Per-instance name, with a leak test. cache=shared stays: a private
in-memory DB is per-connection and `database/sql` pools connections.
- `IngestBenchDir` committed the run row before knowing whether any rows-* file
parsed, so the log said 17 runs while the API returned 42 with 25 empty. The
row is committed only when tasks > 0, with a test asserting the counter and
the table agree.
- `dash.Options.Mode` was set to a hardcoded "active" and never read. It now
carries the real mode and `/api/capture` serves it with the off-path pool's
queue counters, so the UI renders the observe banner the issue asked for —
including `dropped`, which tells the reader the projection UNDERSTATES.
`dash.Options.Preset` was a second copy of a value the capture site already
reads from proxy.Options; deleted rather than wired.
Docs updated where they stated the old overhead figure and the old content
default. The measure-savings and design notes now describe why the cheap half of
a two-part path is the wrong thing to benchmark.
Assisted-By: Claude Opus 5
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
The cumulative-cost chart's caption and the dashboard guide both still stated the rule the pricing fix removed — "baseline prices the tokens we removed at the cache-write rate" — which is the inflated formula described as if it were the behaviour. A dashboard whose caption contradicts its own number is worse than one with no caption, so both now describe the actual split: unique savings at the write rate, the re-sent remainder at the read rate. The overview screenshot is re-shot against a live proxy on port 4193 replaying the same 63 captured SWE-bench requests, because the committed one showed the inflated dollar figures. Verified before committing: no credential-shaped strings in the JPEG bytes, no EXIF/XMP/GPS. UI re-verified against that build with Playwright: 0 console errors, 0 page errors, 0 off-origin requests, all six views render, request detail and drawer work, XSS flags still false with one script tag (the app's own), and no horizontal overflow at 390px. The observe banner is hidden in sync mode and, on a proxy started with --mode observe, renders "You are currently in OBSERVE mode…" with the off-path queue's processed/pending/dropped counters. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
OsherElhadad
force-pushed
the
feat/i30-dashboard-rebased
branch
from
August 10, 2026 17:48
d08e133 to
85e84c4
Compare
Collaborator
Author
|
Superseded by #58 — same five commits rebased onto Three conflicts resolved, one of them consequential: both this branch and #56 changed the same Also worth noting: this PR's own |
github-project-automation
Bot
moved this from New/ToDo
to Done
in Rossoctl Issue Prioritization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #30. Supersedes #38 — same work, rebased onto
main(9 PRs landed since thatbranch forked) with the three review blockers fixed. #38 could not be updated in place: the
rebase rewrote history and force-push is blocked by branch protection.
What changed since #38
🔴 BLOCKER 1 —
net_dollars_savedwas inflated (gross vs unique)Event.Pricebuilt the baseline fromSaved()(tokens_before − tokens_after for thisturn). The agent re-sends its whole transcript every turn, so one compaction was paid for
once per remaining turn — and the tile rendered the correction factor,
overcount_ratio,immediately beside the figure without applying it.
Two errors on one line, the second multiplying the first: non-unique removed tokens would
have been served as cache reads, not writes, so pricing them at
CacheWrite(11.5× aread) inflated on top of the overcount. Now unique savings are priced at the write rate and
the re-sent remainder at the read rate, clamped so a per-component unique figure exceeding a
request's own gross saving cannot drive the remainder term negative.
Every downstream figure — session rows, the savings waterfall, the cumulative baseline chart
— sums the per-row
baseline_cost_usd, so all of them follow from the one fix.Measured on the same 63 replayed SWE-bench requests the review used:
9.3× inflation removed, verified two ways: recomputing from the per-row API against the
served aggregate (exact match to 9 decimal places), and
TestDollarsDeriveFromUniqueNotGrossSavings,which builds a deliberately-overcounted fixture (one 1,000-token compaction re-sent over ten
turns,
overcount_ratio10) and pins the aggregate dollar figure. RevertingPriceto theold formula makes it fail:
$0.0250where$0.0043is correct.🔴 BLOCKER 2 — redaction ran on the request goroutine
capture.finishis called fromserve'sdefer, which runs before the handler returns,so nine regexes over up to 48 blobs were paid by the next request on a keep-alive connection
— every real agent.
BenchmarkRecord's ~175 ns was honest about the channel send and blindto the rest, because it calls
Recorddirectly and neverfinish.Redaction moves to the writer goroutine, which already owns the event. Secrets still never
reach disk: it now runs immediately before the INSERT rather than at the capture site, so
what changed is which goroutine pays. The regex pass also got cheaper (one merged prefix
alternation instead of seven, and a
ContainsAny(":=")prefilter in front of the expensivekey-assignment rules).
Measured with content capture ON, 24-tool-result transcript, one keep-alive connection,
medians of 40 paired and interleaved requests against a dashboard-off handler:
The guard is now
TestDashboardAddsNoRequestLatencyWithContentCaptureat the handler level.Restoring redaction to the request path measures +87 ms and fails it. The pairing matters:
measuring one config fully and then the other attributes machine drift to the dashboard, and
consecutive runs of that shape disagreed by 4×.
🔴 BLOCKER 3 — content redaction leaked 11 of 22 credential shapes
Reproduced (12 with a lowercase variant), now 22/22 redact, pinned by a table-driven
test. The worst was
Authorization: Bearer <token>: the pattern matched\S+after thecolon, so it redacted the word "Bearer" and left the credential rendered in the diff view.
Also fixed: JSON-form
{"api_key": …}(the leading\b[A-Z0-9_]*could not cross thequote, which also let GCP service-account keys through), basic-auth URLs, Azure connection
strings, and the
glpat-/sk_live_/hf_/AIza/ya29.prefixes. Auth headers nowmatch to end-of-line; URL userinfo has its own rule that replaces only the password so the
host stays readable. The config path gets the other half of the same gap — an allowlisted key
like
anthropic_upstreamhad its value passed through verbatim, and an upstream URL isexactly where a
user:password@credential lives.But 22/22 passing does not make a denylist complete, and content is arbitrary agent output
that cannot be allowlisted the way headers and config keys are. So
--dashboard-contentnow defaults OFF. The diff view is still the best thing here; it is opt-in.
Mediums
Open(":memory:")usedfile::memory:?cache=shared, where the name identifies thedatabase — so every in-memory dashboard in the process was the same one. Per-instance name,
with the leak test.
cache=sharedstays: a private in-memory DB is per-connection anddatabase/sqlpools connections.IngestBenchDircommitted the run row before knowing whether anyrows-*.jsonparsed, sothe log said 17 runs while the API returned 42 with 25 empty. Committed only when
tasks > 0. Verified live against/tmp/cg-runs,/tmp/tb-runs: log 17, API 17, 0empty-arm rows (was 42 / 25).
dash.Options.Modewas set to a hardcoded"active"and never read; it now carries thereal mode.
dash.Options.Presetwas a second copy of a value the capture site alreadyreads from
proxy.Options— deleted rather than wired.Observe mode (#43) and
observe_queue(#50)Both merged while #38 was open, so the dashboard now renders them.
/api/captureserves themode and the off-path pool's counters, and the UI shows an unmissable banner: "You are
currently in OBSERVE mode. context-guru did not modify any request…" plus
processed/pending/dropped.droppedis called out specifically, because a droppedobservation means the projection understates what compaction would have saved. Verified
live on a
--mode observeproxy.applyModereturns the zeroTracein observe mode deliberately: the enforced path rannothing, so crediting the off-path projection to a request that was forwarded untouched is
exactly the confusion the
potential_*namespace exists to prevent.Rebase notes
Traceis now embedded inapply.Resultrather than living beside it, soSessionandRunexist once instead of twice — #43'sBodyOpts/Resultrefactor and this branch'sBodyTracehad introduced two copies of both.BodyTraceis gone;BodyOptsis the singleimplementation and
BodyFullstill delegates to it, so the rewrite stays byte-identicalwhether or not anyone reads the trace. Two early returns that built a fresh
Resultwerechanged to assign, or a bypassed request with no
messagesarray would have reported itselfas
no_messages.metrics.Snapshotconflicts were additive on both sides and all fields are kept — the naivekeep-both would have folded the dashboard's token-tier fields into
QueueStats, which isthe interleaving hazard flagged for
components/go. The eight observe/mode keys from #43/#50are added to the golden test's reviewed list;
/statsremains append-only.Gates
CGO_ENABLED=1 go build -tags cg_skeleton ./...·go test -tags cg_skeleton ./...·go test -race ./... -count=2·make lint·mkdocs build --strict— all clean.UI re-verified with Playwright against a live build on port 4193 replaying the 63 captured
requests: 0 console errors, 0 page errors, 0 off-origin requests, all six views render,
request detail and diff work, XSS flags still
falsewith one script tag (the app's own), nohorizontal overflow at 390 px. The overview screenshot is re-shot because the dollar
figures changed, and scanned for credential shapes and EXIF/XMP/GPS before committing.
Not fixed, deliberately
Global-counter LLM attribution, clear-on-overflow vs LRU, and the 1,500-line diff fallback
stay as documented
ponytail:ceilings, per the review.?refresh=1mutating on anunauthenticated GET,
VACUUMin the writer loop, the single-bucket chart degenerating to onepoint,
percentile()'s> 0filter, andcg_latency_msrendering as raw ms in the drawerare all still open and non-blocking.
Original #38 description
Closes #30
context-guru had no dashboard: the entire observability surface was
GET /stats, onein-memory
Aggregatorsnapshot that vanished on restart. A user could not answer thequestion the product exists to answer — what value is context-guru providing?
This adds a persistent dashboard: durable per-request storage, an off-hot-path capture
pipeline, a filterable JSON/SSE API, and an embedded single-page UI.
--dashboardturns iton; without the flag the route table is byte-identical to before and
/statsis unchanged.What is here
Storage — SQLite via
modernc.org/sqlite(pure Go, no extra C toolchain), WAL mode.requests·request_components·request_content(gzip, size-capped, skippable) ·bench_runs/bench_tasks. Epoch-ms timestamps throughout; a formatted locale stringcannot be range-queried, sorted portably, or bucketed. Retention bounded by age and
size. Schema versioned, with a mismatch renaming the old file aside rather than refusing to
boot.
--dashboard-db :memory:for no-persistence, which is also the automatic fallbackwhen the path is unwritable.
Capture off the hot path — the handler builds one struct from values the request path
already computed and does a channel send with a
default:branch. A full queue drops andcounts; the drop count is surfaced in the UI, because a dashboard that hides its own
coverage gaps cannot be trusted about anything else. One writer goroutine batches inserts
and fans summaries to SSE.
API —
/api/stats,/api/series?bucket=,/api/requests(server-side filters +keyset pagination),
/api/requests/{id},/api/sessions,/api/components,/api/facets,/api/config,/api/benchmarks[/{id}/tasks],/api/capture,/api/events(SSE honoring
Last-Event-ID). No rollup tables — series are bucketed in SQL at query time.UI — three files via
go:embed. No npm, no bundler, no framework, and no CDN:context-guru ships into VPC and air-gapped contexts, so a test fails on any external origin
and the assets are served under
default-src 'self'. Charts are hand-drawn SVG. Designtokens are CSS custom properties from line one, so dark mode redefines only tokens.
Metric semantics — the actual point
overcount_ratiosurfacedtoken_accountingper rowcomplete|partial|missing. An unpriceable request reads unknown — never zero, never exacthit · cold_start · ttl_expiry · prefix_change · unknown; cold start is not a failure, TTL wins tiesbypassed · below_trigger · cache_frozen · found_nothing · reverted · no_messagesCost is computed at write time so history does not reprice; percentages at read time
so a filter change needs no rebuild.
The headline feature: the before/after diff
Both reference implementations carry this data and neither renders it.
LCS diff with common head/tail trimmed first, Git-style hunks with line numbers and
collapsed unchanged runs, plus side-by-side and after-only modes. Blocks are ordered
biggest-saving first and the largest opens — leading with an unchanged block buries the
answer.
Per-component economics
On the real traffic above the verdicts do their job:
extractis earning its place,cacheinjectmutates, saves no content (its win is provider-side), and a component thatburned wall time for nothing reads costly and inert.
Benchmarks
Ingested straight from the harness artifacts (
summary.json+rows-<arm>.json) — no newexport format, so 42 historical runs from
/tmp/tb-runsand/tmp/cg-runsingested onfirst boot (652 task rows). Per arm: solve rate, mean reward, total/mean cost, cost per
solve, cache hit rate, exceptions, with a cost-vs-reward scatter and per-task drill-down.
Re-ingesting replaces a run rather than duplicating it.
Sessions, requests, config, dark mode
Measured capture overhead
The issue asks for this explicitly, since a dashboard that costs request latency is a
regression in a tool that sells latency awareness.
~175 ns per request on the request goroutine — against a measured 8.4 s mean upstream
round trip on this workload, that is ~0.000002%.
BenchmarkRecord's 7.5 µs includes thebenchmark's own
&Event{}allocation, which the handler does regardless. The full-queue(drop) path is 388 ns, so an overloaded dashboard degrades instead of becoming a latency
incident. A regression test fails above 50 µs, so moving I/O onto the capture path breaks CI.
Security
Redaction happens before the database, never on read — a secret on disk is a secret
forever, and a redact-on-read filter is one forgotten code path from leaking it. Headers are
blanket-redacted by key against a short allowlist (a denylist fails the moment a gateway
invents a new auth header); config keys are allowlisted with credential-named keys always
withheld; content is pattern-scrubbed then capped, so a secret at the truncation
boundary cannot survive by being cut into an unmatchable prefix.
Per-request content and the effective config are loopback/trusted-CIDR only; aggregates stay
open, because a proxy bound to
0.0.0.0should still report its own numbers. There isdeliberately no "disable observability in production" switch.
Real-data validation (not fixtures)
Everything above is rendered from real traffic through this proxy on port 4107:
claude-codetranscripts, against the live provider — so all four billed token tiers, the prompt-cache
structure, cold starts and TTL expiries are real, not synthesised.
claudeCLI session through the proxy.drill-down verified.
Observed: 4 sessions · 120k → 106k content tokens · 14k gross / 2.0k unique saved
(overcount 7.0×) · 328k cache reads vs 144k writes · $0.46 baseline vs $0.44 actual ·
accounting 32
complete/ 9partial· cache misses attributed 26 hit / 8 unknown /4 cold_start / 3 ttl_expiry.
Tests
dash: schema migration + version-mismatch preservation, retention by age and size(with cascade), keyset pagination covering every row exactly once, every filter
dimension, query-time bucketing, redaction (incl. a canary written through the real
capture path and read back from every column), the capture channel dropping rather than
blocking with the drop counted, SSE fan-out with a hung client evicted,
Last-Event-IDbackfill, cache attribution, benchmark ingest from real harness-shaped artifacts.
proxy: golden/statsshape in both directions (a lost field fails; a new fieldfails until recorded), usage parsing in both dialects, the streaming sniffer, and
end-to-end capture through a real pipeline.
go test -race ./...— 20/20 packages green, including the capture path and SSE hub.make lintclean;mkdocs build --strictclean.Browser verification: 124 checks, 0 failures against the live app — every stat tile's
data-testid, chart rendering and tooltips, filters, pagination, the drawer, all three diffmodes, benchmark drill-down, dark/light, a 390 px viewport, and empty states. Every
screenshot was scanned for credentials byte-wise and by OCR before being committed.
Bugs found and fixed along the way
Four real defects, each with a regression test:
internal/modelinfonever resolved anything in production. The LiteLLM prices mapwas decoded into one typed
map[string]struct{…}, which fails on the document'sprose-filled
sample_specentry and on models that spell an integer as a float —yielding nothing, so every context-window lookup silently returned "unknown" and
fraction-based triggers have never fired. Per-entry decode keeps the ~2,900 good rows;
the resolved rates now match the harness's published price vector exactly. (Pre-existing,
unrelated to the dashboard.)
message_delta, which carries the final completion count — parsed as "no usage".usage for every large streamed response.
app.jsrendered a blank dashboard. Go compiled, every Go testpassed, the HTML served 200. There is now a test that parses
app.js(node when present,a bracket-balance scan otherwise).
Plus two redaction over-reaches caught by the browser checks: redacting the
componentssubtree wholesale made the config view show nothing, and a substring match on
tokenalsoswallowed
max_tokens/min_tokens. Safety that destroys the feature is not safety.Notes for review
apply.BodyFullnow delegates toapply.BodyTrace, so the rewrite is byte-identicalwhether or not anyone is looking. No behaviour change on the compaction path.
metrics.Snapshotgains fields only — never renames or removals — guarded by the goldentest, because
deploy/harbor/*.pyparses that payload by name.modernc.org/sqlite(pure Go).Open concerns
process-global cheap-model counter, so under concurrency a request can be charged a
neighbour's call. Correct in aggregate, approximate per row. Fixing it properly needs the
Modelinterface to carry a per-request label.Recorder.Observe/MarkUniqueclear their maps on overflow rather than evicting LRU(marked with
ponytail:comments). A reset re-reports a cold start, which is honest butloses attribution precision; it needs a real LRU only if session churn gets large.
is O(n·m). A real Myers O(nd) would remove the ceiling.
under different roots collide.
modecolumn and filter are in place, but onlyactive/bypassoccur today —observeneeds feat(proxy): three operating modes — sync, async (cache-safe deferred compaction), and observe (measure without enforcing) #31.