Prune-value dashboard + agent-agnostic metrics (D0–D5) + guard receipts#141
Merged
Conversation
added 10 commits
June 16, 2026 10:02
… path) Add src/cozempic/metrics.py: PruneReceipt schema (v1.0), a PURE build_receipt() rollup of PrescriptionResult, the MetricsAdapter measurement seam + ClaudeMetricsAdapter, privacy-safe hashing, serialization, and validate_receipt. Feeds the prune-value dashboard now; the Codex adapter inherits it by implementing MetricsAdapter. Local-only, no network telemetry. Fleet-review fixes (19 confirmed findings): - token method vocab: the live value is "exact" (not "usage"); confidence map, validator, and tests corrected so exact->high. Previously every highest-confidence prune was silently stamped confidence "none"; the test masked it by hardcoding a value the pipeline never emits. - per-strategy token apportionment uses largest-remainder (Hamilton) so it sums exactly to tokens.reclaimed (plain rounding drifted). - cap trigger.reason / validation.defer_reason at the contract boundary; agent-neutral seam annotations (Sequence[Any]); stronger validate_receipt (method/mode/source enums + strategy item shape). 36 tests; full suite 1439 passing, 0 regressions.
…ard path) Add src/cozempic/receipts.py: write_receipt/emit_receipt persist PruneReceipts to ~/.cozempic/receipts/<session_hash>.jsonl + index.jsonl. Local-only, opt-out via COZEMPIC_NO_RECEIPTS, exception-isolated so a receipt can NEVER break or defer a prune. Wire _emit_prune_receipt into cmd_treat and cmd_reload at the committed and both deferred (lock/conflict) branches. Fleet-review fixes (15 confirmed, all medium-or-below): - index.jsonl is a shared cross-session file with no _PruneLock coverage: append via a single os.write to an O_APPEND fd (atomic under PIPE_BUF) so concurrent sessions cannot interleave/tear lines; docstring corrected (it over-claimed _PruneLock serialization). - _session_stem hardened (strip path separators + leading dots: no escape/dotfile). - trigger.tier falls back to "custom" for non-builtin prescription names instead of emitting a non-tier value; dropped redundant reason=source. - tests: validate_receipt on emitted receipts, deferred-path integration, _session_stem units, corrupt-index tolerance, never-raise asserts no garbage file. Follow-up (tracked): wire guard auto-prune / overflow / precompact entry points and a "failed" receipt on PruneValidationError — separate slice (guard.py is higher-risk). 24 D1 tests; full suite 1455 passing, 0 regressions.
Add src/cozempic/dashboard/ subpackage: load_receipts (tolerant — skips torn/ garbage/non-receipt lines) + a PURE aggregate() producing lifetime totals, a per-strategy leaderboard, per-agent grouping, tier distribution, and per-session timelines (context % over time). Agent-agnostic: Codex receipts aggregate unchanged. Honesty rule: only committed prunes count toward reclaimed savings. Fleet-review fixes (14 confirmed, all medium-or-below): - aggregate() now genuinely defensive: a _d() helper routes every nested access, so a receipt with non-dict session/agent/trigger/model (or non-list strategies, or missing ts) degrades to "unknown"/0 instead of crashing the whole load. - timeline honesty: non-committed prunes show 0 reclaimed (was leaking phantom savings); first/last_ts filter non-str; outer per_session sort None-safe. - tests: malformed-receipt robustness, no-mutation/purity, mixed-agent session, leaderboard ordering, JSON array/scalar load tolerance. 17 D2 tests; full suite 1472 passing, 0 regressions.
…D3/D4 = Claude v1) Add src/cozempic/dashboard/render.py: render_html (self-contained, zero-dep, server-side rendered — CSS bars + inline-SVG sparklines, NO client JS, every value HTML-escaped) + atomic write_dashboard + render_dashboard convenience. Wire a `cozempic dashboard` subcommand that loads receipts -> aggregates -> renders ~/.cozempic/dashboard.html and opens it (--no-open to suppress). Fleet-review fixes (13 confirmed, 1 medium): - render no longer crashes on non-string by_tier keys (sort/coerce via str()). - sparkline caps to last 60 points, drops NaN/inf, clamps y into the viewBox; _bar_rows floors its scale at 0 (all-negative -> empty bars); deferral_rate None-safe; _fmt_tokens rolls 999,999 -> "1.0M". - `dashboard` added to _AUTO_INIT_SKIP_CMDS (report-only, never mutates project state); webbrowser.open False return prints the manual-open hint. - tests: non-string tier key, all-zero/negative/None values, negative-pct sparkline clamp, write-failure cleanup, default browser-open, empty stdout. Ships the Claude prune-value dashboard. 17 D3/D4 tests; full suite 1489 passing.
… Codex inherits)
Add CodexMetricsAdapter (metrics.py): the second MetricsAdapter, proving the
contract is agent-agnostic — a Codex prune emits the SAME PruneReceipt that flows
through write -> load -> aggregate -> render with ZERO changes to those modules.
Add `cozempic dashboard --agent <name>` to filter one agent; Codex sessions appear
tagged agent:codex.
Fleet-review fixes (14 confirmed; verified against real ~/.codex rollouts):
- count_tokens read the WRONG payload shape: real codex 0.139 nests the total at
payload.info.total_token_usage.total_tokens (and the window at
info.model_context_window), so the flat lookup never fired and silently fell to
the byte heuristic. Now reads the nested fields (flat alias kept as fallback);
context_window returns the real model_context_window.
- the --agent isolation test was brittle AND didn't verify isolation (a no-op
filter would pass): replaced with a data-level assertion + assertNotIn("claude").
- cli filter hardened against a non-dict agent; filter-aware empty-state message.
- tests: real nested token_count, flat-alias/bool guard, default window,
unknown-agent empty state.
10 D5 tests; full suite 1499 passing, 0 regressions. Completes the D0->D5 path.
The dashboard's per-prune receipts are new, so a real user opening it sees "1 prune" while their actual history is thousands. Surface cozempic's existing lifetime ledger (helpers.record_savings) as a header band so `cozempic dashboard` leads with TRUE totals (e.g. 456.2M tokens across 3,309 prunes). New dashboard/lifetime.py (tolerant load_lifetime), band in render.py, cmd_dashboard loads + prints it. Band renders even with zero receipts. Also fix a test-hygiene leak the D1 cli wiring introduced: cmd_treat/cmd_reload now emit receipts on execute, so existing treat/reload tests wrote receipts into the developer's real ~/.cozempic. Added an autouse conftest fixture that disables receipt writing suite-wide; receipt tests opt back in within isolated dirs. Fleet-review fixes (10 confirmed, all low): - impossible savings rate (processed<saved, corrupt ledger) suppressed, not shown as "250% reclaimed". - honest labels: "avg reclaimed" -> "reclaimed of processed" (it's saved/processed, not a per-prune average; processed is cumulative-with-overlap); "+N extra turns" -> "~N est. extra turns" (matches the CLI's tilde convention). - --agent filter now suppresses the GLOBAL ledger (was contradictory: "456M reclaimed" beside "no prunes for codex"). - load_lifetime accepts float ledger values (was silently dropping the whole band). - conftest documents the pytest-only hermetic guarantee. 13 lifetime tests; full suite 1512 passing, leak-free.
…ked) Store distinct pruned-session count in ~/.cozempic_savings.json so the dashboard shows the REAL "Nx longer per pruned session" once measured. record_savings gains session_id; threaded through cmd_treat, cmd_reload, guard_prune_cycle. Fleet-review fixes (15 confirmed; the HIGH was caught pre-commit): - multiplier MUST use a forward-only tracked_prunes counter, NOT the lifetime prune_count: dividing lifetime prunes (3,309) by the forward-only session count produced an absurd 117x right at the sessions>=5 gate. Both operands now cover the same window (tracked_prunes/sessions). Regression test pins prune_count=3309 + tracked_prunes=10/sessions=5 -> 1.35x, never 117x. - distinct-session count no longer overcounts past the 50k hash cap (increment only when the hash is actually recorded); O(1) set membership. - honest-blank: the multiplier is suppressed until enough forward data is measured (no estimate, no inflation) per product decision. Session ids stored hashed (sha256[:12]); raw never written. 19 lifetime tests; full suite 1518 passing; tests never touch the real ledger/receipts.
…-prunes Wire _emit_guard_receipt into the guard's confirmed-write path so the dashboard's per-prune views capture AUTOMATIC prunes (guard daemon + overflow), not just the manual treat/reload paths. Emitted from _write_pruned_after_exit right after save_messages succeeds (post-terminate, pre-resume) — the single proven after-disk-write hook; covers the guard terminate-first path AND the auto_reload= False overflow path. Doubly exception-isolated; can never perturb the prune/ terminate/resume cycle. PreCompact is checkpoint/digest, not a prune — not wired. Fleet-review fixes (12 confirmed): - HIGH: unit tests called _emit_guard_receipt directly and missed the real closure wiring (a deleted call left tests green — proven by mutation). Added integration tests driving guard_prune_cycle -> deferred writer -> emit under a temp HOME, asserting a committed receipt lands ONLY on a confirmed write, plus negatives (read-only / un-written deferred prune emit nothing) and overflow source through the real path. Mutation-verified: neutering the emit fails the integration test. - emit moved off the token-delta gate -> EVERY confirmed write emits (byte-only / token-neutral overflow prunes now count), consistent with "committed == written". - trigger.tier normalized to "custom" for non-builtin prescriptions (matches cli). - documented the deferral_rate asymmetry (guard deferrals not yet emitted; tracked). Test-hygiene: the harness patches record_savings (the real _SAVINGS_FILE is a frozen module constant a HOME patch can't redirect) so the suite never touches the real ledger. Full suite 1526 passing; ledger + receipts untouched. Follow-ups (tracked): emit 'deferred' receipts from guard/overflow defer points; extract a shared emit helper between cli and guard.
…factor) origin/main advanced 13 commits while this branch was developed; the textual merge was clean but two semantic conflicts surfaced under the full suite: - #138 renamed the guard's load path to load_messages_and_snapshot + added _guard_tmp_root; the guard-receipt integration harness now patches those. - this branch added session_id= to the record_savings call; #138's test_deferred_writer_calls_record_savings assertion now expects session_id=ANY. Merged suite: 1745 passing, ledger/receipts untouched, emit wiring mutation-verified.
There was a problem hiding this comment.
Pull request overview
This PR introduces a local, privacy-safe “prune-value” dashboard built on an agent-agnostic receipt/metrics contract. It adds receipt emission in the CLI + guard/overflow paths, plus a static self-contained HTML dashboard that aggregates those receipts and can optionally surface existing lifetime totals from ~/.cozempic_savings.json.
Changes:
- Add an agent-agnostic
PruneReceiptcontract (metrics.py) with adapters (Claude + Codex) and structural validation. - Add local-only receipt persistence (
receipts.py) and wire receipt emission into CLI prune flows and guard/overflow confirmed-write hooks. - Add dashboard aggregation + rendering +
cozempic dashboardcommand (including lifetime ledger band support).
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/cozempic/metrics.py |
Defines the receipt schema/contract, adapters (Claude/Codex), and validation/serialization helpers. |
src/cozempic/receipts.py |
Implements local append-only receipt + index writers and a fire-and-forget emission helper. |
src/cozempic/dashboard/aggregate.py |
Loads and aggregates receipt logs into dashboard-friendly views. |
src/cozempic/dashboard/render.py |
Renders aggregated views into static, self-contained HTML and writes it atomically. |
src/cozempic/dashboard/lifetime.py |
Loads and normalizes the existing lifetime savings ledger for the dashboard band. |
src/cozempic/dashboard/__init__.py |
Exposes dashboard entry points via package exports. |
src/cozempic/cli.py |
Emits receipts for manual prune flows, adds dashboard subcommand, and tracks session_id in savings ledger. |
src/cozempic/guard.py |
Emits committed receipts after confirmed guard/overflow writes and tracks session_id in savings ledger. |
src/cozempic/overflow.py |
Tags overflow-triggered prunes with trigger_source="overflow". |
src/cozempic/helpers.py |
Extends lifetime savings ledger to track distinct pruned sessions (hashed) for a measured multiplier. |
tests/conftest.py |
Adds an autouse pytest fixture to prevent tests from writing real receipts to ~/.cozempic. |
tests/test_metrics.py |
Tests receipt construction, privacy hashing, adapter seam, serialization, and validation. |
tests/test_receipts.py |
Tests receipt persistence behavior (index/session logs, opt-out, corruption tolerance). |
tests/test_guard_receipt.py |
Tests guard/overflow receipt emission and the confirmed-write integration path. |
tests/test_dashboard_aggregate.py |
Tests aggregation correctness and robustness against malformed receipt lines. |
tests/test_dashboard_render.py |
Tests HTML rendering, escaping, “no JS/no external refs”, sparklines, and CLI behavior. |
tests/test_dashboard_lifetime.py |
Tests lifetime ledger normalization and multiplier logic + CLI suppression under agent filter. |
tests/test_codex_inherit.py |
Verifies Codex adapter inherits the same receipt → dashboard pipeline and agent filtering works. |
tests/test_cli_receipts.py |
Tests CLI helper receipt emission and failure isolation. |
tests/test_pr138_review_fixes.py |
Updates an assertion to include the newly added session_id parameter. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+72
to
+81
| """Append one line via a single ``os.write`` to an ``O_APPEND`` fd. | ||
|
|
||
| A single write to an O_APPEND descriptor is atomic for payloads under | ||
| PIPE_BUF (>=512 bytes — index summaries and most receipts qualify), so | ||
| concurrent appenders — INCLUDING different sessions racing the shared | ||
| ``index.jsonl`` (which no per-session ``_PruneLock`` protects) — cannot | ||
| interleave or tear those lines. Oversized receipt lines may still split | ||
| under contention; that is loss-tolerant by design — the D2 aggregator skips | ||
| any unparseable line and the prune itself is never affected. | ||
| """ |
Comment on lines
+205
to
+218
| if session_id: | ||
| import hashlib | ||
| h = hashlib.sha256(session_id.encode("utf-8")).hexdigest()[:12] | ||
| seen = data.get("_pruned_session_hashes") | ||
| if not isinstance(seen, list): | ||
| seen = [] | ||
| # numerator: prunes that occurred under session tracking | ||
| data["tracked_prunes"] = data.get("tracked_prunes", 0) + 1 | ||
| # denominator: distinct sessions, counted only when actually | ||
| # recorded (so past the cap `sessions` can't overcount) | ||
| if h not in set(seen) and len(seen) < 50_000: | ||
| seen.append(h) | ||
| data["sessions"] = data.get("sessions", 0) + 1 | ||
| data["_pruned_session_hashes"] = seen |
Comment on lines
+224
to
+234
| def render_dashboard(base_dir: Path | None = None, *, generated_ts: str) -> str: | ||
| """Convenience: load receipts + lifetime ledger -> aggregate -> render HTML.""" | ||
| from .aggregate import aggregate, load_receipts | ||
| from .lifetime import load_lifetime | ||
|
|
||
| return render_html( | ||
| aggregate(load_receipts(base_dir)), | ||
| generated_ts=generated_ts, | ||
| source_label="~/.cozempic/receipts", | ||
| ledger=load_lifetime(), | ||
| ) |
junaidtitan
pushed a commit
that referenced
this pull request
Jun 18, 2026
Bundles two workstreams since v1.8.33 (27 commits): Dashboard / receipts (#141-145): new `cozempic dashboard` static-HTML command, local prune receipts (default-on, COZEMPIC_NO_RECEIPTS opt-out, privacy-hashed), guard/overflow emit committed receipts, lifetime savings band, Codex adapter. Reload-gate & guard hardening (#131-139): phantom clear/terminate block for user-typed notifications (#134), reload-ledger env-knob bounds (#137), teammate-aware agents_active (#131), slug parity (#135), SIGTERM-leak/_pid_is_alive consolidation (#136), _constants.py + attribute-tolerant _TN_*_RE + _persisted_tokens_saved salvages (#132/#139), message_count regression fix, spawn-lock V4 gate repair, test isolation. Version bumped across all release files + packaging mirrors (homebrew/nix/macports). Suite 1745 passed. Pre-release verification fleet: ship, 0 confirmed blockers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A local, privacy-safe prune-value dashboard for cozempic, built on an agent-agnostic metrics contract so Codex (and future agents) inherit it for free.
`cozempic dashboard` → self-contained static HTML (zero deps, no JS) showing lifetime savings, per-strategy leaderboard, per-agent grouping, tier breakdown, and per-session context-trend sparklines. Leads with your real lifetime totals from `~/.cozempic_savings.json`.
How it's built (each slice fleet-reviewed)
Safety
Follow-ups (tracked, not in this PR)