diff --git a/AGENTS.md b/AGENTS.md index bdd60a7..d41b2a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,21 +28,23 @@ tokenix --help | `src/indexer.rs` | File walk + incremental index pipeline. Runs at below-normal OS priority (`lower_process_priority()`, opt-out `--no-low-priority`/`TOKENIX_FOREGROUND`). `decode_text()` handles UTF-16 BOMs (SSMS-saved `.sql`) and skips binary files (NUL in first 8 KiB). Embeds in batches (default 16) with a progress bar; each batch commits to the embedding cache so a killed run resumes via cache hits | | `src/query.rs` | Hybrid semantic/lexical ranking (FTS5 + BM25 + RRF), strict `context` modes, budget enforcement, cross-project search | | `src/pack.rs` | `tokenix pack` — budgeted repo map + focused context, changed-file packs, token maps, and safety report | -| `src/graph.rs` | Symbol graph with PageRank, cycle detection (Tarjan's SCC, homonym-filtered, `path:line`-annotated), tree-sitter references, incremental repair (`update_symbol_graph_incremental` — FTS-narrowed inbound-edge restore; `rebuild-graph` = full escape hatch), file-level import graph (`rebuild_import_graph`, per-language import extraction + path resolution), HTML + Mermaid export | +| `src/graph.rs` | Symbol graph with PageRank, cycle detection (Tarjan's SCC, homonym-filtered, `path:line`-annotated), tree-sitter references, incremental repair (`update_symbol_graph_incremental` — FTS-narrowed inbound-edge restore; `rebuild-graph` = full escape hatch), file-level import graph (`rebuild_import_graph`, per-language import extraction + path resolution), HTML + Mermaid export. Repo-wide overview (`tokenix graph`): `repo_hotspots` (degree + transitive-dependent blast radius, trivial-symbol filtered), `format_repo_report` (god nodes / bottlenecks / blast-radius leaders), `format_edges_dot` (Graphviz of the top subgraph) | | `src/artifacts.rs` | Context artifacts — index non-code files (schemas, API specs, docs) via `.tokenix/artifacts.json` | | `src/hook.rs` | `run_hook()` — called by PreToolUse hook. Tries daemon first for Grep. Thresholds (Read 200 lines / Grep 3 words) overridable via `[hook]` in `.tokenix.toml` (`read_min_lines`, `grep_min_words`) | | `src/daemon.rs` | Background TCP server (port 47392). Holds model + int8-quantized embedding cache (LRU, max 3 projects, content cap 1000). Bounded to 4 handler threads. Protocol: `search`/`health`/`status`; CLI `tokenix daemon status\|stop\|restart` | | `src/compress.rs` | Legacy `PostToolUse` compatibility compression + `tokenix run` command-output compression: ANSI strip, emoji removal, blank-line collapse, repeat grouping, JSON compaction, cargo/git-log heuristics. `tokenix run` only applies command-specific filters to stderr when `filter_stderr=true`; otherwise stderr uses safe generic compression so errors are not turned into success sentinels | | `src/filters.rs` | `FilterDef` (TOML schema), active filter listing, `load_user_filters()`, `load_bundled_filters()` (rust-embed), `apply_filter()`. `find_filter()` matches via `derive_command_candidates()`, which unwraps shell runners, strips `cd`/env prefixes, and `split_on_operators()` splits compound commands quote-aware on `&&`/`\|\|`/`;`/`\|` so anchored `match_command` patterns match a base command in any segment/position | | `src/cmd_filter.rs` | `tokenix filter list/active/generate` + `filter record start/stop/status` subcommands. `generate` prefers `recordings::read_samples` over a re-run, invokes a detected AI CLI, and saves to `~/.tokenix/filters/`; reused by the TUI Studio tab as a foreground drop-out | -| `src/tui.rs` | Interactive ratatui shell shown by a bare `tokenix` / `tokenix filter` in a TTY (else falls back to help / `filter list`). Tab bar (`←`/`→`): **Stats** dashboard (wordmark + version + hook status + index summary, with selectable Index / Install hooks / Install binary actions — Index runs in the foreground with live progress, the two install actions confirm before writing; Install binary self-execs `tokenix install-binary`), **Filters** (3-pane groups · filters · live `apply_filter` input→output preview with a `chunker::count_tokens` gauge line showing `X → Y tokens · % saved` between the panes), **Studio** (surfaces the record→preview→generate filter loop: `r`/`s` arm/stop a `recordings::start`/`stop` session, left column is a unified candidate list from `cmd_filter::suggest_filters` — recordings unioned with the tokens-wasted ranking, badged `⚠` unfiltered sink (biggest waste first) / `✓` already filtered / `●` recorded-only — plus saved `~/.tokenix/filters/*.toml`, right pane previews a `recordings::read_samples` head with a live `apply_filter` before→after `chunker::count_tokens` delta when an active filter matches the base command; `g` sets `request_generate` to run `cmd_filter::cmd_filter_generate` as a foreground drop-out — same pattern as Index — then resumes the TUI; `x` deletes a saved filter with confirm; `Tab` switches pane), **Gain** (native colored render of `gain::compute_gain`: tokens-saved headline with ≈USD at the ★ reference model's input rate, savings-by-source split — semantic index vs command filters — and numbered by command / by project tables with share %, toggles `c`/`a`), **Doctor**/**Tokenmap** (self-exec captured output), **Secrets** (background-threaded `secrets_scan::scan_findings` with spinner; dedup by distinct value + count; `v` reveal, `c` copy raw value to system clipboard via `clip`/`pbcopy`/`wl-copy`/`xclip`/`xsel`, `x` write `[REDACTED]`), **Egress** (background-threaded `egress_scan::scan_findings` with the same 3-pane pattern as Secrets: groups · destinations · occurrence detail; `s` cycles host/rule/agent/file grouping; `r` rescans; host reputation colors: green safe, red dangerous, yellow unknown). Both Secrets and Egress open scoped to the current repo (cwd) and `g` toggles a global all-repos view; scoping filters the raw scan by each finding's attributed `repo` (`is_local` matches exact `cwd` paths plus Claude `~slug:`/Gemini `~dir:` fallback markers against the project root) | +| `src/tui.rs` | Interactive ratatui shell shown by a bare `tokenix` / `tokenix filter` in a TTY (else falls back to help / `filter list`). Tab bar (`←`/`→`): **Stats** dashboard (wordmark + version + hook status + index summary, with selectable Index / Install hooks / Install binary actions — Index runs in the foreground with live progress, the two install actions confirm before writing; Install binary self-execs `tokenix install-binary`), **Filters** (3-pane groups · filters · live `apply_filter` input→output preview with a `chunker::count_tokens` gauge line showing `X → Y tokens · % saved` between the panes), **Studio** (surfaces the record→preview→generate filter loop: `r`/`s` arm/stop a `recordings::start`/`stop` session, left column is a unified candidate list from `cmd_filter::suggest_filters` — recordings unioned with the tokens-wasted ranking, badged `⚠` unfiltered sink (biggest waste first) / `✓` already filtered / `●` recorded-only — plus saved `~/.tokenix/filters/*.toml`, right pane previews a `recordings::read_samples` head with a live `apply_filter` before→after `chunker::count_tokens` delta when an active filter matches the base command; `g` sets `request_generate` to run `cmd_filter::cmd_filter_generate` as a foreground drop-out — same pattern as Index — then resumes the TUI; `x` deletes a saved filter with confirm; `Tab` switches pane), **Gain** (native colored render of `gain::compute_gain`: tokens-saved headline with ≈USD at the ★ reference model's input rate, savings-by-source split — semantic index vs command filters — and numbered by command / by project tables with share %, toggles `c`/`a`), **Usage** (self-exec captured `tokenix usage` via dynamic argv: `s` cycles daily/model/blocks/project/session, `a` toggles all-projects, `r` refresh), **Doctor**/**Tokenmap** (self-exec captured output), **Graph** (self-exec captured `tokenix graph` repo overview — god nodes / bottlenecks / blast radius; `r` refresh), **Secrets** (background-threaded `secrets_scan::scan_findings` with spinner; dedup by distinct value + count; `v` reveal, `c` copy raw value to system clipboard via `clip`/`pbcopy`/`wl-copy`/`xclip`/`xsel`, `x` write `[REDACTED]`), **Egress** (background-threaded `egress_scan::scan_findings` with the same 3-pane pattern as Secrets: groups · destinations · occurrence detail; `s` cycles host/rule/agent/file grouping; `r` rescans; host reputation colors: green safe, red dangerous, yellow unknown). Both Secrets and Egress open scoped to the current repo (cwd) and `g` toggles a global all-repos view; scoping filters the raw scan by each finding's attributed `repo` (`is_local` matches exact `cwd` paths plus Claude `~slug:`/Gemini `~dir:` fallback markers against the project root) | | `src/ui.rs` | Shared terminal-UI vocabulary for human-facing CLI output (`box_header`, `bar`, `section`/`kv`, `format_num`, `table` via `tabled`); LLM/JSON output deliberately does not route through it | -| `src/gain.rs` | `compute_gain()`/`compute_global_gain()`, `GainStats` (incl. `index_saved`/`filter_saved` source split: empty `command` = semantic-index intercept, non-empty = command filter; pre-phase Bash/PowerShell rewrite markers are excluded from `filter_calls`), `MODELS` pricing table (Anthropic/OpenAI/Google). Grep semantic intercepts are logged as neutral usage, not claimed savings, because native grep output is not measured before interception | +| `src/gain.rs` | `compute_gain()`/`compute_global_gain()`, `GainStats` (incl. `index_saved`/`filter_saved` source split: empty `command` = semantic-index intercept, non-empty = command filter; pre-phase Bash/PowerShell rewrite markers are excluded from `filter_calls`), `MODELS` pricing table (Anthropic/OpenAI/Google, with `input`/`output`/`cache_read`/`cache_write` per-1M rates; `price_for` name/prefix match + `usage_cost` per-record helper reused by `tokenix usage`). Grep semantic intercepts are logged as neutral usage, not claimed savings, because native grep output is not measured before interception | +| `src/transcripts.rs` | Shared enumeration of local agent transcript files (`roots` per agent: Claude/Codex/Copilot/OpenAI, `transcript_files` walker). Single source of truth reused by `conversation-audit` and `usage` | +| `src/usage.rs` | `tokenix usage` — absolute token spend + ≈USD cost parsed from transcript `message.usage` blocks (input/output/cache read+write), deduped by `(message.id, requestId)`. Aggregates by `daily\|weekly\|monthly\|session\|model\|project`; rolling 5-hour `blocks` with burn rate + projection; month-end forecast; `--cost-mode auto\|calculate\|display`; `--statusline`; `--all-projects` scope; `--json` | | `src/mcp.rs` | MCP server. `--profile full` exposes all tools; `--profile slim` exposes context/search/call meta-tools for progressive discovery | | `src/mcp_audit.rs` | `tokenix prompt-audit` / `session-audit` — per-agent MCP config discovery (Claude, Codex, Copilot, OpenCode, Antigravity) + minimal synchronous MCP stdio client (`initialize`/`tools/list`) + token scoring/report | | `src/secrets_scan.rs` | `tokenix scan-secrets` — gitleaks-style credential scan of Claude/Gemini/Copilot/Antigravity conversation transcripts under `~`; rules loaded from TOML (`assets/secret-rules/` bundled via `rust-embed`, extended by `/` then `~/.tokenix/secret-rules/*.toml`, later `id` wins), backtracking-free regex + entropy-gated generic rule. Each finding is attributed to its repo + git branch via the transcript line's `cwd`/`gitBranch` (Claude), falling back to the project dir slug. Report supports `--filter` (substring), `--group `, `--reveal` (raw values, default redacted), `--json`; exit 1 on hits. `scan_findings()` returns structured `ScanFinding`s (raw + redacted) for the TUI; `redact_in_files()` rewrites `[REDACTED]` over a value in text files (SQLite DBs skipped) | | `src/egress_scan.rs` | `tokenix egress-audit` — scans Claude/Gemini/Copilot/Antigravity conversation transcripts for external DNS/IP destinations; bundled TOML rules live under `assets/egress-rules/`, local safe hosts are loaded from `~/.tokenix/safe-hosts.toml`, and local blocklist hosts from `~/.tokenix/dangerous-hosts.toml` (`dangerous`, `blocklist`, or `hosts` arrays); report supports `--filter`, `--group `, `--safe`, and `--json`. `scan_findings()` returns structured `EgressFinding`s for the TUI | -| `assets/filters/` | 378 TOML output filters embedded via `rust-embed`, each homologated with ≥2 golden `[[tests]]` cases (realistic success + failure-path inputs; the failure case must prove errors are never masked). 784 cases run through the real `apply_filter` pipeline in `bundled_filters_pass_embedded_golden_tests`; `verbose_real_output_compresses_at_least_70pct` proves ≥70% reduction on realistic verbose output and `match_command_resolves_many_invocation_variants` homologates wrapper/shell/global-opt command variants. User filters in `~/.tokenix/filters/` take priority | +| `assets/filters/` | 386 TOML output filters embedded via `rust-embed`, each homologated with ≥2 golden `[[tests]]` cases (realistic success + failure-path inputs; the failure case must prove errors are never masked). 800 cases run through the real `apply_filter` pipeline in `bundled_filters_pass_embedded_golden_tests`; `verbose_real_output_compresses_at_least_70pct` proves ≥70% reduction on realistic verbose output and `match_command_resolves_many_invocation_variants` homologates wrapper/shell/global-opt command variants. User filters in `~/.tokenix/filters/` take priority | ## SQLite Schema @@ -240,7 +242,7 @@ but a `node` grandchild may linger briefly until stdin EOF. Kill-the-tree **Add a language:** `chunker.rs` — add extension to `INDEXED_EXTS`, add `Lang` variant, map in `detect_lang()`, implement `chunk_()` following `chunk_rust()` pattern (tree-sitter), or `chunk_by_symbol_lines()` with a `_symbol_of()` line matcher when no grammar is bundled (see VB6/SQL). Also add the new `Lang` arms in `graph.rs` (`extract_references_tree_sitter`, `extract_file_imports`). Do NOT add to `INDEXED_EXTS` without a symbol-aware chunker. -**Add a bundled filter:** create `assets/filters/.toml` with **≥2 embedded `[[tests.]]` golden cases** (input/expected — enforced by `bundled_filters_require_minimum_tests`). Filters with an `on_empty` sentinel must NOT also set `passthrough_when_emptied` (they conflict; passthrough wins and the sentinel never fires), and any filter that can empty a failure payload must keep failure markers (`(?i)error|fail|fatal`) or set `passthrough_when_emptied` — else `bundled_filters_never_mask_generic_failure` fails. Rebuild — rust-embed includes it automatically. Homologate with `cargo test --bin tokenix filters::tests::` (golden + 70% economy + never-mask + no-inflate). Currently 378 filters · 784 golden cases. +**Add a bundled filter:** create `assets/filters/.toml` with **≥2 embedded `[[tests.]]` golden cases** (input/expected — enforced by `bundled_filters_require_minimum_tests`). Filters with an `on_empty` sentinel must NOT also set `passthrough_when_emptied` (they conflict; passthrough wins and the sentinel never fires), and any filter that can empty a failure payload must keep failure markers (`(?i)error|fail|fatal`) or set `passthrough_when_emptied` — else `bundled_filters_never_mask_generic_failure` fails. Rebuild — rust-embed includes it automatically. Homologate with `cargo test --bin tokenix filters::tests::` (golden + 70% economy + never-mask + no-inflate). Currently 386 filters · 800 golden cases. **`filter record` token-economy preview:** `recordings::economy()` reconstructs each captured command's raw output (stripping the `$ cmd`/`--- stderr ---`/truncation scaffold), resolves the bundled filter via the real `find_filter`+`apply_filter` path, and reports `raw→filtered` tokens. `record stop`/`status` render it as a per-command compression bar + total via `print_economy_table` in `cmd_filter.rs`. @@ -376,10 +378,22 @@ Narrow context with: ```bash tokenix read --symbol tokenix read --lines N-M +tokenix read --mode signatures # signatures only (no bodies) +tokenix read --mode diff # outline + uncommitted hunks +tokenix read --mode density:40 # keep ~40% highest-entropy lines ``` Only read a full file directly when tokenix shows it is small. +Inspect the symbol graph and spend with: + +```bash +tokenix graph # repo-wide god nodes / bottlenecks / blast radius +tokenix graph --format dot # Graphviz of the top subgraph +tokenix usage # absolute token spend + ≈USD cost (daily) +tokenix usage blocks # rolling 5-hour billing blocks + burn rate +``` + ## Release Releases are automated via GitHub Actions (`.github/workflows/release.yml`). Pushing to `main` auto-creates a version tag and GitHub Release with pre-built binaries for Linux, macOS, and Windows. diff --git a/README.md b/README.md index 794340b..7976747 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ Savings depend on codebase size, AI behavior, and file sizes. Run `tokenix gain` ## 🖥 Interactive Dashboard -Run bare `tokenix` to open a terminal dashboard — eight tabs, zero flags. `←`/`→` switch tabs, `↑`/`↓` move, `q` quits. Piped or non-TTY falls back to `--help`. +Run bare `tokenix` to open a terminal dashboard — ten tabs, zero flags. `←`/`→` switch tabs, `↑`/`↓` move, `q` quits. Piped or non-TTY falls back to `--help`. @@ -50,7 +50,13 @@ Run bare `tokenix` to open a terminal dashboard — eight tabs, zero flags. `← - + + + + + + + @@ -164,6 +170,7 @@ The embedding model (`nomic-embed-text-v1.5`, ~130 MB) is downloaded automatical | **JSON output** | `--json` on `query`, `context`, `explore`, `read`, `symbols`, `callers`, `callees`, `deps` (+ `impact --format json`) for scripts and agent pipelines | | **PC-friendly indexing** | `tokenix index` runs at below-normal OS priority by default so long index runs never starve the machine (`--no-low-priority` opts out) | | **Interactive HTML/Mermaid graphs** | `tokenix impact --format html\|mermaid` exports vis.js / Mermaid flowcharts; `tokenix flow --format mermaid` traces call flow | +| **Repo graph overview** | `tokenix graph` ranks god nodes, bottlenecks, and blast-radius leaders across the whole symbol graph (`--format text\|dot\|json`, `--top N`) | | **Cycle detection** | `tokenix cycles` finds circular dependencies via Tarjan's strongly-connected components algorithm, dropping same-name (homonym) false positives and annotating each node with `path:line` | | **Token map** | `tokenix tokenmap` shows a directory tree with token counts per file/folder | | **Preference memory** | `tokenix memory add/list` stores global and project preferences in editable Markdown; context/explore include saved preferences | @@ -171,11 +178,11 @@ The embedding model (`nomic-embed-text-v1.5`, ~130 MB) is downloaded automatical | **Legacy VB6 + SQL sources** | `.bas`/`.cls`/`.ctl`/`.frm`/`.vbp` and `.sql`/`.fnc`/`.trg`/`.pkg`/`.prc`/`.tab`/`.vw` indexed with symbol-aware heuristic chunking (`Sub`/`Function`/`Property`, `CREATE` objects); UTF-16 SQL files decoded via BOM; binary files (e.g. `.frx`) skipped by a NUL sniff | | **Symbol-aware chunking** | AST Tree-sitter parsers for Rust, Python, TypeScript, JavaScript, Go, C/C++ | | **Multi-agent safe index** | PID-based index lock prevents concurrent reindex; embeddings are committed per batch, so a killed index run resumes from the last completed batch | -| **Smart file reader** | Outlines large files; supports `--symbol` and `--lines` reads | +| **Smart file reader** | Outlines large files; supports `--symbol` and `--lines` reads, plus `--mode full\|outline\|signatures\|diff\|density:X` (signatures-only, changed-hunks, or entropy-filtered reads) | | **Hook-based interception** | `PreToolUse` intercepts large reads and rewrites noisy Bash **and PowerShell** commands before execution; thresholds tunable via `[hook]` in `.tokenix.toml` | | **Structural output compression** | Fuzzy grouping, compact `git`/`cargo` filters, NDJSON/JSON compaction, and ANSI/Emoji stripping | | **Local project filters** | Drop `.toml` files in `.tokenix/filters/` for project-scoped compression rules — highest priority over user and bundled filters | -| **Output filters** | 378 TOML output filters embedded in the binary (each homologated against 784 golden cases) — auto-applied to Bash/PowerShell output for `uv`, `cargo`, `terraform`, `ansible`, `docker`, `kubectl`, `git`, `npm`, `pnpm`, `bun`, `deno`, `vite`, `pip`, `poetry`, `go`, `rust`, `helm`, `apt`, `journalctl`, `trivy`, `semgrep`, `bazel`, `ctest`, `tox`, `conda`, `pulumi`, `dnf`/`yum`, `pacman`, `apk`, `pip-audit`, `ng test`, `bru`, `ps`, and more | +| **Output filters** | 386 TOML output filters embedded in the binary (each homologated against 800 golden cases) — auto-applied to Bash/PowerShell output for `uv`, `cargo`, `terraform`, `ansible`, `docker`, `kubectl`, `git`, `npm`, `pnpm`, `bun`, `deno`, `vite`, `pip`, `poetry`, `go`, `rust`, `helm`, `apt`, `journalctl`, `trivy`, `semgrep`, `bazel`, `ctest`, `tox`, `conda`, `pulumi`, `dnf`/`yum`, `pacman`, `apk`, `pip-audit`, `ng test`, `bru`, `ps`, `cargo tree`, `npm ls`, `kubectl explain`, `lsof`, `ss`, `netstat`, `ip`, `systemctl list-*`, and more | | **Filter generation** | `tokenix filter generate` writes a TOML filter for a command; `tokenix filter record` captures real output for richer generation, with a per-command **token-economy preview** (raw→filtered tokens, % saved, compression bar) shown by `record stop`/`status` | | **GPU acceleration (opt-in)** | Build with `--features directml` (Windows) or `--features cuda` to run embeddings on GPU; GPU is used by default at runtime with automatic CPU fallback, or force CPU with `--only-cpu` | | **Environment diagnostics** | `tokenix doctor` reports the compiled backend, detected GPU, CUDA/cuDNN status, model cache, and daemon | @@ -184,6 +191,7 @@ The embedding model (`nomic-embed-text-v1.5`, ~130 MB) is downloaded automatical | **Graceful fallback** | Exits `0` on errors — your AI session is never broken | | **Token budget** | Results fit within a configurable token budget (default `1200`) | | **Savings analytics** | `tokenix gain` — token summary, savings split by source (semantic index vs command filters), and by-tool histogram; `--cost-estimate` adds a per-model cost table (10 reference models across Anthropic / OpenAI / Google) | +| **Spend analytics** | `tokenix usage` — absolute token spend and ≈USD cost read from agent transcripts, by `daily\|weekly\|monthly\|session\|model\|project\|blocks`; rolling 5-hour blocks with burn rate, month-end forecast, `--cost-mode auto\|calculate\|display`, `--statusline`, and `--json` | | **Slim MCP profile** | `tokenix mcp --profile slim` exposes 3 meta-tools instead of the full tool surface for hosts that support progressive discovery | | **MCP/prompt weight audit** | `tokenix prompt-audit --recommend --profile-impact` connects to configured MCP servers, tokenizes tool schemas, and shows full-vs-slim MCP savings | | **Session audit** | `tokenix session-audit --cache-hygiene` combines index freshness, hook history, MCP/tool weight, and prompt-cache stability risks | @@ -271,6 +279,9 @@ and supports `plan`, `debug`, `audit`, `security`, and `review` modes. Use tokenix read src/auth/middleware.rs # symbol outline tokenix read src/auth/middleware.rs --symbol validate_token # targeted tokenix read src/auth/middleware.rs --lines 45-80 # line range +tokenix read src/auth/middleware.rs --mode signatures # signatures only +tokenix read src/auth/middleware.rs --mode diff # outline + changed hunks +tokenix read src/auth/middleware.rs --mode density:40 # keep ~40% highest-entropy lines ``` ### 6. Symbol graph & maps @@ -285,6 +296,8 @@ tokenix impact update_user --format html --output update_user.html # vis.js gr tokenix deps src/indexer.rs # file-level import dependencies tokenix deps src/store.rs --reverse # who imports this file tokenix deps src/daemon.rs --transitive # follow the import chain +tokenix graph # repo-wide hotspots / blast radius +tokenix graph --format dot --top 20 -o graph.dot # Graphviz of the top subgraph tokenix tokenmap # token tree tokenix rebuild-graph # recompute relationships without re-embedding ``` @@ -302,6 +315,10 @@ tokenix callers run_hook --json tokenix gain # token summary + by-tool histogram tokenix gain --history # include per-call history tokenix gain --cost-estimate # add the per-model cost table +tokenix usage # absolute spend (daily) + ≈USD cost +tokenix usage model # spend by model · also: weekly|monthly|session|project|blocks +tokenix usage blocks # rolling 5-hour billing blocks + burn rate +tokenix usage --statusline # compact one-liner for a status bar tokenix session-audit # index + hook + MCP token-economy health ``` @@ -511,13 +528,14 @@ tokenix install-hook --tool all | `tokenix explore TEXT` | Graph-aware exploration: entry points, relationships, grouped source | | `tokenix query TEXT` | Semantic search over indexed chunks | | `tokenix grep PATTERN` | Exact regex/literal search over indexed content (no embedding) | -| `tokenix read FILE` | Smart reader — outline for large files, full for small | +| `tokenix read FILE` | Smart reader — outline for large files, full for small (`--symbol`, `--lines`, `--mode full\|outline\|signatures\|diff\|density:X`) | | `tokenix symbols QUERY` | Find indexed symbols by name or path (`--kind` filters by symbol type) | | `tokenix callers SYMBOL` | Show symbols that call/reference a symbol | | `tokenix callees SYMBOL` | Show symbols called/referenced by a symbol | | `tokenix deps FILE` | File-level import dependencies (`--reverse`, `--transitive`, `--json`) | | `tokenix impact SYMBOL` | Bidirectional impact graph (`--format html\|mermaid` for vis.js graph or Mermaid flowchart) | | `tokenix flow SYMBOL` | Forward call-flow trace from a symbol (`--depth`, `--format text\|mermaid`) | +| `tokenix graph` | Repo-wide symbol-graph overview — god nodes, bottlenecks, blast-radius leaders (`--format text\|dot\|json`, `--top N`, `--output`) | | `tokenix pack` | Budgeted repo pack for non-hook AI tools (`--mode/--profile`, `--changed`, `--token-map`) | | `tokenix memory add TEXT` | Save a preference (`--global` or `--project`) for future context | | `tokenix memory list` | List global and project preferences | @@ -528,7 +546,7 @@ tokenix install-hook --tool all | Command | Description | |---|---| -| `tokenix` (no args) | Open the [interactive dashboard](#-interactive-dashboard) — Stats · Filters · Gain · Doctor · Tokenmap · Secrets · Egress tabs; piped/non-TTY falls back to help | +| `tokenix` (no args) | Open the [interactive dashboard](#-interactive-dashboard) — Stats · Filters · Studio · Gain · Usage · Doctor · Tokenmap · Graph · Secrets · Egress tabs; piped/non-TTY falls back to help | | `tokenix filter` (no args) | Open the dashboard on the Filters tab; piped falls back to `filter list` | | `tokenix index [PATH]` | Index the repo at PATH (default `.`) | | `tokenix install-hook` | Install assistant hook/instructions (default `--tool all`) | @@ -539,6 +557,7 @@ tokenix install-hook --tool all | `tokenix stop` | Stop the background daemon | | `tokenix daemon status\|stop\|restart` | Inspect (pid, port, uptime, model, cache RAM) or control the daemon | | `tokenix gain` | Token savings analytics with a by-source split — measured Read savings vs command filters; semantic Grep is neutral usage (`--cost-estimate` adds a per-model cost table) | +| `tokenix usage` | Absolute token spend + ≈USD cost from agent transcripts (`daily\|weekly\|monthly\|session\|model\|project\|blocks`, `--since/--until`, `--all-projects`, `--cost-mode`, `--statusline`, `--json`) | | `tokenix stats` | Index statistics (files, chunks, tokens, age) | | `tokenix tokenmap` | Directory tree map with token counts, heaviest paths first, plus a top-10 files summary (`--format html` supported) | | `tokenix benchmark` | Reproducible token-savings and retrieval-quality benchmark — vanilla vs tokenix (`--json`) | @@ -665,7 +684,7 @@ tokenix reduces noisy shell output by rewriting matching `Bash` commands in `Pre 1. **Local project filters** — `.toml` files in `.tokenix/filters/` inside the repo. Scoped to the project, committed to version control. 2. **User filters** — `.toml` files in `~/.tokenix/filters/`. Apply to all projects, override bundled filters. -3. **Bundled filters** — 378 TOML output filters shipped inside the binary (each homologated against 784 embedded golden cases), covering `uv`, `cargo build`/`cargo run`/`cargo audit`, `git`, `gradle`, `terraform plan`, `make`, `npm`/`npm audit`, `pnpm`, `bun`, `deno`, `vite`, `node --test`, `poetry`, `docker`, `kubectl`/`kubectl top`, `helm`, `go`, `rust`, `python`, `dotnet`, `swift`, `apt`/`apt-get`, `journalctl`, `trivy`, `semgrep`, `bazel`, `ctest`, `tox`, `conda`/`mamba`, `pulumi up`/`preview`/`destroy`, `dnf`/`yum`, `pacman`, `apk`, `pip-audit`, `ng test` (Karma), `bru` (Bruno), `ps`, and more. Applied automatically — no setup needed. +3. **Bundled filters** — 386 TOML output filters shipped inside the binary (each homologated against 800 embedded golden cases), covering `uv`, `cargo build`/`cargo run`/`cargo audit`, `git`, `gradle`, `terraform plan`, `make`, `npm`/`npm audit`, `pnpm`, `bun`, `deno`, `vite`, `node --test`, `poetry`, `docker`, `kubectl`/`kubectl top`, `helm`, `go`, `rust`, `python`, `dotnet`, `swift`, `apt`/`apt-get`, `journalctl`, `trivy`, `semgrep`, `bazel`, `ctest`, `tox`, `conda`/`mamba`, `pulumi up`/`preview`/`destroy`, `dnf`/`yum`, `pacman`, `apk`, `pip-audit`, `ng test` (Karma), `bru` (Bruno), `ps`, and more. Applied automatically — no setup needed. ### Filter format @@ -733,7 +752,7 @@ src/ └── mcp_audit.rs Multi-agent MCP config discovery + live tools/list introspection (prompt/session audit) assets/ -└── filters/ 378 TOML output filters (+784 golden cases), embedded in the binary via rust-embed +└── filters/ 386 TOML output filters (+800 golden cases), embedded in the binary via rust-embed ``` ### GPU acceleration (opt-in) diff --git a/assets/filters/cargo-tree.toml b/assets/filters/cargo-tree.toml new file mode 100644 index 0000000..a9a8d12 --- /dev/null +++ b/assets/filters/cargo-tree.toml @@ -0,0 +1,29 @@ +[filters.cargo-tree] +description = "Compact `cargo tree` dependency trees — drop blank lines, cap deep output, truncate long lines." +match_command = "^cargo\\s+tree\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 80 +truncate_lines_at = 140 + +[[tests.cargo-tree]] +name = "keeps tree, strips blank lines" +input = """ +myapp v0.1.0 (/work/myapp) +├── serde v1.0.0 + +└── tokio v1.0.0 + └── bytes v1.0.0 +""" +expected = """myapp v0.1.0 (/work/myapp) +├── serde v1.0.0 +└── tokio v1.0.0 + └── bytes v1.0.0""" + +[[tests.cargo-tree]] +name = "dedup markers preserved" +input = "myapp v0.1.0\n├── serde v1.0.0 (*)\n└── serde_json v1.0.0" +expected = "myapp v0.1.0\n├── serde v1.0.0 (*)\n└── serde_json v1.0.0" diff --git a/assets/filters/ip.toml b/assets/filters/ip.toml new file mode 100644 index 0000000..7bddba6 --- /dev/null +++ b/assets/filters/ip.toml @@ -0,0 +1,29 @@ +[filters.ip] +description = "Compact `ip addr` / `ip route` / `ip link` output — drop blank lines, cap and truncate verbose interface dumps." +match_command = "^ip\\s+(a|addr|address|r|route|l|link|n|neigh)\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 60 +truncate_lines_at = 160 + +[[tests.ip]] +name = "keeps interface lines, strips blanks" +input = """ +1: lo: mtu 65536 state UNKNOWN + inet 127.0.0.1/8 scope host lo + +2: eth0: mtu 1500 state UP + inet 10.0.0.5/24 scope global eth0 +""" +expected = """1: lo: mtu 65536 state UNKNOWN + inet 127.0.0.1/8 scope host lo +2: eth0: mtu 1500 state UP + inet 10.0.0.5/24 scope global eth0""" + +[[tests.ip]] +name = "route table kept" +input = "default via 10.0.0.1 dev eth0\n10.0.0.0/24 dev eth0 proto kernel scope link" +expected = "default via 10.0.0.1 dev eth0\n10.0.0.0/24 dev eth0 proto kernel scope link" diff --git a/assets/filters/kubectl-explain.toml b/assets/filters/kubectl-explain.toml new file mode 100644 index 0000000..10dc2b4 --- /dev/null +++ b/assets/filters/kubectl-explain.toml @@ -0,0 +1,34 @@ +[filters.kubectl-explain] +description = "Compact `kubectl explain` schema docs — drop blank lines and cap the long DESCRIPTION/FIELDS prose." +match_command = "^kubectl\\s+explain\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 50 +truncate_lines_at = 160 + +[[tests.kubectl-explain]] +name = "keeps schema header and fields, strips blanks" +input = """ +KIND: Pod +VERSION: v1 + +RESOURCE: spec + +FIELDS: + containers <[]Object> -required- + nodeName +""" +expected = """KIND: Pod +VERSION: v1 +RESOURCE: spec +FIELDS: + containers <[]Object> -required- + nodeName """ + +[[tests.kubectl-explain]] +name = "single field kept" +input = "KIND: Deployment\nVERSION: apps/v1" +expected = "KIND: Deployment\nVERSION: apps/v1" diff --git a/assets/filters/lsof.toml b/assets/filters/lsof.toml new file mode 100644 index 0000000..ed9b5e4 --- /dev/null +++ b/assets/filters/lsof.toml @@ -0,0 +1,27 @@ +[filters.lsof] +description = "Compact `lsof` open-file listings — drop blank lines, cap and truncate the typically huge table." +match_command = "^lsof\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 80 +truncate_lines_at = 200 + +[[tests.lsof]] +name = "keeps table, strips blanks" +input = """ +COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME +nginx 1234 root 6u IPv4 12345 0t0 TCP *:http (LISTEN) + +redis 5678 redis 6u IPv4 67890 0t0 TCP localhost:6379 (LISTEN) +""" +expected = """COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME +nginx 1234 root 6u IPv4 12345 0t0 TCP *:http (LISTEN) +redis 5678 redis 6u IPv4 67890 0t0 TCP localhost:6379 (LISTEN)""" + +[[tests.lsof]] +name = "single match kept" +input = "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava 9012 app 120u IPv6 24680 0t0 TCP *:8080 (LISTEN)" +expected = "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\njava 9012 app 120u IPv6 24680 0t0 TCP *:8080 (LISTEN)" diff --git a/assets/filters/netstat.toml b/assets/filters/netstat.toml new file mode 100644 index 0000000..001f12b --- /dev/null +++ b/assets/filters/netstat.toml @@ -0,0 +1,27 @@ +[filters.netstat] +description = "Compact `netstat` output — drop blank lines, cap and truncate large connection/route tables." +match_command = "^netstat\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 80 +truncate_lines_at = 200 + +[[tests.netstat]] +name = "keeps connections, strips blanks" +input = """ +Proto Recv-Q Send-Q Local Address Foreign Address State +tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN + +tcp 0 0 10.0.0.5:443 10.0.0.9:51000 ESTABLISHED +""" +expected = """Proto Recv-Q Send-Q Local Address Foreign Address State +tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN +tcp 0 0 10.0.0.5:443 10.0.0.9:51000 ESTABLISHED""" + +[[tests.netstat]] +name = "header banner kept" +input = "Active Internet connections (servers and established)\nProto Recv-Q Send-Q Local Address Foreign Address State" +expected = "Active Internet connections (servers and established)\nProto Recv-Q Send-Q Local Address Foreign Address State" diff --git a/assets/filters/npm-ls.toml b/assets/filters/npm-ls.toml new file mode 100644 index 0000000..6e74b78 --- /dev/null +++ b/assets/filters/npm-ls.toml @@ -0,0 +1,29 @@ +[filters.npm-ls] +description = "Compact `npm ls` / `npm list` dependency trees — drop blank lines and npm warn/notice noise, keep the tree." +match_command = "^npm\\s+(ls|list)\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", + "^npm warn", + "^npm notice", +] +head_lines = 80 +truncate_lines_at = 140 + +[[tests.npm-ls]] +name = "tree kept, warn noise stripped" +input = """ +myapp@1.0.0 /work/myapp +├── express@4.18.2 +npm warn deprecated har-validator@5.1.5: this library is no longer supported +└── jest@29.7.0 +""" +expected = """myapp@1.0.0 /work/myapp +├── express@4.18.2 +└── jest@29.7.0""" + +[[tests.npm-ls]] +name = "errors are not stripped" +input = "myapp@1.0.0 /work/myapp\nnpm error code ELSPROBLEMS\nnpm error missing: lodash@^4.0.0" +expected = "myapp@1.0.0 /work/myapp\nnpm error code ELSPROBLEMS\nnpm error missing: lodash@^4.0.0" diff --git a/assets/filters/ss.toml b/assets/filters/ss.toml new file mode 100644 index 0000000..fb51c77 --- /dev/null +++ b/assets/filters/ss.toml @@ -0,0 +1,27 @@ +[filters.ss] +description = "Compact `ss` socket statistics — drop blank lines, cap and truncate large connection tables." +match_command = "^ss\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", +] +head_lines = 60 +truncate_lines_at = 200 + +[[tests.ss]] +name = "keeps listening sockets, strips blanks" +input = """ +Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port +tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* + +tcp LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* +""" +expected = """Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port +tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* +tcp LISTEN 0 511 127.0.0.1:6379 0.0.0.0:*""" + +[[tests.ss]] +name = "summary line kept" +input = "Total: 245\nTCP: 12 (estab 4, closed 2)" +expected = "Total: 245\nTCP: 12 (estab 4, closed 2)" diff --git a/assets/filters/systemctl-list.toml b/assets/filters/systemctl-list.toml new file mode 100644 index 0000000..84ff008 --- /dev/null +++ b/assets/filters/systemctl-list.toml @@ -0,0 +1,48 @@ +[filters.systemctl-list] +description = "Compact `systemctl list-units` / `list-unit-files` / `list-timers` — keep the table, drop the legend footer and count lines." +match_command = "^systemctl\\s+(list-units|list-unit-files|list-timers|list-sockets)\\b" +passthrough_when_emptied = true +strip_ansi = true +strip_lines_matching = [ + "^\\s*$", + "^LOAD\\s+=", + "^ACTIVE\\s+=", + "^SUB\\s+=", + "^Legend:", + "^\\d+ loaded units listed", + "^\\d+ unit files listed", + "^To show all installed unit files", +] +head_lines = 80 +truncate_lines_at = 160 + +[[tests.systemctl-list]] +name = "keeps unit table, strips legend footer" +input = """ +UNIT LOAD ACTIVE SUB DESCRIPTION +ssh.service loaded active running OpenBSD Secure Shell server +cron.service loaded active running Regular background program + +LOAD = Reflects whether the unit definition was properly loaded. +ACTIVE = The high-level unit activation state. +SUB = The low-level unit activation state. + +Legend: the unit list is sorted by name. +123 loaded units listed. +""" +expected = """UNIT LOAD ACTIVE SUB DESCRIPTION +ssh.service loaded active running OpenBSD Secure Shell server +cron.service loaded active running Regular background program""" + +[[tests.systemctl-list]] +name = "unit-files table kept, footer stripped" +input = """ +UNIT FILE STATE +ssh.service enabled +cron.service enabled + +245 unit files listed. +""" +expected = """UNIT FILE STATE +ssh.service enabled +cron.service enabled""" diff --git a/src/conversation_audit.rs b/src/conversation_audit.rs index 6a83825..870beae 100644 --- a/src/conversation_audit.rs +++ b/src/conversation_audit.rs @@ -10,7 +10,6 @@ use serde::Serialize; use serde_json::Value; use std::collections::HashMap; use std::path::{Path, PathBuf}; -use walkdir::WalkDir; use crate::chunker::count_tokens; use crate::filters; @@ -177,7 +176,7 @@ fn audit(agent: Agent, min_chars: usize, limit: usize) -> Result { if !root.exists() { continue; } - for path in transcript_files(&root, agent_key) { + for path in crate::transcripts::transcript_files(&root, agent_key) { scanned_files += 1; scan_jsonl_file(&path, agent_key, min_chars, &filters, &mut findings); scan_text_file(&path, agent_key, min_chars, &filters, &mut findings); @@ -203,37 +202,14 @@ fn audit(agent: Agent, min_chars: usize, limit: usize) -> Result { } fn roots(home: &Path, agent: Agent) -> Vec<(&'static str, PathBuf)> { - let mut out = Vec::new(); - if matches!(agent, Agent::All | Agent::Claude) { - out.push(("claude", home.join(".claude").join("projects"))); - } - if matches!(agent, Agent::All | Agent::Codex) { - out.push(("codex", home.join(".codex").join("sessions"))); - } - if matches!(agent, Agent::All | Agent::Copilot) { - out.push(("copilot", home.join(".copilot").join("session-state"))); - out.push(("copilot", home.join(".copilot").join("logs"))); - } - if matches!(agent, Agent::All | Agent::OpenAi) { - out.push(("openai", home.join(".openai"))); - } - out -} - -fn transcript_files(root: &Path, agent: &str) -> Vec { - WalkDir::new(root) + crate::transcripts::roots(home) .into_iter() - .filter_map(|e| e.ok()) - .filter(|e| e.file_type().is_file()) - .filter_map(|e| { - let p = e.into_path(); - let ext = p.extension().and_then(|x| x.to_str()).unwrap_or(""); - let name = p.file_name().and_then(|x| x.to_str()).unwrap_or(""); - let keep = matches!(ext, "jsonl" | "json" | "log" | "txt") - && !(agent == "copilot" - && p.components().any(|c| c.as_os_str() == "pkg") - && !name.contains("session")); - keep.then_some(p) + .filter(|(key, _)| match agent { + Agent::All => true, + Agent::Claude => *key == "claude", + Agent::Codex => *key == "codex", + Agent::Copilot => *key == "copilot", + Agent::OpenAi => *key == "openai", }) .collect() } diff --git a/src/gain.rs b/src/gain.rs index 582a96c..55b79b8 100644 --- a/src/gain.rs +++ b/src/gain.rs @@ -4,67 +4,136 @@ use std::path::Path; pub struct ModelPrice { pub name: &'static str, pub input_per_1m: f64, + /// Output (completion) token rate. Used by `tokenix usage` for absolute spend. + pub output_per_1m: f64, + /// Cached-input read rate (typically a fraction of input). + pub cache_read_per_1m: f64, + /// Cache-write (creation) rate (typically a small premium over input). + pub cache_write_per_1m: f64, pub reference: bool, } pub const PRICING_COLLECTED_AT: &str = "2026-06-11"; pub const MODELS: &[ModelPrice] = &[ - // Anthropic (source: platform.claude.com/docs/about-claude/pricing, collected 2026-06-11) + // Anthropic (source: platform.claude.com/docs/about-claude/pricing, collected 2026-06-11). + // Anthropic convention: cache read = 0.1x input, cache write (5m) = 1.25x input. ModelPrice { name: "claude-haiku-4-5", input_per_1m: 1.00, + output_per_1m: 5.00, + cache_read_per_1m: 0.10, + cache_write_per_1m: 1.25, reference: false, }, ModelPrice { name: "claude-sonnet-4-6", input_per_1m: 3.00, + output_per_1m: 15.00, + cache_read_per_1m: 0.30, + cache_write_per_1m: 3.75, reference: true, }, ModelPrice { name: "claude-opus-4-8", input_per_1m: 5.00, + output_per_1m: 25.00, + cache_read_per_1m: 0.50, + cache_write_per_1m: 6.25, reference: false, }, ModelPrice { name: "claude-fable-5", input_per_1m: 10.00, + output_per_1m: 50.00, + cache_read_per_1m: 1.00, + cache_write_per_1m: 12.50, reference: false, }, - // OpenAI (source: developers.openai.com/api/docs/pricing, collected 2026-06-11) + // OpenAI (source: developers.openai.com/api/docs/pricing, collected 2026-06-11). + // OpenAI has no separate cache-write; cached input is a discounted read (~0.25x). ModelPrice { name: "gpt-5.4-mini", input_per_1m: 0.75, + output_per_1m: 3.00, + cache_read_per_1m: 0.19, + cache_write_per_1m: 0.75, reference: false, }, ModelPrice { name: "gpt-5.4", input_per_1m: 2.50, + output_per_1m: 10.00, + cache_read_per_1m: 0.63, + cache_write_per_1m: 2.50, reference: false, }, ModelPrice { name: "gpt-5.5", input_per_1m: 5.00, + output_per_1m: 20.00, + cache_read_per_1m: 1.25, + cache_write_per_1m: 5.00, reference: false, }, - // Google (source: ai.google.dev/gemini-api/docs/pricing, collected 2026-06-11) + // Google (source: ai.google.dev/gemini-api/docs/pricing, collected 2026-06-11). ModelPrice { name: "gemini-3.1-flash-lite", input_per_1m: 0.25, + output_per_1m: 1.00, + cache_read_per_1m: 0.06, + cache_write_per_1m: 0.25, reference: false, }, ModelPrice { name: "gemini-3.5-flash", input_per_1m: 1.50, + output_per_1m: 6.00, + cache_read_per_1m: 0.38, + cache_write_per_1m: 1.50, reference: false, }, ModelPrice { name: "gemini-3.1-pro-preview", input_per_1m: 2.00, + output_per_1m: 8.00, + cache_read_per_1m: 0.50, + cache_write_per_1m: 2.00, reference: false, }, ]; +/// Look up pricing for a model id, matching by exact name then by prefix +/// (transcripts may carry suffixed ids like `claude-opus-4-8-20260101`). +pub fn price_for(model: &str) -> Option<&'static ModelPrice> { + MODELS + .iter() + .find(|m| m.name == model) + .or_else(|| MODELS.iter().find(|m| model.starts_with(m.name))) + .or_else(|| { + // Fall back to family match (e.g. "claude-opus" -> opus entry). + MODELS.iter().find(|m| { + let fam: String = m.name.split('-').take(2).collect::>().join("-"); + !fam.is_empty() && model.starts_with(&fam) + }) + }) +} + +/// Absolute USD cost of a single usage record, per token category. +pub fn usage_cost( + price: &ModelPrice, + input: u64, + output: u64, + cache_read: u64, + cache_write: u64, +) -> f64 { + (input as f64 * price.input_per_1m + + output as f64 * price.output_per_1m + + cache_read as f64 * price.cache_read_per_1m + + cache_write as f64 * price.cache_write_per_1m) + / 1_000_000.0 +} + pub struct CostRow { pub model: &'static str, pub reference: bool, @@ -441,6 +510,30 @@ mod tests { let _ = std::fs::remove_dir_all(&temp_dir); } + #[test] + fn price_for_matches_exact_and_suffixed_ids() { + assert_eq!( + price_for("claude-sonnet-4-6").map(|m| m.name), + Some("claude-sonnet-4-6") + ); + // Suffixed transcript ids resolve by prefix to the base model. + assert_eq!( + price_for("claude-opus-4-8-20260101").map(|m| m.name), + Some("claude-opus-4-8") + ); + assert!(price_for("totally-unknown-model").is_none()); + } + + #[test] + fn usage_cost_sums_all_token_categories() { + let p = price_for("claude-sonnet-4-6").unwrap(); + // 1M each of input/output/cache_read/cache_write at 3/15/0.30/3.75 per 1M. + let cost = usage_cost(p, 1_000_000, 1_000_000, 1_000_000, 1_000_000); + assert!((cost - (3.0 + 15.0 + 0.30 + 3.75)).abs() < 1e-9); + // Zero usage costs nothing. + assert_eq!(usage_cost(p, 0, 0, 0, 0), 0.0); + } + #[test] fn by_command_rollup_aggregates_same_base() { let temp_dir = create_test_temp_dir("rollup"); diff --git a/src/graph.rs b/src/graph.rs index 7c2e685..f3fb37f 100644 --- a/src/graph.rs +++ b/src/graph.rs @@ -491,6 +491,167 @@ pub fn format_relations(relations: &[GraphRelation], title: &str) -> String { out } +/// A repo-wide graph hotspot: a symbol ranked by its connectivity and the +/// number of symbols transitively affected if it changes (blast radius). +#[derive(Debug, Clone, serde::Serialize)] +pub struct Hotspot { + pub name: String, + pub path: String, + pub in_degree: usize, + pub out_degree: usize, + /// Transitive dependents — how many symbols are affected by a change here. + pub blast: usize, +} + +/// Rank the most-connected symbols across the whole symbol graph. Blast radius +/// (transitive dependents) is computed only for the strongest degree candidates +/// to keep this bounded on large graphs. +pub fn repo_hotspots(edges: &[store::GraphEdgeRow], top: usize) -> Vec { + let mut label: HashMap = HashMap::new(); + let mut indeg: HashMap = HashMap::new(); + let mut outdeg: HashMap = HashMap::new(); + // Reverse adjacency: callee -> callers, for blast-radius traversal. + let mut dependents: HashMap> = HashMap::new(); + + for (cid, cname, cpath, eid, ename, epath) in edges { + label + .entry(*cid) + .or_insert_with(|| (cname.clone(), cpath.clone())); + label + .entry(*eid) + .or_insert_with(|| (ename.clone(), epath.clone())); + *outdeg.entry(*cid).or_default() += 1; + *indeg.entry(*eid).or_default() += 1; + dependents.entry(*eid).or_default().push(*cid); + } + + // Rank by degree first; only the strongest candidates get a blast walk. + let mut by_degree: Vec = label.keys().copied().collect(); + by_degree.sort_by(|a, b| { + let da = indeg.get(a).unwrap_or(&0) + outdeg.get(a).unwrap_or(&0); + let db = indeg.get(b).unwrap_or(&0) + outdeg.get(b).unwrap_or(&0); + db.cmp(&da) + }); + + let candidate_cap = (top * 3).max(top); + by_degree + .into_iter() + .filter(|id| !is_trivial_symbol(&label[id].0)) + .take(candidate_cap) + .map(|id| { + let (name, path) = label[&id].clone(); + Hotspot { + name, + path, + in_degree: *indeg.get(&id).unwrap_or(&0), + out_degree: *outdeg.get(&id).unwrap_or(&0), + blast: transitive_dependents(id, &dependents), + } + }) + .collect() +} + +/// Drop graph-extraction noise (single-letter bindings, `_`, language keywords) +/// so the hotspot report surfaces meaningful symbols. +fn is_trivial_symbol(name: &str) -> bool { + name.len() <= 2 || name.chars().all(|c| c == '_') || KEYWORDS.contains(&name) +} + +/// BFS count of unique nodes reachable from `start` over the reverse-edge map. +fn transitive_dependents(start: i64, dependents: &HashMap>) -> usize { + let mut seen = HashSet::new(); + let mut frontier = vec![start]; + while let Some(node) = frontier.pop() { + if let Some(callers) = dependents.get(&node) { + for &c in callers { + if seen.insert(c) { + frontier.push(c); + } + } + } + } + seen.len() +} + +/// Render a compact repo-wide graph report: god nodes, bottlenecks, and +/// blast-radius leaders. Inspired by knowledge-graph overviews but built from +/// tokenix's own symbol graph. +pub fn format_repo_report(edges: &[store::GraphEdgeRow], top: usize) -> String { + if edges.is_empty() { + return "No symbol-graph edges found. Run `tokenix index` first.".to_string(); + } + let node_count = { + let mut s = HashSet::new(); + for (cid, _, _, eid, _, _) in edges { + s.insert(*cid); + s.insert(*eid); + } + s.len() + }; + let spots = repo_hotspots(edges, top); + + let mut out = format!( + "# Repo graph — {} symbols, {} edges\n", + node_count, + edges.len() + ); + + out.push_str("\n## God nodes (most connected)\n"); + let mut god = spots.clone(); + god.sort_by_key(|h| std::cmp::Reverse(h.in_degree + h.out_degree)); + for h in god.iter().take(top) { + out.push_str(&format!( + "- {} (↑{} ↓{}) {}\n", + h.name, h.in_degree, h.out_degree, h.path + )); + } + + out.push_str("\n## Bottlenecks (high fan-in, low fan-out)\n"); + let mut neck = spots.clone(); + neck.sort_by(|a, b| { + let sa = a.in_degree as i64 - a.out_degree as i64; + let sb = b.in_degree as i64 - b.out_degree as i64; + sb.cmp(&sa) + }); + for h in neck.iter().filter(|h| h.in_degree > h.out_degree).take(top) { + out.push_str(&format!( + "- {} (↑{} ↓{}) {}\n", + h.name, h.in_degree, h.out_degree, h.path + )); + } + + out.push_str("\n## Blast-radius leaders (most transitive dependents)\n"); + let mut blast = spots; + blast.sort_by_key(|h| std::cmp::Reverse(h.blast)); + for h in blast.iter().take(top) { + out.push_str(&format!( + "- {} → {} dependents {}\n", + h.name, h.blast, h.path + )); + } + + out +} + +/// Render the most-connected subgraph as Graphviz DOT. Only edges whose both +/// endpoints are among the top hotspots are emitted, keeping the diagram legible. +pub fn format_edges_dot(edges: &[store::GraphEdgeRow], top: usize) -> String { + let spots = repo_hotspots(edges, top); + let keep: HashSet = spots.iter().take(top).map(|h| h.name.clone()).collect(); + let mut out = String::from("digraph tokenix {\n rankdir=LR;\n node [shape=box];\n"); + let mut seen = HashSet::new(); + for (_, cname, _, _, ename, _) in edges { + if keep.contains(cname) && keep.contains(ename) { + let line = format!(" {:?} -> {:?};\n", cname, ename); + if seen.insert(line.clone()) { + out.push_str(&line); + } + } + } + out.push_str("}\n"); + out +} + /// Format graph relations as a Mermaid flowchart diagram. pub fn format_relations_mermaid(relations: &[GraphRelation], title: &str) -> String { if relations.is_empty() { @@ -1570,4 +1731,39 @@ mod tests { "homonym SCC should be dropped" ); } + + #[test] + fn repo_hotspots_ranks_by_degree_and_blast() { + // hub is called by three callers; chain a->b->hub gives hub 2 transitive + // dependents through b plus the direct callers. + let edges = vec![ + edge(1, "caller_one", "src/a.rs:1", 4, "hub", "src/hub.rs:1"), + edge(2, "caller_two", "src/b.rs:1", 4, "hub", "src/hub.rs:1"), + edge(3, "caller_three", "src/c.rs:1", 4, "hub", "src/hub.rs:1"), + ]; + let spots = repo_hotspots(&edges, 10); + let hub = spots.iter().find(|h| h.name == "hub").expect("hub present"); + assert_eq!(hub.in_degree, 3); + assert_eq!(hub.out_degree, 0); + assert_eq!(hub.blast, 3, "three transitive dependents"); + + // Trivial single-letter / keyword symbols are filtered out of ranking. + let noisy = vec![edge(1, "e", "src/a.rs:1", 2, "fn", "src/b.rs:1")]; + assert!(repo_hotspots(&noisy, 10).is_empty()); + } + + #[test] + fn format_edges_dot_emits_only_top_subgraph() { + let edges = vec![edge( + 1, + "alpha_fn", + "src/a.rs:1", + 2, + "beta_fn", + "src/b.rs:1", + )]; + let dot = format_edges_dot(&edges, 10); + assert!(dot.starts_with("digraph tokenix {")); + assert!(dot.contains("\"alpha_fn\" -> \"beta_fn\";")); + } } diff --git a/src/main.rs b/src/main.rs index 6f284a8..11b7200 100644 --- a/src/main.rs +++ b/src/main.rs @@ -21,8 +21,10 @@ mod query; mod recordings; mod secrets_scan; mod store; +mod transcripts; mod tui; mod ui; +mod usage; use anyhow::Result; use clap::{CommandFactory, FromArgMatches, Parser, Subcommand, ValueEnum}; @@ -336,6 +338,11 @@ enum Commands { symbol: Option, #[arg(short, long, help = "Line range e.g. 10-50")] lines: Option, + #[arg( + long, + help = "Read mode: full | outline | signatures | diff | density:X (e.g. density:40)" + )] + mode: Option, #[arg(long, help = "Emit machine-readable JSON instead of text")] json: bool, #[arg(short, long, default_value = ".")] @@ -424,6 +431,21 @@ enum Commands { #[arg(short, long, default_value = ".")] path: PathBuf, }, + /// Repo-wide symbol-graph overview: hotspots, bottlenecks, blast radius + Graph { + #[arg( + long, + help = "Output format: text | dot | json", + default_value = "text" + )] + format: String, + #[arg(long, default_value_t = 30, help = "How many top symbols to show")] + top: usize, + #[arg(short, long, help = "Write output to a file instead of stdout")] + output: Option, + #[arg(short, long, default_value = ".")] + path: PathBuf, + }, /// Detect circular dependencies in the symbol graph Cycles { #[arg(short, long, default_value = ".")] @@ -448,6 +470,32 @@ enum Commands { )] global: bool, }, + /// Show absolute token spend and USD cost from agent transcripts + Usage { + /// Breakdown dimension + #[arg(value_enum, default_value = "daily")] + group: usage::Group, + /// Only count records on/after this date (YYYY-MM-DD) + #[arg(long)] + since: Option, + /// Only count records on/before this date (YYYY-MM-DD) + #[arg(long)] + until: Option, + /// Aggregate across all projects (default: current repo only) + #[arg(long)] + all_projects: bool, + /// How to derive cost: auto | calculate | display + #[arg(long, value_enum, default_value = "auto")] + cost_mode: usage::CostMode, + /// Emit a compact one-line summary for a status bar hook + #[arg(long)] + statusline: bool, + /// Emit machine-readable JSON instead of a table + #[arg(long)] + json: bool, + #[arg(short, long, default_value = ".")] + path: PathBuf, + }, /// Pack focused repository context for AI tools that cannot call tokenix hooks Pack { #[arg(short, long, default_value = ".")] @@ -878,9 +926,17 @@ fn main() -> Result<()> { file, symbol, lines, + mode, json, path, - } => cmd_read(&file, symbol.as_deref(), lines.as_deref(), json, &path), + } => cmd_read( + &file, + symbol.as_deref(), + lines.as_deref(), + mode.as_deref(), + json, + &path, + ), Commands::Symbols { query, limit, @@ -922,6 +978,12 @@ fn main() -> Result<()> { json, path, } => cmd_deps(&file, reverse, transitive, json, &path), + Commands::Graph { + format, + top, + output, + path, + } => cmd_graph(&format, top, output.as_deref(), &path), Commands::Cycles { path } => cmd_cycles(&path), Commands::RebuildGraph { path } => cmd_rebuild_graph(&path), Commands::Gain { @@ -936,6 +998,25 @@ fn main() -> Result<()> { cmd_gain(&path, history, cost_estimate) } } + Commands::Usage { + group, + since, + until, + all_projects, + cost_mode, + statusline, + json, + path, + } => usage::run(usage::Options { + group, + since, + until, + all_projects, + cost_mode, + statusline, + json, + path, + }), Commands::Pack { path, profile, @@ -1743,6 +1824,28 @@ fn cmd_cycles(path: &Path) -> Result<()> { Ok(()) } +fn cmd_graph(format_str: &str, top: usize, output: Option<&str>, path: &Path) -> Result<()> { + if top == 0 { + anyhow::bail!("--top must be >= 1"); + } + let conn = open_existing_index(path)?; + let edges = store::load_all_graph_edges(&conn)?; + let body = if format_str.eq_ignore_ascii_case("json") { + serde_json::to_string_pretty(&graph::repo_hotspots(&edges, top))? + } else if format_str.eq_ignore_ascii_case("dot") { + graph::format_edges_dot(&edges, top) + } else { + graph::format_repo_report(&edges, top) + }; + if let Some(file) = output { + std::fs::write(file, &body)?; + println!("{} graph {} written to {}", "ok".green(), format_str, file); + } else { + println!("{body}"); + } + Ok(()) +} + fn cmd_rebuild_graph(path: &Path) -> Result<()> { let repo_root = find_repo_root(path); let conn = open_existing_index(path)?; @@ -1814,6 +1917,7 @@ fn cmd_read( file: &str, symbol: Option<&str>, lines_range: Option<&str>, + mode: Option<&str>, json: bool, path: &Path, ) -> Result<()> { @@ -1911,6 +2015,50 @@ fn cmd_read( return Ok(()); } + if let Some(mode) = mode { + let m = mode.to_lowercase(); + if m == "full" { + println!("{content}"); + } else if m == "outline" { + println!("{}", chunker::generate_outline(&content, &rel)); + } else if m == "signatures" { + let chunks = chunker::chunk_file(&rel, &content); + for c in &chunks { + let sig = c + .content + .lines() + .find(|l| !l.trim().is_empty()) + .unwrap_or("") + .trim_end(); + println!("L{}: {}", c.start_line, sig); + } + } else if m == "diff" { + println!("{}", chunker::generate_outline(&content, &rel)); + let out = std::process::Command::new("git") + .arg("-C") + .arg(&repo_root) + .args(["diff", "--", &rel]) + .output(); + match out { + Ok(o) if !o.stdout.is_empty() => { + println!("\n# changed hunks"); + println!("{}", String::from_utf8_lossy(&o.stdout)); + } + _ => println!("\n(no uncommitted changes)"), + } + } else if m == "density" || m.starts_with("density:") { + let frac = m + .strip_prefix("density:") + .and_then(|x| x.trim().parse::().ok()) + .map(|p| (p / 100.0).clamp(0.05, 1.0)) + .unwrap_or(0.40); + println!("{}", density_filter(&file_lines, frac)); + } else { + anyhow::bail!("unknown --mode: {mode} (use full|outline|signatures|diff|density:X)"); + } + return Ok(()); + } + if file_lines.len() >= 200 { println!("{}", chunker::generate_outline(&content, &rel)); println!("\nUse --symbol or --lines N-M to read specific parts."); @@ -1920,6 +2068,69 @@ fn cmd_read( Ok(()) } +/// Keep the highest-entropy lines until roughly `frac` of the file's tokens +/// remain, preserving original order and collapsing dropped runs into `…`. +/// Deterministic: ranks by Shannon byte-entropy, ties broken by line length. +fn density_filter(lines: &[&str], frac: f64) -> String { + let total_tokens: usize = lines.iter().map(|l| chunker::count_tokens(l)).sum(); + let budget = ((total_tokens as f64) * frac).ceil() as usize; + + let mut ranked: Vec = (0..lines.len()).collect(); + ranked.sort_by(|&a, &b| { + let ea = line_entropy(lines[a]); + let eb = line_entropy(lines[b]); + eb.partial_cmp(&ea) + .unwrap_or(std::cmp::Ordering::Equal) + .then(lines[b].len().cmp(&lines[a].len())) + .then(a.cmp(&b)) + }); + + let mut keep = vec![false; lines.len()]; + let mut used = 0usize; + for &i in &ranked { + if used >= budget { + break; + } + keep[i] = true; + used += chunker::count_tokens(lines[i]).max(1); + } + + let mut out = String::new(); + let mut gap = false; + for (i, line) in lines.iter().enumerate() { + if keep[i] { + out.push_str(line); + out.push('\n'); + gap = false; + } else if !gap { + out.push_str("…\n"); + gap = true; + } + } + out +} + +/// Shannon entropy (bits) of a line's byte distribution. +fn line_entropy(line: &str) -> f64 { + let bytes = line.trim().as_bytes(); + if bytes.is_empty() { + return 0.0; + } + let mut counts = [0u32; 256]; + for &b in bytes { + counts[b as usize] += 1; + } + let len = bytes.len() as f64; + counts + .iter() + .filter(|&&c| c > 0) + .map(|&c| { + let p = c as f64 / len; + -p * p.log2() + }) + .sum() +} + fn cmd_generate_ignores(path: &Path) -> Result<()> { let repo_root = find_repo_root(path); let gitignore = repo_root.join(".gitignore"); diff --git a/src/transcripts.rs b/src/transcripts.rs new file mode 100644 index 0000000..c4b5a66 --- /dev/null +++ b/src/transcripts.rs @@ -0,0 +1,40 @@ +//! Shared enumeration of local AI-agent transcript files. +//! +//! Several commands (`conversation-audit`, `usage`, secrets/egress scans) need +//! the same list of on-disk history files per agent. This is the single source +//! of truth so the directory layout lives in one place. + +use std::path::{Path, PathBuf}; +use walkdir::WalkDir; + +/// `(agent_key, root_dir)` pairs for every supported agent. Callers should skip +/// roots whose directory does not exist. +pub fn roots(home: &Path) -> Vec<(&'static str, PathBuf)> { + vec![ + ("claude", home.join(".claude").join("projects")), + ("codex", home.join(".codex").join("sessions")), + ("copilot", home.join(".copilot").join("session-state")), + ("copilot", home.join(".copilot").join("logs")), + ("openai", home.join(".openai")), + ] +} + +/// Walk a single agent root and return its transcript files. Copilot package +/// internals are skipped unless the filename looks like a session. +pub fn transcript_files(root: &Path, agent: &str) -> Vec { + WalkDir::new(root) + .into_iter() + .filter_map(|e| e.ok()) + .filter(|e| e.file_type().is_file()) + .filter_map(|e| { + let p = e.into_path(); + let ext = p.extension().and_then(|x| x.to_str()).unwrap_or(""); + let name = p.file_name().and_then(|x| x.to_str()).unwrap_or(""); + let keep = matches!(ext, "jsonl" | "json" | "log" | "txt") + && !(agent == "copilot" + && p.components().any(|c| c.as_os_str() == "pkg") + && !name.contains("session")); + keep.then_some(p) + }) + .collect() +} diff --git a/src/tui.rs b/src/tui.rs index 0479efc..f5b5cb6 100644 --- a/src/tui.rs +++ b/src/tui.rs @@ -30,20 +30,24 @@ enum Cmd { Filters, Studio, Gain, + Usage, Doctor, Tokenmap, + Graph, Secrets, Egress, } impl Cmd { - const ALL: [Cmd; 8] = [ + const ALL: [Cmd; 10] = [ Cmd::Stats, Cmd::Filters, Cmd::Studio, Cmd::Gain, + Cmd::Usage, Cmd::Doctor, Cmd::Tokenmap, + Cmd::Graph, Cmd::Secrets, Cmd::Egress, ]; @@ -58,8 +62,10 @@ impl Cmd { Cmd::Filters => "Filters", Cmd::Studio => "Studio", Cmd::Gain => "Gain", + Cmd::Usage => "Usage", Cmd::Doctor => "Doctor", Cmd::Tokenmap => "Tokenmap", + Cmd::Graph => "Graph", Cmd::Secrets => "Secrets", Cmd::Egress => "Egress", } @@ -217,6 +223,9 @@ struct Shell { /// Set to a base command to run `filter generate` in the foreground (mirrors /// `request_index`); consumed by the event loop after it drops the alt-screen. request_generate: Option, + // Usage tab (spend analytics, dynamic argv) --------------------------- + usage_group: usize, + usage_global: bool, // Report / install pages ---------------------------------------------- reports: HashMap, scroll: u16, @@ -259,6 +268,8 @@ pub fn run() -> Result<()> { stats_sel: 0, stats_confirm: false, stats_msg: None, + usage_group: 0, + usage_global: false, gain_cache: None, gain_cost: false, gain_global: false, @@ -404,6 +415,8 @@ impl Shell { Cmd::Filters => self.key_filters(key.code), Cmd::Studio => self.key_studio(key.code), Cmd::Gain => self.key_gain(key.code), + Cmd::Usage => self.key_usage(key.code), + Cmd::Graph => self.key_graph(key.code), Cmd::Secrets => self.key_secrets(key.code), Cmd::Egress => self.key_egress(key.code), _ => self.key_scroll(key.code), @@ -455,15 +468,71 @@ impl Shell { }; } - /// Lazily capture report output the first time its tab is shown. + /// Lazily capture report output the first time its tab is shown. Tabs with + /// state (Usage group/scope) build their argv dynamically. fn ensure_report(&mut self) { + let idx = self.cmd.index(); + if let Some(args) = self.dyn_argv() { + let refs: Vec<&str> = args.iter().map(|s| s.as_str()).collect(); + self.reports.entry(idx).or_insert_with(|| capture(&refs)); + return; + } let Some(argv) = self.cmd.argv() else { return; }; - let idx = self.cmd.index(); self.reports.entry(idx).or_insert_with(|| capture(argv)); } + /// Per-tab dynamic argv for report tabs whose output depends on UI state. + fn dyn_argv(&self) -> Option> { + match self.cmd { + Cmd::Usage => { + let group = + ["daily", "model", "blocks", "project", "session"][self.usage_group % 5]; + let mut v = vec!["usage".to_string(), group.to_string()]; + if self.usage_global { + v.push("--all-projects".to_string()); + } + Some(v) + } + Cmd::Graph => Some(vec![ + "graph".to_string(), + "--top".to_string(), + "30".to_string(), + ]), + _ => None, + } + } + + fn key_usage(&mut self, code: KeyCode) { + match code { + KeyCode::Char('s') => { + self.usage_group = (self.usage_group + 1) % 5; + self.reports.remove(&self.cmd.index()); + self.scroll = 0; + } + KeyCode::Char('a') => { + self.usage_global = !self.usage_global; + self.reports.remove(&self.cmd.index()); + self.scroll = 0; + } + KeyCode::Char('r') => { + self.reports.remove(&self.cmd.index()); + self.scroll = 0; + } + _ => self.key_scroll(code), + } + } + + fn key_graph(&mut self, code: KeyCode) { + if let KeyCode::Char('r') = code { + self.reports.remove(&self.cmd.index()); + self.scroll = 0; + } else { + self.key_scroll(code); + } + } + /// Compute gain data the first time the Gain tab is shown (or after a toggle /// that changed the project scope cleared the cache). fn ensure_gain(&mut self) { @@ -2676,6 +2745,17 @@ impl Shell { "←→: tab · ↑↓: scroll · c: cost · a: all-projects · r: refresh · q: quit" .to_string() } + Cmd::Usage => format!( + "←→: tab · ↑↓: scroll · s: group · a: {} · r: refresh · q: quit", + if self.usage_global { + "this repo" + } else { + "all projects" + } + ), + Cmd::Graph => { + "←→: tab · ↑↓: scroll · r: refresh · q: quit".to_string() + } Cmd::Secrets if self.secrets.is_none() => { "scanning… · ←→: tab · q: quit".to_string() } diff --git a/src/usage.rs b/src/usage.rs new file mode 100644 index 0000000..57025c2 --- /dev/null +++ b/src/usage.rs @@ -0,0 +1,618 @@ +//! Absolute token-spend analytics from local agent transcripts. +//! +//! This is the spend-side counterpart to `gain` (which measures savings): it +//! reads the real `usage` blocks AI agents write to their on-disk histories and +//! reports how many tokens were consumed and the estimated USD cost, broken down +//! by day/week/month/session/model/project, plus rolling 5-hour billing blocks. + +use anyhow::Result; +use chrono::{DateTime, Datelike, Duration, Local, NaiveDate, TimeZone, Timelike}; +use colored::Colorize; +use serde::Serialize; +use serde_json::Value; +use std::collections::HashSet; +use std::path::{Path, PathBuf}; + +use crate::gain::{price_for, usage_cost}; + +/// Length of a billing block (Anthropic's rolling 5-hour window). +const BLOCK_HOURS: i64 = 5; + +#[derive(clap::ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +#[clap(rename_all = "kebab-case")] +pub enum Group { + Daily, + Weekly, + Monthly, + Session, + Model, + Project, + Blocks, +} + +#[derive(clap::ValueEnum, Clone, Copy, Debug, PartialEq, Eq)] +#[clap(rename_all = "kebab-case")] +pub enum CostMode { + /// Use the cost logged by the agent when present, otherwise calculate it. + Auto, + /// Always calculate from token counts and the bundled pricing table. + Calculate, + /// Only show costs the agent itself logged. + Display, +} + +pub struct Options { + pub group: Group, + pub since: Option, + pub until: Option, + pub all_projects: bool, + pub cost_mode: CostMode, + pub statusline: bool, + pub json: bool, + pub path: PathBuf, +} + +#[derive(Clone, Debug)] +struct Record { + ts: DateTime, + model: String, + project: String, + session: String, + input: u64, + output: u64, + cache_read: u64, + cache_write: u64, + logged_cost: Option, +} + +impl Record { + fn tokens(&self) -> u64 { + self.input + self.output + self.cache_read + self.cache_write + } + + fn cost(&self, mode: CostMode) -> f64 { + let calc = price_for(&self.model) + .map(|p| { + usage_cost( + p, + self.input, + self.output, + self.cache_read, + self.cache_write, + ) + }) + .unwrap_or(0.0); + match mode { + CostMode::Calculate => calc, + CostMode::Display => self.logged_cost.unwrap_or(0.0), + CostMode::Auto => self.logged_cost.unwrap_or(calc), + } + } +} + +#[derive(Default, Serialize)] +struct Row { + key: String, + input: u64, + output: u64, + cache_read: u64, + cache_write: u64, + tokens: u64, + cost_usd: f64, +} + +pub fn run(opts: Options) -> Result<()> { + let records = collect_records(&opts)?; + + if opts.statusline { + print_statusline(&records, opts.cost_mode); + return Ok(()); + } + + if matches!(opts.group, Group::Blocks) { + return report_blocks(&records, &opts); + } + + let mut rows = aggregate(&records, &opts); + sort_rows(&mut rows, opts.group); + + let total = totals(&records, opts.cost_mode); + if opts.json { + let forecast = month_forecast(&records, opts.cost_mode); + let out = serde_json::json!({ + "group": format!("{:?}", opts.group).to_lowercase(), + "rows": rows, + "total": total, + "month_forecast_usd": forecast, + }); + println!("{}", serde_json::to_string_pretty(&out)?); + } else { + print_table(&rows, &total, &opts); + } + Ok(()) +} + +// --------------------------------------------------------------------------- +// Collection +// --------------------------------------------------------------------------- + +fn collect_records(opts: &Options) -> Result> { + let home = dirs::home_dir().unwrap_or_else(|| PathBuf::from(".")); + let since = opts.since.as_deref().and_then(parse_date); + let until = opts.until.as_deref().and_then(parse_date); + let scope = if opts.all_projects { + None + } else { + Some(current_project(&opts.path)) + }; + + let mut records = Vec::new(); + let mut seen: HashSet = HashSet::new(); + + for (agent_key, root) in crate::transcripts::roots(&home) { + if !root.exists() { + continue; + } + for path in crate::transcripts::transcript_files(&root, agent_key) { + let ext = path.extension().and_then(|x| x.to_str()).unwrap_or(""); + if !matches!(ext, "jsonl" | "json") { + continue; + } + parse_file(&path, &mut records, &mut seen); + } + } + + records.retain(|r| { + let d = r.ts.date_naive(); + since.map(|s| d >= s).unwrap_or(true) + && until.map(|u| d <= u).unwrap_or(true) + && scope.as_deref().map(|s| r.project == s).unwrap_or(true) + }); + records.sort_by_key(|r| r.ts); + Ok(records) +} + +fn parse_file(path: &Path, out: &mut Vec, seen: &mut HashSet) { + let Ok(raw) = std::fs::read_to_string(path) else { + return; + }; + let session_fallback = path + .file_stem() + .and_then(|s| s.to_str()) + .unwrap_or("?") + .to_string(); + for line in raw.lines() { + let Ok(v) = serde_json::from_str::(line) else { + continue; + }; + if let Some(rec) = record_from_value(&v, &session_fallback, seen) { + out.push(rec); + } + } +} + +fn record_from_value( + v: &Value, + session_fallback: &str, + seen: &mut HashSet, +) -> Option { + let message = v.get("message"); + let usage = message + .and_then(|m| m.get("usage")) + .or_else(|| v.get("usage"))?; + + let input = u64_at(usage, "input_tokens"); + let output = u64_at(usage, "output_tokens"); + let cache_read = u64_at(usage, "cache_read_input_tokens"); + let cache_write = u64_at(usage, "cache_creation_input_tokens"); + if input + output + cache_read + cache_write == 0 { + return None; + } + + // Dedup replayed lines by (message id, requestId) when both are present. + let msg_id = message + .and_then(|m| m.get("id")) + .and_then(|x| x.as_str()) + .unwrap_or(""); + let req_id = v.get("requestId").and_then(|x| x.as_str()).unwrap_or(""); + if !msg_id.is_empty() && !req_id.is_empty() { + let key = format!("{msg_id}|{req_id}"); + if !seen.insert(key) { + return None; + } + } + + let ts = v + .get("timestamp") + .and_then(|x| x.as_str()) + .and_then(parse_ts) + .unwrap_or_else(Local::now); + + let model = message + .and_then(|m| m.get("model")) + .or_else(|| v.get("model")) + .and_then(|x| x.as_str()) + .unwrap_or("unknown") + .to_string(); + + let project = v + .get("cwd") + .and_then(|x| x.as_str()) + .map(basename) + .unwrap_or_else(|| "?".to_string()); + + let session = v + .get("sessionId") + .and_then(|x| x.as_str()) + .unwrap_or(session_fallback) + .to_string(); + + let logged_cost = v + .get("costUSD") + .or_else(|| v.get("cost_usd")) + .and_then(|x| x.as_f64()); + + Some(Record { + ts, + model, + project, + session, + input, + output, + cache_read, + cache_write, + logged_cost, + }) +} + +fn u64_at(v: &Value, key: &str) -> u64 { + v.get(key).and_then(|x| x.as_u64()).unwrap_or(0) +} + +fn parse_ts(s: &str) -> Option> { + DateTime::parse_from_rfc3339(s) + .ok() + .map(|dt| dt.with_timezone(&Local)) +} + +fn parse_date(s: &str) -> Option { + NaiveDate::parse_from_str(s.trim(), "%Y-%m-%d").ok() +} + +fn basename(p: &str) -> String { + p.replace('\\', "/") + .trim_end_matches('/') + .rsplit('/') + .next() + .unwrap_or(p) + .to_string() +} + +fn current_project(path: &Path) -> String { + std::fs::canonicalize(path) + .ok() + .and_then(|p| p.file_name().map(|n| n.to_string_lossy().to_string())) + .unwrap_or_else(|| basename(&path.to_string_lossy())) +} + +// --------------------------------------------------------------------------- +// Aggregation +// --------------------------------------------------------------------------- + +fn group_key(r: &Record, group: Group) -> String { + match group { + Group::Daily => r.ts.format("%Y-%m-%d").to_string(), + Group::Weekly => { + let iso = r.ts.iso_week(); + format!("{}-W{:02}", iso.year(), iso.week()) + } + Group::Monthly => r.ts.format("%Y-%m").to_string(), + Group::Session => short(&r.session), + Group::Model => r.model.clone(), + Group::Project => r.project.clone(), + Group::Blocks => unreachable!(), + } +} + +fn aggregate(records: &[Record], opts: &Options) -> Vec { + use std::collections::HashMap; + let mut map: HashMap = HashMap::new(); + for r in records { + let key = group_key(r, opts.group); + let row = map.entry(key.clone()).or_insert_with(|| Row { + key, + ..Default::default() + }); + row.input += r.input; + row.output += r.output; + row.cache_read += r.cache_read; + row.cache_write += r.cache_write; + row.tokens += r.tokens(); + row.cost_usd += r.cost(opts.cost_mode); + } + map.into_values().collect() +} + +fn sort_rows(rows: &mut [Row], group: Group) { + match group { + // Chronological keys read best ascending. + Group::Daily | Group::Weekly | Group::Monthly => rows.sort_by(|a, b| a.key.cmp(&b.key)), + // Everything else: biggest spenders first. + _ => rows.sort_by(|a, b| b.cost_usd.total_cmp(&a.cost_usd)), + } +} + +fn totals(records: &[Record], mode: CostMode) -> Row { + let mut t = Row { + key: "TOTAL".to_string(), + ..Default::default() + }; + for r in records { + t.input += r.input; + t.output += r.output; + t.cache_read += r.cache_read; + t.cache_write += r.cache_write; + t.tokens += r.tokens(); + t.cost_usd += r.cost(mode); + } + t +} + +/// Linear month-end projection from spend so far this calendar month. +fn month_forecast(records: &[Record], mode: CostMode) -> f64 { + let now = Local::now(); + let month_cost: f64 = records + .iter() + .filter(|r| r.ts.year() == now.year() && r.ts.month() == now.month()) + .map(|r| r.cost(mode)) + .sum(); + let days_in_month = days_in_month(now.year(), now.month()); + let day = now.day().max(1); + if day == 0 { + return month_cost; + } + month_cost / day as f64 * days_in_month as f64 +} + +fn days_in_month(year: i32, month: u32) -> u32 { + let (ny, nm) = if month == 12 { + (year + 1, 1) + } else { + (year, month + 1) + }; + NaiveDate::from_ymd_opt(ny, nm, 1) + .and_then(|first_next| first_next.pred_opt()) + .map(|d| d.day()) + .unwrap_or(30) +} + +// --------------------------------------------------------------------------- +// Blocks (5-hour billing windows) +// --------------------------------------------------------------------------- + +#[derive(Serialize)] +struct Block { + start: String, + end: String, + tokens: u64, + cost_usd: f64, + active: bool, + burn_per_min: Option, + projected_cost_usd: Option, +} + +fn report_blocks(records: &[Record], opts: &Options) -> Result<()> { + let mut blocks: Vec = Vec::new(); + let now = Local::now(); + let mut iter = records.iter(); + if let Some(first) = iter.next() { + let mut start = floor_hour(first.ts); + let mut tok = 0u64; + let mut cost = 0.0; + let mut latest = first.ts; + let flush = |start: DateTime, + latest: DateTime, + tok: u64, + cost: f64, + now: DateTime| + -> Block { + let end = start + Duration::hours(BLOCK_HOURS); + let active = now < end && now >= start; + let (burn, proj) = if active { + let mins = (now - start).num_minutes().max(1) as f64; + let burn = tok as f64 / mins; + let remaining = (end - now).num_minutes().max(0) as f64; + let proj = cost + (cost / mins) * remaining; + (Some(burn), Some(proj)) + } else { + (None, None) + }; + let _ = latest; + Block { + start: start.format("%Y-%m-%d %H:%M").to_string(), + end: end.format("%H:%M").to_string(), + tokens: tok, + cost_usd: cost, + active, + burn_per_min: burn, + projected_cost_usd: proj, + } + }; + for r in std::iter::once(first).chain(iter) { + if r.ts >= start + Duration::hours(BLOCK_HOURS) { + blocks.push(flush(start, latest, tok, cost, now)); + start = floor_hour(r.ts); + tok = 0; + cost = 0.0; + } + tok += r.tokens(); + cost += r.cost(opts.cost_mode); + latest = r.ts; + } + blocks.push(flush(start, latest, tok, cost, now)); + } + + if opts.json { + println!("{}", serde_json::to_string_pretty(&blocks)?); + return Ok(()); + } + + println!("{}", "Usage — 5-hour blocks".bold()); + if blocks.is_empty() { + println!(" {}", "no usage records found".dimmed()); + return Ok(()); + } + for b in &blocks { + let marker = if b.active { + " ● active".green().to_string() + } else { + String::new() + }; + println!( + " {}→{} {:>10} {:>9}{}", + b.start, + b.end, + fmt_tokens(b.tokens), + fmt_cost(b.cost_usd), + marker + ); + if b.active { + if let (Some(burn), Some(proj)) = (b.burn_per_min, b.projected_cost_usd) { + println!( + " 🔥 {:.0} tok/min · projected {}", + burn, + fmt_cost(proj) + ); + } + } + } + Ok(()) +} + +fn floor_hour(ts: DateTime) -> DateTime { + Local + .with_ymd_and_hms(ts.year(), ts.month(), ts.day(), ts.hour(), 0, 0) + .single() + .unwrap_or(ts) +} + +// --------------------------------------------------------------------------- +// Rendering +// --------------------------------------------------------------------------- + +fn print_table(rows: &[Row], total: &Row, opts: &Options) { + let title = format!("Usage — {:?}", opts.group).to_lowercase(); + println!("{}", title.bold()); + if rows.is_empty() { + println!(" {}", "no usage records found".dimmed()); + return; + } + println!( + " {:<22} {:>10} {:>10} {:>10} {:>10} {:>11}", + "key".dimmed(), + "input".dimmed(), + "output".dimmed(), + "cache".dimmed(), + "tokens".dimmed(), + "cost".dimmed() + ); + for r in rows { + println!( + " {:<22} {:>10} {:>10} {:>10} {:>10} {:>11}", + short(&r.key), + fmt_tokens(r.input), + fmt_tokens(r.output), + fmt_tokens(r.cache_read + r.cache_write), + fmt_tokens(r.tokens), + fmt_cost(r.cost_usd) + ); + } + println!( + " {:<22} {:>10} {:>10} {:>10} {:>10} {:>11}", + "TOTAL".bold(), + fmt_tokens(total.input), + fmt_tokens(total.output), + fmt_tokens(total.cache_read + total.cache_write), + fmt_tokens(total.tokens), + fmt_cost(total.cost_usd).bold() + ); + if matches!(opts.group, Group::Daily | Group::Monthly | Group::Weekly) { + // Forecast needs the unfiltered-by-group records; recompute cheaply here. + let forecast = total_month_forecast(opts); + if let Some(f) = forecast { + println!( + " {} {}", + "month-end forecast:".dimmed(), + fmt_cost(f).yellow() + ); + } + } +} + +fn total_month_forecast(opts: &Options) -> Option { + // Recollect is cheap relative to I/O already done; reuse collection. + let records = collect_records(opts).ok()?; + Some(month_forecast(&records, opts.cost_mode)) +} + +fn print_statusline(records: &[Record], mode: CostMode) { + let now = Local::now(); + let today = now.date_naive(); + let mut cost = 0.0; + let mut tokens = 0u64; + for r in records.iter().filter(|r| r.ts.date_naive() == today) { + cost += r.cost(mode); + tokens += r.tokens(); + } + // Active-block burn rate. + let block_start = records + .iter() + .rev() + .find(|r| now - r.ts < Duration::hours(BLOCK_HOURS)) + .map(|_| floor_hour(now - Duration::hours(BLOCK_HOURS))); + let burn = block_start.map(|_| { + let win_start = now - Duration::hours(BLOCK_HOURS); + let tok: u64 = records + .iter() + .filter(|r| r.ts >= win_start) + .map(|r| r.tokens()) + .sum(); + tok as f64 / (BLOCK_HOURS * 60) as f64 + }); + let mut parts = vec![ + format!("{} today", fmt_cost(cost)), + format!("{} tok", fmt_tokens(tokens)), + ]; + if let Some(b) = burn { + parts.push(format!("🔥{:.0}/min", b)); + } + println!("{}", parts.join(" · ")); +} + +fn short(s: &str) -> String { + if s.len() > 20 { + format!("{}…", &s[..19]) + } else { + s.to_string() + } +} + +fn fmt_tokens(n: u64) -> String { + if n >= 1_000_000 { + format!("{:.1}M", n as f64 / 1_000_000.0) + } else if n >= 1_000 { + format!("{:.1}K", n as f64 / 1_000.0) + } else { + n.to_string() + } +} + +fn fmt_cost(c: f64) -> String { + if c >= 100.0 { + format!("${:.0}", c) + } else if c >= 1.0 { + format!("${:.2}", c) + } else { + format!("${:.4}", c) + } +}
Gain tab
Gain — tokens saved with a reduction bar, split by source and by command/tool. c adds the ≈USD cost table · a all-projects · r refresh.
Filters tab
Filters — browse all 378 bundled filters by tool with a live input → output preview and a per-filter X → Y tokens · % saved gauge.		Usage — absolute token spend and ≈USD cost read from agent transcripts (the spend-side counterpart to Gain). s cycles the breakdown (daily · model · 5-hour blocks · project · session), a toggles this-repo vs all-projects, r refreshes. The active 5-hour block shows burn rate and a projected cost.
Graph — repo-wide symbol-graph overview: god nodes (most connected), bottlenecks (high fan-in / low fan-out), and blast-radius leaders (most transitive dependents). r refreshes.
Filters tab
Filters — browse all 386 bundled filters by tool with a live input → output preview and a per-filter X → Y tokens · % saved gauge.		 Secrets tab
Secrets — credentials leaked across agent transcripts, grouped by rule and attributed to repo + branch. Starts scoped to the current repo; g toggles all repos. v reveal · c copy · x redact.