diff --git a/skills/hyenand-retrofit/evals/README.md b/skills/hyenand-retrofit/evals/README.md index 65022ac1..fede3207 100644 --- a/skills/hyenand-retrofit/evals/README.md +++ b/skills/hyenand-retrofit/evals/README.md @@ -1,16 +1,17 @@ # Skill evals — future CI integration -This directory holds eval cases for the `hyenand-retrofit` skill. As of writing -they function as a **spec** for the skill's intended behavior, not a runnable -test suite — there is no harness in this repo that executes them. This README -is the design for wiring that up later. +This directory holds eval cases for the `hyenand-retrofit` skill. The **correctness** side (the grep assertions in `evals.json`) is still a **spec** — there is no harness that checks pass/fail yet, and the bulk of this README is the design for wiring that up later. The **token-efficiency** side, however, is built and measured: `bench_tokens.py` runs every eval through a real agent and records how much work each retrofit took in tokens, turns, and wall-clock. See [Token-efficiency benchmark](#token-efficiency-benchmark) for the latest numbers. ## What's here -| File | Purpose | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `evals.json` | Five eval cases (native pure swap, native hybrid, foreign 2D ViT, foreign 3D feature-map U-Net, foreign 1D causal LM). Each case has a prompt, the input files the agent should see, an `expected_output` description, and a list of grep-based assertions against the file the agent writes. | -| `inputs/*.py` | Standalone test fixtures the agent retrofits. Each runs end-to-end (`python ` produces a forward-pass shape) so the harness can sanity-check the inputs themselves before evaluating the agent's output. | +| File | Purpose | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `evals.json` | Five eval cases (native pure swap, native hybrid, foreign 2D ViT, foreign 3D feature-map U-Net, foreign 1D causal LM). Each case has a prompt, the input files the agent should see, an `expected_output` description, and a list of grep-based assertions against the file the agent writes. | +| `inputs/*.py` | Standalone test fixtures the agent retrofits. Each runs end-to-end (`python ` produces a forward-pass shape) so the harness can sanity-check the inputs themselves before evaluating the agent's output. | +| `bench_tokens.py` | Token/turn/latency benchmark. Runs each eval through a fresh `claude -p` agent (skill text injected inline, `acceptEdits` so it actually writes the retrofit), parses the JSON usage envelope, and appends one row per run to `bench_results.jsonl`. Restores the working tree after each run. | +| `plot_tokens.py` | Renders `bench_tokens.png` (stacked token breakdown + agent turns per eval) from `bench_results.jsonl`. | +| `bench_results.jsonl` | Raw per-run measurements — one JSON line per run (tokens, turns, duration, model). | +| `bench_tokens.png` | The rendered token-efficiency figure embedded below. | The evals together cover the four-axis grid from `SKILL.md`: @@ -20,8 +21,64 @@ The evals together cover the four-axis grid from `SKILL.md`: | `foreign-3d-feature-map` | 3 | False | feature_map | 0 | | `foreign-1d-causal-lm` | 1 | True | tokens | 0 | -`native-pure-swap` and `native-hybrid` cover the native path where the host -already uses nvSubquadratic builders. +`native-pure-swap` and `native-hybrid` cover the native path where the host already uses nvSubquadratic builders. + +### What each eval retrofits + +| eval | host model | what gets swapped | path & key challenge | +| ------------------------ | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `native-pure-swap` | nvSubquadratic ViT-5 attention config | `build_attention_net` → `build_hyena_net` (pure HyenaND) | native — edit a config, no adapter needed | +| `native-hybrid` | nvSubquadratic ViT-5 attention config | → `build_hybrid_net` with an `HHHA` layer pattern (3 Hyena + 1 attention, ×3) | native — edit a config | +| `foreign-timm-vit` | standalone tiny ViT using `nn.MultiheadAttention`, 64×64 images, 65 tokens | the block's `.attn` → HyenaND (`data_dim=2`) | foreign — adapter reshapes `[B,65,C]`→`[B,8,8,C]`, peels the CLS token, returns `(out, None)` | +| `foreign-3d-feature-map` | standalone 3D U-Net with a `SpatialAttention3D` bottleneck (`F.scaled_dot_product_attention`) | the bottleneck → HyenaND (`data_dim=3`) | foreign — channel-first `[B,C,D,H,W]` ↔ channel-last, no CLS to peel | +| `foreign-1d-causal-lm` | standalone causal char-LM using causal `nn.MultiheadAttention`, sequence length 256 | the block's `.attn` → causal HyenaND (`data_dim=1`) | foreign — `is_causal`, RoPE, `exp_decay` mask, `omega_0=100`, adapter swallows `attn_mask` | + +## Token-efficiency benchmark + +`bench_tokens.py` measures what it takes to invoke this skill: for each eval it spawns a fresh `claude -p` agent with `SKILL.md` injected inline, lets it carry out the retrofit (`--permission-mode acceptEdits`, so it really writes the sibling file), parses the JSON usage envelope Claude returns, and appends a row to `bench_results.jsonl`. `plot_tokens.py` renders the figure below. The working tree is restored after every run, so the benchmark leaves no diff behind. + +**`claude-opus-4-8`, 3 runs/eval, 15 runs** (each cell is the per-run mean): + +| eval | turns | output tok | billed tok | wall (s) | +| ------------------------ | ----: | ---------: | ---------: | -------: | +| `native-pure-swap` | 5.7 | 2,144 | 164k | 31 | +| `native-hybrid` | 9.0 | 2,810 | 287k | 53 | +| `foreign-timm-vit` | 5.0 | 6,751 | 168k | 74 | +| `foreign-3d-feature-map` | 5.3 | 5,792 | 183k | 66 | +| `foreign-1d-causal-lm` | 13.7 | 11,705 | 535k | 355 | + +![Token efficiency per retrofit eval](bench_tokens.png) + +### Reading the table + +**Turns** is the number of model↔tool round-trips the agent took to finish one retrofit — each turn is one model response, usually a `Read`/`Write`/edit cycle. It's the cleanest single proxy for how much work a retrofit was: every extra turn re-reads the cached context, which is what drives **billed tok** up. **Output tok** is what the agent actually generated (the retrofit code plus its reasoning); it's only ~2–3 % of billed tokens, because each cold `claude -p` process re-reads the large system-prompt cache on every turn. Dollar figures are deliberately omitted — see the caveats. + +### Why the evals differ — and why `foreign-1d-causal-lm` is the heaviest + +Turn counts aren't uniform because the evals demand different amounts of work: + +- **Native swaps** (`native-pure-swap`, `native-hybrid`) only edit a config — read one or two files, write a short builder shim. Fewest turns, least output. +- **Foreign retrofits** must read the host model, design and write an adapter module, then reconcile it with the library's API — roughly 3× the output of a native swap. +- **`foreign-1d-causal-lm` is the outlier** (13.7 turns, ~12k output tokens — the most of any eval, ~1.7× the next-highest and ~2.7× the native swaps). It has the most constraints to satisfy at once: `is_causal=True` on `CKConvND` (and *not* `fft_padding='causal'`, which the library rejects), RoPE, an `exp_decay` mask, `omega_0=100`, plus an adapter that swallows `attn_mask`/`need_weights`. The agent frequently trips the `fft_padding='causal'` footgun, hits the validation error, and retries — so it iterates the most and generates the most code. + +A separate 3× rerun of this eval confirmed the instability rather than smoothing it out: **5, 10, and 25 turns** — one run finished cleanly in 5, another ballooned to 25 turns and ~1.56M billed tokens. The skill *can* land this retrofit quickly; it just doesn't reliably steer the agent there. That makes the causal path the highest-value target for tightening `SKILL.md` (e.g. calling out the `fft_padding='causal'` rejection up front). + +### Reproduce + +```bash +python skills/hyenand-retrofit/evals/bench_tokens.py --runs 3 # default model: claude-opus-4-8 +python skills/hyenand-retrofit/evals/plot_tokens.py +``` + +Pin a different model to track drift across releases with `--model claude-sonnet-4-6`. Use `--eval-id 5` to focus one case, or `--dry-run` to preview prompts without calling the API. Re-running appends to `bench_results.jsonl`; delete it first for a clean dataset. + +### Caveats (so the numbers aren't over-read) + +- **No dollar figures.** The CLI's `total_cost_usd` proved non-monotonic with token usage on multi-turn runs (a 629k-token run reported a higher cost than a 1.56M-token one) — cache-write churn across turns isn't captured cleanly — so we report turns and tokens, which are internally consistent. +- **Overhead dominates the billed total.** Each `claude -p` is a cold process that re-reads the full system-prompt cache every turn, so billed tokens are mostly fixed overhead, *not* skill-specific work. Compare **turns** and **output tok** across runs/models/skill revisions, not the billed total. +- **Skill injected inline.** `SKILL.md` (~16 KB) is prepended to each prompt to simulate the skill being loaded, because it lives under `skills/` and isn't auto-discovered by the CLI. +- **`acceptEdits` mode.** File writes auto-accept (the deliverable); Bash is auto-denied, so the agent writes the `__main__` smoke-test block but never runs it. Runtime verification is out of scope — this measures code-generation work only. +- **Cost, not correctness.** This harness does not check the grep assertions in `evals.json` (still a spec — below). ## Why CI for these diff --git a/skills/hyenand-retrofit/evals/bench_results.jsonl b/skills/hyenand-retrofit/evals/bench_results.jsonl new file mode 100644 index 00000000..c298cb5f --- /dev/null +++ b/skills/hyenand-retrofit/evals/bench_results.jsonl @@ -0,0 +1,15 @@ +{"eval_id": 1, "eval_name": "native-pure-swap", "run": 1, "model": "claude-opus-4-8", "wall_sec": 29.1, "ts": 1780694126, "input_tokens": 1652, "output_tokens": 2096, "cache_creation_tokens": 21206, "cache_read_tokens": 114296, "billed_input_tokens": 137154, "billed_total_tokens": 139250, "cost_usd": 0.256294, "duration_ms": 28392, "num_turns": 5, "permission_denials": 0, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 1, "eval_name": "native-pure-swap", "run": 2, "model": "claude-opus-4-8", "wall_sec": 32.9, "ts": 1780694159, "input_tokens": 1654, "output_tokens": 2127, "cache_creation_tokens": 21323, "cache_read_tokens": 151091, "billed_input_tokens": 174068, "billed_total_tokens": 176195, "cost_usd": 0.276183, "duration_ms": 32061, "num_turns": 6, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 1, "eval_name": "native-pure-swap", "run": 3, "model": "claude-opus-4-8", "wall_sec": 31.4, "ts": 1780694190, "input_tokens": 1654, "output_tokens": 2209, "cache_creation_tokens": 1922, "cache_read_tokens": 170598, "billed_input_tokens": 174174, "billed_total_tokens": 176383, "cost_usd": 0.166725, "duration_ms": 30632, "num_turns": 6, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 2, "eval_name": "native-hybrid", "run": 1, "model": "claude-opus-4-8", "wall_sec": 46.4, "ts": 1780694237, "input_tokens": 1658, "output_tokens": 2725, "cache_creation_tokens": 21286, "cache_read_tokens": 220566, "billed_input_tokens": 243510, "billed_total_tokens": 246235, "cost_usd": 0.338142, "duration_ms": 45646, "num_turns": 8, "permission_denials": 4, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 2, "eval_name": "native-hybrid", "run": 2, "model": "claude-opus-4-8", "wall_sec": 72.0, "ts": 1780694309, "input_tokens": 1784, "output_tokens": 3052, "cache_creation_tokens": 4197, "cache_read_tokens": 316055, "billed_input_tokens": 322036, "billed_total_tokens": 325088, "cost_usd": 0.32163, "duration_ms": 71277, "num_turns": 10, "permission_denials": 12, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 2, "eval_name": "native-hybrid", "run": 3, "model": "claude-opus-4-8", "wall_sec": 39.5, "ts": 1780694348, "input_tokens": 138, "output_tokens": 2654, "cache_creation_tokens": 11716, "cache_read_tokens": 276258, "billed_input_tokens": 288112, "billed_total_tokens": 290766, "cost_usd": 0.284368, "duration_ms": 38700, "num_turns": 9, "permission_denials": 3, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 1, "model": "claude-opus-4-8", "wall_sec": 73.3, "ts": 1780694422, "input_tokens": 1654, "output_tokens": 6732, "cache_creation_tokens": 55399, "cache_read_tokens": 107587, "billed_input_tokens": 164640, "billed_total_tokens": 171372, "cost_usd": 0.582579, "duration_ms": 72511, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 2, "model": "claude-opus-4-8", "wall_sec": 82.7, "ts": 1780694504, "input_tokens": 1656, "output_tokens": 7262, "cache_creation_tokens": 6552, "cache_read_tokens": 184163, "billed_input_tokens": 192371, "billed_total_tokens": 199633, "cost_usd": 0.328834, "duration_ms": 81891, "num_turns": 6, "permission_denials": 3, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 3, "model": "claude-opus-4-8", "wall_sec": 65.5, "ts": 1780694570, "input_tokens": 8, "output_tokens": 6260, "cache_creation_tokens": 5355, "cache_read_tokens": 122555, "billed_input_tokens": 127918, "billed_total_tokens": 134178, "cost_usd": 0.257258, "duration_ms": 64819, "num_turns": 4, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 1, "model": "claude-opus-4-8", "wall_sec": 76.4, "ts": 1780694646, "input_tokens": 1656, "output_tokens": 6567, "cache_creation_tokens": 22798, "cache_read_tokens": 184442, "billed_input_tokens": 208896, "billed_total_tokens": 215463, "cost_usd": 0.413177, "duration_ms": 75734, "num_turns": 6, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 2, "model": "claude-opus-4-8", "wall_sec": 59.0, "ts": 1780694706, "input_tokens": 1654, "output_tokens": 5431, "cache_creation_tokens": 4590, "cache_read_tokens": 154656, "billed_input_tokens": 160900, "billed_total_tokens": 166331, "cost_usd": 0.256059, "duration_ms": 58326, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 3, "model": "claude-opus-4-8", "wall_sec": 61.5, "ts": 1780694767, "input_tokens": 10, "output_tokens": 5378, "cache_creation_tokens": 4864, "cache_read_tokens": 156728, "billed_input_tokens": 161602, "billed_total_tokens": 166980, "cost_usd": 0.249263, "duration_ms": 60723, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 1, "model": "claude-opus-4-8", "wall_sec": 294.8, "ts": 1780695062, "input_tokens": 1914, "output_tokens": 11772, "cache_creation_tokens": 39646, "cache_read_tokens": 477955, "billed_input_tokens": 519515, "billed_total_tokens": 531287, "cost_usd": 1.244425, "duration_ms": 294059, "num_turns": 15, "permission_denials": 26, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 2, "model": "claude-opus-4-8", "wall_sec": 558.3, "ts": 1780695620, "input_tokens": 1918, "output_tokens": 12852, "cache_creation_tokens": 29890, "cache_read_tokens": 584403, "billed_input_tokens": 616211, "billed_total_tokens": 629063, "cost_usd": 5.170056, "duration_ms": 557564, "num_turns": 15, "permission_denials": 29, "primary_model": "claude-opus-4-8", "is_error": false} +{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 3, "model": "claude-opus-4-8", "wall_sec": 211.9, "ts": 1780695832, "input_tokens": 1788, "output_tokens": 10492, "cache_creation_tokens": 20896, "cache_read_tokens": 410433, "billed_input_tokens": 433117, "billed_total_tokens": 443609, "cost_usd": 0.838811, "duration_ms": 211110, "num_turns": 11, "permission_denials": 10, "primary_model": "claude-opus-4-8", "is_error": false} diff --git a/skills/hyenand-retrofit/evals/bench_tokens.png b/skills/hyenand-retrofit/evals/bench_tokens.png new file mode 100644 index 00000000..67cd98ed Binary files /dev/null and b/skills/hyenand-retrofit/evals/bench_tokens.png differ diff --git a/skills/hyenand-retrofit/evals/bench_tokens.py b/skills/hyenand-retrofit/evals/bench_tokens.py new file mode 100644 index 00000000..218b34ce --- /dev/null +++ b/skills/hyenand-retrofit/evals/bench_tokens.py @@ -0,0 +1,321 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Benchmark LLM token usage for the hyenand-retrofit skill. + +For each eval case in evals.json, this spawns a fresh, non-interactive Claude +agent (`claude -p`) with the skill text injected inline, lets it actually carry +out the retrofit (read inputs, write the sibling output file), and records the +token / cost / latency figures Claude reports in its JSON result envelope. + +Because every `claude -p` invocation is a cold process, it pays the full +system-prompt cache-creation cost once (no warm cache between runs). That fixed +overhead is part of what "invoking the skill" costs, so we report it explicitly +(cache_creation_tokens) alongside the task-driven input/output tokens. + +Usage: + python bench_tokens.py [--runs N] [--eval-id ID] [--model MODEL] [--dry-run] + + --runs N Repetitions per eval case (default: 3) + --eval-id ID Run only the eval with this integer id (default: all) + --model MODEL Model to pin for reproducibility (default: claude-opus-4-8) + --dry-run Print the prompts and the exact CLI invocation; call nothing + +Each run appends one JSON line to bench_results.jsonl with these fields: + eval_id, eval_name, run, model, input_tokens, output_tokens, + cache_creation_tokens, cache_read_tokens, billed_input_tokens, + billed_total_tokens, cost_usd, duration_ms, num_turns, + permission_denials, ts +""" + +import argparse +import json +import pathlib +import subprocess +import sys +import time + + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[3] +EVALS_FILE = pathlib.Path(__file__).parent / "evals.json" +RESULTS_FILE = pathlib.Path(__file__).parent / "bench_results.jsonl" +SKILL_FILE = pathlib.Path(__file__).resolve().parents[1] / "SKILL.md" + +DEFAULT_MODEL = "claude-opus-4-8" + +# Directories an eval is allowed to write into. Cleanup is scoped to these so we +# never touch the bench script, its results, or unrelated working-tree changes. +WRITE_DIRS = [ + "examples/vit5_imagenet/", + "skills/hyenand-retrofit/evals/inputs/", +] + + +def load_evals(): + """Load eval cases from evals.json.""" + return json.loads(EVALS_FILE.read_text())["evals"] + + +def build_prompt(eval_case: dict) -> str: + """Inject the full skill text + file hints + the eval prompt. + + Injecting SKILL.md inline reproduces the "skill is loaded" condition and + folds the skill's fixed context cost into the measured input tokens. + """ + skill_text = SKILL_FILE.read_text() if SKILL_FILE.exists() else "" + preamble = f"\n{skill_text}\n\n\n" if skill_text else "" + file_hints = "" + if eval_case.get("files"): + file_hints = ( + "Relevant files to read first (paths relative to repo root):\n" + + "\n".join(f" - {f}" for f in eval_case["files"]) + + "\n\n" + ) + return preamble + file_hints + eval_case["prompt"] + + +def porcelain() -> dict[str, str]: + """Map of path -> status code from `git status --porcelain`, scoped to WRITE_DIRS.""" + out = subprocess.run(["git", "status", "--porcelain"], cwd=REPO_ROOT, capture_output=True, text=True).stdout + status = {} + for line in out.splitlines(): + if not line.strip(): + continue + code, path = line[:2], line[3:].strip() + # Rename entries look like "old -> new"; keep the new path. + if " -> " in path: + path = path.split(" -> ", 1)[1] + if any(path.startswith(d) for d in WRITE_DIRS): + status[path] = code + return status + + +def restore_tree(baseline: dict[str, str]): + """Undo whatever the agent did inside WRITE_DIRS, leaving everything else alone. + + - Files newly untracked (not in baseline) are deleted. + - Tracked files the agent modified are restored with `git checkout --`. + """ + after = porcelain() + for path, code in after.items(): + if path in baseline and baseline[path] == code: + continue # unchanged from before this run + full = REPO_ROOT / path + if code.startswith("??"): + if full.exists(): + full.unlink() + else: + # Modified / added tracked file -> restore committed version. + subprocess.run(["git", "checkout", "--", path], cwd=REPO_ROOT, capture_output=True, text=True) + + +def parse_result(stdout: str) -> dict: + """Parse the `--output-format json` envelope (a single JSON object).""" + obj = None + try: + obj = json.loads(stdout) + except json.JSONDecodeError: + # Fallback: scan for the last line that parses and has type==result. + for line in stdout.splitlines(): + line = line.strip() + if not line: + continue + try: + cand = json.loads(line) + except json.JSONDecodeError: + continue + if cand.get("type") == "result": + obj = cand + if obj is None: + return {} + + usage = obj.get("usage", {}) + model_usage = obj.get("modelUsage", {}) or {} + # Primary model = the one that produced the most output tokens. + primary_model = "" + if model_usage: + primary_model = max(model_usage, key=lambda m: model_usage[m].get("outputTokens", 0)) + + inp = usage.get("input_tokens", 0) + out = usage.get("output_tokens", 0) + cc = usage.get("cache_creation_input_tokens", 0) + cr = usage.get("cache_read_input_tokens", 0) + return { + "input_tokens": inp, + "output_tokens": out, + "cache_creation_tokens": cc, + "cache_read_tokens": cr, + "billed_input_tokens": inp + cc + cr, + "billed_total_tokens": inp + cc + cr + out, + "cost_usd": round(obj.get("total_cost_usd", 0.0), 6), + "duration_ms": obj.get("duration_ms", 0), + "num_turns": obj.get("num_turns", 0), + "permission_denials": len(obj.get("permission_denials", []) or []), + "primary_model": primary_model, + "is_error": obj.get("is_error", False), + } + + +def run_one(eval_case: dict, run_index: int, model: str, dry_run: bool) -> dict: + """Run one eval case via `claude -p` and return a result record.""" + prompt = build_prompt(eval_case) + cmd = ["claude", "-p", "--output-format", "json", "--permission-mode", "acceptEdits", "--model", model] + + if dry_run: + print(f"\n--- DRY RUN eval {eval_case['id']} ({eval_case['name']}) run {run_index} ---") + print("CMD:", " ".join(cmd), " (prompt via stdin)") + print("PROMPT (first 400 chars of task portion):") + print(" ", eval_case["prompt"][:400], "...") + return {} + + baseline = porcelain() + t0 = time.monotonic() + try: + proc = subprocess.run(cmd, cwd=REPO_ROOT, input=prompt, capture_output=True, text=True, timeout=900) + except subprocess.TimeoutExpired: + restore_tree(baseline) + return { + "eval_id": eval_case["id"], + "eval_name": eval_case["name"], + "run": run_index, + "model": model, + "error": "timeout", + "ts": int(time.time()), + } + wall_sec = round(time.monotonic() - t0, 1) + + if proc.returncode != 0: + restore_tree(baseline) + return { + "eval_id": eval_case["id"], + "eval_name": eval_case["name"], + "run": run_index, + "model": model, + "error": f"exit {proc.returncode}: {proc.stderr[:160]}", + "ts": int(time.time()), + } + + parsed = parse_result(proc.stdout) + restore_tree(baseline) + + record = { + "eval_id": eval_case["id"], + "eval_name": eval_case["name"], + "run": run_index, + "model": model, + "wall_sec": wall_sec, + "ts": int(time.time()), + **parsed, + } + return record + + +def fmt_int(x): + """Format a number with thousands separators, or return str(x).""" + return f"{x:,}" if isinstance(x, (int, float)) else str(x) + + +def print_table(records: list[dict]): + """Print per-run rows and per-eval averages for successful benchmark records.""" + rows = [r for r in records if "error" not in r] + errs = [r for r in records if "error" in r] + if rows: + hdr = ( + f"{'eval':<24}{'run':>4} {'in':>6} {'cache_cr':>8} {'cache_rd':>8} " + f"{'out':>6} {'billed':>8} {'cost$':>7} {'turns':>5} {'sec':>6}" + ) + print("\n" + hdr) + print("-" * len(hdr)) + for r in rows: + print( + f"{r['eval_name']:<24}{r['run']:>4} " + f"{r['input_tokens']:>6,} {r['cache_creation_tokens']:>8,} " + f"{r['cache_read_tokens']:>8,} {r['output_tokens']:>6,} " + f"{r['billed_total_tokens']:>8,} {r['cost_usd']:>7.3f} " + f"{r['num_turns']:>5} {r.get('wall_sec', 0):>6.1f}" + ) + + # Per-eval averages + by = {} + for r in rows: + by.setdefault(r["eval_name"], []).append(r) + print( + f"\n{'AVG per eval':<24}{'n':>4} {'in':>6} {'cache_cr':>8} {'cache_rd':>8} " + f"{'out':>6} {'billed':>8} {'cost$':>7} {'turns':>5} {'sec':>6}" + ) + print("-" * len(hdr)) + for name, rs in by.items(): + n = len(rs) + + def avg(k): + return sum(r[k] for r in rs) / n + + print( + f"{name:<24}{n:>4} {round(avg('input_tokens')):>6,} " + f"{round(avg('cache_creation_tokens')):>8,} {round(avg('cache_read_tokens')):>8,} " + f"{round(avg('output_tokens')):>6,} {round(avg('billed_total_tokens')):>8,} " + f"{avg('cost_usd'):>7.3f} {round(avg('num_turns')):>5} {avg('wall_sec'):>6.1f}" + ) + total_cost = sum(r["cost_usd"] for r in rows) + print(f"\nTotal cost across {len(rows)} runs: ${total_cost:.3f} (model: {rows[0]['model']})") + for e in errs: + print(f"[ERROR] {e['eval_name']} run {e['run']}: {e['error']}") + + +def main(): + """CLI entry point for the token benchmark.""" + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--runs", type=int, default=3, metavar="N") + p.add_argument("--eval-id", type=int, default=None, metavar="ID") + p.add_argument("--model", default=DEFAULT_MODEL) + p.add_argument("--dry-run", action="store_true") + args = p.parse_args() + + evals = load_evals() + if args.eval_id is not None: + evals = [e for e in evals if e["id"] == args.eval_id] + if not evals: + sys.exit(f"No eval with id={args.eval_id}") + + collected = [] + for ec in evals: + print(f"\n=== eval {ec['id']}: {ec['name']} ({args.runs} run(s), model={args.model}) ===") + for i in range(1, args.runs + 1): + print(f" run {i}/{args.runs} ...", end=" ", flush=True) + rec = run_one(ec, i, args.model, args.dry_run) + if args.dry_run: + continue + collected.append(rec) + with open(RESULTS_FILE, "a") as f: + f.write(json.dumps(rec) + "\n") + if "error" in rec: + print(f"ERROR: {rec['error']}") + else: + print( + f"in={rec['input_tokens']:,} cache_cr={rec['cache_creation_tokens']:,} " + f"out={rec['output_tokens']:,} billed={rec['billed_total_tokens']:,} " + f"${rec['cost_usd']:.3f} turns={rec['num_turns']} {rec['wall_sec']:.0f}s" + ) + + if not args.dry_run: + print_table(collected) + print( + f"\nAppended {len([r for r in collected if 'error' not in r])} rows " + f"to {RESULTS_FILE.relative_to(REPO_ROOT)}" + ) + + +if __name__ == "__main__": + main() diff --git a/skills/hyenand-retrofit/evals/plot_tokens.py b/skills/hyenand-retrofit/evals/plot_tokens.py new file mode 100644 index 00000000..16228bd8 --- /dev/null +++ b/skills/hyenand-retrofit/evals/plot_tokens.py @@ -0,0 +1,156 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Plot LLM token usage / cost for the hyenand-retrofit skill. + +Reads bench_results.jsonl (produced by bench_tokens.py) and renders a two-panel +figure to bench_tokens.png: + + Panel A — stacked mean token breakdown per eval (cache_read, cache_creation, + input, output). The stack height is the billed total; the labels + make clear how much is fixed skill/system overhead (cache) vs. + task-driven generation (output). + Panel B — mean agent turns per eval with std-dev error bars across runs. + Turns track how much back-and-forth the retrofit took and are the + cleanest proxy for work (the CLI's dollar figure is noisy for + multi-turn runs, so it is deliberately not plotted). + +Usage: + python plot_tokens.py [--results bench_results.jsonl] [--out bench_tokens.png] +""" + +import argparse +import json +import pathlib +from collections import defaultdict + +import matplotlib + + +matplotlib.use("Agg") +import matplotlib.pyplot as plt + + +HERE = pathlib.Path(__file__).parent +COMPONENTS = [ + ("cache_read_tokens", "cache read", "#bdd7e7"), + ("cache_creation_tokens", "cache creation", "#6baed6"), + ("input_tokens", "input", "#2171b5"), + ("output_tokens", "output (generated)", "#e6550d"), +] + + +def mean(xs): + """Arithmetic mean; returns 0.0 for an empty sequence.""" + return sum(xs) / len(xs) if xs else 0.0 + + +def std(xs): + """Sample standard deviation; returns 0.0 for fewer than two values.""" + if len(xs) < 2: + return 0.0 + m = mean(xs) + return (sum((x - m) ** 2 for x in xs) / (len(xs) - 1)) ** 0.5 + + +def load(results_path: pathlib.Path): + """Load successful benchmark rows from a JSONL results file.""" + rows = [] + for line in results_path.read_text().splitlines(): + line = line.strip() + if not line: + continue + r = json.loads(line) + if "error" not in r: + rows.append(r) + if not rows: + raise SystemExit(f"No successful runs found in {results_path}") + return rows + + +def aggregate(rows): + """Group rows by (eval_id, eval_name), preserving eval_id order.""" + by = defaultdict(list) + # Preserve eval_id ordering for a stable x-axis. + for r in rows: + by[(r["eval_id"], r["eval_name"])].append(r) + keys = sorted(by, key=lambda k: k[0]) + return keys, by + + +def main(): + """CLI entry point for plotting benchmark token usage.""" + ap = argparse.ArgumentParser() + ap.add_argument("--results", default=str(HERE / "bench_results.jsonl")) + ap.add_argument("--out", default=str(HERE / "bench_tokens.png")) + args = ap.parse_args() + + rows = load(pathlib.Path(args.results)) + keys, by = aggregate(rows) + names = [k[1] for k in keys] + model = rows[0].get("model", "?") + n_runs = max(len(by[k]) for k in keys) + + fig, (axA, axB) = plt.subplots(1, 2, figsize=(13, 5.5)) + + # ---- Panel A: stacked token breakdown ---- + x = range(len(keys)) + bottoms = [0.0] * len(keys) + for field, label, color in COMPONENTS: + vals = [mean([r[field] for r in by[k]]) for k in keys] + axA.bar(x, vals, bottom=bottoms, label=label, color=color, width=0.62) + bottoms = [b + v for b, v in zip(bottoms, vals)] + # Annotate billed total on top of each stack. + for xi, total in zip(x, bottoms): + axA.text(xi, total, f"{total / 1000:.0f}k", ha="center", va="bottom", fontsize=9, fontweight="bold") + axA.set_xticks(list(x)) + axA.set_xticklabels(names, rotation=20, ha="right", fontsize=9) + axA.set_ylabel("tokens (mean of runs)") + axA.set_title("Token breakdown per retrofit eval") + axA.set_ylim(top=max(bottoms) * 1.15) # headroom so top labels clear the legend + axA.legend(fontsize=8, loc="best") # auto-place away from the tallest bar + axA.grid(axis="y", alpha=0.3) + + # ---- Panel B: agent turns per eval with error bars ---- + turns = [mean([r["num_turns"] for r in by[k]]) for k in keys] + turns_err = [std([r["num_turns"] for r in by[k]]) for k in keys] + bars = axB.bar(x, turns, yerr=turns_err, capsize=4, color="#31a354", width=0.62) + for bar, t in zip(bars, turns): + axB.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height(), + f"{t:.1f}", + ha="center", + va="bottom", + fontsize=9, + fontweight="bold", + ) + axB.set_xticks(list(x)) + axB.set_xticklabels(names, rotation=20, ha="right", fontsize=9) + axB.set_ylabel("agent turns (mean ± std)") + axB.set_title("Agent turns per retrofit eval") + axB.grid(axis="y", alpha=0.3) + + fig.suptitle( + f"hyenand-retrofit skill — token efficiency (model: {model}, {n_runs} runs/eval, {len(rows)} runs)", + fontsize=12, + ) + fig.tight_layout(rect=[0, 0, 1, 0.96]) + fig.savefig(args.out, dpi=130) + print(f"wrote {args.out}") + + +if __name__ == "__main__": + main()