Skip to content

skill's token efficiency benchmark #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
farhadrgh wants to merge 1 commit into
base: main
Choose a base branch
from
Open

skill's token efficiency benchmark #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 67 additions & 10 deletions skills/hyenand-retrofit/evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
# Skill evals — future CI integration

This directory holds eval cases for the `hyenand-retrofit` skill. As of writing
they function as a **spec** for the skill's intended behavior, not a runnable
test suite — there is no harness in this repo that executes them. This README
is the design for wiring that up later.
This directory holds eval cases for the `hyenand-retrofit` skill. The **correctness** side (the grep assertions in `evals.json`) is still a **spec** — there is no harness that checks pass/fail yet, and the bulk of this README is the design for wiring that up later. The **token-efficiency** side, however, is built and measured: `bench_tokens.py` runs every eval through a real agent and records how much work each retrofit took in tokens, turns, and wall-clock. See [Token-efficiency benchmark](#token-efficiency-benchmark) for the latest numbers.

## What's here

| File | Purpose |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evals.json` | Five eval cases (native pure swap, native hybrid, foreign 2D ViT, foreign 3D feature-map U-Net, foreign 1D causal LM). Each case has a prompt, the input files the agent should see, an `expected_output` description, and a list of grep-based assertions against the file the agent writes. |
| `inputs/*.py` | Standalone test fixtures the agent retrofits. Each runs end-to-end (`python <file>` produces a forward-pass shape) so the harness can sanity-check the inputs themselves before evaluating the agent's output. |
| File | Purpose |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evals.json` | Five eval cases (native pure swap, native hybrid, foreign 2D ViT, foreign 3D feature-map U-Net, foreign 1D causal LM). Each case has a prompt, the input files the agent should see, an `expected_output` description, and a list of grep-based assertions against the file the agent writes. |
| `inputs/*.py` | Standalone test fixtures the agent retrofits. Each runs end-to-end (`python <file>` produces a forward-pass shape) so the harness can sanity-check the inputs themselves before evaluating the agent's output. |
| `bench_tokens.py` | Token/turn/latency benchmark. Runs each eval through a fresh `claude -p` agent (skill text injected inline, `acceptEdits` so it actually writes the retrofit), parses the JSON usage envelope, and appends one row per run to `bench_results.jsonl`. Restores the working tree after each run. |
| `plot_tokens.py` | Renders `bench_tokens.png` (stacked token breakdown + agent turns per eval) from `bench_results.jsonl`. |
| `bench_results.jsonl` | Raw per-run measurements — one JSON line per run (tokens, turns, duration, model). |
| `bench_tokens.png` | The rendered token-efficiency figure embedded below. |

The evals together cover the four-axis grid from `SKILL.md`:

Expand All @@ -20,8 +21,64 @@ The evals together cover the four-axis grid from `SKILL.md`:
| `foreign-3d-feature-map` | 3 | False | feature_map | 0 |
| `foreign-1d-causal-lm` | 1 | True | tokens | 0 |

`native-pure-swap` and `native-hybrid` cover the native path where the host
already uses nvSubquadratic builders.
`native-pure-swap` and `native-hybrid` cover the native path where the host already uses nvSubquadratic builders.

### What each eval retrofits

| eval | host model | what gets swapped | path & key challenge |
| ------------------------ | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| `native-pure-swap` | nvSubquadratic ViT-5 attention config | `build_attention_net` → `build_hyena_net` (pure HyenaND) | native — edit a config, no adapter needed |
| `native-hybrid` | nvSubquadratic ViT-5 attention config | → `build_hybrid_net` with an `HHHA` layer pattern (3 Hyena + 1 attention, ×3) | native — edit a config |
| `foreign-timm-vit` | standalone tiny ViT using `nn.MultiheadAttention`, 64×64 images, 65 tokens | the block's `.attn` → HyenaND (`data_dim=2`) | foreign — adapter reshapes `[B,65,C]`→`[B,8,8,C]`, peels the CLS token, returns `(out, None)` |
| `foreign-3d-feature-map` | standalone 3D U-Net with a `SpatialAttention3D` bottleneck (`F.scaled_dot_product_attention`) | the bottleneck → HyenaND (`data_dim=3`) | foreign — channel-first `[B,C,D,H,W]` ↔ channel-last, no CLS to peel |
| `foreign-1d-causal-lm` | standalone causal char-LM using causal `nn.MultiheadAttention`, sequence length 256 | the block's `.attn` → causal HyenaND (`data_dim=1`) | foreign — `is_causal`, RoPE, `exp_decay` mask, `omega_0=100`, adapter swallows `attn_mask` |

## Token-efficiency benchmark

`bench_tokens.py` measures what it takes to invoke this skill: for each eval it spawns a fresh `claude -p` agent with `SKILL.md` injected inline, lets it carry out the retrofit (`--permission-mode acceptEdits`, so it really writes the sibling file), parses the JSON usage envelope Claude returns, and appends a row to `bench_results.jsonl`. `plot_tokens.py` renders the figure below. The working tree is restored after every run, so the benchmark leaves no diff behind.

**`claude-opus-4-8`, 3 runs/eval, 15 runs** (each cell is the per-run mean):

| eval | turns | output tok | billed tok | wall (s) |
| ------------------------ | ----: | ---------: | ---------: | -------: |
| `native-pure-swap` | 5.7 | 2,144 | 164k | 31 |
| `native-hybrid` | 9.0 | 2,810 | 287k | 53 |
| `foreign-timm-vit` | 5.0 | 6,751 | 168k | 74 |
| `foreign-3d-feature-map` | 5.3 | 5,792 | 183k | 66 |
| `foreign-1d-causal-lm` | 13.7 | 11,705 | 535k | 355 |

![Token efficiency per retrofit eval](bench_tokens.png)

### Reading the table

**Turns** is the number of model↔tool round-trips the agent took to finish one retrofit — each turn is one model response, usually a `Read`/`Write`/edit cycle. It's the cleanest single proxy for how much work a retrofit was: every extra turn re-reads the cached context, which is what drives **billed tok** up. **Output tok** is what the agent actually generated (the retrofit code plus its reasoning); it's only ~2–3 % of billed tokens, because each cold `claude -p` process re-reads the large system-prompt cache on every turn. Dollar figures are deliberately omitted — see the caveats.

### Why the evals differ — and why `foreign-1d-causal-lm` is the heaviest

Turn counts aren't uniform because the evals demand different amounts of work:

- **Native swaps** (`native-pure-swap`, `native-hybrid`) only edit a config — read one or two files, write a short builder shim. Fewest turns, least output.
- **Foreign retrofits** must read the host model, design and write an adapter module, then reconcile it with the library's API — roughly 3× the output of a native swap.
- **`foreign-1d-causal-lm` is the outlier** (13.7 turns, ~12k output tokens — the most of any eval, ~1.7× the next-highest and ~2.7× the native swaps). It has the most constraints to satisfy at once: `is_causal=True` on `CKConvND` (and *not* `fft_padding='causal'`, which the library rejects), RoPE, an `exp_decay` mask, `omega_0=100`, plus an adapter that swallows `attn_mask`/`need_weights`. The agent frequently trips the `fft_padding='causal'` footgun, hits the validation error, and retries — so it iterates the most and generates the most code.

A separate 3× rerun of this eval confirmed the instability rather than smoothing it out: **5, 10, and 25 turns** — one run finished cleanly in 5, another ballooned to 25 turns and ~1.56M billed tokens. The skill *can* land this retrofit quickly; it just doesn't reliably steer the agent there. That makes the causal path the highest-value target for tightening `SKILL.md` (e.g. calling out the `fft_padding='causal'` rejection up front).

### Reproduce

```bash
python skills/hyenand-retrofit/evals/bench_tokens.py --runs 3 # default model: claude-opus-4-8
python skills/hyenand-retrofit/evals/plot_tokens.py
```

Pin a different model to track drift across releases with `--model claude-sonnet-4-6`. Use `--eval-id 5` to focus one case, or `--dry-run` to preview prompts without calling the API. Re-running appends to `bench_results.jsonl`; delete it first for a clean dataset.

### Caveats (so the numbers aren't over-read)

- **No dollar figures.** The CLI's `total_cost_usd` proved non-monotonic with token usage on multi-turn runs (a 629k-token run reported a higher cost than a 1.56M-token one) — cache-write churn across turns isn't captured cleanly — so we report turns and tokens, which are internally consistent.
- **Overhead dominates the billed total.** Each `claude -p` is a cold process that re-reads the full system-prompt cache every turn, so billed tokens are mostly fixed overhead, *not* skill-specific work. Compare **turns** and **output tok** across runs/models/skill revisions, not the billed total.
- **Skill injected inline.** `SKILL.md` (~16 KB) is prepended to each prompt to simulate the skill being loaded, because it lives under `skills/` and isn't auto-discovered by the CLI.
- **`acceptEdits` mode.** File writes auto-accept (the deliverable); Bash is auto-denied, so the agent writes the `__main__` smoke-test block but never runs it. Runtime verification is out of scope — this measures code-generation work only.
- **Cost, not correctness.** This harness does not check the grep assertions in `evals.json` (still a spec — below).

## Why CI for these

Expand Down
15 changes: 15 additions & 0 deletions skills/hyenand-retrofit/evals/bench_results.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{"eval_id": 1, "eval_name": "native-pure-swap", "run": 1, "model": "claude-opus-4-8", "wall_sec": 29.1, "ts": 1780694126, "input_tokens": 1652, "output_tokens": 2096, "cache_creation_tokens": 21206, "cache_read_tokens": 114296, "billed_input_tokens": 137154, "billed_total_tokens": 139250, "cost_usd": 0.256294, "duration_ms": 28392, "num_turns": 5, "permission_denials": 0, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 1, "eval_name": "native-pure-swap", "run": 2, "model": "claude-opus-4-8", "wall_sec": 32.9, "ts": 1780694159, "input_tokens": 1654, "output_tokens": 2127, "cache_creation_tokens": 21323, "cache_read_tokens": 151091, "billed_input_tokens": 174068, "billed_total_tokens": 176195, "cost_usd": 0.276183, "duration_ms": 32061, "num_turns": 6, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 1, "eval_name": "native-pure-swap", "run": 3, "model": "claude-opus-4-8", "wall_sec": 31.4, "ts": 1780694190, "input_tokens": 1654, "output_tokens": 2209, "cache_creation_tokens": 1922, "cache_read_tokens": 170598, "billed_input_tokens": 174174, "billed_total_tokens": 176383, "cost_usd": 0.166725, "duration_ms": 30632, "num_turns": 6, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 2, "eval_name": "native-hybrid", "run": 1, "model": "claude-opus-4-8", "wall_sec": 46.4, "ts": 1780694237, "input_tokens": 1658, "output_tokens": 2725, "cache_creation_tokens": 21286, "cache_read_tokens": 220566, "billed_input_tokens": 243510, "billed_total_tokens": 246235, "cost_usd": 0.338142, "duration_ms": 45646, "num_turns": 8, "permission_denials": 4, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 2, "eval_name": "native-hybrid", "run": 2, "model": "claude-opus-4-8", "wall_sec": 72.0, "ts": 1780694309, "input_tokens": 1784, "output_tokens": 3052, "cache_creation_tokens": 4197, "cache_read_tokens": 316055, "billed_input_tokens": 322036, "billed_total_tokens": 325088, "cost_usd": 0.32163, "duration_ms": 71277, "num_turns": 10, "permission_denials": 12, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 2, "eval_name": "native-hybrid", "run": 3, "model": "claude-opus-4-8", "wall_sec": 39.5, "ts": 1780694348, "input_tokens": 138, "output_tokens": 2654, "cache_creation_tokens": 11716, "cache_read_tokens": 276258, "billed_input_tokens": 288112, "billed_total_tokens": 290766, "cost_usd": 0.284368, "duration_ms": 38700, "num_turns": 9, "permission_denials": 3, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 1, "model": "claude-opus-4-8", "wall_sec": 73.3, "ts": 1780694422, "input_tokens": 1654, "output_tokens": 6732, "cache_creation_tokens": 55399, "cache_read_tokens": 107587, "billed_input_tokens": 164640, "billed_total_tokens": 171372, "cost_usd": 0.582579, "duration_ms": 72511, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 2, "model": "claude-opus-4-8", "wall_sec": 82.7, "ts": 1780694504, "input_tokens": 1656, "output_tokens": 7262, "cache_creation_tokens": 6552, "cache_read_tokens": 184163, "billed_input_tokens": 192371, "billed_total_tokens": 199633, "cost_usd": 0.328834, "duration_ms": 81891, "num_turns": 6, "permission_denials": 3, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 3, "eval_name": "foreign-timm-vit", "run": 3, "model": "claude-opus-4-8", "wall_sec": 65.5, "ts": 1780694570, "input_tokens": 8, "output_tokens": 6260, "cache_creation_tokens": 5355, "cache_read_tokens": 122555, "billed_input_tokens": 127918, "billed_total_tokens": 134178, "cost_usd": 0.257258, "duration_ms": 64819, "num_turns": 4, "permission_denials": 1, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 1, "model": "claude-opus-4-8", "wall_sec": 76.4, "ts": 1780694646, "input_tokens": 1656, "output_tokens": 6567, "cache_creation_tokens": 22798, "cache_read_tokens": 184442, "billed_input_tokens": 208896, "billed_total_tokens": 215463, "cost_usd": 0.413177, "duration_ms": 75734, "num_turns": 6, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 2, "model": "claude-opus-4-8", "wall_sec": 59.0, "ts": 1780694706, "input_tokens": 1654, "output_tokens": 5431, "cache_creation_tokens": 4590, "cache_read_tokens": 154656, "billed_input_tokens": 160900, "billed_total_tokens": 166331, "cost_usd": 0.256059, "duration_ms": 58326, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 4, "eval_name": "foreign-3d-feature-map", "run": 3, "model": "claude-opus-4-8", "wall_sec": 61.5, "ts": 1780694767, "input_tokens": 10, "output_tokens": 5378, "cache_creation_tokens": 4864, "cache_read_tokens": 156728, "billed_input_tokens": 161602, "billed_total_tokens": 166980, "cost_usd": 0.249263, "duration_ms": 60723, "num_turns": 5, "permission_denials": 2, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 1, "model": "claude-opus-4-8", "wall_sec": 294.8, "ts": 1780695062, "input_tokens": 1914, "output_tokens": 11772, "cache_creation_tokens": 39646, "cache_read_tokens": 477955, "billed_input_tokens": 519515, "billed_total_tokens": 531287, "cost_usd": 1.244425, "duration_ms": 294059, "num_turns": 15, "permission_denials": 26, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 2, "model": "claude-opus-4-8", "wall_sec": 558.3, "ts": 1780695620, "input_tokens": 1918, "output_tokens": 12852, "cache_creation_tokens": 29890, "cache_read_tokens": 584403, "billed_input_tokens": 616211, "billed_total_tokens": 629063, "cost_usd": 5.170056, "duration_ms": 557564, "num_turns": 15, "permission_denials": 29, "primary_model": "claude-opus-4-8", "is_error": false}
{"eval_id": 5, "eval_name": "foreign-1d-causal-lm", "run": 3, "model": "claude-opus-4-8", "wall_sec": 211.9, "ts": 1780695832, "input_tokens": 1788, "output_tokens": 10492, "cache_creation_tokens": 20896, "cache_read_tokens": 410433, "billed_input_tokens": 433117, "billed_total_tokens": 443609, "cost_usd": 0.838811, "duration_ms": 211110, "num_turns": 11, "permission_denials": 10, "primary_model": "claude-opus-4-8", "is_error": false}
Binary file added skills/hyenand-retrofit/evals/bench_tokens.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading