Demo-first onboarding: shareable, scrubbable run-replay artifact (cap-evolve replay)

[OsherElhadad]

Closes #122.

What this is

cap-evolve replay renders a recorded run as one self-contained HTML file that plays the run back over its own virtual timeline — play / pause / scrub / speed. You watch candidates get proposed, accepted or rejected with the gate's reason, the best-so-far advance, and live accepted/rejected/event counters, instead of only seeing the final reduced state that dashboard.html shows today.

How a newcomer opens it with zero setup

cap-evolve replay --demo --open

No project, no capevolve.yaml, no credentials, no model calls, no server, no node, no network. --demo uses a real recorded toy_calc run committed at examples/toy_calc/recorded_run/ (68 KB: events.jsonl + baseline/final/state/splits + rollouts), produced by the actual zero-API mock-optimizer pipeline — not a synthetic fixture. The command prints the file it wrote and its file:// URL; the artifact opens straight from disk.

cap-evolve replay [run_dir] (or with no argument: your newest run) does the same for your own runs. Linked as step 0 of docs/GETTING_STARTED.md and as the first line of the site's "try it" block.

Reduced motion is honoured: under prefers-reduced-motion: reduce the page renders the finished state instead of autoplaying. Long eval pauses are clamped (1.5 s of virtual time per gap) so a 15-minute eval doesn't stall playback.

How backward scrubbing works

read_new_events(path, 0) gives the whole ordered log deterministically; build_replay folds it into a frame list (rel, kind, line, cand, status, val, best) that is inlined in the artifact. Seeking to any time — forwards or backwards — is a linear scan to an array index and a recompute of the counters from frames[0..k]. No seeking, no time→offset index.

That is the deliberate answer to #191's reviewer's noted gap. A single-file shareable artifact has to inline the whole log anyway, so an index would buy nothing: an offset index only pays off when you can avoid reading the bytes, and here they are already in the file. Bound stated in the code (ponytail: comment): events.jsonl is 3.4 KB for toy_calc, ~200 KB for a 500-iteration run — fine in memory and fine inlined. If a run ever logs enough to matter, the fix is to cap the frame list, not to make the reader seek.

Scrubbing (it's shareable, so it's redacted) — canary evidence

Everything goes through the existing dashboard.redact; no second scrubber. Two real leaks were found by the canary test and fixed at the root:

1. A bare high-entropy secret under an innocent key name leaked. _env_secret_values (the Add cap-evolve doctor install/health diagnostic #193/Terminal degradation ladder + redacted forensic crash log (#144) #215 hunk) keys on the key name, so MODEL_ENDPOINT_SUFFIX=canaryDDD… passed straight through — exactly what Terminal degradation ladder + redacted forensic crash log (#144) #215's review predicted. Fixed by also covering values that are opaque by shape (≥20 chars, no whitespace, no /, ≥2 character classes) under any key name. Whitespace// exclusions keep PATH/PWD/prose settings out, so the false-positive cost is "a long opaque config value shows as «redacted» in a report" — cheap next to shipping a live key in a file people paste into issues.
2. Rendering truncated a secret in half before redaction could match it. format_event clips optimizer_error to 200 chars, and half a credential matches no shape regex and no literal env value. build_replay now redacts each event before rendering it.

Also ported the ghp_ / github_pat_ / UUID shapes (identical hunk to #193 and #215 — whichever lands first, the others are a no-op merge).

Six canaries in four shapes planted under both innocent and secret-looking env keys, and inside both model-written fields (reason and optimizer_error.error, the latter with no KEY= prefix to lean on), then grep -c on a real generated artifact:

env key	shape	canary	grep -c
OPENAI_API_KEY	vendor prefix	sk-canaryAAAA…1	0
GITHUB_TOKEN	GitHub PAT	ghp_canaryBBBB…2	0
WATSONX_APIKEY	bare high-entropy (watsonx-style)	canaryCCCC…3	0
MODEL_ENDPOINT_SUFFIX (innocent)	bare high-entropy	canaryDDDD…4	0
DEPLOYMENT_ID (innocent)	UUID	3f2a1b4c-canary-…	0
RUNTIME_PROFILE (innocent)	fine-grained PAT	github_pat_canaryEEEE…5	0

grep -c -i canary over the whole artifact → 0. Full output in the Evidence comment.

Injection — I fixed #209's sink

Not avoided: fixed at the root, because the replay payload rides the same interpolation and every other future payload field would inherit the bug.

dashboard.py:657 was json.dumps(...).replace("</", "<\\/") — a one-sequence denylist. Replaced with one shared dashboard.json_for_html that encodes every <, >, & (plus U+2028/9) as < etc. Inside a JSON string literal those parse back to the original characters, so the data the JS reads is bit-identical while being inert to the HTML parser. Encode-everything, not the sixth denylist in this batch.

Measured in headless Chromium on real generated artifacts, asserting structure, not "no exception":

reason / optimizer_error payload	sections	body chars	replay present
plain text (control)	5	1292	yes
<!--<script> — old sink	0	34	no
<!--<script> — new sink	5	1288	yes
</script><script>…innerHTML=""</script>	5	1368	yes
newline-forged fake FINALIZE line	5	1408	yes

The newline forgery and ANSI/OSC (\x1b]0;pwned\x07\x1b[2J) are already dead via #191's sanitize, which every frame's line goes through by construction — asserted in the test ("\n" not in line, "\x1b" not in line).

Zero external origins

grep -oE "https?://[A-Za-z0-9.-]+" over the built artifact → http://www.w3.org only (the SVG XML namespace, not a fetch). No <link, no cdn., no fetch(, no @import.

Rather than a second guard, I widened the existing one (test_render_html_self_contained_and_parseable) from a marker denylist to a shape check — any absolute http(s) URL, one-host allowlist — and made it assert the Run replay panel. The replay artifact is the same renderer, so it's covered by construction. This is the same denylist→shape lesson as #192's CDN guard.

Expected merge order

1. feat(observability): live terminal progress via --follow and cap-evolve tail #191 (issue Classic cap-evolve run is silent for its entire duration — add --follow / cap-evolve tail #116) — this PR is based on feat/issue-116-follow-tail and consumes eventstream. Merge feat(observability): live terminal progress via --follow and cap-evolve tail #191 first; the base then retargets to main cleanly.
2. Either order with Add cap-evolve doctor install/health diagnostic #193 / Terminal degradation ladder + redacted forensic crash log (#144) #215 — the ghp_/github_pat_/UUID + _env_secret_values hunk is identical in all three, so the later ones are no-op merges. My _looks_opaque extension sits on top and is additive.
3. Independent of Distinguish a stalled/hung run from idle/done (SSE 5-min idle + coarse status heuristic) #118 (PR feat(observability): classify a run as working / stalled / crashed / done #218) and fix(dashboard): render the live event ticker in the SPA; populate the algorithm label #204 — different consumers of the same stream; no shared lines.
4. dashboard/frontend/dist/ is not touched or committed (per Committed dashboard dist/ with hashed filenames makes every concurrent frontend PR conflict, and can silently ship a stale bundle #188).

Verification

$ PYTHONPATH=core python -m pytest core/tests -q
225 passed in 68.85s (0:01:08)

225 = 179 baseline + 38 from #191's base + 8 new in core/tests/test_replay_artifact.py. 0 failed. test_dashboard_launch.py::test_maybe_launch_spawns_when_available (#200, environmentally flaky on port 7878) passed in this environment.

$ python -m compileall -q core skills
compileall clean

Real artifact from a real zero-API run:

$ PYTHONPATH=core python -m cap_evolve.cli replay --demo -o /tmp/replay-demo.html
{"artifact": "/tmp/replay-demo.html", "frames": 15, "run": ".../examples/toy_calc/recorded_run", "url": "file:///private/tmp/replay-demo.html"}

$ grep -oE "https?://[A-Za-z0-9.\-]+" /tmp/replay-demo.html | sort | uniq -c
   1 http://www.w3.org

Headless Chromium, scrubbing the real demo artifact — forward to the end, then backwards to the middle and to zero (state recomputes correctly each time):

sections     = 6            (was 5 before this change: the replay panel)
body_chars   = 2266         (was 913)
log_rows     = 15
at_end        -> 2.8s / 2.8s · 15/15 | [03:25:38] FINALIZE  test=1.0000 (baseline 0.0000, Δ+1.0000)  best=cand_0001
back_to_mid   -> 1.4s / 2.8s ·  6/15 | [03:25:37] ACCEPT  cand_0001  val=1.0000 (parent 0.0000) — paired Δ̄=+1.0000 > 0
back_to_zero  -> 0.0s / 2.8s ·  1/15 | [03:25:36] splits frozen  train=4 val=2 test=2 (test sealed)

Counters at those positions: best so far —/accepted 0/rejected 0 at t=0 → best so far 1.000/accepted 1/rejected 2 at the end, and correctly back down when scrubbed backwards.

No frontend changes (git status --short dashboard/ → empty), so no npm ci / tsc / vitest run is needed.

Full commands and untruncated output in the 🔬 Evidence comment.

Files touched

• core/cap_evolve/dashboard.py — build_replay, json_for_html (Model-controlled event text can blank the generated dashboard.html (incomplete script-context escape at dashboard.py:657) #209 fix), replay player CSS+JS in the template, redact hardening
• core/cap_evolve/cli.py — cap-evolve replay subcommand + DEMO_RUN
• core/tests/test_replay_artifact.py — new, 8 tests
• core/tests/test_dashboard.py — existing self-contained guard widened to a shape check + replay panel
• examples/toy_calc/recorded_run/ — new, the bundled real run for --demo
• examples/toy_calc/README.md, docs/GETTING_STARTED.md, site/index.html, CHANGELOG.md

[OsherElhadad]

🔬 Evidence

All commands run in a clean worktree at origin/feat/issue-116-follow-tail + this commit. Python 3.14, macOS.

1. Full test suite

$ cd /tmp/wt-122 && PYTHONPATH=/tmp/wt-122/core python -m pytest core/tests -q
........................................................................ [ 32%]
........................................................................ [ 64%]
........................................................................ [ 96%]
.........                                                                [100%]
225 passed in 67.84s (0:01:07)

2. The 8 new tests, verbose

$ PYTHONPATH=core python -m pytest core/tests/test_replay_artifact.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.2, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/ce-venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/wt-122/core
configfile: pyproject.toml
plugins: anyio-4.14.2
collecting ... collected 8 items

core/tests/test_replay_artifact.py::test_replay_frames_are_ordered_and_cover_the_run PASSED [ 12%]
core/tests/test_replay_artifact.py::test_replay_survives_a_malformed_event PASSED [ 25%]
core/tests/test_replay_artifact.py::test_replay_is_embedded_in_the_artifact PASSED [ 37%]
core/tests/test_replay_artifact.py::test_artifact_references_no_external_origin PASSED [ 50%]
core/tests/test_replay_artifact.py::test_no_canary_of_any_shape_survives_into_the_artifact PASSED [ 62%]
core/tests/test_replay_artifact.py::test_hostile_event_text_does_not_change_the_artifact_structure PASSED [ 75%]
core/tests/test_replay_artifact.py::test_json_for_html_encodes_every_parser_shifting_char PASSED [ 87%]
core/tests/test_replay_artifact.py::test_replay_demo_builds_the_bundled_artifact_with_no_project PASSED [100%]

============================== 8 passed in 0.30s ===============================

3. compileall

$ python -m compileall -q core skills; echo "exit=$?"
exit=0

4. Produce the real artifact from the real zero-API run

First, the recorded run itself came from the real pipeline (examples/toy_calc + mock optimizer, no API):

$ python -m cap_evolve.cli run --spec $D/.capevolve/project/capevolve.yaml --project $D/.capevolve/project --run-ts demo
{
  "run_dir": ".capevolve/run_demo",
  "best_id": "cand_0001",
  "baseline_val": 0.0,
  "test_reward": 1.0,
  "test_baseline_reward": 0.0,
  "test_delta": 1.0,
  "test_pass_k": {"1": 1.0, "2": 0.0},
  "iterations": 3,
  "dashboard": ".capevolve/run_demo/dashboard.html"
}

That run dir is committed as examples/toy_calc/recorded_run/. Now the artifact:

$ PYTHONPATH=core python -m cap_evolve.cli replay --demo -o /tmp/replay-demo.html
{"artifact": "/tmp/replay-demo.html", "frames": 15, "run": "/private/tmp/wt-122/examples/toy_calc/recorded_run", "url": "file:///private/tmp/replay-demo.html"}

$ ls -l /tmp/replay-demo.html
41625 /tmp/replay-demo.html

5. Zero external origins

$ grep -oE "https?://[A-Za-z0-9.\-]+" /tmp/replay-demo.html | sort | uniq -c
   1 http://www.w3.org

$ for m in "<link" "cdn." "fetch(" "@import" "src=\"http" "href=\"http"; do printf "%-14s %s
" "$m" "$(grep -c -F -- "$m" /tmp/replay-demo.html)"; done
<link          0
cdn.           0
fetch(         0
@import        0
src="http      0
href="http     0

# the one http:// is the SVG XML namespace attribute, not a subresource:
$ grep -oE ".{25}http://www.w3.org.{12}" /tmp/replay-demo.html

The existing guard (widened from a marker denylist to a shape check) covers it:

$ PYTHONPATH=core python -m pytest core/tests/test_dashboard.py -q -k self_contained
.                                                                        [100%]
1 passed, 9 deselected in 0.04s

6. The replay actually steps through the run — headless Chromium

Probe script (/tmp/probe.py) loads the artifact from file://, reads the DOM, then drives the range input forward to the end and backwards to the middle and to zero:

$ python /tmp/probe.py /tmp/replay-demo.html /tmp/replay-demo.png
{
  "sections": 6,
  "body_chars": 2350,
  "replay_present": 1,
  "log_rows": 15,
  "clock_t0": "1.0s / 2.8s · 6/15",
  "now_t0": "[03:25:37] ACCEPT  cand_0001  val=1.0000 (parent 0.0000)  — paired Δ̄=+1.0000 > 0 (SE=0 → STRICT fallback, warned; n=2)",
  "meters_t0": "best so far 1.000\naccepted 1\nrejected 0\nevents 6",
  "errors": [
    "Cannot set properties of undefined (setting 'textContent')"
  ],
  "at_end": {
    "clock": "2.8s / 2.8s · 15/15",
    "now": "[03:25:38] FINALIZE  test=1.0000 (baseline 0.0000, Δ+1.0000)  best=cand_0001",
    "meters": "best so far 1.000\naccepted 1\nrejected 2\nevents 15",
    "cur": "[03:25:38] FINALIZE  test=1.0000 (baseline 0.0000, Δ+1.0000)  best=cand_0001"
  },
  "back_to_mid": {
    "clock": "1.4s / 2.8s · 6/15",
    "now": "[03:25:37] ACCEPT  cand_0001  val=1.0000 (parent 0.0000)  — paired Δ̄=+1.0000 > 0 (SE=0 → STRICT fallback, warned; n=2)",
    "meters": "best so far 1.000\naccepted 1\nrejected 0\nevents 6",
    "cur": "[03:25:37] ACCEPT  cand_0001  val=1.0000 (parent 0.0000)  — paired Δ̄=+1.0000 > 0 (SE=0 → STRICT fallback, warned; n=2)"
  },
  "back_to_zero": {
    "clock": "0.0s / 2.8s · 1/15",
    "now": "[03:25:36] splits frozen  train=4 val=2 test=2 (test sealed)",
    "meters": "best so far —\naccepted 0\nrejected 0\nevents 1",
    "cur": "[03:25:36] splits frozen  train=4 val=2 test=2 (test sealed)"
  },
  "screenshot": "/tmp/replay-demo.png"
}

The single pageerror is pre-existing on main (the chart panel's text-content attribute trick), present in an unmodified dashboard.html too — not introduced here:

$ python /tmp/probe2.py <dashboard.html generated by the run, before this change>
{"sections": 5, "body_chars": 913, "rp": 0, "errors": ["Cannot set properties of undefined (setting 'textContent')"]}

So the artifact goes 5 sections / 913 chars → 6 sections / 2266 chars with a working player, and scrubbing backwards recomputes best-so-far and the accepted/rejected counters correctly at every position.

7. No leak — 6 canaries, 4 shapes, innocent AND secret-looking keys

Planted in reason (as KEY=value) and in optimizer_error.error (bare, no KEY= prefix to lean on), then grepped on the generated artifact:

$ bash /tmp/canary.sh
planted 6 canaries under ['OPENAI_API_KEY', 'GITHUB_TOKEN', 'WATSONX_APIKEY', 'MODEL_ENDPOINT_SUFFIX', 'DEPLOYMENT_ID', 'RUNTIME_PROFILE']
{"artifact": "/tmp/canary.html", "frames": 18, "run": "/tmp/canaryrun/run", "url": "file:///private/tmp/canary.html"}
--- grep -c each canary in the artifact ---
OPENAI_API_KEY           sk-canaryAAAAAAAAAAAAAAAAAAAAAAAA1             -> 0
GITHUB_TOKEN             ghp_canaryBBBBBBBBBBBBBBBBBBBBBBBB2            -> 0
WATSONX_APIKEY           canaryCCCCCCCCCCCCCCCCCCCCCCCCCCCC3            -> 0
MODEL_ENDPOINT_SUFFIX    canaryDDDDDDDDDDDDDDDDDDDDDDDDDDDD4            -> 0
DEPLOYMENT_ID            3f2a1b4c-canary-4d5e-8f90-abcdef012345         -> 0
RUNTIME_PROFILE          github_pat_canaryEEEEEEEEEEEEEEEEEEEEEEEE5     -> 0
--- any 'canary' substring at all ---
0
0

Three of those keys — MODEL_ENDPOINT_SUFFIX, DEPLOYMENT_ID, RUNTIME_PROFILE — contain no key/token/secret/password substring, so a key-name heuristic does not see them. That is what caught the two real bugs fixed here.

Proof the leak was real — with only the #193/#215 key-name-based _env_secret_values (i.e. before _looks_opaque), the test fails on exactly the innocent-key canary:

AssertionError: canary leaked into the artifact under MODEL_ENDPOINT_SUFFIX: canaryDDDDDDDDDDDDDDDDDDDDDDDDDDDD4
  'canaryDDDDDDDDDDDDDDDDDDDDDDDDDDDD4' is contained here:
    NT_SUFFIX=canaryDDDDDDDDDDDDDDDDDDDDDDDDDDDD4 DEPLOYMENT_ID=3f2a1b4c-canary-4d5e-8f90-abcdef012345 RUNTIME_PROFILE=«redacted»", "parent_val": 0.25, ...

And a second real bug, from the same test: format_event truncates optimizer_error to 200 chars, slicing a credential in half so no shape rule and no literal env value matched it afterwards. Fixed by redacting the event before rendering:

AssertionError: assert 'canary' not in '{"graph": {...}'
  'canary' is contained here:
     3f2a1b4c-canary-4d5", "cand": "cand_0002", "status": "rejected", ...

(3f2a1b4c-canary-4d5 — the UUID cut mid-token by the 200-char clip.)

8. Injection is inert — structure asserted, not "no exception"

Four real artifacts built with the hostile string in both reason and optimizer_error.error, each loaded in headless Chromium:

$ python /tmp/probe2.py /tmp/inj-clean.html
{"sections": 5, "body_chars": 1292, "rp": 1, "errors": ["Cannot set properties of undefined (setting 'textContent')"]}
$ python /tmp/probe2.py /tmp/inj-comment.html
{"sections": 5, "body_chars": 1288, "rp": 1, "errors": ["Cannot set properties of undefined (setting 'textContent')"]}
$ python /tmp/probe2.py /tmp/inj-close.html
{"sections": 5, "body_chars": 1368, "rp": 1, "errors": ["Cannot set properties of undefined (setting 'textContent')"]}
$ python /tmp/probe2.py /tmp/inj-forge.html
{"sections": 5, "body_chars": 1408, "rp": 1, "errors": ["Cannot set properties of undefined (setting 'textContent')"]}

payload	sections	body chars	replay panel
a plain reason (control)	5	1292	yes
<!--<script>	5	1288	yes
</script><script>document.body.innerHTML=""</script>	5	1368	yes
newline-forged FINALIZE test=1.0000 … best=FAKE	5	1408	yes

And the old sink, for contrast. I reverted json_for_html to main's .replace("</", "<\/") in place and rebuilt the same <!--<script> artifact:

$ grep -c "OLD #209 sink" core/cap_evolve/dashboard.py
1
$ python /tmp/inject.py "<!--<script>" /tmp/inj-OLD.html && python /tmp/probe2.py /tmp/inj-OLD.html
{"sections": 0, "body_chars": 34, "rp": 0, "errors": []}

0 sections, 34 chars, no player, and no error raised — the silent denial-of-view #209 describes. With the encode-everything fix: 5 sections, 1288 chars, player present. The file was restored immediately after (git diff --stat confirmed the sink back to the fixed form before committing).

9. Frontend untouched (no dist/ commit, per #188)

$ git status --short dashboard/
(empty)

$ git show --stat HEAD | grep -c dashboard/frontend
0
0

10. Commit authorship

$ git log -1 --format="%an <%ae>%n%s"
Osher Elhadad <Osher.Elhadad@ibm.com>
feat(dx): shareable, scrubbable run-replay artifact + `cap-evolve replay` (#122)

$ git log -1 --format=%B | grep -ci "co-authored-by\|generated with"
0
0

$ git show --stat HEAD:

 .../recorded_run/rollouts/test/a8__FINAL__t0.json  |   1 +
 .../rollouts/test/a8__FINAL_seed__t0.json          |   1 +
 .../rollouts/val/a1__cand_0001__t0.json            |   1 +
 .../rollouts/val/a1__cand_0002__t0.json            |   1 +
 .../rollouts/val/a1__cand_0003__t0.json            |   1 +
 .../recorded_run/rollouts/val/a1__seed__t0.json    |   1 +
 .../rollouts/val/a4__cand_0001__t0.json            |   1 +
 .../rollouts/val/a4__cand_0002__t0.json            |   1 +
 .../rollouts/val/a4__cand_0003__t0.json            |   1 +
 .../recorded_run/rollouts/val/a4__seed__t0.json    |   1 +
 examples/toy_calc/recorded_run/splits.json         |  18 ++
 examples/toy_calc/recorded_run/state.json          |  25 +++
 site/index.html                                    |   9 +-
 25 files changed, 819 insertions(+), 9 deletions(-)

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.