feat: comprehensive multi-suite benchmark system (AWFY + CLBG + Wren)

[txloc1909]

Summary

A benchmark system for Lox++ with two complementary harnesses:

1. benchmarks/crosslang.py — apples-to-apples cross-language comparison. Every language runs the identical algorithm at the identical problem size and self-times only the compute region (interpreter/container startup excluded), reported as min-of-N. This is the only cross-comparable harness. Covers 6 languages (lox++, clox, wren, python, lua, node) on 4 book-Lox micro-benchmarks (fib, towers, queens, mandelbrot). lox++ and clox share benchmarks/clox/*.lox; benchmarks/{python,lua,js,wren}/ hold matching ports.
2. benchmarks/runner.py — lox++ suite + AWFY across python/lua/js. Runs lox++ across its own 26-program suite (AWFY / CLBG / Wren ports under benchmarks/lox/), and the AWFY suite for python/lua/js via the SOM harness. The lox++ ports are faithful AWFY translations and manifest.toml sets each benchmark's inner to do the identical work, so all 12 AWFY benchmarks are apples-to-apple across lox++/python/lua/js (bounce, cd, deltablue, list, mandelbrot, nbody, permute, queens, richards, sieve, storage, towers). crosslang.py adds clox + wren on 4 book-Lox micros.

Cross-language results (apples-to-apples, min of 5, compute-only)

All languages produce identical results (fib→9227465, towers→1048575, queens→92, mandelbrot→3959).

benchmark	lox++	clox	wren	python	lua	node (V8)
fib (35)	1004 ms	680 ms	987 ms	845 ms	486 ms	80 ms
towers (hanoi 20)	133 ms	82 ms	96 ms	100 ms	52 ms	7 ms
queens (8)	7.7 ms	5.5 ms	5.9 ms	2.7 ms	3.9 ms	1.9 ms
mandelbrot (100)	37 ms	31 ms	35 ms	28 ms	21 ms	4.9 ms

• lox++ is in the same class as wren (both simple NaN-tagged bytecode VMs).
• It trails its nearest faster peer clox by ~1.4× geomean — a per-opcode-cost gap (same bytecode, slower dispatch/operand handling), not a codegen gap.
• lua (register VM) leads the non-JIT pack; node/V8 JITs to native (the ceiling, not a like-for-like VM comparison).

What's in the suite

• 26 lox++ benchmark programs under benchmarks/lox/ — AWFY (16), CLBG (6), Wren (4). 22 run clean today; binary_trees/fannkuch are allocation-heavy (long), and deltablue (compile error) / havlak (runtime error) need follow-up.
• 4 strict book-Lox programs under benchmarks/clox/ (no lists/maps) shared by lox++ and clox.
• Cross-language ports under benchmarks/{python,lua,js,wren}/ for the 4 micro-benchmarks.
• Container images: Dockerfile.clox (builds clox at a pinned commit), Dockerfile.lua (lua5.4), Dockerfile.wren (builds the wren CLI from wren-lang/wren-cli); python/node use official python:3.12-slim / node:22-alpine. See benchmarks/SOURCES.md.
• manifest.toml (benchmark × language → file) and langs.toml (interpreter configs) for runner.py.

Infrastructure fixes

The third-party plumbing did not run as originally written; the runnable paths are now fixed:

• manifest.toml was invalid TOML (table header + key on one line) — rewritten as inline tables so runner.py can load it.
• runner.py: lox++ uses build-tree paths unconditionally; the AWFY adapter accepts scientific notation (str() prints 1.07e+06 us); the clox/raw adapter reads the program's self-timed elapsed instead of podman wall time.
• Dockerfile.clox: add libc6-dev (was failing on stdlib.h).
• Dockerfile.wren: build the wren CLI from wren-lang/wren-cli (the wren-lang/wren repo builds only a test runner).
• Dockerfile.lua: minimal lua5.4 interpreter image.

Quick start

# Apples-to-apples cross-language matrix (builds/pulls the 6 images)
python3 benchmarks/crosslang.py                 # all 4 benchmarks, all 6 langs
python3 benchmarks/crosslang.py --bench fib --lang lox clox

# lox++ suite tracking
python3 benchmarks/runner.py --lang lox --format markdown

# AWFY suite across languages (needs bench-python / bench-lua / bench-js)
python3 benchmarks/runner.py --suite awfy --lang lox python lua js --format markdown

Known limitations / follow-ups

• havlak lox++ port overflows the 256-frame call stack (recursive doDFS) and uses a non-AWFY signature, so it's lox++-only and currently errors — needs an iterative rewrite to run/compare. fib/earley have no AWFY python/lua/js port.
• binary_trees / fannkuch (CLBG, lox++-only) are slow (allocation-heavy).
• CI nightly runner + lox++ regression tracking — follow-up.

🤖 Generated with Claude Code

[txloc1909]

Benchmark report — lox++ vs candidate languages

Ran the suite on this branch (rebased onto current main, which now includes the profiler #50 and the bit-mask hash probe #92). Honest headline: the only cross-implementation comparison I could actually run is lox++ vs clox — the Python/Lua/JS/Wren container images do not build, and manifest.toml doesn't parse, so runner.py can't drive a full matrix yet (details at the bottom). clox is also the most useful comparison anyway: same language, the canonical bytecode VM, both NaN-boxed.

Environment

• Container loxpp-dev-env (Ubuntu 24.04). lox++: clang-18, Release preset (-O3, NaN-tagging on). clox: munificent/craftinginterpreters@4a840f70, gcc -O3 -march=native, NAN_BOXING on, switch dispatch.
• Self-timed via each program's own clock() (CPU seconds), min of 5 runs. Both interpreters produce identical results on every benchmark.
• Caveat: clox gets -march=native; lox++ uses the stock Release preset (no -march=native), so part of the gap is build flags, not the VM.

lox++ vs clox (the 4 book-Lox benchmarks clox can run)

clox is strict book Lox (no lists/maps), so only these four overlap:

benchmark	lox++ (s)	clox (s)	lox++ / clox
fib	0.983	0.807	1.22×
towers	0.120	0.0785	1.53×
queens	0.00529	0.00504	1.05×
mandelbrot	0.0358	0.0255	1.40×

lox++ is ~1.05–1.53× slower than clox (geomean ≈ 1.3×). queens is essentially at parity; towers/mandelbrot show the widest gaps. From the earlier fib profiling, the gap is per-opcode runtime cost (identical bytecode/opcode counts), not codegen — the remaining levers are registerizing the instruction pointer, 1-byte constant operands, and inlining/unchecking getConstant. (The bit-mask probe in #92 already shaved fib from ~1.38× to ~1.22×.)

lox++ full-suite coverage (all 26 programs, wall-clock, 60s cap)

22/26 run clean. 4 need attention on current main:

status	benchmarks
✅ ok (22)	bounce, cd, earley, fasta, fib, for_in, instantiation, json, k_nucleotide, list, mandelbrot, nbody, permute, queens, reverse_complement, richards, sieve, spectral_norm, storage, string_interning, towers, zoo
⏱️ >60s timeout	binary_trees, fannkuch (allocation/GC-heavy)
❌ compile error	deltablue — [line 869] Error at end: Expect '}'
❌ runtime error	havlak — [line 752] in script (rc=70)

Why the other languages didn't run (multi-language infra is currently broken)

The PR description lists 9 infra fixes, but they don't appear to be in the branch — every non-lox path fails:

• manifest.toml is invalid TOML — [bounce.lox] file = "..." puts a table header and a key on one line; tomllib rejects it at line 12, so runner.py can't even load the manifest. (Needs inline tables, e.g. lox = { file = "bounce.lox" }.)
• runner.py lox path — lox is forced through run_direct, but the host-path override only triggers under --no-container, so a unified run uses container paths (/workspace/...) on the host and fails. (if lang == "lox" and no_container: → if lang == "lox":.)
• AWFY images (python/lua/js) — pinned commit 5e9fa8e… 404s (master HEAD is 74306fec); AWFY filenames are now lowercase (bounce.py, not Bounce.py), and the benchmarks need harness invocation, not direct file exec; alpine busybox tar rejects --wildcards (lua/js).
• clox image — Dockerfile.clox installs gcc make but not libc6-dev, so stdlib.h: No such file. (I built clox directly with the full toolchain for this report.)
• wren image — pinned commit 4a8e084… 404s.

Happy to do the infra repair (manifest + runner + 5 Dockerfiles + harness wrappers) as a follow-up so a true all-languages matrix can run — it's a meaningful chunk of work on its own.

_{🤖 Generated with Claude Code}

[txloc1909]

Apples-to-apples cross-language results

Reworked the cross-language comparison to be truly apples-to-apples (per review): every language runs the identical algorithm at the identical problem size and self-times only the compute region (interpreter/container startup excluded), reported as min of 5 runs. All languages produce identical results (fib→9227465, towers→1048575, queens→92, mandelbrot→3959).

New harness: benchmarks/crosslang.py. lox++ and clox share benchmarks/clox/*.lox; benchmarks/{python,lua,js}/ hold matching ports. Run in pinned containers (loxpp-dev-env, bench-clox, bench-lua, python:3.12-slim, node:22-alpine).

benchmark	lox++	clox	python	lua	node (V8)
fib (35)	953 ms	661 ms	842 ms	492 ms	81 ms
towers (hanoi 20)	125 ms	74 ms	99 ms	50 ms	7 ms
queens (8)	7.7 ms	5.5 ms	2.8 ms	2.9 ms	1.8 ms
mandelbrot (100)	42 ms	34 ms	25 ms	20 ms	4.3 ms

Normalized (× the fastest, node):

benchmark	lox++	clox	python	lua	node
fib	11.7×	8.1×	10.3×	6.1×	1.0×
towers	16.8×	10.0×	13.4×	6.7×	1.0×
queens	4.3×	3.0×	1.6×	1.6×	1.0×
mandelbrot	9.8×	8.0×	5.9×	4.6×	1.0×

Reading it

• node/V8 wins everywhere — it JITs to native; the rest are bytecode interpreters, so this is the expected ceiling, not a like-for-like VM comparison.
• The meaningful peer for lox++ is clox (both are simple switch-dispatch bytecode VMs, both NaN-boxed). lox++ trails clox by 1.2–1.7× (geomean ~1.4×) — consistent with the per-opcode-cost analysis: same bytecode, slower dispatch/operand handling. The merged bit-mask probe (perf: bitmask hash-table probe instead of modulo #92) already closed part of fib's gap.
• lua (optimized register VM) and CPython both beat lox++ here, so there is real headroom. The ranked levers from the fib profiling — registerize ip, 1-byte constant operands, inline/uncheck getConstant — target exactly this.

Infra fixed in this PR (4 commits)

The original multi-language plumbing didn't run at all; the runnable parts are now fixed:

1. manifest.toml rewritten as valid TOML (it didn't parse → runner.py couldn't load it).
2. runner.py: lox++ uses build-tree paths unconditionally; AWFY adapter accepts scientific notation; the clox/raw adapter reads the program's self-timed elapsed instead of podman wall time.
3. Dockerfile.clox: add libc6-dev (was failing on stdlib.h).
4. crosslang.py + python/lua/js ports + a minimal Dockerfile.lua.

Still WIP (documented in SOURCES.md)

• wren: needs wren-lang/wren-cli (the wren-lang/wren repo builds only a test runner, not a CLI), so it's not in the matrix yet.
• The AWFY-harness multi-suite for python/lua/js via runner.py (Dockerfile.python/.js) still pins a 404 commit and needs harness wiring; the apples-to-apples harness supersedes it for cross-language comparison. Adding lua/js/python/wren ports for the remaining benchmarks to crosslang.py is the path to a fuller matrix.

_{🤖 Generated with Claude Code}

[txloc1909]

Full 6-language apples-to-apples matrix (wren added)

Added wren to the cross-language harness — all 6 candidate languages now run the identical program at the identical size, self-timed (startup excluded), min of 5. All produce identical results (fib→9227465, towers→1048575, queens→92, mandelbrot→3959).

The wren CLI is built from wren-lang/wren-cli (make config=release_64bit, pinned) — the wren-lang/wren repo the PR originally used builds only a test runner, not a CLI.

benchmark	lox++	clox	wren	python	lua	node (V8)
fib (35)	1004 ms	680 ms	987 ms	845 ms	486 ms	80 ms
towers (hanoi 20)	133 ms	82 ms	96 ms	100 ms	52 ms	7 ms
queens (8)	7.7 ms	5.5 ms	5.9 ms	2.7 ms	3.9 ms	1.9 ms
mandelbrot (100)	37 ms	31 ms	35 ms	28 ms	21 ms	4.9 ms

Normalized (× fastest = node):

benchmark	lox++	clox	wren	python	lua	node
fib	12.5×	8.5×	12.3×	10.5×	6.1×	1.0×
towers	18.3×	11.4×	13.2×	13.7×	7.2×	1.0×
queens	4.1×	3.0×	3.2×	1.5×	2.1×	1.0×
mandelbrot	7.5×	6.3×	7.1×	5.6×	4.3×	1.0×

Reading it

• node/V8 JITs to native — the expected ceiling, not a like-for-like VM comparison.
• lox++ and wren are in the same class (both simple NaN-tagged bytecode VMs): lox++ is slightly behind wren on towers/queens, roughly even on fib/mandelbrot.
• clox is the nearest faster peer; lox++ trails it by 1.2–1.6× (geomean ~1.4×) — consistent with the per-opcode-cost analysis (same bytecode, slower dispatch/operand handling).
• lua (optimized register VM) leads the non-JIT pack; CPython is competitive on the small/arithmetic benchmarks. So there's real headroom — the ranked levers from the fib profiling (registerize ip, 1-byte constant operands, inline/uncheck getConstant) target exactly the lox++↔clox gap.

Infra delivered in this PR

• crosslang.py apples-to-apples harness + python/lua/js/wren ports (lox & clox share benchmarks/clox/).
• Fixes that made the suite runnable: valid manifest.toml; corrected runner adapters; libc6-dev for clox; correct Dockerfile.wren (wren-cli); minimal Dockerfile.lua.
• Remaining WIP (documented in SOURCES.md): the AWFY-harness multi-suite for python/lua/js via runner.py (superseded by this harness for cross-language comparison).

_{🤖 Generated with Claude Code}

[txloc1909]

AWFY suite wired across python/lua/js

runner.py now runs the AWFY suite for python/lua/js via the SOM harness (images at the valid commit 74306fec; manifest.toml carries a per-benchmark inner matching the lox++ port). Run with:

python3 benchmarks/runner.py --suite awfy --lang lox python lua js --format markdown

Apples-to-apple AWFY rows (lox++ port runs AWFY's exact algorithm at a reproducible size), min-of-5 averages:

benchmark	lox++	clox¹	python	lua	node
bounce	1.1 ms	–	1.5 ms	1.3 ms	1.0 ms
cd	11.8 ms	–	8.9 ms	11.0 ms	2.8 ms
mandelbrot (750)	1.87 s	–	1.95 s	658 ms	81 ms
nbody (250k)	1.47 s	–	1.32 s	725 ms	22 ms
richards	35 ms	–	40 ms	36 ms	3.4 ms

¹ clox can't run the AWFY ports (lists/maps); it's in the crosslang.py micro matrix instead.

lox++ tracks CPython closely here (faster on mandelbrot/nbody/richards, slower on cd), with lua ahead and node/V8 well ahead — consistent with the crosslang.py micro results.

Honest caveat

The remaining lox++ AWFY ports use reduced/ad-hoc sizes that don't map onto AWFY's harness, so those rows are not apples-to-apple and are excluded above: sieve (lox++ 2 ms vs python 3.7 s), queens (lox++ 1.6 ms vs python 8.2 ms), list, permute, storage, towers. deltablue/havlak lox++ ports don't build/run, and fib/earley have no AWFY python/lua/js port. Aligning those lox++ ports to AWFY's standard workloads is follow-up; crosslang.py remains the guaranteed cross-comparable harness.

_{🤖 Generated with Claude Code}

[txloc1909]

AWFY lox++ ports aligned — full suite now apples-to-apple

Root cause of the earlier divergences: the lox++ AWFY ports are actually faithful translations of AWFY's benchmark() — the mismatch was in my manifest.toml inner values. For queens/sieve I'd set inner to the internal size (10, 5000), but those ports already do that work inside benchmark(), so inner must be 1 (it's the repeat count). sieve was running 5000× more work in python than lox++. Also fixed deltablue (it had /* */ block comments, which lox++'s lexer rejects).

All 12 AWFY benchmarks are now apples-to-apple across lox++/python/lua/js (min-of-5 avg):

benchmark	lox++	python	lua	node
bounce	1.1 ms	0.81 ms	1.4 ms	0.75 ms
cd	10.8 ms	8.6 ms	10.4 ms	3.0 ms
deltablue	5.0 ms	5.9 ms	14.3 ms	2.6 ms
list	1.3 ms	0.52 ms	1.7 ms	0.36 ms
mandelbrot	1.66 s	1.89 s	626 ms	80 ms
nbody	1.48 s	1.31 s	730 ms	23 ms
permute	1.7 ms	1.3 ms	1.5 ms	0.69 ms
queens	1.5 ms	0.64 ms	1.7 ms	0.52 ms
richards	36 ms	37 ms	38 ms	3.7 ms
sieve	1.2 ms	0.60 ms	0.82 ms	0.27 ms
storage	6.7 ms	1.9 ms	3.0 ms	0.88 ms
towers	5.8 ms	1.5 ms	2.9 ms	0.84 ms

lox++ vs CPython: roughly even on compute-heavy benchmarks (mandelbrot, nbody, richards, deltablue, cd) and ~2–4× behind on the allocation/list-heavy ones (storage, towers, list, sieve) — pointing at list/object allocation as the next optimization target. lua leads the non-JIT pack; node/V8 JITs to native.

Run it:

python3 benchmarks/runner.py --suite awfy --lang lox python lua js --format markdown

Still lox++-only: havlak (recursive doDFS overflows the 256-frame stack + non-AWFY signature — needs an iterative rewrite), and fib/earley (no AWFY python/lua/js port). crosslang.py remains the 6-language (incl. clox, wren) book-Lox micro comparison.

_{🤖 Generated with Claude Code}