Skip to content

update benchmark data: post-#504/#505/#506 perf wins (Apple Silicon N=10)#507

Merged
cs01 merged 1 commit intomainfrom
bench-refresh-post-perf
Apr 13, 2026
Merged

update benchmark data: post-#504/#505/#506 perf wins (Apple Silicon N=10)#507
cs01 merged 1 commit intomainfrom
bench-refresh-post-perf

Conversation

@cs01
Copy link
Copy Markdown
Owner

@cs01 cs01 commented Apr 13, 2026

Fresh benchmark refresh on Apple Silicon with N=10 bootstrap CIs, measured on current main (includes #503 infra + #504 memchr split + #505 num-to-str fast path + #506 bounds-check elim).

Headline wins visible in the data

bench pre-perf (from #503) post-perf (this run) delta rank change
String Manipulation 0.017s 0.007s −59% #4 → 🥇
String Search 0.020s 0.008s −60% #5 → 🥉
Cold Start 5.9ms 5.8ms ~tie (CI bound) 🥈 → 🥇 (non-overlap vs Go)

New ranking distribution

🥇 on 4 benchmarks: SQLite, JSON Parse, String Manipulation, Cold Start (all 95% CI tied-with-or-beating C).

🥈 on 4: Binary Trees (beats C+Go, loses to Node's V8 escape analysis), Fibonacci, Monte Carlo, Sieve.

🥉 on 5: Matrix Multiply, Quicksort, N-Body, File I/O, String Search.

Nothing below 🥉. Previously the 🥉 tier had String Manipulation at #4 and String Search at #5 — both promoted out.

Against Node specifically

ChadScript now beats Node on every single benchmark except Binary Trees (where V8's JIT escape analysis eliminates the tree-node allocations entirely). Previously chad was losing to Node on String Manipulation and String Search too; #504 and #505 closed both gaps.

Noise on the ones that didn't move

A few benchmarks show 5-6% slippage vs the previous refresh (fib 0.516 → 0.542, monte carlo 0.264 → 0.279, matmul 0.109 → 0.116). These are all well within the N=10 measurement noise band on a warm-laptop environment — same code, different thermal state. No regressions suspected; they'd swing back on the next refresh. Not cherry-picking; committed as-is to preserve honest-measurement discipline.

#506 specifically

PR #506 (bounds-check elimination) landed correctly and is sound — ~60% fewer bounds checks in the hot loops of string benches, and ~5% faster self-compile. But it did NOT move matmul/quicksort/sieve on the benchmark dashboard because those loops use compound indices like `arr[row*N+k]` which #506's narrow pass doesn't handle. That's a follow-up (value-range analysis or user-opt-in unchecked indexing). Already logged as the next target.

Files

  • `docs/public/benchmarks.json` — regenerated with N=10 + CI fields
  • `docs/public/benchmarks-all.json` — same (unfiltered view)
  • `README.md` — headline table refreshed with new numbers; rows re-ordered by vs-Node margin

Reproduce

BENCH_RUNS=10 ./benchmarks/run.sh

Methodology: Apple Silicon M-series, dedicated hardware, N=10 per benchmark, 2000-iteration bootstrap 95% CIs, tie-on-CI-overlap ranking (per #503).

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark Results (Linux x86-64)

Benchmark C ChadScript Go Node Place
Binary Trees 1.420s 1.265s 2.731s 1.177s 🥈
Cold Start 0.9ms 0.8ms 1.2ms 27.7ms 🥇
Fibonacci 0.814s 0.763s 1.559s 3.179s 🥇
File I/O 0.121s 0.094s 0.087s 0.207s 🥈
JSON Parse/Stringify 0.004s 0.005s 0.018s 0.016s 🥈
Matrix Multiply 0.445s 0.972s 0.577s 0.368s #4
Monte Carlo Pi 0.389s 0.410s 0.404s 2.249s 🥉
N-Body Simulation 1.665s 2.124s 2.202s 2.381s 🥈
Quicksort 0.215s 0.246s 0.213s 0.261s 🥉
SQLite 0.357s 0.376s 0.444s 🥈
Sieve of Eratosthenes 0.015s 0.029s 0.021s 0.040s 🥉
String Manipulation 0.008s 0.017s 0.016s 0.035s 🥉

CLI Tool Benchmarks

Benchmark ChadScript grep node xxd Place
Hex Dump 0.427s 1.021s 0.130s 🥈
Recursive Grep 0.019s 0.010s 0.097s 🥈

@cs01 cs01 merged commit 76a1d9e into main Apr 13, 2026
10 of 12 checks passed
@cs01 cs01 deleted the bench-refresh-post-perf branch April 13, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant