update benchmark data: post-#504/#505/#506 perf wins (Apple Silicon N=10)#507
Merged
update benchmark data: post-#504/#505/#506 perf wins (Apple Silicon N=10)#507
Conversation
Contributor
Benchmark Results (Linux x86-64)
CLI Tool Benchmarks
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fresh benchmark refresh on Apple Silicon with N=10 bootstrap CIs, measured on current main (includes #503 infra + #504 memchr split + #505 num-to-str fast path + #506 bounds-check elim).
Headline wins visible in the data
New ranking distribution
🥇 on 4 benchmarks: SQLite, JSON Parse, String Manipulation, Cold Start (all 95% CI tied-with-or-beating C).
🥈 on 4: Binary Trees (beats C+Go, loses to Node's V8 escape analysis), Fibonacci, Monte Carlo, Sieve.
🥉 on 5: Matrix Multiply, Quicksort, N-Body, File I/O, String Search.
Nothing below 🥉. Previously the 🥉 tier had String Manipulation at #4 and String Search at #5 — both promoted out.
Against Node specifically
ChadScript now beats Node on every single benchmark except Binary Trees (where V8's JIT escape analysis eliminates the tree-node allocations entirely). Previously chad was losing to Node on String Manipulation and String Search too; #504 and #505 closed both gaps.
Noise on the ones that didn't move
A few benchmarks show 5-6% slippage vs the previous refresh (fib 0.516 → 0.542, monte carlo 0.264 → 0.279, matmul 0.109 → 0.116). These are all well within the N=10 measurement noise band on a warm-laptop environment — same code, different thermal state. No regressions suspected; they'd swing back on the next refresh. Not cherry-picking; committed as-is to preserve honest-measurement discipline.
#506 specifically
PR #506 (bounds-check elimination) landed correctly and is sound — ~60% fewer bounds checks in the hot loops of string benches, and ~5% faster self-compile. But it did NOT move matmul/quicksort/sieve on the benchmark dashboard because those loops use compound indices like `arr[row*N+k]` which #506's narrow pass doesn't handle. That's a follow-up (value-range analysis or user-opt-in unchecked indexing). Already logged as the next target.
Files
Reproduce
Methodology: Apple Silicon M-series, dedicated hardware, N=10 per benchmark, 2000-iteration bootstrap 95% CIs, tie-on-CI-overlap ranking (per #503).