fasl: record shen.*lambdatable* deltas by name instead of serializing live lambdas - #45
Merged
Conversation
… live lambdas
On the S41.2 kernel every (define ...) in a loaded file runs
shen.update-lambdatable, which (set)s shen.*lambdatable* to an assoc
list whose entries hold live curried lambda functions. The fasl
recorder captured that as an ordinary "g" (set) record, kdata_ser hit
the closure ("unserializable KDATA value: function"), and the whole
file became fasl-uncacheable — in practice ~every stdlib and user file,
so the ~1s stdlib load re-compiled on every boot and every (load) paid
full reader+macro+typecheck cost on every run.
Record a new "lt" record instead: diff the new table against the
current global (the recorder runs before the original set, so the old
table is still live) and store only the NAMES whose entry was
added/replaced. On replay, rebuild each entry from the live defun the
same way shen.update-lambdatable does — shen.lambda-entry (which reads
the arity property, already replayed by the preceding "p" record in
stream order) + shen.assoc->. Mirrors the existing "lf" special case
for the older kernel's shen.lambda-form put path.
Bump FASL_FORMAT to SHENFASL5 so stale caches miss cleanly.
Measured (arm64 macOS, LuaJIT 2.1 rolling): trivial-script startup
1.5s -> 0.34s warm; all 21 StLib files now fasl-hit (previously 19/21
uncacheable). make test 500/0, run-kernel-tests 134/0, warm/cold suite
stdout identical modulo the by-design 'run time' banner drop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the S41.2 kernel, every
(define ...)in a loaded file runsshen.update-lambdatable, which(set)sshen.*lambdatable*to an assoc list whose entries hold live curried lambda functions. The fasl recorder captured that as an ordinaryg(set) record,kdata_serhit the closure (unserializable KDATA value: functionat boot.lua:254), and the whole file became fasl-uncacheable.In practice that was ~every stdlib and user file (19 of 21 StLib files on a cold boot), so:
(load)paid full reader+macro+typecheck cost on every run.This is the dominant share of the
~1.5scold-start reported for a trivial script (vs ~13ms for shen-cl), and a fixed per-invocation tax on every suite run in batch runners like urdr's Bifrost gate.Fix
Record a new
ltrecord instead: diff the new table against the current global (the recorder runs before the originalset, so the old table is still live) and store only the names whose entry was added/replaced. On replay, rebuild each entry from the live defun the same wayshen.update-lambdatabledoes —shen.lambda-entry(which reads the arity property, already replayed by the precedingprecord in stream order) +shen.assoc->. This mirrors the existinglfspecial case for the older kernel'sshen.lambda-formput path.FASL_FORMATbumped toSHENFASL5so stale caches miss cleanly.Measured (arm64 macOS, LuaJIT 2.1 rolling, S41.2)
Verification
make test: 500 pass / 0 fail across 14 specsrun-kernel-tests.lua(canonical certification): 134 passed / 0 failed, okshen/tests/prngsuite: warm/cold stdout identical modulo the by-designrun timebanner drop;ALL PASSboth ways🤖 Generated with Claude Code