fork-instrument: reduce synthetic frame-state locals

[brandonpayton]

Summary

The PR #701 reproducer shows that fork instrumentation increases V8 Wasm stack-frame pressure enough to reduce recursive survival depth. This PR reduces synthetic frame-state locals in wasm-fork-instrument while preserving the existing wpk_fork_* exports and continuation-frame layout.

Current PR head: 64b8977d5bf14f5c70c60beb32f210ec45153e05.

Implemented reductions:

• 97f51fd90: no-catch functions no longer declare unused catch_region_id and exnref_slot locals.
• 7546a7b75: top-level and nested switch dispatch reuse the continuation frame header instead of declaring synthetic frame_ptr and call_idx locals.
• a876580fa: tried packing no-catch zero stores into one i64.store offset=8; this reduced generated bytes but caused the browser staging shell demo regression.
• 64b8977d: keeps the frame-header reuse/local-count result, but restores the no-catch header zeroes to two scalar i32.stores because the packed i64.store was the browser regression boundary.

The current PR result is therefore the local-pressure reduction from frame-header reuse, not the packed-store code-size cleanup.

PR #701 MRE Results

MRE: /tmp/kandelo-pr701-mre, function benchmark_walk, Node v24.15.0. Lower declared locals are a proxy for lower V8 per-frame pressure. max_survived is the largest recursive depth completed before V8 stack overflow.

Local-count progression

Scenario	Commit / state	walk locals	Notes
Baseline, uninstrumented	control	4	Control build.
Original forkinstr before PR #713	historical PR #701 MRE	12	Starting point for this PR.
Catch-state-only cleanup	97f51fd90	10	Removes unused catch-state locals from no-catch functions.
Frame-header reuse	7546a7b75	8	Removes synthetic frame_ptr and call_idx locals by reusing the frame header.
Packed no-catch store attempt	a876580fa	8	Same local count, 10 bytes smaller in the MRE, but browser staging failed.
Current head, scalar no-catch stores	64b8977d	8	Keeps the local-count improvement and restores the previously green scalar store shape.

Size/depth measurements

Scenario	Wasm bytes	Code section	walk locals	max_survived	Context
Baseline, uninstrumented	3882	0x0aec	4	9959	Fresh kd-7d1 store-shape measurement.
Original forkinstr before PR #713	n/a	n/a	12	6639	Historical bead measurement; directional only against fresh rows.
Catch-state-only cleanup	n/a	n/a	10	6865	Earlier fresh local rerun from the PR body; another bead note recorded 7469 in a different run context.
Frame-header reuse, scalar no-catch stores	4646	0x0cc8	8	8536	Fresh kd-7d1 measurement for the 7546a7b75 generated shape; staging run was green.
Packed no-catch store attempt	4636	0x0cbe	8	8536	Fresh kd-7d1 measurement for a876580fa; browser staging later failed.
Current head, scalar no-catch stores	4646	0x0cc8	8	8536	Current head 64b8977d restores the scalar no-catch generated shape from 7546a7b75; no new depth claim beyond the last same-shape measurement.

Comparability note: V8 stack-depth numbers are only strict within the same measurement context. The durable local-count result is stable: original forkinstr 12 -> catch-state-only 10 -> current head 8.

Browser CI Regression And Fix

The packed-store commit a876580fa caused PR #713 staging browser failure in apps/browser-demos/test/kandelo-merge-gate.spec.ts while running the shell demo. The failing job timed out waiting for KANDELO_BASH_OK after the bash array/prompt command was echoed but did not complete.

The failure reproduced locally from the failed CI test-workspace artifact. The regression boundary was the no-catch postamble change from two scalar stores:

(i32.store offset=8 ...)
(i32.store offset=12 ...)

to one packed store:

(i64.store offset=8 ...)

64b8977d restores the two scalar no-catch stores while keeping frame-header reuse and avoiding synthetic frame_ptr / call_idx locals. Tests now reject the packed no-catch i64.store offset=8 shape.

CI status at this update:

• 7546a7b75 staging run: green.
• a876580fa staging run: browser failure reproduced locally.
• 64b8977d staging run 27803748679, attempt 1: cancelled before the browser test gate completed, so it did not prove the post-fix browser result.
• 64b8977d staging run 27803748679, attempt 2: in progress as of this update. Current visible status is 52 successful jobs, 3 skipped jobs, and 3 queued package-matrix jobs (zstd, node-vfs, lamp); the merge-gate context remains pending. This PR body does not claim current-head browser green status yet.

Validation

Passed for the frame-header reuse work:

• bash scripts/dev-shell.sh bash scripts/build-fork-instrument-tool.sh
• bash scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin --test switch_dispatch
• bash scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin --test instrument
• bash scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin --test large_dispatcher
• bash scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin
• bash scripts/dev-shell.sh bash scripts/ci-run-test-suite.sh fork-instrument
• bash scripts/dev-shell.sh bash scripts/check-abi-version.sh
• git diff --check for touched fork-instrument, docs, and reporting-guidance files
• PR Add standalone fork-instrument crash reproducer #701 standalone MRE measurements listed above

Passed after the current-head scalar-store fix:

• Focused browser Playwright repro failed before the fix with the CI symptom, using the failed CI test-workspace artifact.
• cargo test -p fork-instrument --target $HOST_TARGET --test instrument
• bash scripts/dev-shell.sh bash scripts/ci-run-test-suite.sh fork-instrument
• bash scripts/dev-shell.sh bash scripts/check-abi-version.sh
• git diff --check -- crates/fork-instrument/src/instrument.rs crates/fork-instrument/tests/instrument.rs

Not run or not claimed:

• Current-head post-fix browser merge-gate is not claimed green here because attempt 1 of staging run 27803748679 was cancelled before the browser test gate completed and attempt 2 is still in progress.
• A fresh current-head PR Add standalone fork-instrument crash reproducer #701 MRE rebuild was attempted from /Users/brandon/src/kandelo-kd-fbt, but wasm32posix-cc could not use that worktree's incomplete sysroot (sysroot/lib/libc.a missing). The current-head table therefore uses the last measured same generated-code shape for size/depth rather than claiming a new depth sweep.
• cargo test -p kandelo --target aarch64-apple-darwin --lib: not run because this change is limited to the fork-instrument code generator, generated Wasm shape, docs, and reporting guidance; it does not change kernel/libc/syscall behavior.
• cd host && npx vitest run: not run because no host runtime or browser/Node adapter code changed.
• scripts/run-libc-tests.sh: not run because no libc, syscall, or POSIX runtime behavior changed.
• scripts/run-posix-tests.sh: not run because no kernel/POSIX semantics changed.
• cargo fmt --check -p fork-instrument: not run successfully because this dev shell has no cargo fmt subcommand installed.

[github-actions]

Phase B-1 matrix build status — pr-713-staging

ABI v15. 52 built, 0 failed, 52 total.

Package	Arch	Status	Sha
bc	wasm32	built	3a78fb67
bzip2	wasm32	built	2ddc287d
coreutils	wasm32	built	2a18f004
curl	wasm32	built	a1ab622d
dash	wasm32	built	a4ab83b1
diffutils	wasm32	built	89790993
dinit	wasm32	built	b793ac0f
fbdoom	wasm32	built	d9296a33
file	wasm32	built	6a1455c3
findutils	wasm32	built	1237e4d0
gawk	wasm32	built	57c5d5cf
git	wasm32	built	7c88a387
grep	wasm32	built	235ddcd5
gzip	wasm32	built	481eb543
kandelo-sdk	wasm32	built	18869e79
kernel	wasm32	built	7f3002ad
less	wasm32	built	a4f09d75
lsof	wasm32	built	cbd163cd
m4	wasm32	built	f0aff0d3
make	wasm32	built	250d1d25
mariadb	wasm32	built	bccf7260
mariadb	wasm64	built	3df390c0
modeset	wasm32	built	703cfd80
msmtpd	wasm32	built	ec5df629
nano	wasm32	built	1cea23a0
ncurses	wasm32	built	8c54f1c2
netcat	wasm32	built	361622f8
nginx	wasm32	built	d3f35370
php	wasm32	built	c81ee5be
posix-utils-lite	wasm32	built	26ecd803
sed	wasm32	built	03befd20
tar	wasm32	built	bede734f
tcl	wasm32	built	9070fddb
unzip	wasm32	built	6591e7ef
userspace	wasm32	built	c3d7e154
vim	wasm32	built	0d02b67d
wget	wasm32	built	04bb22d8
xz	wasm32	built	f54f1e93
zip	wasm32	built	19152752
zstd	wasm32	built	45fa0e20
bash	wasm32	built	74f8e39c
mariadb-test	wasm32	built	76d2883a
mariadb-vfs	wasm32	built	37cbbded
mariadb-vfs	wasm64	built	a37c83f6
nethack	wasm32	built	51827b74
vim-browser-bundle	wasm32	built	63be56eb
nethack-browser-bundle	wasm32	built	6f83697f
rootfs	wasm32	built	bed032dc
shell	wasm32	built	bc60e6a2
lamp	wasm32	built	5fe3410c
node-vfs	wasm32	built	f286a3c5
wordpress	wasm32	built	33e16f14

_{Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.}

[brandonpayton]

Updated this draft with the deeper frame-header state change in 7546a7b7599496a7c37a35600b8d3dead785473e and force-pushed the existing PR branch after amending the commit body.

Fresh local MRE context: /tmp/kandelo-pr701-mre, benchmark_walk, Node v24.15.0, rebuilt tools/bin/wasm-fork-instrument from the measured branch/tool before each instrumented run.

Scenario	walk locals	max_survived	Context	Comparability note
Baseline, uninstrumented	4	9154	Fresh local rerun for this update	Direct control for the fresh catch-state-only and deeper rows. Historical original-run baseline was 9959.
Original forkinstr before PR #713	12	6639	Historical bead measurement from the PR #701 MRE on Node v24.15.0	Directional only against fresh rows; not a strict same-run comparison.
PR #713 catch-state-only	10	6865	Fresh local rerun using clean commit 97f51fd90 tool against the fresh baseline build	Directly comparable to the fresh baseline and deeper rows. Earlier bead notes recorded 7469.
Deeper frame-header-state reuse	8	7846	Fresh local rerun using amended commit 7546a7b75 tool	Directly comparable to the fresh baseline and catch-state-only rows.

Local validation run:

• bash scripts/dev-shell.sh bash scripts/build-fork-instrument-tool.sh
• focused tests: switch_dispatch 12/12, instrument 58/58, large_dispatcher 5/5 on aarch64-apple-darwin
• bash scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin
• bash scripts/dev-shell.sh bash scripts/ci-run-test-suite.sh fork-instrument
• bash scripts/dev-shell.sh bash scripts/check-abi-version.sh
• git diff --check for touched fork-instrument, fork docs, and reporting-guidance files

Broader kernel/host/libc/POSIX/browser suites were not run because this patch changes wasm-fork-instrument generated Wasm shape/tests/docs/reporting guidance and preserves the wpk_fork_* exports plus existing frame offsets. cargo fmt --check -p fork-instrument could not run because this dev shell has no cargo fmt subcommand installed.

[brandonpayton]

Build-size examples for the wasm-fork-instrument local-pressure changes.

These are raw .wasm byte sizes, not compressed package/archive sizes. Each row uses one uninstrumented input compiled once, then instruments the exact same input with three tool binaries:

• original pre-PR fork-instrument: reduce synthetic frame-state locals #713 tool: fff6e37bf from /tmp/kandelo-pr713-original
• PR fork-instrument: reduce synthetic frame-state locals #713 catch-state-only tool: 97f51fd90 from /tmp/kandelo-pr713-before
• current deeper frame-header-state tool: 7546a7b7599496a7c37a35600b8d3dead785473e from this PR branch

Tool build command for each checkout: bash scripts/dev-shell.sh bash scripts/build-fork-instrument-tool.sh. Input compile command context: scripts/dev-shell.sh with flake LLVM 21.1.7 and WASM_POSIX_LLVM_DIR=$LLVM_BIN so wasm-ld came from the repo dev shell, not Homebrew.

Artifact	Why this artifact	Input source/command	Uninstrumented input	Original tool fff6e37bf	Catch-state-only 97f51fd90	Current 7546a7b75	Current vs catch-only	Comparability note
PR #701 MRE chain.forkinstr.wasm	Required MRE; recursive benchmark_walk shape used for locals/depth evidence	wasm32posix-cc -O2 -Wl,-z,stack-size=67108864 -Wl,--export=benchmark_walk /tmp/kandelo-pr701-mre/chain.c	3,882 B (3.8 KiB)	4,587 B (4.5 KiB)	4,573 B (4.5 KiB)	4,646 B (4.5 KiB)	+73 B	Same uninstrumented input, same local run; size is directly comparable across tool versions.
programs/sh.c	Practical shell-style fork user with command execution/pipelines and many fork-reachable paths	wasm32posix-cc -O2 programs/sh.c	52,136 B (50.9 KiB)	82,433 B (80.5 KiB)	81,979 B (80.1 KiB)	89,207 B (87.1 KiB)	+7,228 B	Same uninstrumented input. This is a useful code-size tradeoff example: fewer declared locals can require more generated frame load/store code.
programs/p_06_fork_from_thread.c	Practical pthread fork path; representative of non-main-thread fork coverage	wasm32posix-cc -O2 programs/p_06_fork_from_thread.c	30,062 B (29.4 KiB)	31,233 B (30.5 KiB)	31,191 B (30.5 KiB)	31,266 B (30.5 KiB)	+75 B	Same uninstrumented input; shows the overhead can be small on simple fork paths.
programs/p_07_recursive_fork.c	Recursive/nested fork stress fixture; not a package, but useful for call-site-heavy generated-code shape	wasm32posix-cc -O2 programs/p_07_recursive_fork.c	27,809 B (27.2 KiB)	45,042 B (44.0 KiB)	44,854 B (43.8 KiB)	48,655 B (47.5 KiB)	+3,801 B	Same uninstrumented input; included to show the byte-size cost can grow with repeated fork-path dispatch sites.

Expected tradeoff: PR #713's first catch-state-only commit usually trims a small number of bytes by deleting unused locals/state handling in no-catch functions. The deeper frame-header-state commit reduces declared Wasm locals and improves the PR #701 stack-depth MRE, but it can increase raw .wasm size because generated code reloads/stores frame header fields from linear memory instead of using cached locals.

Additional candidate measured but not used as an “affected size” example: programs/p_04_popen_pclose.c compiled with wasm32posix-cc -O2 produced 37,414 B after instrumentation for all three tool versions, so it is a practical fork/exec/popen path but not useful for demonstrating this size delta.

Package artifact note: local package outputs such as packages/registry/dash/bin/dash.wasm, packages/registry/bash/bin/bash.wasm, and packages/registry/coreutils/bin/coreutils.wasm are already final instrumented binaries; their matching pre-instrumented inputs were not available in this worktree. I did not run broad package rebuilds just for this comment, because the bead requested representative examples and specifically said not to let this become a broad package rebuild.

[brandonpayton]

Some notes from the agent about further improvement possibilities. I'm currently reviewing this and deciding what might be reasonable to do now.

Wasm Fork PR713 Design Options

Date: 2026-06-18

Context: PR #713 reduces wasm-fork-instrument synthetic local pressure for
the PR #701 recursive MRE by reusing the continuation frame header. The current
design tradeoff is fewer declared locals and better recursive survival depth,
at the cost of more generated frame load/store code. These are the option sets
to keep in view for follow-up design and implementation.

1. Keep Instrumented Code Size Low

• Keep frame_ptr / call_idx locals: smallest generated code, but gives back
  PR fork-instrument: reduce synthetic frame-state locals #713's stack-pressure win. Mostly no-go for PR Add standalone fork-instrument crash reproducer #701 recursion work.
• Pack no-catch header zero stores: replace i32.store +8 and
  i32.store +12 with one i64.store offset=8. Low-risk follow-up.
• Single-load IfElse range test: replace idx >= lo && idx <= hi with
  (idx - lo) <= (hi - lo). Good, but needs a NORMAL-path safety test first.
• Improve indirect-call precision: instrument fewer false-positive
  fork-reachable functions. Good for size when conservative table closure
  over-instruments.
• Liveness-prune frame save/restore: save fewer scalar locals when proven dead
  after a fork-path call. Useful for bytes and frame-buffer size; correctness
  sensitive.
• Internal helper functions: __wpk_current_frame_ptr() /
  __wpk_current_call_index(). Could shrink repeated frame-load code without
  caller locals, but adds cold Wasm calls.
• Tune dispatch bucketing / br_table shape: can reduce very large dispatcher
  output. Existing bucketing helps, but thresholds remain tunable.
• Post-instrument wasm-opt: not a short-term option because fork
  instrumentation currently must run last; needs global-index robustness first.
• Shared postamble helper: mostly no-go because helpers cannot read caller
  locals without huge parameter lists or memory staging.
• Per-function size/stack heuristic: use compact locals for non-recursive
  functions, and lower-local memory/header loads for recursive SCCs. Viable
  research path.

2. Further Reduce Stack Pressure

• Keep PR fork-instrument: reduce synthetic frame-state locals #713 frame-header reuse: current 10 -> 8 local reduction; should
  stay.
• Pure argument-tail materialization: recompute pure call args after POST_K
  instead of spilling them to locals. Best next stack-pressure option.
• Pure condition/carryover materialization: recompute values like eqz(depth)
  instead of spilling. Good follow-up after pure args.
• Scratch pooling by value type: reuse one i32 scratch local across disjoint
  call-result checks. Good and relatively contained.
• Spill-local pooling: share arg/carryover locals across call sites when
  lifetimes cannot overlap. Useful in call-site-heavy functions.
• Liveness-prune synthetic and user local saves: may reduce frame traffic, but
  only reduces stack pressure if it also eliminates declared spill locals.
• Shadow/heap frame with one pointer local: can reduce locals, but adds
  normal-path allocation and memory traffic; not preferred for PR Add standalone fork-instrument crash reproducer #701.
• Function splitting/trampolines: can move locals out of the recursive
  function, but adds extra Wasm call frames; likely bad for the recursive MRE.
• Pack multiple i32 spills into one i64/local: possible but code-heavy and
  engine-sensitive; research only.
• Typed block-param continuations: use structured block params instead of
  locals in some shapes; complex validation work.
• Better fork-reachability analysis: avoids stack pressure in functions that
  should not have been instrumented.
• Host stack flags or skipped instrumentation: no-go as platform fixes.

3. Dedicated Tables For References Restored During Rewind

Constraint: funcref, externref, and exnref cannot be serialized into
linear memory. Table designs are about rooting and replaying references inside
an instance, not storing refs in the linear fork buffer.

• Status quo per-class aux tables: _wpk_fork_funcref_stash,
  _wpk_fork_externref_stash, _wpk_fork_exnref_stash; deterministic
  module-wide slots. Current supported path.
• Split tables by purpose: separate user-local refs, catch exnrefs,
  call-arg refs, and carryover refs. Better lifecycle/debugging, more tables.
• Per-function table ranges: keep one table per ref class but allocate
  dedicated ranges per function. Good compromise over many physical tables.
• Per-function physical tables: clearer isolation, but likely bloats table and
  function metadata.
• Dynamic ref-slot arena: allocate slots during unwind and store slot IDs in
  frame data. General, but needs lifecycle/free/reset correctness.
• Per-frame parallel ref stack: mirror linear continuation frames with table
  slots. Strong model, higher implementation cost.
• Plain-catch ref operand tables: needed to lift the current unsupported
  carve-out for catch operands containing refs.
• Ref-typed call arg/carryover tables: needed if fork-path calls carry
  funcref, externref, or exnref values across rewind.
• Wasm-GC typed tables: future option for any, eq, struct, array, and
  i31 refs; currently rejected.
• Host-owned ref registry: mostly no-go; host cannot portably serialize
  arbitrary Wasm refs across fork instances.
• Linear memory storage: hard no-go for refs; refs cannot be written to linear
  memory.

Practical Sequence

1. Small code-size follow-up: packed no-catch header zero store and single-load
   IfElse range test with focused NORMAL/REWIND fixtures.
2. Stack-pressure follow-up: pure materialization plus scratch pooling.
3. Ref-table expansion only when a real ref-carrying workload needs it.

[brandonpayton]

kd-7d1 PR update is published in a876580fa (fork-instrument: pack no-catch frame header stores).

Scope:

• No-catch postambles now write frame[8..16] with one i64.store offset=8 / i64.const 0.
• Catch-capable postambles still write dynamic catch_region_id and exnref_slot separately with i32.store offset=8 and i32.store offset=12.
• No frame_ptr or call_idx locals were reintroduced; wpk_fork_* exports and frame offsets are unchanged.

PR #701 MRE evidence (/tmp/kandelo-pr701-mre artifacts):

• Baseline chain.baseline.wasm: 3,882 bytes, code section 0x0aec, benchmark_walk locals 4.
• PR fork-instrument: reduce synthetic frame-state locals #713 head before packed cleanup chain.current-rerun.wasm: 4,646 bytes, code section 0x0cc8, benchmark_walk locals 8.
• After packed cleanup chain.pr713-packed.wasm: 4,636 bytes, code section 0x0cbe, benchmark_walk locals 8.
• Fresh Node v24.15.0 depth sweeps: baseline max_survived=9,959; pre-packed forkinstr max_survived=8,536; packed forkinstr max_survived=8,536.

Verification:

• Pre-fix focused test failed against the old generated WAT with separate zero i32.store offset=8 and i32.store offset=12.
• scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin --test instrument postamble -- --nocapture - 6 passed.
• scripts/dev-shell.sh cargo test -p fork-instrument --target aarch64-apple-darwin - full fork-instrument suite passed.
• scripts/dev-shell.sh bash scripts/check-abi-version.sh - ABI snapshot, libc header, TS bindings, and ABI_VERSION are consistent.
• git diff --check - passed.

[github-actions]

prepare-merge: test-gate passed against the synthetic PR merge and binaries-abi-v15. Missing entries were built or promoted from PR staging only when their merged-tree cache key matched; merge-gate=success posted on PR HEAD and squash auto-merge enabled.