quote lowering phase 5: JIT lane — jit_enabled triggers QuotePass, lift -jit folder skips

[borisbat]

Phase 5 of the quote-lowering plan (QUOTE_LOWERING.md; AOT lane = #3109): policies.jit_enabled now triggers the daslib/quote QuotePass, so quote/qmacro code compiles natively under the LLVM JIT instead of panicking the whole file, and the -jit folder skips are lifted.

What changed

• daslib/quote.das — QuotePass gate widens to aot_macros || (jit_enabled && !needMacroModule) || options aot_macros. Three traps found on first contact, all fixed here:
  1. Wrapper return type: the generated `quote`lowered`N function used autoinfer, leaking the concrete node type (quote(0) → ExprConstInt? instead of Expression?) and breaking cond ? clone_expression(x) : quote(0) (30411 — cond arms must match exactly; daslib/typemacro_boost has exactly this construct). Now fn.result = clone_type(expr._type) — identical to the raw quote's static type by construction.
  2. Macro-module programs don't lower under the jit trigger: macro contexts are never JITted (llvm_macro's is_compiling_macros() skip), so lowering there is pure cost — and the lowered construction frames overflowed the macro-context stack at apply time. Infer-time predicate is prog.flags.needMacroModule (is_compiling_macros() is only true during makeMacroModule's simulate, not when QuotePass runs). The ast_infer_type.cpp noAot-skip mirrors the same predicate.
  3. Macro-called non-macro modules (e.g. daslib/linq_fold_decs — plain functions invoked from linq_boost's fold macros) DO lower, and their lowered quotes evaluate at macro-apply time on the calling macro module's context stack. Without headroom: stack overflow while calling @linq_fold_decs::`quote`lowered`17 → 31206 → consumer file fails to compile, order-dependently (the margin varies with module-cache state — single-folder runs passed, full-tree runs failed deterministically; took a long bisect to pin). Fix: Program::makeMacroModule sizes the macro context to max(getContextStackSize(), 1MB) when aot_macros || jit_enabled — the headroom lands exactly where lowered quotes evaluate, for any embedder. (First cut bumped policies.stack in the three jit drivers, mirroring -aot-macros — that regressed wasm_cross: policies.stack also sizes the PRODUCED program's runtime context, and cross-compiled wasm trapped trying to carve 1MB out of linear memory. Reverted in the follow-up commit; the -aot-macros global bump keeps its historical behavior.)
• modules/dasLLVM/daslib/llvm_jit.das — DisableJitVisitor gets preVisitExprQuote (warn + per-function interpreter fallback). Raw quotes can still reach JIT from macro modules' runtime-used functions; previously that failed the whole file's codegen, now just that function stays interpreted (e.g. ast_match's peel_lambda_* helpers).
• Coverage lifted — tests/.das_test -jit skips (quote, ast, ast_match, no_aot, gc) and the matching jit_cache_all_tests --excludes are gone; tests/template/test_push_block_list.das [no_jit] markers removed (its stated reason — JIT can't lower ExprQuote — no longer holds; qmacro_block now JITs).
• No LLVM_JIT_CODEGEN_VERSION bump: emitters unchanged (gate + collection only), and the dll cache hash folds per-function AOT hashes, so lowered ASTs self-invalidate. (CLAUDE.md's stale pointer to the constant fixed — it lives in llvm_jit_run.das.)

Found along the way (pre-existing, not fixed here)

1. LLVM JIT doesn't implement force_escape_free / force_allocate_on_stack — three symptoms in tests/gc: heap grows where interp stays flat; the scope-exit free aborts (deleting ... which is not a chunk pointer) on JIT-allocated owning nodes under persistent_heap; nested new Inner(...) make-struct field arrives null. Scoped per-function [no_jit] markers on the workers in the three escape-analysis test files keep the rest of the gc folder JIT-covered. The JIT gap itself is a separate work item — flagging for a fix-vs-document decision.
2. Local-only: a deliberate panic inside jitted code aborts the process instead of unwinding to recover (exit 127, no diagnostic; LLVM 22.1.5 Windows, opt-level 3). Reproduces with master daslib state and an identical content-addressed dll hash, so unrelated to this PR; CI's LLVM doesn't show it (and the CI step has the --isolated-mode fallback). Affects tests/language/table_operations.das and tests/linq/test_linq_table_source.das locally.

Verification

• Full-tree dastest -jit (CI form, two pre-existing-crash files excluded locally): 10071 tests, 10065 passed, 0 failed, 0 errors, 6 skipped — ~550 more tests under JIT than the skip-era baseline. Re-verified identical after the makeMacroModule fixup. Folder detail: quote 20/20, ast 1/1, ast_match 380/380, flatten/no_aot 14/14, gc 48/48, template 10/10.
• Full-tree interp: 10152 / 10146 / 0 / 0 / 6.
• Full-tree AOT gate (test_aot -use-aot, fresh stubs): 9464 / 9458 / 0 / 0 / 6.
• jit-tool chunk-form spot-check over the formerly-excluded files: green.
• Lint + format clean on all changed .das files (two pre-existing lint findings in utils/jit/main.das cleaned while touching it).

Phase 6 (follow-up, per QUOTE_LOWERING.md): lift options no_aot on the ~41 quote-using test files case-by-case, JIT-of-macro-contexts behind a policy, chunked lowered-construction frames (removes the macro-context special-casing), and the imgui_demo compile-time measurement.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Enables quote lowering in the LLVM JIT lane by treating policies.jit_enabled as a trigger for daslib/quote’s QuotePass, so runtime quote/qmacro code can compile under JIT without failing whole-file codegen. This also removes broad -jit test-folder skips and replaces them with narrower per-function fallbacks/markers where JIT still lacks features.

Changes:

• Expand QuotePass gating to include jit_enabled (with a macro-module exclusion) and fix lowered-wrapper return typing.
• Increase policy stack size to 1MB in JIT-enabled entry points to prevent stack overflows from large lowered-quote construction frames.
• Lift -jit folder skips across tests and add targeted [no_jit] on specific GC escape-analysis workers where LLVM JIT behavior diverges.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/jit/main.das	Ensure standalone jit.exe registers LLVM JIT support and applies a 1MB stack when jit_enabled is used.
utils/daScript/main.cpp	Apply the same 1MB stack bump when running with JIT enabled (not just -aot-macros).
utils/CMakeLists.txt	Lift JIT prewarm excludes that were only needed before JIT quote lowering; keep module-based excludes.
modules/dasLLVM/daslib/llvm_jit.das	Add per-function JIT disable on raw ExprQuote (warning + interpreter fallback).
daslib/quote.das	Trigger quote lowering under jit_enabled (excluding macro-module programs) and preserve wrapper return type.
src/ast/ast_infer_type.cpp	Align noAot marking for quote() with the new jit_enabled lowering gate (incl. macro-module exclusion).
dastest/suite.das	Apply 1MB stack bump for JIT-enabled test compilation to support lowered quote frames.
dastest/dastest.das	Apply 1MB stack bump during JIT-enabled serialization compilation.
tests/.das_test	Remove -jit directory skips (quote/ast/ast_match/no_aot/gc) while keeping --use-aot gating.
tests/template/test_push_block_list.das	Remove [no_jit] markers now that block-form qmacro lowering works under JIT.
tests/gc/test_gc_escape_free.das	Add [no_jit] to the specific worker affected by missing JIT force_escape_free behavior.
tests/gc/test_gc_escape_free_frees.das	Add [no_jit] to escape-free workers (and control) due to JIT divergence/crash cases.
tests/gc/test_gc_allocate_on_stack.das	Add [no_jit] to ascend/owning workers due to missing JIT force_allocate_on_stack support.
skills/writing_tests.md	Update testing guidance to match lifted -jit folder gating and recommend per-function [no_jit].
QUOTE_LOWERING.md	Document Phase 5 completion and key findings for the JIT lane.
CLAUDE.md	Correct the documented location/meaning of LLVM_JIT_CODEGEN_VERSION.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

[borisbat]

Debug CI round (3 red lanes: linux/darwin15/darwin26 Debug) — root-caused, fixed in 93cfca0.

All three lanes trace to one root cause that only Debug builds can see, plus collateral:

1. The real bug — tests/gc is unsound under JIT, and not just for force_escape_free. test_gc_escape_free.das test_escape_free_does_not_free_returned double-freed: heap_collect cannot see a heap pointer whose only reference is a local in a jitted frame (jitted locals live on the native stack, invisible to the collector), so the object is freed as garbage and the test's later delete n is a double free. Debug's memory_model.h:109 assert catches it; Release passes by luck until the slot is reused. Probe (Release-observable, with or without force_escape_free):

var n = make_node(7)            // returned pointer, held in a local
unsafe { heap_collect(true, true) }
// ... churn 1000 allocations ...
print("n.x={n.x}")              // interp: 7    -jit: -1 (slot reused)

That makes GC-semantics tests systemically unsound under JIT — almost certainly why the gc folder was on the -jit skip list originally. So: the folder-level gc skip in tests/.das_test is restored (root cause in the comment), --exclude gc is back on jit_cache_all_tests, and the per-function [no_jit] markers from the earlier round are reverted (tests/gc back to master state). Lifting it again needs the JIT to spill GC roots somewhere the collector can scan — flagged as upstream work alongside the force_escape_free/force_allocate_on_stack gap.

2. The 3 linux-Debug typer_errors "failures" — collateral from a pre-existing CI infra race, not real. The gc crash sent the sweep into the || --isolated-mode fallback. Workers spawn with -jit -jit, whose harness dll hashes differently (0x41e26b231da05490) than the prewarmed outer run's — so all 32 workers cache-missed the same dll path simultaneously and raced writing one .o/.dll: "file format not recognized" (half-written .o), "failed to set dynamic section sizes: file truncated" (rewritten mid-link), and the llvm_jit_run.das:342 post-write verify assert (dll swapped between write and reopen). Pre-existing — any crash in the -jit lane re-exposes it. It heals here because the sweep no longer crashes into the fallback, but worth its own fix: lock/atomic-rename in write_dll, or prewarm the worker-flag-combination dll before fan-out. Noted in QUOTE_LOWERING.md; can file an issue if you want it tracked.

Lesson recorded in skills/writing_tests.md: a green local Release sweep does not prove a skip-lift sound — daslang assert and the C++ memory-model asserts both compile out of Release; Debug CI is the oracle for memory-divergence.

Local gates after the fix: tests/gc interp 48/48; full -jit sweep 10023 tests, 10017 passed, 0 failed, 0 errors, 6 skipped (= previous 10071 minus the 48 gc tests).

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.