quote lowering phase 5: JIT lane — jit_enabled triggers QuotePass, lift -jit folder skips

CLAUDE.md

@@ -310,7 +310,7 @@ Most layout is obvious from `ls`. Non-obvious ones worth knowing:
 - `utils/detect-dupe/` (in-repo dupe finder) and `utils/find-dupe/` (Claude-based judge that needs `daspkg install --root utils/find-dupe` + `ANTHROPIC_API_KEY`) — both also exposed as MCP tools
 - `utils/mcp/`, `utils/daslang-live/`, `utils/daspkg/` — in-tree tools (most also have skills under `skills/`)
 - `tutorials/language/` (language tour) vs `tutorials/<area>/` (per-area) — never put tutorials in `modules/<X>/tutorial/`
-- **Array/dim/vector indexing lives across 5 tiers** — bug fixes (bounds checks, neg-index detection, width-aware bounds) usually need parallel edits in all of them: AOT C++ ([include/daScript/simulate/aot.h](include/daScript/simulate/aot.h)), interpreter non-fused ([include/daScript/simulate/runtime_array.h](include/daScript/simulate/runtime_array.h) + [include/daScript/simulate/simulate_nodes.h](include/daScript/simulate/simulate_nodes.h)), interpreter fused ([src/simulate/simulate_fusion_at_array.cpp](src/simulate/simulate_fusion_at_array.cpp) + [src/simulate/simulate_fusion_at.cpp](src/simulate/simulate_fusion_at.cpp)), JIT ([modules/dasLLVM/daslib/llvm_jit.das](modules/dasLLVM/daslib/llvm_jit.das)), and AST const-fold ([src/ast/ast_simulate.cpp](src/ast/ast_simulate.cpp)). Debug builds bypass the fused permutations — a fix that lands only in the fused path passes Release CI and trips Debug-ARM. Bump `LLVM_JIT_CODEGEN_VERSION` in `modules/dasLLVM/daslib/llvm_macro.das` after JIT changes to invalidate cached `.dll`s
+- **Array/dim/vector indexing lives across 5 tiers** — bug fixes (bounds checks, neg-index detection, width-aware bounds) usually need parallel edits in all of them: AOT C++ ([include/daScript/simulate/aot.h](include/daScript/simulate/aot.h)), interpreter non-fused ([include/daScript/simulate/runtime_array.h](include/daScript/simulate/runtime_array.h) + [include/daScript/simulate/simulate_nodes.h](include/daScript/simulate/simulate_nodes.h)), interpreter fused ([src/simulate/simulate_fusion_at_array.cpp](src/simulate/simulate_fusion_at_array.cpp) + [src/simulate/simulate_fusion_at.cpp](src/simulate/simulate_fusion_at.cpp)), JIT ([modules/dasLLVM/daslib/llvm_jit.das](modules/dasLLVM/daslib/llvm_jit.das)), and AST const-fold ([src/ast/ast_simulate.cpp](src/ast/ast_simulate.cpp)). Debug builds bypass the fused permutations — a fix that lands only in the fused path passes Release CI and trips Debug-ARM. Bump `LLVM_JIT_CODEGEN_VERSION` in `modules/dasLLVM/daslib/llvm_jit_run.das` after JIT emitter/ABI changes to invalidate cached `.dll`s (the cache hash also folds per-function AOT hashes, so AST-level changes self-invalidate without a bump)
 
 ## MCP Server (AI Tool Integration)
 

QUOTE_LOWERING.md

@@ -247,14 +247,94 @@ JIT work in scope before then is the Phase 1 diagnostic (clean `unsupported` mes
 replacing the misleading "Internal jit error" panic). The QuotePass gate stays
 `aot_macros`-only until this phase; extending it to `jit_enabled` lands here.
 
-- [ ] Extend QuotePass gate to `jit_enabled`.
-- [ ] `tests/jit_tests/` coverage for quote under `options jit` (lowered path through
-      LlvmJitVisitor — make-struct of handled types, `to_array_move`, by-name lookups).
+- [x] Extend QuotePass gate to `jit_enabled` (2026-06-12). Two traps found on first contact,
+      both via `daslib/typemacro_boost`:
+      1. **Wrapper return type must be the raw quote's type.** `fn.result = autoinfer` leaked
+         the concrete node type (`quote(0)` → `ExprConstInt?` instead of `Expression?`),
+         breaking `cond ? clone_expression(x) : quote(0)` (30411, cond arms must match
+         exactly). Fix: `fn.result = clone_type(expr._type)` — identity by construction.
+      2. **Macro-module programs must not lower under the jit trigger.** Macro contexts are
+         never JITted (llvm_macro's `is_compiling_macros()` skip), and the lowered
+         construction frames overflowed the macro-context stack at apply time
+         (`stack overflow while calling @typemacro_boost::\`quote\`lowered\`28`). Predicate
+         gotcha: `is_compiling_macros()` reads `Program::isCompilingMacros`, which is only
+         true during `makeMacroModule`'s simulate — at infer time (when QuotePass runs) use
+         `prog.flags.needMacroModule` instead (set at annotation-apply, pre-infer). The
+         infer-time noAot-skip in ast_infer_type.cpp mirrors the same predicate. Raw quotes
+         reaching JIT from such modules fall back per-function via the new
+         DisableJitVisitor `preVisitExprQuote` (warn + disable, replacing whole-file panic).
+- [x] Coverage instead of new jit_tests: lifted the `-jit` folder skips (quote, ast,
+      ast_match, no_aot) in tests/.das_test + the matching `--exclude`s on
+      jit_cache_all_tests, and un-marked `[no_jit]` in tests/template/test_push_block_list
+      (its stated reason — JIT can't lower ExprQuote — is gone; qmacro_block now JITs).
+      Folder results: quote 20/20, ast 1/1, ast_match 380/380 (2 graceful per-function
+      fallbacks in daslib/ast_match — macro module, stays raw by design), flatten/no_aot
+      14/14, template 10/10. The gc skip was lifted too, then RESTORED — see next item.
+- [x] NOT quote-related, surfaced (and re-buried) while lifting the gc skip: tests/gc is
+      unsound under JIT for two reasons. (1) **LLVM JIT does not implement
+      `force_escape_free` / `force_allocate_on_stack`** — heap grows where interp stays
+      flat; scope-exit free aborts "not a chunk pointer" on JIT-allocated owning nodes
+      under persistent_heap; nested `new Inner` make-struct field arrives null. (2) The
+      one Debug CI caught after the per-function `[no_jit]` round: **`heap_collect` cannot
+      see a heap pointer whose only reference is a local in a jitted frame** (native-stack
+      locals are invisible to the collector) — the object is freed as garbage, and the
+      test's later `delete` double-frees (Debug memory_model.h:109 assert; Release passes
+      by luck until the slot is reused — probe: interp prints n.x=7, jit prints n.x=-1,
+      with or without force_escape_free). (2) is systemic for GC-semantics tests, so the
+      folder-level `-jit` skip in tests/.das_test is restored (root cause in the comment)
+      and the scoped `[no_jit]` markers were reverted. Lift again only when the JIT spills
+      GC roots somewhere the collector can scan. Upstream fix is its own work item.
+- [x] Pre-existing CI infra bug exposed by the gc crash (NOT this PR's doing, heals once
+      the gc skip is back): when the main `-jit` sweep crashes, build.yml falls back to
+      `--isolated-mode` — 32 worker subprocesses all spawn with `-jit -jit`, whose harness
+      dll hash differs from the outer run's prewarmed one → every worker cache-misses the
+      SAME dll path simultaneously and they race writing the same `.o`/`.dll`: "file
+      format not recognized" (half-written .o), "failed to set dynamic section sizes:
+      file truncated" (rewritten mid-link), and the llvm_jit_run.das:342 post-write
+      verify assert (dll swapped between write and reopen). Cost 3 collateral
+      typer_errors "failures" on linux Debug. Fix candidates: lock/atomic-rename in
+      write_dll, or prewarm the worker-flag-combination dll before fan-out.
+- [x] Local full-tree `-jit` sweep caveat (NOT this PR's doing): the in-process sweep
+      aborts (exit 127, no diagnostic) at tests/language/table_operations.das
+      `ta_test_lock_panic` — a deliberate table-lock panic inside jitted code fails to
+      unwind to `recover` and kills the process. **Pre-existing**: reproduces with master
+      daslib state, and the content-addressed dll hash is identical with/without the
+      Phase 5 changes (same codegen input). Local-env specific (LLVM 22.1.5 Windows,
+      opt-level 3); master CI Windows -jit is green (different LLVM, plus the
+      `|| --isolated-mode` fallback built into the CI step). Local full-tree signal
+      obtained with `--exclude table_operations --exclude test_linq_table_source` (the
+      second file is just the next deliberate-panic test the abort moves to); CI is the
+      authoritative gate.
+- [x] **Third trap, found only by the full sweep (order-dependent linq failures):** a
+      module that is macro-CALLED but not itself a macro module (daslib/linq_fold_decs —
+      plain functions invoked from linq_boost's fold macros) has `needMacroModule ==
+      false`, so it DOES lower under the jit trigger — and its lowered quotes then
+      evaluate at macro-apply time on the calling macro module's macro-context stack:
+      `stack overflow while calling @linq_fold_decs::\`quote\`lowered\`17` → 31206 → the
+      consumer file fails to compile (tests/linq/test_linq_fold_terminal_select +
+      test_linq_from_decs). Order-dependent because the overflow margin depends on
+      module-cache state at compile time — single-folder runs passed, full-tree
+      root-form runs failed deterministically (bisect: my code at master folder set
+      still failed; zero-predecessor root form still failed → not contamination at all,
+      a per-file stack margin). FIX: `Program::makeMacroModule` sizes the MACRO context
+      to `max(getContextStackSize(), 1MB)` when `aot_macros || jit_enabled` — the
+      headroom lands exactly where lowered quotes evaluate at macro-apply time, for any
+      embedder, with no driver changes. First attempt was a `policies.stack = 1MB` bump
+      in the three jit drivers (main.cpp / dastest / jit.exe), mirroring the
+      `-aot-macros` flow — REGRESSION: `policies.stack` also sizes the PRODUCED
+      program's runtime context, so cross-compiled wasm executables tried to carve 1MB
+      out of wasm linear memory and trapped at runtime (CI `wasm_cross`, f2s.wasm
+      "thrown Wasm exception"). Reverted; the `-aot-macros` global bump keeps its
+      historical behavior. Durable fix remains chunking the lowered construction into
+      bounded frames (Phase 6 candidate) — that would let macro contexts drop the
+      special-casing entirely.
 - [ ] Decide + implement JIT-of-macro-contexts: lift the `is_compiling_macros()` skip in
       llvm_macro.das behind a policy. This is the compile-time payoff twin of AOT'd
       macro modules. Separate PR; needs its own bake time.
-- [ ] `LLVM_JIT_CODEGEN_VERSION` bump only if emitters change (diagnostic-only changes
-      don't).
+- [x] `LLVM_JIT_CODEGEN_VERSION` bump NOT needed: emitters unchanged (gate + collection
+      only), and the dll cache hash folds per-function AOT hashes, so lowered ASTs and
+      changed collection sets self-invalidate. (CLAUDE.md pointer fixed: the constant
+      lives in llvm_jit_run.das, not llvm_macro.das.)
 
 ## Phase 6 — extra coverage + payoff measurement
 

daslib/quote.das

@@ -483,7 +483,10 @@ class QuoteConverter : AstVisitor {
         }
         var fn = new Function(at = expr.at, atDecl = expr.at, name := fname,
             flags = FunctionFlags.generated | FunctionFlags.privateFunction)
-        fn.result = new TypeDecl(baseType = Type.autoinfer)
+        // preserve the raw quote's static type (Expression?) — autoinfer would leak the
+        // concrete node type (e.g. ExprConstInt?) and break consumers that need type
+        // identity, e.g. `cond ? clone_expression(x) : quote(0)`
+        fn.result = expr._type != null ? clone_type(expr._type) : new TypeDecl(baseType = Type.autoinfer)
         var blk = new ExprBlock(at = expr.at)
         emplace_new(blk.list) <| new ExprReturn(at = expr.at, subexpr = conv)
         fn.body = blk
@@ -497,9 +500,16 @@ class QuotePass : AstPassMacro {
     //! Pass macro that processes quoted AST expressions.
     def override apply(prog : ProgramPtr; mod : Module?) : bool {
         // Unwrapping ExprQuote is slow and inefficient, do it only if necessary.
-        // Triggered by the aot_macros policy (`daslang -aot -aot-macros`) or per-module
-        // `options aot_macros` (self-contained tests, interpreted A/B).
-        let lower = compiling_program().policies.aot_macros || (prog._options |> find_arg("aot_macros") ?as tBool ?? false)
+        // Triggered by the aot_macros policy (`daslang -aot -aot-macros`), the jit_enabled
+        // policy (lowered quotes codegen natively, raw ones can't), or per-module
+        // `options aot_macros` (self-contained tests). Macro-module programs are excluded
+        // from the jit trigger: macro contexts are never JITted (see llvm_macro's
+        // is_compiling_macros skip), and the lowered construction frames can overflow the
+        // macro-context stack at apply time. Raw quotes reaching JIT codegen from such
+        // modules fall back per-function via DisableJitVisitor.
+        let lower = (compiling_program().policies.aot_macros
+            || (compiling_program().policies.jit_enabled && !prog.flags.needMacroModule)
+            || (prog._options |> find_arg("aot_macros") ?as tBool ?? false))
         if (!lower) return false // nothing to do
         var astVisitor = new QuoteConverter(mod = compiling_module())
         make_visitor(*astVisitor) $(astVisitorAdapter) {

modules/dasLLVM/daslib/llvm_jit.das

@@ -3391,7 +3391,7 @@ class public LlvmJitVisitor : AstVisitor {
 
 // ExprQuote
     def override preVisitExprQuote(expr : ExprQuote?) : void {
-        unsupported(expr, "quote( ) is not jit-able without aot_macros lowering (daslib/quote)")
+        unsupported(expr, "raw quote( ) is not jit-able — daslib/quote lowering (jit_enabled / aot_macros gates) did not run for this module")
     }
 
 // ExprIfThenElse
@@ -7645,6 +7645,14 @@ class public DisableJitVisitor : AstVisitor {
         }
     }
 
+    def override preVisitExprQuote(expr : ExprQuote?) : void {
+        // jit_enabled programs get quotes lowered by daslib/quote's QuotePass before we
+        // ever run; a raw one means lowering didn't fire for the defining module (macro
+        // modules stay raw by design) — skip the function instead of failing the file
+        to_log(LOG_WARNING, "LLVM JIT: raw quote( ) — daslib/quote lowering (jit_enabled / aot_macros gates) did not run for this module at {expr.at}, falling back to interpreter.\n")
+        disable = true
+    }
+
     def override preVisitExprForBody(expr : ExprFor?) : void {
         for (_, ssrc in expr.iteratorVariables, expr.sources) {
             assume ssrc_t = ssrc._type

skills/writing_tests.md

@@ -21,7 +21,7 @@ directory filter below instead.)
 
 `tests/.das_test` is a daslang script dastest compiles per run; its `can_visit_folder`
 pinvoke gates whole directories per mode — e.g. `no_aot/`, `ast/`, `ast_match/` are
-skipped under `--use-aot` / `-jit`, module dirs (dasSQLITE, dasPUGIXML…) skip when the
+skipped under `--use-aot`, module dirs (dasSQLITE, dasPUGIXML…) skip when the
 module isn't built in. **The filter is looked up only at the `--test` ROOT path** —
 `--test tests` finds and applies it, but `--test tests/flatten` looks for
 `tests/flatten/.das_test` (absent) and walks into `no_aot/` unfiltered, producing
@@ -33,14 +33,20 @@ false `error[50101]` / JIT failures. For AOT/JIT validation, sweep `--test tests
 `tests/.das_test` defines `can_visit_folder(folder_name, result)` — dastest consults it
 per subfolder during file collection (only for the `.das_test` at the `--test <root>`
 argument; directly naming a child folder bypasses it). It gates folders on module
-availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv for `-jit` /
-`--use-aot` (e.g. `ast`, `ast_match`, `no_aot`, `gc`, `quote` skip under `-jit` — runtime
-quote/qmacro the JIT can't codegen). Two traps: a whole-folder JIT/AOT failure usually
-means a missing entry here, NOT a per-file fix; and the `jit_cache_all_tests` prewarm
-target (utils/CMakeLists.txt) does NOT consult it — its `--exclude` list mirrors the
-`-jit` skips manually and must be updated in the same change. Per-function `[test, no_jit]`
-(tests/template/test_push_block_list.das) is the finer-grained alternative when only some
-tests in a kept folder can't JIT.
+availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv — `--use-aot`
+skips `ast`, `ast_match`, `no_aot`; `-jit` skips only `gc` (heap_collect can't see heap
+pointers whose only reference is a local in a jitted frame — native-stack locals are
+invisible to the collector, so GC-semantics tests are interp-only; the other former `-jit`
+skips were lifted once `jit_enabled` started triggering daslib/quote lowering). Two traps:
+a whole-folder JIT/AOT failure usually means a missing entry here, NOT a per-file fix; and
+the `jit_cache_all_tests` prewarm target (utils/CMakeLists.txt) does NOT consult it — its
+`--exclude` list mirrors the skips manually and must be updated in the same change.
+Per-function `[no_jit]` is the finer-grained alternative when only some functions in a
+kept folder can't JIT — put it on the function whose CODE diverges under JIT, not just the
+`[test]` wrapper (JITted callees replace their SimNode bodies, so an interpreted wrapper
+still calls jitted workers). Beware Release-blind divergence: memory bugs (double-free,
+reuse-after-collect) only trip the Debug memory_model.h assert, so a green local Release
+sweep does NOT prove a lifted skip is sound — Debug CI is the oracle.
 
 ## Test file structure
 

src/ast/ast_infer_type.cpp

@@ -1438,9 +1438,12 @@ namespace das {
         expr->type->firstType = new TypeDecl(Type::tHandle);
         expr->type->firstType->annotation = (TypeAnnotation *)Module::require("ast_core")->findAnnotation("Expression");
         // mark quote as noAot, unless daslib/quote lowering will replace it
-        // (aot_macros policy or per-module `options aot_macros` — same gate as QuotePass)
+        // (aot_macros or jit_enabled policy, or per-module `options aot_macros` — same gate
+        // as QuotePass, including its macro-module exclusion for the jit trigger)
         if (func) {
-            if (!program->policies.aot_macros && !program->options.getBoolOption("aot_macros", false)) {
+            if (!program->policies.aot_macros
+                    && !(program->policies.jit_enabled && !program->needMacroModule)
+                    && !program->options.getBoolOption("aot_macros", false)) {
                 func->noAot = true;
             }
         }

src/ast/ast_simulate.cpp

@@ -3532,7 +3532,16 @@ namespace das
 
     void Program::makeMacroModule ( TextWriter & logs ) {
         isCompilingMacros = true;
-        thisModule->macroContext = get_context(getContextStackSize());
+        int macroStackSize = getContextStackSize();
+        if ( policies.aot_macros || policies.jit_enabled || options.getBoolOption("aot_macros", false) ) {
+            // quote lowering (daslib/quote) is active (same triggers as its QuotePass gate,
+            // including the per-module option): a lowered quote evaluates one large
+            // construction frame per quote, and macro-called functions evaluate theirs on
+            // THIS context's stack at macro-apply time. Size only the macro context — a
+            // global policies.stack bump would leak into produced exe/wasm runtime stacks.
+            macroStackSize = das::max(macroStackSize, 1 * 1024 * 1024);
+        }
+        thisModule->macroContext = get_context(macroStackSize);
         thisModule->macroContext->category = uint32_t(das::ContextCategory::macro_context);
         auto oldAot = policies.aot;
         auto oldHeap = policies.persistent_heap;