From 19322eced3c84e9b109677f47a860c1058b40771 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Fri, 12 Jun 2026 01:07:19 -0700 Subject: [PATCH 1/5] =?UTF-8?q?quote=20lowering=20phase=205:=20JIT=20lane?= =?UTF-8?q?=20=E2=80=94=20jit=5Fenabled=20triggers=20QuotePass,=20lift=20-?= =?UTF-8?q?jit=20skips?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit QuotePass now lowers quotes when policies.jit_enabled is set (daslang -jit, dastest -jit, jit.exe), so quote/qmacro code JITs natively instead of panicking the whole file. Three bugs surfaced on first contact: * Wrapper return type: the generated `quote`lowered`N function used autoinfer, leaking the concrete node type (quote(0) -> ExprConstInt? instead of Expression?), which broke `cond ? clone_expression(x) : quote(0)` (30411 — cond arms must match exactly). Now fn.result = clone_type(expr._type) — identity with the raw quote's static type by construction. * Macro-module programs are excluded from the jit trigger: macro contexts are never JITted (llvm_macro's is_compiling_macros skip) and the lowered construction frames overflowed the macro-context stack at apply time. Infer-time predicate is prog.flags.needMacroModule — is_compiling_macros() is only true during makeMacroModule's simulate, not when QuotePass runs. ast_infer_type.cpp's noAot-skip mirrors the same predicate. Raw quotes that still reach JIT (runtime-used functions from macro modules) now fall back per-function via DisableJitVisitor's new preVisitExprQuote (warn + disable) instead of failing the file. * The jit lane gets the same 1MB stack the -aot-macros flow always shipped with (daslang -jit in main.cpp, dastest -jit in suite.das and the ser path, jit.exe). A module that is macro-CALLED but not itself a macro module (daslib/linq_fold_decs — plain functions invoked from linq_boost's fold macros) is not excluded by needMacroModule, so its lowered quotes evaluate at macro-apply time on the calling macro module's context stack; without the bump that overflowed ("stack overflow while calling @linq_fold_decs::`quote`lowered`17" -> 31206 -> consumer fails to compile), order-dependently — the margin varies with module-cache state, so single-folder runs passed while full-tree root-form runs failed deterministically. Macro-module programs inherit consumer policies, so macro contexts get the headroom too. Coverage lifted: the -jit folder skips in tests/.das_test (quote, ast, ast_match, no_aot, gc) and the matching jit_cache_all_tests excludes are gone; tests/template/test_push_block_list [no_jit] markers removed (qmacro_block JITs now). Folder runs: quote 20/20, ast 1/1, ast_match 380/380, flatten/no_aot 14/14, gc 48/48, template 10/10. Not quote-related, found while lifting the gc skip: LLVM JIT does not implement force_escape_free / force_allocate_on_stack (heap grows where interp stays flat; scope-exit free aborts on JIT-allocated owning nodes under persistent_heap; nested new-in-make-struct field arrives null). Scoped [no_jit] markers on the worker functions in the three tests/gc files keep the rest of the folder JIT-covered; the JIT gap is a separate work item. No LLVM_JIT_CODEGEN_VERSION bump: emitters are unchanged (gate + collection only) and the dll cache hash folds per-function AOT hashes, so lowered ASTs self-invalidate. CLAUDE.md's pointer to the constant fixed (it lives in llvm_jit_run.das, not llvm_macro.das). Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 2 +- QUOTE_LOWERING.md | 69 +++++++++++++++++++++++-- daslib/quote.das | 18 +++++-- dastest/dastest.das | 5 ++ dastest/suite.das | 5 +- modules/dasLLVM/daslib/llvm_jit.das | 8 +++ skills/writing_tests.md | 18 ++++--- src/ast/ast_infer_type.cpp | 7 ++- tests/.das_test | 38 ++------------ tests/gc/test_gc_allocate_on_stack.das | 5 ++ tests/gc/test_gc_escape_free.das | 3 ++ tests/gc/test_gc_escape_free_frees.das | 11 ++++ tests/template/test_push_block_list.das | 16 ++---- utils/CMakeLists.txt | 10 ++-- utils/daScript/main.cpp | 7 ++- utils/jit/main.das | 8 +-- 16 files changed, 153 insertions(+), 77 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ad985b387..833faa19f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -310,7 +310,7 @@ Most layout is obvious from `ls`. Non-obvious ones worth knowing: - `utils/detect-dupe/` (in-repo dupe finder) and `utils/find-dupe/` (Claude-based judge that needs `daspkg install --root utils/find-dupe` + `ANTHROPIC_API_KEY`) — both also exposed as MCP tools - `utils/mcp/`, `utils/daslang-live/`, `utils/daspkg/` — in-tree tools (most also have skills under `skills/`) - `tutorials/language/` (language tour) vs `tutorials//` (per-area) — never put tutorials in `modules//tutorial/` -- **Array/dim/vector indexing lives across 5 tiers** — bug fixes (bounds checks, neg-index detection, width-aware bounds) usually need parallel edits in all of them: AOT C++ ([include/daScript/simulate/aot.h](include/daScript/simulate/aot.h)), interpreter non-fused ([include/daScript/simulate/runtime_array.h](include/daScript/simulate/runtime_array.h) + [include/daScript/simulate/simulate_nodes.h](include/daScript/simulate/simulate_nodes.h)), interpreter fused ([src/simulate/simulate_fusion_at_array.cpp](src/simulate/simulate_fusion_at_array.cpp) + [src/simulate/simulate_fusion_at.cpp](src/simulate/simulate_fusion_at.cpp)), JIT ([modules/dasLLVM/daslib/llvm_jit.das](modules/dasLLVM/daslib/llvm_jit.das)), and AST const-fold ([src/ast/ast_simulate.cpp](src/ast/ast_simulate.cpp)). Debug builds bypass the fused permutations — a fix that lands only in the fused path passes Release CI and trips Debug-ARM. Bump `LLVM_JIT_CODEGEN_VERSION` in `modules/dasLLVM/daslib/llvm_macro.das` after JIT changes to invalidate cached `.dll`s +- **Array/dim/vector indexing lives across 5 tiers** — bug fixes (bounds checks, neg-index detection, width-aware bounds) usually need parallel edits in all of them: AOT C++ ([include/daScript/simulate/aot.h](include/daScript/simulate/aot.h)), interpreter non-fused ([include/daScript/simulate/runtime_array.h](include/daScript/simulate/runtime_array.h) + [include/daScript/simulate/simulate_nodes.h](include/daScript/simulate/simulate_nodes.h)), interpreter fused ([src/simulate/simulate_fusion_at_array.cpp](src/simulate/simulate_fusion_at_array.cpp) + [src/simulate/simulate_fusion_at.cpp](src/simulate/simulate_fusion_at.cpp)), JIT ([modules/dasLLVM/daslib/llvm_jit.das](modules/dasLLVM/daslib/llvm_jit.das)), and AST const-fold ([src/ast/ast_simulate.cpp](src/ast/ast_simulate.cpp)). Debug builds bypass the fused permutations — a fix that lands only in the fused path passes Release CI and trips Debug-ARM. Bump `LLVM_JIT_CODEGEN_VERSION` in `modules/dasLLVM/daslib/llvm_jit_run.das` after JIT emitter/ABI changes to invalidate cached `.dll`s (the cache hash also folds per-function AOT hashes, so AST-level changes self-invalidate without a bump) ## MCP Server (AI Tool Integration) diff --git a/QUOTE_LOWERING.md b/QUOTE_LOWERING.md index 5249dd994..5651b20c5 100644 --- a/QUOTE_LOWERING.md +++ b/QUOTE_LOWERING.md @@ -247,14 +247,73 @@ JIT work in scope before then is the Phase 1 diagnostic (clean `unsupported` mes replacing the misleading "Internal jit error" panic). The QuotePass gate stays `aot_macros`-only until this phase; extending it to `jit_enabled` lands here. -- [ ] Extend QuotePass gate to `jit_enabled`. -- [ ] `tests/jit_tests/` coverage for quote under `options jit` (lowered path through - LlvmJitVisitor — make-struct of handled types, `to_array_move`, by-name lookups). +- [x] Extend QuotePass gate to `jit_enabled` (2026-06-12). Two traps found on first contact, + both via `daslib/typemacro_boost`: + 1. **Wrapper return type must be the raw quote's type.** `fn.result = autoinfer` leaked + the concrete node type (`quote(0)` → `ExprConstInt?` instead of `Expression?`), + breaking `cond ? clone_expression(x) : quote(0)` (30411, cond arms must match + exactly). Fix: `fn.result = clone_type(expr._type)` — identity by construction. + 2. **Macro-module programs must not lower under the jit trigger.** Macro contexts are + never JITted (llvm_macro's `is_compiling_macros()` skip), and the lowered + construction frames overflowed the macro-context stack at apply time + (`stack overflow while calling @typemacro_boost::\`quote\`lowered\`28`). Predicate + gotcha: `is_compiling_macros()` reads `Program::isCompilingMacros`, which is only + true during `makeMacroModule`'s simulate — at infer time (when QuotePass runs) use + `prog.flags.needMacroModule` instead (set at annotation-apply, pre-infer). The + infer-time noAot-skip in ast_infer_type.cpp mirrors the same predicate. Raw quotes + reaching JIT from such modules fall back per-function via the new + DisableJitVisitor `preVisitExprQuote` (warn + disable, replacing whole-file panic). +- [x] Coverage instead of new jit_tests: lifted the `-jit` folder skips (quote, ast, + ast_match, no_aot, gc) in tests/.das_test + the matching `--exclude`s on + jit_cache_all_tests, and un-marked `[no_jit]` in tests/template/test_push_block_list + (its stated reason — JIT can't lower ExprQuote — is gone; qmacro_block now JITs). + Folder results: quote 20/20, ast 1/1, ast_match 380/380 (2 graceful per-function + fallbacks in daslib/ast_match — macro module, stays raw by design), flatten/no_aot + 14/14, gc 48/48, template 10/10. +- [x] NOT quote-related, surfaced while lifting the gc skip: **LLVM JIT does not implement + `force_escape_free` / `force_allocate_on_stack`** — three symptoms in tests/gc + (heap grows where interp stays flat; scope-exit free aborts "not a chunk pointer" + on JIT-allocated owning nodes under persistent_heap; nested `new Inner` make-struct + field arrives null). Scoped per-function `[no_jit]` markers on the workers in + test_gc_allocate_on_stack / test_gc_escape_free_frees / test_gc_escape_free keep the + rest of the folder JIT-covered. Upstream fix is its own work item. +- [x] Local full-tree `-jit` sweep caveat (NOT this PR's doing): the in-process sweep + aborts (exit 127, no diagnostic) at tests/language/table_operations.das + `ta_test_lock_panic` — a deliberate table-lock panic inside jitted code fails to + unwind to `recover` and kills the process. **Pre-existing**: reproduces with master + daslib state, and the content-addressed dll hash is identical with/without the + Phase 5 changes (same codegen input). Local-env specific (LLVM 22.1.5 Windows, + opt-level 3); master CI Windows -jit is green (different LLVM, plus the + `|| --isolated-mode` fallback built into the CI step). Local full-tree signal + obtained with `--exclude table_operations --exclude test_linq_table_source` (the + second file is just the next deliberate-panic test the abort moves to); CI is the + authoritative gate. +- [x] **Third trap, found only by the full sweep (order-dependent linq failures):** a + module that is macro-CALLED but not itself a macro module (daslib/linq_fold_decs — + plain functions invoked from linq_boost's fold macros) has `needMacroModule == + false`, so it DOES lower under the jit trigger — and its lowered quotes then + evaluate at macro-apply time on the calling macro module's macro-context stack: + `stack overflow while calling @linq_fold_decs::\`quote\`lowered\`17` → 31206 → the + consumer file fails to compile (tests/linq/test_linq_fold_terminal_select + + test_linq_from_decs). Order-dependent because the overflow margin depends on + module-cache state at compile time — single-folder runs passed, full-tree + root-form runs failed deterministically (bisect: my code at master folder set + still failed; zero-predecessor root form still failed → not contamination at all, + a per-file stack margin). FIX (matches the `-aot-macros` design, which always + shipped with a 1MB stack): the jit lane applies the same bump — `daslang -jit` + (main.cpp), dastest under `-jit` (suite.das + dastest.das ser path), jit.exe + (utils/jit/main.das) — and macro-module programs inherit consumer policies, so + macro contexts get the headroom too. Known sharp edge for embedders: enabling + `policies.jit_enabled` without a stack bump can still overflow on a big quote + evaluated at macro time — durable fix is chunking the lowered construction into + bounded frames (Phase 6 candidate). - [ ] Decide + implement JIT-of-macro-contexts: lift the `is_compiling_macros()` skip in llvm_macro.das behind a policy. This is the compile-time payoff twin of AOT'd macro modules. Separate PR; needs its own bake time. -- [ ] `LLVM_JIT_CODEGEN_VERSION` bump only if emitters change (diagnostic-only changes - don't). +- [x] `LLVM_JIT_CODEGEN_VERSION` bump NOT needed: emitters unchanged (gate + collection + only), and the dll cache hash folds per-function AOT hashes, so lowered ASTs and + changed collection sets self-invalidate. (CLAUDE.md pointer fixed: the constant + lives in llvm_jit_run.das, not llvm_macro.das.) ## Phase 6 — extra coverage + payoff measurement diff --git a/daslib/quote.das b/daslib/quote.das index e9a0739c6..2b3c63b64 100644 --- a/daslib/quote.das +++ b/daslib/quote.das @@ -483,7 +483,10 @@ class QuoteConverter : AstVisitor { } var fn = new Function(at = expr.at, atDecl = expr.at, name := fname, flags = FunctionFlags.generated | FunctionFlags.privateFunction) - fn.result = new TypeDecl(baseType = Type.autoinfer) + // preserve the raw quote's static type (Expression?) — autoinfer would leak the + // concrete node type (e.g. ExprConstInt?) and break consumers that need type + // identity, e.g. `cond ? clone_expression(x) : quote(0)` + fn.result = expr._type != null ? clone_type(expr._type) : new TypeDecl(baseType = Type.autoinfer) var blk = new ExprBlock(at = expr.at) emplace_new(blk.list) <| new ExprReturn(at = expr.at, subexpr = conv) fn.body = blk @@ -497,9 +500,16 @@ class QuotePass : AstPassMacro { //! Pass macro that processes quoted AST expressions. def override apply(prog : ProgramPtr; mod : Module?) : bool { // Unwrapping ExprQuote is slow and inefficient, do it only if necessary. - // Triggered by the aot_macros policy (`daslang -aot -aot-macros`) or per-module - // `options aot_macros` (self-contained tests, interpreted A/B). - let lower = compiling_program().policies.aot_macros || (prog._options |> find_arg("aot_macros") ?as tBool ?? false) + // Triggered by the aot_macros policy (`daslang -aot -aot-macros`), the jit_enabled + // policy (lowered quotes codegen natively, raw ones can't), or per-module + // `options aot_macros` (self-contained tests). Macro-module programs are excluded + // from the jit trigger: macro contexts are never JITted (see llvm_macro's + // is_compiling_macros skip), and the lowered construction frames can overflow the + // macro-context stack at apply time. Raw quotes reaching JIT codegen from such + // modules fall back per-function via DisableJitVisitor. + let lower = (compiling_program().policies.aot_macros + || (compiling_program().policies.jit_enabled && !prog.flags.needMacroModule) + || (prog._options |> find_arg("aot_macros") ?as tBool ?? false)) if (!lower) return false // nothing to do var astVisitor = new QuoteConverter(mod = compiling_module()) make_visitor(*astVisitor) $(astVisitorAdapter) { diff --git a/dastest/dastest.das b/dastest/dastest.das index 49c77004d..086ac5831 100644 --- a/dastest/dastest.das +++ b/dastest/dastest.das @@ -377,6 +377,11 @@ def serialize_path(ctx : SuiteCtx, files : array, out_file : string) { cop.fail_on_no_aot = ctx.use_aot cop.threadlock_context = true cop.jit_enabled = jit_enabled() + if (jit_enabled()) { + // jit_enabled triggers quote lowering; lowered construction frames + // need the same stack bump suite.das applies (see suite.das) + cop.stack = 1024u * 1024u + } compile_file(file, access, unsafe(addr(mg)), cop) $(ok, program, output) { if (!ok) { log::error("Failed to compile {file}\n{output}") diff --git a/dastest/suite.das b/dastest/suite.das index 1b8dbe6f5..079e9ae79 100644 --- a/dastest/suite.das +++ b/dastest/suite.das @@ -161,9 +161,10 @@ def test_file(file_name : string; var ctx : SuiteCtx; var file_ctx : FileCtx) : cop.aot = ctx.use_aot cop.fail_on_no_aot = ctx.use_aot cop.aot_macros = ctx.aot_macros - if (ctx.aot_macros) { + if (ctx.aot_macros || jit_enabled()) { // a lowered quote evaluates one large make-struct construction frame - // (same bump the `daslang -aot -aot-macros` flow applies) + // (same bump the `daslang -aot -aot-macros` flow applies); jit_enabled + // triggers the same lowering, and macro contexts inherit this stack size cop.stack = 1024u * 1024u } cop.threadlock_context = true diff --git a/modules/dasLLVM/daslib/llvm_jit.das b/modules/dasLLVM/daslib/llvm_jit.das index 4b9d13254..952f05e8f 100644 --- a/modules/dasLLVM/daslib/llvm_jit.das +++ b/modules/dasLLVM/daslib/llvm_jit.das @@ -7645,6 +7645,14 @@ class public DisableJitVisitor : AstVisitor { } } + def override preVisitExprQuote(expr : ExprQuote?) : void { + // jit_enabled programs get quotes lowered by daslib/quote's QuotePass before we + // ever run; a raw one means lowering didn't fire (e.g. a context compiled without + // the policy) — skip the function instead of failing the whole file's codegen + to_log(LOG_WARNING, "LLVM JIT: quote( ) without aot_macros lowering at {expr.at}, falling back to interpreter.\n") + disable = true + } + def override preVisitExprForBody(expr : ExprFor?) : void { for (_, ssrc in expr.iteratorVariables, expr.sources) { assume ssrc_t = ssrc._type diff --git a/skills/writing_tests.md b/skills/writing_tests.md index 45a2c32e2..9b53ed68d 100644 --- a/skills/writing_tests.md +++ b/skills/writing_tests.md @@ -21,7 +21,7 @@ directory filter below instead.) `tests/.das_test` is a daslang script dastest compiles per run; its `can_visit_folder` pinvoke gates whole directories per mode — e.g. `no_aot/`, `ast/`, `ast_match/` are -skipped under `--use-aot` / `-jit`, module dirs (dasSQLITE, dasPUGIXML…) skip when the +skipped under `--use-aot`, module dirs (dasSQLITE, dasPUGIXML…) skip when the module isn't built in. **The filter is looked up only at the `--test` ROOT path** — `--test tests` finds and applies it, but `--test tests/flatten` looks for `tests/flatten/.das_test` (absent) and walks into `no_aot/` unfiltered, producing @@ -34,13 +34,15 @@ false `error[50101]` / JIT failures. For AOT/JIT validation, sweep `--test tests per subfolder during file collection (only for the `.das_test` at the `--test ` argument; directly naming a child folder bypasses it). It gates folders on module availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv for `-jit` / -`--use-aot` (e.g. `ast`, `ast_match`, `no_aot`, `gc`, `quote` skip under `-jit` — runtime -quote/qmacro the JIT can't codegen). Two traps: a whole-folder JIT/AOT failure usually -means a missing entry here, NOT a per-file fix; and the `jit_cache_all_tests` prewarm -target (utils/CMakeLists.txt) does NOT consult it — its `--exclude` list mirrors the -`-jit` skips manually and must be updated in the same change. Per-function `[test, no_jit]` -(tests/template/test_push_block_list.das) is the finer-grained alternative when only some -tests in a kept folder can't JIT. +`--use-aot` (e.g. `ast`, `ast_match`, `no_aot` skip under `--use-aot`; the former `-jit` +skips were lifted once `jit_enabled` started triggering daslib/quote lowering). Two traps: +a whole-folder JIT/AOT failure usually means a missing entry here, NOT a per-file fix; and +the `jit_cache_all_tests` prewarm target (utils/CMakeLists.txt) does NOT consult it — its +`--exclude` list mirrors the skips manually and must be updated in the same change. +Per-function `[no_jit]` is the finer-grained alternative when only some functions in a +kept folder can't JIT — put it on the function whose CODE diverges under JIT, not just the +`[test]` wrapper (JITted callees replace their SimNode bodies, so an interpreted wrapper +still calls jitted workers); see the `force_escape_free` workers in tests/gc. ## Test file structure diff --git a/src/ast/ast_infer_type.cpp b/src/ast/ast_infer_type.cpp index fa25ad6b1..905c0fe91 100644 --- a/src/ast/ast_infer_type.cpp +++ b/src/ast/ast_infer_type.cpp @@ -1438,9 +1438,12 @@ namespace das { expr->type->firstType = new TypeDecl(Type::tHandle); expr->type->firstType->annotation = (TypeAnnotation *)Module::require("ast_core")->findAnnotation("Expression"); // mark quote as noAot, unless daslib/quote lowering will replace it - // (aot_macros policy or per-module `options aot_macros` — same gate as QuotePass) + // (aot_macros or jit_enabled policy, or per-module `options aot_macros` — same gate + // as QuotePass, including its macro-module exclusion for the jit trigger) if (func) { - if (!program->policies.aot_macros && !program->options.getBoolOption("aot_macros", false)) { + if (!program->policies.aot_macros + && !(program->policies.jit_enabled && !program->needMacroModule) + && !program->options.getBoolOption("aot_macros", false)) { func->noAot = true; } } diff --git a/tests/.das_test b/tests/.das_test index e8ecd3772..af409948f 100644 --- a/tests/.das_test +++ b/tests/.das_test @@ -39,45 +39,15 @@ def can_visit_folder(folder_name : string; var result : bool?) { } } // A meta-test that requires the compiler frontend at runtime (e.g. daslib/flatten -> - // macro_boost) can't AOT-link in the full-tree batch, and one that runs qmacro at runtime - // (flatten_optimize re-entry) carries quote nodes LLVM JIT can't codegen — interp only. - if (folder_name == "no_aot") { + // macro_boost) can't AOT-link in the full-tree batch. Under -jit it runs fine: quote( ) + // nodes are lowered by daslib/quote when jit_enabled is set. + if (folder_name == "no_aot" || folder_name == "ast_match" || folder_name == "ast") { let args <- get_command_line_arguments() for (arg in args) { - if (arg == "--use-aot" || arg == "-jit") { - *result = false - return - } - } - } - if (folder_name == "ast_match" || folder_name == "ast") { - let args <- get_command_line_arguments() - for (arg in args) { - if (arg == "--use-aot" || arg == "-jit") { - *result = false - return - } - } - } - if (folder_name == "gc") { - let args <- get_command_line_arguments() - for (arg in args) { - if (arg == "-jit") { - *result = false - return - } - } - } - // The A/B baseline fixture keeps raw quote( ) nodes LLVM JIT can't codegen. - // AOT stays on (the lowered twins are the point). Lift with the JIT lowering lane. - if (folder_name == "quote") { - let args <- get_command_line_arguments() - for (arg in args) { - if (arg == "-jit") { + if (arg == "--use-aot") { *result = false return } } } - } diff --git a/tests/gc/test_gc_allocate_on_stack.das b/tests/gc/test_gc_allocate_on_stack.das index 764778528..480460053 100644 --- a/tests/gc/test_gc_allocate_on_stack.das +++ b/tests/gc/test_gc_allocate_on_stack.das @@ -55,6 +55,10 @@ def sum_plain_new() : int { } // `new Node(x = .., y = ..)` - ExprAscend over make-local -> built directly in the frame +// no_jit: LLVM JIT doesn't implement force_allocate_on_stack for the ascend lowering — +// it heap-allocates (failing the flat-heap assert), and the owning variant's scope-exit +// free then aborts on the mismatched pointer +[no_jit] def sum_ascend_new() : int { var total = 0 for (i in range(N)) { @@ -65,6 +69,7 @@ def sum_ascend_new() : int { } // shell on the stack; the owned array is still on the heap and is freed at scope exit +[no_jit] // same force_allocate_on_stack gap as sum_ascend_new — crashes under JIT def sum_owning() : int { var total = 0 for (i in range(N)) { diff --git a/tests/gc/test_gc_escape_free.das b/tests/gc/test_gc_escape_free.das index ae0894ecd..2f1165501 100644 --- a/tests/gc/test_gc_escape_free.das +++ b/tests/gc/test_gc_escape_free.das @@ -24,6 +24,9 @@ struct Node { struct Inner { vals : array } struct Outer { tag : int; child : Inner ? } +// no_jit: LLVM JIT doesn't implement force_escape_free — here the nested `new Inner` +// field comes back null (sibling files document the heap-grow / bad-free symptoms) +[no_jit] def churn_nested() : int { var acc = 0 for (i in range(500)) { diff --git a/tests/gc/test_gc_escape_free_frees.das b/tests/gc/test_gc_escape_free_frees.das index 25a834487..1c7197472 100644 --- a/tests/gc/test_gc_escape_free_frees.das +++ b/tests/gc/test_gc_escape_free_frees.das @@ -34,7 +34,13 @@ struct OwningNode { var g_list : array +// All workers carry [no_jit]: LLVM JIT doesn't implement force_escape_free — loops grow +// the heap (failing the flat-heap asserts) and the owning variant's scope-exit free aborts +// on the mismatched pointer ("not a chunk pointer"). The control is marked too so the +// grows/stays-flat comparison runs in one engine. + // every use of `p` is a field access -> p does not escape -> freed at scope exit +[no_jit] def sum_nonescaping() : int { var total = 0 for (i in range(N)) { @@ -45,6 +51,7 @@ def sum_nonescaping() : int { } // multi-variable let: each variable gets its OWN fresh allocation, so both are freed (no alias) +[no_jit] def sum_multi() : int { var total = 0 for (i in range(N)) { @@ -55,6 +62,7 @@ def sum_multi() : int { } // the free still runs in a function with try/recover (on the normal path) +[no_jit] def sum_with_try() : int { var total = 0 for (i in range(N)) { @@ -65,6 +73,7 @@ def sum_with_try() : int { } // passed only to a pure built-in (test_escape_pure: no side effects, scalar return) -> freed +[no_jit] def sum_pure_builtin() : int { var total = 0 for (i in range(N)) { @@ -75,6 +84,7 @@ def sum_pure_builtin() : int { } // non-escaping owner of a heap field; the raw collect frees the owned array too +[no_jit] def sum_owning() : int { var total = 0 for (i in range(N)) { @@ -84,6 +94,7 @@ def sum_owning() : int { return total } +[no_jit] def sum_escaping() : int { var total = 0 g_list |> reserve(length(g_list) + N) diff --git a/tests/template/test_push_block_list.das b/tests/template/test_push_block_list.das index a94c7cfde..a2515dbca 100644 --- a/tests/template/test_push_block_list.das +++ b/tests/template/test_push_block_list.das @@ -11,16 +11,10 @@ require dastest/testing_boost public // result into an ``array``, cloning each. Use in place of a // cluster of ``stmts |> push <| qmacro_expr() { single_stmt }`` calls when // emitting multi-statement snippets. -// -// All tests carry ``[no_jit]`` because ``qmacro_block`` codegens to an -// ``ExprQuote`` wrapping a multi-statement block, which the LLVM JIT does -// not lower (interp + AOT lanes handle it fine). Single-expression -// ``qmacro(...)`` calls in the peer ``tests/ast_match/test_peel_lambda_*`` -// tests DO JIT — only the block form is affected. // ── Multi-statement append ────────────────────────────────────────────── -[test, no_jit] +[test] def test_appends_all_statements(t : T?) { ast_gc_guard() { var stmts : array @@ -36,7 +30,7 @@ def test_appends_all_statements(t : T?) { // ── Single-statement (degenerate but legal) ──────────────────────────── -[test, no_jit] +[test] def test_single_statement(t : T?) { ast_gc_guard() { var stmts : array @@ -50,7 +44,7 @@ def test_single_statement(t : T?) { // ── Composes with prior + subsequent pushes ──────────────────────────── -[test, no_jit] +[test] def test_interleaves_with_other_pushes(t : T?) { ast_gc_guard() { var stmts : array @@ -71,7 +65,7 @@ def test_interleaves_with_other_pushes(t : T?) { // ── Pipe form ─────────────────────────────────────────────────────────── -[test, no_jit] +[test] def test_pipe_form(t : T?) { ast_gc_guard() { var stmts : array @@ -86,7 +80,7 @@ def test_pipe_form(t : T?) { // ── Null input is a silent no-op ──────────────────────────────────────── -[test, no_jit] +[test] def test_null_block_is_noop(t : T?) { ast_gc_guard() { var stmts : array diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index f373b3d03..186fbb0f7 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -104,13 +104,13 @@ foreach(util ${DAS_UTILS}) endforeach() add_dependencies(all_utils_exe dastest_exe) -# Prewarm the JIT dll cache for the whole test tree, except the folders dastest -# also excludes under -jit (gc, ast_match, ast, no_aot, quote — they use runtime quote/qmacro -# the JIT can't lower). Reuses the jit_exe target built by the foreach above (bin/jit.exe, built -# with --jit-register-all-modules). Run explicitly: +# Prewarm the JIT dll cache for the whole test tree, except module-gated folders +# (mirrors tests/.das_test availability checks; quote/qmacro folders prewarm fine now +# that jit_enabled triggers daslib/quote lowering). Reuses the jit_exe target built by +# the foreach above (bin/jit.exe, built with --jit-register-all-modules). Run explicitly: # ninja jit_cache_all_tests add_custom_target(jit_cache_all_tests - COMMAND ${UTIL_BIN_DIR}/jit.exe ${PROJECT_SOURCE_DIR}/tests --parallel 0 $,--jit-opt-level=0,--jit-opt-level=3> --exclude gc --exclude dasHV --exclude dasPUGIXML --exclude dasSQLITE --exclude stbimage --exclude live_host --exclude audio --exclude strudel --exclude ast_match --exclude ast --exclude no_aot --exclude quote + COMMAND ${UTIL_BIN_DIR}/jit.exe ${PROJECT_SOURCE_DIR}/tests --parallel 0 $,--jit-opt-level=0,--jit-opt-level=3> --exclude dasHV --exclude dasPUGIXML --exclude dasSQLITE --exclude stbimage --exclude live_host --exclude audio --exclude strudel DEPENDS jit_exe WORKING_DIRECTORY ${UTIL_BIN_DIR} COMMENT "JIT-compiling tests/ into the .jitted_scripts cache" diff --git a/utils/daScript/main.cpp b/utils/daScript/main.cpp index 64232c177..9a5ff29f9 100644 --- a/utils/daScript/main.cpp +++ b/utils/daScript/main.cpp @@ -286,8 +286,11 @@ int compile_and_run ( const string & fn, const string & mainFnName, bool outputP } policies.fail_on_lack_of_aot_export = false; policies.aot_macros = aotMacros; // -aot-macros: force quote lowering (daslib/quote) in a normal run - if ( aotMacros ) { - policies.stack = 1 * 1024 * 1024; // a lowered quote evaluates one large construction frame + if ( aotMacros || jitEnabled != JitMode::None ) { + // a lowered quote evaluates one large construction frame; jit_enabled triggers the + // same lowering, and the frame can evaluate at macro-apply time on a macro-context + // stack sized from these policies + policies.stack = 1 * 1024 * 1024; } policies.version_2_syntax = version2syntax; policies.gen2_make_syntax = gen2MakeSyntax; diff --git a/utils/jit/main.das b/utils/jit/main.das index 20b276763..966729b8b 100644 --- a/utils/jit/main.das +++ b/utils/jit/main.das @@ -5,10 +5,9 @@ require daslib/clargs require daslib/fio require daslib/jobque_boost require daslib/strings_boost -require math require llvm/daslib/llvm_jit_cli -require llvm/daslib/llvm_jit_run +require llvm/daslib/llvm_jit_run // nolint:STYLE030 registration side effect: bakes the native llvm module into a standalone -exe build (see simulate() comment below) [CommandLineArgs] struct JitToolArgs { @@ -256,6 +255,9 @@ def jit_compile_one(input : string; tool : JitToolArgs; jit : JitCliOptions) : b // can't emit IR for them -> "Failed to get IR for functions ..."). cop.aot_module = true cop.threadlock_context = true + // jit_enabled triggers quote lowering; lowered construction frames need + // the same stack bump the -aot-macros / dastest flows apply + cop.stack = 1024u * 1024u if (tool.exe) { cop.jit_exe_mode = true cop.jit_dll_mode = false @@ -287,7 +289,7 @@ def jit_compile_one(input : string; tool : JitToolArgs; jit : JitCliOptions) : b // marked no-jit correctly), which a post-simulate run_jit call // would miss. `require llvm_jit_run` above bakes the native llvm // module into a standalone -exe build of this tool. - simulate(program) $(sok, context, serrors) { + simulate(program) $(sok, _context, serrors) { if (!sok) { to_log(LOG_ERROR, "jit: failed to simulate {input}\n{serrors}\n") success = false From 51bb453d7a47a41a155e341a5fb4b33eb136ae69 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Fri, 12 Jun 2026 01:22:31 -0700 Subject: [PATCH 2/5] jit-lane stack headroom moves into makeMacroModule (fixes wasm_cross) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first cut bumped policies.stack to 1MB in the three jit drivers (daslang -jit, dastest -jit, jit.exe), mirroring the -aot-macros flow. But policies.stack also sizes the PRODUCED program's runtime context: cross-compiled wasm executables tried to carve 1MB out of wasm linear memory and trapped at runtime (CI wasm_cross, f2s.wasm "thrown Wasm exception"). Reverted the driver bumps. Program::makeMacroModule now sizes the macro context to max(getContextStackSize(), 1MB) when aot_macros or jit_enabled is set — the headroom lands exactly where lowered quotes evaluate at macro-apply time, for any embedder, without touching the produced program's stack. The -aot-macros global bump keeps its historical behavior. Full-tree dastest -jit re-verified green with the macro-context-only fix: 10071 tests, 10065 passed, 0 failed, 0 errors, 6 skipped. Co-Authored-By: Claude Fable 5 --- QUOTE_LOWERING.md | 20 ++++++++++++-------- dastest/dastest.das | 5 ----- dastest/suite.das | 5 ++--- src/ast/ast_simulate.cpp | 10 +++++++++- utils/daScript/main.cpp | 7 ++----- utils/jit/main.das | 3 --- 6 files changed, 25 insertions(+), 25 deletions(-) diff --git a/QUOTE_LOWERING.md b/QUOTE_LOWERING.md index 5651b20c5..4ec562d81 100644 --- a/QUOTE_LOWERING.md +++ b/QUOTE_LOWERING.md @@ -299,14 +299,18 @@ replacing the misleading "Internal jit error" panic). The QuotePass gate stays module-cache state at compile time — single-folder runs passed, full-tree root-form runs failed deterministically (bisect: my code at master folder set still failed; zero-predecessor root form still failed → not contamination at all, - a per-file stack margin). FIX (matches the `-aot-macros` design, which always - shipped with a 1MB stack): the jit lane applies the same bump — `daslang -jit` - (main.cpp), dastest under `-jit` (suite.das + dastest.das ser path), jit.exe - (utils/jit/main.das) — and macro-module programs inherit consumer policies, so - macro contexts get the headroom too. Known sharp edge for embedders: enabling - `policies.jit_enabled` without a stack bump can still overflow on a big quote - evaluated at macro time — durable fix is chunking the lowered construction into - bounded frames (Phase 6 candidate). + a per-file stack margin). FIX: `Program::makeMacroModule` sizes the MACRO context + to `max(getContextStackSize(), 1MB)` when `aot_macros || jit_enabled` — the + headroom lands exactly where lowered quotes evaluate at macro-apply time, for any + embedder, with no driver changes. First attempt was a `policies.stack = 1MB` bump + in the three jit drivers (main.cpp / dastest / jit.exe), mirroring the + `-aot-macros` flow — REGRESSION: `policies.stack` also sizes the PRODUCED + program's runtime context, so cross-compiled wasm executables tried to carve 1MB + out of wasm linear memory and trapped at runtime (CI `wasm_cross`, f2s.wasm + "thrown Wasm exception"). Reverted; the `-aot-macros` global bump keeps its + historical behavior. Durable fix remains chunking the lowered construction into + bounded frames (Phase 6 candidate) — that would let macro contexts drop the + special-casing entirely. - [ ] Decide + implement JIT-of-macro-contexts: lift the `is_compiling_macros()` skip in llvm_macro.das behind a policy. This is the compile-time payoff twin of AOT'd macro modules. Separate PR; needs its own bake time. diff --git a/dastest/dastest.das b/dastest/dastest.das index 086ac5831..49c77004d 100644 --- a/dastest/dastest.das +++ b/dastest/dastest.das @@ -377,11 +377,6 @@ def serialize_path(ctx : SuiteCtx, files : array, out_file : string) { cop.fail_on_no_aot = ctx.use_aot cop.threadlock_context = true cop.jit_enabled = jit_enabled() - if (jit_enabled()) { - // jit_enabled triggers quote lowering; lowered construction frames - // need the same stack bump suite.das applies (see suite.das) - cop.stack = 1024u * 1024u - } compile_file(file, access, unsafe(addr(mg)), cop) $(ok, program, output) { if (!ok) { log::error("Failed to compile {file}\n{output}") diff --git a/dastest/suite.das b/dastest/suite.das index 079e9ae79..1b8dbe6f5 100644 --- a/dastest/suite.das +++ b/dastest/suite.das @@ -161,10 +161,9 @@ def test_file(file_name : string; var ctx : SuiteCtx; var file_ctx : FileCtx) : cop.aot = ctx.use_aot cop.fail_on_no_aot = ctx.use_aot cop.aot_macros = ctx.aot_macros - if (ctx.aot_macros || jit_enabled()) { + if (ctx.aot_macros) { // a lowered quote evaluates one large make-struct construction frame - // (same bump the `daslang -aot -aot-macros` flow applies); jit_enabled - // triggers the same lowering, and macro contexts inherit this stack size + // (same bump the `daslang -aot -aot-macros` flow applies) cop.stack = 1024u * 1024u } cop.threadlock_context = true diff --git a/src/ast/ast_simulate.cpp b/src/ast/ast_simulate.cpp index effde64ea..94f10a447 100644 --- a/src/ast/ast_simulate.cpp +++ b/src/ast/ast_simulate.cpp @@ -3532,7 +3532,15 @@ namespace das void Program::makeMacroModule ( TextWriter & logs ) { isCompilingMacros = true; - thisModule->macroContext = get_context(getContextStackSize()); + int macroStackSize = getContextStackSize(); + if ( policies.aot_macros || policies.jit_enabled ) { + // quote lowering (daslib/quote) is active: a lowered quote evaluates one large + // construction frame per quote, and macro-called functions evaluate theirs on + // THIS context's stack at macro-apply time. Size only the macro context — a + // global policies.stack bump would leak into produced exe/wasm runtime stacks. + macroStackSize = das::max(macroStackSize, 1 * 1024 * 1024); + } + thisModule->macroContext = get_context(macroStackSize); thisModule->macroContext->category = uint32_t(das::ContextCategory::macro_context); auto oldAot = policies.aot; auto oldHeap = policies.persistent_heap; diff --git a/utils/daScript/main.cpp b/utils/daScript/main.cpp index 9a5ff29f9..64232c177 100644 --- a/utils/daScript/main.cpp +++ b/utils/daScript/main.cpp @@ -286,11 +286,8 @@ int compile_and_run ( const string & fn, const string & mainFnName, bool outputP } policies.fail_on_lack_of_aot_export = false; policies.aot_macros = aotMacros; // -aot-macros: force quote lowering (daslib/quote) in a normal run - if ( aotMacros || jitEnabled != JitMode::None ) { - // a lowered quote evaluates one large construction frame; jit_enabled triggers the - // same lowering, and the frame can evaluate at macro-apply time on a macro-context - // stack sized from these policies - policies.stack = 1 * 1024 * 1024; + if ( aotMacros ) { + policies.stack = 1 * 1024 * 1024; // a lowered quote evaluates one large construction frame } policies.version_2_syntax = version2syntax; policies.gen2_make_syntax = gen2MakeSyntax; diff --git a/utils/jit/main.das b/utils/jit/main.das index 966729b8b..ff556fc3b 100644 --- a/utils/jit/main.das +++ b/utils/jit/main.das @@ -255,9 +255,6 @@ def jit_compile_one(input : string; tool : JitToolArgs; jit : JitCliOptions) : b // can't emit IR for them -> "Failed to get IR for functions ..."). cop.aot_module = true cop.threadlock_context = true - // jit_enabled triggers quote lowering; lowered construction frames need - // the same stack bump the -aot-macros / dastest flows apply - cop.stack = 1024u * 1024u if (tool.exe) { cop.jit_exe_mode = true cop.jit_dll_mode = false From 3e025033e612310e2d788b9adb00681c2cfde094 Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Fri, 12 Jun 2026 01:40:13 -0700 Subject: [PATCH 3/5] review r1: quote-lowering diagnostics name both gates; .das_test doc de-jits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DisableJitVisitor warning and the codegen backstop said "without aot_macros lowering" — misleading in the jit lane, where jit_enabled is the usual trigger. Both now name the lowering generically with its gates (jit_enabled / aot_macros) and state that lowering didn't run for the defining module. skills/writing_tests.md no longer leads with "-jit / --use-aot" argv scanning for the .das_test filter — only --use-aot checks remain after the skip lift. Co-Authored-By: Claude Fable 5 --- modules/dasLLVM/daslib/llvm_jit.das | 8 ++++---- skills/writing_tests.md | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/modules/dasLLVM/daslib/llvm_jit.das b/modules/dasLLVM/daslib/llvm_jit.das index 952f05e8f..0ecb11409 100644 --- a/modules/dasLLVM/daslib/llvm_jit.das +++ b/modules/dasLLVM/daslib/llvm_jit.das @@ -3391,7 +3391,7 @@ class public LlvmJitVisitor : AstVisitor { // ExprQuote def override preVisitExprQuote(expr : ExprQuote?) : void { - unsupported(expr, "quote( ) is not jit-able without aot_macros lowering (daslib/quote)") + unsupported(expr, "raw quote( ) is not jit-able — daslib/quote lowering (jit_enabled / aot_macros gates) did not run for this module") } // ExprIfThenElse @@ -7647,9 +7647,9 @@ class public DisableJitVisitor : AstVisitor { def override preVisitExprQuote(expr : ExprQuote?) : void { // jit_enabled programs get quotes lowered by daslib/quote's QuotePass before we - // ever run; a raw one means lowering didn't fire (e.g. a context compiled without - // the policy) — skip the function instead of failing the whole file's codegen - to_log(LOG_WARNING, "LLVM JIT: quote( ) without aot_macros lowering at {expr.at}, falling back to interpreter.\n") + // ever run; a raw one means lowering didn't fire for the defining module (macro + // modules stay raw by design) — skip the function instead of failing the file + to_log(LOG_WARNING, "LLVM JIT: raw quote( ) — daslib/quote lowering (jit_enabled / aot_macros gates) did not run for this module at {expr.at}, falling back to interpreter.\n") disable = true } diff --git a/skills/writing_tests.md b/skills/writing_tests.md index 9b53ed68d..35ec23b65 100644 --- a/skills/writing_tests.md +++ b/skills/writing_tests.md @@ -33,9 +33,9 @@ false `error[50101]` / JIT failures. For AOT/JIT validation, sweep `--test tests `tests/.das_test` defines `can_visit_folder(folder_name, result)` — dastest consults it per subfolder during file collection (only for the `.das_test` at the `--test ` argument; directly naming a child folder bypasses it). It gates folders on module -availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv for `-jit` / -`--use-aot` (e.g. `ast`, `ast_match`, `no_aot` skip under `--use-aot`; the former `-jit` -skips were lifted once `jit_enabled` started triggering daslib/quote lowering). Two traps: +availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv — currently +only `--use-aot` checks remain (e.g. `ast`, `ast_match`, `no_aot`); the former `-jit` +skips were lifted once `jit_enabled` started triggering daslib/quote lowering. Two traps: a whole-folder JIT/AOT failure usually means a missing entry here, NOT a per-file fix; and the `jit_cache_all_tests` prewarm target (utils/CMakeLists.txt) does NOT consult it — its `--exclude` list mirrors the skips manually and must be updated in the same change. From 7a9c2eb76c6b4e6ee68996c589aaf34e7e1f7f1c Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Fri, 12 Jun 2026 01:50:32 -0700 Subject: [PATCH 4/5] review r2: macro-context stack bump also honors per-module options aot_macros makeMacroModule's 1MB headroom triggered on policies only; a macro module opting into lowering via `options aot_macros` would still lower its quotes and could overflow its macro context at apply time. The condition now mirrors the full QuotePass gate (same getBoolOption clause the ast_infer_type.cpp noAot-skip uses). Co-Authored-By: Claude Fable 5 --- src/ast/ast_simulate.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/ast/ast_simulate.cpp b/src/ast/ast_simulate.cpp index 94f10a447..5be767ce7 100644 --- a/src/ast/ast_simulate.cpp +++ b/src/ast/ast_simulate.cpp @@ -3533,8 +3533,9 @@ namespace das void Program::makeMacroModule ( TextWriter & logs ) { isCompilingMacros = true; int macroStackSize = getContextStackSize(); - if ( policies.aot_macros || policies.jit_enabled ) { - // quote lowering (daslib/quote) is active: a lowered quote evaluates one large + if ( policies.aot_macros || policies.jit_enabled || options.getBoolOption("aot_macros", false) ) { + // quote lowering (daslib/quote) is active (same triggers as its QuotePass gate, + // including the per-module option): a lowered quote evaluates one large // construction frame per quote, and macro-called functions evaluate theirs on // THIS context's stack at macro-apply time. Size only the macro context — a // global policies.stack bump would leak into produced exe/wasm runtime stacks. From 93cfca03e5900255ff47aaa650f7348e3a2db82a Mon Sep 17 00:00:00 2001 From: Boris Batkin Date: Fri, 12 Jun 2026 02:59:30 -0700 Subject: [PATCH 5/5] =?UTF-8?q?Debug=20CI:=20restore=20the=20gc=20folder?= =?UTF-8?q?=20-jit=20skip=20=E2=80=94=20GC=20can't=20see=20jitted-frame=20?= =?UTF-8?q?locals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Debug lanes caught what Release hid: heap_collect frees a heap object whose only reference is a local in a jitted frame (native-stack locals are invisible to the collector), so test_gc_escape_free's later `delete n` double-frees and trips the memory_model.h assert. Probe-proven independent of force_escape_free (interp keeps n.x=7 across collect+churn, jit reads the reused slot). That makes GC-semantics tests systemically unsound under JIT — restore the folder-level skip in tests/.das_test (root cause in the comment), mirror --exclude gc on jit_cache_all_tests, and revert the per-function [no_jit] markers the skip supersedes (tests/gc back to master state). The 3 linux-Debug typer_errors "failures" were collateral: the gc crash sent CI into the --isolated-mode fallback, whose 32 workers all cache-miss the same harness dll (`-jit -jit` hashes differently than the prewarmed outer run) and race writing one .o/.dll path — half-written-object, truncated-file, and post-write-verify assert. Pre-existing infra bug, noted in QUOTE_LOWERING.md; it heals here because the sweep no longer crashes into the fallback. skills/writing_tests.md updated: gc keeps its -jit skip; a green local Release sweep does not prove a lifted skip sound — Debug CI is the oracle. Co-Authored-By: Claude Fable 5 --- QUOTE_LOWERING.md | 35 +++++++++++++++++++------- skills/writing_tests.md | 12 ++++++--- tests/.das_test | 14 +++++++++++ tests/gc/test_gc_allocate_on_stack.das | 5 ---- tests/gc/test_gc_escape_free.das | 3 --- tests/gc/test_gc_escape_free_frees.das | 11 -------- utils/CMakeLists.txt | 11 ++++---- 7 files changed, 54 insertions(+), 37 deletions(-) diff --git a/QUOTE_LOWERING.md b/QUOTE_LOWERING.md index 4ec562d81..99e3a7fce 100644 --- a/QUOTE_LOWERING.md +++ b/QUOTE_LOWERING.md @@ -264,19 +264,36 @@ replacing the misleading "Internal jit error" panic). The QuotePass gate stays reaching JIT from such modules fall back per-function via the new DisableJitVisitor `preVisitExprQuote` (warn + disable, replacing whole-file panic). - [x] Coverage instead of new jit_tests: lifted the `-jit` folder skips (quote, ast, - ast_match, no_aot, gc) in tests/.das_test + the matching `--exclude`s on + ast_match, no_aot) in tests/.das_test + the matching `--exclude`s on jit_cache_all_tests, and un-marked `[no_jit]` in tests/template/test_push_block_list (its stated reason — JIT can't lower ExprQuote — is gone; qmacro_block now JITs). Folder results: quote 20/20, ast 1/1, ast_match 380/380 (2 graceful per-function fallbacks in daslib/ast_match — macro module, stays raw by design), flatten/no_aot - 14/14, gc 48/48, template 10/10. -- [x] NOT quote-related, surfaced while lifting the gc skip: **LLVM JIT does not implement - `force_escape_free` / `force_allocate_on_stack`** — three symptoms in tests/gc - (heap grows where interp stays flat; scope-exit free aborts "not a chunk pointer" - on JIT-allocated owning nodes under persistent_heap; nested `new Inner` make-struct - field arrives null). Scoped per-function `[no_jit]` markers on the workers in - test_gc_allocate_on_stack / test_gc_escape_free_frees / test_gc_escape_free keep the - rest of the folder JIT-covered. Upstream fix is its own work item. + 14/14, template 10/10. The gc skip was lifted too, then RESTORED — see next item. +- [x] NOT quote-related, surfaced (and re-buried) while lifting the gc skip: tests/gc is + unsound under JIT for two reasons. (1) **LLVM JIT does not implement + `force_escape_free` / `force_allocate_on_stack`** — heap grows where interp stays + flat; scope-exit free aborts "not a chunk pointer" on JIT-allocated owning nodes + under persistent_heap; nested `new Inner` make-struct field arrives null. (2) The + one Debug CI caught after the per-function `[no_jit]` round: **`heap_collect` cannot + see a heap pointer whose only reference is a local in a jitted frame** (native-stack + locals are invisible to the collector) — the object is freed as garbage, and the + test's later `delete` double-frees (Debug memory_model.h:109 assert; Release passes + by luck until the slot is reused — probe: interp prints n.x=7, jit prints n.x=-1, + with or without force_escape_free). (2) is systemic for GC-semantics tests, so the + folder-level `-jit` skip in tests/.das_test is restored (root cause in the comment) + and the scoped `[no_jit]` markers were reverted. Lift again only when the JIT spills + GC roots somewhere the collector can scan. Upstream fix is its own work item. +- [x] Pre-existing CI infra bug exposed by the gc crash (NOT this PR's doing, heals once + the gc skip is back): when the main `-jit` sweep crashes, build.yml falls back to + `--isolated-mode` — 32 worker subprocesses all spawn with `-jit -jit`, whose harness + dll hash differs from the outer run's prewarmed one → every worker cache-misses the + SAME dll path simultaneously and they race writing the same `.o`/`.dll`: "file + format not recognized" (half-written .o), "failed to set dynamic section sizes: + file truncated" (rewritten mid-link), and the llvm_jit_run.das:342 post-write + verify assert (dll swapped between write and reopen). Cost 3 collateral + typer_errors "failures" on linux Debug. Fix candidates: lock/atomic-rename in + write_dll, or prewarm the worker-flag-combination dll before fan-out. - [x] Local full-tree `-jit` sweep caveat (NOT this PR's doing): the in-process sweep aborts (exit 127, no diagnostic) at tests/language/table_operations.das `ta_test_lock_panic` — a deliberate table-lock panic inside jitted code fails to diff --git a/skills/writing_tests.md b/skills/writing_tests.md index 35ec23b65..aa2e8bd30 100644 --- a/skills/writing_tests.md +++ b/skills/writing_tests.md @@ -33,16 +33,20 @@ false `error[50101]` / JIT failures. For AOT/JIT validation, sweep `--test tests `tests/.das_test` defines `can_visit_folder(folder_name, result)` — dastest consults it per subfolder during file collection (only for the `.das_test` at the `--test ` argument; directly naming a child folder bypasses it). It gates folders on module -availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv — currently -only `--use-aot` checks remain (e.g. `ast`, `ast_match`, `no_aot`); the former `-jit` -skips were lifted once `jit_enabled` started triggering daslib/quote lowering. Two traps: +availability (`dasHV`, `dasSQLITE`, …) and on sweep mode by scanning argv — `--use-aot` +skips `ast`, `ast_match`, `no_aot`; `-jit` skips only `gc` (heap_collect can't see heap +pointers whose only reference is a local in a jitted frame — native-stack locals are +invisible to the collector, so GC-semantics tests are interp-only; the other former `-jit` +skips were lifted once `jit_enabled` started triggering daslib/quote lowering). Two traps: a whole-folder JIT/AOT failure usually means a missing entry here, NOT a per-file fix; and the `jit_cache_all_tests` prewarm target (utils/CMakeLists.txt) does NOT consult it — its `--exclude` list mirrors the skips manually and must be updated in the same change. Per-function `[no_jit]` is the finer-grained alternative when only some functions in a kept folder can't JIT — put it on the function whose CODE diverges under JIT, not just the `[test]` wrapper (JITted callees replace their SimNode bodies, so an interpreted wrapper -still calls jitted workers); see the `force_escape_free` workers in tests/gc. +still calls jitted workers). Beware Release-blind divergence: memory bugs (double-free, +reuse-after-collect) only trip the Debug memory_model.h assert, so a green local Release +sweep does NOT prove a lifted skip is sound — Debug CI is the oracle. ## Test file structure diff --git a/tests/.das_test b/tests/.das_test index af409948f..cf0422277 100644 --- a/tests/.das_test +++ b/tests/.das_test @@ -50,4 +50,18 @@ def can_visit_folder(folder_name : string; var result : bool?) { } } } + // GC tests are interp-only under JIT: jitted frames keep locals on the native stack, + // so heap_collect can't see a heap pointer whose only reference is a jitted local and + // frees it (double-free / reuse-after-collect; Debug memory_model.h assert catches it, + // Release passes by luck). Lift only once the JIT spills GC roots somewhere the + // collector can scan. + if (folder_name == "gc") { + let args <- get_command_line_arguments() + for (arg in args) { + if (arg == "-jit") { + *result = false + return + } + } + } } diff --git a/tests/gc/test_gc_allocate_on_stack.das b/tests/gc/test_gc_allocate_on_stack.das index 480460053..764778528 100644 --- a/tests/gc/test_gc_allocate_on_stack.das +++ b/tests/gc/test_gc_allocate_on_stack.das @@ -55,10 +55,6 @@ def sum_plain_new() : int { } // `new Node(x = .., y = ..)` - ExprAscend over make-local -> built directly in the frame -// no_jit: LLVM JIT doesn't implement force_allocate_on_stack for the ascend lowering — -// it heap-allocates (failing the flat-heap assert), and the owning variant's scope-exit -// free then aborts on the mismatched pointer -[no_jit] def sum_ascend_new() : int { var total = 0 for (i in range(N)) { @@ -69,7 +65,6 @@ def sum_ascend_new() : int { } // shell on the stack; the owned array is still on the heap and is freed at scope exit -[no_jit] // same force_allocate_on_stack gap as sum_ascend_new — crashes under JIT def sum_owning() : int { var total = 0 for (i in range(N)) { diff --git a/tests/gc/test_gc_escape_free.das b/tests/gc/test_gc_escape_free.das index 2f1165501..ae0894ecd 100644 --- a/tests/gc/test_gc_escape_free.das +++ b/tests/gc/test_gc_escape_free.das @@ -24,9 +24,6 @@ struct Node { struct Inner { vals : array } struct Outer { tag : int; child : Inner ? } -// no_jit: LLVM JIT doesn't implement force_escape_free — here the nested `new Inner` -// field comes back null (sibling files document the heap-grow / bad-free symptoms) -[no_jit] def churn_nested() : int { var acc = 0 for (i in range(500)) { diff --git a/tests/gc/test_gc_escape_free_frees.das b/tests/gc/test_gc_escape_free_frees.das index 1c7197472..25a834487 100644 --- a/tests/gc/test_gc_escape_free_frees.das +++ b/tests/gc/test_gc_escape_free_frees.das @@ -34,13 +34,7 @@ struct OwningNode { var g_list : array -// All workers carry [no_jit]: LLVM JIT doesn't implement force_escape_free — loops grow -// the heap (failing the flat-heap asserts) and the owning variant's scope-exit free aborts -// on the mismatched pointer ("not a chunk pointer"). The control is marked too so the -// grows/stays-flat comparison runs in one engine. - // every use of `p` is a field access -> p does not escape -> freed at scope exit -[no_jit] def sum_nonescaping() : int { var total = 0 for (i in range(N)) { @@ -51,7 +45,6 @@ def sum_nonescaping() : int { } // multi-variable let: each variable gets its OWN fresh allocation, so both are freed (no alias) -[no_jit] def sum_multi() : int { var total = 0 for (i in range(N)) { @@ -62,7 +55,6 @@ def sum_multi() : int { } // the free still runs in a function with try/recover (on the normal path) -[no_jit] def sum_with_try() : int { var total = 0 for (i in range(N)) { @@ -73,7 +65,6 @@ def sum_with_try() : int { } // passed only to a pure built-in (test_escape_pure: no side effects, scalar return) -> freed -[no_jit] def sum_pure_builtin() : int { var total = 0 for (i in range(N)) { @@ -84,7 +75,6 @@ def sum_pure_builtin() : int { } // non-escaping owner of a heap field; the raw collect frees the owned array too -[no_jit] def sum_owning() : int { var total = 0 for (i in range(N)) { @@ -94,7 +84,6 @@ def sum_owning() : int { return total } -[no_jit] def sum_escaping() : int { var total = 0 g_list |> reserve(length(g_list) + N) diff --git a/utils/CMakeLists.txt b/utils/CMakeLists.txt index 186fbb0f7..671ca4097 100644 --- a/utils/CMakeLists.txt +++ b/utils/CMakeLists.txt @@ -104,13 +104,14 @@ foreach(util ${DAS_UTILS}) endforeach() add_dependencies(all_utils_exe dastest_exe) -# Prewarm the JIT dll cache for the whole test tree, except module-gated folders -# (mirrors tests/.das_test availability checks; quote/qmacro folders prewarm fine now -# that jit_enabled triggers daslib/quote lowering). Reuses the jit_exe target built by -# the foreach above (bin/jit.exe, built with --jit-register-all-modules). Run explicitly: +# Prewarm the JIT dll cache for the whole test tree, except module-gated folders and gc +# (mirrors tests/.das_test: gc stays interp-only under -jit — heap_collect can't see +# jitted-frame locals; quote/qmacro folders prewarm fine now that jit_enabled triggers +# daslib/quote lowering). Reuses the jit_exe target built by the foreach above +# (bin/jit.exe, built with --jit-register-all-modules). Run explicitly: # ninja jit_cache_all_tests add_custom_target(jit_cache_all_tests - COMMAND ${UTIL_BIN_DIR}/jit.exe ${PROJECT_SOURCE_DIR}/tests --parallel 0 $,--jit-opt-level=0,--jit-opt-level=3> --exclude dasHV --exclude dasPUGIXML --exclude dasSQLITE --exclude stbimage --exclude live_host --exclude audio --exclude strudel + COMMAND ${UTIL_BIN_DIR}/jit.exe ${PROJECT_SOURCE_DIR}/tests --parallel 0 $,--jit-opt-level=0,--jit-opt-level=3> --exclude gc --exclude dasHV --exclude dasPUGIXML --exclude dasSQLITE --exclude stbimage --exclude live_host --exclude audio --exclude strudel DEPENDS jit_exe WORKING_DIRECTORY ${UTIL_BIN_DIR} COMMENT "JIT-compiling tests/ into the .jitted_scripts cache"