diff --git a/QUOTE_LOWERING.md b/QUOTE_LOWERING.md new file mode 100644 index 000000000..5249dd994 --- /dev/null +++ b/QUOTE_LOWERING.md @@ -0,0 +1,279 @@ +# Quote lowering: AOT + JIT support for `quote` / `qmacro` + +**Goal:** `ExprQuote` (and therefore the whole qmacro family) compiles under AOT and JIT. +Mechanism: revive the existing infer-time lowering in `daslib/quote.das` that replaces +`quote(tree)` with generated AST-reconstruction code (`ExprMakeStruct`/`ExprAscend`/…), +which is plain das code and compiles in all three tiers. + +## Decisions (settled) + +- **Revive, don't rewrite.** `d38cd728f` ("enable ast quote in aot", Jul 2025) is the right + design: reflection-driven converter walking the quoted tree via `BasicStructureAnnotation` + field enumeration + `walk_and_convert` for leaf data. No per-node emitter to maintain. +- **Stays in `daslib/quote.das`.** +- **Wiring = option (a):** static `require daslib/quote` from `daslib/templates_boost.das`. + Activation stays cop-driven via the existing `policies.aot_macros` check + (quote.das:410) — the pass is inert when the policy is off; `module quote shared` makes + the inert require near-free. (Rejected: root-level `addExtraModule` injection — extras land + on the program root only, but QuotePass must fire inside each macro module's own + compilation, where `[infer_macro]` visibility follows that module's require chain.) +- **Tests are mandatory before anything ships.** Basic suite first; lifting `no_aot`/jit + gates on existing tests is *extra* coverage afterwards, not a substitute. + +## Current-state map (verified 2026-06-11) + +- `ExprQuote` infers to `Expression?` handle; quoted subtree is never inferred + (`canVisitQuoteSubexpression` defaults false). Infer marks the enclosing function + `noAot` unless `policies.aot_macros` (src/ast/ast_infer_type.cpp:1440). +- Interpreter: `SimNode_AstGetExpression` holds a raw `Expression*` and returns + `expr->clone()` per evaluation (src/ast/ast_simulate.cpp:1280). +- qmacro expands to `apply_qmacro(quote(tree), rules-block)` (templates_boost.das:1179), + so lowering ExprQuote covers qmacro; the rules block is ordinary code. +- `daslib/quote.das`: `QuotePass` (`[infer_macro]` AstPassMacro) + `QuoteConverter` + replaces each ExprQuote via `convert_quote_to_expression`. Managed entities resolve + by name at runtime (`get_module` / `module_find_annotation` / `find_call_macro` / + `find_unique_function_ptr` / `module_find_structure` / `clone_file_info`). + **Orphaned:** nothing requires it (docs tooling only), zero tests. +- Lossiness blacklist (quote.das:151): `Function.annotations`, `EnumEntry.value/.at`, + `Enumeration.list`, `CaptureEntry.at`, `MakeFieldDecl.at`, `AnnotationArgument.at`; + `AnnotationArgumentInitData` carries only string/bool/int/float; FileInfo rebuilt by + name+tabSize (loses identity of `LineInfo.fileInfo`). +- AOT: `daslib/aot_cpp.das` has **no** ExprQuote handling — with `-aot-macros` set but + the pass not required, codegen emits garbage with no diagnostic. +- JIT: no `visitExprQuote` in `LlvmJitVisitor` → parent `getE` → `failed_E "unresolved + expression"` (llvm_jit.das:7017) → `g_errors` → `panic("Internal jit error…")` + (llvm_jit_run.das:295). Loud but misleading; all-or-nothing. +- JIT of macro contexts is separately disabled: `jit_llvm` simulate-macro skips when + `is_compiling_macros()` (llvm_macro.das:49). +- CLI: `daslang -aot … -aot-macros` sets `aot_macros` + `export_all` + 1MB stack + (utils/daScript/main.cpp:65). +- Policy-option promotion precedent: `policies.threadlock_context |= options.getBoolOption(…)` + (src/ast/ast_parse.cpp:1416). +- Blast radius: every in-tree quote user requires templates_boost (defer, decs_boost, + lpipe, static_let, typemacro_boost, interfaces, linq_fold_common, type_traits, …). + 41 test files use quote/qmacro (ast_match suite, linq_fold, 13× dasPEG, flatten/no_aot, + dasSQLITE/test_07_macros, fake_numeric, …), heavily overlapping the `no_aot`-gated set. + +## Phase 0 — empirical baseline (in progress) + +Worktree build (fresh master; both existing binaries are skewed vs today's +annotation-info merge), then: + +- [x] Interpreted probes: raw `quote`, `qmacro` incl. `$c` splice — green + (`SimNode_AstGetExpression` path intact). +- [x] Lowered probes (`options aot_macros`, interpreted): green after the bitrot fixes + below — consts, op2, calls, strings, `$c` splice, `qmacro_block`, `qmacro_type`, + make-struct, lambda. Probes in `probe_quote/` (become `tests/quote/` in Phase 3). +- [x] A/B: identical `describe_expression` output lowered vs unlowered (6 shapes). +- [x] `daslang -aot -aot-macros`: emits real reconstruction C++ + (`das_ascend_handle::make`, `clone_line_info`/`clone_file_info`), exit 0. +- [x] Reach probe: root-level `require daslib/quote` does NOT lower quotes inside a + required module (pass-macro visibility = the module's own require chain) — wiring + (b) rejected on evidence; with the templates_boost require the same module lowers. +- [x] Regression smoke (unlowered path + new require wiring): ast_match 380/380, + template 10/10, linq 1971/1971. + +### Bitrot fixed during Phase 0 + +1. **`apply_to_vec` gc_node spellings** (src/builtin/module_builtin_ast.cpp): dispatched + on smart_ptr-era `describe()` strings (`"ast::MakeStruct*"` — dead since the ast-module + removal). Fixed: one generic `tPointer` → `vector` branch (post-gc_node every + AST node vector is a plain raw-pointer vector); `"ast_core::MakeFieldDecl?"` keeps the + MakeStruct multiple-inheritance base-adjust special case. +2. **`convert_vec_expr` retarget stomp** (daslib/quote.das): `dst_tp != src_tp` was a + POINTER compare — always true after `clone_type` — so for pointer-element vectors the + code reinterpreted an `ExprAscend` as `ExprMakeStruct` and wrote `makeType` through the + wrong layout (memory stomp → gc_collect AV; survived 2025 only by layout luck). Fixed: + semantic compare (`describe(dst) != describe(src)`) + `arg_expr is ExprMakeStruct` guard. +3. **`cvt_to_mks` move_new** (daslib/quote.das): smart_ptr-era `move_new((*res)[i]) <| arg` + on what is now a raw-pointer slot. Fixed: plain assignment. + +## Phase 1 — plumbing + loud diagnostics (DONE 2026-06-11) + +- [x] `require daslib/quote public` in templates_boost (public: the generated + reconstruction calls — `cvt_to_mks`, `clone_line_info`, … — must resolve in the + qmacro-using module's scope). +- [x] `options aot_macros` per-module trigger: the option name is already valid + (CodeOfPolicies fields are auto-registered option names via + `getCodeOfPolicyOptions()`). QuotePass gate reads + `policies.aot_macros || prog options aot_macros`; the infer-time noAot-skip + (ast_infer_type.cpp) got the identical option clause — no parseDaScript + promotion needed. +- [x] QuotePass gate stays `aot_macros`-only (JIT extension deferred to Phase 5). +- [x] `aot_cpp.das` diagnostics — NOT a panic: a panic inside a make_visitor-driven + visitor is swallowed by `runMacroFunction`/`runWithCatch` (verified — error filed + on a program nobody reads post-compile, tool exits 0). Instead: `NoAotMarker` + gets `preVisitExprQuote` → marks the function `noAot` + `to_log(LOG_ERROR, …)` + naming file:line, cause, and fix (silent when the function is already noAot, i.e. + the normal no-aot_macros flow). Function falls back to interpreter; + `fail_on_lack_of_aot_export` escalates for exports. CppAot keeps a backstop + `visitExprQuote` that writes `#error` into the generated C++ (deterministic, + unlike panic). Verified on a negative probe: clean LOG_ERROR, broken emission gone. +- [x] `llvm_jit.das`: `preVisitExprQuote` → `unsupported(expr, "quote( ) is not jit-able + without aot_macros lowering (daslib/quote)")`, modeled on the try-catch case. + Lints clean; behavioral check needs an LLVM-enabled build (Phase 5). +- [x] Formatter verified + lint clean on all four changed das files (llvm_jit.das lint + needs `lib/LLVM.dll` present — dasbind resolves `getDasRoot()/lib/`). + +## Phase 2 — make the lowering correct (IN PROGRESS) + +### Done so far (2026-06-11) + +- **Suite-wide forcing flags**: `daslang -aot-macros