quote: revive aot_macros lowering — quote/qmacro now compile and run under AOT#3109
Merged
Conversation
…ing, loud diagnostics (phases 0-1) Revives daslib/quote.das (d38cd72) so quote()/qmacro lower to plain AST-reconstruction code under aot_macros, making them AOT-able. - apply_to_vec: replace dead smart_ptr-era describe() cases with a generic tPointer -> vector<void*> branch (all AST node vectors are raw-pointer vectors post-gc_node); keep MakeStruct base-adjust for MakeFieldDecl? - quote.das convert_vec_expr: dst_tp != src_tp was a pointer compare (always true after clone_type), reinterpreting ExprAscend as ExprMakeStruct and stomping memory; now describe-compare + is-ExprMakeStruct guard - quote.das cvt_to_mks: move_new on a raw-pointer slot -> plain assignment - templates_boost: require daslib/quote public (QuotePass fires for every qmacro user; public so generated reconstruction calls resolve there) - gate: policies.aot_macros || per-module `options aot_macros` (same clause in the infer noAot-skip), enabling self-contained interpreted A/B tests - aot_cpp: NoAotMarker excludes functions with surviving quotes + LOG_ERROR (panic in make_visitor visitors is swallowed by runMacroFunction); CppAot writes #error into generated C++ as backstop - llvm_jit: preVisitExprQuote -> clean `unsupported` diagnostic Verified: lowered-vs-unlowered describe output identical across 6 shapes; -aot -aot-macros emits reconstruction C++; ast_match 380/380, template 10/10, linq 1971/1971 with the new wiring. Plan: QUOTE_LOWERING.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…te (phases 2-3)
- QuoteConverter wraps each lowered construction in a generated
`quote`lowered`N private function and replaces the quote with a call.
The inline form inflated every CALLER's stack frame by the sum of
construction temporaries — fatal in recursive macros (flatten's
lower_stmt overflowed any reasonable stack); now the big frame is a
single non-recursive leaf
- audit hammer: `daslang -aot-macros <script>` (normal run) and
`dastest --aot-macros` (suite.das cop + 1MB stack) force lowering
program-wide. FULL tests/ tree lowered: 10103 tests, identical verdict
to the unlowered baseline
- anonymous-module entities: quoted type<LocalEnum>/type<LocalStruct>
generated get_module("") -> null deref. Enumeration joins the managed
by-name set (new builtin module_find_enumeration); TypeDecls referencing
anonymous-module enums/structs reconstruct as make_alias_type_decl
(the parser's unresolved-name alias shape, re-inferred at splice site)
- resolve_file_info: per-file interned dummy FileInfo per context instead
of a fresh allocation per LineInfo per evaluation
- QuotePass gate also honors per-module `options aot_macros` (CodeOfPolicies
field names are auto-registered option names), enabling self-contained
interpreted A/B tests
- tests/quote/: twin fixture modules with identical bodies (SimNode path vs
lowered) compared per shape by describe output — consts/escapes/non-ASCII,
calls, splices, blocks+finally, make-struct, named-module enum/struct types,
nested container types, table/tuple literals, capture lambdas, generators,
qmacro_block_to_array — plus anonymous-main-module alias coverage. 20/20.
Registered for AOT (test_aot_quote, test_aot_quote_modules)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(phase 4) - module_find_enumeration gets its DAS_CC_API declaration in aot_builtin_ast.h (AOT emits direct C++ calls to bound builtins) - blacklist ExprBlock.stackCleanVars in the reconstruction: post-infer allocateStack bookkeeping, always empty in quoted trees, and its vector<pair<uint32,uint32>> type has no AOT mapping - dastest suite.das: cop.stack literal was int into uint (lint error); STYLE027/STYLE030 cleanups in the quote fixtures (comprehension form, nolint for require-only-used-inside-quotes) - tests/README.md: quote/ section Verified with test_aot (fail_on_no_aot active, so linkage is real): tests/quote 20/20, tests/aot 134/134, full tests/ tree 10123 tests / 10117 passed / 0 failed / 6 skipped — identical skip set to the interpreter baseline. Formatter + lint clean across all touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lowered reconstruction bakes the quoted file's path as a string constant (FileInfoInitData.name). dasAot generation and runtime can open the same file via different spellings (D:/fwd vs d:\back) — different constant, different own-hash, error[50101] on every `quote`lowered`N. Caught by the CI-form invocation (-use-aot ... --use-aot); the flagless test_aot run does not enable AOT linking and had masked it. normalize_source_path (separators to '/', drive letter uppercased) at the point the constant enters the tree — the documented fix for this mismatch class. CI-form full tree under test_aot: 9439 tests, 9433 passed, 0 failed, 0 errors, 6 skipped. Docs: das2rst groups for resolve_file_info / make_alias_type_decl / module_find_enumeration + handmade description for the new builtin. das2rst clean (no stubs, no Uncategorized); sphinx latex+html clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR revives and hardens daslib/quote.das quote/qmacro lowering so ExprQuote can be replaced with plain AST-reconstruction code that compiles and runs under AOT (with improved diagnostics for AOT/JIT). It also adds a focused A/B test suite to ensure lowered vs interpreter quote output matches.
Changes:
- Add
aot_macrosgating (policy + per-module option) and wiring (require daslib/quote publicvia templates_boost) so quote lowering runs in macro modules and resolves generated reconstruction calls. - Fix/refactor AST reflection helpers and add new builtin
module_find_enumerationto support reconstructing enum references by name. - Add
tests/quote/A/B fixtures + AOT registration, plus diagnostics for survivingquote()during AOT and clearer JIT “unsupported” reporting.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/daScript/main.cpp | Adds -aot-macros flag in normal runs and bumps stack when enabled. |
| tests/README.md | Documents the new tests/quote/ suite. |
| tests/quote/test_quote_lowering.das | New A/B equivalence test comparing lowered vs SimNode quote output. |
| tests/quote/test_quote_lowered_main.das | New tests covering anonymous-main-module entity reconstruction and splice substitution under lowering. |
| tests/quote/_quote_shapes.das | Reference (unlowered) quote shape fixture module. |
| tests/quote/_quote_shapes_lowered.das | Lowered twin of _quote_shapes using options aot_macros. |
| tests/quote/_quote_entities.das | Shared named-module enum/struct for stable describe output in A/B comparisons. |
| tests/aot/CMakeLists.txt | Registers quote tests for AOT and AOT-lib generation of fixture modules. |
| src/builtin/module_builtin_ast.cpp | Fixes/refactors vector reflection dispatch; adds module_find_enumeration builtin binding. |
| src/ast/ast_infer_type.cpp | Aligns noAot marking for quote() with the aot_macros policy/option gate. |
| QUOTE_LOWERING.md | Design/worklog for quote lowering implementation and rollout plan. |
| modules/dasLLVM/daslib/llvm_jit.das | Emits a clearer unsupported diagnostic for ExprQuote under LLVM JIT. |
| include/daScript/simulate/aot_builtin_ast.h | Adds AOT declaration for module_find_enumeration. |
| doc/source/stdlib/handmade/function-ast-module_find_enumeration-0xb93136d08d96a621.rst | Documents module_find_enumeration. |
| doc/reflections/das2rst.das | Updates doc grouping to include module_find_enumeration and new quote helpers. |
| dastest/suite.das | Adds --aot-macros passthrough to compilation policies and stack bump when forced. |
| dastest/dastest_clargs.das | Adds CLI flag for forcing quote/qmacro lowering in test compilation. |
| daslib/templates_boost.das | Requires daslib/quote public so lowering and reconstruction helpers are visible to macro modules. |
| daslib/quote.das | Core lowering fixes: move_new removal, semantic type compare guard, FileInfo interning/normalization, alias fallback, per-quote generated functions, and new enum support. |
| daslib/aot_cpp.das | Adds deterministic diagnostics for surviving quote() under aot_macros (noAot marker + #error backstop). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ed paths; Copilot fixes CI fix: tests/quote joins the .das_test -jit folder-skip family (raw-quote A/B fixture can't JIT-codegen until the Phase 5 lowering lane; AOT coverage stays on) + jit_cache_all_tests prewarm gets --exclude quote (its list manually mirrors .das_test). Drops a pre-existing unused 'require strings' the lint flagged. Hash fix: normalize_source_path now canonicalizes baked source paths to dasroot-relative (trim_prefix(get_das_root()) after slash/drive normalization) -- dasAot gets dasroot-absolute paths from CMake while a runtime '--test tests/...' compile is repo-relative, which desynced the lowered functions' own-hash (error 50101). Baked constants are now machine-independent as a side effect. Copilot round 1: resolve_file_info interns per (name, tabSize) instead of name only; apply_to_vec keeps the exact-type vector<const char*> cast for the string case (the generic vector<void*> pun is for AST pointers only); QUOTE_LOWERING.md line-wrap typo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…contract The shared A/B fixture header claimed "interpreter SimNode path" verbatim in BOTH twins; now describes both roles so the byte-identical-except-two-lines discipline keeps it accurate in each file. (Re-learned: even comment-only edits to a lowered fixture shift baked LineInfo constants -> stub purge + rebuild; recorded in skills/aot_testing.md.) module_find_enumeration handmade doc now states the non-null module precondition alongside the returns-null-when-not-found behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Revives
daslib/quote.das(d38cd72, orphaned since Jul 2025) soquote(...)— and therefore the whole qmacro family — lowers underaot_macrosinto plain AST-reconstruction code that compiles in all execution tiers. Plan/worklog:QUOTE_LOWERING.md.What was broken
apply_to_vec(C++ reflection helper) still dispatched on smart_ptr-eradescribe()spellings ("ast::MakeStruct*") — fatal on the very firstargumentsvector. Replaced with one generictPointer → vector<void*>branch (post-gc_node every AST node vector is a raw-pointer vector); theMakeFieldDecl?multiple-inheritance base-adjust stays as the lone special case.convert_vec_expr'sdst_tp != src_tpwas a pointer compare — always true afterclone_type— so pointer-element vectors had anExprAscendreinterpreted asExprMakeStructandmakeTypewritten through the wrong layout (gc_collect AV; survived 2025 only by layout luck). Now a semantic compare +is ExprMakeStructguard.cvt_to_mksused smart_ptr-eramove_newon a raw-pointer slot → plain assignment.Design changes
`quote`lowered`Nprivate function + a call. The inline form inflated every caller's frame by the sum of construction temporaries — fatal in recursive macros (flatten'slower_stmt). 1MB macro-context stack (the bump the-aot -aot-macrosflow already used) then suffices everywhere.require daslib/quote publicin templates_boost — pass-macro visibility follows each module's own require chain (verified empirically; root-level injection does not reach macro-module compiles), andpubliclets the generated reconstruction calls resolve in qmacro-using modules.policies.aot_macrosor per-moduleoptions aot_macros(CodeOfPolicies field names are auto-valid option names) — the latter enables self-contained interpreted A/B tests. Infer's noAot-skip uses the same condition.type<LocalEnum>/type<LocalStruct>used to emitget_module("")→ null deref.Enumerationjoins the managed by-name set (new builtinmodule_find_enumeration+ AOT decl), and TypeDecls referencing anonymous-module entities reconstruct asmake_alias_type_decl("<name>")— the parser's unresolved-name shape, re-resolved at splice-site infer.resolve_file_info: per-file interned dummy FileInfo per context (was: fresh allocation per LineInfo per evaluation). Baked source paths are normalized (separators, drive-letter case) so dasAot-generation and runtime spellings of the same file hash identically — without this, every lowered function failed AOT linking (error 50101) under the CI invocation.NoAotMarkerexcludes functions with surviving quotes + LOG_ERROR naming file/cause/fix (a panic inside make_visitor visitors is swallowed byrunMacroFunction); CppAot writes#errorinto the generated C++ as a deterministic backstop; llvm_jit gets a cleanunsupportedmessage instead of the misleading "Internal jit error" panic.daslang -aot-macros <script>anddastest --aot-macrosforce lowering program-wide for A/B validation.ExprBlock.stackCleanVars— post-infer allocateStack bookkeeping, always empty in quoted trees, no AOT type mapping.Tests
tests/quote/: twin fixture modules with identical bodies (SimNode path vs lowered) compared shape-by-shape via describe output — consts/escapes/non-ASCII, calls, splices, blocks+finally, make-struct, enum/struct/container types, table/tuple literals, capture lambdas, generators,qmacro_block_to_array— plus anonymous-main-module alias coverage. Registered for AOT (test_aot_quote+test_aot_quote_modules).Validation
test_aot -use-aot … --use-aotfull treeJIT support is deliberately out of scope (follow-up phase per
QUOTE_LOWERING.md); the only JIT change here is the diagnostic. Themodules/dasLLVM/daslib/llvm_jit.dasedit is lint/compile-verified; no current CI JIT lane reaches a quote (functions containing them are noAot/interpreted until the follow-up).🤖 Generated with Claude Code