Skip to content

compiler: separate library for runtime#2380

Merged
borisbat merged 8 commits into
GaijinEntertainment:masterfrom
aleksisch:feature/split-library
Jun 11, 2026
Merged

compiler: separate library for runtime#2380
borisbat merged 8 commits into
GaijinEntertainment:masterfrom
aleksisch:feature/split-library

Conversation

@aleksisch

Copy link
Copy Markdown
Collaborator

No description provided.

@aleksisch aleksisch force-pushed the feature/split-library branch 8 times, most recently from 254404b to e916b6b Compare April 3, 2026 10:47
@aleksisch aleksisch changed the title [WIP] compiler: separate library for runtime compiler: separate library for runtime Apr 3, 2026
@aleksisch aleksisch force-pushed the feature/split-library branch from e916b6b to fc264b7 Compare April 3, 2026 11:13
@aleksisch aleksisch changed the title compiler: separate library for runtime [WIP] compiler: separate library for runtime Apr 3, 2026
@aleksisch aleksisch force-pushed the feature/split-library branch 17 times, most recently from ea9bb55 to 72bb0a2 Compare April 8, 2026 15:04
@aleksisch aleksisch force-pushed the feature/split-library branch from d8e8b27 to e799654 Compare April 15, 2026 14:55
@aleksisch aleksisch force-pushed the feature/split-library branch from e799654 to 3d5822c Compare June 2, 2026 14:14
@aleksisch aleksisch force-pushed the feature/split-library branch 8 times, most recently from 8831e8a to 5bb0940 Compare June 4, 2026 18:31
@aleksisch aleksisch force-pushed the feature/split-library branch from 5bb0940 to 2c8e0ce Compare June 10, 2026 22:30
@aleksisch aleksisch changed the title [WIP] compiler: separate library for runtime compiler: separate library for runtime Jun 10, 2026
@borisbat borisbat requested a review from Copilot June 10, 2026 23:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR restructures daScript’s build to separate a runtime-only library from the compiler library, moving compiler-only functionality (parsing/type inference/optimization/AST builtins) out of the runtime binary and updating exported APIs, tests, tutorials, and docs accordingly.

Changes:

  • Split build outputs into libDaScript_runtime (runtime) and libDaScript/libDaScriptDyn (compiler), relocating parser/formatter/infer/optimize/AST builtins into the compiler library.
  • Move “compile-from-source” builtins (compile, compile_file, make_file_access) from rtti_core into ast_core, and update .das tests/tutorials to require daslib/ast instead of daslib/rtti.
  • Adjust symbol visibility/export macros (DAS_API vs DAS_CC_API) and update various headers/functions for cross-library linking.

Reviewed changes

Copilot reviewed 55 out of 59 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
utils/daspkg/package_runner.das Switches package runner to require daslib/ast.
utils/dasFormatter/formatter.cpp Resets formatter state’s program pointer during init.
utils/dasFormatter/fmt.h Exports formatter CLI entrypoint via compiler-lib API macro.
tutorials/language/44_compile_and_run.das Updates tutorial requires to the AST module.
tutorials/integration/cpp/19_class_adapters.cpp Updates builtin module compilation call signature.
tutorials/integration/cpp/16_sandbox.cpp Updates .das_project sandbox FsFileAccess construction for new API.
tests/typer_errors/test_alias_message.das Updates requires from RTTI to AST.
tests/module_tests/test_modules.das Updates requires from RTTI to AST.
tests/language/reflection.das Updates requires and modernizes several block-call sites.
tests/jit_tests/jit_exe.das Updates requires and block-call syntax; refactors skip condition.
tests/jit_tests/dll_cache.das Updates requires and ignores unused callback vars.
tests/gc/gc_typedecl.das Updates requires and ignores unused compile result vars.
tests/flatten/test_flatten_errors.das Updates requires from RTTI to AST.
src/simulate/simulate_fn_hash.cpp Exports collectDependencies for cross-lib use.
src/simulate/fs_file_info.cpp Updates FsFileAccess ctor to accept a compiled Program.
src/hal/project_specific_file_info.cpp Removes old get_file_access; exposes project override getter for compiler side.
src/hal/performance_time.cpp Exports ref_time_delta_to_usec for compiler-lib consumers.
src/builtin/module_jit.cpp Removes AST builtin include dependency.
src/builtin/module_file_access.cpp Changes ModuleFileAccess ctor to consume an already-compiled Program.
src/builtin/module_builtin_rtti.cpp Moves adapter helpers and exports getCodeOfPolicyOptions; removes compile builtins from RTTI module.
src/builtin/module_builtin_fio.cpp Exports retry_pending_dynamic_modules for compiler side.
src/builtin/module_builtin_ast.cpp Hosts compile builtins + make_file_access in AST module (compiler side).
src/ast/dyn_modules.cpp Imports/export-adjusts retry hook for dynamic module scanning.
src/ast/ast.cpp Moves isLocalOrGlobal out of the general AST TU.
src/ast/ast_typedecl.cpp Exports aotSuffixNameEx.
src/ast/ast_simulate.cpp Moves stack-ref helper methods out of simulate TU.
src/ast/ast_program.cpp Removes Program::optimize method (now free function).
src/ast/ast_parse.cpp Adds compiler-side get_file_access, compileBuiltinModule, and switches to free inferTypes/optimizeProgram.
src/ast/ast_optimize.cpp Introduces optimizeProgram(Program*,...) implementation.
src/ast/ast_module.cpp Exports appendBuiltinModuleContent; removes Module::compileBuiltinModule method.
src/ast/ast_lint.cpp Imports getCodeOfPolicyOptions with export.
src/ast/ast_infer_type.cpp Refactors type inference entrypoints to free functions taking Program*.
src/ast/ast_infer_type_report.cpp Moves reportTrait implementation into infer-type reporting TU.
src/ast/ast_handle.cpp Removes duplicated reportTrait implementation.
src/ast/ast_allocate_stack.cpp Relocates setRefSp implementations for stack allocation helpers.
modules/dasUnitTest/test_handles.cpp Updates builtin module compilation call signature.
modules/dasLLVM/daslib/llvm_jit_run.das Bumps LLVM JIT cache version.
modules/dasLLVM/daslib/llvm_exe.das Updates “whole lib needed” logic for ast/network compiler modules.
include/daScript/simulate/fs_file_info.h Updates FsFileAccess ctor signature to accept Program.
include/daScript/simulate/debug_info.h Updates ModuleFileAccess ctor signature; forward-declares Program.
include/daScript/simulate/aot_builtin_ast.h Tweaks VisitorAdapter export declaration (compiler/runtime split).
include/daScript/misc/uric.h Exports Uri class for DLL builds.
include/daScript/misc/sysos.h Exports normalizeFileName.
include/daScript/misc/network.h Ensures platform macros are available; exports Server.
include/daScript/misc/lexer_alloc_track.h Exports lexer allocation tracker entrypoint for compiler lexer.
include/daScript/das_project_specific.h Exposes project-specific file access callback getter.
include/daScript/ast/dyn_modules.h Marks dynamic module require as compiler-lib API.
include/daScript/ast/ast.h Reassigns many APIs to runtime vs compiler export macros; moves compileBuiltinModule to free function; adjusts exports.
include/daScript/ast/ast_visitor.h Exports generateAstContext.
include/daScript/ast/ast_typedecl.h Exports several type utilities.
include/daScript/ast/ast_infer_type.h Adds declarations for free inferTypes entrypoints.
include/daScript/ast/ast_generate.h Re-marks numerous AST generation helpers as compiler-side exports or internal.
include/daScript/ast/ast_expressions.h Exports constant-expression helpers and MakeFieldDecl.
doc/source/stdlib/handmade/function-ast-make_file_access-0x3f72c00df56d8c61.rst Adds stdlib docs for make_file_access.
doc/source/stdlib/handmade/function-ast-compile-0xbea6fab989dcfbdf.rst Adds stdlib docs for compile.
doc/source/stdlib/handmade/function-ast-compile-0x97cb4ad1839fc0a.rst Adds stdlib docs for compile overload.
doc/source/stdlib/handmade/function-ast-compile_file-0x60781bcd3310a035.rst Adds stdlib docs for compile_file.
doc/reflections/das2rst.das Groups AST module docs for compilation/file access.
CMakeLists.txt Implements the runtime/compiler library split and realloc tracker override duplication for compiler libs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread utils/dasFormatter/fmt.h
Comment thread include/daScript/simulate/aot_builtin_ast.h Outdated
Comment thread tutorials/integration/cpp/16_sandbox.cpp Outdated
Comment thread tutorials/integration/cpp/16_sandbox.cpp Outdated
Comment thread tutorials/integration/cpp/16_sandbox.cpp Outdated
Comment thread include/daScript/misc/uric.h
Comment thread src/ast/ast_infer_type_report.cpp
@aleksisch aleksisch force-pushed the feature/split-library branch from 2c8e0ce to 8d68f46 Compare June 11, 2026 07:39
aleksisch and others added 8 commits June 11, 2026 11:42
dasFormatter's file-scope static State retained a ProgramPtr after
destroy(). The Program got torn down during C++ static destruction,
where ~ptr_ref_count locks the global ref_count_mutex — already
destroyed on macOS (cross-TU order) → EINVAL → abort.
Releasing the Program in destroy() keeps daslang objects from outliving the runtime.
Move the 6 ast_infer_type*.cpp and src/parser from the
runtime lib into the compiler lib.

Continues series of pull request to keep minimal runtime library.
dyn_modules.cpp calls compileDaScript (compiler) and its only caller
require_dynamic_modules is the exe/daScriptC, so it belongs in the
compiler lib. Moving it removes one runtime->compiler edge.

Export adjustments (macro matches where the symbol is defined):
- DAS_CC_API on require_dynamic_modules (now compiler-defined, called
  from the exe main.cpp)
- DAS_API on normalizeFileName (sysos) and retry_pending_dynamic_modules
  (module_builtin_fio), runtime symbols now called from compiler-side
  dyn_modules

Note: this alone does not turn the macOS build green - compileDaScript
and the other cross-lib symbols still have additional runtime callers
(module_file_access, module_builtin_ast, ast_program, ...). macOS
rejects those undefined dylib symbols where Linux defers them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the optimization-pass orchestration off the Program class into a
free function optimizeProgram(Program*, ...) in a new compiler source
file src/ast/ast_optimize.cpp. Sole caller (ast_parse.cpp, compiler)
updated to optimizeProgram(program.get(), ...).

Move ast_const_folding, ast_block_folding, ast_unused and ast_lint into
the compiler lib. Enabled by the prior optimizeProgram extraction: the
optimization* methods are now called only from compiler-side
optimizeProgram, and the PassVisitor/FoldingVisitor base (defined in
ast_const_folding) plus all its subclasses (block_folding, unused,
InferTypes) are now together in the compiler lib.
Move the ast-reflection module group (module_builtin_ast + serialize stays
runtime, flags, annotations*, adapters) and module_builtin_network to the
compiler lib. Relocate the compile-at-require path off the runtime lib:

- compileBuiltinModule: ast_module.cpp -> ast_parse.cpp (compiler);
  appendBuiltinModuleContent un-static'd + DAS_API so it stays runtime
- rtti_builtin_compile / _ex / _file + make_file_access: bindings and impls
  moved from the rtti module (runtime) to the ast module (compiler). These
  builtins now live in ast_core instead of rtti_core
- ModuleFileAccess / FsFileAccess ctors take a pre-compiled Program instead
  of calling compileDaScript; the compile moves to the caller
- get_file_access (the pak compiler) moved from HAL to ast_parse.cpp
  (compiler); the project-specific override stays in HAL, reached via the
  new get_project_specific_file_access() getter

Export tags follow "macro matches defining lib": DAS_CC_API on
compiler-defined symbols called from exe/runtime (compileBuiltinModule,
get_file_access); DAS_API on runtime symbols now called from the moved
compiler code (Server, getFunctionHash/getFunctionAotHash/getAotHashComment,
generateAstContext, collectDependencies, getProjectSpecificFileAccess,
appendBuiltinModuleContent).

Result: zero runtime->compiler edges. libDaScriptDyn(_runtime) shared and
the static daslang_static both link with 0 undefined. The runtime lib no
longer depends on the compiler lib (one-way compiler->runtime), which also
fixes the macOS dylib self-link and the static-archive circular dependency.

API note: compile / compile_file / make_file_access are now in the ast_core
module, not rtti_core. Callers reaching them via daslib/ast (or ast_boost)
are unaffected; any script requiring only rtti for these needs require ast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RelWithDebInfo leak tracking reported thousands of cross-module orphan
frees (and spurious "leaked lexer NAME tokens" / heap-leak dumps) during
AOT generation after the runtime/compiler library split.

DAS_TRACK_ALLOC propagates PUBLIC from libDaScript*_runtime to the
compiler libs, so the compiler binary compiles tracked code — but the
global operator new/delete override TU (alloc_tracker_overrides.cpp) was
only compiled into the runtime libs. On Windows each DLL resolves
operator new per-CRT, so libDaScriptDyn.dll's parser/infer/ast
allocations escaped the tracker and read as cross-module orphan frees.

Compile the override into the compiler libs too, mirroring the runtime
libs (1134-1135) and the dynamic-module macro (444). Static-archive
copies stay dormant (daslang_static provides the single definition), so
no duplicate-symbol risk.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
>=2 used = aggregation facade (daslib/ast = ast_core + rtti)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aleksisch aleksisch force-pushed the feature/split-library branch from 8d68f46 to f044fd0 Compare June 11, 2026 08:58
@borisbat borisbat merged commit c165bc1 into GaijinEntertainment:master Jun 11, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants